Can AI Write Code? My Experiments with Developer AI Tools

AI | Design & Development

Artificial Intelligence, AI, is the buzzword that permeates advertising these days.  Suddenly all products must have AI, whether anyone asked for it or not.  But what are we talking about here?  AI is a broad term that encompasses many areas, including Machine Learning (ML) and Natural Language Processing (NLP), which is what most advertisers are actually referring to.  

Over the past year or so, as these technologies have gained popularity, there has been growing talk about how it will affect various industries, particularly “knowledge workers”, such as developers.  Some claim AI will reduce the need for professionals, such as accountants and software developers (gasp).  While others laud the potential for increased productivity of those same professionals.  I don’t think I will be able to determine if the end is near or if I will suddenly start cranking out websites in a fraction of the time, but I do want to get a sense of where we are with the technology and see ways I can start leveraging “AI” at work.

My hypothesis is that AI is like other advanced tools, really powerful, but you still need a skilled operator.  

I briefly played around with a chatbot and found it interesting.  One of the requests I made was to generate a unit test for a well-known (and thoroughly tested) Drupal module.  The results were pretty promising, the test looked a lot like the existing test provided by the module.  But then, that shouldn’t be surprising since the module itself is open source, and the existing tests are readily available on the Web.  It’s pretty well known that Large Language Models (LLMs) use open-source code repositories for training.  This isn’t an ethics discussion, so we’ll leave the training of these tools aside for now.  Notably, along with the test, ChatGPT provided some generic instructions on how to configure PHPUnit and run this particular test.  A novice developer could probably ask enough questions to eventually get the test to run.  For a seasoned developer, it definitely saves time.

For my next experiment, I tested the AI Assistant integrated into my IDE.  This time, instead of testing public code, I requested a test of some code I was working on.  Again, the resultant test seems reasonable (@todo runs the tests), but actually implementing the tests is not trivial.


Next, I wanted to try out some of the other features.  There is a “Generate Code” feature, where a developer can interactively request some code.  I removed a block of code and explained to the prompt what I wanted it to write.  A few things struck me.

  1. To get the results I wanted, it took more time than writing the original code.  It took several attempts to craft a prompt that was precise enough to accomplish my goal.
  2. The final block of code was almost exactly what I had written.  It was actually a bit tidier than my original.
  3. The code insertion was pretty clumsy.  The way the AI injected the code, it messed up some of the existing code.  This is either a bug or more likely, my inexperience with the tool.

I tested out a few other features, including the Explain Code, and Find Problems. Pretty much everything worked reasonably well.  While this isn’t a product review, and my experience is still pretty limited, I don’t think the tools are quite refined enough yet to be something I use on a daily basis.  I think at this point, we can’t rely on AI for development alone, an experienced developer should review what is being generated.  

However, I can see the growing potential.  Many times, development involves a lot of “boilerplate”, and if an AI tool can do that for us, and we focus on the important business logic, there is definitely efficiency to be gained.  Likewise, a new developer can get the benefit of having AI explain code so they can level up quicker.  I think the risk is if we expect an inexperienced developer to produce complex code, and they don’t have the knowledge to audit what the AI produces effectively.