We have using GitHub Copilot probably for 2 years now. First used as a tool for code view, and then expanded as plug-in inside the IDE (both IntelliJ and VS Code). We can ask all kinds of questions in the conversation dialog. Personally I used it to generate unit test code a few months ago.
While the tool is helpful both in code review and in software (or application) development in general, recently I also noticed two issues, both are annoying. And it seems the issues are somewhat related to what Martin Fowler said in his recent blog post “I don’t like LLMs“.
Issue one: it gave totally irrelevant code review comments. I don’t know where the comments originated from, but the tone it used again reminds me of the Mr. Fowler’s post. For me the only solution is add my comments in the Copilot code review comments and move on. It’s not the only time Copilot code review provided this kind of comments. The overall tone from Copilot is
Issue two: this one honestly stumbled upon me, and took me a while to understand the rationale of Copilot’s mistake. I asked Copilot to fix an issue of “unnecessary mockito stubbing” during unit testing. Note the code was generated by Copilot, and what it does was: to remove 2 lines of code which is not supposed to remove, while the correct solution is to remove the same 2 lines of code in a similar function. The main difference between the 2 functions here is one fucntion is with togglz off, while the other is with togglz on. It kept doing that for a few times, until I realized I can manually do it by removing those two lines in the correct function. I think in this case Copilot still is helpful to me – but this small incident reinforced my thinking of “trust but verify” for AI generated code.
At the same time, I read a bit of “When AI writes almost all code, what happens to software engineering?” (Substack link, I came across it from LinkedIn) by Pragmatic Programmer aka Gergely Orosz. It sounds like he is more bullish on AI or AI generating code. Note the Substack article was written in last Nov or Dec.
Another related observation is, I notice sometimes colleagues would say: let AI do the algorithm or something like that (the implied meaning seems to me is we are not as smart as AI).
One counter argument is: the big tech companies are the ones leading the charge using AI to generating code, but at the same time their code quality sometimes are horrendous, such as the Uber Driver App – e.g., this personal experience I had when I was delivering for Uber Eats.
Last but not least let me share a short talk I came across from my friend and a professor in management science at my alma mater Huazhong University of Science and Technology, as shown below. In Chinese. I assume you can ask ChatGPT or Google Gemini for translations if needed.
