I’ll not be the first to say it: AI sucks. I’ve been messing around with neural nets since 2001, and they are, generally, pretty terrible things. They can be beneficial in some endeavors but aren’t that smart. The limitations are the same, too, only shrinking marginally from a practical standpoint in over 20 years. Yet, since the birth of ChatGPT, people have raved over it, and I finally decided to give it a serious chance.
Firstly, I was invited to GitHub’s CoPilot software a while ago. I installed the plugin into my IDE and got more than a few good laughs at its suggestions. In all the months I trialed it, I think it only gave suitable suggestions when doing what could be done with copy/paste and some logic. Occasionally, it would offer up some exciting things when I figured out that I could write a comment, and it would use the comment to fill in the following few blocks of code. It was still subtly wrong enough times that I couldn’t take it seriously.
My experience with CoPilot left me amused but unconvinced about its utility, yet it paved the way to trying other tools. I eventually uninstalled it and installed Jetbrain’s AI assistant at the beginning of my winter break a few weeks ago. I was writing a lot of Go and C during this time, investigating a memory leak in FrankenPHP. I would have many conversations with the AI Assistant about the code I was reading, suggestions for improvement, and some back-and-forth about why its suggestion was wrong.
JetBrains AI Assistant
It was at this moment that I truly understood the power of AI. It was like I was talking to an inverse me — I deeply understood the program and the goals but was unfamiliar with the language. At the same time, the AI was deeply familiar with the language but had issues grasping the program and the goals I was trying to achieve. It was this balance that I found quite powerful. I almost felt like I was on a team where I had to explain a concept in several ways for others to get it right. Sometimes, the simple act of explaining it and someone not “getting it” led me to search for better ways to explain what I was doing and led to better discoveries.
I eventually discovered the source of the memory leak, with very little “actual” help from the AI Assistant, though having a rubber duck that talked back was immensely helpful.
One of the fantastic features of AI Assistant was the code review and refactoring tools. I’ve mostly only played with these features so far; however, it made compelling suggestions on code I had no reason to change (yet).
Since I’m writing my framework for giggles, I let it refactor some code that uses it. It butchered the code to hell and back. Quickly enough, I could write a prompt informing it of the codebase’s coding conventions, and it could apply refactorings correctly.
Finally, as a birthday present to myself, I paid for a month of ChatGPT. So far, I’ve talked to ChatGPT about ideas and research on other unfinished blog posts. It’s like having a dumb assistant that aims to tell me what I want to hear. Getting it to say bad or negative news has been challenging, though I think I’ve gotten a decent prompt that sometimes gets it okay – at least, it is much more critical now.
Will I pay for another month of ChatGPT? It is unlikely. The utility of ChatGPT isn’t that great when it comes to more nuanced things (which is what I am trying to use it for). However, I suspect (like most neural networks) it is excellent for what it was trained to do … have conversations. It always expects a back-and-forth conversation and will do anything to continue it. If you can do anything in the context of a conversation, ChatGPT can likely help tremendously. Still, as a resource of knowledge and applying that knowledge … it has massive shortcomings because that isn’t what it was trained to be.
Skeptical but Amused
My experience has been a mixed bag. While I was initially skeptical and amused by AI’s suggestions, I have come to appreciate the unique value they provide. Despite its shortcomings in understanding the full scope of a project, the AI Assistant proved to be a valuable counterpart in coding, offering a different perspective that spurred deeper understanding and problem-solving. The refactoring tools, though imperfect, demonstrated potential in aiding code development. ChatGPT, while limited in its ability to handle nuanced tasks beyond simple conversations, still serves as a helpful resource for idea generation and discussion. Ultimately, these AI tools are not replacements for human expertise but a supplement that can offer a different viewpoint, encourage exploration, and occasionally assist in unexpected ways. The future of AI in programming is promising, yet it is clear that its role is to augment, not replace, the human element in the creative process. Yet.