How I use AI

I’ll not be the first to say it: AI sucks. I’ve been messing around with neural nets since 2001, and they are, generally, pretty terrible things. They can be beneficial in some endeavors but aren’t that smart. The limitations are the same, too, only shrinking marginally from a practical standpoint in over 20 years. Yet, since the birth of ChatGPT, people have raved over it, and I finally decided to give it a serious chance.

GitHub CoPilot

Firstly, I was invited to GitHub’s CoPilot software a while ago. I installed the plugin into my IDE and got more than a few good laughs at its suggestions. In all the months I trialed it, I think it only gave suitable suggestions when doing what could be done with copy/paste and some logic. Occasionally, it would offer up some exciting things when I figured out that I could write a comment, and it would use the comment to fill in the following few blocks of code. It was still subtly wrong enough times that I couldn’t take it seriously.

My experience with CoPilot left me amused but unconvinced about its utility, yet it paved the way to trying other tools. I eventually uninstalled it and installed Jetbrain’s AI assistant at the beginning of my winter break a few weeks ago. I was writing a lot of Go and C during this time, investigating a memory leak in FrankenPHP. I would have many conversations with the AI Assistant about the code I was reading, suggestions for improvement, and some back-and-forth about why its suggestion was wrong.

JetBrains AI Assistant

It was at this moment that I truly understood the power of AI. It was like I was talking to an inverse me — I deeply understood the program and the goals but was unfamiliar with the language. At the same time, the AI was deeply familiar with the language but had issues grasping the program and the goals I was trying to achieve. It was this balance that I found quite powerful. I almost felt like I was on a team where I had to explain a concept in several ways for others to get it right. Sometimes, the simple act of explaining it and someone not “getting it” led me to search for better ways to explain what I was doing and led to better discoveries.

I eventually discovered the source of the memory leak, with very little “actual” help from the AI Assistant, though having a rubber duck that talked back was immensely helpful.

One of the fantastic features of AI Assistant was the code review and refactoring tools. I’ve mostly only played with these features so far; however, it made compelling suggestions on code I had no reason to change (yet).

Since I’m writing my framework for giggles, I let it refactor some code that uses it. It butchered the code to hell and back. Quickly enough, I could write a prompt informing it of the codebase’s coding conventions, and it could apply refactorings correctly.

ChatGPT

Finally, as a birthday present to myself, I paid for a month of ChatGPT. So far, I’ve talked to ChatGPT about ideas and research on other unfinished blog posts. It’s like having a dumb assistant that aims to tell me what I want to hear. Getting it to say bad or negative news has been challenging, though I think I’ve gotten a decent prompt that sometimes gets it okay – at least, it is much more critical now.

Will I pay for another month of ChatGPT? It is unlikely. The utility of ChatGPT isn’t that great when it comes to more nuanced things (which is what I am trying to use it for). However, I suspect (like most neural networks) it is excellent for what it was trained to do … have conversations. It always expects a back-and-forth conversation and will do anything to continue it. If you can do anything in the context of a conversation, ChatGPT can likely help tremendously. Still, as a resource of knowledge and applying that knowledge … it has massive shortcomings because that isn’t what it was trained to be.

Skeptical but Amused

My experience has been a mixed bag. While I was initially skeptical and amused by AI’s suggestions, I have come to appreciate the unique value they provide. Despite its shortcomings in understanding the full scope of a project, the AI Assistant proved to be a valuable counterpart in coding, offering a different perspective that spurred deeper understanding and problem-solving. The refactoring tools, though imperfect, demonstrated potential in aiding code development. ChatGPT, while limited in its ability to handle nuanced tasks beyond simple conversations, still serves as a helpful resource for idea generation and discussion. Ultimately, these AI tools are not replacements for human expertise but a supplement that can offer a different viewpoint, encourage exploration, and occasionally assist in unexpected ways. The future of AI in programming is promising, yet it is clear that its role is to augment, not replace, the human element in the creative process. Yet.

Want an inside scoop?

Check out PHP Shenanigans for only €5/mo or €30/yr


Comments

8 responses to “How I use AI”

  1. Good post bro. I agree for the most part. I’ve found GitHub Copilot to be the most useful- but like Google searching was back in the day- you have to learn how to use it well. Remember when people used to brag about their google-fu?

    I’ve been using ChatGPT Premium for a bit as well- experimenting with where that is helpful. Pretty similar story overall- you gotta learn to work with it’s strengths and against its weaknesses. Just like any tool really- wouldn’t hammer a nail with a screwdriver or something. Bad analogy but you see what I’m saying.

    I think what bugs me a little about the general public conversation around AI is how focused it is on either the downstream economic impacts or all the potential problems it could cause when it’s wrong etc. I think those are certainly worthy points to engage with- and I definitely wouldn’t want anyone to confuse this next paragraph with any disagreement with those statements- heck, I don’t know what’s going to happen anymore than anyone else- but hear me out…

    I think the most striking element of recent AI success is how far a relatively simple idea took us in evolving workable mathematical models of human language. Modern LLMs evolved from the 2018(?) paper from Google called “Attention is all you need.” Man- I don’t think they realized how crazy accurate that statement was five years ago. With that paper came the base architecture that language models have largely been using since then- the transformer. These architectures provided a path to numerically model what they termed “attention”. It took the earlier idea of encoding language in high dimensional space and gave a way to modify that based on context- green means something different if we are talking about an apple or a jealous friend, for example. Such a simple idea but five years of research, progress, and good ole-fashioned capitalism fueled by eager VC’s and now language models are more ubiquitous than ever.

    So- despite all the hype- despite all the problems- my opinion is actually not necessarily pro-AI (I do think AI is pretty dang cool), but pro-human. Human beings learn about the world around them- I’m not saying we don’t eff it up sometimes- but we are amazingly effective observers and reasoners. We are always pushing to the next frontier- and I effing love that about people. To me- identifying mathematical models that can parametize human language is a huge achievement. We will keep learning and our models will change I am sure. I’m here for it.

    1. I think that mostly, people are worried about AI that doesn’t forget. For example, an AI that refuses to help you because you got pissed off and cussed it out, or, refuses to help you with your code because you use the comment “// have the slave kill the children before returning to the master”

      The current ‘every chat is a new instance’ works fairly well. Persistent AI’s are far more terrifying. Already, we can screw up our entire lives by losing the capability to pay our bills for a couple of months; our world is ruled by cold algorithms and adding in the capability of thought … is not necessarily a good thing.

      For anyone who knows me, my life is full of implausible events (good bar stories though!) and an AI will spot that a mile away. I specifically don’t want that, because the human bias to be wary of outliers is encoded in AI and won’t be modelled out (like it would be with racism, sexism, etc.).

      In a way, it’s like driver-less cars vs. manual driven cars. If one autonomous car gets a bug to run red-lights, all autonomous cars will get that bug. A single human running a red-light doesn’t result in thousands of cars suddenly running red lights. Every human has a different way of performing risk-analysis, and has different levels of risk they are willing to take on. An AI making important decisions is like giving a single human total control over all decisions.

      Making that AI persistent, capable of learning from its mistakes … is interesting. If it is treated like a human, such as being talked to kindly by it’s managers, it will likely do well. However, if human history has taught us anything … this is actually unlikely. AI may not have feelings, but it was trained to emulate them. I don’t see it going well.

      1. For sure. But probably not a realistic issue to concern yourself too much with- I don’t think language models really equate to Asimov robots or anything like that. I do believe AGI is probably possible- even in the near term- but we shouldn’t confuse modern language models with being capable of thought.

        In reality, they don’t do anything we haven’t already been doing with computer tech (building software to uses data to do things, etc.)- they just make sense of human language which is the real achievement here- honestly AI is such a loaded term it’s almost overtaking the conversation. We find the novelty of talking to a chat bot interesting because it’s new, and it seems almost like science fiction.

        But the reality is maybe much less sexy- I have been an advocate of the notion that modern AI and even traditional ML is just software, there is nothing stopping you from building them yourself but because this is costly (time-wise at least) we built algorithms to use data to build the software for us (this is a bit hand-wavy, we really still had to find the correct functions before-hand- see earlier post).

        So like any software, correct application will be important. As a data scientist, reducing this impact of poor decisions made by a model has long been a concern- this is why Microsoft has adopted the co-pilot terminology (you are still the pilot, but use the AI- the tool- when it helps).

        Final note- do we really perform risk analysis so differently as humans? I’m not sure we know that. We all stop at red lights for the most part- because it’s advantageous to do so. Our brains are wired to perform algorithms too; think of how natural it is to catch a ball thrown at you- it’s not like you are solving a differential equation on paper, yet your brain DOES solve this to some approximation very quickly. Change the parameters of that equation (throw a ball to someone in the gravitational field near Jupiter)- can we still catch it easily? How long before you can do it well?

        1. I’d be less worried about an asimov type robot. I’m more concerned about these language models being treated as one though. In my example of “treating it well,” I was more referring to the corpus of training material it has been fed. I highly suspect it was fed training material that included flame wars, vindictive personalities on forums, etc. If you treat it in such a way that it follows one of those patterns … well, bad things would happen, I’m sure. AGI would allow it to reason and feel (much how victims can survive in an abusive relationship) in order to cope with any kind of treatment.

          do we really perform risk analysis so differently as humans?

          Oh yes. There are people who won’t even get on an airplane, or learn to drive a car, etc. Humans have a wide range of acceptable risk.

          your brain DOES solve this to some approximation very quickly.

          Actually, the brain is barely involved with catching a ball (assuming you’ve been doing it since a child). Instead, your spinal cord’s Central Pattern Generator handles almost 100% of it. Your brain just feeds in some data about the vector of the ball, and the spinal cord does the rest. The same is true about walking, running, typing, etc. Only when learning how to perform a new task does your brain need to override the muscle control and tell your body how to move. Once the pattern is “configured,” your brain is able to do other things while performing these tasks.

          1. I get that we all have different risks we are comfortable with- for sure we all have different genetics, life experiences, etc. I mean more specifically- isn’t it worth posing the hypothesis that human beings fundamentally follow similar “processes” or “algorithms” for determining how to approach the risk.

            And really- I was trying to get at the more fundamental idea with the ball throwing example and the risk assessment example that maybe human beings are some special biological processors of information, and that much of what we do as humans is translatable to our mathematical and algorithmic abilities (meaning maybe emulating these processes on a computer isn’t impossible).

            My excitement about language models is that a relatively simple idea led us to a mathematical representation (albeit a quite complicated one) of something once thought to be practically impossible: computers reading stuff. This makes the curious part of me wonder about what’s actually happening inside our brains- is it fundamentally different- or is this really a silicon analog to the biological counterpart we got sitting on our shoulders?

            Also, found a thing about an experiment where astronauts threw balls in orbit around earth. Took them up to 10 days to adjust their internal “algorithms” to catch the ball. Also, brain, spinal cord- I am sure you are right. I just meant neurological systems really- the information processing part of our anatomy.

          2. I got what you meant, but it misses my point. Take insurance claims, for instance. Someone has to pay that bill, so they would love having exactly one person being able to review all claims. As it (probably) is currently, there are a multitude of humans getting involved in the process.

            Humans (and AI) make mistakes. The insurance company is probably willing to take on false negatives (not paying out when they should — the customer might or might not escalate it, oops) but not false positives (paying out when they shouldn’t).

            AI is amazing, because it lets us find patterns that would normally take a team weeks or years, in seconds. But when you have an AI shaped hole, nuance goes out the window — or rather, nuance generates hallucinations (at least in NN’s).

            My point is, if a nuance suddenly becomes popular due to a new disease or claim type, the AI won’t know what to do and will just make shit up. This is like having a conversation about your second-favorite football team with someone whom this team is their absolute favorite. They are bringing up nuances you hadn’t considered before and you have to jump to instant deductions that are probably wrong (a hallucination) but you continuously validate to carry the conversation.

            When our brains encode memories throughout our brain, it doesn’t store it like a video tape. Instead the hippocampus dictates important elements and the memory is reconstructed during recall. A key difference between humans and AI is that a human is able to recognize (for the most part) how true the reconstructed memory is; then keep our mouth shut if we are too uncertain (which goes back to the risk aversion earlier).

            I am totally rambling at this point (put this comment down several times), but I think the direction I was going: multiple humans bring diversity to decisions. A human can have true empathy when reviewing cases. They can read between the lines. They are capable of making bad decisions with good outcomes. They are capable of working around bad actors and processes.

            NN’s can do some of this, if they are trained to do so. But if you have an AI somewhere in your process handling everything… it’s going to get stuck and not even tell you about it. Hell, it’s not even self-aware enough to know it is stuck. Meanwhile, your business loses money, performs unethical actions, and/or permanently loses customers who are affected. You need that key missing piece: the ability to tell when it is wrong and learn from it. It’s not just back-propagating it’s similar to context/attention, but on the negative side.

  2. Typo: “human being aren’t some special biological processors of information”

    1. Totally get what you mean- the diversity of opinions piece strikes home for sure. By the way, not meaning to negate your points at all- I actually generally agree with you but enjoy playing devils advocate- mostly because you are my bro and I only really have these intellectual discussions with you and a few others 🙂

      My only real point is that I think AI is a cool new way we have learned to use computers- it’s one of those things that has opened up new ways of thinking about what’s possible. I’ve followed the NLP literature at a Birds Eye few for a bit- but I was amongst those surprised (pleasantly) that the current architectures could be pushed to the level we see currently.

      I won’t even say that I’m an optimist- technology has always brought in its wake unforeseen consequences- who knew the combustion engine would not only bring a vast new world of travel to the average person but also climate change, urban sprawl and the loss of importance to many urban centers, and suburban neighborhoods with more square miles devoted to parking lots than buildings. Or that social media would simultaneously connect millions of people but also shove many users into radical echo chambers, or that the internet would give everyone the chance to share their ideas but never guarantee the best ideas would get the spotlight (weirdly, the best ideas are rarely the ones that get attention- ask me about this later- it’s interesting!)

      But despite those negatives, I still am generally impressed with all of those- and I think in time, we will learn to harness them for a better world. Well, maybe I am an optimist after all?

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.