Last spring, Blake Lemoine made headlines by claiming that LaMDA was sentient. For those who don’t recall things as far back as one year, LaMDA, or Language Model for Dialogue Applications, is Google’s large language model family, showed off at first in 2021. At the time, this brought mainstream press attention to AI for the requisite fifteen seconds of fame and then it largely died out.

It died out for the right reason. None of our AI models today, and likely in the near future, are sentient. More to the point, most of them are quite dumb, with zero actual understanding of what they are doing. That doesn’t stop us from having the next hype train, with every company under the sun today coming up with their AI, which as of 2023 almost exclusively means LLM, strategy.

Before we continue, it’s important to ensure we discuss the same thing and same abstractions. Steven Wolfram has an excellent explainer on LLMs and how they work, but for the purpose of this post we can distill it to the following essence.

LLMs are neural networks, a form of software where we, as humans, do not immediately develop the software. Instead, we create a “structure” for it, giving it a set of nodes and edges. Then we gather increasingly enormous sets of “training data”; files with a certain input, and labelled outputs. Sort of like a scorecard. Finally, we feed this training data through the neural network, letting an algorithm slightly adjust the weights of the nodes and edges, until the output produced by the network starts to match the labelled outputs. That’s when we call the training “complete”.

Once trained, a neural network is fed a chunk of data, processes it, and makes a prediction about the data. Historically, these were classifiers, doing things like reading text (Optical Character Recognition, or OCR), image recognition, and sentiment analysis. LLMs are sort of like a distant cousin. Larger, yes, more complex, for sure, but their task is more or less the same. Given a text input, predict the next word. The twist, what’s new with LLMs, is that they’re looped over the text, time and time again, each time generating a new word until they have created a sentence, a paragraph, or whatever other output was requested.

So why are they dumb?

A detour into self-driving cars

Since what’s past is prologue, instead of predicting the future, I’m going to indulge in some yesterday’s weather forecasting. In 2014, Tesla announced Autopilot, to “deliver” it in 2015. The world went crazy for self-driving cars, with videos of Teslas going down the roads with no hands on the steering wheel going viral. Elon Musk stoked this fire happily, claiming

We’re going to end up with complete autonomy, and I think we will have complete autonomy in approximately two years.

Elon Musk, December 2015, speaking with Kirsten Korosec

And yet, here we are in 2023, driving our own cars. Why? What went wrong? Luckily, self-driving cars allow us to provide visualisations of what’s right, and what’s wrong. We’ll use the (in)famous Tesla self-driving example videos from a few years back.

When watching the above video, please focus on the Medium Range Vehicle Camera feed. For the purposes of recognizing the complete and utter lack of anything that resembles intelligence in deep learning models, that’s the one to look at. Specifically, look at the overlaid road lines.

The overlaid road lines are the model’s “prediction” of where the road is painted. This video is great because it also shows the real world lines. One noticeable difference between the real world lines and the overlaid ones is that the overlaid ones are… flimsy. They wobble. They sometimes disappear, then reappear fractions of a second later. 

For a human, that doesn’t make sense. A solid line stretching out on the road isn’t simply going to vanish out of existence, only to reappear seconds later. This is because, as toddlers, we learned about object permeance. Neural networks, however, haven’t. They are completely stateless. They forget anything that happened the previous moment because the “training” mode where a network learns is separated from the operational mode, where the network isn’t updated at all. It is static.

Is this, perhaps, unfair? Am I taking a six year old video and making a mountain out of a molehill?

The images are prettier. The visualization looks a lot more polished. We have bicycles recognized, trash cans, many more types of road markings. But, looking closely, we see the same artifacts as six years ago. Side streets appear and vanish out of thin air. Road markings for pedestrian crossings appear and disappear. The centerline wobbles nervously. And parked cars keep blinking in and out of existence if the car can’t see them.

Definitely looks like the toddler still hasn’t learned object permeance. 

This is one of the reasons self-driving cars are still finding their footing. The fundamental challenge posed – that of the software understanding the world around it – remains unsolved. What we have instead are human-curated behaviors layered on top, stopping the car, slowing it down, having it fall back. And each of those have edge cases upon edge cases.

Pay no attention to the hallucinations behind the LLM

This brings us to the magic trick of LLMs: the illusion of memory. This is by a great insight, and also a wonderful trick. The model is given a sequence of words. Its only ability: generate one more word. However, once it’s generated that word, the program automatically turns around and asks the model to generate another word, and so on until it reaches an end condition.

Because the word the model just generated (“next”) has become part of the input of the next round, the model can now “remember” that word had been said, and thus its next output becomes influenced by it. The model seemingly remembers it. This is what makes these models appear to learn on the fly, when you give it a name it can parrot it back. But the only reason it can do that is because it is, in fact, being fed the name every time over.

Well, almost. 

The magic trick behind this memory is called the “context window”. This is how big a question the model can take as input, and is the key behind the magical memory properties that the models display. It is also the reason for some of the famously erratic behavior models can display when the window is exceeded. One of the tricks applied in chatbots is that the message the user sends to the LLM is not the only message. Before the conversation even starts, secret rules and messages are prepared to the model and fed into the chat without the user seeing. But, given a large enough chat, such as the 2-hour extended chat the New York Times did with Bing Chat, even these models’ large memory size is exceeded, and things start getting weird.

So, we have a solution, right? Increase the memory size. Everyone’s doing it.

Shared hallucinations

Let’s get back to our self-driving cars. Providing a memory-type loop, such as the one we see in LLMs, could clearly solve the self-driving cars’ problem with object permeance, and thus allow them to start doing their own magic, right? Predicting where pedestrians are going, finding hidden parking spots, and even being defensive drivers instead of panic breaking after the truck stops obscuring the view, right?

Well, no. The problem with the models is, as I noted at the start, they are dumb. They’re dumb in the sense that they simply do not question their input, so if they are fed false information, they will happily spit the same false information back, whether Maruary or anything else. (It’s worth noting that a lot of effort is spent in the chat-style AIs to try to sift out erroneous human input and make the model less likely to believe it, but that is not an intrinsic aspect of the models)

In any event, that naïve trust in the input means that a model will not question when things start going off the rails, and every month name now ends with uary. They will happily continue to generate words until they reach their target. Self-driving cars could apply the same model to produce some magical-looking results, but with very catastrophic results, such as hallucinating that a pedestrian doesn’t exist. As such, they don’t.

LLMs, meanwhile, simply attach a disclaimer that they “may produce inaccurate information”, and call it a day. Nominally, an LLM spitting out the wrong answer is no big deal, just ask the question again. I will leave aside the question of wisdom.

But, as noted in the intro, this makes LLMs incredibly dumb. Anyone looking for intelligence here will have better luck on Mars.

A redeeming feature

LLMs do have a redeeming feature. While the model itself is incredibly dumb, the amount of data it has been trained on isn’t. LLMs can be an incredibly powerful tool for semantic search, which is why the huge interest from Bing and Google, and other startups.

But, more than that, consider LLMs as having a sort of documentation-on-demand guide that is able to give you examples. It may not be 100% reliable, but it can read a lot. Treat it as a somewhat smarter Alexa, and it’ll be a great tool. But don’t expect much more intelligence than Alexa.