Amid all the hype about AI it sometimes seems as though the world has lost sight of the fact that software such as ChatGPT contains no intelligence. Instead it’s an extremely sophisticated system for extracting plausible machine generated content from the corpus on which it is trained. There’s a long history behind machine generated text, and perhaps the simplest example comes in the form of a Markov chain. [Ben Hoyt] takes us through how these work, and provides some Python code so that you can roll your own.
If you’re uncertain what a Markov chain is, consider the predictive text on your phone. It works by offering the statistically most likely next word in your sentence, and should you accept all of its choices it will deliver sentences which are superficially readable but otherwise complete nonsense. He demonstrates with very simple short source texts how a collocate probability map is generated for two-word phrases, and how from that a likely next word can be extracted. It’s not AI, but it can be a lot of fun to play with and it opens the door to the entire field of computational linguistics. We haven’t set one loose on Hackaday’s archive yet but we suspect it would talk a lot about the Arduino.
We’re talking about Markov chains here with respect to language, but it’s also worth remembering that they work for music too.
Header: Bad AI image with Dall-E prompt, “Ten thousand monkeys with typewriters”.
More like “Associated Press Intelligence” (an oxymoron if I’ve ever heard one)
“…Amid all the hype about AI it sometimes seems as though the world has lost sight of the fact that software such as ChatGPT contains no intelligence. Instead it’s an extremely sophisticated system for extracting plausible machine generated content from the corpus on which it is trained. …” – thank you. This needs to be repeated often.
Yeah. Still cannot generalize. CCRU is sad
It could also be said of many people.
Thank you! This can not be overstated.
Markov chains are a fun way to play with generating varying styles of content when combined with context. Yet AI generally refers not just to generation of contextual content, but often inclusion of abstract logical problems to be solved. Such a process requires both recognition and comprehension of the question being asked, problem solving and presentation and explanation of potential solutions. You would be sitting far from reality to portray AI as simply something that reformats text. Yet as AI evolves rapidly and increasingly self programs, your answer will become more and more rapidly incorrect :)
It depends on the level of analysis (high or low). Thus while the brain may be said to show evidence of intelligence, individual neurons, or even clusters of them, most certainly do not. Thus it is unfair to judge an AI system by its computational elements (words and associations). It should instead be judged by its outputs.
But that makes mediocore writers scared.
The ‘G’ in ChatGPT indeed stands for ‘Generative.’ While it’s crucial to recognize the differences in capabilities between AI and human intelligence, it’s also important not to oversimplify AI as being ‘limited to only what it was trained on.’ ChatGPT can, in fact, generate content beyond its training data, demonstrating a level of creativity and prediction.
In a similar vein, humans can be seen as highly sophisticated systems that extract plausible sequential event patterns from chaotic input streams, forming narratives. We’re essentially prediction engines, not unlike these AI systems. Both are adept at pattern recognition and generating predictions, albeit with differing complexities and nuances.
I beg to differ. Animal intelligence is just a sophisticated system for extracting plausible animal generated content from the corpus on which the person was trained. Organic brains form pathways based on stimulus and then those pathways are used to generate novel outputs. Saying AI has not intelligence is easy if you don’t bother defining what intelligence actually is in a way that excludes AI without excluding the average animal. People insisting AI isn’t intelligence are generally just demonstrating their ignorance of how modern AI works or the centuries of philosophical debate to define intelligence, usually both. Usually it’s just “this AI makes mistakes, which humans never do!” as if an AI trained for 2 weeks weren’t already performing better than a newborn animal after 2 weeks carrying an organic network 1,000x the size of the AI’s.
Define intelligence in a way that doesn’t exclude a single living human being, but doesn’t include any AI models, and I’ll consider that AI has no “true” intelligence and maybe I’m not a “true” Scotsman.
So you think ChatGPT is intelligent?
I think he’s saying “humans aren’t special”. If chatGPT is just a highly sophisticated mimic, maybe humans are too?
I think he’s right. What LLM’s do is pretty similar to what animal brains do. They’re both just neural nets, doing their neural net thing. They say, “AI’s are terrible because they sometimes make things up!” Well, so do humans. They say, “Sometimes AI’s can’t answer questions.” Well, sometimes humans can’t either. They say, “AI’s are no good because they can’t tell you HOW they arrived at an answer.” Well, a lot of the time humans can’t either.
I’m not saying AI’s are as intelligent as people, I’m saying people aren’t very intelligent either.
My opinion is that AI’s have a chance to make people better thinkers because AI’s have no “limbic system” or “emotional brain”. They think without emotion, which makes a very nice compliment to humans, who DO think with emotion.
“My opinion is that AI’s have a chance to make people better thinkers because AI’s have no “limbic system” or “emotional brain”. They think without emotion, which makes a very nice compliment to humans, who DO think with emotion.”
Hm. Reminds me of the classic verbal battles between Mr. Spock and Dr. McCoy. 🙂
Optionally at least. I have to confess I sit on the fence when it comes to emotion. It makes life more interesting, yet I do enjoy the clarity of suppressing it. Emotion creates a “noisy” picture with unnecessary inputs.
Well except one is a much simplified version of the other.
Fundamentally the average human being is barely self-aware. That’s not a sign of intelligence, that’s the minimum. They barely achieve it.
ChatGPT isn’t self aware, but unlike my coworkers it can answer questions requiring more than knowing the alphabet.
“They think without emotion, which makes a very nice compliment to humans, who DO think with emotion”
Yes, but also no. If an LLM training data is poisoned by something like people squabbling on Twitter, it will use emotionally charged language.
LLMs are the sum of their training data and the algorithms that process it. They may not have ‘real emotions’ but don’t get to thinking they are perfectly logical Vulcans.
The old computing mantra is “garbage in, garbage out”.
comment glitch #12938908904590 but in short, AI not handling emotions is a flaw not a strength.
It passes the Turing Test easily. And it does demonstrate some AGI in my opinion. I recently had an electronic problem where certain numerical inputs failed to produce expected outputs. GPT4 detected that the binary common element was in the second data line based on what I was telling it. Very hard to see that coming from some statistical rehash of tons of verbiage.
You could start by defining intelligence yourself (spoiler alert, you cant), or maybe the conclusion philosophy has arrived at after thousands of years of searching (spoiler alert, no such conclusion exists), or how you know that intelligence is just organic reinforcement learning (spoiler alert, you don’t, and it isn’t). Its easy to handwave it all away when you generalize about “performing better” if you ignore the obvious, like the fact that a simple calculator can ‘perform better’ than every mathematician on earth put together despite having what, a millionth the computational prowess, a billionth? Are calculators intelligent? If so, can you define it in such a way that excludes the abacus?
Reminds me of the saying “all generalizations are false”.
Intelligence is the ability to learn and apply that knowledge to generate new insights. “AI” cannot do this, yet ants and squirrels and squid and humans can.
AI (currently) has no care or method to know when it is wrong or to generate new insights based on existing knowledge. It cannot make abstractions.
ChatGPT is simply a permutation generator with a filter to match styles, there is no intelligence there.
+1 ….. I was only saying the other day to someone how intelligent the Arduino was. Obviously not as intelligent as a human, but possibly more so than an earthworm. I’m quite happy to say to anyone that chatGPT is intelligent, but perhaps not quite aware of it’s own existence yet. Then again, is a mouse self aware? Does a dog have emotional intelligence? (obviously not a pitbull terrier type). Defining intelligence is multi-facted and inevitably there’s a broad spectrum where even a certain Republican politician often in the news (and the courts) fits in.
“Then again, is a mouse self aware? Does a dog have emotional intelligence? (obviously not a pitbull terrier type). Defining intelligence is multi-facted and inevitably there’s a broad spectrum where even a certain Republican politician often in the news (and the courts) fits in. ”
Every pet owner would like to confirm that.
Pets who have the ability to dream (rapid eye movement) surely are mentally advanced.
But here we’re crossing the realms of philosophy. A field who has been neglected quite a while.
Personally, based on pure observation, I would attribute cognitive abilities to some animals than their researchers. They’re less robotic in their acting.
Most pet owners anthropomorphize their animals and see in them thought that doesn’t exist. See e.g. the studies about dogs looking guilty.
…and read the reply by the study author who said that the headline simplification you’re regurgitating isn’t what the study showed.
Just flat out wrong. Dogs have attained genius level emotional intelligence regarding humans, as they have been manipulating humans into taking care of them for thousands of years. You are going to try and tell me a dog who gets all excited , acting like it is so happy to see the guy on the couch, and snuggles up to him so much that it seems to fall off the couch and hits what looks to be a birthday cake on the way down. The dog just looks clumsy. Until the slow motion footsge gets played back, and you see that the dog knew which way it was falling and where the cake was, and twisted itself during the fall to be able to swipe a huge bite from the cake. And without the slow motion replay, nobody would have been the wiser. That is forethought of the consequences vs reward, planning, and execution. Smarter than some people I know. And it was definitely not accidental coincidence. You can see the dogs eyes lock on the cake as it twists its head around to snag the bite.
Spiders dream…
What baffles me is how often emotions are being left out.
Even plant life has the ability to feel, despite being based on a more primitive nervous system.
So how comes that artifical intelligence is in the center of attention, but not emotional “intelligence” ?
One of the conditions of self-awareness is the ability to “feel” the surroundings, as a way of interaction.
Real people personality maybe?
There was someone who asked, what does it feel like to be a proton?
Mostly positive.
Best comment.
Window sticker: have you hugged your plant today?
Intelligence is the ability to learn, ie pattern recognition. So a neural network that’s can’t change itself or use some form of memory is just a fancy algorithm, so is bacteria. So I’d say something like chatgpt is somewhat intelligent since it is able to understand context, but not in a “smart” way like how humans can learn and reason about things quickly, especially since llms can only really process language. I wonder, if you include the training algorithm for a neural network as part of the AI, then does that make the whole thing intelligent regardless of how simple the neural net is?
btw
explain the pattern and find the next entry
0bcd
a1cde
ab2def
abc3efg
abcd4fgh
abcde5ghi
abcdef6hij
abcdefg7ijk
abcedfgh8jkl
chatgpt:
The given pattern seems to follow a sequence where each entry consists of the letters “a” through “h” in alphabetical order, with each letter corresponding to a position in the entry, and the number at the end is the position of the last letter in the alphabet in that entry.
Let’s break down the pattern:
0bcd
a1cde
ab2def
abc3efg
abcd4fgh
abcde5ghi
abcdef6hij
abcdefg7ijk
abcedfgh8jkl
In each entry:
The first letter is “a.”
The second letter is “b.”
The third letter is “c.”
The fourth letter is the position of the last letter in the entry.
Based on this pattern, the next entry would follow the same sequence:
abcdefgh9jklm
In this entry, the first four letters are “a,” “b,” “c,” and “d,” and the last letter’s position in the alphabet is “9,” so the last letter is “i” (since it is the 9th letter in the alphabet).
Therefore, the next entry in the pattern is “abcdefgh9ijklm.”
“Based on this pattern, the next entry would follow the same sequence:
abcdefgh9jklm”
Why? It seems to me like the next entry should be abcdefghi9klm.
I’m pointing out it’s wrong
Put your back into it. It was only born yesterday. – https://chat.openai.com/share/71888f01-6897-4caa-b9e5-e39415b57b3d
—
… The following is a pattern that will be new to you…
Is There a Pattern?
…
Add limerick goofy comments to the side of each entry to help you keep track as you re-write the entire list along with the next 5 entries.
ChatGPT
Pattern Analysis and Extension
0bcd – Zero leads the charge, with b, c, and d in tow.
a1cde – One steps in, right after a, b’s now a no-show.
ab2def – Two’s in the mix, with c, d, e, f in line.
abc3efg – Three jumps the queue, where b, c, and d find.
abcd4fgh – Four’s at the door, e’s gone, f, g, h are fine.
abcde5ghi – Five’s alive, after e, f, g, h, and i.
abcdef6hij – Six in the mix, g’s out, h, i, j comply.
abcdefg7ijk – Seven’s in heaven, h takes a hike, i, j, k.
abcdefgh8jkl – Eight’s not late, i’s gone, j, k, l in the strike.
Next Five Entries
abcdefghi9klm – Nine’s divine, j’s out, k, l, m in the rhyme.
abcdefghij10lmn – Ten’s a hen…(goes on 3 more)
It’s worth saying that there are different gpt models, and not all models perform the same on every task. Also, even in the example where it gave a wrong answer it had given a pretty strong analysis of the pattern even so.
“Intelligence is the ability to learn, ie pattern recognition. So a neural network that’s can’t change itself or use some form of memory is just a fancy algorithm, so is bacteria”
Which type of intelligence, though?
Is there one universal type or ar there more?
What about cognitive reflection? Thinking out of the box?
Simple example :
If a group of musicians needs 5 minutes to play a song, how long needs one musician?
If the context is not understood, I think, math or an algorithm can’t solve it properly.
Exactly, my answer is my opinion, but it’s hard to define exactly what intelligence is.
Here is what gpt model 3.5 (not the latest version and significantly less capable) has to say about that question:
If a group of musicians needs 5 minutes to play a song, and there are multiple musicians in the group, it can be assumed that they play the song simultaneously. Therefore, the time it takes for one musician to play the song would still be 5 minutes.
Even an algorithm that is programmed to change isn’t doing so freely. How it changes is based on yet another algorithm. It’s just kicking the philosophical can further down the road.
An algorithm like ChatGPT doesn’t understand anything. To say you understand something implies that you already know what you’re about to talk about before you say it, whereas ChatGPT does it backwards: it looks at what was already written and then finds what syllables and words fit in next, like a person assembling a jigsaw puzzle without knowing what the picture is going to be.
> defining what intelligence actually is in a way that excludes AI without excluding the average animal
Why is that a problem? It’s not implausible that the average animal is not intelligent.
John Searle already pointed out that if intelligence should mean something, a fixed algorithm or a program isn’t it. Such things can have syntax, but not semantics, meaning that they cannot create or have meaning and therefore do not even understand what they’re doing. Whether they can create “novel outputs” is no different than flipping coins, so calling something like that intelligent is misleading.
“Why is that a problem? It’s not implausible that the average animal is not intelligent. ”
You should see all the comments in the animal shorts on YT. Invariably people will say look at how intelligent the animal is. Or worse say how dumb people can be.
People anthropomorphize the wind and see intent in inanimate objects to explain natural events.
I for the most part agree. However, the one additional point I would like to make is in context to the design of that intelligence. I would say that AI, in its current form of prompt to response, is, while maybe somewhat intelligent, not conscious or sentient. However, I do think a modified version of the an AI with the capabilities of our current model that did not exist only in a prompt to response form (Maybe with a feedback loop of some kind) and had some form of longterm memory could be much closer than we currently believe we are to a completely intelligent and conscious AI.
Cad makes a very valid point. Ignorance is bliss for many. Yet AI is also evolving far more rapidly than many realise. Any statement made today, under pressures of constant learning and global input, is likely to be incorrect by tomorrow. To make a blanket statement that AI is not intelligent would be ignorant at best, yet I have seen few making such statements who truly understand their own intelligence and underlying processes. There are going to be some real eye openers in the near future.
Well it is the core of the world empire, so of course they would have their commissars make sure the thinking is correct. Would be silly if they did not
The Emacs editor has a variant on Markov chain (the “Dissociated Press” function) which takes even less code. Load your source document into a buffer. Randomize a starting location. Output some number of characters, maybe with some random variation. Take the last N characters output and search for another instance of them in the buffer, wrapping the search around to the start of the buffer if needed. Output, skip. Output, skip. Repeat until bored.
Complete Shakespeare (with indentation), 4-character overlap:
THE SON of sixteen,
That makes civil hast action? Hence;
Let witchcraft thus have marked matron, so you would give thee,
from our royal hand of the King’s day have you require in press’d
down with pinches;
and die;
How her price is fall into Enforc’d to scold.
CRESSIDA. The midnight bell.
Bene. Then the former words
On any play with my masted our reward,
To bed, to bed, and die!
[To RICHMOND, afterwards Eminence that our own;
So should say you, Simon Catling?
1. Mus. Faith, Captain, these are a queen of Naples.
‘The riot of those to take up their nurse; therefore precise me
where he hath said to be and Claudio, as–in love of
Stop to climate business. Will be here and not the thou wert us’d.
SHall we fall foul offender an oak whose antique fables, nor shall
we sit in England’s neck. Being an ugly deed look faith, this word
shall give you?
Mer. The hand of Griefs would make use her well; the world
If we be not, should utter, with white roses her against some
storm,
She speak not. Mark a large mouth, i kiss me, sweet an
opposition.’ Say? Nay, good King? Look you ther function. Every
true man’s feather of
Gonzago’?
1. Play. Ay, I thanks I give
And then, an he could.
This apish again]
GLOUCESTER. Mark what mercIALLY. Why then, alas, sweet wife!
People forget, we all live in a “simulation” anyway…. so any “AI” is just an interface to that simulation.
It can’t be any smarter than the core softare that we’re running on.
Or, it’s just the next “big thing” that turns out to be nothing anyway.
Cool!
Also, if you want a quick, zero-effort play with Markov chains, check out Garkov—
Randomly written Garfield comics from the early 80’s, trained from the comics from that era.
http://www.joshmillard.com/garkov/
It’s neat to reload the page and see how semi-coherent the output is.
Well, trained is the wrong word, but you get the idea.
I’ve referred to ChatGPT as a “glorified Markov chain” since it was released.
I was referring to humans as “glorified Markov chains” ever since I realised how primitive and predictable the wast majority of people are.
Life tries to turn the surrounding random chaos of nature into meaningful non-random action.
AI is coming from the other end of the problem, trying to create meaningful action by starting from non-random rules derived ex-nihilo – from nothing.
This is not how generative transformers work. Not in a slightest.
finally someone who understands Ai. Ai can’t make a decision on it’s own. It can only make decisions within the limits of it’s programming. Ai is what your math teacher told you not to do and that is take an average of a set of averages. Think of it as a PID control loop. It’s answer is continually corrected till the deviation is close to zero
I think the idea of simply selecting a random instance of a word following a bigram is not the best way to form a markov’s chain that will faithfully reflect English syntax is incorrect.
The resulting generated text is much more syntactically correct (if still nonsensical) when we construct a table that calculates the relative frequency of each word after a bigram, and then weight the random function based on that frequency. In this way, if the word ‘and’ follows a given word pair 35% of the time, and ‘it’ follows 15% of the time, the. weighting the random function by these percentages will more often produce text that sounds like the original corpus, than if you simply selected words at random.
Mike
Actually there is AI in Markov Ch*AI*n… sorry couldn’t resist.
“and should you accept all of its choices it will deliver sentences which are superficially readable but otherwise complete nonsense.”
There is a film crew in database over the same lettuce and I have a few questions about the car park in the morning and I will be back in the morning for the next few weeks. I will let you have a good time in your office today.
Q.E.D.
“It’s not AI”
Back in the day, Markov models were very much considered AI techniques.
My final year MSc project was implementing Hidden Markov Models for use in speech recognition.
It so happens that, over time, mainstream computer science has assimilated these, and similar, techniques so they are no longer considered AI.
If we add weights to the inputs and outputs of the nodes of a Markov chain, and the weights can update with more inputs, is this not an AI neuron?
Understanding emotions would be a step up. Most kids learn that when their parents say violent things after someone almost wrecks into them on the road, they don’t actually mean it but they’re upset. AI or kids who don’t read the emotional context may learn a different lesson from that. If a person isn’t smart enough to understand something they’ve never felt, they might be smart enough to understand something they have felt. The unfeeling AI has a higher bar to reach there.
It’s not just emotions that work this way to be fair, but some amount of miscommunication happens because people get used to being understood by people who are able to fill in the blanks automatically by making basic assumptions. Then they encounter someone with different experiences or something like a computer that doesn’t do that, and all of a sudden “nothing is getting through” or “it won’t just do what i tell it” when the problem is on the other end.
comment glitch #12938908904590
“Contains no intelligence” is clearly going to far here. Intelligence is generally defined as the ability to acquire and apply knowledge and skills. It’s not about how, but if and how well. And it’s a very, very low bar. A wide variety systems have shown such abilities for decades, many decades IMO. There is really no question ChatGPT has such abilities.
Now there is the word intelligent, which often implies a high level of intelligence. It’s used very subjectively. So subjectively that one can easily argue that something is highly intelligent, or not intelligent at all, often depending on circumstance, but even in the same circumstance from different perspectives. It’s important we do not to confuse the two.