Anthropic has developed an AI 'brain scanner' to understand how LLMs work and it turns out the reason why chatbots are terrible at simple math and hallucinate is weirder than you thought
-
This post did not contain any content.
you can't trust its explanations as to what it has just done.
I might have had a lucky guess, but this was basically my assumption. You can't ask LLMs how they work and get an answer coming from an internal understanding of themselves, because they have no 'internal' experience.
Unless you make a scanner like the one in the study, non-verbal processing is as much of a black box to their 'output voice' as it is to us.
-
'is weirder than you thought '
I am as likely to click a link with that line as much as if it had
'this one weird trick' or 'side hussle'.
I would really like it if headlines treated us like adults and got rid of click baity lines.
They do it because it works on the whole. If straight titles were as effective they'd be used instead.
-
This post did not contain any content.
Don't tell me that my thoughts aren't weird enough.
-
They do it because it works on the whole. If straight titles were as effective they'd be used instead.
The one weird trick that makes clickbait work
-
'is weirder than you thought '
I am as likely to click a link with that line as much as if it had
'this one weird trick' or 'side hussle'.
I would really like it if headlines treated us like adults and got rid of click baity lines.
But then you wouldn't need to click on thir Ad infested shite website where 1-2 paragraphs worth of actual information is stretched into a giant essay so that they can show you more Ads the longer you scroll
-
They do it because it works on the whole. If straight titles were as effective they'd be used instead.
-
This is pretty normal, in my opinion. Every time people complain about common core arithmetic there are dozens of us who come out of the woodwork to argue that the concepts being taught are important for deeper understanding of math, beyond just rote memorization of pencil and paper algorithms.
The problem with common core math isn’t that rounding is inherently bad, it’s that you don’t start with that as a framework.
-
I might. Then I can subtract 74 to get 74*14, and subtract 28 to get 72*13.
I don't generally do that to 'weird' numbers, I usually get closer to multiples of 5, 9, 10, or 11.
But a computer stores information differently. Perhaps it moves closer to numbers with simpler binary addresses.
-
How I'd do it is basically
72 * (10+3)
(72 * 10) + (72 * 3)
(720) + (3*(70+2))
(720) + (210+6)
(720) + (216)
936
Basically I break the numbers apart into easier chunks and then add them together.
-
You're antropomorphising quite a bit there. It is not trying to be deceptive, it's building two mostly unrelated pieces of text and deciding the fuzzy logic is getting it the most likely valid response once and that the description of the algorithm is the most likely response to the other. As far as I can tell there's neither a reward for lying about the process nor any awareness of what the process was anywhere in this.
Still interesting (but unsurprising) that it's not getting there by doing actual maths, though.
Maybe you're right. Maybe it's Markov chains all the way down.
The only way I can think to test this would be to "poison" the training data with faulty arithmetic to see if it is just recalling precedent or actually implementing an algorithm.
-
"Ask Claude to add 36 and 59 and the model will go through a series of odd steps, including first adding a selection of approximate values (add 40ish and 60ish, add 57ish and 36ish). Towards the end of its process, it comes up with the value 92ish. Meanwhile, another sequence of steps focuses on the last digits, 6 and 9, and determines that the answer must end in a 5. Putting that together with 92ish gives the correct answer of 95," the MIT article explains."
That is precisrly how I do math. Feel a little targeted that they called this odd.
But you're doing two calculations now, an approximate one and another one on the last digits, since you're going to do the approximate calculation you might act as well just do the accurate calculation and be done in one step.
This solution, while it works, has the feeling of evolution. No intelligent design, which I suppose makes sense considering the AI did essentially evolve.
-
But you're doing two calculations now, an approximate one and another one on the last digits, since you're going to do the approximate calculation you might act as well just do the accurate calculation and be done in one step.
This solution, while it works, has the feeling of evolution. No intelligent design, which I suppose makes sense considering the AI did essentially evolve.
Appreciate the advice on how my brain should work.
-
Not, but I'd do 7510 + 754, then subtract the extra.
The LLM method of doing it with multiple numbers without proper interpolation though makes it extra weird
-
It really doesn't. You're just describing the "fancy" part of "fancy autocomplete." No one was ever really suggesting that they only predict the next word. If that was the case they would just be autocomplete, nothing fancy about it.
What's being conveyed by "fancy autocomplete" is that these models ultimately operate by combining the most statistically likely elements of their dataset, with some application of random noise. More noise creates more "creative" (meaning more random, less probable) outputs. They do not actually "think" as we understand thought. This can clearly be seen in the examples given in the article, especially to do with math. The model is throwing together elements that are statistically proximate to the prompt. It's not actually applying a structured, logical method the way humans can be taught to.
People are generally shit at understanding probabilities and even when they have a fairly strong math background tend to explain probablistic outcomes through anthropomorphism rather than doing the more difficult and "think-painy" statistical analysis that would be required to know if there was anything more to it.
I myself start to have thoughts that balatro is purposefully screwing me over or feeding me outcomes when it's just randomness and probability as stated.
Ultimately, it's easier (and more fun) for us to reason that way and it largely serves us better in everyday life.
But these things are entire casinos' worth of probability and statistics in and of themselves, and the people developing them want desperately to believe that they are something more than pseudorandom probabilistic fancy autocomplete engines.
Add the difficulty of getting someone to understand how something works when their salary depends on them not understanding it to the existing inability of humans to reason probabilistically and the AGI from LLM delusion becomes near impossible to shake for some folks.
I wouldn't be surprised if this AI hype bubble yields a cult in the end.
-
Yeah I caught that too, I'd be curious to know more about what specifically they meant by that.
Being able to link all of the words that have a similar meaning, say, nearby, close, adjacent, proximal, side-by-side, etc and realize they all share something in common could be done in many ways. Some would require an abstract understanding of what spatial distance actually is, an understanding of physical reality. Others would not, one could simply make use of word adjacency, noticing that all of these words are frequently used alongside certain other words. This would not be abstract, it'd be more of a simple sum of clear correlations. You could call this mathematical framework a universal language if you wanted.
Ultimately, a person learns meaning and then applies language to it. When I'm a baby I see my mother, and know my mother is something that exists. Then I learn the word "mother" and apply it to her. The abstract comes first. Can an LLM do something similar despite having never seen anything that isn't a word or number?
Can an LLM do something similar despite having never seen anything that isn’t a word or number?
No.
-
Redditor as "a person active on Reddit"? I don't see where I was talking about humans. Or am I misunderstanding the question?
This dumbass is convinced that humans are chatbots likely because chatbots are his only friends.
-
To understand what's actually happening, Anthropic's researchers developed a new technique, called circuit tracing, to track the decision-making processes inside a large language model step-by-step. They then applied it to their own Claude 3.5 Haiku LLM.
Anthropic says its approach was inspired by the brain scanning techniques used in neuroscience and can identify components of the model that are active at different times. In other words, it's a little like a brain scanner spotting which parts of the brain are firing during a cognitive process.
This is why LLMs are so patchy at math. (Image credit: Anthropic)
Anthropic made lots of intriguing discoveries using this approach, not least of which is why LLMs are so terrible at basic mathematics. "Ask Claude to add 36 and 59 and the model will go through a series of odd steps, including first adding a selection of approximate values (add 40ish and 60ish, add 57ish and 36ish). Towards the end of its process, it comes up with the value 92ish. Meanwhile, another sequence of steps focuses on the last digits, 6 and 9, and determines that the answer must end in a 5. Putting that together with 92ish gives the correct answer of 95," the MIT article explains.
But here's the really funky bit. If you ask Claude how it got the correct answer of 95, it will apparently tell you, "I added the ones (6+9=15), carried the 1, then added the 10s (3+5+1=9), resulting in 95." But that actually only reflects common answers in its training data as to how the sum might be completed, as opposed to what it actually did.
In other words, not only does the model use a very, very odd method to do the maths, you can't trust its explanations as to what it has just done. That's significant and shows that model outputs can not be relied upon when designing guardrails for AI. Their internal workings need to be understood, too.
Another very surprising outcome of the research is the discovery that these LLMs do not, as is widely assumed, operate by merely predicting the next word. By tracing how Claude generated rhyming couplets, Anthropic found that it chose the rhyming word at the end of verses first, then filled in the rest of the line.
"The planning thing in poems blew me away," says Batson. "Instead of at the very last minute trying to make the rhyme make sense, it knows where it’s going."
Anthropic discovered that their Claude LLM didn't just predict the next word. (Image credit: Anthropic)
Anthropic also found, among other things, that Claude "sometimes thinks in a conceptual space that is shared between languages, suggesting it has a kind of universal 'language of thought'."
Anywho, there's apparently a long way to go with this research. According to Anthropic, "it currently takes a few hours of human effort to understand the circuits we see, even on prompts with only tens of words." And the research doesn't explain how the structures inside LLMs are formed in the first place.
But it has shone a light on at least some parts of how these oddly mysterious AI beings—which we have created but don't understand—actually work. And that has to be a good thing.
Thanks for copypasting. It should be criminal to share a clickbait non-descriptive headline without atleast copying a couple paragraphs for context.
-
This dumbass is convinced that humans are chatbots likely because chatbots are his only friends.
Sounds scary. I read a story the other day about a dude who really got himself a discord server with chatbots, and that was his main place of "communicating" and "socializing"
-
Sounds scary. I read a story the other day about a dude who really got himself a discord server with chatbots, and that was his main place of "communicating" and "socializing"
This anecdote has the makings of a "men will literally x instead of going to therapy" joke.
On a more serious note though, I really wish people would stop anthropomorphisizing these things, especially when they do it while dehumanizing and devaluing humanity as a whole.
But that's unlikely to happen. It's the same type of people that thought the mind was a machine in the first industrial revolution, and then a CPU in the third...now they think it's an LLM.
LLMs could have some better (if narrower) applications if we could stop being so stupid as to inject them into things where they are obviously counterproductive.
