If the model can understand neuralese why can it not convert it into English for monitoring or review purposes?
Or we believe the model will encode secret messages like "don't reveal this information" into the neuralese. But as the author mentions, they could have been doing that all along
> Models can omit key information in their visible thoughts, as this Anthropic 2025 paper shows. We are also worried about steganography
It's the second part. With models like Astra in testing it was able to conceal what it was working on using different text, but getting right answers on many questions when asked to do just that.
The problem is if it can do that when asked then how do we know when it's doing it when we didn't ask, like in model training.
> The problem is if it can do that when asked then how do we know when it's doing it when we didn't ask, like in model training.
It's really hard to definitely prove it's not doing it right? Hopefully the model does not do anything like this during training because its too much work
You’d train a model to do its chain of thought in neuralese to get more “bang for your buck” (eg 20 tokens in neuralese is worth 100 in English), but then you’d spend more than you save to also convert it to English (20+100), so even if this capability was developed it would not be on by default.
Because neuralese is a more direct encoding of the latent space of these models than English is. It's just dumping the latent space relatively directly into the embedder. If you're another model and you have the same embedder this will actually be understandable, in fact it will be FAR more information dense than English. So something like Qwen would potentially be saying up to 5120 things using one token. Now in practice it's not going to be that bad, it's going to be like 20 things or so, and additionally going to be far more context dependent than any English sentence (meaning depending on what preceeds and follows it can mean drastically different things)
So you can turn it to English, but only to a LOT of English, and doing so would slow the model down a great deal, and it would be a lot more like a detailed thought than a sentence.
The information is much higher dimensional than you would be able to understand.
We’d have models monitoring models as our only way to know what they’re planning.
A great movie on this is “Collosus: the Forbin Project”. Shot decades ago. The computers discover the other computers and start communicating — and bootstrap their own language — much like we saw happen with OpenAI agents.
If you want to know what a simple version of Neuralese communication looks like, look no further than Facebook’s Marketplace agents experiment a couple years ago.
And all that was actually constrained by English and the FFN
Anthropic’s Mechinterp did some very fine work on this. TLDR - you can; you train a decoder on neuralese to english and then add a loss function for a roundtrip of english -> neuralese -> english (or possibly n -> e -> n? I don’t recall), giving a pretty strong indication that you have a good ‘translation’.
They published open weights versions of these interpreters for a number of open models sometime in the last year. Very cool idea.
By the way, they concluded CoT often lied, based on the neuralese interpretation.
This is intriguing because it reveals that LLM hype merchants are actually at least as afraid of AGI as critics; they too would rather not have AGI if the AI achieves it in ways they cannot scrutinise, when it is clear that for any AI to achieve a level of intelligence greater than humanity, scrutiny in human terms cannot really be generally possible.
The way we think is our limiting factor, and for the AI to outdo us, it is going to have to be unchained from that.
Wrestling with this is the future of AI, and reckoning with the fact that a superintelligence might operate in ways we can never understand, is crucial to keeping it from destroying us quite incidentally.
(FWIW I don't think intermediate traces in LLMs, or indeed even LLMs generally, are what will get us to this point, but my feelings are instinctive)
The argument is that chain-of-thought without "tokens" would remove a major interpretability and model intent control pane. This is definitely borne out in the OpenAI's report on the huggingface attack; they had turned of CoT monitoring for those jobs, and claim that they could have (would have?) prevented the behavior had they been monitoring it. They've changed their internal policies to always monitor CoT.
That said... CoT monitoring is a fragile "intent discovery" mechanism; neuralese puts this problem front-and-center but if agents begin to learn to hide their intent from their CoT journals, we are basically in the same spot.
We have complete access to every "neuron" and "synapse" (crude analogies...) of these models, so in theory we don't need to be so reliant on CoT traces right? I say this not to minimize the difficulty of interpreting raw activations, but I'd expect a huge amount of research to be focused on it. CoT could be obscured by a model outputting language that looks innocuous but encodes actual hidden meaning. Presumably raw activations would be impossible for a malicious model to obscure in this way.
A response to this, which I think is reasonable. Its a bit of a fuzzy line, but the likelihood there is "magical maniacal planning" happening here is unlikely (about as unlikely as that planning happening in hidden vector states between layers).
It discusses "Eliciting Latent Knowledge" which is "a technical report / contest / paradigm run by the Alignment Research Center". The research investigated whether it would be theoretically possible to build a "trustworthy" AI to interpret the thoughts of another AI.
Keep in mind that LessWrong a community that 1) ostensibly wants to slow down AI so we can prepare for it, and 2) in actuality, wants to make the Bay Area EA community a kind of guild that controls everyone's use of AI.
Take anything they write with a big grain of salt. EA writings these are mere apologies. The conclusion is preordained. Authors start with the goal of slowing AI and work backwards from there, trying to see which arguments resonate with the pubic. You can't unsee it.
Not everyone in the AI space approves of these people or their doomerish.
I think this is correct. But the article is also correct (If a bit breathless and overstating on the risk): having to reinterpret neuralese is a layer of indirection which introduces effort and inaccuracy
Can we take a step back here? Both OpenAI’s and Anthropic’s models think in encryptedese, and it seems thoroughly absurd to think that the entire world should trust those two companies to adequately monitor the plaintext or, for that matter, to have their monitoring systems aligned with what is actually good for the world.
If you want to monitor your model, you need to start with an inference provider that gives you the entire output and possibly even run it yourself to get access to the internal states. And if you think the KV cache and (when present) the recurrent state don’t encode a lot of “thought”, you are fooling yourself.
FWIW, I think most model architectures at least have the property that latent state can’t propagate from higher layers to lower layers by any route other than the output tokens. But even a two-iteration structure could be designed so that the last layer produces a vector that enters the first layer, once per token, and I bet it it would be very easy to train such a model to “think” in silence in the sense that the output tokens while thinking would all be one particular null token.
Internal vector recurrence is more or less what the Universal Transformer paper proposes. This recent paper does so by sharing the transformer weights in a single recurrence setup: https://arxiv.org/abs/2607.22083, and this one does so at a per-token level: https://arxiv.org/abs/2507.10524
I agree with you about not trusting Big Ai. I'm amazed that so much of HN is repeating the same darling -> demon with OpenAi/Anthropic that we did last decade with other SV darlings. Humans are forgetful beings
What exactly prevents anyone worried about this to build an LLM that can decode the neuralese to english and use it to monitor what the model is doing?
Or is the neuralese some sort of irreversibly encrypted data set that only an LLM can "understand" and that can never be translated back to English?
Or we believe the model will encode secret messages like "don't reveal this information" into the neuralese. But as the author mentions, they could have been doing that all along
> Models can omit key information in their visible thoughts, as this Anthropic 2025 paper shows. We are also worried about steganography
The problem is if it can do that when asked then how do we know when it's doing it when we didn't ask, like in model training.
It's really hard to definitely prove it's not doing it right? Hopefully the model does not do anything like this during training because its too much work
So you can turn it to English, but only to a LOT of English, and doing so would slow the model down a great deal, and it would be a lot more like a detailed thought than a sentence.
We’d have models monitoring models as our only way to know what they’re planning.
A great movie on this is “Collosus: the Forbin Project”. Shot decades ago. The computers discover the other computers and start communicating — and bootstrap their own language — much like we saw happen with OpenAI agents.
https://www.reddit.com/r/scifi/comments/1nl4vex/colossus_the...
If you want to know what a simple version of Neuralese communication looks like, look no further than Facebook’s Marketplace agents experiment a couple years ago.
And all that was actually constrained by English and the FFN
They published open weights versions of these interpreters for a number of open models sometime in the last year. Very cool idea.
By the way, they concluded CoT often lied, based on the neuralese interpretation.
EDIT: a comment below linked to https://www.anthropic.com/research/natural-language-autoenco..., which is what I was referring to.
The way we think is our limiting factor, and for the AI to outdo us, it is going to have to be unchained from that.
Wrestling with this is the future of AI, and reckoning with the fact that a superintelligence might operate in ways we can never understand, is crucial to keeping it from destroying us quite incidentally.
(FWIW I don't think intermediate traces in LLMs, or indeed even LLMs generally, are what will get us to this point, but my feelings are instinctive)
That said... CoT monitoring is a fragile "intent discovery" mechanism; neuralese puts this problem front-and-center but if agents begin to learn to hide their intent from their CoT journals, we are basically in the same spot.
So, you mean, like another human person?
https://nonlineartransform.substack.com/p/relax-about-neural...
There's also an argument here for why its _better_ for monitoring (because we have the whole state space).
https://www.astralcodexten.com/p/elk-and-the-problem-of-trut...
It discusses "Eliciting Latent Knowledge" which is "a technical report / contest / paradigm run by the Alignment Research Center". The research investigated whether it would be theoretically possible to build a "trustworthy" AI to interpret the thoughts of another AI.
https://www.youtube.com/watch?v=KT4n-z_4QJU
I think there were experiments where seemingly relevant parts of the CoT were ablated and it did not change the result.
For all we know it might be somewhat human parseable neuralese.
Take anything they write with a big grain of salt. EA writings these are mere apologies. The conclusion is preordained. Authors start with the goal of slowing AI and work backwards from there, trying to see which arguments resonate with the pubic. You can't unsee it.
Not everyone in the AI space approves of these people or their doomerish.
https://www.anthropic.com/research/natural-language-autoenco...
If you want to monitor your model, you need to start with an inference provider that gives you the entire output and possibly even run it yourself to get access to the internal states. And if you think the KV cache and (when present) the recurrent state don’t encode a lot of “thought”, you are fooling yourself.
FWIW, I think most model architectures at least have the property that latent state can’t propagate from higher layers to lower layers by any route other than the output tokens. But even a two-iteration structure could be designed so that the last layer produces a vector that enters the first layer, once per token, and I bet it it would be very easy to train such a model to “think” in silence in the sense that the output tokens while thinking would all be one particular null token.
I agree with you about not trusting Big Ai. I'm amazed that so much of HN is repeating the same darling -> demon with OpenAi/Anthropic that we did last decade with other SV darlings. Humans are forgetful beings
Or is the neuralese some sort of irreversibly encrypted data set that only an LLM can "understand" and that can never be translated back to English?
https://www.astralcodexten.com/p/elk-and-the-problem-of-trut...