A very neat problem and result. I often find myself swinging between "It's so over" and "We're so back" - some days I roll out of bed thinking I could have Claude solve some random unproven OEIS sequence before breakfast; other days, I wake up in a cold sweat worried about the fate of humanity and what the world might look like in a decade. I think it's that I don't have a very high p(doom) or p(utopia), and I don't really have any solid conviction on how this whole thing is going to go, so my vibe-o-meter jitters between 'fine' and 'not fine' constantly. It's just such an unpredictable moment. Anyways: really neat to see this use case. I myself recently used Claude to finally do an relatively exhaustive study of the location of heretofore-unlisted formal gardens in Ireland in the early 1800s and early 1900s, by having Claude write the tooling for me to manually annotate a few dozen on tiles of historic maps, and then running some CV model across the rest of the tiles using my input. I'd been planning to do this project for over a decade, but I could never find the time (or the enthusiasm) to learn all the details of how to do it myself. It took me a weekend with Claude and continues to bring me joy.
> Caveats, stated plainly. [from the Fable transcript pasted in the article]
I've done something similar to your formal garden map. It's work that no professional historian would ever do because the data entry would be such a slog for a relatively small reward. GPT reduced the task from "infeasible" to "annoying", and once I had the data transcribed I learned a few things, so I walked away happy. Whatever happens commercially, these models have been a real boon to hobby projects.
> I told it to look online at some of Fable’s strongest feats, especially the math problems it has solved, and that something like this should be easy in comparison.
Wait. Wait wait wait. Are we supposed to be giving them pep talks?
on older gemini models ide have to actively give them encouragement and/or easy bait problems that they can correctively solve without issue to avoid runaway spiraling into "i'm useless and i want to kms" behaviour with complex use case.
That's an artifact of it not using A-Z as it's alphabet. What you type gets translated before the AI sees it. The seventeen word the AI sees does only contain three 'e' s.
Is this true? If so, does it mean anything? Sure, it is tokenized in processing (tokens don't really have an alphabet either), but this means it did not correctly parse the problem at all if this is the case.
Reading your comment, I asked it to count how many characters are in the word. It answered correctly: 9. It then also said it skipped an E in the prior chat and corrected it count to 4. It spun for 52 seconds so maybe some Python was involved.
I tried too. It got it. Maybe more importantly, who cares?
For example, I'm a nerd. I'm bad at baseball. I lack that kind of intelligence, even though it's more common than the ability to program. That doesn't also imply that you can't trust my Python code.
We found a cipher my dad had written as a child with no obvious key or anything. Chatgpt was able to crack it in 20 minutes and figure out the message, and we knew it was right because it mentioned names of children he went to school with.
> Historically, many of these problems were bottlenecked by human attention. Someone had to care enough to spend hours or days reading obscure material, testing unpromising ideas, tracing references, and trying things that might go nowhere
I wonder how many of the recent results are due to the fact that very few looked at the problem to start with. Still great results, but the general impression is that it's more about the so many low-hanging fruits than the actual capability.
> I told it to look online at some of Fable’s strongest feats, especially the math problems it has solved, and that something like this should be easy in comparison.
Fascinating. I wonder if you could show "fake news" to a weaker model and get it to be more ambitious in its attempted solutions, even if it's not fundamentally any smarter.
I forget the story, but isn't this the origin story of one of solutions to one of the hard problems in mathematics? The story goes that the student shows up late, and misinterprets the final slide to be homework, and it turns out that the professor was showcasing a hard problem. Thinking that the slide was homework, the student takes it home and solves it.
EDIT:
In 1939, George Dantzig was a graduate student at UC Berkeley studying under the statistician Jerzy Neyman. He arrived late to class one day, saw two problems written on the blackboard, assumed they were homework assignments, copied them down, and turned in solutions a few days later. He apologized for being late -- the problems had seemed "a little harder than usual."
I really love this idea given the recent controversy around mathematics solutions.
It seems like a "mere suggestion" of success has a positive impact on finding solutions.
We know this technique works in humans, from which this is all derived from, so it seems to make sense.
Its a bit sus since there doesn't really seem to be much discourse on this either. Like okay, it solved the puzzle but the puzzle was just a key cipher with plain text? And how is this verified or even matter in terms of what it reveals? Seems more like a marketing fun post than anything susbtantial.
They published this on 31 aug and nobody in that community cared and no news covered how this 300+ years mystery was solved?
I was watching Shatner's "Unexplained" the other day on this topic, and it hit me; there are mountains of these old mysteries out there that could be solved in an afternoon now with frontier LLMs as soon as anyone took the time to bother. Exciting times.
> Caveats, stated plainly. [from the Fable transcript pasted in the article]
I had a visceral reaction to these three words.
> I told it to look online at some of Fable’s strongest feats, especially the math problems it has solved, and that something like this should be easy in comparison.
Wait. Wait wait wait. Are we supposed to be giving them pep talks?
I have not seen this in other models.
That ability to create ad hoc tools makes up for a lot of shortfalls.
For example, I'm a nerd. I'm bad at baseball. I lack that kind of intelligence, even though it's more common than the ability to program. That doesn't also imply that you can't trust my Python code.
I wonder how many of the recent results are due to the fact that very few looked at the problem to start with. Still great results, but the general impression is that it's more about the so many low-hanging fruits than the actual capability.
Fascinating. I wonder if you could show "fake news" to a weaker model and get it to be more ambitious in its attempted solutions, even if it's not fundamentally any smarter.
EDIT: In 1939, George Dantzig was a graduate student at UC Berkeley studying under the statistician Jerzy Neyman. He arrived late to class one day, saw two problems written on the blackboard, assumed they were homework assignments, copied them down, and turned in solutions a few days later. He apologized for being late -- the problems had seemed "a little harder than usual."
They published this on 31 aug and nobody in that community cared and no news covered how this 300+ years mystery was solved?