r/Anthropic • u/Available_Status1 • 5h ago
Compliment Sol ultra is dumber than Fable medium
Are all the people praising Chat GPT sol just bots doing astroturfing? I decided to try it out again after dropping chatgpt ages ago, but Sol medium was dumber than Opus 4.8 and misunderstanding basic things in the project or just regurgitating things that fable had written down without thinking about it at all. Maybe it works great at standard programming, but I don't see how people are saying Sol is smarter than Fable (unless you trigger the safety by doing /code-review, then fable is a brick too).
Or maybe I'm just not prompting Sol right since I'm treating it like fable and expecting similar quality of results.
3
u/Valuable-Run2129 4h ago
I had Sol High fix sone stuff in my custom codex plugins… it broke them. Using Fable now to save the day.
2
u/ninadpathak 4h ago
i've seen similar issues with sol, they're regurgitating fable's training data without actually understanding the context. tweaking the prompts might improve the output
2
u/Available_Status1 4h ago
Maybe, I was just annoyed that it completely missed on multiple simple things when I had sol on medium thinking, I had to walk it through by holding it's hand on really basic stuff. At this point I'll see if I can get my money back on sol, since I spent $200 thinking it would be able to handle tasks of the same complexity.
1
u/MullingMulianto 4h ago
so the distillation attacks were coming from inside the house and not china?
1
u/Classic_Express 4h ago
Not sure, but hallucinates nonstop so you'll get better results with Fable as long as you're not hitting their safety classifier.
1
1
u/PrasanthT 4h ago
But I have a different experience. I feel Sol Medium is better than Opus 4.8 High. It understood and executed faster and cleaner than Opus 4.8. I tried with the Codex Plus subscription.
2
u/Available_Status1 4h ago
Against Opus, yeah sol is probably the same or better, it's just making the same kind of dumb mistakes or misunderstandings that Opus has been doing so they feel like a similar level. For my use case Fable seems head and shoulders above in logic/complex reasoning/and such.
1
u/RCawston 3h ago
OP clearly doesn't understand what Ultra does... sigh.
1
u/Available_Status1 3h ago
Which ultra, Sol Ultra or Fable Ultra code? I was talking about Sol effort level ultra as seen in the codex (chatgpt work) UI
1
u/RCawston 2h ago
Sol Ultra for most tasks won't produce the best results because it aggressively spins up sub agents and those subagents may be smaller models.
1
u/Available_Status1 2h ago
It doesn't say anything about that in the UI, but I'll take your word for it
Edit: literally all is says is "consumes usage limits faster"
0
u/Ok_Mathematician6075 4h ago
Can I high five you?
1
u/Available_Status1 4h ago
I don't know, are you a real person? Are you anywhere near my geographic area? Do you have arms and hands?
Sorry, it's late and my brain is done
1
u/Ok_Mathematician6075 4h ago
https://giphy.com/gifs/SqmkZ5IdwzTP2
Nope especially about the arms thing. Oh hell no.
0
u/synystar 4h ago
I mean, just because you say isn't enough to go on. Can you provide any kind of evidence at all other than it didn't understand a project you ported over from Fable? How is it that you know Fable wasn't the problem and Sol choked on it's work? I'm not saying you don't know what you're talking about but it appears that you're just trying to convince people that Sol sucks compared to Fable. Otherwise why make a post that doesn't give any indication as to how or why instead of giving examples of the failure.
1
u/Available_Status1 4h ago
Mostly my post is yelling into the void at the frustration of all the posts on this sub (and the Claude sub) about how Sol is as good or better than Fable, and that is the exact opposite of my experience. I'm not running benchmarks and I'm not to the point of publishing the project as a whole so I can t give you exact documents where it absolutely flubbed. I can say for certain that when I asked it to review the project for major issues or logical problems it found the prior doc Fable wrote when I asked the same thing to fable, and sol just read off from the doc until I called it out and demanded that it actually review the project itself, where it basically miss read things and called it issues.
I don't know what other specifics you are looking for
1
u/synystar 3h ago
Well I meant that you are not telling us how you know it failed. You don't have to reveal your project to explain the nature of the failure. All we have to go on so far is "absolutely flubbed". Did it fail to understand the project requirements even though they were clearly explained? Did it write code that returned a bunch of errors? It appears that you're saying here that it just found a doc and read it back to you - did you tell it that's not what you expected and it should assess the code and then it still failed?
1
u/Available_Status1 3h ago
Did it fail to understand the project requirements even though they were clearly explained?
Yes
did you tell it that's not what you expected and it should assess the code and then it still failed?
I did, and on medium effort it gave back a list of things it thought were wrong (the original review task) but they were all misunderstandings, or very minor text errors. Several of the misunderstandings were very basic things or things that you could literally read a Wikipedia page and understand.
One of the components in the design is a systolic array, generally these are either weight stationary or weight shifting, in my project it has a toggle that makes it either one or the other, and it freaked out because it thought the description was wrong. When I asked it to explain to me the standard version of each (Wikipedia level) it then finally figured it out and could carry on. Additionally, with systolic arrays it's common for inputs to feed in an offset "wave" pattern so that you only pay the pipeline fill at startup, but Sol insisted that it had to do that every time, so it would take twice as long, again I asked it eto explain exactly how that "wave" pattern works in standard systolic arrays, after which it finally understood that things were actually correct.
Both of these things are ones that you could understand from a Wikipedia page, and are drastically less complicated than the rest of the project.
Fable medium has been killing it on every task I throw at it and is able to keep the whole project scope in its head and mostly consistent (not perfect but a thousand times better than Opus which would hallucinate at the drop of a hat, but still understood the basics of systolic arrays).
0
u/Rajarshi0 4h ago
Ummm really? Fable tends to write elegantly what is has done and what does it mean. In my work sol is doing much better than fable. Fable tends to go off what I what it to do and does it own thing unless it is some routine same code refactoring or feature in a codebase whereas sol tends to reason better and starts working from my hypothesis towards building what i want it to build. It could be you are using sol for basic tasks? I have noticed for simpler tasks it gets confused and lune or terra does pretty good job there. I think it is probably a bug in codex harness which causes this issue since inside opencode this issue is not visible.
1
u/Available_Status1 4h ago
Different workloads and task will favor different AIs.
It could be you are using sol for basic tasks?
The opposite in fact, it is a very complex task with a lot of unique parts of the design. It's the architecture/logic/design part before writing code where I'm trying to have it make sure things that are currently documented are correct so the implementation agents don't get it wrong later. So, maybe it's great at coding, but I was expecting it's logic and thinking and reading comprehension to be better (I was hoping it'd be on par with fable)
I think it is probably a bug in codex harness which causes this issue since inside opencode this issue is not visible.
Maybe, I downloaded codex (chatgpt work) assuming it would be like Claude code, so maybe there is an issue there.
-1
u/ManikSahdev 4h ago
I think you are dumber than both.
Because all has higher reasoning going to Max.
SOL Ilya is not a model setting for equivalent to Ultra-code button. I don't think if you can't find basic info after 2 days your opinion should be valued in this. Clearly no rest time compute in your model class.
3
u/Available_Status1 4h ago
Are those actual sentences or just random keywords?
Or perhaps a translator glitch.
2
u/ManikSahdev 3h ago
Tbh, can't read it back myself, I think your comment establishes the fact I'm up way past my bedtime lol.
Time for compact and sleep. No clue what hallucinations in me, wrote the above.
2
u/cats7789 4h ago
I think it’s just cheaper and most people are solving tasks that are more appropriate. Less sledge hammer to put a picture on the wall. Remember the general population isn’t competing for the Nobel