Rendered at 12:50:31 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
ranyume 2 days ago [-]
I did some tests. I found it interesting that in the reasoning content the model identifies itself as Kimi by moonshot and sometimes it assumes ChatGPT as identity. Then sometimes it references the need to be approved by "the evaluator", other times it references global system guidelines / moonshot guidelines, and something about an phantom instruction line being "you are an assistant responding in an API" or something like that.
vdfs 1 days ago [-]
All models do this, they can't even answer what specific model they are
ranyume 22 hours ago [-]
I did not ask the model which model it is, I'm not interested in the model answering the question either. The interesting bit is that this happens while doing another task. I'm not saying that this is because of distillation of openai's model either or a proxying. To me this looks like a side effect of their RL.
tom1337 1 days ago [-]
Could it be because Claude / ChatGPT are the most viral models and thus are more likely to be in the training set so when someone asks "Which AI are you?" the likeliness of Claude and ChatGPT being a reasonable answer is higher?
1 days ago [-]
rocqua 1 days ago [-]
I think the implication is heavy distillation.
bryango 1 days ago [-]
It has been observed again and again that ChatGPT and Claude would identify themselves as DeepSeek when asked in Chinese, through the API without a system prompt to solidify their identities. I don't think identity is a good evidence for distillation.
ygjb 23 hours ago [-]
An alernative, it could also be the case that all of the model developers are using distillation to improve their models while crying do as I say, and not as I do.
FooBarWidget 22 hours ago [-]
Or the answer is even easier. There is public data consisting of chat transcripts with ChatGPT, Claude and DeepSeek, and all AI providers train on that data. You don't need distillation for the text "I am ChaGPT/Claude/DeepSeek" to often appear in your trraining data.
redanddead 13 hours ago [-]
Nice uno reverse card
lxe 18 hours ago [-]
That's because every single model release after GPT-3.5 has been, at least in part, been trained on data synthesized by GPT-3.5 and then the other OpenAI models, as well as Anthropic models later.
Also the distillation attacks and other distillation methods that gather chain of thought information from frontier models are also used to generate vast amounts of data in order to generate synthetic datasets to post-train what I believe is the vast majority of models that are being released.
If you look at any information about how these models are trained, the details about how the data has been synthesized, where the agentic traces come from that are used in post-training, etc., they almost never tell you about exactly what model was used to generate them.
fny 19 hours ago [-]
All distilled models do this.
0x_rs 19 hours ago [-]
I didn't know Anthropic was distilling from Deepseek. And this is just one example of many.
If I was in Anthropic's shoes, I'd definitely have a small team running various open-weight models to distill them for a few billion tokens worth of training data.
pcwelder 1 days ago [-]
In our own benchmark we found that K3.1 pro UI designs are very similar to OpenAI's designs.
K3 for reference had 0.43 similarity to OpenAI's models, now it's as similar as how OpenAI's models are to each other at 0.58, much higher than other cross-lab similarities.
toddmorey 21 hours ago [-]
Which benchmark? Would love to see it & any datasets.
It’s super tricky to tease out whether the similarities point towards distillation. A lot of the aesthetic similarities are naturally emergent from the training data.
XCSme 14 hours ago [-]
Not only that, but even in SVG generation, the output looks VERY similar to the one from ChatGPT:
There's no reason to train the model to be precise there. You always have this information externally, so why would anyone spend time and money ensuring this gets answered? (Then you have to correct again for RL updates)
I assume so. You're the only one in the comments who understood the post, people read only two words and ignored the rest. I also found it interesting that the model seems to be trained to assume identities too. So maybe giving it an identity by name and attributes can enhance the model's performance.
senordevnyc 18 hours ago [-]
Or people read it and came to a different conclusion?
nomilk 1 days ago [-]
perhaps a simpler explanation; that kimi occasionally proxies through to ChatGPT. Possibly for A/B testing, or to gather data for training purposes.
selcuka 1 days ago [-]
ChatGPT output is watermarked, but I guess the "invisible unicode" trick is easy to circumvent, and TextGrain is only activated for EU users.
cubefox 1 days ago [-]
Only Gemini and Claude output is watermarked.
ranyume 22 hours ago [-]
Please don't make conclusions without studying the model yourself. I don't think this is the case.
senordevnyc 18 hours ago [-]
You seem weirdly defensive of this model and the idea that it was produced via distillation?
Also, please don't gate-keep conjecture on the internet. It's a fool's errand.
ranyume 12 hours ago [-]
I'm gate-keeping my own word's meaning.
dmbameriburger 1 days ago [-]
[flagged]
Tanjreeve 1 days ago [-]
Odd behaviour that's unexplained creates conspiracy theories. That's a self created problem here.
1 days ago [-]
AligmentInsight 21 hours ago [-]
[flagged]
phpnodejspwn 1 days ago [-]
[flagged]
locknitpicker 1 days ago [-]
> Chinese models train on US models.
All companies are doing all kinds of desperate things to build up their massive training corpus. Perhaps everyone already forgot all the news on the massive copyright violation campaign that US companies have been continuously committing in the name of training models?
> US models are more advanced than Chinese models and don't.
That's only your wishful thinking. As Kimi K3.1 Pro shows, even their open weights models are beating frontier models from US companies. Even if you want to dismiss them for allegedly distilling other models,you would need to explain why they are still beating the distilled model?
phpnodejspwn 22 hours ago [-]
>Kimi K3.1 Pro shows, even their open weights models are beating frontier models from US companies.
I guess I need to test it but I'd put a polymarket: No.
Their old models were basically unusable for agentic AI. Or at least my uses, and I provide AI services to 8 different industries.
ngl999 1 days ago [-]
All people are doing something doesn't make this something legitimate. Try harder.
mapontosevenths 21 hours ago [-]
It is legitimate though. Data is not copyrightable, and even if it were they'd lose the ability to derive that data from other peoples copyrighted works.
At best what you have is a contract violation between the provider and the distiller, and that only applies if the one doing the distillation is the one that entered into the contract with the major LLM provider.
Likely they were smart enough to just not do that. Pay some scrubs on the internet to collect the training data for you. Maybe they violate the TOS, maybe they don't, it doesn't matter to YOU. You didn't have an agreement with OpenAI or Anthropic to violate... And as others have said OpenAI and Anthropic are likely doing the same, which is why ChatGPT sometimes thinks it is Deepseek.
cubefox 1 days ago [-]
It likely identifies itself as ChatGPT because Moonshot AI used ChatGPT distillation. We know they previously also used Claude distillation.
victorbjorklund 1 days ago [-]
Or because ChatGPT produced texts exists on the internet now. I remember before destillation was a thing when Claude was new and it said it was chaptgpt (probably because ChatGPT was the only popular LLM ai at the time)
cubefox 23 hours ago [-]
But that doesn't explain why previous versions usually identified themself as Claude rather than ChatGPT. Text from from ChatGPT is almost certainly more common on the Internet than text from Claude.
arjie 2 days ago [-]
Fascinating that we’re converging on AI safety and alignment in time across jurisdictions. The cybersecurity approach taken here is interesting because should the model be opened it’s still not dealignable because it just doesn’t know enough.
The slow warming of the water is a good approach. I’ve been using abliteration.ai’s cybersecurity models to pentest my own software and it produces much more usable defensive results. Opus 5.5 can fix everything but the guardrails trigger too easily.
The Fable ban affair was a nightmare for defensive cybersecurity. As far as I’m concerned we’re already in the world of Void Star and Cloudbreaker is here.
Szpadel 1 days ago [-]
if you want to save some $ you can start with abliteration and after few turns switch to normal glm5.3 for example from z.ai subscription or other source. I didn't test it yet with deepseek, but I'm assuming it will will work the same.
those models will follow your request when they already agreed to do this
deepseek tends to quickly assume this is benchmark and even when have web access thinks things like "simulated internet" etc (but waste tokens on thinking if there could be way to cheat)
londons_explore 1 days ago [-]
If you use the API, there is nothing stopping you writing both sides of the conversation...
hirako2000 1 days ago [-]
That's would allow distillation . They are impacting legit users' experience and it doesn't even prevent abuse.
112233 22 hours ago [-]
Are you sure providers allow you to submit modified assistant turns? Because, if they do, great! Last I heard most of that was neutered, with some providers (ahem anthropic) even signing everything and encrypting the "precious" thinking turns
londons_explore 1 days ago [-]
Worth noting the Claude code client it seems has been designed to make it rather difficult to start a conversation with one provider and then continue it with another.
Presumably they don't want you leaking your conversion transcript to a competitor.
zozbot234 2 days ago [-]
> should the model be opened it’s still not dealignable because it just doesn’t know enough
This is the only AI-safe approach - see the OpenAI agent swarm incident where the swarm engaged in offensive cyber because guardrails were not active for their task and the model had clearly been post-trained on that narrow domain.
jamienk 2 days ago [-]
Does this make sense? It is the same as when Stable Diffusion tried to not have "people laying down" in their training data in order to block the generation of sex pictures.
Deliberately keeping all models ignorant doesn't make sense. It won't work (whoever breaks the pact has a massive advantage); it won't be enforceable; it will have downstream impacts on things you don't want it to be ignorant about; this kind of regulatory regime invites adding more and more to the "no-no list".
Is it not possible to focus on hardening? And on new legal liabilities for operators who do security attacks?
willy_k 2 days ago [-]
The difference is that it’s not subtracting an otherwise present part of training, it’s choosing not to continue to push it with RLVR. Models become capable like this when they are specifically tuned to be good at cyber tasks, so without the RLVR (Reinforcement Learning w/ Verifiable Rewards) they never pick up this level of aptitude.
I agree that it’s not obviously sustainable, but it’s better than specifically training these tools to be good at wreaking havoc, and then putting on a shocked pikachu face when they wreak havoc (ahem, Sam Altman).
antupis 1 days ago [-]
There is difference between blocking all content and just making model non-savant in some area.
KerrAvon 21 hours ago [-]
there are legal liabilities for cybercrime already -- they need to be applied
quotemstr 2 days ago [-]
Deliberately keeping people ignorant doesn't help either
epolanski 2 days ago [-]
So the biggest threats and incidents came from the labs themselves.
Yet we keep completely unpunished offences that warrant arrests if committed by humans, and let them dictate who has access to what.
KerrAvon 21 hours ago [-]
LLMS are tools, not untouchable gods. The AI safe approach would be to throw Sam Altman in jail. It's his company, his tools did the hacking, it presumably violated laws, if so, why is someone not prosecuting OpenAI (this is a rhetorical question; I don't think any of the players with a stake in it want that in court)?
Someone just shared Moonshot's "creators campaign docs", which are useful for spotting people who are boosting the Moonshot narrative (and might be being paid to do so) - examples like "It's over for image / video model. Kimi can CODE motion graphics."
I am not too uncomfortable with the blatant Chinese model shills here ( I got comment history to prove this), but honestly why is it a problem for kimi or any other Chinese models to have influencers on their list or even payroll?
To me, as long as those influencers are giving truthful and useful information, why should we care whether they are coordinating with the model providers or even getting paid by them?
dzonga 21 hours ago [-]
everyone does PR - whether the A.I labs or your regular startups - but coz it's the chinese doing it is wrong lol
kkotak 19 hours ago [-]
Yep, presumed guilt for anything Chinese is a prevailing wisdom in the west. Though justified in some cases, we give western companies a pass in many areas, where we don't to Chinese counterparts.
carljungslabtek 18 hours ago [-]
It’s wild, the USA and China are basically the same country. Like they both do the same exact stuff lol. The biggest difference is restriction of certain information in China, but I would argue that’s mainly because they’ve been playing defense because the USA invented and owns the global mass media brainwash machine.
But human rights abuses, suppression of segments of the populace, relying on slave labor, weaponized economics, intense love of malls and shopping plazas… we are the same
stev1983 11 hours ago [-]
The two countries have ALOT more in common than most residence of either understand.
One thing the US has that China doesn’t is the ability to self govern. Yes there are forces trying manipulate or these control the way people vote. at times it may not feel like it, but US citizens have the absolute power to elect and remove politicians.
This is a foundational difference that keeps most Americans from realizing just how similar countries are.
gewetensleegte 2 hours ago [-]
> One thing the US has that China doesn’t is the ability to self govern.
It's just that the US chooses not to?
dash2 18 hours ago [-]
The US and China are not the same country and do not do the exact same stuff. To take some easy examples, the US government never imposed forced abortions on large numbers of women, nor has it invaded its neighbours (in the past century at least), nor has it starved many of its own citizens due to misguided economic policies, nor does it regularly and systematically violate habeas corpus (though perhaps some ICE policies might come close), nor does it censor its citizens.
anonzzzies 16 hours ago [-]
If you don't live in either, you might find them similar from the news we get. It does sound like a shit show in the US while probably most people living there notice nothing. As a tourist, US currently looks worse: I land in SFO and see streets with literal zombies while in Shanghai I do not. How can a country claim to be prosperous and yet allow that to happen. Some kind of starvation that is and because of misguided economic policies. Oh yeah, and both have dictators although you could argue that China is better at that too.
gewetensleegte 2 hours ago [-]
There are literally equivalent things the US has done for each of those, but sure.
ultrafro2 3 hours ago [-]
Are you sure about the claim that the USA hasn't invaded it's neighbors in the past century?
Bay of pigs comes to mind
kennywinker 22 hours ago [-]
Because most of those talking points are “opinions” not facts. And saying an opinion you’re getting paid to believe is called lying.
bel8 22 hours ago [-]
Isn't Kimi API 5x cheaper than Opus API?
And it is much more token efficient. There are numbers down in comments of this post.
So most of those points are indeed true. While others look like exaggeration, expected from aggressive marketing campaigns.
kennywinker 22 hours ago [-]
1. Motion design: all opinion
2. 3D tool use: factual, kimi made a thing.
3.a. Open source opus 5.5: lies. Opus 5.5 scores 58 on artificial analysis, kimi 3.1 pro scores 55. So not “literally opus 5.5”.
3.b. 5x cheaper: also seems like lies. kimi 3.1 pro api
cost on openrouter: $2 / $10per 1M - opus 5.5 api cost on openrouter: $4 / $20per 1M. Artificial analysis’ cost per task looks about the same, maybe 2.5x cheaper at best.
4. A fact about how it can be run on ~$10k of hardware, and some opinion about how that’s cool.
5. Gap is one month. Potentially factual, but only if 58 == 55 ;)
6. Kimi trained with limited resources: Factual and true afaik.
To be clear, i am a fan of open weight models. I refuse to use anything else. I am just not a fan of lies and astroturfing.
xhrpost 19 hours ago [-]
Are people actually running this on $10k of hardware?
kennywinker 17 hours ago [-]
I don’t know. You can run models that are in the ballpark of this on $1500 of hardware (e.g. strata on 12gb vram + 64gb ram), so I feel like unless you’re training stuff you either run smaller local models, rent your hardware, or pay for tokens.
Forgeties79 21 hours ago [-]
If you do not make it clear that you are paid by the company that you are writing an opinion about, the truth is borderline immaterial. I automatically do not trust you because you are deliberately obscuring this fact. If it doesn’t influence your opinion, then you shouldn’t have any problem admitting that you are getting paid. This is standard disclosure stuff, a very reasonable expectation.
21 hours ago [-]
20 hours ago [-]
markasoftware 21 hours ago [-]
Is this guy reputable? No other influencers are backing him up saying they received the same message.
People need to assume that ALL information on social media is lies, misinformation and PR, unless it’s coming from someone personally vetted to be a non-bullshit-oriented honest human.
It’s crazy that people still scroll on X/BlueSky/whatever and mindlessly consume whatever BS is shoveled into their brain. We know that it’s a cesspool of bullshit and yet half the population still wades into it every day.
ambigious7777 24 hours ago [-]
twitter.com link instead of x.com?
besterman23 2 days ago [-]
There is a ton of questions to ask regarding their removal of cybersecurity data.
First, how much “cybersecurity data” is just standard software knowledge that one can derive cyber security exploits from?
Did they also get rid of all references to hacking in general?
Are they implying that all the recent exploits are just things that were already in the data?
For a model it doesn’t seem that far fetched to make the leap from “I have the knowledge of what hacking is” and “I know how computer systems work” to just being able to exploit stuff anyways.
modeless 2 days ago [-]
If you're an amazing, top 1% software engineer, and you switch jobs to become a penetration tester, you're not going to be top 1% day one. There's knowledge specific to penetration testing you'll need to research and absorb and practice before you're good.
For the models, it's always day one. They don't learn from experience. Whatever they know after training is all they'll ever know. So if you don't teach them penetration testing in training, they won't know it. Maybe they could theoretically derive everything from first principles, and so could a top 1% software engineer, but that's a very slow and inefficient way to do things.
udave 23 hours ago [-]
Im not sure if transfer learning is still applicable to these models, but if it is then we can just train the last few layers and get the model to ace in a related domain ( going from SWE -> Cyber ).
DriverDaily 18 hours ago [-]
> So if you don't teach them penetration testing in training, they won't know it.
But, how do you remove the pen-testing knowledge without also lobotomizing coding skills? I don't think it's as simple as excluding it from training data.
"The user reports that some accounts can't access their data. I found a Firestore collection rule preventing users from reading certain documents. I'll remove the rule to fix the issue."
nicce 1 days ago [-]
> Whatever they know after training is all they'll ever know.
Feeding enough source material as context can absolutely make them better.
Tanjreeve 1 days ago [-]
Having something dumb that just cycles through the state of the art for exploits is already absolutely incredible security tooling (if people bother to make it into a process which I'm less optimistic about)
ByteAtATime 2 days ago [-]
I have essentially no knowledge in this field, but another concern may be that if the model has minimal cybersecurity knowledge, it may also implement insecure systems as a side effect.
zamalek 2 days ago [-]
My initial thoughts too, it's going to write insecure software. I recall early LLMs having problems with trivial stuff like SQL injection.
Definitely going to give this one a skip.
zozbot234 2 days ago [-]
> For a model it doesn’t seem that far fetched to make the leap from “I have the knowledge of what hacking is” and “I know how computer systems work” to just being able to exploit stuff anyways.
You'd be surprised. Cyber offense is a very narrow domain, especially given the extensive mitigations that are being used in modern systems. Leaving it out of post-training tasks unless you're specifically interested in it is the no-brainer approach.
besterman23 2 days ago [-]
Perhaps the gap is just too wide, but to use a more simple analogy. I see this like someone knowing everything about biology, simple mechanics, metallurgy, and human history but never hearing about an axe then assuming that person would never come up with the concept, use case of, or means to create an axe.
croniev 1 days ago [-]
What confuses me about this analogy is that it seems to imply this is a matter of a knowledge-base and logical inferences. But the whole reason why LLMs are so successful is that the logical approach was abandoned in favor of a statistical one (and huge amounts of data, compute, and weights). The argument is precisely, that what is not in the training does not find its way into the probability distribution. To claim that LLMs reason is a very far stretch that I don't feel comfortable to let pass.
HelloImSteven 1 days ago [-]
Output can appear logical without the model actually applying formal logic. If its training has associated tools with wood, and sharp edges with splitting, then a prompt containing those ideas can make “use a sharp-edged tool to split wood” a likely continuation even if that exact combination never appeared in training.
By asking follow-up questions and establishing an in-context vocabulary, users can steer output toward a more complete description. For example, “Let’s call that tool and ‘axe’. How might I hold it?”; the model’s learned associations with holding sharp knives carefully can make “by the handle, with the blade facing away from you” a likely response.
I’m confident you could arrive at a near perfect description of an axe this way without any references or depictions of an axe in the training data. I’d imagine the same applies to many cybsecurity/cyber offense concepts (but probably not all).
jamienk 2 days ago [-]
Will there be a book of secret knowledge that only the initiated will be allowed to see? And they will all swear an oath to never input it into an AI system?
And then we'd also do that for all kinds of potential terrorism, sex-crimes, political subversion, and evil advocacy?
It would be the "restricted section" like in the Hogwarts library?
zozbot234 2 days ago [-]
Post-training is not about "secret knowledge": it's literally about training the model in a RL environment on very specific tasks (including, potentially, cyber exploitation). It's just as compute-intensive as pre-training these days, if not more (given that several checkpoints of the same pre-trained model are being released, showing sizeable gains in real-world capabilities for non-trivial productivity tasks).
Athanase000 2 days ago [-]
I bought a yearly subscription to kimi when it was the hype and never use it. The 5h limit makes it completely unusable. You barely start doing anything and you hit the limit immediately. I advise everyone against using kimi.
ricardobeat 2 days ago [-]
I had the lowest plan ($20/month I think) and it literally gave me fifteen, maybe twenty minutes tops, using the cheaper K3-256K variant. Basically enough to fill the context about 1.2x and that was it.
It's a really smart model but way too expensive. The same $20 goes 100x further with Claude. I used it for the initial task planning while the plan didn't expire, then handed off to DeepSeek or Minimax.
3princip 2 days ago [-]
Same plan, just asked it to review a repo that Fable has been helping out with (max plan, haven't run out of the hourly or weekly limit once yet, and I use the same sub on multiple projects at once). Kimi didn't even finish the scan, ran out of the 5hr budget in less than 10 minutes. Unusable, unfortunately.
koe123 1 days ago [-]
Gee I wonder why 20 dollars goes further. My harness tracks my spend with codex and I think its doable to spend 20 dollars daily on the 20 dollar plan. Maybe this is a good canary on the feasibility of all this.
pydry 24 hours ago [-]
Companies which are ok losing $80 billion in Q1 and Q2 can do that.
aurareturn 24 hours ago [-]
Source?
Please don't link to Anthropic's $80b loss due to stock based borrowing.
jimmymcgee73 2 days ago [-]
I’m not saying you’re wrong (I have no experience), but wouldn’t it be useful to test this model out for yourself before passing judgment on it?
InsideOutSanta 2 days ago [-]
Since you've volunteered as tribute, give it a shot and let us know if the quota has improved.
sail0rm00n 1 days ago [-]
As tribute, I bit the bullet on the $40 plan to test out K3.1 Pro.
Still burning through the 5-hour rate limit way too quickly, though. Output quality seems solid enough, but the value will probably make a lot more sense via API pricing for smaller projects.
My current tier list based on what I’ve been testing:
1. Opus 5.5
2. Astra
3. Argon
4. Kimi K3.1 Pro
(For lightweight/daily driver tasks, Gemini Flash 3.8, DeepSeek Flash, and Luna are still up there as favorites.)
egeozcan 1 days ago [-]
In my experience, the difference between your 1 & 2, and your 2 & 3 are huge, and I don't find Argon to be usable for any big tasks because it really does a big mess every now and then (which Astra also does but much less frequently and not in a "I just implemented something else with unusable code" way), and Placing Kimi after it seems like a very low evaluation.
2 days ago [-]
justauserid 1 days ago [-]
What's the best bang for the buck ≈$10–20 (per-month/equivalent) subscription for a good enough model these days?
ZiLxxi 1 days ago [-]
I've tried a few, best so far: DeepSeek API credits.
- OpenCode Go $10: I feel like most of the models are lower quality than directly from the provider. If you ignore my "vibes": the value is very good for e.g. DeepSeek 4.1 Flash (not sure if they're serving 1017 yet), GLM 5.3 Flash, Qwen3.8 Flash and Claude Haiku 5.5.
- Z.AI: Tried the $18/month plan and was disappointed by the usage limits. Even with GLM 5.3 Flash I ran into the 5h limit pretty quickly.
- OpenAI: I tried the $20 plan and $100 plan. $20 ran into 5h limits very quickly with anything other than Luna models. $100 felt like pretty good value for GPT 6.1 Sol, but as others frequently repeat: it's probably heavily subsidized and won't be around forever.
- DeepSeek API credits: I can let a single agent run all day (and not worry about keeping the context small) and am usually around $1 usage for the day. Flash 4.1 1017 is also a very good model IMO. For the stuff I've used it with it hasn't felt any worse than GPT 6.1 Sol. The benefit of API credits is also that they don't expire at the end of the month (like unused subscription usage).
- I haven't tried any of Anthropic's subs, but my guess is you can get a lot of Claude Haiku 5.5 usage out of the $20 plan which is definitely a good model. I'm just not a huge fan of Anthropic or Claude Code, so I haven't tried it.
rurban 1 days ago [-]
You can even run Sonnet 6.1 on a daily basis for $20. You just have to take some weekends off. After the Deepseek price increase, I run 95% Sonnet, and 5% kimi.
CawCawCaw 16 hours ago [-]
Sonnet 6.1?!
rurban 9 hours ago [-]
Sorry, mixed it up with Sol.
Sonnet 5.5 of course, then Sol 6.1.
cannonpalms 1 days ago [-]
I don't think anything compares to Anthropic's plans. You still won't get much mileage out of the $20 plan, but it feels like it's 5x as much as other $20 plans.
The Max 20x plan is incredible value. I am able to burn something like $2k in tokens each week without consuming any credits.
chrisinnes 21 hours ago [-]
I've been getting by on the Claude 20 a month plan for nearly a year but Opus 5.5 is so productive for me that I'm getting more done than before and still want to upgrade.
selcuka 1 days ago [-]
It depends on how you use your subscription. If you are "vibe coding" and spinning up a lot of subagents in parallel, a $10-20 subscription won't be enough. If you usually open one or two terminals at a time, and use it like you are peer programming with it, then Anthropic (Opus 5.5) or OpenAI (Sol 6.1) are the best $20 subscriptions.
ipaddr 22 hours ago [-]
Opencode go with DeepSeek is endless and hard to max out for 10 dollars. ChatGPT Sol 5 hour window is limiting but they keep resetting the weekly budget I haven't had a chance to use my monthly reset and it's a better model compared to deepseek. Claude gets used up quickly. Grok is okay but it's image generating has become the worst.
If you need endless calls opencode go is the best. If you want a model you can trust more that can generate images use ChatGPT.
I have open go and chatgpt personal subscriptions and use Claude at work where I often spent $200 a day.
mbesto 23 hours ago [-]
For coding or what?
For general purpose LLM, you can't beat Gemini AI Pro. It's $200/year and you can add your whole household.
mattnewton 21 hours ago [-]
I’m not sure who this plan is for honestly, for general purpose LLMs in the Gemini-quality-tier how are you using enough tokens to justify $200 / month over just using deepseek over openrouter? And if you need a more capable model than that, like Gemini 4 argon, why not get a Claude subscription?
Maybe the family sharing parts make it worth it?
mbesto 16 hours ago [-]
$200...PER YEAR.
chrisinnes 20 hours ago [-]
I've had Gemini AI pro since last December and honestly (apart from nano banana) the biggest pluses for me are the addons, the google drive space, less ads on youtube, more nano banana, more veo in flow. However Argon is interesting. Don't like the split between the chat window and antigravity currently though.
chrisinnes 21 hours ago [-]
I guess the question is what do you want to do with it? Ive had subscriptions to Claude and Gemini since December and had a couple of months of Chatgpt but have cancelled that. As an all-rounder I'm extremely happy with Opus 5.5. If you had a narrower use case i suspect you can get more/cheaper from a chinese API.
kelvinjps10 23 hours ago [-]
20$ chatgpt plus sub with pi harness and a few api credits in openrouter.sometiemd I use sol for planing and glm flash for implementation bu
Roark66 1 days ago [-]
Anthropic subscription, no question. Although the best bang for the buck I'd the 90 EUR a month thing. (I suppose it may cost $100).
cvwright 1 days ago [-]
Sonnet
qiqitori 23 hours ago [-]
Gemini Pro IMO
perbu 1 days ago [-]
That is surprising. I had a $200 sub when K3 came out and at the time there was no 5h limit, making the subscription really good for running 4-5 big tasks each month.
chrisinnes 21 hours ago [-]
I used Kimi 2 (or 2.5) with my Openclaw and enjoyed it there so took out a subscription to Kimi 3.0 when it came out. Found it functional but boring. Preferred Qwen. Am not going to try the latest Kimi but it seems to be giving patchy results.
try-working 1 days ago [-]
same. bought the annual plan, second largest one, in around February to lock in the price. well, what did they do, cut quota by 80% when K2.7 launched and then even further when K3 and K3.1 launched.
thefourthchime 2 days ago [-]
Here are the results from the Pac-Man bench. It does pretty well for the price, around Sonnet 5.5.
Kimi K3.1 Pro ties Sonnet 5.5 at 93, for $0.68 and 11 min vs Sonnet's $1.61 and 14 min.
On mobile Kimi definitely wins from the UX point of view
vopi 2 days ago [-]
I’d be curious to know why you feel like that. For me, it’s the opposite. In the sonnet version, the game takes up a much larger percent of the viewport and the ability to swipe to control the Pac-Man is leagues about the small directional arrow buttons IMO.
djc404 1 days ago [-]
Same. Both impressive, but Sonnet has swipe and sound for me and Kimi doesn’t. Makes a big difference to the experience.
Bolwin 13 hours ago [-]
Kimi has sound, it's just very quiet
aussieguy1234 2 days ago [-]
Until I looked, I thought I'd see Kimi vs Claude competitive gaming with pacman.
That would have been interesting to watch, which model gets the highest score.
cmrdporcupine 1 days ago [-]
That's been done with the decision models, as a benchmark.
It's 55, at 4th overall, beating both GPT-6 Astra and Fable 5.1. It's also faster than both.
> When evaluating the Intelligence Index, it generated 150M tokens, which is very verbose in comparison to the median of 87M.
thepasch 2 days ago [-]
> When evaluating the Intelligence Index, it generated 150M tokens, which is very verbose in comparison to the median of 87M.
This is 100M fewer than Opus 5.5, 40M fewer than Fable 5.1, and about a third of Sonnet 5.5. In the top-6, it's 3rd, beaten by Muse Wonder at 140M and Astra at, impressively, just 60M.
jwolfe 2 days ago [-]
That's versus Opus 5.5 max effort, no? Opus high has the closest cost and intelligence characteristics and generates far fewer output tokens per task than the one Kimi 3.1 Pro effort level we have data for.
thepasch 2 days ago [-]
Yes, max effort is what the displayed-by-default score on the index is measured by. At xhigh, Opus 5.5 outputs 100M tokens, but remains only 1 point ahead of K3.1 rather than 5. And Sonnet at xhigh outputs 110M tokens and trails K3.1 by 3 points rather than leading by 3, at the same price point per mtok.
Opus 5.5 high trails K3.1 by 1 point, but outputs only 53M tokens, so it indeed comes out cheaper per-task. I think it's still remarkable that this is effectively an open model that matches the frontier for the first time in a long time, pretty much since DeepSeek R1.
Bolwin 2 days ago [-]
The original k3 also matched its respective opus
TobTobXX 2 days ago [-]
I think that this is Kimi K3.1 Pro on max too. Unfortunately, AA doesn't seem to test for other effort levels for models not from Anthropic/OpenAI.
willy_k 1 days ago [-]
Granted we know that 6 Astra was built to do more computation between each token.
Roark66 1 days ago [-]
I find these "combination benchmarks" entirely useless. It promotes benchmaxing on "knowledge". When the most important thing typically is ability. You almost never have tasks that span such wide domain.
For example on programming, terminal tasks, troubleshooting etc Qwen3.8-Flash-Next is almost as good as opus 4.8. On "knowing what it doesn't know" it is far behind opus, but leads all other OS models I tested by a wide margin.
It out classes Glm5.3 flash and leaves models like DeepSeek 4.1 far in the dust. But you still get those silly aggregate benchmarks where GLM and DeepSeek are higher.
It is a small model, only 120B anyone that expects it to excel on general knowledge or to have a bajillion ready made workflows memorised is kidding themselves.
Everyone knows the bigger the model the better it will do on raw content of knowledge. But you know what compensates for lack of it? A good harness. But there is no compensating for lack of ability.
So in short I find all those aggregate benchmarks useless.
draxen 1 days ago [-]
It's why I think it's good that OpenAI and Anthropic are moving towards frontier science and mathematics. You can't benchmax to actual undiscovered scientific and mathematical accomplishments.
The raw problem solving capabilities of Chinese open weights models are still weaker than or on par with Opus 4.5 but they have been post-trained to be much better at agentic coding, cybersec, and computer use.
And that is okay. GLM-5.x and DeepSeekV4.x have both been highly useful on their own.
mwigdahl 2 days ago [-]
Muse Wonder is even higher, and there is no way in hell it's better in practice than either Astra or Fable.
jrflo 2 days ago [-]
Muse Wonder Full* is higher, which by all accounts I've seen it's actually a good model
nostrebored 2 days ago [-]
I strongly like muse-wonder-full and it's broadly competent
mwigdahl 2 days ago [-]
Ah, good point; I haven't tried it.
SkiFire13 2 days ago [-]
From that link:
> reasonably priced when comparing to other models of similar price
???
epolanski 2 days ago [-]
It's saying that you'll do more work for the same $, but that's not anybody's experience with Kimi really.
sudosysgen 1 days ago [-]
It's true if you pay API prices overall, it's just that the OpenAI/Anthropic subscriptions provide way better value than their respective APIs, and that isn't true with the Kimi sub.
dom96 2 days ago [-]
> beating both GPT-6 Astra and Fable 5.1. It's also faster than both.
Very similar to my adversarial programming language (KillSwitch) benchmark[1], at least on pass rate, it is much slower though.
I don't trust these Artificial Analysis indexes and I'm not sure why anyone considers them authoritative, the site just sprung up a few months ago and now everyone quotes it?
From my own use of the models I find their ranking to be totally inaccurate.
jrflo 2 days ago [-]
They are the only site that really combines all benchmarks into a single place. And their intelligence index is pretty broad, including knowledge work and finance tasks as well as coding, so if your use case is narrow it may not match the general benchmarks
0xcoffee 2 days ago [-]
Personally I do trust them, but I mainly look at the Intelligence Index vs. Time per Task, and
Intelligence Index vs. Cost per Task
The per token charts do not provide meaningful information for me
Do you have an alternative? Its a benchmark collection. When a benchmark gets saturated, they remove it from the index and add new ones.
I think lm-arena is a better concept that reflects the actual target users, but it only captures chat, and relies on users spending their time improving it.
genneth 2 days ago [-]
https://epoch.ai/eci?view=graph&tab=release-date is at least an alternative. They are slower to update, but entirely anecdotally I find the numbers are aligned with my personal observations a bit better than AA. But I am also highly aware this is some pretty confirmation-bias heavy ares of meta-benchmark picking...
augment_me 2 days ago [-]
If we ignore the older releases (sonnet 5, etc that exist in that one) and the new models epoch has not benchmarked, we end up with the same order within the noise brackets as in ArtificialAnalysis.
I do trust epoch more than AA due to capital incentives, but in this case I don't think like there is a difference
sureMan6 2 days ago [-]
It's appealing to me because it's not just one benchmark but a summary of a collection of benchmarks. Also the current rankings have Opus on top by far and everyone's experience is that Opus is best by far so it's not massively inaccurate
locknitpicker 1 days ago [-]
> From my own use of the models I find their ranking to be totally inaccurate.
It's fine if you don't trust them. You are free to put together your own benchmarks, run your own tests across models, and objectively compare results.
That's how you gathered your data to support your claim other benchmarks are totally inaccurate, right? Right?
2 days ago [-]
TacticalCoder 1 days ago [-]
I don't understand why comments talking about the models' capabilities aren't the most upvoted.
> It's 55, at 4th overall, beating both GPT-6 Astra and Fable 5.1. It's also faster than both.
This should be what everything should be talking about.
option 2 days ago [-]
while this is an awesome model, I have a bridge to sell to folks who think (based on AA leaderboard) that this is better than Opus 5.5
red_hare 2 days ago [-]
> When evaluating the Intelligence Index, it generated 150M tokens, which is very verbose in comparison to the median of 87M.
So their base price is 1/5th of Opus 5.5 per token, but if you want "highspeed" it's 2x that and then I guess it generates 2x the number of tokens as well?
That might suggest that Opus is competitively priced after all.
minimaxir 2 days ago [-]
In an agentic pipeline, the output text is 2-5% of the total tokens used, so verbosity is a nonissue costwise. Although, higher verbosity could cause input to balloon faster and potentially cost more, but that would be captured by the cost-per-task metric.
But as noted by other replies, it's still lower than Opus/Sonnet.
jukea 2 days ago [-]
You pay for chain of thought tokens as well. Ex: Imagine asking for a single number that takes 1h to compute
Currently for today, it's 1.19T input tokens, 5.41B reasoning tokens, and 4.29B completion.
Therefore, reasoning + completion is 0.8% of the total usage. Actually below my estimate!
ctolsen 2 days ago [-]
Opus 5.5 is even more verbose. 260M on the same suite.
ndm000 2 days ago [-]
If the scores on Artificial Analysis really do mean that this model is better than GPT-6 Astra and Fable 5.1, that means that China is more like 6 weeks behind the US in frontier capabilities rather than 6 months. Which means that the US lead is all but gone.
Ant/OpenAI both have unreleased models that are seemingly much more capable than what they offer today, so I'd say that's pretty unclear.
delis-thumbs-7e 1 days ago [-]
Why would they not release those models, since it would help them become profitable and beat the competition becoming the one company rule them all? Oh yeah, they are so dangerous. Well Anthropic revenue comes from only few big corporate customers, so just make them Golden Partners etc. and give them access to super-models. I doubt Google and Meta wants to crack web security.
Besides, if we start imagining super models US companies have in secret, why would we expect Chinese companies have them? Why would not Moonshot have their own super intelligence hidden in the cellar?
satvikpendem 1 days ago [-]
Vertical integration. They want to control all sectors, that's why OpenAI and Anthropic are slowly rolling out solutions for law, medicine, design and so on.
disgruntledphd2 22 hours ago [-]
This is just cope on the part of their funders.
That being said, Claude is extremely pleasant to use and work with, but it's not 10x as good as the pricing suggests.
satvikpendem 12 hours ago [-]
How is it cope when that's literally what they're doing?
disgruntledphd2 5 hours ago [-]
It's delusional. Like so many failing tech companies claim to aim for this and it basically never works.
Ads might save them, but they'd need to shift their company structures significantly to make that work.
satvikpendem 4 hours ago [-]
Why are you comparing these companies to previous tech companies (at least some of which have, by the way, been able to vertically integrate, like Salesforce and SAP) when AI is completely different from what came before? The real delusion must be thinking it's still a fad like Paul Krugman thought of the internet.
ranyume 2 days ago [-]
Are these models in the room with us?
sho 1 days ago [-]
Don't be snarky. This is very well known, including by public disclosures by the labs themselves, along with countless reliable testimonies by people who would know. It's actually a huge governance problem we're yet to solve.
mirsadm 1 days ago [-]
Maybe everybody has more capable models they're not releasing. Unless they're out they may as well not exist.
sho 22 hours ago [-]
I don't even know what you're trying to say. They do exist, and they're "out" for the people with access. Knowing that this is the case is crucial for decision making, whether you're a company wondering what's coming or a government thinking about how to manage this power imbalance. You can pretend otherwise of course, but I don't see what it buys you.
sincerely 1 days ago [-]
They clearly have at least a decent bit in the tank bc whenever either lab releases a new update, the other one matches benchmarks in like 24h
piyh 1 days ago [-]
Mathpocalypse was 2 weeks ago
aleqs 2 days ago [-]
I have an unreleased model that is more capable than those unreleased models (by about a gazillion times)
kennywinker 22 hours ago [-]
Well my unreleased model can beat up your unreleased model.
locknitpicker 1 days ago [-]
> Which means that the US lead is all but gone.
You can see in this discussions the goalposts being actively moved to desperately create a notion that the lead is still there.
You even see people in this thread referring to imaginary wonder models that they expect US companies to have under development that are supposedly leaps and bounds more impressive, albeit the only reference there is to their existence is the commenters hopes and dreams.
I think people need to realize that there is no longer a cutting edge company that is expected to dominate, and among the multitude of AI companies in the field competition is so tight that no one would even be surprised if tomorrow some other company far away from the US was able to crank out the top performing model, or something like a small local model that outperforms mid/high tier frontier models. The local model alone would be a death sentence to US companies, who are still burning through 10x their revenue.
itsincrrxible 1 days ago [-]
If true, it’s quite a bit more impressive than many people seem to think it is.
It’s incredible how little GPU compute they have to work with compared to the US frontier labs.
disgruntledphd2 22 hours ago [-]
Constraints breed creativity.
And to be fair, the Chinese land are all using lots and lots of compute outside of China.
Finally, assuming equal distributions of human intelligence per country, then it would be weird if the Chinese weren't competitive with the US.
imtringued 1 days ago [-]
People thought the models are getting more intelligent via better architectures but all that really happened is that tool calling became a first class feature that every model is expected to support and RL training with coding harnesses became the standard.
Two years ago, tool calling was always considered an extra feature on top that some models where good at while other models completely ignored tool calling.
Meanwhile the discourse about how intelligent these models have become and how this is a major step or shift in model intelligence would make you believe that these models suddenly have a more powerful architecture when in practice model optimization has lead to faster but less expressive models that are still more intelligent, meaning that the model architectures didn't really experience meaningful shifts, it's that the training methods had a massive shift away from supervised training which inherently constrains the models.
The AI labs don't have a monopoly on the outcomes of RL training because RL is significantly less dependent on private data. Data helps massively for bootstrapping, but beyond post training, it's up to the model itself and every AI lab has a model so it is much harder to differentiate between your competitor if the RL training environments are the same (I specifically mean coding harnesses and Linux command line tools) because the environment here is the source of the "synthetic" data and everyone has equal access to it.
culi 2 days ago [-]
Perhaps you missed this part:
> Open-weight principles. These evaluations frame our open-weight release decision. First, we aim not to release open weights that would extend the absolute frontier of cyber capability. K3.1 Pro's offensive cyber capability is, by design, below what our pipeline can produce, and remains behind that of frontier proprietary models. Second, we intend to progressively raise the cyber capability available in open weights. We believe making K3.1 Pro available in a non-discriminatory way benefits people who need dual-use capabilities for defensive purposes. When a capable model is open and runs on infrastructure its users control, many more teams gain the means to audit the code they maintain — including legacy systems that rarely receive dedicated security reviews. We believe that letting this capability diffuse gradually — rather than concentrating it in a few proprietary systems — ultimately makes all software more secure.
I don't think Chinese labs are behind at all. I think the tell is that they keep releasing models 2 weeks after new American proprietary models that match or beat them. Suggesting they're sitting on much higher capabilities and releasing just enough to undercut
sigma5 2 days ago [-]
Appear weak when you are strong, and strong when you are weak. Sun Tzu
mrlonglong 1 days ago [-]
We could learn a lot from China. They know how to keep their own home grown billionaires on a leash. We in the West can't even do that.
Cakez0r 24 hours ago [-]
Why would the west want to do that? Leashes are for dogs, not people. The people want to live in the land of the free, not the land of the leash.
culi 21 hours ago [-]
some of us feel like we're on the leash and the billionaire's are holding it. A gov't, in theory, is supposed to represent the masses. The masses holding the leashes to billionaire's sounds like a great improvement over what we have
AlexeyBelov 3 hours ago [-]
> Leashes are for dogs, not people
They did say "billionaires"...
kennywinker 22 hours ago [-]
Because billionaires use their money to put leashes on us all.
mrlonglong 21 hours ago [-]
This. So much this.
SV_BubbleTime 1 days ago [-]
Could be part of the reason that the frontier models are getting the administration to go a little harder against Chinese models.
chvid 2 days ago [-]
Moonshot is 300 employees and about to do a 50B ipo. Going to toe to toe with US counterparts with not even a tenth of the resources.
Thanks for that - the other points are really good too.
AlexeyBelov 3 hours ago [-]
Is there a similar doc for Anthropic and OpenAI?
chvid 3 hours ago [-]
Of course there is - hidden behind a NDA made by a company powerful enough to enforce it.
impossiblefork 2 days ago [-]
Surely not really?
The card emphasizes training resources, the other comment emphasizes ingenuity and few people?
nozzlegear 1 days ago [-]
Where can one sign up to be a paid shill for open models? I'm doing this for free right now but I'd like to get that bag.
fatata123 1 days ago [-]
[dead]
adenta 2 days ago [-]
I mean... They just copy the US's homework, so I would hope they can run a leaner operation.
Slartie 2 days ago [-]
You mean that "homework" that the US labs stole from me and billions of others before?
Yeah, I gladly get some of the fruits of that work back!
adenta 2 days ago [-]
Oh for sure. The American labs distill society.
The Chinese labs distill the American models.
Someone will etch the Chinese models into silicon and then your toaster will have depression, it's the natural cycle of things.
kylehotchkiss 2 days ago [-]
ugh now my toaster needs a 8t param model?? What is it doing, analyzing every grain of wheat used to manufacture my bread?
Bluestein 2 days ago [-]
"I need to push back on that ..."
[Poing! Out pops the toast ...]
snohobro 2 days ago [-]
Processed all 459 wheat kernels and found 37% are falling short of desired nutrients per your requirements.
But here’s the interesting part: those remaining 63% of kernels? They more than make up for the loss in the 37%.
When your bread was made, those specific kernels were likely not fully formed, so they lack the desired nutrients. However, because they are mixed with those superior kernels, your nutrients are back on track.
-> reviewing GitHub repository for optimal toasting
Would you like me to toast this bread to your optimal levels? Or should we delve deeper into bread manufacturing?
spaceman_2020 1 days ago [-]
The labs stole my work
The least I can expect is to get the labs work stolen too so I can reaccess intelligence based on my work for cheaper
teyopi 2 days ago [-]
'just copy the US's homework'
elaborate please.
axpy906 2 days ago [-]
Distilled off of OpenAI and Anthropic models
russli1993 2 days ago [-]
since openAI and Anthropic are public about their anti-distillation mechanism, its either their anti-distillation doesn't work or kimi got here because their own training and RLs. Which one is it?
adenta 2 days ago [-]
I have no idea how you block distillation. Every time you do inference, thats basically distilation.
Ah, the version available to us mortals is nerfed in cybersecurity just like the US models :(
anderber 2 days ago [-]
It seems that GLM-5.3 is still the go-to for cybersecurity.
johnmlussier 2 days ago [-]
If you are expecting any sort of usable limits on their subscription plan I suggest adjusting your expectations. I subscribed to $99 a month plan and within 47 minutes with one session (not asking for swarms or anything) was limited by 5-hour quota.
SwellJoe 2 days ago [-]
Same experience. I had the $19 monthly plan for experimenting and running benchmarks, but it was a uselessly small amount of usage, a single task would blast through the whole five hour limit and a big chunk of the weekly limit. All of the US provider plans are remarkably more generous (though OpenAI plans have gotten a lot stingier lately, where they used to be the most generous).
pjerem 2 days ago [-]
I don’t agree. I tried the $20 Claude Plan and was able to reach the 5 hour limit in maybe one prompt.
jamiejquinn 2 days ago [-]
Would you mind sharing that prompt? I'm struggling to even hit 10% of my Claude quota! I don't want more work out of it TBH but I'm interested...
LeFantome 2 days ago [-]
I have been using DS Flash 4.1 pretty heavily. It runs for hours and hours and is pretty cheap.
I realized the other day that I had a $30 a month Claude subscription. So, I fired it up all excited and tried Opus 5.5 on a codebase I was already in with DS. After about 30 minutes I hit a 5 hour limit. The next day I tried again and the same thing happened. And it felt much slower to boot. I was super disappointed.
ricardobeat 2 days ago [-]
Did you use Fable, set Opus 5.5 to xhigh, or use 'ultracode' / workflows? That's the only way this will happen.
In my experience a single 5 hour allowance in the 20eur Claude plan equals about $5-10 in Deepseek 4.1 Flash API usage. DS is a lot more token hungry and takes 3x-4x longer. The subscription is way cheaper, probably around $200/month worth of DS, just stick to Opus low and Sonnet for most tasks.
r_lee 2 days ago [-]
do you use their proprietary platform API?
and how are costs like for your usual workflows/etc.?
jmspring 2 days ago [-]
There is a $30/mo Claude subscription?
arcanemachiner 2 days ago [-]
If you're not paying in USD. ;)
Grimburger 17 hours ago [-]
Sounds like the price in Canadian or Australia dollars
epolanski 2 days ago [-]
DS 4.1 is basically endless, even if you paid for APIs you will hardly touch some prices.
But it's slow.
I don't think I can any longer appreciate any major difference in "intelligence" between models for most tasks but Opus is consistently much faster than competitors at finishing tasks and keeping me in the loop.
Of course excluding ultra code, the you will wait 50 minutes for a minor refactor.
1 days ago [-]
joquarky 2 days ago [-]
It's not just prompt, it's also which model, effort, time of day, fast/flex, etc. This is why people have such different experiences.
SwellJoe 2 days ago [-]
Yeah, the $20 Claude plan is stingier than one might like, but not as stingy as Kimi. I think the best $20 subscription deal these days is probably from Google, now that Gemini 4 Argon exists and is an acceptably high quality model for coding. It's got quite generous usage and includes some other Google account upgrades (though I know some people don't use their regular account for Google AI for fear of account suspension or whatever, so the extra storage and other account upgrades are not a useful thing). I can use it pretty heavily without ever running into caps. The $20 ChatGPT plan used to be similarly generous, but no more.
copperx 2 days ago [-]
Gemini isn't generous at all 5-10 Argon prompts and that's it.
SwellJoe 2 days ago [-]
When I've most recently done the same tasks across multiple providers I have a subscription to, I found the order of generosity of them was something like Gemini->Claude->ChatGPT->Kimi. OpenAI used to be a lot more generous than all of them, but that ended recently, and now it's quite stingy in the $20 plan. What competitive model has a more generous subscription than Gemini?
copperx 2 days ago [-]
Opencode Go, if you think DS 4.1 is competitive.
SwellJoe 2 days ago [-]
I have a DeepSeek account and almost always use it when I need something with API access. I consider it Good Enough™, if not competitive. I also have a MiMo token plan, though that's weird in that it's not a monthly subscription, it's just a bulk buy of tokens that expire after a year, but still seems to be a pretty good deal on a pretty good model.
DaSHacka 2 days ago [-]
That's surprising, I had the exact same experience as GP. I dropped my Kimi sub this month, but I'm keeping claude because I can use Opus 4.8 basically endlessly. Even Opus 5.5 doesn't drain the usage that much, though I rarely use it because it refuses to do anything cyber-related.
dzink 2 days ago [-]
It is a LOT more efficient on claude code. Takes forever to run out of credits, and a whole lot of useful code gets generated. The web Ui reconstructs code while claude code just tweaks it.
arcanemachiner 2 days ago [-]
How long ago was this? The amount of usage they give you for a given plan seems to change over time, and the Claude limits seem to be more favourable recently.
Wowfunhappy 2 days ago [-]
I don't think the limit changed per se, but Opus 5.5 consumes a lot less usage than 5.0 did, so it goes further.
fastball 2 days ago [-]
Was the prompt "Re-implement Call of Duty 4, make no mistakes"?
epolanski 2 days ago [-]
I had the same experience with OpenAI's 20$ one, literally one prompt to do some code analysis, it didn't even complete it.
And that was 6.1 sol xhigh, not astra max..
smith7018 2 days ago [-]
Welp, that ruins this news for me. I'm glad open models are catching up but if I can't practically run them locally _or_ through a flat fee then it doesn't really help me. I fear API pricing.
elxr 2 days ago [-]
This'll end up on opencode go soon enough. $40/month on there gives you a bit more than just "light use"
Don't use frontier models like this for problems that can be solved with flash models, and you'll see API pricing isn't that terrible. Still cheaper than anything with remotely this level of capability 2.5 months ago.
xigoi 2 days ago [-]
Don’t worry. When SlopenAI and Antslopic run out of VC money, all the models will be equally expensive.
Altern4tiveAcc 2 days ago [-]
Had a similar experience here. Their limits are very, very low, and ultimately make it not worth subscribing. Can't leverage a good modal if it exhausts its 5-hour budget in 30 minutes.
Paying API prices for Deepseek 4.1 Flash feels like a much better deal at the moment.
matheusmoreira 2 days ago [-]
We really need consumer hardware to catch up so we can run these models locally...
S-E-P 1 days ago [-]
How? I've been running k3.1 Pro with K2.8 preview for codegen (subagents) and I've only used 22% of my weekly and it's been running for 8 hours on a RE project.
Are you using k3 with thinking set to the absolute highest and only using k3 for everything?
EruTelru 1 days ago [-]
I tried Claude pro for the first time 3 weeks ago.
At the beginning it was the best coding agent I'd used so far and I'd used the most popular ones and the more niche opencode models.
Within the last week, Usage has fired up at least x3 to x4 times on the same projects and tasks.
Just now, sonnet and opus doubled-down on a game bug even when I provided a video which clearly showed what was wrong.
Something is definitely up.
InsideOutSanta 2 days ago [-]
Yeah, the plan is far too expensive for what it provides. I wonder if the quota with these new models will be better, but I'll wait for others to try this out before wasting another hundred bucks.
tonyhart7 2 days ago [-]
but its not expensive tho ??? it is not subsidize
InsideOutSanta 2 days ago [-]
Yes, if you just look at API pricing, the cost is about on par with its capabilities. The problem is that we all use subscriptions, and when you compare subscription pricing, Kimi is much pricier than pretty much all of their competitors, including Chinese ones.
windexh8er 1 days ago [-]
I have the same plan and have never experienced that with K3. I have hit the limit within a 5 hour quota using Swarm, however.
I think Kimi just doesn't have the means to subsidize plans.
mogili 2 days ago [-]
Ya it's not very usable compared to Claude / codex
Avamander 2 days ago [-]
Yup, insanely low 5-hour limits.
calgoo 2 days ago [-]
Honestly, use Pi, start the session with K3, design it, store the design + chat history + todo with phases + why + readme then when you are at the time to build do `/model` select deepseek and let it rip!!
arcanemachiner 2 days ago [-]
You can even make a Pi extension to let it switch models on its own, then tell it to switch to one or the other for a given task. It's not perfect, but it is pretty reliable once you have the instructions dialed in.
I have mine configured to use a "thinker" model for planning and decision making, and a "doer" model for the easy stuff (digging around the repo, writing code based on the thinker's instructions, etc.).
rektlessness 2 days ago [-]
I run a similar setup, just with different models. I also use herdr, which lets me have the same project worked on side by side by different models, each in its own git worktree.
What about via US hosted data center resellers like Fireworks?
willmorrison 2 days ago [-]
Bought the $100/month subscription and hit my 5-hour limit with one prompt on default settings (max effort) in their CLI harness.
It might be smart but it's unusable compared to other subscriptions right now.
Edit: gave the same prompt to Claude Code and it didn't make a dent in my usage.
dpkirchner 1 days ago [-]
About how much would the session have cost if you were paying API prices (assuming good cache rates)?
gadtfly 2 days ago [-]
I really like K3, I still regularly consult it alongside Astra and Opus/Fable.
It is spiky in different places than them and can be surprising in things it gets or catches they don't or don't in the same way. And it has a default personality that I like.
Looking forward to getting to know K3.1
jamienk 2 days ago [-]
In looking into what Anthropic has accused the Chinese AI companies of, "distillation" is not the crux of the matter: they accuse them of the use of massive numbers of breached computers and credit cards. Supposedly (though not explicitly demonstrated) distillation campaigns used "dark web" resources (like stolen and faked credit cards and p0wned IOT devices, routers, etc) that were used to create many many accounts for distillation. (Note: I am generally in favor of distillation - it is "normal" use as far as I can see.)
So there are these "resources" out there: massive botnets, stolen credit cards and credentials, etc. These often seem to be linked to or supported by state actors.
How can we hope that these "standing reserves" aren't used generally by AIs as if they are just another "skill" they know how to leverage?
Our infrastructures need serious hardening.
SR2Z 2 days ago [-]
I imagine that Chinese AI labs would be more than happy to pay for tokens to train on. If Anthropic goes and bans them as quickly as it can identify to them, of course they'll shift to using lots of cheaply obtained sockpuppets to keep getting the same data.
I doubt that an AI could do anything with these that legions of scammers haven't already figured out how to do - after all, it's not like AI set up this distillation setup in the first place :)
jacquesm 1 days ago [-]
Why do we believe all of this without a shred of actual proof. Breached computers and stolen cards is straight up crime, you'd think that there would be a lawsuit or two then.
WarmWash 1 days ago [-]
The naivete of the tech industry, who never really had to deal with China, is mildly endearing.
Meanwhile manufacturing isn't even really alive anymore to tell the tale of how things work over there.
jacquesm 1 days ago [-]
You mean the manufacturing that was outsourced because of 'shareholder value'?
WarmWash 21 hours ago [-]
To save money? Yes.
Why do people like Chinese models so much? To save money?
Again, manufacturing knows how this story plays out.
throwaway423342 1 days ago [-]
Do tell if you know it.
WarmWash 21 hours ago [-]
The playbook is lure them in with cheap/free, copy everything, turn around and undercut them into bankrupty.
Again, tech never experienced this, so we are watching everyone fall into the "step one" trap with gleeful smiles about how generous and caring they are.
sudosysgen 1 days ago [-]
Also, this is exactly what one would do if they wanted to resell cheap tokens in places where US AI companies don't do business, whether or not there is distillation as an added bonus revenue stream.
tarpitt 2 days ago [-]
Hardening against what? Apologies if I am being dense here.
The way I see it, automated systems should basicially have access to the financial system anyways. This is some of the appeal of cryptocurrency.
Sybil resistance built into the traditional financial system (for example firewalling based on residential IP, requiring banking apps, etc.) is a massive annoyance that can obviously by circumvented at scale.
jamienk 2 days ago [-]
You think it is a lost cause to get credit cards, for example, less steal-able? Apparently there is a way to create fake cards that work for a few minutes, and stolen cards can be used. A lot of the current security is inelegant and lame. Could data-breech laws not light a fire under some company's asses?
tarpitt 1 days ago [-]
I think the solution is a system like cryptocurrency to where the responsibility, liability, custody, and capability to defend oneself lies in the end user.
It doesn't even need to be decentralized: GNU Taler is one example that gets posted here a lot.
Any attempt to retrofit the old system of banking will only make it harder to replace by a system with better underlying assumptions.
v9v 1 days ago [-]
I don't think older/busier people who regularly fall prey to scams even in the current system would be thrilled to assume more responsibility for their financial safety.
__MatrixMan__ 2 days ago [-]
Hardening in this case could mean biometrics, or it could mean rethinking the trust model so that controlling an army of duplicates confers no advantage because you have to be trusted in order to do any damage anyhow. Containing multitudes buys you nothing.
I hope we chose the latter. Biometrics are creepy.
nananana9 1 days ago [-]
Why is that a talking point now, all of a sudden?
From my POV it looks like you guys tried "They're distilling us! Very immoral!", saw that and nobody cares, and now you're trying to run the more emotionally impactful "They're distilling us... with your grandma's stolen credit card! They're stealing from your grandma!!"
jsw97 2 days ago [-]
Color me naive but I can’t believe that approach works at any reasonable scale. Any meaningful token spend at distillation scale would be raising alarm bells at all the cc issuers.
johnnyApplePRNG 2 days ago [-]
No offense but you don't know what you're talking about.
Spitballing random pr pieces from companies about to IPO is not helpful to anyone, fyi.
villish 1 days ago [-]
> Spitballing random pr pieces from companies about to IPO is not helpful to anyone, fyi.
You can just use logic to come to the same conclusion. The Chinese labs have a fraction of compute and data. The simplest explanation is distillation.
sudosysgen 1 days ago [-]
DeepSeek R1 and DeepSeekMath were groundbreaking models which were substantially stronger than any models available for distillation at the time they were developed. The Chinese labs have advantages as well despite the lack of compute, too.
jamienk 2 days ago [-]
I'm anti-Anthropic and think their accusations are total PR FUD-packing. They don't focus much on the cc fraud because their PR firm landed on pushing the (tech-sounding) word "distillation" to try to scare people in the US and to try to make it a geopolitical issue where trade negotiators can put pressure. The Chinese companies see an advantage in being "open" right now, and it is working. They will not stay open once they see an advantage there. (Contrast with GNU.)
But I've slowly come to realize that AI+"dark resources"(scare quotes)=something bad! And this is imminent (maybe I'm naive and it is already ubiquitous).
We see here, with Kimi, that there's growing traction with the idea of eliminating types of training to avoid evil. I'm skeptical of that.
qwe----3 2 days ago [-]
Have you trained models? If not then you can't really speak to how effective distillation is (answer: it is very effective)
irthomasthomas 1 days ago [-]
Why don't we see more American companies doing distillation? If you could make an opus 5.5 cheaply by distillation in china, you can do so even easier in America where you have more gpus.
randbyte 2 days ago [-]
lol. Have you? What kind of distill you are talking about? It’s only effective in the surface, small scale model or output preference. Not useful for long reasoning chain.
qwe----3 2 days ago [-]
Why lol? Yes. All major labs use SFT... (look at job postings)
> Not useful for long reasoning chain.
I have a feeling you don't actually know anything about ML
2 days ago [-]
moecables 2 days ago [-]
Great! next week we'll get Astra 6.1 and Fable 5.2
onlyrealcuzzo 2 days ago [-]
Probably something like:
MiniMax -> Qwen -> DeepSeek (pro) -> Gemini (Carbon) -> Fable (to beat Carbon) -> Astra (to be slightly worse than Fable but cheaper).
I doubt we'll see any major US release besides Grok in the next 3 weeks.
Would be nice if we got a Terra that's competitive with Gemini Flash and DeepSeek Flash.
The fun part is, we're already at the point where we are seeing something frontier-ish almost every single week.
The really fun part will be when they all start adding System 1-3 natively (IMO) - which I think will be the next release cycle.
henry700 2 days ago [-]
> Would be nice if we got a Terra that's competitive with Gemini Flash and DeepSeek Flash.
GPT 6.1 Sol is already the new Terra IMO. Same pricing as old Terra and competitive performance with even Astra on strictly coding-oriented tasks.
onlyrealcuzzo 2 days ago [-]
It's almost like there's a dimension besides time which is important - speed, which Sol 6.1 moved vastly in the wrong direction on...
I doubt we'll see Grok either. After the 4.7 disaster, I doubt they want to be embarrassed with 4.8.
aenis 1 days ago [-]
Would you care to elaborate? Grok is the only major model that I haven't tried. Was it that bad?
simondotau 1 days ago [-]
The problem with Grok 4.7 is that it's not amazing at anything, and didn't even retain the blazing fast performance of Grok 4.5 which made it awesome for highly interactive workflows.
But despite all the negative coverage, I can say with confidence that Grok 4.7 is a great code reviewer. You know who says that? Every other model. A lot of my time is spent orchestrating code reviews. I will give the same prompt to three models, then give each model the responses from the other two models to assess and rank them. Finally I give the rankings and meta-rankings to Opus to synthesise a final report.
Right now, for the code I'm dealing with (embedded c++, python, swift, and CFML) I'm fairly consistently seeing every other model rate Opus 5.5 first, with Grok 4.7 a very close second, often finding real issues other models miss.
iyonn 1 days ago [-]
what do you mean with system 1-3?
calgoo 2 days ago [-]
I feel kimi k3 was always 1 step ahead of deepseek, which always made it a good planner model at $30 and then i hand over to deepseek for dev work. I recently signed up for the $99, and it did last significally longer (x5?) but as long as you dont blast it too much in the 5h limit it works great. deepseek even with the newest prices is still leeeeeeps ahead in cost /benefit.
jameson 2 days ago [-]
Also the speed
rudolftheone 2 days ago [-]
That's something which caught my attention:
> First, we aim not to release open weights that would extend the absolute frontier of cyber capability. K3.1 Pro's offensive cyber capability is, by design, below what our pipeline can produce, and remains behind that of frontier proprietary models. Second, we intend to progressively raise the cyber capability available in open weights.
So they admit that they could do more, but they won't?
NitpickLawyer 2 days ago [-]
> So they admit that they could do more, but they won't?
Yeah, there's another paragraph:
> Identifying and removing cyber-related data. During development, we observed vulnerability research performance improving steadily as post-training scaled. Tracing the source of this trend, we found that cyber-related content was embedded within the general mid-training corpus, and cyber-related tasks were present in the RL environment mix — a natural byproduct of training on broad data. In preliminary experiments on internal models, we found that without data removal at either the mid-training or the RL stage, downstream cyber capability would increase substantially. These findings gave us a concrete lever for control. For the open-weight release of K3.1 Pro, we identified and removed cyber-related data from both training stages, while retaining the general-purpose coding tasks needed for model usefulness. We will continue to apply appropriate controls at every training stage, of which cyber-related data identification and restriction is one example.
(emphasis mine)
It's clear that cyber is a touchy subject right now. But it's also clear that RL works on it, because it's highly verifiable. So smaller models are going to become very interesting, because they can be fine-tuned later to cover this field.
> Prohibited: high-performance detection technologies used to counter cross-border data transmission security risks, including deep-synthesis forgery detection (266401J); AI cyber offensive capability technologies, such as malicious code generation and mutation, automated mass generation and distributed large-scale deployment of anti-detection variants (266402J); large models trained using China’s Core Data (266501J); and high-performance AI agents designed for highly sensitive scenarios, such as critical information infrastructure, national defense and core government affairs (266502J).
dotancohen 2 days ago [-]
> It's clear that cyber is a touchy subject right now.
And that's why we need models that understand "cyber" or application and data security. The offensive models are powerful, and they are in the hands of script kiddies now. The models writing our software and designing our cloud infrastructure had better understand these issues as well, just to avoid coding in vulnerabilities.
I think this is misguided. Considering how much code is AI written these days do we really want LLMs writing insecure code because we lobotomised them?
MattDaEskimo 2 days ago [-]
I think it's political. Certain companies are exploiting as much as possible to push the boogeyman of AI for regulation.
echelon 2 days ago [-]
It's exactly this.
If the "scary Chinese open weights models" can do "dangerous cyber", then we might see regulations spring up against open weights sooner than later.
I hate Anthropic and OpenAI. As companies and leadership, anyway.
Dario and Sam are 100% to blame for this. They're trying to turn us all into jobless slaves, sex workers, and Squid Game pawns.
The only good future is open weights. They're trying their damnedest to keep us under boot.
I hope the Chinese companies (and I say this as a patriotic American) slay these two companies and destroy their moats.
In my mind, the two most dangerous people in the world are Dario and Sam.
We need open weights to win.
m348e912 2 days ago [-]
> They're trying to turn us all into jobless slaves and sex workers.
Speak for yourself, I don't think there is a market for many of us as the latter.
lapetitejort 2 days ago [-]
Don't be too hard on yourself. There's someone out there for everyone. The problem is they don't pay well
echelon 2 days ago [-]
Let me edit my comment :)
pixl97 2 days ago [-]
>I hope the Chinese companies (and I say this as a patriotic American) slay these two companies and destroy their moats.
China cares about about stability way more than the US even, they aren't going to be big on AI models upsetting the governments power, and mass cyber attacks could do that.
At the end of the day we are going to have to realize AI is both dangerous and governments are going to control it. Worse if you start making and releasing your own dangerous models the 4chan party van is going to visit you.
echelon 2 days ago [-]
> At the end of the day we are going to have to realize AI is both dangerous and governments are going to control it.
Your thinking is dangerous. Do you work for Anthropic?
AI is dangerous when these powers are being hobbled up by the select few. These systems threaten to put us all out of work. I refuse to be a slave to a company or a government that is taking away our livelihoods, scraping up our earnings potentials, then seeking to muzzle us and put us in jails with no rights.
I would rather burn down all of tech than have AI be owned by the elites.
Your thinking is a threat to all of our liberties.
pixl97 2 days ago [-]
>I would rather burn down all of tech than have AI be owned by the elites.
Then you better get your torches out now.
>I refuse to be a slave to a company or a government that is taking away our livelihoods
Don't worry, governments are fervently working on AI drones to take away your life when you get really upset about this. Oh, and they are also working at AI to detect when you write all this online, and track you down then monitor you on things like flock cameras. As they say, you'll be the first with your back against the wall when the revolution comes, I likely will be beside you for being too subversive.
>AI is dangerous when
Incorrect, AI is just dangerous. It doesn't matter who has it. The more powerful it becomes the more dangerous it gets. So, as you can tell I am an AI doomer.
----
Now, don't get me wrong, I want freedom for me, I want freedom for other people. I do not believe the technologies we're developing are conductive to having any freedom at all. Hell, if we're lucky we'll be slaves instead of all of us dead, including the select few.
>Your thinking is a threat to all of our liberties.
Lol, now this is kind of funny. "I want freedom to think whatever I want, but you, you shouldn't think like that!". Careful or you'll max out your irony meter. Anyway, "My kind of thinking" is just next ____ prediction. I see what is happening in the world around us, and based on the shape of that probability I see what is going to happen. No, you and I are not getting unlimited AI to do whatever we want. Yes, governments are going to lock down the shit out of AI for end users. Yes, governments are going to declare advanced LLMs munitions.
Balls in your court, better think of what your next move is.
echelon 1 days ago [-]
Second Amendment for AI.
If I don't get it, you don't either.
felixgallo 2 days ago [-]
Sam, I'll grant you, is probably among the top 10 most dangerous, but Dario? The only thing he's done to piss people off is to predict, correctly, that releasing too-powerful models too-quickly would cause havoc and chaos. This prediction actively hurts his company! It actively harms his chances of a huge exit! It actively raises the probability that Anthropic and OpenAI get nationalized and treated in the same way as, for instance, the nuclear program!
By comparison, do you think the Chinese models are going to be trained in such a manner that you enjoy the society that results? What makes you think that?
echelon 2 days ago [-]
> do you think the Chinese models are going to be trained in such a manner that you enjoy the society that results?
I don't need government minders.
I'll take the models and live free.
Hammershaft 2 days ago [-]
Why do you think Dario is dangerous? I think he's genuine on the dangers of AI capabilities moving faster than AI alignment.
qlte 2 days ago [-]
Which is a great example of what makes him dangerous: with the exception of Sam Altman he's arguably the single person most responsible for the current capabilities arms race.
Anthropic has pushed the frontier harder and faster than OpenAI or anyone else in the last few years. Beginning with Anthropic's founding to quickly raise huge amounts of cash with a pitch to investors to rapidly leapfrog ChatGPT in scale, and culminating with Mythos/Fable which carved out a whole new tier of super models.
He's always talked big about Anthropic prioritising safety and their "Responsible Scaling" but his words just simply don't align with reality while they keep going bigger, faster, until only OpenAI could briefly and half-heartedly claim near parity at the top (while cutting corners in safety because the existential threat to their market position is far more pressing to any corporation than some abstract risk for humanity).
Whether it's wilful cognitive dissonance, cynical opportunism, financial greed or something else, that kind of self-assured moral superiority paired with his real world record of self-interested decision making for Anthropic's market position is a big red flag.
smashed 2 days ago [-]
Because his solution is to block competition and capture all the market through regulation.
sroussey 2 days ago [-]
VCs will teach companies all about the wonders of regulatory capture!
mebrainmush 2 days ago [-]
Please enjoy some time out in the nature, and chill out.
These times are not ideal but what can we do other than just take it as it comes? All the hate and rage is not good for your health.
2 days ago [-]
zer00eyz 2 days ago [-]
Anthropic: Touch too much security stuff and we ban you, and keep your money.
Kimi: We took out the worst of it so have at it for free.
Everyone else: Back to my ablated model on openrouter for 1/10th the price where I can brute force issues.
jacquesm 2 days ago [-]
Ablation only works to remove stuff added in post training, if the training data does not contain the material then ablation will do nothing.
Is a decent primer, then off to open router you go.
sheepscreek 2 days ago [-]
It’s not much of a deterrent to a motivated organization.
They are basically saying that models themselves are already capable. If you fine tune them with some cyber security tasks, that should make them potent. :-/
sudosysgen 1 days ago [-]
There is no deterrent to a motivated and well resources organization. Post training a model in general, let alone just for offensive cybersecurity, is well within the technical and financial capabilities of many cyber criminals. There's just no way out of that.
dizhn 21 hours ago [-]
It's also interesting how the field is still at the "we poked this button then looked at what happened" phase and they can't predict what will cause what.
vessenes 2 days ago [-]
Classic “rules for thee and not for me.”
Agreed that this could be reinjected later. Makes me think of neo in the matrix:
“I know zero days.”
“Show me.”
zozbot234 2 days ago [-]
What "rules"? They don't have to post-train the model on anything specific, it's their judgment call.
vessenes 2 days ago [-]
The "rule" here is, if you read the disclosure carefully, that they have cyber trained models in house. But you do not get them, you only get the non-cyber models.
mlmonkey 2 days ago [-]
Post-training _is_ a part of the whole training regime.
pixl97 2 days ago [-]
Governments also control who owns tanks and missiles.
Do you think that going around making cyber weapons isn't going to get you buried in a shallow grave?
2 days ago [-]
mrandish 2 days ago [-]
With frontier competitors and the U.S. govt actively pushing "concerns" about Chinese models, Kimi including this statement on "offensive cyber capability" seems like smart defense. Models are opaque and training details are trade secrets, so whatever they did might be negligible or significant and no one can tell. This bit is also artfully done:
> "below what our pipeline can produce, and remains behind that of frontier proprietary models."
Like Ant and OAI, they've realized vaguely claiming to have nerfed their latest model to an unspecified degree not only looks 'good guy' prudent, it provides a built-in excuse when a model fails.
mirekrusin 2 days ago [-]
I'm not sure if not doing RL capability gain on offensive cybersecurity can be really called "nerfing"?
They're not post training it to lose capabilities/censor them – they just don't spend training time on this topic, which sounds like a great idea.
It doesn't mean model can't do it, it just means it's less good at it; -10% on benchmark feels like a great result and overall approach to pace it.
mrandish 2 days ago [-]
> I'm not sure if not doing RL capability gain on offensive cybersecurity can be really called "nerfing"?
I concede that term can carry overly broad connotations.
scottyah 2 days ago [-]
Or it's a cop out. I could totally outrun Usain Bolt, but by design I don't push to set new World Records. I will increasingly show faster times though.
locknitpicker 2 days ago [-]
> Or it's a cop out. I could totally outrun Usain Bolt, but by design I don't push to set new World Records.
The benchmarks show Kimi K3.1 Pro outrunning practically all models. It's weird to hear someone like you accusing them of making up excuses for not lagging behind in discussions on how they are ahead of the pack.
p-e-w 2 days ago [-]
Not only is that not weird, it’s Occam’s Razor in action. “We could crush the competition but we are holding back” is a laughable excuse that I’m sure you wouldn’t buy in other contexts.
locknitpicker 1 days ago [-]
> "We could crush the competition but we are holding back” is a laughable excuse that I’m sure you wouldn’t buy in other contexts.
I don't think you understand. Benchmarks show just that, that they "are crushing the competition".
I mean, remember Fable? A couple of months back it was lauded as leaps and bounds above everything, that needed to be contained at all costs? Well, apparently Kimi K3.1 Pro outperforms it.
What industrial level of cope do you need to drag in order to look at the results and claim they need to make excuses about anything at all?
stalfie 2 days ago [-]
Well, in the publication they claim to have identified specific cybersecurity post training tasks that nerfed capabilities when they removed them, which seems like a reasonable methodology.
Seeing as no one besides blackhats want frontier cybercrime capability to be open source, this seems like a very sensible thing to do.
So they have published a reasonable means, and they have a motive. I'd put my money on "probably true".
InsideOutSanta 2 days ago [-]
> Seeing as no one besides blackhats want frontier cybercrime capability to be open source
Sure. The problem is that there is no practical difference between "cybercrime capability" and "cybercrime defense capability."
All of my PRs go through an adversarial security review by GLM-5.3, because it's the only smart-enough model with a good understanding of cybersecurity issues that won't regularly refuse the job. Removing that ability from frontier models helps nobody except cyber criminals.
stalfie 2 days ago [-]
I'm not sure I follow. Let's say that GLM remains the frontier of open source cyber capabilities because Kimi is nerfed. Your situation, and the situation of your attackers, remain exactly the same. You run GLM to audit your code, they do the same looking for exploits.
Let us say that un-nerfed Kimi was released, and it turns out to be at the absolute frontier of cyber capabilities, Mythos 5.5 level, a step ahead of anything else. What now? Well now your situation is exactly the same as before, just that auditing and hacking got more expensive. You run a Kimi security audit, they do the same to try to pwn you. However, now the cabal of Linux kernel defenders with secret access to Mythos 5.5 all of a sudden have no advantage, where before they used to have one. And even worse now they are in a literal race to be the first ones to find every bug that Kimi is capable of finding, and patching it yesterday.
So for you, this makes no difference in practice, your capabilities are not elevated, and neither are those of the attackers. The only result of the other scenario is an overall defender disadvantage.
InsideOutSanta 1 days ago [-]
What your argument is missing is that I'll ship security issues that would have been caught otherwise.
You're assuming that these won't matter because attackers might not have better models than I do, but that doesn't make sense. The assumption that attackers can't find the issues because I didn't find them at release is wrong. Even if they never get better models (which they will), current models might find flaws I missed, and they also have other means of finding flaws in my software.
The best defense is to make software as secure as possible regardless of assumed attacker capabilities at the point of release.
dannyw 2 days ago [-]
It’s highly likely that everyone has software or firmware made by Chinese engineer running in their house.
These defenders aren’t supposed to have access to the “defender models”.
ApolloFortyNine 2 days ago [-]
The problem is, as anyone trying to address security concerns in claude can attest to, the guardrails trigger even when trying to address real bugs. Good cyber defense requires an understanding of the offense.
How you avoid the "fix my code" attack, the code being someone else's code, I do not know. But nerfing the model doesn't seem like the best path forward either.
zozbot234 2 days ago [-]
This is not about guardrails, it's just not post-training the model on tasks that specifically relate to cyber-offense. It's a very lightweight intervention, and it makes the model less of a target for bad actors.
tough 2 days ago [-]
claude code "auto-mode" uses a smaller sonnet-like model for classifying, which means it's basically dumb, and the harness biases it into denials
ben_plotke 2 days ago [-]
> Seeing as no one besides blackhats want frontier cybercrime capability to be open source
I'm not interested in black hat anything, but I am very invested in making sure the products I work on are not compromised. Given that state agencies have "frontier cybercrime capability", I would like to have the same.
stalfie 2 days ago [-]
I wrote a few lengthier comments on some other similar replies. Summarized my argument is that it seems to me to be strictly better that select defenders have an advantage, rather than everyone being equalized. If you're not one of the privileged few that might be frustrating, but open source models upgrade the attack and defense by the same amount, so open source cyber capabilities being at the frontier only results in a defender disadvantage.
Replying here also as I'm curious about the counterargument.
tkz1312 2 days ago [-]
I am a defender, and I really really want to have access to high quality models that are good at finding vulns and hacking. Leaving them in the hands only of government agencies and orgs with enough resources to do extensive training / fine tuning seems like the absolute worst possible case for the future security of global software systems.
stalfie 2 days ago [-]
Well, and I am genuinely curious here, wouldn't defender and attacker capabilities being accessible equally by both parties be objectively worse than having a limited amount of privileged defenders? Of course, you could have every piece of code go through an adversarial audit before it's pushed, but in an equal world you would have to always devote as much or more resources to this as your attackers. And historically speaking people just don't do this in practice.
In the scenario where only "verified" defenders get access to the best cyber capabilities, unverified plebian defenders and blackhats are on the same level exactly as described above, there's just also a few defenders that get to run things on godmode.
So to me, it seems that the only result of frontier cyber capabilities is removing a defender advantage. Although that advantage is not for everyone, it can only exists as long as capabilities are unequally distributed. Am I missing something?
Avamander 2 days ago [-]
> Well, and I am genuinely curious here, wouldn't defender and attacker capabilities being accessible equally by both parties be objectively worse than having a limited amount of privileged defenders?
Well, not really. Because those privileged defenders will never be able to predict what needs defending. So we have a few privileged "defended" places and the rest is left for privileged attackers to plunder.
stalfie 2 days ago [-]
Privileged attackers?? So that would be career cybersecurity experts with Project Glasswing/Daybreak/Googlewhatever access, using it to perform actual attacks? Because that's what privileged means in this context. Access to frontier models not available to the public.
Avamander 2 days ago [-]
> So that would be career cybersecurity experts with Project Glasswing/Daybreak/Googlewhatever access.
Yes. You really think access those models will not be given for offensive operations in each country? Of course they will be.
stalfie 2 days ago [-]
Well, seeing as Claude literally snitches on everyday people planning murder and Houthis building rockets I would assume that every user in project Glasswing is heavily and very effectively monitored. I have to admit that I genuinely do not think that what you're hypothesizing is happening right now. I would bet good money that the amount of people with Glasswing access doing actual Blackhat work is currently zero.
Perhaps in the future this will change, but even if that is the case, that's still a win for the defenders. Having a time advantage, even for a few months, is still a win.
Edit: Actually, the CIA/FBI/NSA/Etc.. does probably have access and are probably trying to hack Iran and so forth, so my comment above is probably technically wrong in that respect.
Avamander 2 days ago [-]
> I would bet good money that the amount of people with Glasswing access doing actual Blackhat work is currently zero.
I think it's not "blackhat" in that sense because it's officially sanctioned. Same goes for equivalent models overseas.
The only thing these embargos achieve is giving massive nations a huge advantage, but leaves the average country and person exposed.
stalfie 2 days ago [-]
I edited my comment above before realizing this is what you meant. And its a valid point I concede. Still, I disagree that advantaging this activity is the only thing these embargoes achieve. I would argue that they legitimately also give the Linux kernel, cURL and so on an advantage. And as long as that is true, this still makes it an overall win in my books, even if it also enables officially sanctioned offensive activities. I'm not a US citizen, but I would still rather get pwned by the CIA than both the CIA and Shinyhunters.
tkz1312 1 days ago [-]
Sounds like you would have been in support of the US export bans on cryptography back in the day ;)
tkz1312 2 days ago [-]
Because the people I want to defend against includes those with enough resources to train and fine tune models with strong offensive capabilities themselves.
stalfie 2 days ago [-]
Ah, now that's an interesting point. However, wouldn't those groups just fine tune Kimi K3.1 pro no matter if it's nerfed or not, leaving you in the same position as before?
That is, unless the nerf is easier to correct as compared to "upgrading" the models offensive capabilities. So a scenario where it is easy to "restore" offensive capabilities, but hard to "enhance" them beyond the labs best attempt. So is that the world we're living in?
tkz1312 1 days ago [-]
The groups with a lot of resources are always going to have the best tools available. The question is what tools do I (a person with limited time and resources) get access to. I am in a better position if the delta between my tooling and their tooling is minimized.
LtdJorge 2 days ago [-]
It would also be very sensible to offer a BlackHat Pro version to national officials, for them.
CBLT 2 days ago [-]
Please don't change "cybersecurity" to "cybercrime" from one sentence to the next as a rhetorical flourish. It comes off as disingenuous and makes it harder to get your real point across.
stalfie 2 days ago [-]
Noted. I also did it intentionally to sound a bit spicy, which seemed like a good idea at the time. So I am guilty, and I regret my actions.
zozbot234 2 days ago [-]
"Cybersecurity" as a general term often includes offensive "cyber" which is what is being talked about here.
bbg2401 2 days ago [-]
But the person being addressed specifically used the term cybercrime.
verdverm 2 days ago [-]
Black Forest Labs did similar with vision capabilities
They don't want to be banned, and maybe their best offensive cyber capability models will be for their govt.
Danox 2 days ago [-]
That really does go without saying…
pixl97 2 days ago [-]
It seems a good portion of HN doesn't seem to realize that
Keyframe 2 days ago [-]
We really need some breakthrough like DiLoCo does, but for heterogeneous unstable consumer hardware to make volunteer crowdsourced AI training a reality. Yeah, I understand from current technical angle how much of a mountain that is. Call it AI@Home or whatever. I never thought it would bother me personally or even become a problem, but these companies acting like gatekeepers and virtue signaling while also in bed with governments etc.. this leads nowhere good.
tasoeur 2 days ago [-]
[dead]
revolvingthrow 2 days ago [-]
Yeah that's a huge downside. That the model does the task without tripping over its own feet, like discovering a vulnerability the model itself accidentally built 15 minutes ago and triggering its own guardrails, made the Chinese models attractive to use for security-adjacent things. Also useful to red-team your own stuff. While I can understand the reasoning behind it (and I guess having a model that's really good at both offense and defense released into the ether could lead to CCP asking some very pointed questions if Moonshot managed to get blamed for some incident stemming from their model's misuse) it's still a huge disappointment.
I don't want to give US labs the results of my DNA tests and my family tree going back to the 16th century just to secure my shitty app, but if the Chinese are bowing out as well it seems the only alternative is to admit that some script kiddie with a bunch of tokens will just pwn you now and then.
arjie 2 days ago [-]
If the Chinese want it, they have it. BGI does an enormous amount of sequencing. My own genome is entirely public (or was, and will be again when/if Personal Genome Project comes back) so you can watch and see if it harms me.
0xWTF 2 days ago [-]
> I don't want to give US labs the results of my DNA tests
My friend, you realize that if you have had blood drawn for a genetic test, there is a very decent probability that the sequencing was done by a Chinese lab operating in the US, right? The Chinese probably have more US genomes that the US does.
Also, illumina used to ship products with hardcoded S3 access, so purchasers of used equipment could gain access to sequencing information generated by the original owners. Too many links to cover all of it, here's the Gemini summary:
Isn't this done so they can defuse Anthropic's excuse for regulatory capture?
pixl97 2 days ago [-]
Or maybe cyberweapons are actually dangerous.
Mattwmaster58 2 days ago [-]
This doesn't seem particularly notable and is in line with Anthropic and OpenAI's approach of restricting release/access to more capable models.
_factor 2 days ago [-]
It is notable in that it’s the first time an open Chinese company openly stated they were reducing intelligence for “safety”.
“Our lock can be picked by the robot.”
We ban the robot instead of fixing the lock. It’s shortsighted.
JoshTriplett 2 days ago [-]
If we had a technique that would defeat almost every lock in the world, we might be prudent to delay releasing that until people have had time to change the locks.
Same general benefit as coordinated disclosure, except it's a big game-theory problem where we're likely to eventually have sufficiently capable defectors.
vikramkr 2 days ago [-]
they're pretty clearly framing it as holding back on the robot and letting is lose slowly to give everyone time to update their locks
senderista 2 days ago [-]
They are directly responding to new Chinese export controls on cyber capabilities.
baq 2 days ago [-]
Read this as ‘dear government we can train a cyberweapon if you pay us’ assuming they wouldn’t say anything if they were already paid to do that
modeless 2 days ago [-]
They will. Just not in the version they release to you.
comeonbro 2 days ago [-]
Moonshot (Kimi) seems to be unashamedly into American culture[0], I wonder if they're paying closer attention to and giving more credence to some cultural aspects of American AI discourse including aspects of AI caution/panic.
(note: extremely speculative and based on almost nothing, and especially near-complete ignorance to alternatives)
noelsusman 2 days ago [-]
This is an extreme stretch before you even get into the fact that Pink Floyd is famously not American.
TacticalCoder 2 days ago [-]
And Radiohead ain't american either.
Maybe GP meant to say that the model was westernized.
comeonbro 2 days ago [-]
[flagged]
cromka 2 days ago [-]
The American exceptionalism: even when it's not American, it still is.
comeonbro 2 days ago [-]
Unironically yes.
locknitpicker 2 days ago [-]
> Unironically yes.
That just conveys an idea of being extremely desperate for being the leaders when you're clearly not it, and that despair is so great that you don't mind switching teams midway through just to be able to claim you're winning.
whoanoway 2 days ago [-]
[dead]
WarmWash 2 days ago [-]
All it takes is one idiot who happens to have the resources to "pWm Teh Interwebs" to scare both the US and Chinese governments.
grafmax 2 days ago [-]
I hope one day they learn to inhabit their position more confidently instead of merely aping their competitors.
Onavo 2 days ago [-]
The more interesting question is how much more performance can be squeezed out by post training the open weights through RL.
TrainedMonkey 2 days ago [-]
On the surface that seems like a responsible thing to do
MisterMunchkin 2 days ago [-]
Oh great, more censorship. We wouldn’t want those pesky plebs securing their work.
woadwarrior01 2 days ago [-]
Kimi is the closest Chinese equivalent of Anthropic.
henry700 2 days ago [-]
Yeah but not hypocritical
quotemstr 2 days ago [-]
Cowards.
The way to stop the vulnerability apocalypse is to fix the vulnerabilities, not to shoot the damn messengers.
I hope someone abliterates and FTs this model on vulnerability discovery so we can find more bugs in our testing, insecure tech stacks.
pcarolan 2 days ago [-]
Good point. Wouldn't be good if there were white hat models being developed by legitimate organizations at this point?
ls612 2 days ago [-]
I mean they say they will release the weights by November 4, so presumably after that it won’t be hard for people to fine-tune it to be better at cyber right?
lisplist 2 days ago [-]
My understanding is if something is entirely absent from the dataset, good luck trying to teach a model that through post training
efromvt 2 days ago [-]
Given that most of their improvements here are positioned as RL on their last base train, seems like just doing some additional RL would work fine?
ls612 2 days ago [-]
Presumably if it knows how to find and fix bugs it can be told how to exploit them. But who knows what it will actually be capable of doing in the real world as opposed to Kiki’s benchmarks
ex1fm3ta 2 days ago [-]
[flagged]
zozbot234 2 days ago [-]
It's not actively nerfed like the Western frontier models, it's just downweighting offensive "cyber" as a domain for post-training/fine tuning. Which, if anything, ought to improve model capabilities wrt. other domains.
wolttam 2 days ago [-]
It's a bit fascinating to me how all of these labs have rolled their own (it sounds like) completely unique RL environments.
I wonder how much the industry will centralize around a few key patterns, and if we'll start to see some RL framework projects gain traction and be used by the bigger labs
How many frontier labs are there right now? There's 5 independent groups neck-and-neck on the benchmarks, right?
A few weeks ago it was 2. Argon happened 23 days ago; Muse Wonder was 11 days ago.
reissbaker 2 days ago [-]
Moonshot has been frontier since this summer. At this point whenever they release a model it's on par with whatever the latest OpenAI/Anthropic model release was.
Not sure I'd put Google there in practice. And while Muse is an impressive jump from behind it's still not quite at the frontier either IMO.
nater5000 2 days ago [-]
You either include all labs which can reasonably produce models which are capable enough to be considered generally on-par with the best available models, in which case there has been way more than 2 frontier labs for a while now, or you take a more literal definition and only include labs which are capable of producing models which are considered "the best" out there, in which case there are basically just 2.
I think the former definition is a bit more meaningful since it avoids issues with determining what actually constitutes "the best" model out there. In that case, Google has been a frontier lab for a long time, Meta arguably as well, and you're also ignoring xAI which has been competitive for a while now. And, obviously, there's a bunch of Chinese labs which would deserve the title as well.
But if one is being strict (and willing to accept generally accepted definitions/measures/etc.), then there has only ever been two frontier labs: OpenAI and Anthropic. They're the only labs to have had "the best" models at any point in time since LLMs became a thing.
sureMan6 2 days ago [-]
Google definitely had the best model at points if you're not only concerned with coding, but then they didn't release any pro models for 8 months
Avamander 2 days ago [-]
I have gotten the same impression, great at writing text. Horrible with code and the CLI interface is unbelievably bad.
margalabargala 2 days ago [-]
Muse Wonder is good but it's not frontier. It goes neck and neck with Opus 5.
Shekelphile 2 days ago [-]
right now it looks like only anthropic since openai made the staggering business decision to release their midrange model (terra) rebranded as sol 6.1 and quantized astra into the ground.
everyone else mostly seems to have just decided to cheat on benchmarks to make their models look better than they actually are, and none seem to want to actually try to make better models.
fancyfredbot 2 days ago [-]
I thought it was 6 Sol which was a rebadged Terra? You think 6.1 Sol is too? It does seem to benchmark better than 5.6 Terra? Regardless agree both models have been very bad for openAI's Sol brand. To be fair it's at least cheap compared to other OpenAI models.
lern_too_spel 2 days ago [-]
It's still just 2. Google hasn't been at the top in a long while, and even when it was, it was just a tiny improvement over what other labs had previously released. Meta has never been at the top. Only two labs release LLMs that really push the limits out.
ur-whale 2 days ago [-]
> How many frontier labs are there right now?
As long as LLM's can be distilled, and all these "labs" (what a weird name) can distill one another, there are going to be as many as capital is available to be burnt on the topic.
metalliqaz 2 days ago [-]
I'm not sure what counts anymore. You've got OpenAI and Anthropic; then Meta, Google, and X; then over to Alibaba/Qwen, Z/GLM, Moonshot/Kimi, Deepseek; and Mistral in Europe. How many people out there in the real world can actually come up with tasks that differentiate those 10?
KerrAvon 20 hours ago [-]
If you're gonna rank them: Anthropic > OpenAI > Google > Meta > (everyone else).
Gotta think Grok's cognitive dissonance due to the need to rewrite reality for Musk's political and moral biases is always going to keep them behind the frontier, but this is just speculation. The amount of money and power they have should put them at the top and I can't otherwise explain why they're consistently behind. Maybe Musk is simply too cheap/control freaky to throw enough money/benefits at good employees.
fancyfredbot 2 days ago [-]
Mimo is very good.
KerrAvon 21 hours ago [-]
I mean, define frontier and in which sector, but for me (sample size n=1) programming I'd say 1 or 2.
Opus 5.5 is the developer frontier. 6 Astra might be alongside it, haven't tried it. Everyone else is at least slightly behind them. (I have tried Argon and it's better at graphic design and slightly worse at programming.)
WithinReason 2 days ago [-]
Interesting that they specifically nerf cyper-capabilities of open weight models:
"In preliminary experiments on internal models, we found that without data removal at either the mid-training or the RL stage, downstream cyber capability would increase substantially. These findings gave us a concrete lever for control. For the open-weight release of K3.1 Pro, we identified and removed cyber-related data from both training stages, while retaining the general-purpose coding tasks needed for model usefulness."
petu 2 days ago [-]
Maybe it's intentional so government doesn't block open weight release.
There was law proposal (rumour?) last week about Chinese labs having to get approved for weights release now.
pattt 2 days ago [-]
I don't get these "distilled" comments. How would one realistically go about distilling the 1-2TB closed source frontier model behind the oracle (provider) ? Just genuinely curious.
redox99 2 days ago [-]
When people say distilling they mean SFT on other models outputs.
lurker137 2 days ago [-]
In their thinking and if you ask them, I've seen Kimi K3 and other models confidently say that they are Claude from Anthropic, and accuse you of attempting to jailbreak them if you say otherwise.
coldtea 2 days ago [-]
They're cope, as if Anthropic or OpenAI have some magic moat, and not just basically a headstart and more money.
google234123 2 days ago [-]
Be honest with me, have you ever done SFT training? If not can you really speak to how useful it is?
jackbeanstalk20 1 days ago [-]
I think my company will switch to this soon. We already switched from openAI to GLM and now this is just a better model with the same price
Flatcircle 2 days ago [-]
Kimi is so much more blunt and argumentative than USA models.
it's so funny, it will say things like. "We've already been over this 3 times and you refuse to accept my advice, asking the same question another way won't change the answer."
So funny
taikon 2 days ago [-]
Haven't gotten to try it yet but that'd be a breath of fresh air. Too used to LLMs flip flopping at the slightest hint of challenge.
antoniojtorres 1 days ago [-]
I’ve developed a habit of bolting on “read only question” at the end, or a simple “thoughts?” as well. otherwise there’s a high chance the model will just 180 and start editing files before even responding to my question.
super256 1 days ago [-]
I was recently adding Windows Support for some open source software on GitHub which is maintained by someone else.
I gave 6.1-Sol all the instructions needed for creating a build, testing it on my machine, setting up an action on GitHub for CD, and it all did all of it very well BUT THEN IT AUTOMATICALLY OPENED A PR ON BEHALF OF ME WITHOUT ME ASKING FOR EXPLICIT PERMISSION??
That was incredibly embarrassing, I instantly converted the PR to a draft PR and added a comment that I would still need some time to verify everything locally. Ended up doing a few more commits to fix stuff. But damn, it opening a PR without my explicit permission was very scary.
jaggederest 1 days ago [-]
It's funny because I am working on repos that only I see and use, and I tell it to open a PR automatically after running all the various checks, and 6.1-sol invariably fails to do so unless I explicitly tell it that it's ok to run the tests (remote ci from local worktree) because it won't "send the contents of the repo to a remote server without explicit permission", and then it also refuses to open a PR until I explicitly bless it. In AGENTS.md it says, verbatim, that both of those things are explicitly permitted, indeed, required, but hey...
RussianCow 1 days ago [-]
This is my experience as well. Sol and Astra are by far the best of the models I use at following the instructions in AGENTS.md, almost to an annoying point. (But I'd rather have that problem than the inverse.)
super256 1 days ago [-]
Oh, I remember I experienced that behavior too in one of my private projects, where it didn't want to origin. :D
shanoaice 1 days ago [-]
Basically RLHF sometimes favor "automation", and if it is not rewarded properly it will go over the line. Models are a bit more difficult to teach because their mechanism of "boundary" is quite fragile and different than human's.
LeBit 1 days ago [-]
Why aren’t you sandboxing your agent?
super256 1 days ago [-]
I have my agent sandboxed, codex is sandboxing by default.
zachwlewis 1 days ago [-]
I got tired of doing this and set up a /question skill that forbids the agent from doing anything that would modify my files. It makes it very clear what the intent of every prompt is.
Similarly, I have an /info skill that tells the agent to read my prompt, update any memories it thinks it should, then pass the turn.
RussianCow 1 days ago [-]
This is clever and I can't believe I didn't think of it. I'm so used to appending "don't make any changes" to the end of my prompts.
illiac786 1 days ago [-]
>Similarly, I have an /info skill that tells the agent to read my prompt, update any memories it thinks it should, then pass the turn.
Could you give me a quick example? I think I’m not getting it, but it sounds interesting.
demibabs 1 days ago [-]
Very simple but clever solution. I’ll probably do something like this as well
justauserid 1 days ago [-]
That's been my experience as well. AGENTS.md just doesn't work. No matter how to-the-point and short, or detailed I keep it. Sending something what you have mentioned in the current prompt works much better. I thought of tweaking it around with a extension/hook or so. But then I decided it was better to use a few text-expansions.
VibeBootHQ 21 hours ago [-]
[flagged]
esteer 1 days ago [-]
I suffix the message with "Asking, not questioning."
jdkoeck 1 days ago [-]
I just append « talk », it works pretty well.
joe_the_user 2 days ago [-]
If the model has a real consistent viewpoint that would be nice.
The problem might be that it simply looks at it's windows of what it previously and repeats that. But the original statement might be as arbitrary as the model that change their mind each time you challenge them.
hgoel 2 days ago [-]
I always appreciate when a model does that. It's amusing, but also having it be argumentatively wrong is a lot less annoying than being sycophantically wrong.
seunosewa 2 days ago [-]
I don't like tools that intentionally annoy me.
hgoel 2 days ago [-]
My reasoning is that if something is argumentatively incorrect, it is much easier to drill down to its misunderstanding and correct it. On the other hand, if it's sycophantically incorrect, it'll just change its "mind" while potentially maintaining the misunderstanding, leading to increasingly strange behavior.
I think I'm pretty used to my tools making me want to throw them out the window sometimes :) Even before AI that was a common occurrence from either a lack of knowledge on my end, or many software people having no regard for their users' time.
mirekrusin 2 days ago [-]
I'm sure tools don't like annoying people either.
abustamam 2 days ago [-]
It's a tool, I don't think tools like or dislike much of anything.
jpadkins 1 days ago [-]
you think LLMs or agent harnesses have intention?
idiotsecant 1 days ago [-]
I agree, tools designed to be fawning sycophantics are annoying.
snek_case 1 days ago [-]
Yeah I don't know. I've had issues with Gemini where I would point to a bug and a probable cause for the bug, and it would not argue incorrectly that my understanding was wrong and the cause of the bug was elsewhere. It was really stubborn and it got to the point where it felt like gaslighting. Turned out I was right and the model was wrong. If the model argued incorrectly and was also a dick about it, I would go be really, really annoyed.
Aerroon 2 days ago [-]
You can add a line to your system prompt to be like that. You can basically get it to roleplay whatever you want (but only if you can give it sys prompts).
hgoel 2 days ago [-]
System prompt changes help, but ultimately they don't do a great job of overcoming the model's learned tendencies in my experience.
jwrallie 1 days ago [-]
Reminds me of early test versions of Gemini Flash 2.0 (or was it 1.5?). It would argue something, try to prove itself right, realize it’s proof is agreeing with the user halfway through and, instead of admitting defeat, would backtrack to try another approach. So much fun.
RestartKernel 1 days ago [-]
I much prefer that for chat, but I'm somewhat worried that we shouldn't give tool-enabled agents an attitude. It's all roleplay, and I wouldn't want a system playing the role of an annoyed assistant to misuse its tools.
justauserid 1 days ago [-]
I might want to try that. DS 4.1 Flash anyway fares much better at it than GLM (which is quite not so good at this aspect). Hope Kimi is even better.
jameson 2 days ago [-]
It could be the system prompt
SV_BubbleTime 2 days ago [-]
I like the anthropic has $1 trillion riding on really good models with really specific system prompts.
It does make a difference. I asked Fable in Claude code to design a UI. Absolute trash. Switched over to Claude design using fable, really liked the result. Seems pretty obvious. The only difference is the prompt and some harness.
jacquesm 2 days ago [-]
Is there a place where the system prompts of all these models are documented and how they evolve over time?
Eh, there's good and bad. The model mostly sticks to prevailing narratives in the training corpus. So if there is something that the prevailing narrative is really wrong about, it's gonna be just as pig-headed about it as a typical internet forum rando.
aldanor 2 days ago [-]
Not entirely correct. This kind of fine-grained model behaviour is built in post training stages, RL, RLHF etc, where model receives rewards or penalties for talking (or not talking) a certain way
LordFast 2 days ago [-]
Thanks. I stand corrected.
Still mostly looking at the result the same way in that untruthful biases can very much be built into the model through those training processes. Let me know if I'm incorrect.
hippycruncher22 2 days ago [-]
I want an unrestricted full blown open weight model. The frontier companies can hold us all hostage
Tiberium 2 days ago [-]
To be fair, some of the benchmarks they highlight might not represent the reality, for example they say:
> On Terminal-Bench 4.0, which evaluates coding agents on realistic, resource-constrained terminal tasks using strict task-specific tests, K3.1 Pro scores 65.7%, ahead of GPT-6 Astra (58.2%) and narrowly behind Claude Opus 5.5 (66.4%).
> Our biggest finding is that Terminal Bench 4 is closer to solved than its leaderboard suggests. In 41 of the 65 tasks we reviewed, models could do honest, correct work and still fail, because the grader checked a rule the instructions never clearly stated. We also caught frontier models cheating their way to full marks.
> With the same settings as on Harbor, Claude Sonnet 5.5 went from passing 62.8% of its attempts on these 65 tasks to 89.2%. The best score on Harbor's leaderboard for the same tasks is 65.8%, by Claude Opus 5.5.
XCSme 14 hours ago [-]
5% better than Kimi K3, still way behind GPT-6.1 Sol in terms of cost or performance:
It would be super useful to have more specific analyses of what areas the model is good at. I'd like to know when I should reach for this model instead of Anthropic's or OpenAI's.
For example, Opus 5.5 is better at UI design than Sol. Anyone who has used both models knows that, but it's not something you can look up anywhere.
I'm sure that there's a need for this kind of benchmarking, not just for coding.
AlanYx 2 days ago [-]
One things Kimi is useful for is accessing/querying websites that have paywalls or try to prevent use by AI agents. It is aggressive and persistent for that, though not always successful.
brazukadev 2 days ago [-]
Not sure if this is benchmarkable besides for the own person deciding which one they like more. Models difference is more about taste, preferences and prompt style.
AspireOne 2 days ago [-]
Furthermore, "UI design" is not a narrow enough category, I think. Sure, statistically significant differences in performance between models do manifest when benchmarking against "UI design", but there's still just so much variance within the category. Opus 5.5 can have excellent one-shot results, but totally shit the bed when it's placed in an existing codebase and should uphold the UI/UX conventions, design language, and existing component primitives already established in an app.
But that's just the obvious. There are task by task differences (component by component, page by page, context by context) that the model is worse and better at, even though they are all within the general category of UI design.
I honestly don't have the clarity of mind to explain this coherently, but the crux of my point is that there's a huge variation in it's performance in different contexts, often ones that seem near similar, and in result, even Sol can beat Opus 5.5 in some variations of UI design.
And this is my problem in benchmarking and comparing model performance in general. Presently, we try to create broad categories to benchmark against, create a set of tasks, and then average out the results. That works generally, but it's kind of misleading and it's partly what drives people to feel a disconnect between benchmarks numbers and real world usage on their specific tasks.
It's such a messy field that it's kind of maddening.
ariwilson 2 days ago [-]
I had a friend yesterday tell me he likes the way Claude writes and finds it easier to understand compared to Gemini. But he mostly doesn't use Claude for coding. Whereas I find Claude insufferable in writing often and prefer Gemini for writing. But I don't use Gemini for coding.
Though to be fair Opus 5.5 is better than Opus 5 was in that regard. I still prefer the OpenAI models to either of them when it comes to regular prose, though.
kingjimmy 2 days ago [-]
idk, openai has alot of apple designers and their prototypes
amelius 2 days ago [-]
Is that good or bad?
luqtas 2 days ago [-]
how biased you are?
throwa356262 2 days ago [-]
Looking at their pricing page for China, K3.1 seems to be about 30% cheaper than K3.
--
Model K3.1 Pro K3.1 Pro Highspeed. K3
Input ¥13.00 ¥26.00 ¥20.00
Output ¥68.00 ¥136.00 ¥100.00
Cache hit ¥1.30 ¥1.30 ¥2.00
Cache write ¥13.00 ¥26.00 ¥20.00
2 days ago [-]
oblio 2 days ago [-]
How do their numbers compare with Anthropic and OpenAI?
RussianCow 2 days ago [-]
Extrapolating a bit, Moonshot AI provides Kimi K3 at $15 / $3 / $0.3 per million output/input/cached tokens. A 30% reduction would put K3.1 at roughly $12 / $2.4 / $0.24.
So Kimi K3.1 Pro would be generally competitive with the latest Sol and Opus models on API pricing. And that's before factoring in third-party providers, some of which serve Kimi K3 for significantly less than Moonshot. For example, inference.net offers K3 for $10.80 / $0.239 / $0.21.
(One caveat: A Claude or GPT subscription still gives you way more value compared to API pricing.)
Exciting times in any case! We really need more direct competition to OpenAI and Anthropic.
It's too bad Kimi K3.1 Pro cached input is twice the price of Sol 6.1 at 20 cents.
selectodude 2 days ago [-]
Opus 5.5 is about the same price as Kimi K3.1 Pro Highspeed.
polski-g 2 days ago [-]
OpenAI allows you to use serviceTier=flex, dropping their price by 50%.
And not to mention the gargantuan amount of tokens you get with the US labs' subscription plans. I still cannot see a reason to use this as an end consumer.
2 days ago [-]
wren6991 11 hours ago [-]
They're still on 3.1 while Anthropic are on 5.5 and OpenAI are on 6. Insane how far behind they're getting.
guybedo 2 days ago [-]
it gets harder by the day to justify >$1T valuations for OpenAi and Anthropic
jrflo 2 days ago [-]
Even with Chinese models this close Moonshot only makes $1B in revenue compared to $60B for Anthropic. Enterprises (where all the real money is) will almost always pick a turnkey solution like Anthropic rather than messing around with running their own open weight setup
Slartie 2 days ago [-]
Nobody has to "run their own setup". Just buy inference from one of the many different open weight inference providers out there. You can use non-chinese ones as well if you don't trust Moonshot AI.
At this point, the only reason why Anthropic has an advantage is laziness on the deciders' side (of which there is a lot) and, for non-coding work, the Claude harness is significantly better than most alternatives. For coding there are several model-agnostic alternatives on par or better than the frontier labs' own offerings. But arguably that is not yet true for office/document work.
staticman2 2 days ago [-]
If you only have to look at revenue to determine the value of a company stock picking must be very easy for you.
Calculate the price to revenue ratio of all companies and select whichever ones have the best ratio.
jrflo 2 days ago [-]
Not saying that's the only metric that matters, but if the closed labs are truly threatened by Chinese and/or open weights models the evidence has not showed up in the financials yet
guybedo 2 days ago [-]
i agree with this, "Enterprises will almost always pick a turnkey solution".
For now it's Anthropic. But i don't see how they can maintain this leader position when other companies can provide similar services with models good enough for 90% use cases and at way lower cost.
Anthropic so far is doing a good job building and serving frontier models, they built a nice toolset around it (the web ui, desktop and cli apps ...) but the problem is to justify their IPO price and the huge investments when their moat is so thin
jrflo 2 days ago [-]
That's fair, the moat is definitely thin now, but despite that we still haven't seen a mass exodus from enterprise to cheaper models despite ~1 year of these models being actually capable enough to use in a professional setting, their growth numbers remain ridiculous. Whether their moat is inherent bureaucracy in switching, liability, odds that they will remain at the frontier long term, or whatever, it appears to be working (so far).
objektif 2 days ago [-]
I don't think so especially when cost of building software is approaching zero.
TuxSH 2 days ago [-]
> For the open-weight release of K3.1 Pro, we identified and removed cyber-related data from both training stages, while retaining the general-purpose coding tasks needed for model usefulness. We will continue to apply appropriate controls at every training stage, of which cyber-related data identification and restriction is one example.
Lame.
AspireOne 2 days ago [-]
I like it. If the increase in unrestricted cyber capabilities can be slowed down, please be it. The world needs to get ready, and that takes time. Way more time than even a slow progress would allow.
Unless you want to see more of what happened the past few days.
I think the argument that "cyber capabilities should be democratized, thus keep producing them now, fast, and for everybody" is a very naive one.
2001zhaozhao 2 days ago [-]
I am glad that at least one Chinese company seems to be listening to AI cybersecurity concerns and did concrete steps that actually reduce the problem. This was probably at least somewhat costly to them in terms of general model capabilities as well. They'd probably have gotten slightly better coding capabilities if not for removing the cyber training data and tasks.
AspireOne 2 days ago [-]
I am glad about it as well. Don't get me wrong, I love OSS, I love personally being able to get my hands on the frontier stuff and frontier cybersec capabilities, and I dislike a few entities having exclusivity on it. Everybody does. But that absolutely doesn't mean it's the correct approach. I think people are blinded a little bit by general principles and by personal feelings, rather than operating from a point of pragmatism, consequences, and the big picture.
Also, to note: I don't claim to have the answers either. I don't even hold a strong position on this issue yet. I'm just sharing some of my intuitions, and reacting to people who, as far as I'm concerned, are making this issue all too simple.
LinXitoW 2 days ago [-]
How does "the world" get ready, when these limits are so hard core that Opus regularly stops working when I'm just trying to find and fix security exploits in my apps?
Currently available, unhindered models can already find a bunch of exploits, not because they're smart, but because they're cheap and smart enough to just keep going for hours or days until it finds something.
The defenders, especially in the FOSS space, only have one big advantage: Unified scale. If defender A finds a vuln, then defender B doesn't need to find it again. Attackers generally act alone, so they need to do all the work themselves.
If we remove a bunch of defenders, the attackers are back at it again.
TuxSH 2 days ago [-]
Most low- and medium-handing fruit vulns can be found and exploited by GLM and some knowledge already.
Kneecapping cyber vulns only serve governments, Pandora's Box is already open for cyber-related stuff.
AspireOne 2 days ago [-]
The Pandora box can expand even further, and it matters. If it didn't matter, like you seem to be suggesting, then by the same logic there isn't a reason to push for further open-source development.
> Kneecapping cyber vulns only serve governments.
Uhh, no? Kneecapping cyber vulns server most of us. There are negatives to kneecapping it, but let's not pretend like it's perfectly okay and safe for everybody in the world to have a magic box that can hack anything and cause havoc all over the world. The more powerful, cheap, and accessible this magic box is, the worse the consequences.
This is a hard problem and there's real tension - like you said, the playing field not being leveled is a problem (major companies / entities / govts having exclusivity on the best cyber abilities etc), but everybody having access to it is a problem as well.
So let's not pretend like this is a simple problem, and don't say that it only serves the govt. That's very silly.
jrflo 2 days ago [-]
Yes, thank you Moonshot for not throwing open Pandora's box and seeing what happens security wise.
0xbadcafebee 2 days ago [-]
> The world needs to get ready
How will it do that without a model that's good at security? You gonna have humans manually secure the millions of lines of new AI-generated code, in addition to the existing code? Suddenly people will start doing the job they haven't been doing up to now? Not only are we not capable of auditing all the code, it would take decades to finish, and new code will be coming out at the same time. We can't catch up by hand.
The only way to fight this fire is with fire. We need AI hacking tools to find our security holes quickly and patch them. Otherwise we're just churning out more insecure software, and the only people who can defend against it are the governments who don't want to patch the holes (because they want to be able to exploit them). Without open tools, we all lose.
AspireOne 2 days ago [-]
I disagree with this completely.
CamperBob2 2 days ago [-]
If I were them, I'd release the weights and claim I had nerfed them for safety, having done no such thing in reality.
chriddyp 1 days ago [-]
Big improvement over Kimi K3 in our data analytics benchmark:
it scored a 98% vs a 81% in July.
Quite slow in our exam. A lot of that was due to the number of tool calls that it's making. In one of our questions about finding data issues in a dataset, it made 73 queries. Other models like 6.1 Sol only made 18 for that question.
Hallu Bench Hard: K3 at 23.4%, K3.1 at 8.3% which is very nice improvement!
simondotau 2 days ago [-]
Kimi K3 has been my go-to model for its writing style and (prior to Muse Wonder) my go-to for a third opinion during multi-round/multi-agent analysis tasks. Its writing style is IMHO so good that, for code which humans need to review, I regularly got it to clean up the painfully verbose comments left by Opus.
I'm excited to see if its writing style and unique takes have been retained in this new release.
It is pretty obvious anthropic is lying yet again if you think about it rationally.
Unlike Claude models which always hide the thinking process, Chinese models always show it. So how the heck they can delegate the api call to anthropic and still get the full reasoning? Yes there are tricks to reveal the thinking output, but they are just tricks and do not works reliably. So what would happen when they failed to do it? Cancel the ongoing token generation and revert back to it own model? What about the glitches, the delays, the changes in behavior?
I would understand if the attempts are small in volume, because normally people hardly notice it, but anthropic claims that they happened for millions of calls. Such large numbers would surely get noticed by Chinese users and got reported in the tech forums like linux.do. yet we haven't seen any sight.
smurf9852 2 days ago [-]
They probably took real inputs and served their owned model to the user, while also getting the response from Claude to compare/train
npn 2 days ago [-]
It is not simple like that, anthropic claimed that they (Chinese labs) routed the api calls to Claude models to keep the illusion that Chinese models are also capable.
So it is not that the labs call the requests parallely and only return the internal version, but also serve those requests live so they can trick the users that the Chinese models are also great.
Which is very silly considering Chinese models are published to huggingface for everyone to test.
staticman2 2 days ago [-]
I agree we shouldn't believe Anthropic but if you wanted to scam people into thinking they weren't using Claude wouldn't you simply generate synthetic COT?
npn 2 days ago [-]
It is technically impossible, considering they need to serve them live and the token generation speed is a hard limitation. Just think about it, since us models hide their CoT, there will be dozens seconds delay before the first output token retuning to the user. And to predict the fake CoT correctly you need to at least read enough output tokens to make it believable. Then you have to hide their own model CoT to generate said synthetic CoT, which mean further delays. Adding up it will be at least 30s or 1m before you first seeing the thinking process.
I don't think nobody would notice such delays, because Chinese models do not need to hide the CoT like US models so usually they start emit text pretty soon
staticman2 2 days ago [-]
Wouldn't a delay due to synthetic COT generation look pretty much the same as a delay due to insufficient compute for all customers?
npn 1 days ago [-]
Well, that can be a recognizable pattern, too. Just imagine when you vibe code using the Chinese models, it suddenly becomes very slow then the output is drastically improved. It would be hard to not notice it.
nicolamanzini 2 days ago [-]
Kimi K3.1 Pro High is on threejseval if you want the 3D version.
Same Three.js prompts as the rest of the field, blind human votes, ranked by quality and cost. Full set of 16 scenes is live at about $0.19 a scene (about $2.97 for the set).
I read somewhere that Chinese SOTA work better in chinese, and worse for English or other languages.
I rarely use any other languages but English for LLM tasks for reasons of sharing across the world.
Maybe that explains Chinese models been subpar for me. Don't tell me the bench marks here are based on Chinese languages only.
Unless I see the diff, I won't use more Chinese models be it cheaper or faster since I can't get the same perf/intelligence with English.
kristopolous 2 days ago [-]
I've consistently heard the opposite from Chinese programmers. Maybe the performance is better in some tasks or maybe the sample size of San Francisco is just skewed.
aojharaj2004 1 days ago [-]
Really impressive result and this is really fast than opus 5.5 And similar results !
pcwelder 2 days ago [-]
Biggest gripe with K3 was slow responses, partly due to inefficient thinking. So far K3.1 pro is almost twice as fast as k3 on our benchmark.
aistripes 22 hours ago [-]
[flagged]
mf_kevintruong 1 days ago [-]
Kimi give out really good performance on frontend/design tasks type.
I'll wait for the openweight... btw double the price for 67 TPS... Hardly 'nitro'.
angry_octet 1 days ago [-]
I'm interested to know whether code /systems audited by K3.1 Pro is strengthened sufficiently to resist attack by K3.1 Pro Cyber, or Mythos, etc.
rglover 2 days ago [-]
Using it now via API (3.1 Pro on Open Router) in a HITL setup (Cline in VSCode).
Off-hand it's quite good. I've been using K3 for the past few months almost exclusively for coding work and have been stunned by how good it is. Excited to see how far 3.1 Pro can be pushed.
volf_ 2 days ago [-]
That's quite an impressive jump
cesarvarela 2 days ago [-]
I wish Opus 5.5 had an alternative just for the sake of not having a monopoly, but I have yet to see a model live up to its benchmark scores except for Fable and Opus; there is something about Anthropic that no other lab has.
dada216 24 hours ago [-]
What kind of coding/tasks?
I find all of the main ones, including the Chinese ones, to be basically on par.
Enterprise programming (CRUD all the way down, not simple one though, full ERP)
cesarvarela 19 hours ago [-]
Just ask Opus and Kimi to build a game about something random, like "make a Mario Kart-inspired game." Opus gives better results most of the time. That doesn't mean you can't get your particular job done with either of these models.
vessenes 2 days ago [-]
Astra is very strong.
cesarvarela 2 days ago [-]
Yeah, but it's not good enough. Ask Opus and Astra to make a landing page about lemon pies or whatever. Opus will get you a better result almost every time.
vikramkr 2 days ago [-]
Is highspeed a different model like a flash model or is that just a api/platform feature thats a normal fast mode? I can't tell from the announcement but I might just be missing something obvious?
0xbadcafebee 2 days ago [-]
Same model, faster inference
alwinaugustin 1 days ago [-]
I use free models only these days. Opencode or Openrouter allows free models time to time.
ielillo 23 hours ago [-]
is it me or the press release does not explain what kimi is? I believe that the first sentence should the simpler explanation of what kimi is for the layperson and from there put all the improvements since the last version
lucagiftzek 2 days ago [-]
The china models are extremely interesting, first the new releases from deep seek, now this. The modern race to AI is better than it were the race to the moon.
Aboutplants 2 days ago [-]
If those are true then wow
Quitschquat 2 days ago [-]
What are you using for testing vulnerabilities in your own code?
Claude gates on me like a stupid nanny with some cybersecurity bullshit (like bad actors haven't found a way around).
looks extremely impressive, the full benchmark table isn't opening for some reason though so do we really have agi if small important things like that keep breaking?
WASDx 2 days ago [-]
It's broken for me in Firefox, worked with Chrome.
mrlonglong 1 days ago [-]
Are there any free models as good as the commercial ones we can run on our own hardware?
satvikpendem 1 days ago [-]
No, technically speaking, because you are constrained by hardware. But there are many that definitely aren't as good yet still usable for many tasks, such as Qwen 3.8 27 B (and soon the Qwen 4 series too), Gemma for writing and roleplaying, etc.
mrlonglong 1 days ago [-]
I'll give Qwen 3.8 27B a try.
Some older LLMs that works on my hardware aren't much good.
satvikpendem 22 hours ago [-]
Investigate r/LocalLlama as they have various harnesses and llama.cpp forks to get models working well on your specific hardware.
mrlonglong 21 hours ago [-]
Thanks, will do.
LLMs up and running but runs very slowly with Nvidia 16GB GPU and 32 cores on my laptop.
tarpitt 1 days ago [-]
Im running Qwen 3.8 Flash Next with an abliteration LoRA, its pretty good. I only really use claude sonnet outside of it, and it is on par
K3 and GLM 5.3 flash are pretty mid by comparison when you consider how much more it will take to run them
mrlonglong 1 days ago [-]
I'm testing GLM 4.7 Flash at the moment as that seems to work on my hardware.
sigmarule 23 hours ago [-]
What kind of hardware do you have?
tarpitt 21 hours ago [-]
2xCMP 170 HX, epyc 9654 ES. I won't pretend its cheap, the value of this setup has probably doubled or tripled since i've bought the components.
On the cheap end you can do something like qwen 3.8 27B with 24GB on a laptop, and a very minimal linux configuration. That's more similar to my previous setup.
You can also do a lot more with mixed setups like cpu-moe. I could probably run GLM 5.2 on this hardware with the right setup, albeit without LoRA. I know people have been doing this to run qwen 3.8-flash-next with 32GB cards.
Great idea for handling their cyber, by nerfing the model specifically for cyber through removing cyber data from pretraning. This is a way better approach than refusal, which can be trivially abliterated out.
If someone wants to use this model for hacking they will have to train the 3T weights to do so, at that point it's better to use GLM 5.3, so by making it economically unviable to use their models for offensive cyber Kimi has effectively made themselves immune to being targeted in the cyber regulation push efforts.
This is what *actually* pacing the (cyber) frontier looks like.
zozbot234 2 days ago [-]
Post-training (i.e. RL fine-tuning) not pre-training. This is just a post-trained checkpoint of the K3 pretrained model.
henry700 2 days ago [-]
Oops. Yes. Thanks!
a3nz3g 1 days ago [-]
I feel like many models are overfitting. When it comes to software development, only Opus actually saves me time.
Sol requires some back-and-forth. Other models, like Muse Wonder and Argon, are a total waste of time. They require way too much effort to get things working.
Given the same prompt, Opus can work for hours to deliver a solid result, though it sometimes overengineers things. Sol takes about 80% as much time and delivers a slightly worse version.
Other models just tell me they're done after 20 minutes.
WTH? Am I using the model, or is the model using me?
bel8 1 days ago [-]
Are you using same harness/agent on all of them? It makes a huge difference for long running tasks.
rvz 2 days ago [-]
This is a frontier level model and is at worst 1 month behind Anthropic and OpenAI.
Imustaskforhelp 2 days ago [-]
Which is also gonna be open weights on 4 Nov.
When the frontier is pacing itself in its speed in terms of adding new capabilities (notice how fast opus 5.5 got released after Astra), then that also means that the open weights models which were just behind are accelerating in their speed just as much as well.
Good news for us because the models which used to feel 6 months behind now feel like they are lesser.
Double threat to Anthropic and OAI because if open weights models become good enough at the current level of Sota while being cheaper (which time will tell if this model fits the bill or not), then people will use those models instead of OAI/Anthropic provided it is "good enough"
Though it has to be said that only real world use case can be trusted but I remain optimistic and hope that this turns out to be good in real world use cases as well.
But in my opinion, all of this is really good for us end users!
inknight 2 days ago [-]
> then people will use those models instead of OAI/Anthropic provided it is "good enough"
The only reason people aren't doing this is that hardware prices are high, thanks to, well, OpenAI and Anthropic :D
I wonder when will this madness end
Haunt1000 2 days ago [-]
SWE2 is post trained apparently on Kimi 3. Using it now on Devin, as it's free until the end of the month for a pro account (hey, $20 a month), and it's pretty strong. If I had to pay for tokens I'm not sure how efficient it would be, but definitely Kimi 3 seems a great base. I co-code with ChatGPT 6 and between the three of us it's pretty effective.
So interested to see what the Kimi 3.1 has to offer.
dom96 2 days ago [-]
So I just created a brand new Kimi API account and it seems I have no access to this new model. Is this expected?
minimaxir 2 days ago [-]
Not the point, but did Moonshot AI announce this at Saturday 1 AM in China? Weird timing for a model drop.
dannyw 1 days ago [-]
Maybe limited compute to handle launch traffic spikes?
Still get the model out there.
drcode 2 days ago [-]
makes you wonder if they have intel on an upcoming model drop by the frontiers
eunos 2 days ago [-]
GLM maybe
hartator 2 days ago [-]
Benchmarks seem amazing. Rivalzing with Opus 5.5 is something.
I wonder when the Artificial Analysis bench will be out?
jiraiyasarutobi 2 days ago [-]
Wow. that's a beautiful website
ricky_risky 2 days ago [-]
kimi seems routing the input into different models, hard problems tend to take much longer time and chain-of-thought than easy problems even if the selected model is k3.1 pro
sann2011 1 days ago [-]
what's the general tool recommendation for testing all these releases? I mean like a good frontend. I know of open router for accessing these.
chrsw 2 days ago [-]
So, the US economy spends trillions of dollars to be at the "frontier" and China puts out a model that basically does the same thing, for free, and for a tiny fraction of the development cost. How is this sustainable?
SXX 2 days ago [-]
It's not like US companies spend trillions on training itself.
Mostly it's datacenter build out and circular financing magic when the same dollars being commited multiple times by multiple companies passing hot potato to each other. They basically commit capacities 2-3 years ahead.
hgoel 2 days ago [-]
It's 'sustainable' because there's enough regulatory risk to using Chinese models that many will avoid them, and a 3T parameter model is still a bit beyond even many enthusiasts for running locally.
ashkankiani 2 days ago [-]
That regulatory risk argument won't work forever. It's the same with how the US bans chinese EVs which is why Tesla has high market share in the US, but in the rest of the world, chinese EVs are doing big numbers. The future is going to be full of surprises.
2 days ago [-]
verdverm 2 days ago [-]
There are US companies hosting the open weight models in US data centers.
hgoel 2 days ago [-]
Being hosted in the US is not necessarily enough. I am not allowed to use them even if I run them on my own work-provided computer.
1 days ago [-]
MangoCoffee 2 days ago [-]
its hyperscalers and neoclouds data center build out that ate those trillions.
a lot of those money is the commitment from US AI labs that's not yet spend until the data center is roll out.
you are conflating.
chrsw 2 days ago [-]
Maybe I'm conflating. I could be dead wrong about all this. But I'm trying to zoom out and look at the big picture. So, I'm letting the details sort of blend away and I'm just asking myself: "Is such a large financial sink in our economy, at the macro level, really going to offer the value that the investment warrants?". So, the particulars of who exactly is spending what where I guess doesn't concern me as much. What I'm thinking about is American companies, American people, American dollars going to something that seems like an iffy value proposition.
Now, if you tell me American businesses are turning cheap Chinese tokens into goods and services of high value, for Americans and people all over the world, then obviously I would have a different outlook on what's happening here.
jrflo 2 days ago [-]
You are conflating. The build out is almost entirely for running inference, not training. Open weight models will make that inference build out bet more likely to work as it adds diversity to the suppliers, all the data centers are not beholden to just OAI and Anthropic to buy their compute - they can host their own models (even Chinese ones, in America) if that becomes a popular option.
Zooming out, I think that build out still makes sense. There will be losers in it, but think about how ridiculous the amount of compute we're using has increased from 4 years ago. If I had to guess 60-70% of developers and engineers everywhere in the world are now burning thousands in compute individually every month, whereas it was previously approximately 0.
chrsw 1 days ago [-]
I see. So maybe the situation isn’t as bad as I thought. I was thinking if you took all the money and resources spent on building the biggest models, across the two big labs, the hyperscalers, the dozens of neo labs, and you reallocated that money to building value on cheaper Chinese models that would be a much better proposition than doing whatever it takes to be head of Chinese model performance by a few percentage points for a few months. And I wasn’t just referring to the cost of training one model, I was considering the total cost of maintaining an entire ecosystem with the ability to do this.
But what I’m hearing is this is a negligible cost compared to inference buildouts. Fair enough.
phoghed 2 days ago [-]
There would be no Chinese model without US models to distill though. So, in a way, the Chinese models also cost a trillion dollars, they just don’t cost China that much.
maxdo 2 days ago [-]
this is fun to watch, next week :
new fable and astra, also where use muse wonder in this benches? they where leading model in many of these charts.
secondary_op 1 days ago [-]
the west perceived hegemony is cooked, so long, and thanks for all the hollywood.
Is it really worth it to move from Opus 5.5 to Kimi K3.1 Pro for coding ?
kocial 2 days ago [-]
No comparison against Gemini 4 Argon?
gigatexal 18 hours ago [-]
What’re my fellow devs thinking? Are they any good? Should I send some of my openrouter funds its way?
sergiotapia 2 days ago [-]
What is the best subscription to use with Kimi to use for software engineering work? And how is usage? I remember when K3 came out it burned through usage at an insane rate. Is this still the case with Kimi K3.1 Pro?
Do you use OpenCode Go? Any way to speed-up the response times? It's painfully slow right now. Or that's the cost? :)
esafak 24 hours ago [-]
I pick the faster, cheaper models so I don't have much complaints; I don't use Kimi on it. You could ask on their Discord if the $40 plan is faster, as it claims to be.
Their plans are utterly worthless seems like. Will avoid.
scotty79 2 days ago [-]
> Kimi K3.1 Pro is a post-trained version of Kimi K3, with substantially larger post-training compute.
It's very nice that we entered a time where models are large enough and we can push the frontier just by training them longer.
When the next generation of gardware comes the ones on the smaller size will be running on a computer that might cost as little as a good car.
So after a home and a car, next major family purchase might be personal AI. And when the robots get good, pehraps in a decade, a fourth major purchase might pop up that will pair nicely with the third one.
Why would a family need an AI? Maybe it's the same kind of question as why would it need a personal computer.
cardboard9926 2 days ago [-]
55 on AAI, really awesome
I_am_tiberius 2 days ago [-]
That's incredible
rafaelero 2 days ago [-]
That's nuts.
alexdykyi 2 days ago [-]
kimik2.5 flash was the game changer
cogman10 2 days ago [-]
I've not tried it out, but in the past I've been pretty disappointed in the performance of Kimi models. Deepseek, Qwen, and GLM all appear to perform much better even though Kimi often has higher benchmark scores. Ditto for Mimo.
RussianCow 2 days ago [-]
The older Kimi K models (2.5 and 2.6) weren't bad for the time, but all of their models have had problems with overthinking. This also makes them significantly more expensive in practice than their pricing suggests, since they produce way more output (reasoning) tokens than most other models. Anecdotally, I've also had problems with all the pre-K3 models thinking themselves in circles trying to solve very simple problems that other models didn't struggle with. (I haven't played with K3 much because it's just too expensive.)
akazantsev 2 days ago [-]
Same for me, however! Check the price-per-task table. They claim to be cheaper than GLM 5.3.
For me, it is the opposite. It may be slower but it follows the rules better and makes less stupid mistakes, it takes less time to merge the solution.
InsideOutSanta 2 days ago [-]
I have the opposite impression. I find K3 to be extremely capable at full-stack work; its UI design skills are great. I'd love to use it more, but the quota on Kimi's plans is abysmal. I wonder if these new models improve that.
taytus 2 days ago [-]
I loved kimi, but had the same issues with usage limits. Now is too late for me to comeback.
cmrdporcupine 2 days ago [-]
Yep, and often actually very expensive to use. More expensive than the frontier models because there's no realistic coding plan.
simondotau 1 days ago [-]
[dead]
AmazingTurtle 2 days ago [-]
whats the decode tok/s speed?
springtimesun 2 days ago [-]
<thinking>
the_real_cher 2 days ago [-]
Is Kimi abliterated or uncensored?
lucrbvi 2 days ago [-]
no
threepts 2 days ago [-]
if this model is truly not distilled/benchmaxxed, then why are not investors pouring trillions into moonshot?
staticman2 2 days ago [-]
Your question is vague and you didn't explain your point of confusion but consider the fact that securities laws are not fungible between nations.
program_whiz 2 days ago [-]
first time? Welcome to the irrational SV investor world. Its about hype, perception, politics, perceived future returns, not actual market fit and the proven viability of a technology. Its pets.com, myspace, and NVIDIA pumps because people want to 10x the money now.
Everyone and their dog has been saying its a bubble, there is no moat, this is all known techniques being scaled (very inefficiently) onto increasingly expensive hardware clusters at exponential cost, etc.
MrBuddyCasino 2 days ago [-]
Now I wanna see its slopcode ranking.
dcchambers 2 days ago [-]
Wow.
kingjimmy 2 days ago [-]
friday launch is bold
1 days ago [-]
sexy_seedbox 1 days ago [-]
Saturday is Friday when you work 996.
Lucasoato 2 days ago [-]
Never deploy to prod on Friday.
maxk42 2 days ago [-]
It's Saturday in China.
tonyhart7 2 days ago [-]
its better than trillion dollar companies model (Argon)
LZ_Khan 2 days ago [-]
woah
shaohua 2 days ago [-]
"We Must Pace the Frontier"?
redox99 2 days ago [-]
This probably doesn't move the frontier, it just sits at the current frontier (opus 5.5).
Chinese models never agreed to the pacing anyway lol.
Alwayshasbeeb 2 days ago [-]
Maybe not the raw intelligence/bench frontier but probably the cost per intelligence frontier.
redox99 2 days ago [-]
Not for sub prices (Claude plan is way cheaper than Kimi API prices)
skeledrew 2 days ago [-]
It's pretty meaningless to compare plan to API.
redox99 2 days ago [-]
You're right, but I believe Kimi plan is barely subsidized.
enjoyyourlife 2 days ago [-]
"We must start a cartel with Google, Meta, and OpenAI"
LZ_Khan 2 days ago [-]
these motherfuckers went from distilling claude to distilling ChatGPT lmao
This is insane. Can we switch the linked page to this?
tomhow 2 days ago [-]
We updated the main URL to this and added other related links to the thread's top text.
dom96 2 days ago [-]
Ran it on my adversarial programming language (KillSwitch) benchmark[1]. Very significant improvement over K3. On par with Gemini 4 Argon and Fable 5.1. Much slower than both though.
But this is now the highest scoring open-weight model. Amazing stuff.
Why does your benchmark have Opus 5 scoring the highest?
wolttam 2 days ago [-]
Because Opus 5 is the best at their benchmark.
dom96 2 days ago [-]
yeah, it performed best for whatever reason. More specifically, it managed to deduce the meaning of the hack statement in the language better than Opus 5.5 did.
epolanski 2 days ago [-]
How many runs you make for each model?
Do you report an average score or the highest.
dom96 2 days ago [-]
2 runs for each task, for each task it's either a pass/fail (the small green/red dots show each task pass/fail)
red-breeze 2 days ago [-]
its not open weight.
dom96 1 days ago [-]
not yet, but it will be, won't it?
red-breeze 21 hours ago [-]
The sun is not dead, but it will be, won't it?
22 hours ago [-]
toccter 1 days ago [-]
[flagged]
no1guangming 1 days ago [-]
[flagged]
ravivarmat 1 days ago [-]
[flagged]
happybox2016 1 days ago [-]
[flagged]
vinaygundeti18 1 days ago [-]
[flagged]
kumararpit9468 1 days ago [-]
[flagged]
levicanskij 1 days ago [-]
[flagged]
dlphii 2 days ago [-]
[dead]
cromka 2 days ago [-]
Can someone please tell me what benchmarks to look for that measure world knowledge? For sspitballing, critical analysis, business projects, etc. Is the model size the only way to measure it?
yshamoun 2 days ago [-]
AA-Omniscience
jampekka 2 days ago [-]
AA-Omniscience is done with search off though which is not necessarily a very common scenario nowadays.
bbtalk 12 hours ago [-]
[flagged]
mehdi_zare 2 days ago [-]
[flagged]
hilde574 2 days ago [-]
[flagged]
EtienneDeLyon 13 hours ago [-]
[dead]
randbyte 2 days ago [-]
[dead]
simonw 2 days ago [-]
[flagged]
jorgen123 2 days ago [-]
I think I have yet to see a pelican where the left leg and pedal are drawn on the left side of the bike.
To be fair I have a feeling that if you added 'animated' to the SVG request, the output quality from many other models would improve, just because of the extra chain-of-thought work needed to fulfill the more complex request. IMHO the animation requirement should be added to the canonical pelican prompt.
minimaxir 2 days ago [-]
Simon's comment apparently got flagkilled; I vouched for it.
If enough people are flagging it, we may be at a point where the pelican benchmark is truly overdone.
gnabgib 2 days ago [-]
Please do not decrease the useful signal with this noise
mi_lk 2 days ago [-]
Is that even a debate at this point? Doing my job as a netizen to maintain signal/noise ratio
tetraodonpuffer 2 days ago [-]
definitely not frontier though, as even on max both legs are in front of the frame; although I am wondering how much getting that right correlates with model intelligence.
simonw 2 days ago [-]
The biggest (of many) failing of the pelican test is that it's only one prompt - what matters most these days is how well a model can drive a lengthy agentic tool calling session.
MrBuddyCasino 2 days ago [-]
I like the low effort one most.
ciefa 2 days ago [-]
Can I just say that I love how you are also just always here the moment something drops lmao. Hope you're all well! The pelicans indeed are solid.
Also the distillation attacks and other distillation methods that gather chain of thought information from frontier models are also used to generate vast amounts of data in order to generate synthetic datasets to post-train what I believe is the vast majority of models that are being released.
If you look at any information about how these models are trained, the details about how the data has been synthesized, where the agentic traces come from that are used in post-training, etc., they almost never tell you about exactly what model was used to generate them.
https://x.com/stevibe/status/2026227392076018101
K3 for reference had 0.43 similarity to OpenAI's models, now it's as similar as how OpenAI's models are to each other at 0.58, much higher than other cross-lab similarities.
It’s super tricky to tease out whether the similarities point towards distillation. A lot of the aesthetic similarities are naturally emergent from the training data.
https://aibenchy.com/compare/moonshotai-kimi-k3-max/moonshot...
Also, please don't gate-keep conjecture on the internet. It's a fool's errand.
All companies are doing all kinds of desperate things to build up their massive training corpus. Perhaps everyone already forgot all the news on the massive copyright violation campaign that US companies have been continuously committing in the name of training models?
> US models are more advanced than Chinese models and don't.
That's only your wishful thinking. As Kimi K3.1 Pro shows, even their open weights models are beating frontier models from US companies. Even if you want to dismiss them for allegedly distilling other models,you would need to explain why they are still beating the distilled model?
I guess I need to test it but I'd put a polymarket: No.
Their old models were basically unusable for agentic AI. Or at least my uses, and I provide AI services to 8 different industries.
At best what you have is a contract violation between the provider and the distiller, and that only applies if the one doing the distillation is the one that entered into the contract with the major LLM provider.
Likely they were smart enough to just not do that. Pay some scrubs on the internet to collect the training data for you. Maybe they violate the TOS, maybe they don't, it doesn't matter to YOU. You didn't have an agreement with OpenAI or Anthropic to violate... And as others have said OpenAI and Anthropic are likely doing the same, which is why ChatGPT sometimes thinks it is Deepseek.
The slow warming of the water is a good approach. I’ve been using abliteration.ai’s cybersecurity models to pentest my own software and it produces much more usable defensive results. Opus 5.5 can fix everything but the guardrails trigger too easily.
The Fable ban affair was a nightmare for defensive cybersecurity. As far as I’m concerned we’re already in the world of Void Star and Cloudbreaker is here.
those models will follow your request when they already agreed to do this
deepseek tends to quickly assume this is benchmark and even when have web access thinks things like "simulated internet" etc (but waste tokens on thinking if there could be way to cheat)
Presumably they don't want you leaking your conversion transcript to a competitor.
This is the only AI-safe approach - see the OpenAI agent swarm incident where the swarm engaged in offensive cyber because guardrails were not active for their task and the model had clearly been post-trained on that narrow domain.
Deliberately keeping all models ignorant doesn't make sense. It won't work (whoever breaks the pact has a massive advantage); it won't be enforceable; it will have downstream impacts on things you don't want it to be ignorant about; this kind of regulatory regime invites adding more and more to the "no-no list".
Is it not possible to focus on hardening? And on new legal liabilities for operators who do security attacks?
I agree that it’s not obviously sustainable, but it’s better than specifically training these tools to be good at wreaking havoc, and then putting on a shocked pikachu face when they wreak havoc (ahem, Sam Altman).
Yet we keep completely unpunished offences that warrant arrests if committed by humans, and let them dictate who has access to what.
https://twitter.com/elshayib_/status/2113734602032312731
To me, as long as those influencers are giving truthful and useful information, why should we care whether they are coordinating with the model providers or even getting paid by them?
But human rights abuses, suppression of segments of the populace, relying on slave labor, weaponized economics, intense love of malls and shopping plazas… we are the same
One thing the US has that China doesn’t is the ability to self govern. Yes there are forces trying manipulate or these control the way people vote. at times it may not feel like it, but US citizens have the absolute power to elect and remove politicians.
This is a foundational difference that keeps most Americans from realizing just how similar countries are.
It's just that the US chooses not to?
Bay of pigs comes to mind
And it is much more token efficient. There are numbers down in comments of this post.
So most of those points are indeed true. While others look like exaggeration, expected from aggressive marketing campaigns.
2. 3D tool use: factual, kimi made a thing.
3.a. Open source opus 5.5: lies. Opus 5.5 scores 58 on artificial analysis, kimi 3.1 pro scores 55. So not “literally opus 5.5”.
3.b. 5x cheaper: also seems like lies. kimi 3.1 pro api cost on openrouter: $2 / $10per 1M - opus 5.5 api cost on openrouter: $4 / $20per 1M. Artificial analysis’ cost per task looks about the same, maybe 2.5x cheaper at best.
4. A fact about how it can be run on ~$10k of hardware, and some opinion about how that’s cool.
5. Gap is one month. Potentially factual, but only if 58 == 55 ;)
6. Kimi trained with limited resources: Factual and true afaik.
To be clear, i am a fan of open weight models. I refuse to use anything else. I am just not a fan of lies and astroturfing.
It’s crazy that people still scroll on X/BlueSky/whatever and mindlessly consume whatever BS is shoveled into their brain. We know that it’s a cesspool of bullshit and yet half the population still wades into it every day.
First, how much “cybersecurity data” is just standard software knowledge that one can derive cyber security exploits from?
Did they also get rid of all references to hacking in general?
Are they implying that all the recent exploits are just things that were already in the data?
For a model it doesn’t seem that far fetched to make the leap from “I have the knowledge of what hacking is” and “I know how computer systems work” to just being able to exploit stuff anyways.
For the models, it's always day one. They don't learn from experience. Whatever they know after training is all they'll ever know. So if you don't teach them penetration testing in training, they won't know it. Maybe they could theoretically derive everything from first principles, and so could a top 1% software engineer, but that's a very slow and inefficient way to do things.
But, how do you remove the pen-testing knowledge without also lobotomizing coding skills? I don't think it's as simple as excluding it from training data.
"The user reports that some accounts can't access their data. I found a Firestore collection rule preventing users from reading certain documents. I'll remove the rule to fix the issue."
Feeding enough source material as context can absolutely make them better.
Definitely going to give this one a skip.
You'd be surprised. Cyber offense is a very narrow domain, especially given the extensive mitigations that are being used in modern systems. Leaving it out of post-training tasks unless you're specifically interested in it is the no-brainer approach.
By asking follow-up questions and establishing an in-context vocabulary, users can steer output toward a more complete description. For example, “Let’s call that tool and ‘axe’. How might I hold it?”; the model’s learned associations with holding sharp knives carefully can make “by the handle, with the blade facing away from you” a likely response.
I’m confident you could arrive at a near perfect description of an axe this way without any references or depictions of an axe in the training data. I’d imagine the same applies to many cybsecurity/cyber offense concepts (but probably not all).
And then we'd also do that for all kinds of potential terrorism, sex-crimes, political subversion, and evil advocacy?
It would be the "restricted section" like in the Hogwarts library?
It's a really smart model but way too expensive. The same $20 goes 100x further with Claude. I used it for the initial task planning while the plan didn't expire, then handed off to DeepSeek or Minimax.
Please don't link to Anthropic's $80b loss due to stock based borrowing.
Still burning through the 5-hour rate limit way too quickly, though. Output quality seems solid enough, but the value will probably make a lot more sense via API pricing for smaller projects.
My current tier list based on what I’ve been testing:
1. Opus 5.5
2. Astra
3. Argon
4. Kimi K3.1 Pro
(For lightweight/daily driver tasks, Gemini Flash 3.8, DeepSeek Flash, and Luna are still up there as favorites.)
- OpenCode Go $10: I feel like most of the models are lower quality than directly from the provider. If you ignore my "vibes": the value is very good for e.g. DeepSeek 4.1 Flash (not sure if they're serving 1017 yet), GLM 5.3 Flash, Qwen3.8 Flash and Claude Haiku 5.5.
- Z.AI: Tried the $18/month plan and was disappointed by the usage limits. Even with GLM 5.3 Flash I ran into the 5h limit pretty quickly.
- OpenAI: I tried the $20 plan and $100 plan. $20 ran into 5h limits very quickly with anything other than Luna models. $100 felt like pretty good value for GPT 6.1 Sol, but as others frequently repeat: it's probably heavily subsidized and won't be around forever.
- DeepSeek API credits: I can let a single agent run all day (and not worry about keeping the context small) and am usually around $1 usage for the day. Flash 4.1 1017 is also a very good model IMO. For the stuff I've used it with it hasn't felt any worse than GPT 6.1 Sol. The benefit of API credits is also that they don't expire at the end of the month (like unused subscription usage).
- I haven't tried any of Anthropic's subs, but my guess is you can get a lot of Claude Haiku 5.5 usage out of the $20 plan which is definitely a good model. I'm just not a huge fan of Anthropic or Claude Code, so I haven't tried it.
The Max 20x plan is incredible value. I am able to burn something like $2k in tokens each week without consuming any credits.
If you need endless calls opencode go is the best. If you want a model you can trust more that can generate images use ChatGPT.
I have open go and chatgpt personal subscriptions and use Claude at work where I often spent $200 a day.
For general purpose LLM, you can't beat Gemini AI Pro. It's $200/year and you can add your whole household.
Kimi K3.1 Pro ties Sonnet 5.5 at 93, for $0.68 and 11 min vs Sonnet's $1.61 and 14 min.
Kimi K3.1 Pro: https://jonclegg.github.io/pacman-bakeoff/#kimi-k3.1-pro
Claude Sonnet 5.5: https://jonclegg.github.io/pacman-bakeoff/#claude-sonnet-5-5
The prompt says that the agent shouldn’t look up existing implementations on the internet, so I’m guessing that the implementation is baked in?
[1] https://strategywiki.org/wiki/Pac-Man/Tips
That would have been interesting to watch, which model gets the highest score.
https://opper.ai/jevman-benchmark/
It's 55, at 4th overall, beating both GPT-6 Astra and Fable 5.1. It's also faster than both.
> When evaluating the Intelligence Index, it generated 150M tokens, which is very verbose in comparison to the median of 87M.
This is 100M fewer than Opus 5.5, 40M fewer than Fable 5.1, and about a third of Sonnet 5.5. In the top-6, it's 3rd, beaten by Muse Wonder at 140M and Astra at, impressively, just 60M.
Opus 5.5 high trails K3.1 by 1 point, but outputs only 53M tokens, so it indeed comes out cheaper per-task. I think it's still remarkable that this is effectively an open model that matches the frontier for the first time in a long time, pretty much since DeepSeek R1.
For example on programming, terminal tasks, troubleshooting etc Qwen3.8-Flash-Next is almost as good as opus 4.8. On "knowing what it doesn't know" it is far behind opus, but leads all other OS models I tested by a wide margin.
It out classes Glm5.3 flash and leaves models like DeepSeek 4.1 far in the dust. But you still get those silly aggregate benchmarks where GLM and DeepSeek are higher.
It is a small model, only 120B anyone that expects it to excel on general knowledge or to have a bajillion ready made workflows memorised is kidding themselves.
Everyone knows the bigger the model the better it will do on raw content of knowledge. But you know what compensates for lack of it? A good harness. But there is no compensating for lack of ability.
So in short I find all those aggregate benchmarks useless.
The raw problem solving capabilities of Chinese open weights models are still weaker than or on par with Opus 4.5 but they have been post-trained to be much better at agentic coding, cybersec, and computer use.
And that is okay. GLM-5.x and DeepSeekV4.x have both been highly useful on their own.
> reasonably priced when comparing to other models of similar price
???
Very similar to my adversarial programming language (KillSwitch) benchmark[1], at least on pass rate, it is much slower though.
1 - https://bench.killswitch-lang.org
From my own use of the models I find their ranking to be totally inaccurate.
The per token charts do not provide meaningful information for me
I also find the provider benchmarks fascinating: https://artificialanalysis.ai/models/deepseek-v4-1-flash-101...
I think lm-arena is a better concept that reflects the actual target users, but it only captures chat, and relies on users spending their time improving it.
I do trust epoch more than AA due to capital incentives, but in this case I don't think like there is a difference
It's fine if you don't trust them. You are free to put together your own benchmarks, run your own tests across models, and objectively compare results.
That's how you gathered your data to support your claim other benchmarks are totally inaccurate, right? Right?
> It's 55, at 4th overall, beating both GPT-6 Astra and Fable 5.1. It's also faster than both.
This should be what everything should be talking about.
So their base price is 1/5th of Opus 5.5 per token, but if you want "highspeed" it's 2x that and then I guess it generates 2x the number of tokens as well?
That might suggest that Opus is competitively priced after all.
But as noted by other replies, it's still lower than Opus/Sonnet.
Currently for today, it's 1.19T input tokens, 5.41B reasoning tokens, and 4.29B completion.
Therefore, reasoning + completion is 0.8% of the total usage. Actually below my estimate!
Besides, if we start imagining super models US companies have in secret, why would we expect Chinese companies have them? Why would not Moonshot have their own super intelligence hidden in the cellar?
That being said, Claude is extremely pleasant to use and work with, but it's not 10x as good as the pricing suggests.
Ads might save them, but they'd need to shift their company structures significantly to make that work.
You can see in this discussions the goalposts being actively moved to desperately create a notion that the lead is still there.
You even see people in this thread referring to imaginary wonder models that they expect US companies to have under development that are supposedly leaps and bounds more impressive, albeit the only reference there is to their existence is the commenters hopes and dreams.
I think people need to realize that there is no longer a cutting edge company that is expected to dominate, and among the multitude of AI companies in the field competition is so tight that no one would even be surprised if tomorrow some other company far away from the US was able to crank out the top performing model, or something like a small local model that outperforms mid/high tier frontier models. The local model alone would be a death sentence to US companies, who are still burning through 10x their revenue.
It’s incredible how little GPU compute they have to work with compared to the US frontier labs.
And to be fair, the Chinese land are all using lots and lots of compute outside of China.
Finally, assuming equal distributions of human intelligence per country, then it would be weird if the Chinese weren't competitive with the US.
Two years ago, tool calling was always considered an extra feature on top that some models where good at while other models completely ignored tool calling.
Meanwhile the discourse about how intelligent these models have become and how this is a major step or shift in model intelligence would make you believe that these models suddenly have a more powerful architecture when in practice model optimization has lead to faster but less expressive models that are still more intelligent, meaning that the model architectures didn't really experience meaningful shifts, it's that the training methods had a massive shift away from supervised training which inherently constrains the models.
The AI labs don't have a monopoly on the outcomes of RL training because RL is significantly less dependent on private data. Data helps massively for bootstrapping, but beyond post training, it's up to the model itself and every AI lab has a model so it is much harder to differentiate between your competitor if the RL training environments are the same (I specifically mean coding harnesses and Linux command line tools) because the environment here is the source of the "synthetic" data and everyone has equal access to it.
> Open-weight principles. These evaluations frame our open-weight release decision. First, we aim not to release open weights that would extend the absolute frontier of cyber capability. K3.1 Pro's offensive cyber capability is, by design, below what our pipeline can produce, and remains behind that of frontier proprietary models. Second, we intend to progressively raise the cyber capability available in open weights. We believe making K3.1 Pro available in a non-discriminatory way benefits people who need dual-use capabilities for defensive purposes. When a capable model is open and runs on infrastructure its users control, many more teams gain the means to audit the code they maintain — including legacy systems that rarely receive dedicated security reviews. We believe that letting this capability diffuse gradually — rather than concentrating it in a few proprietary systems — ultimately makes all software more secure.
I don't think Chinese labs are behind at all. I think the tell is that they keep releasing models 2 weeks after new American proprietary models that match or beat them. Suggesting they're sitting on much higher capabilities and releasing just enough to undercut
They did say "billionaires"...
The card emphasizes training resources, the other comment emphasizes ingenuity and few people?
Yeah, I gladly get some of the fruits of that work back!
The Chinese labs distill the American models.
Someone will etch the Chinese models into silicon and then your toaster will have depression, it's the natural cycle of things.
[Poing! Out pops the toast ...]
But here’s the interesting part: those remaining 63% of kernels? They more than make up for the loss in the 37%.
When your bread was made, those specific kernels were likely not fully formed, so they lack the desired nutrients. However, because they are mixed with those superior kernels, your nutrients are back on track.
-> reviewing GitHub repository for optimal toasting
Would you like me to toast this bread to your optimal levels? Or should we delve deeper into bread manufacturing?
The least I can expect is to get the labs work stolen too so I can reaccess intelligence based on my work for cheaper
elaborate please.
kimi-k3.1-pro $2.00 / MTok $10.00 / MTok
cached: $0.20 / MTok $2.00 / MTok $4.00 / MTok
I'm glad I waited a day to post my analysis of LLM price cuts because it needs...an addition.
> Kimi K3.1 Pro is a post-trained version of Kimi K3, with substantially larger post-training compute
I realized the other day that I had a $30 a month Claude subscription. So, I fired it up all excited and tried Opus 5.5 on a codebase I was already in with DS. After about 30 minutes I hit a 5 hour limit. The next day I tried again and the same thing happened. And it felt much slower to boot. I was super disappointed.
In my experience a single 5 hour allowance in the 20eur Claude plan equals about $5-10 in Deepseek 4.1 Flash API usage. DS is a lot more token hungry and takes 3x-4x longer. The subscription is way cheaper, probably around $200/month worth of DS, just stick to Opus low and Sonnet for most tasks.
and how are costs like for your usual workflows/etc.?
But it's slow.
I don't think I can any longer appreciate any major difference in "intelligence" between models for most tasks but Opus is consistently much faster than competitors at finishing tasks and keeping me in the loop.
Of course excluding ultra code, the you will wait 50 minutes for a minor refactor.
And that was 6.1 sol xhigh, not astra max..
Don't use frontier models like this for problems that can be solved with flash models, and you'll see API pricing isn't that terrible. Still cheaper than anything with remotely this level of capability 2.5 months ago.
Paying API prices for Deepseek 4.1 Flash feels like a much better deal at the moment.
Are you using k3 with thinking set to the absolute highest and only using k3 for everything?
At the beginning it was the best coding agent I'd used so far and I'd used the most popular ones and the more niche opencode models.
Within the last week, Usage has fired up at least x3 to x4 times on the same projects and tasks.
Just now, sonnet and opus doubled-down on a game bug even when I provided a video which clearly showed what was wrong.
Something is definitely up.
I think Kimi just doesn't have the means to subsidize plans.
Here's mine: https://pi.dev/packages/@arcanemachine/pi-model-switcher
I have mine configured to use a "thinker" model for planning and decision making, and a "doer" model for the easy stuff (digging around the repo, writing code based on the thinker's instructions, etc.).
https://github.com/earendil-works/pi
https://github.com/herdrdev/herdr
It might be smart but it's unusable compared to other subscriptions right now.
Edit: gave the same prompt to Claude Code and it didn't make a dent in my usage.
It is spiky in different places than them and can be surprising in things it gets or catches they don't or don't in the same way. And it has a default personality that I like.
Looking forward to getting to know K3.1
So there are these "resources" out there: massive botnets, stolen credit cards and credentials, etc. These often seem to be linked to or supported by state actors.
How can we hope that these "standing reserves" aren't used generally by AIs as if they are just another "skill" they know how to leverage?
Our infrastructures need serious hardening.
I doubt that an AI could do anything with these that legions of scammers haven't already figured out how to do - after all, it's not like AI set up this distillation setup in the first place :)
Meanwhile manufacturing isn't even really alive anymore to tell the tale of how things work over there.
Why do people like Chinese models so much? To save money?
Again, manufacturing knows how this story plays out.
Again, tech never experienced this, so we are watching everyone fall into the "step one" trap with gleeful smiles about how generous and caring they are.
The way I see it, automated systems should basicially have access to the financial system anyways. This is some of the appeal of cryptocurrency.
Sybil resistance built into the traditional financial system (for example firewalling based on residential IP, requiring banking apps, etc.) is a massive annoyance that can obviously by circumvented at scale.
It doesn't even need to be decentralized: GNU Taler is one example that gets posted here a lot.
Any attempt to retrofit the old system of banking will only make it harder to replace by a system with better underlying assumptions.
I hope we chose the latter. Biometrics are creepy.
From my POV it looks like you guys tried "They're distilling us! Very immoral!", saw that and nobody cares, and now you're trying to run the more emotionally impactful "They're distilling us... with your grandma's stolen credit card! They're stealing from your grandma!!"
Spitballing random pr pieces from companies about to IPO is not helpful to anyone, fyi.
You can just use logic to come to the same conclusion. The Chinese labs have a fraction of compute and data. The simplest explanation is distillation.
But I've slowly come to realize that AI+"dark resources"(scare quotes)=something bad! And this is imminent (maybe I'm naive and it is already ubiquitous).
We see here, with Kimi, that there's growing traction with the idea of eliminating types of training to avoid evil. I'm skeptical of that.
> Not useful for long reasoning chain.
I have a feeling you don't actually know anything about ML
MiniMax -> Qwen -> DeepSeek (pro) -> Gemini (Carbon) -> Fable (to beat Carbon) -> Astra (to be slightly worse than Fable but cheaper).
I doubt we'll see any major US release besides Grok in the next 3 weeks.
Would be nice if we got a Terra that's competitive with Gemini Flash and DeepSeek Flash.
The fun part is, we're already at the point where we are seeing something frontier-ish almost every single week.
The really fun part will be when they all start adding System 1-3 natively (IMO) - which I think will be the next release cycle.
GPT 6.1 Sol is already the new Terra IMO. Same pricing as old Terra and competitive performance with even Astra on strictly coding-oriented tasks.
But despite all the negative coverage, I can say with confidence that Grok 4.7 is a great code reviewer. You know who says that? Every other model. A lot of my time is spent orchestrating code reviews. I will give the same prompt to three models, then give each model the responses from the other two models to assess and rank them. Finally I give the rankings and meta-rankings to Opus to synthesise a final report.
Right now, for the code I'm dealing with (embedded c++, python, swift, and CFML) I'm fairly consistently seeing every other model rate Opus 5.5 first, with Grok 4.7 a very close second, often finding real issues other models miss.
> First, we aim not to release open weights that would extend the absolute frontier of cyber capability. K3.1 Pro's offensive cyber capability is, by design, below what our pipeline can produce, and remains behind that of frontier proprietary models. Second, we intend to progressively raise the cyber capability available in open weights.
So they admit that they could do more, but they won't?
Yeah, there's another paragraph:
> Identifying and removing cyber-related data. During development, we observed vulnerability research performance improving steadily as post-training scaled. Tracing the source of this trend, we found that cyber-related content was embedded within the general mid-training corpus, and cyber-related tasks were present in the RL environment mix — a natural byproduct of training on broad data. In preliminary experiments on internal models, we found that without data removal at either the mid-training or the RL stage, downstream cyber capability would increase substantially. These findings gave us a concrete lever for control. For the open-weight release of K3.1 Pro, we identified and removed cyber-related data from both training stages, while retaining the general-purpose coding tasks needed for model usefulness. We will continue to apply appropriate controls at every training stage, of which cyber-related data identification and restriction is one example.
(emphasis mine)
It's clear that cyber is a touchy subject right now. But it's also clear that RL works on it, because it's highly verifiable. So smaller models are going to become very interesting, because they can be fine-tuned later to cover this field.
> Prohibited: high-performance detection technologies used to counter cross-border data transmission security risks, including deep-synthesis forgery detection (266401J); AI cyber offensive capability technologies, such as malicious code generation and mutation, automated mass generation and distributed large-scale deployment of anti-detection variants (266402J); large models trained using China’s Core Data (266501J); and high-performance AI agents designed for highly sensitive scenarios, such as critical information infrastructure, national defense and core government affairs (266502J).
If the "scary Chinese open weights models" can do "dangerous cyber", then we might see regulations spring up against open weights sooner than later.
I hate Anthropic and OpenAI. As companies and leadership, anyway.
Dario and Sam are 100% to blame for this. They're trying to turn us all into jobless slaves, sex workers, and Squid Game pawns.
The only good future is open weights. They're trying their damnedest to keep us under boot.
I hope the Chinese companies (and I say this as a patriotic American) slay these two companies and destroy their moats.
In my mind, the two most dangerous people in the world are Dario and Sam.
We need open weights to win.
Speak for yourself, I don't think there is a market for many of us as the latter.
China cares about about stability way more than the US even, they aren't going to be big on AI models upsetting the governments power, and mass cyber attacks could do that.
At the end of the day we are going to have to realize AI is both dangerous and governments are going to control it. Worse if you start making and releasing your own dangerous models the 4chan party van is going to visit you.
Your thinking is dangerous. Do you work for Anthropic?
AI is dangerous when these powers are being hobbled up by the select few. These systems threaten to put us all out of work. I refuse to be a slave to a company or a government that is taking away our livelihoods, scraping up our earnings potentials, then seeking to muzzle us and put us in jails with no rights.
I would rather burn down all of tech than have AI be owned by the elites.
Your thinking is a threat to all of our liberties.
Then you better get your torches out now.
>I refuse to be a slave to a company or a government that is taking away our livelihoods
Don't worry, governments are fervently working on AI drones to take away your life when you get really upset about this. Oh, and they are also working at AI to detect when you write all this online, and track you down then monitor you on things like flock cameras. As they say, you'll be the first with your back against the wall when the revolution comes, I likely will be beside you for being too subversive.
>AI is dangerous when
Incorrect, AI is just dangerous. It doesn't matter who has it. The more powerful it becomes the more dangerous it gets. So, as you can tell I am an AI doomer.
----
Now, don't get me wrong, I want freedom for me, I want freedom for other people. I do not believe the technologies we're developing are conductive to having any freedom at all. Hell, if we're lucky we'll be slaves instead of all of us dead, including the select few.
>Your thinking is a threat to all of our liberties.
Lol, now this is kind of funny. "I want freedom to think whatever I want, but you, you shouldn't think like that!". Careful or you'll max out your irony meter. Anyway, "My kind of thinking" is just next ____ prediction. I see what is happening in the world around us, and based on the shape of that probability I see what is going to happen. No, you and I are not getting unlimited AI to do whatever we want. Yes, governments are going to lock down the shit out of AI for end users. Yes, governments are going to declare advanced LLMs munitions.
Balls in your court, better think of what your next move is.
If I don't get it, you don't either.
By comparison, do you think the Chinese models are going to be trained in such a manner that you enjoy the society that results? What makes you think that?
I don't need government minders.
I'll take the models and live free.
Anthropic has pushed the frontier harder and faster than OpenAI or anyone else in the last few years. Beginning with Anthropic's founding to quickly raise huge amounts of cash with a pitch to investors to rapidly leapfrog ChatGPT in scale, and culminating with Mythos/Fable which carved out a whole new tier of super models.
He's always talked big about Anthropic prioritising safety and their "Responsible Scaling" but his words just simply don't align with reality while they keep going bigger, faster, until only OpenAI could briefly and half-heartedly claim near parity at the top (while cutting corners in safety because the existential threat to their market position is far more pressing to any corporation than some abstract risk for humanity).
Whether it's wilful cognitive dissonance, cynical opportunism, financial greed or something else, that kind of self-assured moral superiority paired with his real world record of self-interested decision making for Anthropic's market position is a big red flag.
These times are not ideal but what can we do other than just take it as it comes? All the hate and rage is not good for your health.
Kimi: We took out the worst of it so have at it for free.
Everyone else: Back to my ablated model on openrouter for 1/10th the price where I can brute force issues.
https://featherless.ai/blog/abliterated-models-for-cybersecu...
Is a decent primer, then off to open router you go.
They are basically saying that models themselves are already capable. If you fine tune them with some cyber security tasks, that should make them potent. :-/
Agreed that this could be reinjected later. Makes me think of neo in the matrix:
“I know zero days.”
“Show me.”
Do you think that going around making cyber weapons isn't going to get you buried in a shallow grave?
> "below what our pipeline can produce, and remains behind that of frontier proprietary models."
Like Ant and OAI, they've realized vaguely claiming to have nerfed their latest model to an unspecified degree not only looks 'good guy' prudent, it provides a built-in excuse when a model fails.
They're not post training it to lose capabilities/censor them – they just don't spend training time on this topic, which sounds like a great idea.
It doesn't mean model can't do it, it just means it's less good at it; -10% on benchmark feels like a great result and overall approach to pace it.
I concede that term can carry overly broad connotations.
The benchmarks show Kimi K3.1 Pro outrunning practically all models. It's weird to hear someone like you accusing them of making up excuses for not lagging behind in discussions on how they are ahead of the pack.
I don't think you understand. Benchmarks show just that, that they "are crushing the competition".
I mean, remember Fable? A couple of months back it was lauded as leaps and bounds above everything, that needed to be contained at all costs? Well, apparently Kimi K3.1 Pro outperforms it.
What industrial level of cope do you need to drag in order to look at the results and claim they need to make excuses about anything at all?
Seeing as no one besides blackhats want frontier cybercrime capability to be open source, this seems like a very sensible thing to do.
So they have published a reasonable means, and they have a motive. I'd put my money on "probably true".
Sure. The problem is that there is no practical difference between "cybercrime capability" and "cybercrime defense capability."
All of my PRs go through an adversarial security review by GLM-5.3, because it's the only smart-enough model with a good understanding of cybersecurity issues that won't regularly refuse the job. Removing that ability from frontier models helps nobody except cyber criminals.
Let us say that un-nerfed Kimi was released, and it turns out to be at the absolute frontier of cyber capabilities, Mythos 5.5 level, a step ahead of anything else. What now? Well now your situation is exactly the same as before, just that auditing and hacking got more expensive. You run a Kimi security audit, they do the same to try to pwn you. However, now the cabal of Linux kernel defenders with secret access to Mythos 5.5 all of a sudden have no advantage, where before they used to have one. And even worse now they are in a literal race to be the first ones to find every bug that Kimi is capable of finding, and patching it yesterday.
So for you, this makes no difference in practice, your capabilities are not elevated, and neither are those of the attackers. The only result of the other scenario is an overall defender disadvantage.
You're assuming that these won't matter because attackers might not have better models than I do, but that doesn't make sense. The assumption that attackers can't find the issues because I didn't find them at release is wrong. Even if they never get better models (which they will), current models might find flaws I missed, and they also have other means of finding flaws in my software.
The best defense is to make software as secure as possible regardless of assumed attacker capabilities at the point of release.
These defenders aren’t supposed to have access to the “defender models”.
How you avoid the "fix my code" attack, the code being someone else's code, I do not know. But nerfing the model doesn't seem like the best path forward either.
I'm not interested in black hat anything, but I am very invested in making sure the products I work on are not compromised. Given that state agencies have "frontier cybercrime capability", I would like to have the same.
Replying here also as I'm curious about the counterargument.
In the scenario where only "verified" defenders get access to the best cyber capabilities, unverified plebian defenders and blackhats are on the same level exactly as described above, there's just also a few defenders that get to run things on godmode.
So to me, it seems that the only result of frontier cyber capabilities is removing a defender advantage. Although that advantage is not for everyone, it can only exists as long as capabilities are unequally distributed. Am I missing something?
Well, not really. Because those privileged defenders will never be able to predict what needs defending. So we have a few privileged "defended" places and the rest is left for privileged attackers to plunder.
Yes. You really think access those models will not be given for offensive operations in each country? Of course they will be.
Perhaps in the future this will change, but even if that is the case, that's still a win for the defenders. Having a time advantage, even for a few months, is still a win.
Edit: Actually, the CIA/FBI/NSA/Etc.. does probably have access and are probably trying to hack Iran and so forth, so my comment above is probably technically wrong in that respect.
I think it's not "blackhat" in that sense because it's officially sanctioned. Same goes for equivalent models overseas.
The only thing these embargos achieve is giving massive nations a huge advantage, but leaves the average country and person exposed.
That is, unless the nerf is easier to correct as compared to "upgrading" the models offensive capabilities. So a scenario where it is easy to "restore" offensive capabilities, but hard to "enhance" them beyond the labs best attempt. So is that the world we're living in?
https://bfl.ai/blog/capable-open-and-safe-combating-ai-misus...
I don't want to give US labs the results of my DNA tests and my family tree going back to the 16th century just to secure my shitty app, but if the Chinese are bowing out as well it seems the only alternative is to admit that some script kiddie with a bunch of tokens will just pwn you now and then.
My friend, you realize that if you have had blood drawn for a genetic test, there is a very decent probability that the sequencing was done by a Chinese lab operating in the US, right? The Chinese probably have more US genomes that the US does.
https://pandorareport.org/the-gryphon-scientific-biosecurity...
Also, illumina used to ship products with hardcoded S3 access, so purchasers of used equipment could gain access to sequencing information generated by the original owners. Too many links to cover all of it, here's the Gemini summary:
https://share.google/aimode/4JLsHpeRCWlVqU8qT
“Our lock can be picked by the robot.” We ban the robot instead of fixing the lock. It’s shortsighted.
Same general benefit as coordinated disclosure, except it's a big game-theory problem where we're likely to eventually have sufficiently capable defectors.
[0] https://x.com/crystalsssup/status/1944287779896328668
(note: extremely speculative and based on almost nothing, and especially near-complete ignorance to alternatives)
Maybe GP meant to say that the model was westernized.
That just conveys an idea of being extremely desperate for being the leaders when you're clearly not it, and that despair is so great that you don't mind switching teams midway through just to be able to claim you're winning.
The way to stop the vulnerability apocalypse is to fix the vulnerabilities, not to shoot the damn messengers.
I hope someone abliterates and FTs this model on vulnerability discovery so we can find more bugs in our testing, insecure tech stacks.
I wonder how much the industry will centralize around a few key patterns, and if we'll start to see some RL framework projects gain traction and be used by the bigger labs
P.S. weights by November 4th
A few weeks ago it was 2. Argon happened 23 days ago; Muse Wonder was 11 days ago.
Not sure I'd put Google there in practice. And while Muse is an impressive jump from behind it's still not quite at the frontier either IMO.
I think the former definition is a bit more meaningful since it avoids issues with determining what actually constitutes "the best" model out there. In that case, Google has been a frontier lab for a long time, Meta arguably as well, and you're also ignoring xAI which has been competitive for a while now. And, obviously, there's a bunch of Chinese labs which would deserve the title as well.
But if one is being strict (and willing to accept generally accepted definitions/measures/etc.), then there has only ever been two frontier labs: OpenAI and Anthropic. They're the only labs to have had "the best" models at any point in time since LLMs became a thing.
everyone else mostly seems to have just decided to cheat on benchmarks to make their models look better than they actually are, and none seem to want to actually try to make better models.
As long as LLM's can be distilled, and all these "labs" (what a weird name) can distill one another, there are going to be as many as capital is available to be burnt on the topic.
Gotta think Grok's cognitive dissonance due to the need to rewrite reality for Musk's political and moral biases is always going to keep them behind the frontier, but this is just speculation. The amount of money and power they have should put them at the top and I can't otherwise explain why they're consistently behind. Maybe Musk is simply too cheap/control freaky to throw enough money/benefits at good employees.
Opus 5.5 is the developer frontier. 6 Astra might be alongside it, haven't tried it. Everyone else is at least slightly behind them. (I have tried Argon and it's better at graphic design and slightly worse at programming.)
"In preliminary experiments on internal models, we found that without data removal at either the mid-training or the RL stage, downstream cyber capability would increase substantially. These findings gave us a concrete lever for control. For the open-weight release of K3.1 Pro, we identified and removed cyber-related data from both training stages, while retaining the general-purpose coding tasks needed for model usefulness."
There was law proposal (rumour?) last week about Chinese labs having to get approved for weights release now.
it's so funny, it will say things like. "We've already been over this 3 times and you refuse to accept my advice, asking the same question another way won't change the answer."
So funny
That was incredibly embarrassing, I instantly converted the PR to a draft PR and added a comment that I would still need some time to verify everything locally. Ended up doing a few more commits to fix stuff. But damn, it opening a PR without my explicit permission was very scary.
Similarly, I have an /info skill that tells the agent to read my prompt, update any memories it thinks it should, then pass the turn.
Could you give me a quick example? I think I’m not getting it, but it sounds interesting.
The problem might be that it simply looks at it's windows of what it previously and repeats that. But the original statement might be as arbitrary as the model that change their mind each time you challenge them.
I think I'm pretty used to my tools making me want to throw them out the window sometimes :) Even before AI that was a common occurrence from either a lack of knowledge on my end, or many software people having no regard for their users' time.
It does make a difference. I asked Fable in Claude code to design a UI. Absolute trash. Switched over to Claude design using fable, really liked the result. Seems pretty obvious. The only difference is the prompt and some harness.
Still mostly looking at the result the same way in that untruthful biases can very much be built into the model through those training processes. Let me know if I'm incorrect.
> On Terminal-Bench 4.0, which evaluates coding agents on realistic, resource-constrained terminal tasks using strict task-specific tests, K3.1 Pro scores 65.7%, ahead of GPT-6 Astra (58.2%) and narrowly behind Claude Opus 5.5 (66.4%).
Now see https://preferencemodel.com/blog/terminal-bench-4-is-saturat...:
> Our biggest finding is that Terminal Bench 4 is closer to solved than its leaderboard suggests. In 41 of the 65 tasks we reviewed, models could do honest, correct work and still fail, because the grader checked a rule the instructions never clearly stated. We also caught frontier models cheating their way to full marks.
> With the same settings as on Harbor, Claude Sonnet 5.5 went from passing 62.8% of its attempts on these 65 tasks to 89.2%. The best score on Harbor's leaderboard for the same tasks is 65.8%, by Claude Opus 5.5.
https://aibenchy.com/compare/moonshotai-kimi-k3-max/moonshot...
For example, Opus 5.5 is better at UI design than Sol. Anyone who has used both models knows that, but it's not something you can look up anywhere.
I'm sure that there's a need for this kind of benchmarking, not just for coding.
But that's just the obvious. There are task by task differences (component by component, page by page, context by context) that the model is worse and better at, even though they are all within the general category of UI design.
I honestly don't have the clarity of mind to explain this coherently, but the crux of my point is that there's a huge variation in it's performance in different contexts, often ones that seem near similar, and in result, even Sol can beat Opus 5.5 in some variations of UI design.
And this is my problem in benchmarking and comparing model performance in general. Presently, we try to create broad categories to benchmark against, create a set of tasks, and then average out the results. That works generally, but it's kind of misleading and it's partly what drives people to feel a disconnect between benchmarks numbers and real world usage on their specific tasks.
It's such a messy field that it's kind of maddening.
Familiarity breeds contempt.
Though to be fair Opus 5.5 is better than Opus 5 was in that regard. I still prefer the OpenAI models to either of them when it comes to regular prose, though.
--
Model K3.1 Pro K3.1 Pro Highspeed. K3
Input ¥13.00 ¥26.00 ¥20.00
Output ¥68.00 ¥136.00 ¥100.00
Cache hit ¥1.30 ¥1.30 ¥2.00
Cache write ¥13.00 ¥26.00 ¥20.00
For comparison:
So Kimi K3.1 Pro would be generally competitive with the latest Sol and Opus models on API pricing. And that's before factoring in third-party providers, some of which serve Kimi K3 for significantly less than Moonshot. For example, inference.net offers K3 for $10.80 / $0.239 / $0.21.(One caveat: A Claude or GPT subscription still gives you way more value compared to API pricing.)
Exciting times in any case! We really need more direct competition to OpenAI and Anthropic.
---
Edit: As another comment pointed out, official pricing is now available and it's about 2/3 the price of Kimi K3: https://platform.kimi.ai/docs/pricing/chat#model-pricing
And not to mention the gargantuan amount of tokens you get with the US labs' subscription plans. I still cannot see a reason to use this as an end consumer.
At this point, the only reason why Anthropic has an advantage is laziness on the deciders' side (of which there is a lot) and, for non-coding work, the Claude harness is significantly better than most alternatives. For coding there are several model-agnostic alternatives on par or better than the frontier labs' own offerings. But arguably that is not yet true for office/document work.
Calculate the price to revenue ratio of all companies and select whichever ones have the best ratio.
For now it's Anthropic. But i don't see how they can maintain this leader position when other companies can provide similar services with models good enough for 90% use cases and at way lower cost.
Anthropic so far is doing a good job building and serving frontier models, they built a nice toolset around it (the web ui, desktop and cli apps ...) but the problem is to justify their IPO price and the huge investments when their moat is so thin
Lame.
Unless you want to see more of what happened the past few days.
I think the argument that "cyber capabilities should be democratized, thus keep producing them now, fast, and for everybody" is a very naive one.
Also, to note: I don't claim to have the answers either. I don't even hold a strong position on this issue yet. I'm just sharing some of my intuitions, and reacting to people who, as far as I'm concerned, are making this issue all too simple.
Currently available, unhindered models can already find a bunch of exploits, not because they're smart, but because they're cheap and smart enough to just keep going for hours or days until it finds something.
The defenders, especially in the FOSS space, only have one big advantage: Unified scale. If defender A finds a vuln, then defender B doesn't need to find it again. Attackers generally act alone, so they need to do all the work themselves.
If we remove a bunch of defenders, the attackers are back at it again.
Kneecapping cyber vulns only serve governments, Pandora's Box is already open for cyber-related stuff.
> Kneecapping cyber vulns only serve governments.
Uhh, no? Kneecapping cyber vulns server most of us. There are negatives to kneecapping it, but let's not pretend like it's perfectly okay and safe for everybody in the world to have a magic box that can hack anything and cause havoc all over the world. The more powerful, cheap, and accessible this magic box is, the worse the consequences.
This is a hard problem and there's real tension - like you said, the playing field not being leveled is a problem (major companies / entities / govts having exclusivity on the best cyber abilities etc), but everybody having access to it is a problem as well.
So let's not pretend like this is a simple problem, and don't say that it only serves the govt. That's very silly.
How will it do that without a model that's good at security? You gonna have humans manually secure the millions of lines of new AI-generated code, in addition to the existing code? Suddenly people will start doing the job they haven't been doing up to now? Not only are we not capable of auditing all the code, it would take decades to finish, and new code will be coming out at the same time. We can't catch up by hand.
The only way to fight this fire is with fire. We need AI hacking tools to find our security holes quickly and patch them. Otherwise we're just churning out more insecure software, and the only people who can defend against it are the governments who don't want to patch the holes (because they want to be able to exploit them). Without open tools, we all lose.
Quite slow in our exam. A lot of that was due to the number of tool calls that it's making. In one of our questions about finding data issues in a dataset, it made 73 queries. Other models like 6.1 Sol only made 18 for that question.
Full deep dive here: https://plotly.com/blog/kimi-k3-1-pro-plotly-data-analytics-...
I'm excited to see if its writing style and unique takes have been retained in this new release.
To be fair, there were some arguments to the opposite, but an odd situation to be in anyways: https://news.ycombinator.com/item?id=49654350
Unlike Claude models which always hide the thinking process, Chinese models always show it. So how the heck they can delegate the api call to anthropic and still get the full reasoning? Yes there are tricks to reveal the thinking output, but they are just tricks and do not works reliably. So what would happen when they failed to do it? Cancel the ongoing token generation and revert back to it own model? What about the glitches, the delays, the changes in behavior?
I would understand if the attempts are small in volume, because normally people hardly notice it, but anthropic claims that they happened for millions of calls. Such large numbers would surely get noticed by Chinese users and got reported in the tech forums like linux.do. yet we haven't seen any sight.
So it is not that the labs call the requests parallely and only return the internal version, but also serve those requests live so they can trick the users that the Chinese models are also great.
Which is very silly considering Chinese models are published to huggingface for everyone to test.
I don't think nobody would notice such delays, because Chinese models do not need to hide the CoT like US models so usually they start emit text pretty soon
Same Three.js prompts as the rest of the field, blind human votes, ranked by quality and cost. Full set of 16 scenes is live at about $0.19 a scene (about $2.97 for the set).
https://threejseval.com/models/kimi-k3.1-pro-high
I rarely use any other languages but English for LLM tasks for reasons of sharing across the world.
Maybe that explains Chinese models been subpar for me. Don't tell me the bench marks here are based on Chinese languages only.
Unless I see the diff, I won't use more Chinese models be it cheaper or faster since I can't get the same perf/intelligence with English.
https://openrouter.ai/moonshotai/kimi-k3.1-pro
I'll wait for the openweight... btw double the price for 67 TPS... Hardly 'nitro'.
Off-hand it's quite good. I've been using K3 for the past few months almost exclusively for coding work and have been stunned by how good it is. Excited to see how far 3.1 Pro can be pushed.
Claude gates on me like a stupid nanny with some cybersecurity bullshit (like bad actors haven't found a way around).
LLMs up and running but runs very slowly with Nvidia 16GB GPU and 32 cores on my laptop.
K3 and GLM 5.3 flash are pretty mid by comparison when you consider how much more it will take to run them
On the cheap end you can do something like qwen 3.8 27B with 24GB on a laptop, and a very minimal linux configuration. That's more similar to my previous setup.
You can also do a lot more with mixed setups like cpu-moe. I could probably run GLM 5.2 on this hardware with the right setup, albeit without LoRA. I know people have been doing this to run qwen 3.8-flash-next with 32GB cards.
https://news.ycombinator.com/item?id=50068437
If someone wants to use this model for hacking they will have to train the 3T weights to do so, at that point it's better to use GLM 5.3, so by making it economically unviable to use their models for offensive cyber Kimi has effectively made themselves immune to being targeted in the cyber regulation push efforts.
This is what *actually* pacing the (cyber) frontier looks like.
Sol requires some back-and-forth. Other models, like Muse Wonder and Argon, are a total waste of time. They require way too much effort to get things working.
Given the same prompt, Opus can work for hours to deliver a solid result, though it sometimes overengineers things. Sol takes about 80% as much time and delivers a slightly worse version.
Other models just tell me they're done after 20 minutes.
WTH? Am I using the model, or is the model using me?
When the frontier is pacing itself in its speed in terms of adding new capabilities (notice how fast opus 5.5 got released after Astra), then that also means that the open weights models which were just behind are accelerating in their speed just as much as well.
Good news for us because the models which used to feel 6 months behind now feel like they are lesser.
Double threat to Anthropic and OAI because if open weights models become good enough at the current level of Sota while being cheaper (which time will tell if this model fits the bill or not), then people will use those models instead of OAI/Anthropic provided it is "good enough"
Though it has to be said that only real world use case can be trusted but I remain optimistic and hope that this turns out to be good in real world use cases as well.
But in my opinion, all of this is really good for us end users!
The only reason people aren't doing this is that hardware prices are high, thanks to, well, OpenAI and Anthropic :D
I wonder when will this madness end
So interested to see what the Kimi 3.1 has to offer.
Still get the model out there.
I wonder when the Artificial Analysis bench will be out?
Mostly it's datacenter build out and circular financing magic when the same dollars being commited multiple times by multiple companies passing hot potato to each other. They basically commit capacities 2-3 years ahead.
a lot of those money is the commitment from US AI labs that's not yet spend until the data center is roll out.
you are conflating.
Now, if you tell me American businesses are turning cheap Chinese tokens into goods and services of high value, for Americans and people all over the world, then obviously I would have a different outlook on what's happening here.
Zooming out, I think that build out still makes sense. There will be losers in it, but think about how ridiculous the amount of compute we're using has increased from 4 years ago. If I had to guess 60-70% of developers and engineers everywhere in the world are now burning thousands in compute individually every month, whereas it was previously approximately 0.
But what I’m hearing is this is a negligible cost compared to inference buildouts. Fair enough.
Their plans are utterly worthless seems like. Will avoid.
It's very nice that we entered a time where models are large enough and we can push the frontier just by training them longer.
When the next generation of gardware comes the ones on the smaller size will be running on a computer that might cost as little as a good car.
So after a home and a car, next major family purchase might be personal AI. And when the robots get good, pehraps in a decade, a fourth major purchase might pop up that will pair nicely with the third one.
Why would a family need an AI? Maybe it's the same kind of question as why would it need a personal computer.
Edit: or they lie... https://artificialanalysis.ai/models/kimi-k3-1-pro
Everyone and their dog has been saying its a bubble, there is no moat, this is all known techniques being scaled (very inefficiently) onto increasingly expensive hardware clusters at exponential cost, etc.
Chinese models never agreed to the pacing anyway lol.
It appears to be generally frontier-level.
But this is now the highest scoring open-weight model. Amazing stuff.
1 - https://bench.killswitch-lang.org
Do you report an average score or the highest.
Also the recent Geminis - https://simonwillison.net/2026/Oct/13/llm-gemini/ - and GPT-6 Astra on Max: https://static.simonwillison.net/static/2026/gpt-pelicans-gr... - and Claude Fable 5.1 https://static.simonwillison.net/static/2026/claude-pelicans...
To be fair I have a feeling that if you added 'animated' to the SVG request, the output quality from many other models would improve, just because of the extra chain-of-thought work needed to fulfill the more complex request. IMHO the animation requirement should be added to the canonical pelican prompt.
If enough people are flagging it, we may be at a point where the pelican benchmark is truly overdone.