ByteDance Is Training a 10-Trillion-Parameter Model To Chase the Frontier 30
ByteDance is reportedly training an AI model with roughly 10 trillion parameters as it tries to close the gap with leading frontier systems such as Anthropic's Mythos. The model is still in early pre-training, and its eventual performance will depend on more than scale alone, but the project underscores how aggressively Chinese firms are pushing frontier AI despite limits on access to advanced chips. The Next Web reports: The size is itself the statement. At roughly 10 trillion parameters, the model would be more than three times as large as Moonshot's Kimi K3, which sits among the biggest Chinese models today at about 2.8 trillion. [...] Parameter count is not everything, of course. Bigger models are not automatically better, and the industry has learned that data quality, training technique and efficiency often matter as much as raw scale. Even so, committing the compute to train a model this size is a declaration in its own right, a signal that ByteDance wants to compete at the very top rather than ship a capable also-ran.
More Trustworthy than American Models (Score:1)
Re: More Trustworthy than American Models (Score:1)
Why? (Score:5, Interesting)
That said, the huge Chinese models we've seen lately aren't single models, they're MOEs - Effectively a hive-mind of many smaller models working in concert. Bytedance hasn't mentioned that detail yet but it would somewhat mitigate the training issue. This is still absurdly over-spec'd though. There's simply no way to train it any more than all the rest starving for data.
Re: (Score:2)
Re: (Score:2)
If you try to map this to evolution of brains, consider the biggest problem.
Where garbage in == garbage out, much more garbage in == much more garbage out. Size of a model doesn't inherently make it more useful, just as a huge volume of brain mass doesn't make you hyper-intelligent.
This isn't even able to be mapped to Nyquist thinking, or von Neumann. Bigger isn't better; better is better.
Re: (Score:2)
I don't know if the Chinchilla scaling law (which is what I assume you are referring to) includes RL-based post-training, or even non-text input tokens for that matter?
The base model certainly makes a difference, but for areas like math, coding and hacking (to compete with and counter Mythos) the power of the end model mostly comes from the RLVR post-training, and these are all areas where they can generate synthetic data, as well as pay for human-generated data if they care/need to.
So much for the US attem
Re: (Score:2)
starve? no sir, merely rerouting of transport to make sure certain new ventures for certain family members were tied in the supply chain. FTWof few
Re: (Score:2)
Probably the same reason why GPT-3.5 was 1.4T and now much better models fit into 4B (or possibly less): Scaling up by increasing the number of parameters is simple, getting the same in fewer parameters is harder.
One can hope they optimize for the next models to be smaller, because 10T is surely larger than the level of knowledge/intelligence they will achieve in a reasonable training time. And 10T will be very costly to host as well. I don't think anyone will even pay the self-cost price for inference with
Re:Why? (Score:4, Insightful)
Not for lack of will or money, but for lack of data - We're already training top models with effectively the entire digitally-available corpus of human knowledge and need about a thousand times more.
This has to be nonsense, fundamentally speaking. "Data" is not some kind of fungible resource. Humans are 'trained' (in school) using a much smaller set of data for many subjects and given that we haven't reached AGI-levels of intelligence for those yet data clearly cannot be the bottleneck.
You can argue that we are lacking data for understanding/mastering of specific aspects of the universe (touch, motor control, etc.), but for many, many things a larger network might still allow capturing more subtle relations in the dataset or conversely more data might not improve anything.
Re: (Score:2)
The key difference is, humans aren't "trained" primarily via a clean library of tagged data (which in itself is a learned skill). We experience the world as a more-or-less continual stream of noisy data from our several senses. We self-train how to interpret (and primarily, filter) sensory data; and more importantly, how to derive new training data from it.
To be clear, I'm not saying we have so many neurons to support input filtering (
Re: (Score:2)
You ignored the difference in subjects and treated data exactly as a fungible resource. Walking around and looking at the sky and grass is not the same as learning about quantum mechanics. Think about the _value_ of the data you yourself ingest: Is all that sensory data from you taking your morning shit really improving your understanding of the universe?
Also, "current top models" famously do NOT train using a clean library of tagged data. The next token in the input is the output, remember?
Finally, a lot o
Re: (Score:2)
Current top models are already undertrained by multiple orders of magnitude vs their number of parameters. Not for lack of will or money, but for lack of data
Because of bureaucracy. Individuals all up and down the hierarchy are trying to meet arbitrary metrics set by someone who doesn't understand the situation. The ones who meet the metrics get promoted, even if the metrics make no sense.
Of course, there are smart scientists in China who try to do good research, but as is true everywhere, bureaucracy is a hindrance.
Re: (Score:2)
There is plenty of data the west has no access to.
For example all the not yet scanned (or if scanned not published) Chines texts.
And that is basically true for every esoteric language like the main Thai languages (they have 63), Indian or Indonesian or Filipino or god forbid: everything from Africa.
Or the corporate data of BYD, CATL or GWM etc.
Then again: depending on what you want to train, you can use synthetic data.
if some is good more is better (Score:2)
More is not always better.
A flattening curve (Score:4, Insightful)
We are reaching the flat part of the curve, where a change in the order of magnitude in model size is yielding only slight, incremental benefit. We are reaching the hard limits of what large language models can do.
Re: (Score:2)
AI is, and always has been, exclusively nothing but exponentially-increasing requirements for logarithmically-plateauing results.
Any mathematician will tell you what that means in the end.
Re: (Score:2)
AI is, and always has been, exclusively nothing but exponentially-increasing requirements for logarithmically-plateauing results.
I was going to say this is way too general, and AI involves many algorithms that are not LLMs.
Then I started thinking through the algorithms of the past, and many of them have indeed shown great promise in the beginning, then exponentially-increasing requirements.
Re: (Score:2)
I was saying this 20+ years ago when I was studying AI and its history back then, long before LLMs existed.
Re: (Score:2)
How long can the people who have invested billions in OpenAI et al keep stringing this out before everyone and his broker finally realizes that LLMs are dead ( in terms of investment payback, not as a useful tool ) and the money goes "poof" ?
model measuring contest (Score:3)
It's not the size of the model, but what you do with it. or so I am told...
How much? (Score:2)
With enough parameters (Score:2)
What is missing from existing models? (Score:2)
Right now my company gives access to the best models the US provides . I get as-good-as-human results for coding and general information.
What is it that training with more parameters is going to give? What's the goal?
Is this just marketing and AI has already reached its limit?