pcalau12i

That’s literally China’s policies. The problem is most westerners are lied to about China’s model and it is just painted it as if Deng Xiaoping was an uber capitalist lover and turned China into a free market economy and that was the end of history.

The reality is that Deng Xiaoping was a classical Marxist so he wanted China to follow the development path of classical Marxism (grasping the large, letting go of the small) and not the revision of Marxism by Stalin (nationalizing everything), because Marxian theory is about formulating a scientific theory of socioeconomic development, so if they want to develop as rapidly as possible they needed to adhere more closely to Marxian economics.

Deng also knew the people would revolt if the country remained poor for very long, so they should hyper-focus on economic development first-of-foremost at all costs for a short period of time. Such a hyper-focus on development he had foresight to predict would lead to a lot of problems: environmental degradation, rising wealth inequality, etc. So he argued that this should be a two-step development model. There would be an initial stage of rapid development, followed by a second stage of shifting to a model that has more of a focus on high quality development to tackle the problems of the previous stage once they’re a lot wealthier.

The first stage went from Deng Xiaoping to Jiang Zemin, and then they announced they were entering the second phase under Hu Jintao and this has carried onto the Xi Jinping administration. Western media decried Xi an “abandonment of Deng” because western media is just pure propaganda when in reality this was Deng’s vision. China has switched to a model that no longer prioritizes rapid growth but prioritizes high quality growth.

One of the policies for this period has been to tackle the wealth inequality that has arisen during the first period. They have done this through various methods but one major one is huge poverty alleviation initiatives which the wealthy have been required to fund. Tencent for example “donated” an amount worth 3/4th of its whole yearly profits to government poverty alleviation initiatives. China does tax the rich but they have a system of unofficial “taxation” as well where they discretely take over a company through a combination of party cells and becoming a major shareholder with the golden share system and then make that company “donate” its profits back to the state. As a result China’s wealth inequality has been gradually falling since 2010 and they’ve become the #1 funder of green energy initiatives in the entire world.

The reason you don’t see this in western countries is because they are capitalist. Most westerners have an mindset that laws work like magic spells, you can just write down on a piece of paper whatever economic system you want and this is like casting a spell to create that system as if by magic, and so if you just craft the language perfectly to get the perfect spell then you will create the perfect system.

The Chinese understand this is not how reality works, economic systems are real physical machines that continually transform nature into goods and services for human conception, and so whatever laws you write can only meaningfully be implemented in reality if there is a physical basis for them.

The physical basis for political power ultimately rests in production relations, that is to say, ownership and control over the means of production, and thus the ability to appropriate all wealth. The wealth appropriation in countries like the USA is entirely in the hands of the capitalist class, and so they use that immense wealth, and thus political power, to capture the state and subvert it to their own interests, and thus corrupt the state to favor those very same capital interests rather than to control them.

The Chinese understand that if you want the state to remain an independent force that is not captured by the wealth appropriators, then the state must have its own material foundations. That is to say, the state must directly control its own means of production, it must have its own basis in economic production as well, so it can act as an independent economic force and not wholly dependent upon the capitalists for its material existence.

Furthermore, its economic basis must be far larger and thus more economically powerful than any other capitalist. Even if it owns some basis, if that basis is too small it would still become subverted by capitalist oligarchs. The Chinese state directly owns and controls the majority of all its largest enterprises as well as has indirect control of the majority of the minority of those large enterprises it doesn’t directly control. This makes the state itself by far the largest producer of wealth in the whole country, producing 40% of the entire GDP, no singular other enterprise in China even comes close to that.

The absolute enormous control over production allows for the state to control non-state actors and not the other way around. In a capitalist country the non-state actors, these being the wealth bourgeois class who own the large enterprises, instead captures the state and controls it for its own interests and it does not genuinely act as an independent body with its own independent interests, but only as the accumulation of the average interests of the average capitalist.

No law you write that is unfriendly to capitalists under such a system will be sustainable, and often are entirely non-enforceable, because in capitalist societies there is no material basis for them. The US is a great example of this. It’s technically illegal to do insider trading, but everyone in US Congress openly does insider trading, openly talks about it, and the records of them getting rich from insider training is pretty openly public knowledge. But nobody ever gets arrested for it because the law is not enforceable because the material basis of US society is production relations that give control of the commanding heights of the economy to the capitalist class, and so the capitalists just buy off the state for their own interests and there is no meaningfully competing power dynamic against that in US society.

China does tax the rich but they also have an additional system of “voluntary donations.” For example, Tencent “volunteered” to give up an amount that is about 3/4th worth of its yearly profits to social programs.

I say “voluntary” because it’s obviously not very voluntary. China’s government has a party cell inside of Tencent as well as a “golden share” that allows it to act as a major shareholder. It basically has control over the company. These “donations” also go directly to government programs like poverty alleviation and not to a private charity group.

Personally I think general knowledge is kind of a useless metric because you’re not really developing “intelligence” at that point just a giant dictionary, and of course bigger models will always score better because they are bigger. In some sense training an ANN is kinda like a compression algorithm of a ton of knowledge, so the bigger the parameters the less lossy the compression it is, the more it knows. But having an absurd amount of knowledge isn’t what makes humans intelligent, most humans know very little, it’s problem solving. If we have a problem solving machine as intelligent as a human we can just give it access to the internet for that information. Making it bigger with more general knowledge, imo, isn’t genuine “progress” in intelligence. The recent improvements by adding reasoning is a better example of genuine improvements to intelligence.

These bigger models are only scoring better because they have just memorized so much they have seen similar questions before. Genuine improvements to intelligence and progress in this field come when people figure out how to improve the results without more data. These massive models already have more data than ever human could ever have access to in hundreds of lifetimes. If they aren’t beating humans on every single test with that much data then clearly there is something else wrong.

That’s just the thing, though, the point I am making, which is that it turns out in practice synthetic data can give you the same effect as original data. In some sense, training an LLM is kind of like a lossy compression algorithm, you are trying to fit petabytes of data into a few hundred gigabytes as efficiently as possible. In order to successfully compress it, it has to lose specifics, so the algorithm only captures general patterns. This is true for any artificial neural network, so if you train another neural network with the data yourself, you will also lose specifics in the training process and end up with a model that only knows general patterns. Hence, if you train a model using synthetic data, the information lost in that synthetic data will be information the AI you are training would lose anyways, so you don’t necessarily get bad results.

But yes, when I was talking about synthetic data I had in mind data purely generated from an LLM. Of course I do agree translating documents, OCRing documents, etc, to generate new data is generally a good thing as well. I just disagree with your final statement there that it is critical to have a lot of high-quality original data. The notion that we can keep making AIs better by just giving them more and more data, this method is already plateauing in the industry and showing diminishing returns. ChatGPT 3.5 to 4 was a massive leap but the jump to 4.5, which uses an order of magnitude more compute mind you, is negligible.

Just think about it. Humans are way smarter than ChatGPT and we don’t require the energy of a small country and petabytes of all the world’s information to solve simple logical puzzles, just a hot pocket and a glass of water. There is clearly an issue in how we are training things and not the lack of data. We have plenty of data. Recent breakthroughs have come in finding more clever ways to use the data rather than just piling on more and more data.

For example, many models have recently adopted reasoning techniques, so rather than simply spitting out an answer it generates an internal dialog prior to generating the answer, it “thinks” about the problem for a bit. These reasoning models perform way better on complex questions. OpenAI first invented the technique but kept it under lock and key, and the smaller company DeepSeek managed to replicate it and made their methods open source for everyone, and then Alibaba put it into their Qwen model in a new model they call QwQ which dropped recently and performs almost as well as ChatGPT 4 on some benchmarks yet can be run on consumer-end hardware with as little as 24GB of VRAM.

All the major breakthroughs happening recently are coming from not having more data but using the data in more clever ways. Just recently a diffusion LLM dropped which creates text output but borrows the same techniques used in image generation, so rather than doing it character-by-character it outputs a random sequence of characters all at once and continually refines it until it makes sense. This technique is used with images because uncompressed images take up megabytes of data while LLM outputs only output a few kilobytes in a response, so it would just be too slow to use the same method for image generation, yet by applying the image generation method to do what LLMs do it makes it produce reasonable outputs faster than any traditional LLM.

This is a breakthrough that just happened, here’s an IBM article on it from 3 days ago!

https://www.ibm.com/think/news/diffusion-models-llms

The breakthroughs are really not happening in huge data collection right now. Companies will still steal all your data because big data collection is still profitable to sell to advetisers, but it’s not at the heart of the AI revolution right now. That is coming from computer science geniuses who cleverly figure out how to use the data in more effective ways.

Eh, individuals can’t compete with corpos not just because they have access to more data but because making progress in AI requires a large team of well-educated researchers and sufficient capital to be able to experiment with vast technology. It’s a bit like expecting an individual or small business to be able to compete with smartphone manufacturers. It really is not feasible not simply because smartphone manufacturers are using dirty practices but because producing smartphones requires an enormous amount of labor and capital and simply cannot be physically carried out by an individual.

This criticism might be more applicable to a medium-sized business like DeepSeek that is not really “small” but smaller than the others (and definitely not a single individual) and still big enough to still compete, and we can see they still could compete just fine despite the current situation.

The truth is that both USA and China recognize all purely AI-generated work as de facto public domain. That means anything ChatGPT or whatever spits out, no matter what their licensing says, is absolutely free to use however you wish and you will win in court if they try to stop you. There is a common myth that training AI on synthetic data will always be negative. It’s actually only sometimes true if you train the AI on its own synthetic data, but through a process they call “distillation” you can train a less intelligent AI on synthetic data from a more intelligent AI and it will actually improve its performance.

That means any AI made by big companies can be distilled into any other AI to improve its performance. This is because you effectively have access to all the data the big companies have access to but indirectly through the synthetic data their AI can produce. For example, if for some reason you curated the information the AI was trained on so it never encountered the concept of a dog, it simply wouldn’t know what a dog is. If it encountered it a lot, it would know what a dog is and could explain it if you asked. Hence, that information is effectively accessible indirectly by simply asking the AI for it.

If you use distillation then you should can make effectively your own clones of any big company’s AI model and it’s perfectly legal. Not only that, but you can make improvements to it as well. You aren’t just cloning models, but you have the power to modify them. during this distillation process.

Imagine if the initial model was trained using a particular technique that is rather outdated and you believe you’ve invented a new method that if re-trained would produce a smarter AI, but you simply lack access to the original data. What you can instead do is generate a ton of synthetic data from the AI and then train your new AI using the new method on that synthetic data. Your new AI will have access to most of the same information but now trained on a superior technique.

We have seen some smaller companies already take pre-existing models and use distillation to improve them, such as DeepSeek taking the Qwen models and distilling R1 reasoning techniques into them to improve their performance.