MODEL COLLAPSE
On systems that eat themselves, and why this matters beyond neural networks.
When I started training generative models on my own image archive, I expected the technical challenges. What I didn't expect was how much the failures would teach me about systems in general. About feedback loops. About inbreeding. About what happens when a system only sees itself.
What Is Model Collapse?
In machine learning, mode collapse happens when a generative model stops producing variety. Instead of learning the full distribution of the training data, it collapses onto a single mode, a single pattern, a single answer to every question.
You ask for blue glass. You get the same blob. You ask for red ceramic. Same blob. You ask for something organic, something geometric, something dark, something bright. Same blob. Always the same blob.
The model has found a local minimum. A safe answer that satisfies the discriminator. It's not wrong, technically. It's just dead. It has stopped exploring.
The generator says: "Then why try anything else?"
And the conversation ends.
The Technical Reality
In a GAN, two networks fight each other. The generator tries to create images. The discriminator tries to detect fakes. They should push each other toward excellence.
But sometimes the discriminator gets too good, too fast. It learns to reject everything the generator tries. The generator, desperate for any approval, finds one trick that works. One pattern. One blob. And it clings to it.
The balance breaks. The feedback loop becomes a death spiral.
In My Project:
- V10: 76 features. Too many signals, not enough data. Collapsed immediately.
- V11: Lowered learning rate. Still collapsed. The problem wasn't speed.
- V15: D/G loss ratio dropped to 0.08. Discriminator dominance. Generator gave up.
- V16-V18: ADA-p dropped from 0.8 to 0.02. The discriminator was winning too hard.
But It's Not Just Neural Networks
The more I watched my models collapse, the more I started seeing the same pattern everywhere. Systems that stop exploring. Feedback loops that eat themselves. The death of variety.
GENETICS: Inbreeding Depression
A small population breeding only with itself. Each generation, the gene pool shrinks. Harmful recessive traits accumulate. Vigor decreases. The population becomes fragile, unable to adapt. Same principle: limited input, repeated feedback, collapse.
ASTRONOMY: Closed Systems
A star cluster isolated from the galaxy. No new material flowing in. Over billions of years, the heavy elements concentrate. Star formation changes. The system evolves toward a single end state. Entropy wins when there's no outside input.
SOCIETY: Echo Chambers
A group that only talks to itself. Algorithms that show you what you already believe. Each interaction reinforces the same patterns. Dissent is filtered out. The discourse collapses onto a single mode. We call it polarization.
ECONOMY: Monopolies
When one player dominates, competition dies. Without competition, innovation stops. The market collapses onto a single mode of production. Efficiency in the short term, fragility in the long term. Same pattern.
Micro and Macro
What strikes me is how the pattern scales. The same dynamic plays out at every level:
- Molecular: Crystals forming the same lattice over and over. Prions forcing proteins into a single shape.
- Cellular: Cancer: cells that have forgotten how to differentiate.
- Organismal: Inbred populations. Monocultures. Species that over-specialize.
- Social: Cults. Totalitarian states. Markets without competition.
- Computational: GANs. Language models trained on their own output.
- Cosmic: Heat death. Maximum entropy. The universe's final mode collapse.
The scale changes. The math changes. But the shape of the problem remains: a system that loses access to external variety, that feeds only on itself, will collapse onto a single mode.
Diversity is oxygen.
Without it, systems suffocate.
The Incest Metaphor
I keep coming back to incest, as a structural category.
When my GAN trains only on outputs from itself, it's the computational equivalent of a family tree that folds back on itself. Each generation, the pool of possibilities shrinks. The model becomes more confident and less capable. It knows exactly what it is, and that's all it can ever be.
This is why AI researchers worry about "model collapse" in large language models. As the internet fills with AI-generated text, and new models train on that text, we risk creating an ouroboros. A snake eating its own tail. A system that has forgotten what human thought looked like.
The Habsburg jaw of machine learning.
What Saves Us?
In genetics, the answer is outbreeding. New genes from outside the population. In astronomy, it's gas clouds. Material from elsewhere falling into the system. In society, it's the outsider. The immigrant. The heretic. The one who hasn't heard what everyone already knows.
In my project, the answer was simpler: more data, fewer features, and knowing when to stop.
I went from 76 features to 6, which was what my dataset could actually support. I stopped asking the model to learn things it couldn't possibly learn from the data I had.
Sometimes the solution to collapse is less complexity, fresh input, and stopping before the system eats itself.
A Final Thought
I started this project trying to teach a machine to see like me. What I learned instead is how systems die.
They don't die from lack of intelligence or lack of resources. They die from lack of variety. From talking only to themselves. From optimizing so hard for one metric that they forget all the others.
The collapsed model isn't stupid. It found a solution. It just found the only solution it could see. And that's the tragedy: it will never know what else was possible.
Maybe that's what intelligence really is: the ability to resist collapse. To keep exploring when you've found something that works. To stay hungry when you could be satisfied.
The generator that keeps trying new things, even when the discriminator keeps saying no.