# Vision models can converge on the same features

Markdown copy of https://realalpha.app/s/vision-models-can-converge-on-the-same-features/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

- Speaker: [Joscha Bach](https://realalpha.app/people/joscha-bach/) · Founder, California Institute for Machine Consciousness · Developed cognitive architectures and leads CIMC's research into machine consciousness and models of the mind.
- Host: [Tim Scarfe](https://realalpha.app/people/tim-scarfe/) · Host, Machine Learning Street Talk
- Show: Machine Learning Street Talk
- Episode: [Why Consciousness Might Not Be Hard At All (Joscha Bach)](https://realalpha.app/v/yW_ngvMFlYs/), Nov 2, 2026
- Moment: 41:01 to 49:30 of the episode (8 min), playing on https://realalpha.app/s/vision-models-can-converge-on-the-same-features/; the episode from there: https://realalpha.app/v/yW_ngvMFlYs/?t=2461

## Quote

> “it really depends on the **structure of the problem** that you are trying to learn.”
> — [Joscha Bach](https://realalpha.app/people/joscha-bach/) at [42:02](https://realalpha.app/v/yW_ngvMFlYs/?t=2522) · Machine Learning Street Talk

## Takeaways

1. Different architectures and datasets can produce similar vision features.
2. Bach says task structure may shape representations more than model choice.
3. Picbreeder's human raters and tailored activations may bias its results.

## About this segment

Vision models can learn similar features across architectures and datasets, Joscha Bach says. He argues Picbreeder's human raters and tailored activations may explain its strong representations.

From “Why Consciousness Might Not Be Hard At All (Joscha Bach)”, Nov 2, 2026. [Watch the full episode](https://realalpha.app/v/yW_ngvMFlYs/)

## Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

[41:14](https://realalpha.app/v/yW_ngvMFlYs/?t=2474) That's a super interesting question. I suspect that at some point it might be cheaper and easier to start again because true machine perception has never been tried. The data that we give the LLM is stuff that is either produced by humans or it's made for human media consumption, which means it's adapted for our mostly visual and auditory sensors. But there is no reason why you should be limited to the human sensorium, right?

[41:37](https://realalpha.app/v/yW_ngvMFlYs/?t=2497) You could have something that has much higher frame rates and looking at very different frequency ranges all at once. And so you can, in principle, build a much richer, more detailed world model that is coarse graining in a different way. You mentioned the universality hypothesis indirectly, this work by Chris Ola. What his team discovered is that when you are looking at the different vision models, you find that they all learn the same features.

[42:02](https://realalpha.app/v/yW_ngvMFlYs/?t=2522) And that's regardless of the architecture and the particular learning algorithm and even the details of the data. And so the idea behind this is that the structure of your model does not really depend on the algorithm or on the data, but it really depends on the structure of the problem that you are trying to learn. And you find that despite the learning in the transformer algorithm or state-space model or other perceptron algorithms is very, very different from the way in which the brain is learning, the visual architecture of the brain is also very similar to a computer vision model.

[42:35](https://realalpha.app/v/yW_ngvMFlYs/?t=2555) That is, you have at the lowest level colors and spatial frequencies, and then you build this into textures that are combined, and then at some point you have geometries and so on. And the deep question is, at which level does this stop? The universality hypothesis would be universally true. It would mean that if... You train a model on your own in input and output, and you train it for long enough, you will wake up inside of that model.

[43:04](https://realalpha.app/v/yW_ngvMFlYs/?t=2584) And it's a quite mind-blowing idea to me. And I don't know if that is actually the case. And there might be a reason why it's not the case. Maybe language is insufficiently constraining the causal structure of the system that is producing language. Maybe there are many ways in which the same set of linguistic utterances that a human being is producing can be produced. But if you think about it, a human being ultimately is just an end-to-end trained learning system, end-to-end trained over an evolutionary time span, but also your biography.

[43:31](https://realalpha.app/v/yW_ngvMFlYs/?t=2611) And these together produce your behavior, and you would assume that biology roughly gets to the simple solution to produce your output, given your input. And so if you take a neural network that is trained to produce similar output when given the same input, you would also think that this new algorithm that we have implemented is not doing any kind of magic that ends up with something that is way more complicated to produce the same output, but ultimately it's going to converge to a structure that is similar to yours, to a similar causal structure.

[44:00](https://realalpha.app/v/yW_ngvMFlYs/?t=2640) But there might be conditions when that is not the case. For instance, maybe certain things run well on the brain, but they don't run well on a GPU. Or vice versa, there is a lot of stuff that is easy to run on a GPU and almost impossible to run on the brain. So the brain is going to use workarounds. In fact, I believe that our GPUs are more powerful than brains. Basically, the set of algorithms that can run on GPUs is not limited in any way.

[44:23](https://realalpha.app/v/yW_ngvMFlYs/?t=2663) You can impose the same constraints as you have in biology in principle. These are the limits of the resolution, the speed, the size of the memory that you have, and so on. But if you have a hardware that is fast enough, you can simulate an arbitrary natural system to an arbitrary degree of fidelity, whereas your brain cannot simulate a digital computer, simply because it's very mushy. So your brain is limited to the way in which it works.

[44:47](https://realalpha.app/v/yW_ngvMFlYs/?t=2687) And so if it turns out that our own consciousness is an artifact of our brain trying to make complicated intelligent stuff work, and the computer does not need the same artifact, it could be that the computer comes up with something that is more clever and more elegant than our own consciousness. yes okay so i wanted to give a little bit of challenge on that so i was speaking with um akash kumar and he was the co-author with kenneth stanley of this fractured entangled representations paper and um i don't know if you've heard of pick breeder which is this thing that he did ages ago and basically it's like an evolutionary open-ended search and the punchline is it found much better representations in a neural network than a normal neural network does that when you changed some of the parameters because it was carving the world up by the joints it would actually do semantical

[45:33](https://realalpha.app/v/yW_ngvMFlYs/?t=2733) things you know that the skull's mouth would open and you could make it wink and stuff like that and maybe there is something wrong about pig breeder but the way i understand that it works it's using a neural network that has been specifically tinkered for making representations like this so it's something that is not simply using relu's for its output but it might be using gaussians or sign functions and so on as the output function they can be combined to make patterns that repeat or they can be make a pattern that are symmetric and so basically they tweak the neural network that they're using from the beginning to be suitable for this the other one is that the way in which pig breeder is trained is not automatically it's trained with a population of human raters so the fitness function is basically humans who look at the output of

[46:21](https://realalpha.app/v/yW_ngvMFlYs/?t=2781) pig breeder and say this is a good start and this is how we should be proceeding and so i think that this process of the construction of the pictures together with the mechanism that produces the patterns is biasing pig breeder to make stuff that is more organic that is more controllable and so on but i believe this could be imitated right we could automate the procedure by which human beings are selecting pictures by using a diffusion model that is rating the output for similarity to human output so we could use an ai that is in principle already tweaked to imitate the behavior of the human juries that are performing this thing.

[46:57](https://realalpha.app/v/yW_ngvMFlYs/?t=2817) And we could also evolve architectures that have a larger variety of activation functions that are more suitable for this and ultimately outperform pig breeder by this. I think that Sutton's bitter lesson is also going to be true for this one. So if we basically take the human out of the equation and instead just evolve the search space or search the space of solutions automatically, we are also going to end up with something that's more elegant than the hand-tinkered solution in the case of pig breeder.

[47:25](https://realalpha.app/v/yW_ngvMFlYs/?t=2845) All of that is absolutely true. i mean yes so cppns they they are sparse and sparse networks are great if only we could train sparse networks right and i think that we want to have ai which can actually be built and that you know so you know rather than neural networks now they're a bit like a pile of sand they have too many degrees of freedom and we kind of like sparsify and we turn them into a sand castle and wouldn't it be better if we had the lego analogy where we just put little blocks of lego we're in now and we have models that carve the world up by the joints in the case of pig breeder we have a system that is inherently limited because these networks are very small right they're very sparse and so you have relatively few parameters that you need to optimize and you set up the model in such a way that most of the parameters that you're playing with are meaningful

[48:10](https://realalpha.app/v/yW_ngvMFlYs/?t=2890) on the other hand it's limited because you're not going to get pig breeder to produce something that looks arbitrarily complicated and if you want to move in these realms then you have to have tools that are much more complex it's similar to a print would use very good user generated content using blender versus minecraft so with minecraft it's very easy for people to produce meaningful stuff and it's almost difficult not to produce something with minecraft that looks somewhat meaningful that you can play around with and walk around in and have fun in and if you have a complex 3d program the inverse is true because most of the stuff that you're doing with this complicated thing is not going to be interesting and it's not going to look exciting but it will look like oh my god this person has no ability to use its tools and so that is one of the issues

[49:00](https://realalpha.app/v/yW_ngvMFlYs/?t=2940) when we are building learning systems when we make them very simple it's easier to control the complexity similar as it is in art and it can be exciting to work with constraints and if you move the constraints the task becomes much more frustrating because you have to suddenly to conquer an enormously large space, and this space might be simply too large for human beings.