Ilya Sutskever – We're moving from the age of scaling to the age of research

Dwarkesh Patel (Host) 00:00.100

Yeah. Um, final question. What is research taste? You're obviously the person in the world who is considered to have the best taste in doing research in AI. You were uh the co-author on many of the biggest the biggest things that have happened in the history of deep learning.

Dwarkesh Patel (Host) 00:23.540

learning from Alex Net to GPT-3 to so on. What is it that how do you characterize how you come up with these ideas?

Ilya Sutskever (Co-founder and Chief Scientist) 00:32.100

I can answer so I can comment on this for myself. I think different people do it differently. But one thing that um guides me personally is an aesthetic of how AI should be by thinking about how people are. But thinking correctly. Like, it's very easy to think about how people

Ilya Sutskever (Co-founder and Chief Scientist) 00:56.140

are incorrectly. But what does it mean to think about people correctly? So I'll give you some examples. The idea of the artificial neuron is directly inspired by the brain. And it's a great idea. Why? Because you say, "Sure, the brain has all these different organs, it has the

Ilya Sutskever (Co-founder and Chief Scientist) 01:12.880

folds, but the folds probably don't matter." Why do we think that the neurons matter? Because there is many of them. Kind of feels right, so you want the neuron. Yeah.

Dwarkesh Patel (Host) 01:22.280

You

Ilya Sutskever (Co-founder and Chief Scientist) 01:22.360

want some kind of local learning rule that will change the connections. You want some local learning rule rule that will change the connections between the neurons. Right? It feels plausible that the brain does it. The idea of the distributed representation. The idea that the

Ilya Sutskever (Co-founder and Chief Scientist) 01:37.320

brain, you know, the brain responds to experience and neural net should learn from experience, not response. The brain learns from experience, then neural network experience. And you kind of ask yourself, is some is something fundamental or not fundamental? How things should be?

Dwarkesh Patel (Host) 01:52.600

Yeah. And

Ilya Sutskever (Co-founder and Chief Scientist) 01:53.640

I think that's been guiding me a fair bit, kind of thinking from multiple angles and looking for almost beauty, beauty, simplicity, ugliness, there's no room for ugliness. It's just beauty, simplicity, elegance, correct inspiration from the brain and all of those things need to

Ilya Sutskever (Co-founder and Chief Scientist) 02:10.200

be present at the same time and the more they are present, the more confident you can be in a top down belief. And then a top down belief is the thing that sustains you when the experiments contradict you. Because if you just trust the data all the time, well sometimes you can

Ilya Sutskever (Co-founder and Chief Scientist) 02:26.280

be doing a correct thing but there's a bug. But you don't know that there is a bug. How can you tell that there is a bug? How do you know if you should keep debuging or you conclude it's the wrong direction? Well, it's the top down. Well, how should you can say the things have

Ilya Sutskever (Co-founder and Chief Scientist) 02:38.680

to be this way. Something like this has to work. Therefore, we got to keep going. That's the top down. And it's based on this like multifaceted beauty and inspiration by the brain.

Dwarkesh Patel (Host) 02:50.560

All right. We'll leave it there.

Ilya Sutskever (Co-founder and Chief Scientist) 02:52.560

Thank you so much. Thank

Dwarkesh Patel (Host) 02:53.520

you so much.

Ilya Sutskever (Co-founder and Chief Scientist) 02:55.800

All right. Appreciate it.

Dwarkesh Patel (Host) 02:56.800

That was great. Yeah.

Ilya Sutskever (Co-founder and Chief Scientist) 02:58.040

I

Ilya Sutskever (Co-founder and Chief Scientist) 02:58.080

enjoyed it.

Dwarkesh Patel (Host) 02:58.720

Yes, me too. Hey

Dwarkesh Patel (Host) 03:00.160

everybody. I hope you enjoyed that episode. If you did, the most helpful thing you can do is just share it with other people who you think might enjoy. It's also helpful if you leave a rating or a comment on whatever platform you're listening on. If you're interested in

Dwarkesh Patel (Host) 03:14.280

sponsoring the podcast, you can reach out at dwarkesh.com/advertise. Otherwise, I'll see you on the next one.