GDC Luminaries AI panel answers questions from game devs | full session

The Luminaries event within the Game Developers Conference featured an AI panel at the Yerba Buena Theater in San Francisco. What was interesting about the session was that the panel of experts only answered questions submitted by the audience, which included executives and seasoned game developers.

Julian Merceron, CEO of Orion Productions, moderated the session and the speakers included Ross O’Dwyer, AI lead at Meta’s Horizon Worlds project; Zhen Zhai, head of the central AI team that helps game developers at Blizzard; Alexandre Moufarek, project lead at Google DeepMind; and Bryan Cantanzaro, vice president of Applied Deep Learning Research at Nvidia.

The first question from the audience focused on thoughts on players and AI sharing agency in a game. What would that be like?

The discussion included technical topics like “engineless” approaches like Google Genie, and the AI projects that didn’t work. The session also covered new experiences for players and new ways of creating content and doing programming. Merceron also asked about things that are now possible only because of AI. Another asked how game development is going to benefit in the coming years from the use of AI.

Another question posed whether game consoles and PC gaming hardware will become better because of neural-rendering techniques and the use of large language models. Two questions focuseded on the surprises that have come up in the last few months. Another question focused on what would a dream game development graduate be now that AI has come on the scene and what are the implications of AI for education. And the compelling question: Can AI become creative in and of itself?

Here’s an edited transcript of the session.

Julien Merceron: It’s going to be your session. I have this iPad here to read your questions. We’re not going to talk for an hour and then at the last minute, ask for questions. You can start writing your questions right now. I’ll receive them and read them. I have tons of questions for the panel. If you’re not into posting questions, no worries. We have a lot of great things to talk about. But the priority is you guys. It’s your session. We’ll listen to what you want to hear about. We might not have all the answers to the questions you have, but we won’t shy away.

First of all, let’s get started with introductions. Who do we have here?

Ross O’Dwyer: I’m the AI lead on Meta’s Horizon Worlds project.

Zhen Zhai: I’m from Blizzard. I lead the central AI team to help game development and AI production.

Alexandre Moufarek: I’m a product lead at Google DeepMind. We’re exploring breakthrough experiences for developers.

Bryan Catanzaro: I work at Nvidia on DLSS and our Nemotron AI models.

Merceron: Players and AI in the same world sharing agency. What are your thoughts on that?

Catanzaro: This is a really interesting question for me, but I’m not a game developer, so maybe the other people on the panel can answer it better than I do. But I’ve often felt like we’re still trying to discover how AI is going to build worlds that are controllable, collaborative, and consistent. Those are the three things I’m always going for.

Controllability meaning that the people building the environment get to express a point of view, an artistic direction that’s telling a story. Consistency–when you look at it from different angles, when you leave and come back, is it still the same thing? And collaboration–there’s something about games that’s tied to people, groups of people. Telling stories, listening to stories, engaging with stories, I feel like as AI develops and has a bigger impact on games, it needs to allow that to continue. People, I think, are going to be more interested in playing games that they can share, and also playing games that have been built by teams that express a joint vision, as opposed to everyone doing their own thing. I don’t know if that’s true or not, but that’s where I start.

O’Dwyer: No, I think that’s absolutely right. I think what we see is that when you introduce a technology like this, the first point you said, controllability, is key. It’s that idea that you can have an opinion – you have to have an opinion – but then you use the technology to explore that space. But then you have to actually bound those. We saw that early in the 2000s or so with stuff like physics. We introduced that, but teams kept it within bounds in terms of how much you could do. Otherwise it would get too chaotic. The actual experience would break down.

Here, with AI, what we have to see in that kind of shared agency is–can we build frameworks around it? Can we come up with a taxonomy to allow people to describe what they want to achieve, but then put a boundary around it so you can explore that space?

Merceron: How do you see this kind of taxonomy?

O’Dwyer: I think that’s fundamentally really challenging. If you go back to 1999 or so and you’re thinking about game design, the game industry was even talking about it back then. “We don’t have a consistent way to talk about systems that are used to make games.” Everybody once in a while pops up and says, “Hey, we should actually have the same words.” That’s getting much more important when we try to explore using models in the game space. When I say one thing and you say another thing and I’m not quite sure what you mean there–if we can come up with a consistent taxonomy to talk about, what does it mean for agency? What does it mean when we say a player explores a space? How do we come up with a consistent language that we work together as an industry to help guide the models?

Moufarek: There’s definitely something about teams finding their own language and way of iterating together. We’ve found that in our team–at DeepMind we have many former game developers, many people who used to be producers in the industry, many game designers and tech artists. Often they find themselves making up their own words that only make sense for their one team. But we go with the flow because we’re defining a new vocabulary for new experiences.

In the end that’s what we’re pushing for: get these core capabilities that will enable creativity to be expressed with these technologies, but also create new things that will only be possible with AI, rather than optimizing things that we do with other technologies that already exist and are really well-designed for that. In the process of exploring these new experiences, sometimes people come up with their own ways of describing different things that make perfect sense to them.

Merceron: We have a question about the field of education. What do you recommend we teach college students in game development when it comes to AI technology? What do we teach and what should we avoid?

Catanzaro: In the age of AI, ideas are cheap. But good ideas are much more valuable. What I mean by that–there used to be things where we had these blank spaces that needed to be filled. Education was often about teaching people how to write an essay that was 10 pages long, or how to draw a picture to fill a page. Now it’s very easy to fill space with ideas, with images. The question is, are they good?

That critical thinking, human critical thinking coupled with the generative powers of AI, I hope it allows us to explore more ideas and express them in more compelling ways, which then amplifies the value of those good ideas. But that means that the purpose of education has to change. Rather than being about scarcity – we have a blank page and it needs to be filled – the purpose of education needs to be more about verification. You’re given an idea. The question is, what does it mean? What message is it sending? What’s the implication of that idea? How can we use that to change the world, to do something a bit better? I think that’s a different kind of education than we’ve had in the past.

Merceron: Zhen, from a studio perspective, who would be your dream graduate? What would you recommend?

Zhai: The core education in game design, in producing content, in cohesive storytelling, I don’t think that’s changing, or will change any time soon. I do think the tools that are coming in will help people get there faster, better design, more cohesive design. They’ll unlock more. I’d encourage students to hone the core idea of what makes a game fun. That’s going to continue to be important. The tools evolve and change so fast nowadays that it’s really hard to catch up. But I believe the tools that are coming to market will help us have more space to focus on designing games. What makes it fun? What makes it attractive to our players? That’s the core.

Merceron: What has surprised you in terms of innovation with AI in games in the past one to three months? This week we’re seeing a lot of talks about AI in games. Is there something that you’ve been able to see for yourself? What kind of innovations have you seen come up in the last few months?

O’Dwyer: Some of the guided storytelling has been pretty interesting to see. Generally, from an AI perspective, seeing the models start training themselves in the last iteration means we’ve kind of hit that inflection point. But for games specifically, teams like Iconic have explored things like The Oversight Bureau. They’re exploring and getting into that space. It’s not just talking about it. Teams are actually trying to have an opinion and build that stuff.

Moufarek: One of the biggest trends I’ve seen is starting to tackle how to give players unlimited agency. Really figuring out how open-ended a game can be while remaining directed and fun to play. Not just doing anything without reasoning or motivation. We’re getting to a point where a lot of the limitations were maybe about how much content we could produce, and how much of that content was personalized to the player in direct reaction to player choice. Now we’re getting to a point where a lot of models are good enough, cheap enough, and most important can generate in real time or close to real time. You can say, “I don’t have the piece of content for that specific reaction, but the model can create something within the world of my game, with intent, without losing creative direction.”

We’re not seeing full games yet, but I think we’re seeing quite a few more short games and prototypes this year. I’ve seen some in the last few months. It’s exciting to see this sort of thing coming up. A lot of people might have an idea, and we’re seeing others building proofs of concept that are playable here at GDC.

Catanzaro: I wanted to talk about graphics a bit. I come from Nvidia. I’ve been working on DLSS for the past 10 years. It’s kind of amazing to think about that, that we started this vision of–how are we going to transform graphics and visual storytelling with neural rendering and AI models? We’re currently on version 4.5, where 15 out of 16 pixels a gamer sees while they’re playing with DLSS turned on are generated by our model. Because of that, it actually looks better than native rendering. A magazine just did a blind test where it found gamers actually prefer AI-rendered worlds over traditionally rendered worlds, and it’s much more efficient. Up to 10 times more efficient. We’re making a small GPU look like a big one through AI.

I’m super excited about the progress we’re making as a community, and especially at Nvidia, in generative rendering with AI. In the past month the things I’ve seen have blown my mind. I honestly think that generative graphics is the most important update to the way that graphics are rendered in at least the last decade, if not more. To the question about what I’ve seen in the past month, I can’t tell you exactly what I’ve seen, but you’ll find out very soon, and it’s going to blow your mind.

O’Dwyer: When you think about changes to the stack, when you think about impression, the ability to describe the objects you want to render–you still have a very traditional pipeline right now. We’re still creating triangles. We’re still modeling things and then compressing them and downloading them. Is there room for us to pass the seed? What does that future look like in terms of hardware on device, where we’re passing through descriptions to the model and it can generate it there?

Catanzaro: The future is yet to be written. There’s a lot of debate about that, about where we’re going. Personally, I am on the side of gradual evolution of the game engine toward generative rendering, rather the side that wants to throw away the game engine and replace it with a fully generative world model. The reason for that has to do with what I said before about controllability, consistency, and collaboration. The thing that’s so amazing about a game engine is it makes all three of those things possible. I think all three of those things are needed.

Now, the question is, could it be a lot cheaper? The drawback of today’s game engines is that to make a triple-A title, you have to spend enormous amounts of energy, time, and money to create the assets that go into a game. It’s definitely going to be the case that generative AI changes the economics of what you have to build in order to make a triple-A game. Will we get rid of the game engine altogether? I think not for a while.

Merceron: I’d like to stay on hardware for a bit, because I think we’re touching on something important. A lot of the effort around DLSS and so on, neural rendering–you can definitely see how this could be pervasive on a wide range of hardware. Consoles, mobile, others. Now the question is, based on all the breakthroughs we’re seeing right now in AI, all the opportunities that exist for the game industry, are we seeing some platforms potentially becoming better than others because they’re better equipped to leverage the innovations that are going to flourish in terms of AI?

For example, on PC right now you can leverage a lot of LLMs. You have advantages in terms of the hardware environment and connectivity. You have a level of freedom in what you can do. Are we going to see other platforms being less structured to access these technical advantages? What do you think?

Catanzaro: One thing I think definitely has an advantage–in a world where you have generative rendering and some sort of sophisticated game AI, whether that’s an LLM or a world model or something else running in real time to make an experience, memory capacity becomes a really important issue for the client. Because it’s definitely true that the more parameters we have to work with, either in the intelligence of how the game works or what kind of scenario the game is constructing for me to interact with, that’s going to be better with more memory.

We’ve already been on a journey where graphics have moved from being triangle and texture and lighting heavy, raytracing heavy, to machine learning heavy. With DLSS we’re pushing more and more graphics into the same computational engine that’s used for all the rest of AI. What that means is we have more flexibility if we have more memory to trade off fidelity between these different competing applications of AI that are going to be running in the games, and if we have one pool of memory instead of two. Today’s systems are built – for good reason – with two pools of memory. Traditional graphics is really a bandwidth-oriented problem where you’re just trying to stream as many triangles through the GPU as possible.

In the future, I wouldn’t be surprised if the traditional graphics loads, although they’ll still be present–again, the three Cs of the game engine that they provide, I think those are very important. But we could deliver better experiences on a unified memory platform, even if there’s less memory bandwidth, but there’s more compute. Nvidia has been working toward this. We have our DGX Spark, which is a little machine right now with an SOC and unified memory, 128 gigabytes. Obviously it’s not cheap. But I hope that over time, this kind of approach becomes more mass market in the PC. It’s already there in the console. All the consoles are built this way. Most Macs are built this way as well. It seems to me like forces are pushing the design of systems toward unified memory platforms, because that flexibility is going to make for better experiences.

Merceron: Zhen, I know you’re very much involved in building tools related to development and leveraging techniques related to AI. We had a question around how game development is going to benefit, in the coming years, from the advance of AI. Based on your work right now, where do you see things going?

Zhai: Right now, we’re trying to leverage a lot of the models and tools out there that can integrate into how games are made today. Meaning that we’re not trying to rewrite how people are making games today. We’re trying to facilitate. We’re trying to find tools for them that can come into how they already make games. The way games are made now has been through years of iteration, to the point today where teams are very comfortable in each of their disciplines, whether it’s design, engineering, art. How do they go from having ideas to going into production and eventually to a game that’s playable for our players? The pipelines are mature. We try to have tools in there to help speed up iteration, get faster feedback from our players, and allow better communication between our teams. That’s where we are right now on AI models coming into play. That’s also how we integrate them into our pipeline to drive real value today.

Merceron: Alex, I have a question for you. Do you think AI can eventually become creative in and of itself?

Moufarek: It’s an interesting question. I always wonder what it would look like in practice. Let’s say you get a magical AI that can make a game with the push of a button. Where’s the fun in that? It’s hard to make games, but the craft of making games is enjoyable itself. It’s an incredibly fulfilling experience working with a team and their different perspectives and so on. I’m more inclined to think that even if the models could do it, we wouldn’t want to use them in that way. There’s a lot more fun and creativity that comes out of a team with a vision and passion that can be empowered and augmented with different tools, including AI.

Catanzaro: Just to back you up on that, we don’t have a problem of lack of games. There are so many games. There are way more games than any of us could ever play. If we had a magic machine where you pushed a button and a game popped out, would it change the ecosystem? I don’t think so. We already can’t play all the good games.

Moufarek: Right. Exactly. Also, the creative process and the art you’re experiencing and consuming–it’s very much the human connection. Knowing someone, one of us, a mortal, has allocated some of their limited time to create this thing for you to discover and experience, I think that’s extremely powerful. Even if the models could be fully creative, end to end, I don’t think that’s necessarily something we want to optimize for.

O’Dwyer: We actually haven’t seen, when people are playing with the technology, that that’s how they intend to use it. They come with an intent. This is the point you were making with smaller teams. The teams are moving faster. They’re multidisciplinary. They’re coming together. But they have an intent. They have an opinion and they’re trying to express that. It’s not like we see teams saying, “Hey, can you solve all this for me and deliver this?” No. They’re leveraging the tools to try to move faster, but with creative intent and with a story they’re trying to tell.

Moufarek: What we say today is that when you take these very powerful models and put them in the hands of someone like one of the game designers on my team, there’s an incredible difference in the quality and the design direction. As there should be. It’s raising the floor for everyone, rather than just creating more of the same in large quantities. That’s not a problem we have or want to solve.

That reminds me of an experiment, a behavioral economics study of products. Part of it was a cake mix. It had only one ingredient. Nothing else. You just added water and you made a cake. It was incredibly unsuccessful. Nobody bought it. There was this assumption that it would be popular because it would save people time and so on. But what the study found is that people had trouble saying, “I baked you a cake” after they’d used this, because they knew they hadn’t actually done anything. They changed one thing in the new iteration of the product. They removed the eggs from the mix. You didn’t have to do much, but you had to choose which eggs, how many, crack them, put in a bit of effort, and then you’d made a cake.

That doesn’t seem like much, but it demonstrates how much, as creators, we care about the time we put into something. We feel very proud to say, “I did this for you.” When you put this thing out there, you might not have made that game for everyone, but there was someone you might have done it for. Probably starting with yourself. But you wanted to find someone who, like you, would enjoy this fully, who would have this before and after moment.

Catanzaro: My kids were so excited about the new Hollow Knight game, Silksong. The developer spent a long time making it. It’s kind of defined my kids’ childhood, that they got to play Hollow Knight, and then they waited and waited while this team poured their heart and soul into the next version. I don’t think their lives would have been as good if there was a new Hollow Knight every week. I don’t think that would make things better. There’s a human part of the culture, about how we share stories, that I think matters a lot.

Zhai: The craft, I think–when you’re playing a game, what the player experiences comes from the designer’s craft. Everybody working on it as a team is expressed in that product. They’re sharing an experience with the player, and players can see that when they play the game. If you have a machine that does that, you wouldn’t have that communication channel through games.

O’Dwyer: The technology allows people with an opinion to get together–if you go back to the 1990s and early 2000s, teams were much smaller, and they were iterating much more quickly. When studios scaled–I remember I was doing some work on the original Godfather. EA split it all into pods, because they wanted to get the multidisciplinary groups together to be able to move quickly. But we had to get so specific with the scale of games that were going on–no, I’m just working on this particular part of the art pipeline. I’m just coding this one specific thing. Teams lost that ability have a collective input into the creative process. I think we’re actually returning some of that. This allows us all to come together and say, “No, we really care. We’re telling a story together.”

Zhai: That cohesiveness.

Merceron: Have we already found examples of things that are not possible without AI? Do we have the belief that right now, based on the different AI techniques that are becoming available, we have opportunities to go into a space that was not possible before? Do you see games like that?

Moufarek: There’s one thing for me, and it touches a bit on creative education as well. Can we personalize the experience in real time? Rather than having a single golden path. In some cases we want that, and that’s totally fine in a specific type of experience. It’s not to say that everything needs to become personalized. But if there’s value in presenting things slightly differently, in telling the story at a different pace as players experiment and discover the world–that’s very hard to do today. It’s easier to optimize for a few options, to find the best middle ground. You playtest your way to something that will work for most players.

What if the tutorial was completely bespoke? What if I didn’t need the tutorial at all, because I just played the prequel and could just jump in? What if I stopped playing for three months and I don’t remember any of the buttons, but I don’t want to do the tutorial again? I’ve played that before. I want something just for me. We’ve been experimenting with this. There’s a lot of this that’s extremely interesting in the context of education. Can we learn from the player’s experience in real time, and then adjust some of the pacing, some of the content, and generate on the fly, when needed or on demand, without requiring developers to think about all the possibilities in advance?

Thinking back to what I was saying about player agency, if I can do anything, but the world doesn’t react to any of it, then what’s the point? Seeing this kind of personalization come in, memory, these kinds of capabilities, is going to unlock some of that.

Zhai: On the tooling side, last year we came and gave a talk about how Blizzard is training internal models because we have so much data. Back then, there were a lot of things–when Blizzard was working on machine learning before the arrival of generative AI, we required a lot of data to have a machine learning model that could do the things our game teams wanted it to do in our pipeline. But with generative AI, diffusion models, all the foundational models of the past couple of years, fine tuning them requires a much smaller amount of data. We don’t need to start a model from scratch. That allows us to do a lot more.

Even though a company that’s so old has a lot of data–I was talking about this yesterday. We don’t use 25-year-old data to make new data. A lot of the things we’ve done over the years–we’re trying to invent new things. We’re making more modern art with greater fidelity. We want to carry that forward. We’re not saying we’re using all of our 25-year-old data to train a model to do future art. It doesn’t work that way. It’s not that we as Blizzard have a lot of data to use. We have to pick what we want to carry forward to create our new content. Generative AI models, foundational models, can unlock that for us to move forward with those capabilities.

Merceron: We’re talking about new experiences for players, new ways of creating content. There was a question here in the list that I thought was interesting. We haven’t talked a lot about programming. Has AI changed the way programmers work? I see that around me, in terms of how much code is generated.

Catanzaro: It’s totally transformed the way that we work. There’s nobody at Nvidia that’s not using AI to help them write their code. The productivity gains have been astonishing. Now, it’s true that if you use AI in a dumb way, you’ll get a dumb result. But if you’re a smart person and you’re using AI to help you solve problems, you can be dramatically more productive.

One game-adjacent example of this from my team recently is that we’ve been working on a number of simulation environments for robotics. We’re trying to teach robot hands how to manipulate objects. I have this dream of a clothes-folding robot. I really want one. I have four kids. I don’t want to spend time folding laundry. We have amazing AI, so we should be able to have robots that cook food and fold clothes and all that. But we need better dexterity to do that, so we have these simulators where we’re teaching robots how to solve tasks in a simulator. We need to have thousands of simulators so the robot can learn thousands of tasks in all sorts of environments.

The people building those environments–it’s kind of similar to building a level in a game. They told me they just got more than a 10X productivity boost. In the past month we went from having 100 environments to having 1,000 environments. Now we’re able to use those environments to help make our AI better. That’s just one example, but I’m seeing programmers across the entire stack at Nvidia, from transistor designers to people inventing graphics algorithms, that are dramatically more productive because of AI.

O’Dwyer: Across the entire software industry, it’s a situation now where if you’re just typing code, you’re probably doing it wrong. But that comes back to the ability–that those who are succeeding have an opinion, and are able to describe what success looks like. They describe what the outcomes look like. You still need to be able to think through the problem space. You need to be able to describe it really well. If you can do that, then you’re working with an AI framework that can execute inside that. You can talk about how to test.

It still requires the ability that you have when you think about engineering. I think about the problem. I think about how I test it. I think about I measure it. I go back to this book that came out in the early 2000s, The Pragmatic Programmer. There’s this section inside of it called Rubber Ducking, where you have to be able to explain the problem. You learn to explain the problem. Again, we’re back to that state where, when you explain the problem really well and you work with the system, you’re able to achieve incredible velocity.

Zhai: I feel like it’s transitioned the work from programming more to software engineering. The engineering part is where you design the software. How the components work together, how to modularize them, how to optimize the runtime and all of that. That all still comes from our engineers. But they probably spend less time just coding the things they have in their heads, and more on architecting how it works optimally. That’s definitely shifted.

Moufarek: I’ve seen how we can have a lot more overlap in skill sets between team members, so they can share a lot more ideas, and share them faster, with no ego around who’s going to code the tool or the frontend piece we need to be able to try the idea. Now a lot more people can code it without having to take time away from the best software engineer on the team to build that specific piece of code. Instead they can focus on the thing they’re uniquely skilled to do, with or without AI. They’ll have expertise in building something that maybe another engineer wouldn’t be able to do, even if they’re using the same model to do it.

That’s been fun to see. You can also just get more time back to focus on doing the thing you enjoy doing the most. Regardless of what we do day to day, we spend an incredible amount of time writing emails. Is that a core skill set, or something we particularly enjoy doing? No, but we have to do it. It doesn’t mean it doesn’t need to be done, and done well, but if I can spend 80% less time doing that and put that toward the stuff I enjoy doing, the quality of everything goes up. You enjoy your work a lot more. You have a lot more fun. You also find yourself trying things you’d never dare to try, because it’s too risky. You don’t have the time. Now we can try that, because we can very quickly test our ideas.

This is what I find most interesting, how fast we can iterate from idea to prototype. My team, our mantra has been, “Show, don’t tell.” Now it’s so much easier, when I have an idea, to try it out. I can build it really quickly and get other people to try it. If it’s amazing, I don’t need to build a slide deck to convince anyone that we should do something with it. Equally, if it’s horrible, no slides will save me. I’m not going to convince you that the bad experience you just had was actually a good one. It’s been very good at accelerating ideas to proof of concept, and then freeing up time for the experts to really go deep on their unique skill sets.

O’Dwyer: It’s exactly the same for us. It’s all demos. It’s not decks. You can actually build something.

Merceron: There are some approaches that have been called “engineless,” a little like Google Genie. What is the potential they have to be able to provide game systems? Progression, robust game mechanics, multiplayer. How far can we go with this kind of approach?

Moufaron: World models are systems that can simulate the dynamics of a world. What you’re looking for is the consistency we were referring to earlier. You want things to behave in the way you expect them to, so that they’re trustworthy simulations. But then the question is, how rich is the simulation, how complex? What kind of interactions, what affordances do you get in them? Today we’re very much at the early stages of what world models can do. They’re already quite impressive, and we’re making fast progress, but there’s so much more to do.

If use Genie as an example, for us, Genie was first and foremost research into building something that will enable us to create a diverse set of realistic or fantasy worlds that are consistent, so we can train general AI systems in them safely, rather than having to deploy them on robots in the real world in a risky way. We want this ability to train up general agents that can navigate virtual worlds, generate worlds that the agents have never seen before – because they’re created on the fly – and then you have the ultimate scientific test. You can’t have seen this before. You can’t have trained on it. But we expect you to be general and adapt to this new environment.

Where do we go from here? What are the core requirements to make these kinds of things potentially usable in entertainment? You touched on one thing when you said that if you don’t have consistency, you can’t have control. You’re generating pixels, not intent and experiences. We’ve seen that we went from being able to have consistent generation of only 2D for about five seconds to 3D across any sort of environment, photorealistic or not, for about a minute. That took about 18 months. With these things progress is often not linear. Expect to go faster from here when it comes to the capabilities of the models in terms of memory and so on.

That leads to the harder question. Can we get to a point where not only is it good enough, but it’s scalable in terms of inference? Can we provide this on a device? Can we add more creative control in how you can condition the generation? At the start of the experience, during, as a reaction to the player, just from the richness of the simulation?

O’Dwyer: You spoke as well about the fact that–when people see world models now they assume that you’re going to have to spend the inference to generate those images. But actually, you can decide not to generate the images and still have the concept of what’s going on inside the world, which could be used to drive other rendering forms, and still end up with something that’s compelling from the world model.

Moufaron: That’s right. What you’re looking for is the ability of the model to imagine what’s about to happen. It doesn’t mean you have to render that. It might not be a requirement for your experience, but it could be very useful for other systems. To go back to your question, it will only get better from here in terms of these fundamentals. But the reason I can’t answer when and if certain tech will land–I think there’s still a lot of research problems ahead of us. We’re still this early, very exciting frontier space where there’s a lot of uncertainty.

For some crazy people like myself and a lot of my colleagues, this is where we like to be, and so we’ve been pushing really hard on this, because a core aspect of our mission at DeepMind is to build artificial general intelligence safely. This is a key component of that. We’ve made a lot of progress on agents, but not really on world models. This is the other side of that coin for us.

Merceron: Is it something that Nvidia is exploring, Bryan, this kind of engineless approach?

Catanzaro: Yes. We’re very inspired by the work that DeepMind and others are doing. There’s a lot of amazing things happening. My personal bias is toward a more evolutionary approach. For example, DLSS started out as super resolution. Then we did frame generation. Then we did ray reconstruction. Future DLSS is going to do a lot more things, a lot more generation. A lot more realism is going to come from taking simple–you can think of the game engine as providing a prompt to a rendering system that’s actually going to generate the pixels. But the game engine providing that prompt is helpful, because it’s a skeleton that’s consistent, controllable, collaborative, and all that.

One thing that I wanted to mention regarding the hardware question, the on-device question–I don’t know the answer to this, but I wouldn’t be surprised if, as we move toward fully AI-generated worlds, we move business models as well. I think the current business model where you buy a piece of hardware that you own, that sits in your living room or in your hand, isn’t very well-suited to AI. AI is fundamentally much more efficient in the cloud. When you batch, when you have multiple people playing the same game, or asking a question together, you can easily get a 10X efficient boost from just using the hardware more efficiently. Then you have the question of, if you buy a console, what fraction of your time are you playing video games? It would be nice if it was 100%, but we have to sleep and eat and go to work and all that other stuff.

If you think of AI as compute-limited, which I do–at Nvidia that’s what we see. You get more capabilities from your AI when you put more compute into it. Then you have a 20X or 50X compute advantage by moving to the cloud. That seems like something that’s going to change the business model, as opposed to, you buy a machine, put it in your house, and then buy a game the way I did with my Nintendo in 1987. It’s going to be something more like Netflix, where you have a subscription. The subscription will power AI inference for you. The latency concerns about how this is going to work for games are much more solvable than technical concerns about how we’re going to get this much compute into the hands of consumers.

O’Dwyer: You guys have explored that, but still, the DLSS model you’ve spoken about, you’re doing that on-device at the end of the chain. What does it look like, from your perspective, when–the challenge I see is that you can talk about what chip you might stick into a modem so we all have some last line of compute. Everyone’s excited about what that will look like, and in three months it will look like something entirely different. But how do you think about that?

Catanzaro: Maybe DLSS or neural rendering evolves into a really awesome compression system for hybrid systems. We have a world model in the cloud that’s bringing to bear the most frontier capabilities, to ensure the artistic vision and the consistency of the world is absolutely amazing. And then it’s sending over, basically, skeletons that are elaborated on device by some sort of neural rendering system. I think that’s probably going to work better, because the cost of running the world model is going to be basically unbounded. I believe the amount of intelligence we’re going to want to put into that world model is unbounded, therefore the compute is unbounded, therefore it won’t be a great thing for a consumer to try to buy. But they can rent some slices of time on it. It’s going to be 50 times more efficient, which means a way better experience running that expensive thing in the cloud.

Merceron: With all the opportunities that AI is bringing and all the cool things that we think we’ll be able to do, could we touch base on things that we’ve tried and actually didn’t work? Or some of the big challenges we see remaining ahead of us?

Zhai: We’ve definitely had our share of failures. We try a lot of things. If we take a step back and look back at why they didn’t work–our team, my team at Blizzard, is centralized. We have a lot of researchers. We look at this new technology and what it can unlock from our perspective. The things we do, a lot of the time we’re thinking, “This could solve this problem for this group.” But when we actually talk to that group, it might not be the pain point that they want us to tackle. Even if we have new algorithms coming out that might solve a particular problem, it’s not the thing they want us to work with them on.

A lot of the time, the things that have worked at Blizzard happen when the dev team – engineers, designers, and artists alike – comes to us and says, “We have this pain point. Can you help us with this?” Sometimes we say, “No, the technology’s not here yet.” Sometimes we’ve tried it out, but then we can go back in two years and realize we have a better solution because the technology has evolved. Sometimes we can say, “That’s the perfect problem for us to come in and help you solve with the data and the approaches we have.” That’s what works. When we’re trying to bring the technology in, though, most of the time that doesn’t work.

O’Dwyer: For us the piece has been learning to skate beyond where the puck is going. If you actually work with the models as they exist today, the limitations that are there now, you will immediately be surpassed by the models, surpassed by some other technology, surpassed by something else that’s there. The thing we’ve had to learn to change in the way we approach this–assume that if it’s obvious, it’s likely that the one of the models will build that core capability. Think about what you would have to plug in when that’s happened. By the time you’ve built that thing, generally if it’s a larger system, we’ve seen that come up.

That’s been true for–like, we’re going to do a whole bunch of context management. That’s no longer necessary. The fundamental model has gotten more robust. Now we’re going to try to structure work in this particular way. No, actually the agentic tooling has gotten to a better place as well. There’s been wasted work from that perspective. Sure, it was super useful for about a day, but very quickly the capabilities are coming out. We’re just trying to assume that those pieces are coming out and that’s worked well.

Catanzaro: Almost everything we try fails at first. But if we believe in what we’re doing and we keep iterating, we usually make it work. These days I feel like the possibilities are so expanded. It’s so much more clear – how AI is going to change the way we work – than it was a few years ago. It’s kind of astonishing to me. I remember at GDC a few years ago I was talking about how AI was going to change gaming. Back then it felt kind of theoretical. Now it feels like, “This is how it works. This is the day to day.” That doesn’t mean it’s easy. A lot of things fail when we first try. DLSS itself, the first version wasn’t great. But we made the second version a lot better and went from there.

O’Dwyer: Do you think there’s a lesson about perseverance there? I think that’s one of the challenges we see right now. Because it’s so easy to come up with ideas – ideas are cheap, as you said, but good ideas are still really valuable – you pull something in and you try to think, “How do I turn this into an actual product?” That’s where the real work is. You have to have that energy.

Catanzaro: AI makes demos easy, but it sometimes makes products harder. We’ve seen that over the past couple of years. There have been so many mind-blowing demos that never made it. I have no doubt they will one day, if people keep pushing. But yeah, there’s a gap.

Moufaron: We had a product we worked on called Project Astronaut, which was a model that could see, hear, and speak. We were using this as a kind of real time assistant on your phone. Then one day we said, “Why not try this in a video game? You could have your own sort of gaming companion all the time.” It failed miserably. The reason why–it wasn’t the technology. It was because the experience was not the right one.

In the context of productivity, you have a one on one relationship with an agent. I have this task. I want to solve it. You give the model some context, you get some output, and you use that to solve the problem. Whereas here there is no problem to solve. I’m enjoying this video game. Then you have this AI that comes in as a third party in the middle of that experience. What felt like an obvious thing that would be super fun with the technology we have turned out to be extremely hard, because the problem wasn’t getting it plugged in and running. The problem was, “How do you make this fun? Can it even be fun?” Then we gave up.

But I think that will come back, because there’s something we’ve learned from this experience. There’s still something on our minds about how this can be transformational. We might have started in the wrong place. I think there’s potential around making games more accessible to people who are blind, low vision, or other disabled players. We can uniquely enable something like that with an AI in the loop. You have a friend next to you on the couch who’s your eyes, who can play in a certain way with you. I think there’s something there.