Simulated Worlds

It’s a surprisingly old thought. We hardly had to wait for virtual reality (VR) to come along or Hollywood to dream up this dystopian semireligious tale: humans being kept in an unconscious state by intangible evil machines. The “dream argument” is (at least) as old as Western thought, formulated in antiquity by Plato and Aristotle. And in Meditations, Descartes famously wrote, “On many occasions I have in sleep been deceived by similar illusions, and in dwelling carefully on this reflection I see so manifestly that there are no certain indications by which we may clearly distinguish wakefulness from sleep that I am lost in astonishment.”

In a more contemporary form, as “simulation theory,” the idea has been particularly popular in recent decades, especially in Silicon Valley circles. Its proponents include major tech figures, the most famous perhaps being Elon Musk. “If you assume any rate of improvement at all, games will eventually be indistinguishable from reality,” Musk recently said in a podcast, before adding: “We’re most likely in a simulation.” The idea: Some higher life-form runs a simulation – the motivations behind this differ – and we are nothing more than artifacts in this simulated world. The evolution as an experiment, the world a petri dish. We are made to believe that we exist – but actually, we are not even a dream. We are bits and bytes, that’s all. Reality is somewhere else, on a server farm in a different dimension maybe.

Sounds a bit like Second Life. Remember Second Life? The simulated world made featuring clumsy graphics and oversaturated colors, where people wandered around as avatars? “Second Life is still there, actually,” says Jean-François Lucas, external collaborator at EPFL’s Urban Sociology Lab and expert for digital cities, the sociology of innovation as well as virtual worlds and video games. He studied the phenomenon back in the day and has since turned to other interests, but he estimates the number of regular users is still around 50,000. “There’s a range of different motivations to spend time in such virtual spaces,” he says, one of them a very social one – it’s about meeting people. He thinks that such second versions of the world will always be complementary; they will never replace the “first” version. He doesn’t believe that “we could build a perfect representation of the world, perfect for every single one of us.” What would perfection mean in that context, anyway? A perfect copy or a perfected, upgraded version of the more or less defective world out there? This is all getting very philosophical again, as in Luis Borges’s short story On Exactitude in Science. In the story, Borges imagines the quest for a perfect world map: “In time, the Cartographers Guilds struck a Map of the Empire whose size was that of the Empire, and which coincided point for point with it.”

The mirror world is taking shape

Point for point, pixel for pixel. What if it’s not really a coincidence that simulation theory has attracted such a following in the years since The Matrix? Because, in fact, we are actually building these simulations of the world – for real. The mirror world is slowly taking shape. The term “mirror world” was first coined by Yale computer scientist David Gelernter and made famous by Kevin Kelly, founder of Wired, when he put it on the cover of the March 2019 issue of the magazine. 

The mirror world is actually more than a map, and is not just an updated version of Second Life. In his Wired piece,
Kelly wrote of “emerging digital landscapes” that will feel real: “they’ll exhibit what landscape architects call placeness.” What he meant is that in this second reality, overlying the one we know, representation of things will be more than mappings of the real counterpart. “A virtual building will have volume, a virtual chair will exhibit chairness, and a virtual street will have layers of textures, gaps, and intrusions that all convey a sense of ‘street.”

Gaming companies at the forefront

Science fiction? Recently, a range of companies have come up with very impressive mapped worlds, some of them representations of ours, others dreamlands. There’s a strikingly common point with all these companies: they all have to do with the gaming industry. The first reason for that is obvious. The technology behind games has evolved so rapidly over the last decade that suddenly game environments are starting to feel like whole worlds. And thanks to artificial reality (AR), some gaming experiences are actually spilling over into the real world. Imagination is merging with – or actually becoming – reality. And as users, we are actively helping to build these simulations: Niantic, the company behind Pokémon Go, is currently building a 3D map of the world, hand in hand with its player base. As John Hanke, founder of Niantic, told Wired: “If you can solve a problem for a gamer, you can solve it for everyone else.”

It’s thus not really surprising that the entertainment industry is behind some of the most daring developments in world simulations. The Unity engine, first developed strictly as a platform for games, is continuously extending its range to other applications – in 2019 Disney used it to create backgrounds for The Lion King. And the blockbuster game Fortnite recently launched a series of big live concerts. The biggest one, by Travis Scott, attracted more than 12 million viewers. It might not be so far-fetched to imagine movies and games converging into one and the same genre soon.

Wenzel Jakob is not entirely convinced, however. Jakob, who leads the Realistic Graphics Lab at EPFL’s School of Computer and Communication Sciences, has helped develop some of the algorithms used in rendering these digital realities. “Yes, we have become very good at rendering photo-realistic images – you can see that in the cinema.” But this process is resource-heavy and expensive, says Jakob: “It can take up to eight hours for a single image.” Hollywood can do it, but to achieve the same level of photo-realism in games would require another quantum leap in rendering algorithms. Nonetheless, watching the latest demos using Nvidia’s ray-tracing technology (a novelty that “shook the world” – at least Jakob’s) and Unity’s Unreal Engine 4 feels like a glimpse into this future. “Games are maybe ten years behind, it’s probably only a question of time,” says Jakob. Meanwhile, he’s moved one step further already (see box).

The second reason why development of the mirror world is being pushed by the gaming industry is much less intuitive. It has to do with AI. Nvidia might have started their graphics processing unit business mainly in the gaming industry sector, but it has developed into a crucial provider for AI hardware. In what looks like yet another instance of worlds converging, the company recently announced its plans to build a “metaverse”: “Every single factory and every single building will have a digital twin that will simulate and track the physical version of it. Always,” said Nvidia CEO Jensen Huang in an interview in Time. These twins will not only serve as testing grounds for software. According to Huang, every code and its function will first be simulated and optimized in the digital world before being downloaded into the physical version. It will also become a more and more valuable training ground for AIs.

Layers over layers over layers

I first came across this idea at Applied Machine Learning Days in 2020. Danny Lange, vice president of AI and Machine Learning at Unity Technologies, gave a fascinating talk titled “Simulations – the New Reality for AI” at the SwissTech Convention Center. He explained how real-time 3D video gaming technology can be used to generate “practically infinite amounts of synthetic training data whether for supervised learning in computer vision or unsupervised reinforcement learning.” Anyone familiar with the bottleneck of collecting enough quality data for the training process will realize how much this could change AI paradigms in the future.

So maybe we’re doing this for the machines rather than for us. Whoever benefits most, the result will be total interconnectedness, Kelly believes: “Everything connected to the Internet will be connected to the mirror world. And anything connected to the mirror world will see and be seen by everything else in this interconnected environment.” That might in turn give machines acting in the real world superhuman abilities. They will have a networked super-perception: when a robot is finally able to walk down a city street, it will not see our world, but the mirror world version of that street. It will have devoured previously mapped contours of the city landscape and will be able to merge thousands of sensor perceptions. It will be able to look around corners and through walls, because other robot eyes will already have been there. Simulations overlaying synthetic data overlaying the Internet of Things.

Sounds a lot like a robot overlord tale from Hollywood. Or some techno-utopian vision of an über-world. Jean-François Lucas knows the routine: “We keep reactivating old myths about some super-reality; it’s basically the same story over and over again in different disguises. Technology has advanced, but the narration pretty much stayed the same.”

Other tales sound familiar too, but strike darker chords. Remember the famous Black Mirror episode “Be Right Back,” a modern version of Frankenstein? Again, fiction seems to become reality, as some companies are starting to offer customized chatbots imitating loved ones that have passed away. And with the recent – and truly astonishing – progress of language models (GPT3 as the current state of the art), we can expect to engage deeply with virtual characters in games as well as in our day-to-day realities. As we tend to live more and more of our social lives on digital platforms, these characters won’t actually need an embodiment. Machines are already responsible for the majority of social media content. We know that, and we tend to believe we can live without bots easily, but that is bound to change in the years to come. So, inevitably, our reality will feel more and more like a mixture of the real and the simulated.

That, by the way, touches on the eternal problem of VR: How far away from the real can simulated worlds shift? Some say the actual breakthrough will come with AR – in that case, we will have to wait for the return of Google Glass or a competitor. Others already call it XR: mixed reality. But we still have to learn about these realities, and some valuable lessons might also come from the Immersive Interaction Research Group at EPFL, led by Ronan Boulic. A visit to his lab can be a strange experience. In The Matrix, Choi has a good answer to Neo’s question, by the way: “All the time. It’s called mescaline, it’s the only way to fly.” The mescaline of our days may well be VR goggles. Or at least they can give you pretty trippy experiences. Like seeing your hand as a digital copy, lying on a table, just the way it actually does in front of you. Except that if you lift your index finger, what you see is your middle finger going up. And vice versa. Try commanding the two fingers for a while, and something in your brain goes haywire. WYSINWYG: What you see is not what you get. The aim of the Immersive Interaction Research Group is a mixture of neuroscience and practical VR research, as doctoral student Loën Boban explains. How much “unrealness” can one tolerate in VR and still believe the simulation? The glove experiment, as simple as it is, shows that there’s certainly no clear line to draw here. Boulic believes that given our current technological means, we are still “far away from the matrix.” For him, the “hard frontier” is a mechanical system to actually act in: “Huge progress has been made in tricking the audiovisual perception channels, but that’s only part of the felt reality; it’s another story to simulate the senses of balance, body movement and interaction with the environment without risking actual pain.” In other words: there’s always a wall or a chair in the way when you want to dive into and run or fly around in realistic VR worlds. And how do you simulate a steep hill climb if you’re at home in your small apartment?

Virtual worlds with no exits

“Mirror worlds immerse you without removing you from the space,” writes Keiichi Matsuda, former creative director for Leap Motion, a company that develops hand gesture technology for AR. “You are still present, but on a different plane of reality. Think Frodo when he puts on the One Ring. Rather than cutting you off from the world, they form a new connection to it.”

That’s a vision slightly different from the one science fiction writer Stanisław Lem imagined in 1964. In the sixth chapter of his highly readable collection of essays titled Summa Technologiae, Lem imagined a technology called “phantomatics” that “stands for creating situations in which there are no ‘exits’ from the worlds of created fiction into the real world.” No red pill, that is. But would that be all that bad if the illusion were pleasurable?

What can a person connected to a phantomatic generator experience? Everything. He can climb the Alps, wander around the Moon without a spacesuit or an oxygen mask, conquer medieval towns or the North Pole while heading a committed team and wearing shining armor. He can be cheered by crowds as a marathon winner or the greatest poet of all time and receive a Nobel Prize from the hands of the Swedish king; he can love Madame de Pompadour and be loved back by her.

Sounds great, no? But we are back to the dream argument. Will we even want to keep living in a deficient reality if there’s a much better simulation? And could we be tricked into believing that the simulation is actually real? Loën Boban is a bit at a loss when a visitor brings up the topic. A specialist in robotics, control and intelligent systems, she does not understand the fears; rather, she sees an incredible opportunity: “We could create a world where everybody has superpowers; we could visit places we otherwise would never have the chance to see; we could have close connections with people far away.” She doesn’t believe that this upgrade would be a bad thing. Indeed, she can well imagine exchanging our real experience of the world with this virtual one. But until then, there’s a lot of basic research to be done. “I would be very proud and very happy if I could contribute to the development of such a technology.”