Image: Disney/Marvel
Welcome to your multiverse.
At the end of May, I participated in a fascinating experiment: Disney, Marvel, and Industrial Light & Magic teamed up to produce a mixed VR/AR immersive gaming experience for the Apple Vision Pro based on their What…If? TV series.
Also: If you have an Apple Vision Pro, Marvel’s ‘What…If?’ is a must download – and it’s free
Dave Bushore, director and executive producer at Marvel Studios
I consider this an experiment because it is as much a concept prototype as it is a game. First, it’s exclusive to the $3,500 Apple Vision Pro headset. Second, the game is free. And third, the game’s interface is based on the headset’s ability to recognize hand gestures. Participants move their hands in emulation of what the heroes of the Marvel universe (particularly Doctor Strange) do in the movies.
As games go, it’s fun, but nothing to write home about. The gameplay feels fairly on-rails. But the immersive aspects — which are impressive and thought-provoking — demonstrate one take on how games and UI elements are likely to evolve in a spatial computing environment.
But I wanted to know more. What were the design decisions? The challenges? What was Disney thinking? I mean, really. What kinds of things was the team thinking?
So I asked. Dave Bushore, director and executive producer at Marvel Studios, and My-Linh Le, associate producer at ILM Immersive, were kind enough to answer.
Let’s get started.
ZDNET: What inspired the creation of an immersive version of the “What…If?” series for the Apple Vision Pro?
Dave Bushore: I believe that “What If…?” is the perfect entry point. It begs the question to the audience from the beginning, and it allows for characters and crossover points to play however best fits the story. In this case, it was developing a story for an entirely new piece of hardware that allowed for that to be authentic to the series in a new way.
ZDNET: What were some of the biggest challenges in adapting the “What…If?” series for a mixed reality format?
DB: The biggest challenge was landing this new conversation that we’re having with the audience across their journey on this new format. We knew that we needed to develop a story with pacing that really lands with the audience, despite not knowing where people will be looking or how they are participating.
My-Linh Le: We wanted the audience to feel like the hero of the story [and it does —DG] and how they could impact these different multiverses, whether good or bad. It was a challenge balancing trying to show as much of the “What If…?” Universe as possible while also keeping the audience engaged in their role as the hero.
My-Linh Le, associate producer at ILM Immersive
ZDNET: How do you see mixed reality changing the landscape of entertainment over the next five years?
DB: Adoption. As the boundaries continue to get pushed technically, as the hardware, software, and devices continue to get better, it will open the door for bigger and better stories that we can tell — and that the audience will want — in mixed reality.
ML: I think and hope that mixed reality will be able to bring worlds that we’ve always wanted to go to into our homes, where we can talk and hang out with our favorite characters and create our own stories and adventures with them.
ZDNET: My-Linh, that’s such a compelling mission statement. I love it! What advancements in mixed reality technology do you anticipate will be available in the next decade?
ML: Over the next decade, I would say better spatial room and object recognition. I think once we can do that, we can personalize experiences more and make them even more engaging.
DB: The development of photoreal characters in real time will be an emotional game changer. Because the technicalities are so hard, when that happens a story can truly be attached in people’s minds that they lived it or are currently living it.
ZDNET: How do you envision the integration of mixed reality in other Marvel and Disney properties?
ML: I would love to see how we can make fans be a part of the actual story, influencing outcomes, being a menace if they want to, and having experiences with their favorite characters that are unique to them.
DB: I truly think as things develop, it’s a universe of unlimited possibilities… It will be defined only by the limit of how far we can push the ideas people can participate in. The next best story in this space is only limited to what we can imagine.
ZDNET: Can you discuss the collaborative process between Disney, Marvel, and ILM in developing this project?
DB: The collaboration has been ongoing for many years regarding the possibilities of mixed reality and virtual reality stories being told from Disney and Marvel. ILM Immersive has been a great partner and leader in this space, which gave us a huge tailwind both technically and creatively with so many talented and passionate people on this project.
ML: With Dave guiding and leading us through his creative vision, we were in constant communication with each other at all times. Whether that was through story meetings, creative reviews, headset reviews, etc., we felt like one cohesive team even though we were across multiple studios.
ZDNET: What are the key elements that make a mixed reality project successful in terms of audience engagement?
DB: The biggest element that I believe is holistically a huge component in these types of projects is presence. Feeling the presence of the characters in a realistic way, which we’ve been able to do with ILM Immersive successfully over the years.
Also: How to use your Meta Quest in a moving car (as a passenger!)
With Apple Vision Pro, it takes it to a whole new level. As the audience, participant, hero… feeling the performances of the characters and moving through the environments authentically creates an opportunity to create a presence that you are living in this reality. To feel that, on some level, it is real and opens the audience’s minds to engage with it on that level.
ML: I also think that anything that makes folks experience something new is key. If they can walk away from it saying that they’ve never done something like that before, that’s an important part of what makes a mixed reality project successful with audiences.
ZDNET: Well, that was certainly the case here! How did you ensure that the immersive experience remains true to the original “What…If?” series?
ML: We had a lot of guidance from Dave and the “What If…?” team. However, a lot of us at ILM Immersive were already huge Marvel and “What If…?” fans, so a lot of people would regularly chime in with their Marvel knowledge throughout development.
DB: When developing the story, our writers David Dong and Phil McCarty were focused on the key attributes of the Disney+ series, but also the core of the “What If…?” comics and the fun that those stories always unlocked. We worked off a few key ideas that we wanted to accomplish creatively with ILM Immersive, worked those into the main arcs for the characters, and then had a lot of fun saying “What if…” in every meeting for a few months!
Also: Headset making you sick? Here’s an unlikely cure for VR-induced nausea
ZDNET: How do you balance technological innovation with storytelling in mixed reality projects?
DB: For me as the director working with our writers, my producer, the creative team, and within our Marvel Studios teams, it was about which elements of the story were the most important to the characters and you, the hero, that could create an emotional journey for all. Then there was layering in the wish fulfillment possibilities that this technology is beginning to unlock. There were so many ideas that we wanted to try from an innovation standpoint, but I think you’re successful when you can combine the tech with the emotionality of the entire journey you’re trying to create.
ML: I think we generally approach it from what would make an incredible story and experience, and then assess the feasibility. We always try to push the boundaries of what the tech can do and have a “let’s explore everything mentality” with every project, but first and foremost we need it to make sense for the story that we’re telling.
ZDNET: I want to highlight your phrase, “the wish fulfillment possibilities that this technology is beginning to unlock.” That’s deep — and somewhat unique to this medium. What lessons have you learned from this project that will influence future mixed reality endeavors?
DB: The biggest lesson I learned was that the future belongs to those who continue to combine genres, formats, developmental modalities, gaming characteristics, and storytelling, as it’s the biggest opportunity in this space.
It is both entirely new to teams and audiences and at the same time not, creating unique challenges and opportunities. The chalkboard is blank and future projects and endeavors by everyone will only expand the innovations and audiences’ excitement for more content hopefully.
ML: To add to that, we did a lot of test explorations for this project that I think we’ll definitely see in the future. There’s so much potential for what can be done.
ZDNET: How does ILM’s expertise in special effects contribute to the development of immersive experiences?
ML: Our art team is incredible. We get to work with some of the best in the business, with many people at ILM Immersive coming from more traditional VFX work as well. I am genuinely impressed everyday with the quality and fidelity of what our team is able to create, and I think the experience speaks to that level of work.
ZDNET: What are the biggest opportunities and challenges for mixed reality in the entertainment industry?
DB: The biggest opportunities are to have more watershed creative moments, audience participation with the hardware, and social repositioning through ecosystem expansion.
We need more genre-defining content that people can share with their friends, allowing stories to cross over the boundary of niche entertainment to undeniable experiences that are generationally defining to both older and younger audiences.
ML: Accessibility and reach is also a major challenge in this space. A lot of folks haven’t been able to explore mixed reality yet because it can often come with a large financial barrier. I hope in time it becomes even more accessible, as that will really allow adoption of mixed reality storytelling to boom.
ZDNET: What advice would you give to other companies looking to venture into mixed-reality entertainment?
ML: The best advice I would give is to push boundaries as far as you think the tech can go. You’ll never know what’s possible until you know how far you’re able to go.
DB: Focus on the question of “why?” It greatly helped us narrow the focus and create a great sandbox for a “What If…?” story to live in, both from an innovation standpoint and also a narrative one. Why are we telling this story? Why are these characters going on this journey? Why are we using this gesture, this model, this location? If it matters to you, it will matter to someone else.
Also: The best VR headsets: Expert tested and reviewed
Food for thought
This interview gave me a lot of food for thought, and I hope it did for you as well. I’ve been exploring XR and spatial computing for quite a while, but when some of the best storytellers in history explain why it excites them, a deeper understanding snaps into place.
I look forward to seeing where this team takes their explorations next. What do you think? Have you tried the What…If? Immersive experience? Have you dived into VR or AR? Do you have a Apple Vision Pro or Meta Quest 3 headset? Who is your favorite Marvel character? Let us know in the comments below.
You can follow my day-to-day project updates on social media. Be sure to subscribe to my weekly update newsletter, and follow me on Twitter/X at @DavidGewirtz, on Facebook at Facebook.com/DavidGewirtz, on Instagram at Instagram.com/DavidGewirtz, and on YouTube at YouTube.com/DavidGewirtzTV.