vfx animation

The crazy multiverse shot in ‘Doctor Strange in the Multiverse of Madness’ had its own intense VFX journey

In what may well be the stand-out sequence in Sam Raimi’s Doctor Strange in the Multiverse of Madness, Strange (Benedict Cumberbatch) and America Chavez (Xochitl Gomez) are transported through multiple portals and multiverses before arriving at Earth-838.

Framestore, led by visual effects supervisor Alexis Wajsbrot, orchestrated the sequence, landing on an array of different-looking multiverses but all based on the characters traveling through a ‘similar’ New York street environment. The environments ranged from mirrors to paint and even honeycomb, all requiring significant art direction, asset building and effects simulation, plus a unique approach to compositing to bring them all together.

Wajsbrot worked closely with production VFX supervisor Janek Sirrs and production VFX producer Cyndi Ochs to design the sequence and explore methods for making it as dynamic as possible, even contemplating the shooting of live-action elements of Cumberbatch and Gomez with a high speed robot camera, although ultimately the characters were 100% CG in the final 40 second sequence.

befores & afters talks to Wajsbrot about the long journey involved in making the incredible multiverse shot, from tests, to shooting explorations, to how his team pulled those insane environments off. He also reveals several Easter eggs to look out for. You rarely get an insight into the crafting of a visual effects sequence from go to whoa like this. Come along for the ride.

Testing the multiverse, and how it had to differ from what came before

Alexis Wajsbrot: This all went back a long time. Janek and Cyndi contacted me not that far after Spider-Man: Far From Home [Wajsbrot and Framestore worked on the stunning illusion battle]. So, I knew pretty fast that we all wanted to work together again on Doctor Strange in the Multiverse of Madness. At that time, the movie was still directed by Scott Derrickson and even in this very first iteration, there was already some kind of a ride that would mimic the sequence of the first Doctor Strange–the Magical Mystery Tour.

Of course, it needed to be different but it would echo the craziness of the Magical Mystery Tour. We didn’t work on that sequence, but I was always amazed by the amount of creativity in it, and how hard it is to do a long shot. So, it was the end of 2019 when we started doing tests on the multiverse scene.

One test we did was where Doctor Strange was melting and his skin was spilling into liquid. It was not yet ‘paint world’, it was a touch different, but we knew that they were going to be falling. What we didn’t know yet was that it was going to be just America and Strange. In the very first iteration it was, I think, all the key characters of Doctor Strange, there were models, there were stranger, they were all falling into some kind of a freefall.

In fact, the big difference between some of that older ride was that we really wanted to affect the characters so when they entered a world, it would affect them with a kind of tornado of smoke and the character would also merge into this gas and smokey stuff.

Still a portal ride, but things begin to take shape

When Sam took over as director, we kept the idea of doing a portal ride, and named the sequence ‘America Portal Ride’. The only thing that Janek felt strongly about was it should be a freefall, it should be like when a base jumper falls from the plane and the jump is actually really, really fast. You’d have a camera that sometimes can be far away or sometimes can be quite close to them.

We started doing previs here at Framestore directly by the VFX team. I felt in order to sell that previs we may have needed a bit more FX than usual. So, I wanted to involve our new visdev department, which is capable of putting some moving images together very quickly. Even though they work really fast, they still make a moving image that is a really, really good starting point. In fact, I felt that we were missing that department during the first Doctor Strange when we were doing Mandelbroting and the magic effects.

The ‘one New York street’ idea

We started with previs of a freefall, and at that point the shot was only 15 to 20 seconds long. Animation supervisor James King and I were saying, ‘Oh, it would be really cool to mimic the Magical Mystery Tour,’ where it started fast and then at some points it stops, and starts up again. And the world where they stop, you’d see a wide view of the Multiverses. But, you know, what does a Multiverse look like? It’s not a lot of Earths because that would be ‘multi-Earth’. It’s ‘Multiverse’. There was also ‘where’ this all happened. They were always starting in Kamar-Taj and they were supposed to finish in an alleyway. Of course, in the movie they finished on the rooftop, but that was a really late addition. That was changed about three months before delivery!

The point is, we were designing previs for a while, going through these ‘worlds’ in slow-mo and then we received a note from Marvel saying, ‘It’s a bit deja vu, let’s do something else.’ And I’m not sure where it came from, but there was a reference from a movie where a guy is in front of a window of a store and you see the different seasons and you see the mannequins in the store changing clothes, changing very quickly, and you see the same store changing through the window.

We thought, ‘What if the same world is just continuously changing as they go through the multiverse?’ So, because we were supposed to arrive in some kind of alleyway of New York we said, ‘Well, let’s do that one street of New York that they built for the Gargantos scene. Let’s use that street and just change that street many, many times.’ So, in the end, what we did was have the same New York environment that changed from world to world.

How to film actors in freefall

Our original previs had Strange tackling America, but it was all a little static as they fell. Everyone felt that, on the freefall, it would be great if they were moving fast and very action-y. So we said, ‘Well, we can tackle faster and they can transition so they are now moving a bit more within the different worlds.’ That’s when we started techvis, to try and work out how to shoot this, knowing that there’d be close-up moments on America and Strange.

Now, I was keen to shoot as much as possible, even if at the end we were only going to be using some of it as reference. I was really keen on shooting almost the whole shot if possible. We took the Bolt X robot, which is a rig that can move very, very fast with a camera on it. The robot was supposed to move really fast around them and we were supposed to shoot in slow-mo.

We went as far as doing some tests with the Bolt X with some actors. At some point, though, it was felt it could be a bit too dangerous to move this robot so close from Benedict Cumberbatch and Xochitl Gomez. Janek also said, ‘I can promise you, Alexis, it’s going to change. The camera is going to change.’ So we completely changed the idea. We abandoned the idea of shooting the sequence like this.

Strange and America: all CG

At that stage, I made the call that it was going to be full CG. This would give us complete freedom. We knew we were going to build our digi-doubles to a very high standard for other shots in the film. We’d already worked out that the super close-up of America waking up underwater and Doctor Strange transforming into a cube would be CG, too.

We discussed it with Janek and we still felt we should shoot something. Back on Geostorm, we had the idea that if you shoot with 15 cameras surrounding the actor, you can do a photoscan per frame and you can then re-project the 15 cameras. We created with Rob Duncan and Richard Graham a setup called doppelgänger, which is a Framestore thing that we created, where we shoot with 15 cameras all around Gerard Butler and we created this digital scan per frame, where it was a changing topology per frame, and we were projecting that to do all the astronauts in that movie.

Here we shot with five ARRI Alexas as an array. It’s not enough to create photogrammetry per frame, it’s just not enough cameras. But what we could do, since our digi-double rig was so high res, we knew we could do a great facial track and still use the re-projections.

We shot Benedict and Xochitl in different lighting conditions. At that stage, I said to Janek, ‘I’ll be honest, I’m not sure anymore I’m going to use any of that. I think it’s going to be more work to use any of that rather than for me to do it in full CG.’ But for me it was such a great reference for facial animation. And in the end, I only took it as a facial reference, we didn’t do the facial tracking and re-projections. It’s fully digital, fully 3D.

Alleyway or rooftop, re-visited

Initially for all the worlds they go through, we were all thinking we would be doing a lot of 2.5D where we would do basic geometry and a lot of DMP and touch up on top. But as soon as the camera starts to freefall, and as soon as the tackle was a lot more violent–where they were now moving in space a lot more–we knew that we couldn’t do anything like that, and that it was going to be 3D for all the worlds.

It was at that point, too, that things changed. The whole concept was that they are freefalling into different kinds of worlds. They were always supposed to go from New York to New York in different styles and then at some point they were supposed to land into an alleyway, as I mentioned.

At some point somewhere, they said, ‘Oh, actually now they are going to land on a rooftop.’ We were doing so much work already where we were at New York at ground level, but suddenly the whole ride would need to be at rooftop level, to end up on the rooftop. I thought it might change the dynamics of the shot, partly because it would only be sky that you see. You won’t read the speed as much because you’re not close to the ground. And then we all said, ‘Well, you know what? Let’s find a transition world that could go from street level to roof level.’

We said, ‘Okay, what if we create some kind of ancient world where suddenly the geometry is a bit twisted so suddenly you are at rooftop level because of that transition world, which was ancient world.’ At some point, the ancient world became an ancient desert world–the feedback was, ‘We need to be careful of not making that exactly like a world from Doctor Strange.’ We couldn’t mimic anything from Doctor Strange, it needed to be different, needed to be fresh.

So, we had to create that transition world to go into. I’m not even sure when you watch the sequence you realize at first that you are at street level and then you end up at rooftop level. It just goes by so quick. But it worked well for us to construct it like that.

World-building: breaking down just a few of the key multiverses

Living Tribunal Statue World: We wanted to start in Kamar-Taj so there was always going to be some kind of a view of Kamar-Taj that was very telescopic. We created a concept very early on, that was back on the first version, which was called Giant Temple world or Giant Indian statue world. And at some point, it was changed to the Living Tribunal state world, which is a world that actually exists in the Marvel comics.

Onslaught Canyon World: At some point this world was almost dropped because it was a bit reminiscent of the Grand Canyon shots in Spider-Man: No Way Home. This one is now a bit more like the movie 127 Hours with James Franco.

Honeycomb world: At some point our line producer said, ‘Oh, what about Honeycomb world?’ We all said, ‘Hey, that’s cool. Let’s do a concept of Honeycomb World.’ And everyone loved it. In the last three weeks we started feeling, ‘What does this world need?’ As artists, we were like, ‘It’s boring, we need to bring some life.’ We then thought if something lives in Honeycomb World, it must be giant bees. So, we started to populate giant bees. We ended up putting in some water as well for the transition to work better with the next one when we are underwater.

Mirror World: Animator Mike Brunet pitched the idea of a machine world. Which became a glass world, which ended up being slightly more economic, since for glass world, of course, you have to build the inside of every building.

Cube World: That was one we added from the very beginning – a world where suddenly they all transformed into cubes. It was very technical to make the hair attach to the cubes and to slice it while it was still animating. We did lots of different versions of how we would cubify things. First, we were emitting small cubes and we sliced Strange, but that was too gory, so we changed the approach.

Paint World: Paint World is obviously a big simulation for the characters, and there is some simulation for the background as well.

Comic Book World: Interestingly, this was never supposed to be a What If…? Easter egg. It was always supposed to be a comic book Easter egg. That’s why the design of the Comic Book World is very different to What If…?, which is a lot more stylized. This is a lot more about taking influence from some of the comic books. If you read some Spider-Man comic books or Doctor Strange you’ll see the colors are very vibrant, so this world was always supposed to be this way. There is a lot of compositing of DMP of some of the shadows and the lines and all of that because that was the best way to do this world.

Post-apocalyptic World: At some point, we felt it would be cool that, what if in Avengers, they lost and so New York has been completely crushed.

Underwater World: For Underwater World, we go from real-time to slow-mo, and we go underwater! We had to go close-up on America’s face, and we had to simulate that cloak. It was really challenging.

Alt New York: In this world there was no crowd and no cars for a long time, and then in the last three weeks we said, ‘Well, what can we do to bring life?’ So, we added the fish coming from Underwater World which fall onto cars and the road. That’s an idea from Janek to make sure that the fishes fall into the windscreen onto New York. The fish falling into the windscreen were done in the last two weeks of production. We also added cars with people in the cars and crowd, that was very, very late in the day. There’s an Eastern egg here from Sam Raimi and Bob Murawski, the editor, they have this company Grindhouse Releasing, which is on one of the taxis.

Pipe World: An Easter egg here are the drones from Spider-Man: Far From Home, and what we did there was transition from the New York taxi to the drones, they are positioned in the same way.

Savage Land: We were going to have a jungle world from the get-go. Then in talking with Marvel, it appeared that there was a similar world that already existed in the Marvel universe called Savage World, where there were giant dinosaurs. It seemed like a really cool idea, so we made a classic T-Rex versus Triceratops fight.

Hydra World: This is like an old-time New York world. If you watch it in IMAX, there is an Easter egg which is a Hydra Easter egg on the giant blip. It’s not visible in the 2:35 release, it’s only visible in IMAX. We did put a Hydra logo on it. Our original idea was a world that got stuck in the ’30s and that Hydra took over.

A world that didn’t make it: One world we had was a ‘James Turrell’ world, who is an artist who does very simple pieces with light. It’s a shame because it was very simple to do, but it didn’t make the final cut.

One sequence: 20 shots

We worked in animation and in simulation as one shot. Then we were rendering each world as a different shot. That gave us 20 shots. The pipeline we followed was that each time the animator baked the animation, they populated the bake into 20 shots. And so, we had 20 different light rigs so the characters are always lit completely differently from world to world–it’s a completely different lighting setup each time. Having the 20 environments within one scene was just impossible, it was just way too heavy on everything. So, we treated it as 20 different shots.

We did comp 20 different shots. There were at least seven or eight different comp artists on the 20 shots, and then there was one artist who put everything together, which was a massive deal because of all of these portals. We added a crazy number of handles per world. Most of the shots have more than 100 frame handles for the portals to make sure that we have enough of the world before, and the world after. Then there was all of this portal trickery about the fact that we always need to see them as they went through the portal was very challenging. There was almost a recipe per transition.

Final thoughts

This sequence is every possible thing you can imagine–hi-res digital doubles, a lot of FX, and then combining all of that together on a Marvel timeline. I think it’s definitely a shot we were not capable of achieving when we were doing Doctor Strange with that kind of time constraints and that kind of cost efficiency. We were not able to do that five years ago and I think it’s great to see how far we have come and how efficient we are. I’m really proud of the team.