rotoscopy services, Rotoscoping services

Part 2 of our coverage of the invisible effects behind the ’Shang-Chi’ bus fight.

‘How did they shoot this in San Francisco?’

 

Simu Liu as Xu Shang-Chi / Shaun during shooting of the bus fight.

Bus breakdown

Part 2 of our coverage of the invisible effects behind the ’Shang-Chi’ bus fight.

In part 1 of befores & afters’ coverage of the visual effects for the bus fight in Shang-Chi and the Legend of the Ten Rings, we talked to production visual effects supervisor Chris Townsend about planning and shooting certain elements for the fast-moving brawl.

Now we turn to VFX studio Luma Pictures, which crafted the sequence’s final visual effects shots to make it appear as if the action scene really was all filmed in San Francisco. The studio brought together live-action location plates, reference, survey data, LIDAR scans and bluescreen photography, and built a range of CG elements such as bus, car and garbage truck vehicles, environments and even digital-doubles, to make the final dynamic shots possible, including the menacing blade wielded by Razor Fist.

Luma visual effects supervisors Andrew Zink and Alex Cancado explain further.

b&a: Luma has done some great ‘transportation’ sequences in recent Marvel films, like Black Panther and Captain Marvel. What had you learnt from those scenes that perhaps helped here?

Andrew Zink: I think that’s in part why [production VFX supervisor] Chris Townsend and [production VFX producer] Damien Carr. wanted to reach out to us for this sequence in particular—because of the work that happened on Captain Marvel and because of the work that happened on Black Panther. Just like with anything, you learn the pitfalls as you go along, and what to start to look out for ahead of time, what to ask for, or anticipate as you’re starting to do your prep and execution. But every show is a snowflake. Every one of them has their own unique challenges and barriers that you need to break through and this was no different from Captain Marvel or Black Panther.

The bus and garbage truck at the Sydney shoot.

They had done two separate shoots; one that was based in Sydney on their sound stage with the bus on two different gimbal setups, and then they needed to do their on-set acquisition so that we could make this world fit together. This meant that we had to really try and prep and plan as best we could. What do we need them to shoot? What angles do we need them to capture at? What’s the best kind of array set-up that we can shoot for a maximum resolution?

The whole sequence is travelling from uptown to down to the bay. So, in terms of just data to process and handle it, it’s thousands of thousands of frames because we have an eight-camera ARRI array that’s shooting 4K and we need to try and process this, and stitch this, and you need to just use it to choreograph your way through the city. We were able to get that up and running after having done that with some of our work on Open World (the working title for Captain Marvel), and some of our work on Black Panther.

We knew what to ask for, at least, starting with, ‘Hey, can we get this? Hey, can we get that for the acquisition shoot?’ That was one of the best things that we could have been ready for, was just good prep.

b&a: With this sequence, it felt like there were multiple ways that the action was brought together, i.e. on the bluescreen stage with the bus pieces, the live-action photography, plus the CG builds. What did go into the CG bus? What did go into the digital environments?

Alex Cancado: Linking to the previous question that you had, I think that we’ve learned a lot about what we need to build in CG. I think that when we started on Black Panther, and then when we went to Captain Marvel, we are always learning and building new tools, building new setups.

One of the things that we discussed a lot with [associate VFX supervisor] Joe Farrell and Chris was the lighting, and how to look for things, and how to make sure they had great reference that they shot on the environments, and that they shot on stage. They even had some phone footage that we used, that they gave to us, that showed from the outside and the inside of the bus. It really helped us connect all the worlds and make sure that we had everything.

Building the environment is all about reference. Because we have been doing that for a while with Marvel, there was a lot of really great stuff that was gathered that we were able to use, things like great LIDAR, and amazing photo references. The way we’re shooting the images, and the way we’re projecting them, and the way we’re building our environment, was all part of a learning experience. It didn’t just happen on this show. It was something that took us five or six years from Black Panther to now to get to that point.

For the bus, we ended up building the whole vehicle in CG because there were moments that we were partially using the plate and partially using CG. Then there were moments that were full CG. We also ended up having to build the whole thing inside and outside of the bus. We built all the streets. We had props, we had trees, we had the whole environment. Part of it involved using some projections with photos that we acquired, and part of it was just full-on CG. There was a little bit of everything that we used there.

The cool thing is that it’s not one of those visual effects sequences that you’re trying to come up with the look of something. It’s what we call an invisible effects sequence, which for me is really fun to do because it’s the kind of sequence that some people could watch it and be like, ‘Oh yeah, I love that kind of stuff because there’s no visual effects’ and you’re like, ‘Ha! Sure!’

Andrew Zink: ‘Invisible effects’—I think that’s what a lot of filmmakers want to head towards nowadays, with VFX sometimes having such bad connotations of just being too over the top, too overdone, not looking real. And so, from the very get-go, the brief was that the audience needs to feel as though when they’re walking out of the theater, they’re asking, ‘How did they shoot this in San Francisco?’

With the bus, in particular, it started off with good acquisition. It started off with a base LIDAR scan and we just went to town with building it. And as the project started to branch out, we decided we needed to go even further into detail. We said, ‘Okay, we need to build out the interior.’ Even to the point where we matched all the ads for the bus that are running along the top.

We also wanted to build out the handrails, the bolts and the brackets that the handrails are attached to. The reason was, funnily enough, that when they were filming the choreographed fight sequence with all the stunts, they’d be taking stuff out of the bus so that they can get the camera and the crew in there. So, across the entire sequence, there’s shots that were captured that don’t have handrails, don’t have seats which means we had to put those in there as well. And you’ll never know it, no one will ever know it.

It’s like, ‘Ah, yeah, that’s a CG handrail!’ You know, it’s not sexy. It’s part of filmmaking. It’s just what you have to do for continuity’s sake. There was also damage continuity that we had to track since there were three buses that we were dealing with, technically. It was the two in Sydney that were a relatively identical match. And then there was the one that was in Los Angeles that was slightly different. If they did a re-shoot on the LA bus, it was slightly different, and the exterior shots were slightly different. So we always had to try and come at it from, well, we need to modify it so that it suits these surrounding shots so we can keep that continuity and then blend back into when we’re using our Sydney buses. It’s a very active organism, that bus always evolved.

b&a: What about when you had to animate it? Is there something tricky about animating something that’s a rigid body thing and that has the accordion connection?

Andrew Zink: I think the first thing that’s probably the most tricky for everyone is just not going beyond what the bus is capable of doing. What Chris, and Joe, and Damien were very adamant about was, it needs to feel real, it needs to feel visceral. We were going back to old 80s films and looking at when they were really doing live action stunts with smashing cars on the streets and, going back and looking at Speed, oddly enough, and seeing what does the bus look like when Keanu Reeves is in it? We were using all these older film references of when they were really practically stunt driving. The difference with ours is that ours was a double carriage bus connected with that accordion that in itself was particularly tricky to be able to animate.

A lot of times we were always trying to keep what was happening in the plates with the accordion, because it’s a very complex object. It’s not something that’s easily animatable because it’s flexing, it’s kind of contouring. It’s also flapping because they’ve got wind and dynamics that are happening to it. And so, we would say, ‘It looks awesome in the plate, let’s try and retain that.’ And in some instances we just had to blow it away just because the shot just didn’t allow for it, and we had to recreate it. We’re doing automation over the top of it, then we’re doing key animation over the top of that. Then we’re simulating on top of that. And then comp magic after the fact as well. A lot of different layers would go into it to try and make it feel as real as possible.

Alex worked on a particular shot where the bus is going over the top of the hill, and man, I don’t know how many iterations we went through.

Alex Cancado: It’s one of the shots where Shang-Chi is starting to come out of the bus. I’m going to quote our animation director, Raphael Pimentel. At some point he joked that animating a vehicle can be actually harder than a character. ‘You’ve got to keep the weight, you’ve got to keep the realism.’ It’s one of those things where everybody knows how it moves. So if you just change slightly how it realistically moves, everybody’s going to notice. Keeping all of those aspects with the suspension and the wheels and the way it moves, the weight, was hard. You’re trying to keep it very visceral and trying to find that fine line between super active, but at the same time, real.

b&a: I want to also talk about Razor Fist who has the blade for a hand. What was the methodology behind that VFX work in terms of replacing his arm?

Andrew Zink: It was a lot of work. Florian Munteanu, the actor, did a great job with his performance. He’s basically just trying to pantomime fighting Shang-Chi with this half-meter or so razor blade on the end of his arm. That’s pretty difficult, just working out how much does he need to actually accommodate for? It extends quite far beyond his fist. So, in some of the photos you’d see, he’s literally trying to keep his hand like a blade—T-1000 style—as if he would really be having that on there.

Simu Liu as Xu Shang-Chi / Shaun during shooting of the bus fight.

Then, for us, we had to do a lot of clean-up. Sometimes if you’re trying to make the best composition for the shot, Florian might have had his hand a certain way but we want to turn the perspective of the blade. Well, if we do that, now we’ve got to reconstruct all of this plate back here that we didn’t have to previously. So there were aspects of that across the entire sequence of recreating, rebuilding Florian from behind where he may have been blocking himself where our blade wasn’t covering, or rebuilding the set, or rebuilding a passenger, or rebuilding Shang-Chi.

For the design of the blade, we got some really, really early concept art from Marvel to start from. It had a couple flavors of what the energy of the blade might look like: red, blue, green, purple, and then we took that as a baseline and just started blocking it out in 3D.

We had to think about a lot of aspects. It’s a telescoping blade, how do we make that work? Where does it go when it goes back inside of the cuff? No one really has the answer to that. Everyone just accepted that’s how it worked and we didn’t really quite ask beyond that. How is it powered? We don’t know. It’s just how it ends up happening.

We also started off with a mishmash of FX elements that went into it. Sparks, sound waves, solar flare elements, heat wave, smoke, embers, ash, all sorts of different bits and bobs that went into it at the start. And then slowly we paired it back and kept pairing it back.

And I couldn’t tell you how many but we went into the hundreds of iterations on that blade in terms of submitting it to Chris and Joe and trying to get their take on it and Destin, the director, as well. What started off as what would’ve been fully coated in heat on both sides of the blade and flickering and all these bits and bobs coming out, ended up being just the sharpened side of the blade with the heat treatment on it, with the occasional element of something similar to welding sparks whenever you get a flare up—you might emit a few sparks or embers off of that. And then it’d have this unique smoke trail off of that. I think they got to a point where they’re like, ‘We just want it to feel more realistic rather than fantastical, even though it is a giant blade attached to a man’s arm.’

b&a: How did you tackle some of the destruction that he wreaks havoc on in the bus as well?

Andrew Zink: We had some reference of what does it look like when Razor Fist punches through metal? Do we want it to be kind of more lightsaber-ish? Does it leave a melted bead along the path of where it’s cutting? Is it glowing red hot? Does it slowly melt? We went through all these types of design phases and ideas to kind of conceptualize that. We ended up with a happy medium where he basically can slice through anything without any real struggle. And when he does slice through it, it was a combination of either we’re going through a simulation, we’re going through art directed modelling, where we might just model a predetermined path and then dress it up with the welding beads because we’ve already animated it and we know the path that the blade’s going to be following down. And then it’s just blend shapes and triggering it whenever it needs to be released, and then comp revealing it.

It was a pretty simple technique, but an effective technique because it happens within just a few frames. You can kind of cheat it and you’ve got a nice looking slice. Then we just dress it up with some effects, still glowing, some smoke and sparks that are coming off of it. It’s a pretty powerful effect at the end of the day.

The other thing was continuity. Razor Fist would be cutting through the concertina/accordion area, and from that point on in every shot after that would need to have the damaged concertina, all flapping and animated. Or, he’ll cut through the floor in a shot and from this point on, we need to make sure that the damage continuity is tracking throughout the rest of the shots beyond it. It was a fun exercise and just really critiquing and analyzing every plate.

b&a: I did see on some footage from San Francisco of a practical effects stunt involving the bus smashing into a whole bunch of cars. Can you talk about that?

Alex Cancado: Yes, they built the whole rig and the setup. They had a ramp where the bus would go over. A lot of that was actually used. Then some of the shots where they were closeups or they wanted to see the actors, we would mix in some CG or we would mix in some of the plates from Australia. But yes, there was some pretty intricate photography from the street where they destroyed maybe 10 or 15 cars, including a crazy expensive Audi.

The whole sequence was very interesting for us, in terms of going through everything with building the whole bus. We first had to figure out how we’re building it, how we’re putting it together. And then once animation started blocking things, we actually understood there was so much stuff moving. You were looking at so many things at the same time. You had the destruction side, you had the built area outside, you had the truck, you had the bus. Everything basically had to be reconstructed.

They shot some of the 2D elements of the characters, where we had to just put those together, match some of the previs and make sure that everything was working together with one camera motion. And they do things like shoot multiple plates of some of the same characters and we had to blend between different plates and then just coordinate everything so that Shang-Chi goes from kicking one guy and then pushing and coming toward us, and then he lands on the front of the bus. All of these things had to be put together and I think that was definitely one of those aspects of working in visual effects where you have to use every trick in the book that you’ve ever learned to put it together.

Source:https://beforesandafters.com/2021/09/23/how-did-they-shoot-this-in-san-francisco/