Season 2, Episode 9: Exploring Virtual Production with Brenda Medina

Stacey Copeland: Welcome to FCAT after school, a podcast project from SFU's Faculty of Communication, Art, and Technology. In each episode, we join student hosts in conversation with alumni as they explore career journeys since graduation and gather advice for the next generation. In this episode Center for Digital Media Master's student, Marshall McCann catches up with Departure Lounge virtual producer and CDM alum, Brenda Medina, Volumetric capture, LED stages, and real-time rendering engines. At Departure Lounge, these tools are transforming the way Film, TV, commercials, and music videos are produced and experienced. Departure Lounge is a local Vancouver company nestled beside SFU Center for Digital Media. And there, Brenda is at the cutting edge of Metaverse and web three innovation. Recorded in the offices of departure lounge, we join Marshall and Brenda in exploring why these technologies are so pivotal and how they're being used right now.

Brenda Medina: I don't know how long whoever's listening has been in any kind of creative industry. But we're always craving for something different, or I want to be completely out of the box. This is a moment where everybody gets out of the box. Virtual production has still many branches. The main one is definitely it's pushing boundaries in filmmaking methods. So it's all about Computer Aided productions, with different tech tools. And I would say they're still emerging tech tools, most of them for creating new ways of doing filmmaking and filmmaking, I think it's the main bridge for any other kind of audio visual media formats. So just to give you an example, the main service, or asset we have here at the petrol launch is called it's a volumetric stage for volumetric capture, which means that, we have, it's having like 106 cameras, volumetric cameras, only for capturing volume, completely synchronized to then capture shapes and shapes, to then create a 3d, fully 3d hologram of a person. So it's like the new green screen, basically.

Marshall McCann: So just just so I can understand, you have these cameras that basically catch a 360 degree view of an object or a person in in a volume, which is the space in which those cameras are pointing

Brenda Medina: Exactly.

Marshall McCann: And they capture performance or the image of this thing, and they create a 3d object in.

Brenda Medina: Exactly, so you capture a real person, they can be singing, they can be dancing, they can be talking, and this will be fully captured 360 from every angle possible with this 106 cameras. And that will become like a hologram of this person doing exactly what they're they were doing on stage. And that we call it that an asset, that asset which can be in different formats, normally using 3d software's like obj that will be imported into a game engine software like Unreal or Unity. And then you can place this person hologram in absolutely anything you want.

Marshall McCann: Cool. So there was a project at the Center for Digital Media that kind of bridge you into that. Can you talk a bit about what that project was?

Brenda Medina: Yeah, definitely. So I think it was a sum of the three projects I did. The first one was all about product design for a digital product. It's the whole process from concept making all the way to development, create a prototype, to lead a team on how to do it, step by step. Next project was about building visual effects for a, it was like a digital theater work. And the third one was working with the departure lounge, actually as an industry project. And we worked on creating a concept of how, again related to theater, which Shakespeare how to bring that experience into new platforms, like the metaverse into immersive experiences and how people would still have that feeling of theater in a very new format for a lot of people. So that's how we started to work and then they were like, how do we keep you?

Marshall McCann: Okay, and this this might be a dumb question, but why? Why do you use volumetric capture volcat over motion capture mocap?

Brenda Medina: So the main difference between we call it volcat for volumetric volcap and mocap is that mocap, what does is it's registering the movement of, you know, the person who's doing it and that will become just a skeleton that then will be rigged into a 3d human or a fictional character, to then give that that exact movement you were trying for that. So that's what mocap does right now. And volumetric capture captures not just your movement, not just the skeleton move, it captures everything. So it captures the textures, it captures the shapes. So if you're wearing, you know the shirt Marshall is wearing right now it will capture every fold, the color, even how the fabric reflects, a will capture the details of his hair, his face to sir. So that's the thing, that the main difference between mocap and volcap.

Marshall McCann: That's cool, but nobody wants to see me as a 3d object.

Brenda Medina: You never know.

Marshall McCann: Another thing that you mentioned in kind of the tools was was the LED stage, I want you to kind of expand on what that is, and...

Brenda Medina: For sure, so when we were mentioning that virtual production is about using different tech tools in order to do different filmmaking methods. One was the volumetric capture. The other is motion capture. And another is the LED screen, which here we call a narwhal. But basically is a massive, LED screen 4k resolution that you can use to build backgrounds for cinematic purposes. And it blends the best way possible with what you have in front of which could be the actor. So this massive screen can bring in any digital environment background, you're imagining you create it. And then sometimes you just feel like a tiny part of the of the set in front of it. Maybe let's say that, you know, the actor is going through a door, and then into a spaceship. That's the way that the list screen works. So a good example is Mandalorian Star Wars. Maybe you've seen it. If not, I don't know what you're expecting, anyhow, like gosh, you should go and watch it right now. But there's this scene, right, you're in Tatooine, it's this planet that doesn't exist, it has his major spaceships, so you can imagine how much it would cost to create the entire spaceship, just for this. So they don't create it, it's just virtually is just an image in a massive LED screen, they just created like the part of the door where then the actor goes in into the spaceship for the viewer. It all blends perfectly as to just one single image.

Marshall McCann: And so I guess what I'm taking away is that is that this is transitional and massive for the industry is because these tools are basically cutting out like major costs in terms of how you interact with like digital objects in in real space for the LED stage, it's like, let's say you wanted to add something, you know, to an area and change the lighting, you just do that on the LED stage, you don't have to wait for that to happen in a real set or change out all the lightings because it's lit by the LED stage.

Brenda Medina: So, the main thing, this is kind of breaking barriers in the whole filmmaking industry is one is way more cost efficiency, by all the things you just said, just by set building by the amount of talent and people you need in a normal set the other ways using tech, like the lead screen. For example, if I have, you know, a hundred fictional soldiers, I can have them in the screen. And I just need real free soldiers in front of it just to give that sense of depth, right in the in the camera image. Also, in terms of you know, they say it's way more sustainable. Because of the same you know, every set they build, you have to then destroy it and get rid of it. Now every environment you build digitally, you can keep it like library, you can reuse it for so many different projects. You can adjust whatever you want from you know, like the things around it, the lighting, just lighting cameras and hardware is equipment is so expensive. Virtual production is all about building it on engine. So the VFX supervisor from Lion King, I don't know if you saw it, apparently...

Marshall McCann: Never heard of her.

Brenda Medina: Of course. But this guy was saying that virtual production is creating a game called filmmaking because you're building the world of film inside of it. So you're not just creating your film on the you know, an engine, you're also creating your your film set, you have your cameras, you have your lights, you have your set design, and like 80% of your production is being created virtually. So that is how it's completely changing the game in film industry.

Marshall McCann: That's really cool. I'm as well extremely interested in how your role kind of differs from film production or like traditional film production to virtual production. You know, I know that you have a bit of that background, you know, producing in those different areas. What have you seen are some of the major differences maybe in the skill set or in daily operations for your role as a virtual producer?

Brenda Medina: Just so different to think how you would break down whatever you want to do for film in a much more efficient way where I'm telling you you're going to do on tape and with that you're going to create all the scenes you we're thinking, so we're all kind of like a consultant in that way. But also because just filming is just the, like not even half of the process. The rest of it is a pipeline of working on game engines. Normal producer filmmaking, I've probably never even, you know, touched a computer if it's needed. Beyond editing, just parts of video, here is a whole other process to get from just filming all the way to the final thing. So you need different techniques you do you need different knowledge in order to get from, you know, the start of the project all the way to the end, which could be just a 2d music video. But it also could be a full immersive XR experience.

Marshall McCann: So I yeah, I guess there's a lot of overhaul when it comes to trying to get people from other, I guess, more traditional types of media, and these people have tons of experience in those fields. But, you know, I really liked what you said about, you're getting all these different pieces of coverage or footage in one shot. And so how do you kind of take storytelling with that in consideration?

Brenda Medina: That's the fun part, right? I don't know how long whoever's listening has been in any kind of creative industry. But we're always craving for something different, or I want to be completely out of the box. This is a moment where everybody gets out of the box, when I completely change their process, the way they do things the way they think creatively. When I tell them, you know, with one take, you can do as many shots as you want then, you can think okay, so how much can I do with the final thing? How much environments or settings or places can characters go? Or how do they go there, so their creative freedom just expands. And that's where I think all this stuff is really cool for storytelling, Because I just gave you so much new tools to do whatever you're looking for.

Marshall McCann: Yeah, there's a lot to explore. I, you know, out of interest, I think what you're saying about the the stage sort of having not one place to look but sort of it's it's a stage right, it feels more like a stage for theatre. I'm interested in, in whether you know, dancers or theater, traditional theater actors actually thrive in that space, because of how it's set up. I don't know if you guys have worked with them too much.

Brenda Medina: For sure. Actually, we really embrace lino movement in this terms of capturing, whether it be motion capture, or volumetric capture, that's where we want to get because at the end, you know, no matter how perfect 3d rigger you are, you won't get it as the same way a dancer who's been doing it for 20 years does it right. So that's one thing. And also, you know, what musicians, so we actually did a test here, of different musicians with their very different instruments here on our stage, because if you know, or if you don't know, still, in this kind of capturing technology, the biggest problem is capturing hands, fingers, fingers, is really hard to get us, you know, high resolution as possible, especially when they're moving. So that's why we've been playing with specific like dancing, and also a lot of playing instruments to see how we can improve as much as possible, that hand capturing, because if you guys, we got a guy playing like bass, I want to see his, you know, his hand slapping the bass, or we had a guy playing a flute, and then all instruments you can imagine, I want to see that detail of his art. So that's one thing, we're still testing, figuring out how to do it the best way possible. And for dancers it's the same. Some of the, you know, contemporary dance sold about the hands or you know, theater, when we were doing some motion capture. You need to be way more expressive with your body, his body language to understand better if I'm doing like an immersive experience, right? So we were playing with that of how can we embrace as much as possible body movement, or when you're in an immersive or game experience? You feel whatever you need to feel of that performance?

Marshall McCann: That's cool. You mentioned a little bit about the game engine and how that sort of being incorporated and all these pipelines, I'm interested in a. the way that it's incorporated in pipelines, what's its what's its function using a real time rendering engine and b. kind of what are some of the difficulties involved with having that?

Brenda Medina: Yeah, so game engines have become core of this new process of when I'm saying virtual production, I can assure you like, 90% of time, you're gonna use game engine in some way. Right? So you definitely need to know how to use them. What are their abilities, their capabilities or constraints as well, because you have first the technical part, right? At the end of the day, this is a software and there's constant bug fixing and troubleshooting and understanding how anything you integrate into it like is, you know, whatever you capture in mocap or volcap will react to whatever you're implementing on it. So as you know, you do some visual scripting ain't like I want this, Marshall, volumetric asset to go to this place and then transition to this place. There's a lot of technical stuff to think about. Because let's call it layers, right? Maybe Photoshop or whatever you want, as much layers you have, the more you need to learn how to blend them. So for your eyes, it just looks like one, one dynamic one single smooth transition, right. So that's one of the main things we've been struggling working with. For example, I can tell you, I can tell you this detail because it was just released. So I can totally talk about it.  We just released a music video, it was full virtual production. So the artists, the performers were captured volumetrically. And the entire video was created on game engine, which wasn't real. So if you can imagine, maybe the director wants to go from you know, this steak to transition into this take, I have to do it the most smooth where you would with a real camera and real person. But sometimes there's just technical difficulties on it. The assets can spawn inside the engine or in the environment, or you know, sometimes it's too heavy, it's just data we're talking. So just rendering this, I have to consider so much time in my project just for rendering, which is not something you really need when it's just video like After Effects or Premiere. This is like rendering on game engine that could take days, hours, depending on how heavy is so tell your client, you know, like, hey, I need just a week just for rendering. So there's a lot of technicality you need to consider when playing with game engine, doesn't matter what you're doing. But also there's a part of, and I think that's the most interesting thing is in the game industry, they're actually demanding more like cinematic storytelling. So if you go to companies that builds games like EA, they're actually looking for more people that comes from the film background. But at the same time, the film industry is looking for more like state of the art technology that is made by games engines. So they're really blending in into one same thing, even though it's different outcomes, but they're becoming dependent of each other.

Marshall McCann: For someone, and probably for some listeners, who are looking to enter the industry, and maybe specifically in a production role, what are some of the skill sets or tools that you think that they should learn before kind of entering whether it's in your role or another role?

Brenda Medina: Yeah.

Marshall McCann: And kinda working in virtual production?

Brenda Medina: I think that's a great question. Because we've been saying this is all very new, this is emergent tech, this is breaking barriers. So there's not like one still, you know, like resume you need to have in order to get into this, it's more about the mix of capabilities of interest of knowledge. So you can dive into this kind of thing. So one, obviously, technical experience, you need to have some game or filmmaking background, you definitely need some onset experience. When I'm talking about being a producer, you definitely need some onset, because there's still so many traditional processes implemented, you know, like creating a shot list, managing a stage, a whole crew, etc. You also need a lot of time and task management, like anything, but because here you have two different teams and pipelines, one is just being on a stage and filming, capturing whatever you want to call it. And then you have the other pipeline with the creative, the designer, the developers for the second part. So you need to be as fluent as you can between two different kinds of worlds, in a way. And lastly, yeah, of coordinating pipelines and teams and clients and all that kind of management.

Marshall McCann: You touched on something really briefly there that I thought was really interesting, there's sort of two planes of creation happening. There's the one that that's happening on set where it's, it's similar to traditional film, in that, you know, you want to capture a certain performance from your actors and, and you're sort of trying to build that space for them on a, on a, on a stage, right. And then you have the second field of creation, which is happening with your designers and your your programmers and all these people on the back end. How have you found it is I guess, producing both sides of that?

Brenda Medina: It can be very exhilarating, but at the same time very stressful. Because the structure is similar, like you said to traditional one, I think director and producer have to be fluent in the entire process with every team just to get to the final vision. But the difference is now that this second team specifically, it involves very different people you normally had in film. So just talking about with a developer of coding, it wasn't not a normal thing, right? So being able to at least understand the person their needs, what they need to code, what you're trying to achieve, how his dependencies build with a designer, maybe the 3d the environmental design, then the visual effects guy and you know what the most interesting thing as a producer in this kind of things, I really need to understand as much as possible the entire workflow, because there's a lot of dependencies for the pre production. So you know how they say in filmmaking that will fix it in post, that's the most classic phrase you listen to. Now, with virtual production, it's let's fix it in prep. As much as I can get in pre production, I'll get the better outcome because not everything can be fixed in post, especially if I'm talking about volumetric capture, you know, it's just, it's so much data, it's so much another organization, prep, just, whatever, whoever you're capturing, and I can change that after, I really need to know everything, my designers, my developers, my post, my digital compositors will need. So I capture it, when I'm on the filming part of the project.

Marshall McCann: You only have so many chances of getting like the right capture in the volume with with the actor with the talent.

Brenda Medina: I think that's the best best thing to explain of being a virtual producer is the mindset used to be I'll find a way to fix it later. Yeah, you now have to find a way to fix it before.

Marshall McCann: One last question. And you don't have to be a prophet about it. But you know, this is this is a young side of the industry, right? And there's there's a lot of things that are being explored for the first time you're talking about pipelines, and you're talking about sort of the way two planes of creation are happening at different times. What do you think, is another application of virtual production that you'll see in the future, what's kind of the next piece of the industry, it's going to extend out and grab?

Brenda Medina: Obviously, anything related to creating audio visual content. So right now, I, you know, the ones with the bigger budget is film making, is video games. But it's starting to spread out a little bit more into another part like advertising, I actually come from that background as well. And, you know, that's one of the places where they have so much creative vision two, you know, you're creating now like 30 second kind of commercials or videos. But there's not enough obviously, there's not enough we're using all the tools in filmmaking. So this opens the creative field so much for advertising and advertising now is fully dependent on technology and how even technology becomes a new use for the users not just like watching something is how this brand, whatever the brand is, impacts my life, just to say go right. So it's gonna really gonna hit advertising is starting to hit others like music video. So we just created a music video, fully virtual production, and for them is just a new way to reach out to their audiences, right? Because we're actually going to build we're building the VR immersive version now of this music video, where people can actually immerse themselves into this music video. So the artist is very excited, you can get even more closer to whatever their vision was, you know, for this album for the song for this collaboration. So that's hitting that part as well. And obviously, anything audio visual, it will start to find a way. Anyone who's interested in emerging tech and what can it be done. I would definitely say go for it! Because like you said, it's very new. Possibilities are endless. But definitely, it takes a lot of because of it patience, and being able to flex and adapt as much as possible because there's not a manual of how to do things. You're figuring out how to do the things. So that's the fun part of being involved in this. So yeah, that's my main advice. Just be excited for the unknown.

Marshall McCann: Thank you so much for having me in the offices today.

Brenda Medina: You're welcome. Anytime!

Marshall McCann: Very official. We even had your boss pretty good.

Brenda Medina: Yeah. Anytime you're welcome, Marshall.

Stacey Copeland: Interested in learning more about the FCAT community? Stay tuned for a brand new episode of FCAT after school, hitting your feeds every other Wednesday this season. A big thanks to Brenda Medina for joining us here on the show. You'll find links to resources mentioned and more info on Brenda and SFU's Center for Digital Media master's program in the show notes. Our host for this episode was Marshall McCann. Production by Marshall and me, Stacy Copeland. FCAT After School respectfully acknowledges the Musqueam, Squamish, Tsleil-Waututh, Katzie, Kwikwetlem, Qayqayt (kakite), Kwantlen, Semiahmoo and Tsawwassen peoples, on whose unceded traditional territories our three campuses reside, and where many of the stories shared in ourseries take place. Make sure to rate us and subscribe to FCAT After School in your podcast appof choice, so you don't miss any of our upcoming episodes. And you can follow us on socialmedia at FCAT at SFU. That's F C A T @ SFU on Twitter and Instagram. See you next time.