Interactive Volumetric VR for Quest? No problem! w/ Louise Liu and Jeremy Dalton, PwC UK
Hey friends, and welcome back to the Alex Makes VR podcast. Today's episode is so, so special. If you've been listening to the podcast for a while, you will know that I am a huge fan of virtual reality training. I think that training is one of the leading sectors that is really showcasing the promise of virtual reality technology and personally, I love working on those kind of projects. One of my favourite clients and I'm not just saying that because they're probably listening to this, but one of my favourite clients is PwC. UK, more specifically, the XR team ran by Jeremy Dalton and Louise Lui. They are a team that are constantly pushing the boundaries and coming up with innovative ideas for using VR with within training. Now when they approached me last year about a diversity and inclusion project that they wanted to make as an interactive VR experience that was captured using volumetric technology, and was going to be rolled out across the whole PwC UK firm on Oculus Quests...well, I could not turn that down. And it went on to be probably one of the most exciting projects I've ever worked on.
In this episode, we talk about everything, how the project came about, what the brief was, how we went about it, why we used volumetric, some of the challenges that came along with that some of the interesting anecdotes that we found, that kind of show, things that we found out along the way. And most importantly, the kind of impact that this piece is now having. I think this is going to be a really interesting insight for some of you looking to create virtual reality training and if you've got any questions, and I would love, love, love to hear your feedback on the on this episode and any things that come up that you think might be interesting exploring in future episodes, please reach out to me, it's @alexmakesvr across all of the social medias. So please enjoy my conversation with Jeremy and Louise, from PwC.
This is a full transcription of the podcast episode. Instead listen to podcast below:
Alex: Thank you so much for joining me for this first episode hopefully of what will be a series of kind of a behind the scenes of creating a really amazing piece of VR training that we worked on together last year. Thank you so much for joining me for this first episode hopefully of what will be a series of kind of a behind the scenes of creating a really amazing piece of VR training that we worked on together last year. But before I dive into that, before we kick off this conversation, Louise, this is your first time on the podcast, why don't you introduce yourself?
Louise: So my name is Louise Lu and I work as head of operations and delivery for PwC and I look after the actual team or the different peoples in there and help with project managing the projects for our clients and our lines of services.
Alex: And Jeremy, I know you're an old timer at this point on this podcast, but why don't you quickly introduce yourself for people that haven't heard any of your episodes before?
Jeremy: Always happy to have thanks a lot for having me on the show Alex again. It's, it's always a pleasure. So Hi, everyone. For those who don't know me, my name is Jeremy Dalton. And I take care of the XR team for PwC. UK talking to clients about all things virtual reality and augmented reality.
Alex: Amazing again, so succinct. It's so funny, because when you do like events with, you know, when it's like directors or writers or like creatives, they literally list off their history of credits. And I am guilty of that, too. So it's so refreshing to have like a literal two sentence bio. Guys, thank you so much for joining me. So I'm really, really excited to chat to you both about this project, because I've said this before on the podcast, but it's so rare that you get to go behind the scenes of some of these big commercial projects, because, you know, often that NDA, often they're being used internally. And the project we're going to talk about today is not only really really exciting because of the context of what the piece actually is, but it's exciting because this was probably one of the biggest projects that we have worked on collectively. And and the fact that we get to talk about it and talk about our approach to it and some of the interesting choices that we made and the challenges that we had to overcome, I think can be really valuable for people. So either of you, I guess take the floor and just give us a bit of an overview about in my shoes. What is it and how did it come about?
Jeremy: Louise, I'll leave this one with you. You are much more heavily involved in the project than I was.
Louise: Thank you You. So it came about with what's been happening with black lives matter and the firm, the diversity and inclusion team came to us and said - we want to create an experience to train all of our staff in PwC on understanding diversity inclusion, we want to understand also, with this experience, how people are feeling within the firm, we want to help our staff to understand how to conduct themselves with other team members or you know, to understand people's cultures. And so we really want to make sure that everyone, women, PwC fits in and are happy working in the in the office with each other and understand issues that they have as well with each other. So when we started doing this, we looked at the different things, we pull some team together, people together from different ethnicity, background, ethnic background, sorry, and looked at had a chat with them about how they felt what experiences they encountered. And from there, we were able to think about putting the script together and put a storyboard together basically, of a scenario of I'm bringing all these real life scenarios really into the experience and recording and putting this into this experience so that we can put other people have a different a different minority or ethnicity within the experience so that they can experience that same, those same issues that a lot of people in the film are experiencing every day of their lives.
Alex: I mean, the reason it's called in my shoes is because we're literally putting trainees, I guess, or employees through an experience seeing through the eyes of someone who's culturally very different to them, most likely and seeing the repercussions of that. And, Jeremy, why don't you talk a little bit about the technology that we used for this project, which is pretty dare I say it groundbreaking?
Jeremy: Absolutely. So from from a technology perspective, very early on, in the thinking behind this project, we as the the XR team virtual reality and augmented reality got brought in to discuss how VR could potentially be used on this project and be valuable for it. And it became pretty clear to all the stakeholders involved that as a lot of us know, in the virtual reality industry, VR is an incredibly impactful and effective technology to to drive empathy to drive a greater sense of, of understanding or feeling for different scenarios, different communities, different different lives ultimately. So where virtual reality very quickly got agreed as a as as the base kind of technology that we would use the internet to build the experience and to introduce it to others. However, another question then quickly followed behind that around what type of content should we or rather, what content type should we produce within this virtual reality application, because you have a number of choices, you can, you can choose to build a completely digital avatar that is not actually based on anyone any real world person, and you can animate that manually or with facial capture technology, you can bring import someone effectively into the virtual world. So you can use photogrammetry technology to to capture their likeness and and bring that in digitally. And you can of course, take different approaches bits with style, very stylistic, almost cartoony like characters. And and then finally, if you wanted to go really sort of photorealistic you can adopt a volumetric capture technology. Now, volumetric capture, is kind of like taking a three dimensional video. So if you imagine being in a studio with hundreds of cameras all around you, capturing two bits of information. One is the how far away or 1000s 10s of 1000s of points are on your body, to the cameras that are that are scanning you. And then secondly, a second set of regular video cameras that are that are capturing video from different angles. Now what the former does is it creates a 3d model of the individual to their scale into their likeness, and then you with the video texture, and you just slap that on to the 3d model. And the result is you get this really photorealistic human being. And that is the technology that we ended up going with in the end. And the main reason for this apart from it just looking photorealistic it really was important to capture the new ones. of emotion in such a scenario, and because it's being videotaped, effectively you are you are capturing absolutely every single point every sort of, you know, passive aggressive smirk every eyebrow raise, everything is being captured by this video camera and presented to you in a realistic way in the virtual environment. And that was and because it was so important in this particular scenario, we're talking about microaggressions, when we're talking about different aspects of communication and culture to capture those points. And that's why we went with volumetric video at the end of the day.
Alex: Amazing What an amazing kind of overview as well as the different technologies. And naturally, you know, we ended up using what was best for the piece, but also happen to inadvertently be the most technically challenging given that this was scripted as an interactive piece. Now, can you talk to me a little bit about the choice for it to be interactive, Louise? Because obviously, you and I worked very closely on on the storyboard and the script. And we're obviously advised by a specific committee that was put together to help on that side of things. But obviously, what what was the reason? Do you think that we went down the interactive route?
Louise: I think what we wanted to do is give people an option of decision points of where what they thought would be the right answer. We want to give people a chance to actually think about, well, how would I deal with this? Would I do this way? Or will I do that way, and it will take them to a different scenario or different setting, in that experience, so that they can see, whichever choice they made, a different outcome will come about because of it as well. And that really helps people go through a more deeper experience as well.
Alex: I don't know whether this actually exists. But I'd be interested to see if there was research around interactive versus just passive. VR off all three, six experiences, in fact, because you guys have done obviously, both in the past, do you find that interactive does outperform in terms of engagement from from trainees?
Jeremy: I can, I can mention the feedback that we've received on some projects before. So back in the very early days of the XR team at PwC, we we produce some passive 360, video bass productions. And they worked quite well, they were very impactful. But a lot of the feedback that we got was that the users wanted more forms of interactivity, they wanted to be able to, you know, make a decision in the environment. They wanted to, to be able to do something to affect what their experience was going to be like. And you can do that to some extent, in 360, passive, and in the sense that you can obviously look around and you can, you can have a different experience each time depending on what part of the scene you take in, obviously depends on how it's been directed and and script written. But there is a base level of interactivity, we should say. So perhaps more accurately, what they were asking for is a was a greater level of interactivity. So that's why we were driven to include more interactivity in the form of decision points in in further 360 Productions that were involved in. And if you think about it, in terms of how you can bring interactivity into 360, you may have, you know, more views and extensive views on this, Alex, but decision points and decision trees, this ability to select a point and choose your own adventure, so to speak, is a great way of making 360 video, an interactive content type within VR.
Alex: Definitely, definitely. And we're exploring voice interactivity a lot now. So even if it is a passive experience, that extra level of engagement, basically just getting someone actively involved in the story, because that is the power of VR, isn't it? That's why we use it because it does have that extra level of engaging ness. So yeah, no, I totally agree with you on that. But it's funny because in my mind, absolutely, there was no question that this had to be an interactive piece, but then when we went down the road of choosing to do this as a volumetric piece, specifically knowing that it would need to be distributed on something like a quest, a standalone headset. I'm not gonna lie guys, I was very very, very sceptical that we would be able to pull off such an amazing technical feat because natori asleep volumetric capture. I mean, again, coming back to Jeremy's point of hundreds of care cameras recording data at the same time, we're talking gigabytes upon gigabytes upon terabytes, probably just for one scene. So how did you guys feel about tackling that? Because I'm not aware of many projects that have pulled off an interactive volumetric VR piece for a quest.
Jeremy: I was personally sweating bullets, but I'm gonna talk about this question because it was really, it was really challenging. And I remember Louise had to bear the brunt of it. And a lot of the a lot of the the, the pre production and even production stages. So Louise, feel free to, to let it rip with all the challenges.
Louise: It was very challenging, I've never had anything this challenge in my life, actually. It, it was really good, it's actually quite exciting. That's the thing. And I was also quite nervous and cautious about how this how this will work. But the planning stage was so hard, because we really need to plan way in advance, you know, like you can get the right wardrobe for the characters was really important. And we do the testing on the day with them as well. And the hand is last minute buy a new pair of shoes for this actor because those shoes didn't work, and so on. So a lot of testing and planning. And even on the day working with the studio, the volumetric studio was quite challenging, because we had a lot of scenes to shoot. And we had a two metre radius space to work in. And that was it. And having to put certain amount of characters in that space is quite hard. And we had to limit the the movement and so on. So we had to adjust the script slightly as well in how far they walk or which way to the exit here, or do they do an exit? Or do we take a character route and so on. So that was quite interesting. And how we can do that, because I've never had to do it that way before. So that was a big learning curve as well. And then in post production is putting two separate videos of three characters together. So with the matter of record was basically two people in that space. So when we had the scene with three characters, we had to work out where those characters will be standing, while recording it and actually merging it together. So that was amazing, as you seen that come together with the help of the studio and our developers in house developers putting our scenes together. And really the amount of data I was bit worried that we didn't have enough space in our drives as well. So that was quite a challenge. But no, it just came so well together and having the expertise from both parties from different people, different teams helping us really, really made that journey much smoother and less stressful for me as well.
Jeremy: And it was super challenging for the the actors as well. I mean, imagine it you know, what you were saying, Louise's we we have this two metre space to work with. And there were no there were no props, really, we had like we had a green a green seats, you know, or stool more accurately, I should call it for actors to to sit on, you know, so that they could be in a seated position for some of the scenes. But apart from that, they wouldn't know tables, and actors had to be in front of laptops, which didn't exist, they had to have conversations with other actors that weren't in the scene with them. So like I remember lining up eye contact with an invisible person you're speaking to was, was a really an interesting challenge for them. But also for us because we couldn't be in the same room with them while this volumetric capture was happening. And Alex, you can speak more about it. You know, you were there looking at the cameras, but it's really difficult to get their eyeline right knowing that you're going to have to, as Louise was saying piece together, these two people speaking to a third person, and the two and the one were recorded separately and didn't get to see each other in the same scene. It was really odd.
Alex: I mean, it's so funny, I have to admit this project is probably my favourite project bar none I've ever worked on because like we said, It was exciting, like having all of these things like yeah, how are we going to do that? How are we going to make sure that these characters look like they're actually looking at each other? How are we going to get them to look as if they're actually their body language? Is that that it's it would suggest they're in the same space? How are we going to get them to have that this full blown conversation with someone that's not actually there? And for me, it's director like, how am I going to ensure because my job on set is obviously to ensure that we've got the shot. No one's going home until we know that we've got the shot and we don't need to come back and do reshoots, because that gets very, very expensive. So for me not having a playback solution, not having a way to Uh, kind of very easily be able to tell whether or not we got the shot was really interesting. And it took me back to my first days in 360, where I was using the the GoPro rigs, which was just like six GoPros stuck together. And there was no way to be able to really quickly and easily show what that looked like. Now obviously, we've got these live previews of 360s. From 360 cameras, you can see a basic stitch, even if you wanted, even if you couldn't have the basic preview, if you wanted to see it in a headset, you can quickly stitch something and get it on a headset within half an hour or so. But this that wasn't an option. So I remember towards the end of the day, we were remember we were getting, we were basically taking almost like screen recordings of the camera of one camera angle and bear in mind, there's hundreds of camera angles, and then we would bring it into premiere. And then we would basically look just crop it to like line up just to make sure that it did look like the body language and the eye contact was right. And that the more importantly that the audio was matching it because again, there was always going to be in scenes with more than two people, there was always going to be one actor that had to perform to the other actors recorded take. So they were listening to like playback of that scene. And they were having to perfectly time what they were saying, which was something that we rehearsed in, in the online rehearsals, but because of COVID we weren't allowed to kind of rehearsing past. And so that made it so fascinating. And I'm sure like, I feel like the actors themselves found that quite exhilarating to have to be able to do that because it comes much more of a technical practice, not just a performance, I guess. But coming back to coming back to the kind of the post production side of things. And talk to me about getting it getting the environments, right, because obviously this is volumetric capture of the actors, but then the scenes themselves were were computer generated. So talk to me about that process. Louise, how did the pose How did the development team deal with creating environments that then would match and ultimately have to work in tandem with the actors to then run on a quest?
Louise: Yeah, so with that part, we actually had to get an artist come in touch, he created the scenes, whoops, so sorry, of my mother, sorry, we had to get an artist such come in to create these scenes so that our developers have an idea of what the space look like how much space they have an understanding that, I suppose what helped is the actor didn't have to move around too much. So that really helped him measuring how big the room could be, for example, in the office, and so on. There were few scenes where the actors walked in. And so they need to think about where do we put that wall? Do we move it from that side to here, so shows like a corridor, and so on. So then the when the video starts, they come walking in, it looks like they come through a corridor or for that particular doorway, and so on. So it's a bit challenging for our developers, and the early stages of thinking word for the desk. If they're standing here, and we're putting these videos together, which were they coming in, and so on, and how do we hide certain parts where maybe the feet of someone, and so on. So it's challenging in that way, but we're really lucky to have to have some really good 3d artists, some really good unity developers to help us build the space a bit more to get the lighting, right. And that was quite a challenge as well, from daytime to evening time, because this whole experience was from morning through tonight, it does day in the shoes of that person. So it was quite challenging to make sure that these scenes related to certain parts of the day and the light moves throughout the day as well. So they had other factors to work with as well, as well as just the videos itself.
Jeremy: And to add to that complexity, every minute of volumetric video is costing the project 1000s and 1000s of dollars. So we didn't want to have any, any extra video than we actually needed. So once the the actor had left the scene, we wanted to cut it immediately. Because even if you're only talking about three or four seconds here, if you're talking about three or four seconds, and you've got you know 2030 different cuts and scenes and and different versions, then then we're talking about costs that rack into the the 1000s or even, you know, the $10,000 plus mark for just those extra little bits of footage, you know, for safety at the end. But of course you need that safety because as Louise is saying, you know, once you've got an actor leaving the scene, if they're going right if they're going out the door and straight to the right down a corridor, then you can you can cut the volumetric video capture earlier than if they were walking down the corridor. And you had a view of them constantly as they went down the corridor. So we had to take that into account for extra complexity and constraints on the project as well.
Alex: It's so funny because you've literally just triggered a memory of as going through the script. Louise, I believe it was mean you it was a late night, we were going classic late night. And we were we were going through the list of, of characters and working out how long roughly they were on screen based on based on the script. And we figured out that there was I think there was two or three characters that were in there as almost like extras like non speaking actors that were there just to make it feel a bit more realistic, you know, a character in the lift a character, you know, who's in the death of their desk, just like typing away. And we worked out that that if we had kept those characters in, it would have literally cost 10s of 1000s of dollars, and our members both having that realisation of like, okay, yeah, maybe we don't need those characters.
Louise: Absolutely, that was it was surprising, shocking that time because it's just eyes all time. And you know, even when we looked at it, it was just not feasible, chatty half the man, ideally, it'd be quite nice to have more people more characters in experience, because it looks more realistic, like in the office space. But unfortunately, with the cost factor involved in this, she took that out the window, basically.
Alex: And this is so important, then this is something that I love talking about on this podcast, to all of you listening, I know that some of you can get carried away with all of the kind of crazy, you know, detail in your pieces. And I know, if you are very creatively driven, that you'll want to push those boundaries, you'll want to like add all of this extra complexity, I want to try this camera shot, I want to try this one, try that. Sometimes, especially when you're working on a project that's being funded by a client, you really, really, really need to look at your script. Consider the technology and the capture format that you're using and say, does this add anything? Does this add so much value to this project that is worth the extra budget? Because oftentimes, it's not because when we're talking about this piece, for example, the The point is that you are in the shoes of someone and you are going through their experience, and you are making choices, and you're interacting, is someone in the background typing gonna make or break your experience, or you all of a sudden going to not believe this piece, because there wasn't that extra three people in the background? Absolutely not. But could that have been a massive hurdle to the production because we've you know, spent, however much percentage of the budget on these characters. Yeah, that could have really put the project in jeopardy. So really, when you're thinking about creating these pieces, if you are especially looking at the training market, just make sure that especially in these early days of VR, yes, push the boundaries, yes, be innovative, and be brave with your decisions. But also make sure that you're not unnecessarily adding things that will cause kind of complexity. And that goes the same with Louise mentioned about the wardrobe and even like hairstyles and even like body gestures, I remember there's one scene in particular in this piece where the man couldn't, he couldn't use his arms basically, because the cameras couldn't pick up the detail when his arm separated from his his suit jacket. And that's again that's like these, these technical decisions that if you push down was like no, definitely I'm not doing the scene unless they are, you know, they can do X, Y and Zed or if I have x, y and Zed then you might not end up making the project because it can get very, very expensive. So that's my kind of my two cents on that. But I guess we're nearing the kind of the end of like this first initial chat about this conversation and we've touched on so many different topics that I think people will be like head spinning, don't even know where to start. They want to hear all of the details. But if you guys could pick one element of this project that you're really proud of that you think really has pushed the boundaries that is innovative. What I guess what is that piece, what makes you kind of proud I guess of this project?
Jeremy: So technically for me, the the ability to to use volumetric capture technology, which is a very intensive type of content and optimising it so that it could run on effectively. What is a mobile phone processor in a very realistic looking virtual environment while while maintaining a smooth frame rate and not making anyone sick.
Alex: When you put it like that, yeah, that's definitely a proud moment. Amazing. And Louise?
Louise: I'm looking more from a people angle, I feel. And I'm really proud that This experience has really been able to show the a user, what it is like for a person of a different ethnic minority, what they have to go through in life, every day of the week, really, you know, there's a lot of things that I can relate to, that I've been through growing up and even still encounter now. So for me, it's I'm really proud that I'm able to show this I create this experience so that other people can actually, you know, go through it and actually understand and really feel what it's like so that, you know, they really make them think, a bigger picture of like, what can they do afterwards? You know, what's the next step to help others as well?
Alex: Yeah, really, really powerful. And and can you share? I know, obviously, the piece hasn't hasn't had its major rollout. Yeah. But can you share any feedback that you have gotten since the the project is being complete?
Louise: We've tested it internally with a couple of people as well. And they were very moved. They were we had a few people come out crying. And because there was just so so powerful, the experience that they're going through, and everyone needed some time, there's literally complete silence afterwards, when they take the headset off. And you're just worried that someone's going to say something bad, but it's wasn't as because they need the time to reflect before they can talk. And that to me was like, wow, this is really powerful, this is going to really have a lot of impact, once it goes out to the wider audience, you know?
Alex: I we have such like a great dynamic when we work together as a group. And we never really get to do this never get to hang out outside of the context of usually being very stressed trying to pull some impossible task off. It's so much fun.
Good times. Well, thank you again, both so much for joining me.
So there you have it. That was my conversation with Louise and Jeremy about the in my shoes training project for PwC. uk. Honestly, there is so much that went into that project that we could do a whole series on that. And you know what, I'm quite tempted. So if there's elements of this project that you would like to hear more about, please, please, please reach out to me. You can email me Alex mains email@example.com or reach out on social media. Let me know what your thoughts are. Let me know if you'd be interested in me doing a breakdown the step by step process similar to what I'm doing in my how to make a virtual reality film series on Friday. Because Jeremy and Louise, they love talking about this as much as I do. And this is the first project that we can really go in detail and talk about very openly. So let me know. Thank you again so much for listening. And until next time, have a great day wherever you are in the world.
Listen to the Alex Makes VR podcast here
Subscribe to the newsletter here
Follow Alex on Instagram here