• alexandraruhl25

My jump from 360 video to volumetric video

Today's episode is gonna be super off the cuff just you and me so grab a cup of tea. Let's have a chat. I have no idea what I'm going to talk about in this episode but the truth of the matter is, when this episode goes live at 6am on Monday14th of September 2020, I will be in the thick of probably one of the most technically comprehensive and innovative boundary pushing client projects I have ever worked on. I'm directing a volumetric VR project, a branch narrative, volumetric project.

Read on for the full transcription or listen to the podcast episode here

For those of you who don't know what that means, volumetric video is basically the absolute opposite to 360 video. If you think of 360 video as you place the camera in the centre and because it's got lenses all over, the camera body captures the whole scene. So it's recording inside out, isn't it? You put the camera in the middle records everything around it. Volumetric video, is the absolute opposite. Your performers stand in the centre of a totally green screen studio, a massive stage and around that stage are 106 cameras, I think it's infrared and RGB cameras that are capturing the action simultaneously.

106 cameras all capturing the action and the reason why you would film this way and the reason why we're doing it is because what that does is records your performance from every single angle. From above, from below, from the side, upwards, from side downwards, literally everywhere the whole studio is a fully spherical studio. And when I go for a take, when I shout action, that stage will be completely closed off. So it's a fully round green screen studio with cameras all around. It’s quite impressive.

It's been a real learning experience for me actually, it’s been a real like honour, genuinely to like be writing and directing a project like this because it really is pushing the boundaries. It's over a 20 minute long branch narrative piece, which for something like volumetric video, because it's such a new and intense and technically comprehensive medium to capture in, oftentimes you will have maybe like two to five minutes.

To give you an idea, the studio that we're filming at, they filmed certain scenes for Bohemian Rhapsody and then what that allowed them to do is because the scene - I don't know which thing it was - but because those scenes were captured from every single angle, then the director of that film can go in and choose exactly the camera movement that they want. One of the most prolific movies I guess to ever use volumetric video that I know of anyway is The Matrix. For those of you who know that film, obviously there's these iconic fight scenes where Neo is bending backwards and it seems impossible and the the camera seems to be like spinning around him, but he's frozen and you're like wow how is that happening? So that, from my knowledge anyway, that's one of the earliest most famous cases of volumetric video because what that was…I don't know how many cameras they use for that shot in particular, but that was basically just a ring of cameras around him, and that was capturing it from every angle at the same time. So in post production, they could go in there and they could like, you know, jump from camera to camera to give the illusion of going round really fast. But technically, that is literally the same frame just from 100 different angles. And so that's how it's kind of used in traditional. And still to this day, that's often how they use volumetric video for those kind of cases. But in our case, the reason we're doing this is because when you bring volumetric video into a VR headset, what that allows you to do is, it allows you to fully interact with the environment.

So whereas 360 is obviously very static, and it's very, someone puts on a headset, and they can look around them but they can't move back and forward, they can't get closer to the characters, they can't move away from the characters, they have no control over their own kind of sense of space. Whereas with volumetric video, because you've captured an actor from all angles, you now have a fully 3D model of that person. So now, you could put those actors in the centre of a room, and you could have people walking all the way around the scene. And those actors wouldn't move, they would still be performing in exactly the same place but you all of a sudden have agency.

When we talk about this idea of in VR, the director almost becomes irrelevant, like in terms of a traditional film director, because traditional film directors are choosing those frames, those sequence of frames that pull together to make an experience, really, you know, cinematic and make you feel a certain way, with VR the audience becomes the director in a sense. All you can do as a director is, you know, make sure the actions are as good as it can be, choose the locations, like you obviously want to give some kind of structure, and you would hope that they would follow a certain path. But you have to account for the fact that they might not and so much of VR is about giving the audience the freedom to explore.

And with volumetric video, that's taken to a whole new level where you can actually fully interact. if I want to be close to that action, if I want to be intimate with those characters, I can get up close, if I want to pull back and walk away, because actually, you know, it's a really intimate scene, it actually feels a bit strange to be too close to them, I can pull right back. It also accounts for things like… if you're a six foot two man experiencing something in a headset, and that the same experience is being experienced simultaneously by a five foot two woman and a headset, that when you shoot in 360, obviously, the height of the camera is fixed point. So the perspective of the camera is fixed point. Whereas all of a sudden, when you're working in a what we call a six degrees of freedom environment, aka, you know, you don't only get to look around, but you can also move up and down and forward and back and you've got all of the control over a space.

Now, all of a sudden, even if you're watching the same scene, and you're both sat down, that's going to feel much more natural for you if you're a six foot two man and a five foot two woman because, the camera positioning will adjust to your height. So that's quite an interesting thing to think about. It's been a really interesting process, because this is the first volumetric project that I've ever directed and it's been a really interesting process kind of thinking about all these different things.

I did a previous episode about how creative decisions can blow your VR budgets and this is absolutely that project. From just writing in one simple detail was going to cost over 30 grand and that's because I've never worked in this medium before, I had no idea what some of the implications would be because I didn't know the technical setup. And because it's such a comprehensive setup, it's really interesting for me as a director, but also a total tech nerd, to go in there and be like…wow holy shit this is fascinating. So, you know, when I've written in this action for this actor, even yesterday, again, I shouldn't be sharing too many details but some things even now that I've written into the script, they're just not technically possible. Because when we got on the stage to start blocking everything out, it's just not possible anymore. Because of the nature of the filming and that's so interesting to me. And I love this state of learning, I feel like one of my takeaways from this experience has been like, you know, I preach the whole don't be scared to fail and always be learning and don't judge yourself and, in fact, don't judge yourself. I don't know whether I've done a whole episode on that yet, but if not, it might be about the next episode. You know, having that open mind of wanting to be challenged, wanting to learn new things is so important, especially when you're working in a medium like ours, where things are changing so rapidly, it's so important for you to have that beginner's mindset at all times.

Doing these podcasts and things, I’ve gone back to being a beginner and having to learn the ropes but with VR and because I've done the same kind of thing for a while now, and haven't been challenged in the same extent that I have been on this project the whole process has been really interesting.

Listen to the Alex Makes VR podcast here

Subscribe to the newsletter here

Follow Alex on Instagram here

  • LinkedIn
  • Twitter
  • Instagram