Episode 14 - Building Multiplayer Virtual Reality Applications in Unreal with Eric Le Saux
Navyata Bawa, a Developer Advocate at Meta Open Source, chats with a VR Developer Environment Software Engineer Eric Le Saux about the SharedSpaces sample and how it demonstrates the Platform SDK Multiplayer features.
Summary​
In this episode of The Diff podcast, host Navyata Bawa chats with VR Developer Environment Software Engineer Eric Le Saux about the SharedSpaces sample and how it demonstrates the Platform SDK Multiplayer features. Eric dives deep into the challenges and objectives of the sample and the choices his team made to create an app that demonstrates the Multiplayer features in an easy to use way. We discuss use cases that developers should keep in mind when designing multiplayer VR applications, best practices, recommended resources and learning material to help you create your own multiplayer VR experiences.
Social Accounts​
Episode Transcript​
Building Multiplayer Virtual Reality Applications in Unreal with Eric Le Saux
[00:00:00] Navyata Bawa: Hi everyone. Thank you so much for joining us. In this episode of the diff we will discuss a rather new topic and that is building multiplayer virtual reality applications. Today we will learn about the platform. SDK multiplayer features the open source, shared space sample and hand gameplay showcase sample.
[00:00:17] We will also have another episode where we'll dive deeper into the SDK, the shared spaces sample in unity and so much. And with me today is Eric, who is a software engineer on the VR developer environment engineering team. To tell us more about building multiplayer VR apps in unreal. Hi Eric. I'm super excited to have you here with us.
[00:00:36] Could you please tell us a little about yourself and what you work on?
[00:00:40] Eric Le Saux: Yeah. Um, why are you in Nava? So, uh, yeah, Eric Luso, um, I've worked in many industries, uh, over the years. And, um, in 2002, I started working in, uh, game development, uh, worked on massively multiplayer, online games, mostly, [00:01:00] uh, a reboot of mountain online first, um, at origin systems.
[00:01:07] Uh, then I joined, um, uh, Sony online entertainment. I worked on a number of expansions for, uh, ever quest. And also, um, working on other projects there. Um, other MMOs, then I joined, uh, red five studios where, um, we were building yet another MMO. Um, uh, there, I built, uh, the game AI systems, um, with it's own language.
[00:01:36] It's, uh, it's on the, uh, peculiar, um, uh, Uh, system that, uh, was really, uh, interesting reusable, lot of, lot of components that you could build. I, I could go for hours on that project. um, so after, after working on in at those three companies, uh, I joined Google for on few years, close to five [00:02:00] years and, um, uh, worked on, on things that were not game related.
[00:02:06] I still managed to do some, uh, um, some 3d graphics and stuff for, for my graphs, but, um, um, and, uh, a few years ago, a bit more than two years ago, I joined, uh, Facebook at the time was called Facebook on meta now, uh, At the Oculus, uh, in the Oculus division and more specifically in Def tech, which is a team that has two fronts.
[00:02:36] One is, um, so mostly the, the overall goal is, is to ease. Um, Uh, developer, uh, pain points. Uh, one, one side of the group works on optimizations and, uh, profiling and, and making things easier, making the, uh, the tool set better. And the other side, uh, [00:03:00] that I joined, uh, works on, uh, samples and showcases to try to demonstrate what you can do and, uh, you know, uh, good examples of, uh, how, how you can achieve some important results.
[00:03:15] And, uh, so yeah, actually that's, that's what we're going to talk about today. Right? And we are going to talk about, uh, our, uh, latest, uh, project, which is, uh, about. How to get together in VR easily. using the real engine. That's what we are talking about today.
[00:03:34] Navyata Bawa: yeah, absolutely. Thanks for the introduction, Eric.
[00:03:36] And yes. So let's talk about the samples. So the shared space is samples. So tell us what the sample is about and what motivated its creation.
[00:03:45] Eric Le Saux: Yes. So, um, So there, there are different things that can motivate, uh, the team to start working on a project. In this case, uh, there was a, um, a, uh, different teams that, um, uh, were brought [00:04:00] together to, uh, work on, uh, an overall banner under a banner that called, uh, multiplayer red to green program.
[00:04:09] The idea was, uh, okay. We have, we have our dev services. They have created those new ways of, um, Identifying where people are in VR. And, you know, like at the beginning of a metaverse idea where you can be in different applications, but you, you have something that binds them together where you know, about specific titles, specific destinations inside those titles.
[00:04:36] Um, and, uh, so we joined that, that, um, um, That project, uh, we start working on a sample. So as we, as we do this, um, we start digging deeper and deeper into what we have and what are the available services? So on the one side foreign real, we have the label of VR platform, [00:05:00] the low level C APIs that are, uh, ready, available.
[00:05:05] And those are kept more or less up to date on our U E four branch. So, so that's one thing on top of that. Um, the, um, Oculus had built a, an unreal, uh, online subsystem call that OSS. So the Oculus OSS, uh, was available and it's a, it's a layer of adaptation to those label VR platform, um, uh, interfaces. Uh, so what those interfaces provide you are, uh, ways to communicate with each other, you know, peer to peer communication, uh, some VO support and a number of other things like, uh, you know, leaderboards and, um, uh, challenges, uh, Uh, to notion to create rooms and, and send invites and a bunch of other things.
[00:05:55] So the OSS is a layer of adaptation on top of that. [00:06:00] And what we noticed pretty early was that, um, that was not a good place for the new APIs that we had in label VR platform. And, uh, so that, that was kind of a problem for us differently.
[00:06:15] Navyata Bawa: Yeah. So you, you talked about all the initial issues that y'all faced, but, uh, what were some of the things you encountered while you were working on this project?
[00:06:25] Like, what were the other things that came up and you like didn't expect it at that time,
[00:06:30] Eric Le Saux: uh, that were not expected? Um, well, we, we, you, when you from the get go, it, it would not be a trivial project because multiplayer is hard period. Uh, there are so many aspects that are difficult in, in multiplayer. So, so one first suggestion is definitely, if somebody wants to build a multiplayer project, they should, from, from the ground up from the very beginning, it should be a multiplayer project [00:07:00] as you start, because the first things you, you will encounter as difficulties are, what do I replicate between players?
[00:07:08] Um, who's authoritative for these. Um, do I trust my players so many questions, so, um, I will not get into all that just now, but, um, yeah, those, those were definitely some of the main challenges. Uh, and another big challenge was the OSS. What do we do with that? Do, do we keep it, do we keep the USS? And then we have some library on the side, and then we have two ways to talk to, um, our li VR platform.
[00:07:39] Um, so, so tho those were the questions. Another was a, do we make something that is actually playable, you know, something fun with a bunch of different experiences. So that was the long term goal, but the initial goal was not that the initial goal was, uh, reduced. Now what, what we want is to show how, how the mechanics of [00:08:00] connecting players and, uh, determining where the server is running and so on.
[00:08:06] Um, uh, so those were the, the aspects that we wanted to solve first. Uh, so yes. Um, so one solution or the first solution or the, the first decision we made was, um, we are going to drop the Oculus OS. It's not maintained. Uh, it's a bad fit for all these other APIs. And, uh, it's hard to maintain. One of the reasons it's hard to maintain is because it's a layer of adaptation.
[00:08:39] So, uh, what, what I, I, I could say about how it works on our side of things is that we have a number of APIs that have been created for the li of your platform that are cogen. And we have a number of targets. So, uh, someone has written a target as C sharp for unity. And, [00:09:00] um, so, uh, C API calls or interfaces.
[00:09:05] And, uh, all that was working really well. And we wanted the same thing for unreal, and that's what we have been working on. But for shared spaces, uh, the, the initial goal was just, let's start with a simple plugin that would, uh, target just the APIs that we need. And we are going to make it really tight, uh, around those APIs and, and work just for that.
[00:09:30] So that's how we started. So no OSS, we started plugin and we start implementing all the, uh, the interfaces that we needed. And with the proper callbacks and all those nice things. So that's, that's how we started.
[00:09:45] Navyata Bawa: Awesome. That is so interesting. I'm so curious to know more about how the sample handles all these moving parts that you mentioned.
[00:09:51] Can you tell us more about like how the sample is organized and like how it handles all of this?
[00:09:56] Eric Le Saux: Yes. So, um, the project [00:10:00] itself, so, you know, shared spaces is, is a pretty simple. Um, a group of rooms, you know, it's laid out as, uh, a, a, an initial room that is kind of a lobby. Uh, um, so I, what I should say, I should backtrack a little bit, I should talk about, you know, how our dev services, I have created the notion of destinations and presence.
[00:10:29] So inside a title you can create, um, Places that, uh, represent well known destinations in your title. What you want to do with that is, is left to you. It's very flexible, but to simplify things, uh, what we did is, okay, so each destination that we're going to create for shared spaces is going to correspond to a specific level in, uh, in our, [00:11:00] in our example,
[00:11:04] So, so that's for the destinations. Now on top of destinations, you have two IDs that you can set to indicate where you are in the world. That's totally left to your application, by the way, the way you set them, the way you interpret them. So you, you tell the platform where, what, what those values are, and it's up to you to decide how this is going to be handle.
[00:11:30] It's very flexible, but basically there's the usual concept of lobby and of match. The concept of lobby, uh, is, uh, you can think of it as a group of people that you want to play with over a series of matches. The concept of match is one specific instance of. A match , whatever that means in your title. [00:12:00] Um, and so now let, let's talk about how we have mapped that in, in, in shared spaces.
[00:12:06] So first of all, You, when you start on your own, uh, shared spaces, uh, the game opens on a gray level with a bunch of doors on the left and the right, um, called the lobby. Yeah. It's written in big letters. Cannot miss it. So you are, you are in the lobby. So the idea there is, uh, when, when you're in the lobby, you can invite other people to join.
[00:12:32] So if, if they join you and they are in your lobby, uh, they accept the invitation. Um, now you have a group of people that share that identity of, of people in this lobby that are going to play together for a little while now, for the concept of matches. What we have done is simply, okay, we're going to create a number of other levels that represent, um, matches.
[00:12:59][00:13:00] Uh, you, uh, and to go there from a given lobby, you have to cross a number of those, uh, colored doors, uh, that you see in shared spaces, uh, for, for people that listen to this and have not seen it, uh, on YouTube, we have a video, uh, where all this would make sense, so you can visualize what we're talking about.
[00:13:21] So, uh, those doors lead to private matches that are linked to your. So anyone that has been invited to a given lobby and crossed those doors, they will end up in the same private match. So it's that red door for this specific lobby and, uh, you know, the green room and the, the blue room. Um, So here's at works.
[00:13:44] You invite people, you are with them. You can go wherever you want. Uh, one guy can go to the red room. Uh, girl can go to the, uh, blue room and they are in separate rooms. They come back to the lobby. They are together again, they can go together in, uh, different rooms and so on. And, [00:14:00] uh, in theory, the, each of these, uh, rooms would lead to, I don't know, a chess game, a bowling game, a dark game, whatever it said, that was the idea at the time we would have.
[00:14:11] Sorts of mini games. We didn't have time to implement all that, but anyway, I'm sure people on the outside, when they start implementing, um, their own version of, uh, starting from this, they, they will come up with very fun stuff. Actually. I know you did.
[00:14:29] so, um, So the idea of match is, is really an instance where okay, we're together and we can play together. And, um, um, and we're in a specific space for that. The notion of lobby is okay. When I leave that match, I will be in that specific lobby. And I will be with the people that are allowed to be in that lobby.
[00:14:50] Now, imagine if I go to the red room. And now I invite someone that is not from my lobby. I, you can do that the way we do it is [00:15:00] simply we invite that person. If that person accepts the invitation. So it, it pops up on their headset that they have been invited. So, and so is inviting you to join. And if you accept by clicking on the system, uh, UI that pops up, uh, you are trans you, the application is started with the information to, to launch.
[00:15:21] And the way shared spaces interprets that is, oh, you, you have been invited to a match. Okay. You don't have the same lobby ID. That's fine. I give you your own unique lobby ID. However, I will give you the same match ID because you accepted to join that and you end up being the same space. So you are together in VR, you have been connected together.
[00:15:43] We'll see how that works. And um, and those two people can play the, let's say game of. And when they leave, they each go back to their own respective lobbies. So, um, at that point you may invite that [00:16:00] person to your lobby. If you want to play another, uh, you know, series of matches or whatnot, it's just to illustrate how those concepts of destination lobby and match session IDs work.
[00:16:12] And, uh, so it's very flexible. It's very open. You can do a lot of things. Uh, the, the way you want. The only thing that is a bit of a system, uh, meaning for lobbies is that whenever you bring a roster panel, which is, uh, one of those system UI that you can bring up in the title, the names that you will see there as part of your roster are people that share the same lobby ID, but that's the only, um, uh, the, the only.
[00:16:41] Not constraint, but, um, I mean, um, uh, meanings, uh, technical meanings of, of the, the, the, the lobby itself. Otherwise it's up to you. You can have multiple destinations in one, given unreal level. You, you can have multiple matches in the same level. You [00:17:00] it's up to you. It's, uh, it's a, it's not a one to one relationship with a level and a destination and a, a lobby or a match.
[00:17:08] It's really up to you it's, uh, up to your title and how it works.
[00:17:13] Navyata Bawa: Yeah. I think that is really interesting. The fact that it is so flexible that people can actually make different, uh, instances, different games out of it. I think that's really nice and flexible. Yes. Um, so you mentioned there are a bunch of layers that are responsible, like for matching, for invites for just hap the networking layer.
[00:17:29] Like, could you tell us a little more about what are the, what are the layers that are there and like, yes. Yeah. And what control people have over
[00:17:36] them?
[00:17:37] Eric Le Saux: yes, you can. Uh, so, uh, for people that have, um, downloaded the, the project, uh, that's on GitHub, by the way, um, the, the project will, when you open the, the, uh, shared spaces, game instance, Uh, you will see how so.
[00:17:56] So that's where most of the logic is. Uh, so a [00:18:00] game instance, uh, in unreal is ki kind of a Singleton. It, it, it lives beyond loading different levels. It stays around so you can keep stay there. And, uh, the logic of, uh, uh, where you are. Um, and that, uh, that blueprint is split into a number of different layers.
[00:18:21] Uh, there's a social layer, uh, that, um, is characterized by what the O VRR platform. Offers in terms of, uh, you know, what title you're currently running, what is the current destination? What are your match and, um, what your lobby and match, uh, session IDs. And, uh, so, so that's your social presence. So that's the social aspect of things.
[00:18:48] It's that layer that will allow you to invite your friends and to join you. It's that layer that will also pop up bunch of, uh, you know, system UIs for you [00:19:00] to accept invitations. Um, By the way that system is really flexible. Just the apprentice, this here, the, uh, there's many different ways you can send an invitation to, to someone.
[00:19:12] Um, you can even send them via, um, you know, some, some messaging chatting system and people will receive the, uh, the invite in some different way. They click on a link on the webpage. It brings up, uh, some page on, uh, the Oculus website. And then you can launch on your headset. So many different ways to, to organize this and they are still working on other ways.
[00:19:38] So that's, uh, there's a bright future in that direction. Um, so that's the social layer. So that's the first layer where you, you define those things. Uh, next layer is the transport layer. Uh, the transport layer is, um, um, Oculus offers one that was part [00:20:00] of the old OSS. Naturally the first, very first version of shared spaces, uh, that was not published was using the room system from OCUL.
[00:20:11] So we were using that to connect and so on. So that works. Uh, people could use that potentially, but, um, they should probably not. Let's explain why later, but, uh so what, what we had as a recommendation from, um, another team at calculus, the developer relations guys, they work with game studios and, uh, you know, they want big titles and stuff and stuff.
[00:20:34] And, um, so. What the industry typically wants for, uh, when they develop titles is a more, um, um, system agnostic, transport layer, not something that is specific for Oculus, for instance. Uh, so what do you often recommend is photo. So the, um, [00:21:00] photon real time SDK, uh, is one of them. And, um, um, that's, that's a layer that is very commonly used by people and, uh, that's what they recommend, uh, their third party studios to use.
[00:21:16] Uh, I think there are many historical reasons for. I don't know all the answers to why, why things are, are the way they are. But, um, uh, regardless we decided, okay. Yeah, sure. We are going to make an example that demonstrates how to use full time. Uh, we had discussions with, uh, the photon team. Uh, we, we needed to talk to those guys because, uh, um, we could not ship the, uh, sample with their SDK without their permission.
[00:21:45] So we, we came to an agreement with them, really nice guys, and, um, um, So default on layer. What does it do? Uh, there are, uh, uh, well, Z Zach is the expert on this. I, I [00:22:00] think you will interview him, uh, too, as there will be another part to, to this, uh, this presentation. So I'll, I'll let him talk about that in details, but basically your connection between players and the difficulty here is, um, So when you build a title, one, one big problem you have is, uh, a, a multiplayer title.
[00:22:23] You have to decide where your server will live, um, in this case, because it's a simple example and, uh, the load would not be too big and also, uh, running a dedicated server for unreal. Um, in a data center somewhere, uh, you know, people will use today, uh, the Amazon, uh, uh, services, uh, web services and all, all sorts of, uh, PlayFab, uh, bunch of, uh, uh, backend infrastructure things.[00:23:00]
[00:23:00] Um, not play FFA. Playfab's more the middleware, but, uh, the, uh, I mean the, the data centers that you can use today, you can use Google. You can use also some other guys, Microsoft, but, um, The difficulty there is, is yet another level of complexity that we didn't want to tackle. So what we opted to do is to run the server on headset, uh, for unreal, what you, uh, you call that a listen client.
[00:23:28] So there's basically a client that not only, um, Executes the code for your own player, but also runs the logic for the host. Um, and we do that on headset. So first of all, we were not totally convinced that we had all the budget on the device to be able to connect a lot of players, but, um, we tried it and Hey, this works actually pretty nicely, so yay.
[00:23:54] Let's do that. So the, the idea was really to, uh, to be able to have. [00:24:00] Um, the host running on headset and other players connecting to. So that's where photon is really, really useful because, um, um, if, if you try to connect to the headset of someone running, uh, in their own, uh, in their own, uh, home network, uh, you, you can't reach them, right.
[00:24:19] It's blocked, uh, you, um, to be able to address them, you need mechanisms, uh, to go through their firewalls and do all sorts of. So there are Alsos of mechanisms to, to, uh, the tricks of the trade to go through those firewalls. Uh, one, uh, simpler option is to, uh, in the case of photon. Um, is to use one of their servers as a, a, um, a reflector, right?
[00:24:49] You, you talk to their servers and their server knows where to send those packets. It doesn't care what the packets contain. It's just a pipe, uh, to, uh, [00:25:00] propagate, uh, packets around. Now, uh, unreal will understand those packets and will do something with them, but, uh, photon, uh, really doesn't care. So at that level, at the, at the photon layer, what you really care about is to be able to identify, first of all, you need to be able to start a space that you can connect to.
[00:25:20] So in our case, when you start, um, when you join a room, whether it's a lobby or a match, doesn't matter is the same. What you do is you, you connect to photon and you say, Hey, create or join a room of this specific name. If the room exists, you just join it as a client. And if it doesn't, it's created for you and you join it as the master client.
[00:25:51] So you're, you're the main client on, on that room. So when I started the game for the first time and there's nobody around, I start my lobby. I'm in the [00:26:00] lobby. I call photon photon. I tell it, Hey, create the lobby, such number that is mine. And that corresponds to my lobby session ID at the social layer. My lobby session ID is used for the name of the room and that's that I created the room I'm connected to it.
[00:26:18] Now, if I invite someone to join me, they, their lobby session ID will be set. And they will contact photon, Hey, join or create room with this lobby ID happens to be mine already exists. They connect with it as a client. And now the pipe is established between my headset that runs the host and the server, and, uh, that other guide that I invited and now connecting to me now, the communication is established.
[00:26:49] So that's how it works. And, uh, for matches, same. Uh, let's say I go to the red room. So I leave the lobby. I join the red [00:27:00] room. I set my match ID for the red room. So it'll be called, let's say red room for lobby. So and so, so a specific, unique name linked to my lobby. I ask photo town to create that room.
[00:27:12] It'll create it. It's a nobody's there yet. So I create it again. If that person follows me to that room, same thing happens. Leaves that room joins mine. We connect through another room, another connection. So really, um, a, a neat little system there, um, uh, that, uh, that works really nicely. And one key point and, and need to talk about that.
[00:27:38] So you may be wondering, okay, well fine. But, uh, what happens when you leave for that red room? Well, what happens to the guy that you leave behind? Uh, is he just, uh, disconnected and back to. Back to his home environment. So the way we handle this, and that's probably one of the key things that people need to [00:28:00] implement whenever they do hosting on headset, uh, running server on headset is to do the handoff of who's got running the host for that specific room.
[00:28:13] So the cool thing about photon is that for every room you have, that notion that I talked about of master client. If the master client leaves, everybody's notified everybody that is currently connected to that room is notified that, Hey, um, there's a change on master client, somebody left, uh, here's the new master client.
[00:28:37] So it's basically the, um, it's a queue. So all this person in the room after the, the, uh, the current host leaves. Uh, becomes the new master client. So there's no negotiation for us. It's simple. We get the notification at the photon layer that we have a change of host every [00:29:00] player reconnects to that, to that room, but as, uh, either a client or either as a new, um, master, uh, client.
[00:29:10] So. Uh, the, the room never ceases to exist. It's still there in photon. But what changes is, uh, at the unreal level who's hosting, uh, or who's a client. So people that were already clients, they need to reestablish client connection with the new host. So everybody reconnects. You know that if you're the master client, you need to start a listen server or a listen client, you be host.
[00:29:37] And first client of your host. You tell that to photon. Well, you don't have to tell anything you just connected already. So, but the other guys, uh, have to reestablish connection and they will reconnect to you. And again, um, yeah, now I started touching the lower level, which is the unreal layer. Um, so let's [00:30:00] recap.
[00:30:00] We have the social layer where you set your destination, your match, your lobby. You have the layer that we extensively talked about just now the photon layer. Uh, that's uh, the one that establishes the actual pipe for your packets to fly around. And, uh, that also tells you who should be host, who should not.
[00:30:22] And finally the third layer is the actual replication, the game replication layer, which is, uh, very important of course. Uh that's um, the layer that establishes, um, how you start your server, how, how you start your client. Uh, how you replicate things between, um, people and, uh, how you should do it and so on.
[00:30:47] So that's very complex. Uh, it's, it's a lot of work, uh, be warned. It's not easy to make a multiplayer game, but at that level, um, at the very basic [00:31:00] connection time, it's simple. It's a, I'm the master client. I start the. I'm not, I establish a client connection and that's it. So, so that's, that's the, the, uh, the, the connectivity side of things.
[00:31:14] Now, of course, uh, as I said, in the replication layer, you have to decide, okay, what, what do I need to transmit? Uh, do I want the other people see? Um, so in the case of shared spaces, we kept it really, really simple. Uh, we have, of course the location and orientation of your character that needs to be replicated.
[00:31:37] Uh, we, more or less got that for free because we reused the, uh, unreal mannequin that has already all the, um, uh, very interesting components that allow for nice and smooth replication of move. So we will use that on top of that. Uh, the color of your avatar is being replicated, [00:32:00] uh, your name and, um, that that's more or less it.
[00:32:05] Yeah, , that's very simple for, but, um, of course a real game will have much more that you want to replicate. And, and how you do that is, is a, is, is a very, very interest. And difficult topic. Uh, there's a lot of stuff you can read online, uh, that will help you. You need to get familiar with that. That's for sure.
[00:32:28] Um, and, uh, so that's why again, do it from the very beginning. Don't start building a single player game and then migrate it to multiplayer. That is hard. Uh, now start from the very beginning, start simple, understand what you're doing, understand how replication works. Um, You cannot, you can replicate just by having, oh, those values are going to be replicated that rhythm to all clients on a regular basis.
[00:32:58] Um, if you don't get [00:33:00] all the replications, it's fine. Ju you just want to know the latest more or less. There are other calls that are, um, Ity calls remote Ary calls, right? You call a function and you really want that to be replicated to all, and they need to know that the specific event happens. And, uh, that's how you do things.
[00:33:21] Um, and, and then you have to manage all that. You have different options for a number of different things. um, some things that are not necessarily trivial for people to understand is that sometimes you can, you could use for instance, a, a, a remote procedure call to tell everybody this is happening from now to, uh, forever.
[00:33:47] Okay. Example, uh, this elevator will go up and down forever at this rhythm. If everybody knows how that this is happening. And, uh, the elevator is never going to [00:34:00] stop. Let's say, uh, let's keep it simple. Then you just need to send a message once and everybody can do it in parallel. And as long as you keep your clocks in sync, you should have the elevator more or less at the same place for everybody.
[00:34:15] And at that point, you have lowered your bandwidth usage, right? You just, you just told everybody what to do. And, uh, so if you do it well and you verify from time to time that everything is in sync, that's, that's a good way to do things. Another option is just to replicate the location of that elevator all the time to everyone.
[00:34:33] Um, this has some advantages. Um, so first of all, kind of guarantees more precision about where the elevator is, and also you can stop it at any time and you can tell the server, I just blocked it and, uh, everybody's listening to that and oh, they stop it and they, they stop it more or less at the same place, but you can achieve that again.
[00:34:53] Um, by events more than by replication of location. Uh, so you have a lot, lot of, lot to explore and [00:35:00] understand to, to make an, an online game. Um, so yeah, those three levels, right? Uh, social level saw that. Those rooms that we have in shared spaces represent those different places. The, um, uh, the photon layer, the pipes, you know, how you things, how people get together.
[00:35:19] And finally, the, uh, replication, the unreal engine. Replication layer game replication. Um, that's where you probably will have a lot to learn if you're not familiar with those systems.
[00:35:32] Navyata Bawa: Yeah, absolutely. There seem to be so many moving parts, like, um, I'm kind of curious, like, what are some of the use cases that they developers should keep in mind when they're designing some of these VR multiplayer applications?
[00:35:43] I know you gave some use cases, so like, are there some points that they should keep in mind when they design multier VR apps, like right from the get.
[00:35:51] Eric Le Saux: Oh, yes. Um, first of all, it's um, when you develop for quest, um, [00:36:00] the, you have constraints, right? You have memory constraints, uh, you have, um, processing power constraints.
[00:36:09] So, uh, shared spaces with a thousand players, uh, not so much, no, that won't work you cannot handle on one headset, all that traffic, all those simulations, and, and think that you're going to be able to replicate that down to everybody else. So, first, first question you have. What can I run in one, given headset in a multiplayer title?
[00:36:36] How many people can be together? Uh, how can I limit, uh, the number of players I see at any given time, because, uh, may, maybe I allow up to, I don't know, number like this I'm I'm not sure. It depends. It really depends on how taxing it is on the CPU, how much traffic you send back and forth between players.[00:37:00]
[00:37:00] And how precise you want your title to be. There's so many questions, right? So people have to build it and they have to optimize and they have to assess, okay, this is doable. This is not doable. Um, so that's the first question, you know, where, where is the code going to run? Do I need a dedicated server running in a data center?
[00:37:22] Can I run on headset? Can I run part of the experience with host on headset and other experiences on a dedicated server, you can mix and match. You can do all sorts of things like that. You could have, for instance, a small lobby where you allow up to eight people. And that will run on headset. Fine. They just talk to gutter and, uh, let, what, what do we do next?
[00:37:44] All that on headset and then, oh yeah, let's start a game of this and that. And they start, uh, that in entitled room, uh, that will spin a, spin up a server on a, [00:38:00] in a data center and then they connect to it and you can do. So a lot of questions, uh, just to, uh, to get started in terms of performance memory, uh, where do I run my service?
[00:38:12] So, so if that's the very first question, um, of course they have the usual other, um, content questions but so that's, that's the same for multiplayer games, you know, um, the complexity of your environments, the complexity of your textures. You have to, you have to deal with that. Do I bake everything? Do I like the unreal engine?
[00:38:37] Uh, do all that magic. Uh, can I afford it on my headset? Uh, so that's those, those are very important questions, uh, for content, of course, for gameplay, uh, gameplay that's, that's a key, right? It has to be. It has to serve its purpose. If it's a game, it has to be a game you want to play. Right. has to be fun. It has to be, it's [00:39:00] hard.
[00:39:00] It's not easy. Uh, what is it? What is fascinating with VR is that a number of new experiences are possible. And, uh, there are a number of things that you could not do before that, uh, are entirely doable. Um, I think another project we worked on was, uh, doing hand tracking and that project was fascinating for, because the outward looking cameras on your headset, uh, you know, those guys, you have a bunch of cameras everywhere and they look at your hands and, uh, Uh, another team worked on, on those, um, machine learn models, uh, that would recognize, you know, that would interpret that as, uh, you know, bone rotations and, and positions and so on.
[00:39:45] And we did a number of interesting things with that. We built some plugin that would, uh, recognize, uh, poses like, uh, you know, thumbs up. Which is different from, uh, some gun pose or, you know, that type of thing. [00:40:00] And also we built another system, another layer on top with that, with, uh, recognize gestures. Uh, that's that's on the hand game play, uh, it's it's in there.
[00:40:09] You, people can go and explore that and, uh, see what we did there. Uh, they can use that as a starting point and start exploring what's possible. Uh, it's not easy because as soon as you leave the field of view of the camera now, oh my God. It's, it's start to have a lot of errors and eventually it doesn't see anything.
[00:40:26] So it tells you so, uh, but when it's in front of you, it works pretty well. We had some. Um, sign language recognition at some point, which was pretty cool. Um, yeah, I think there's a future for that for people that, uh, would be able to communicate using their hands and it, maybe it would generate voice. Uh, that would be interesting.
[00:40:46] I think somebody should work on that somewhere.
[00:40:49] Navyata Bawa: yeah, absolutely. I mean, ex it's going make it definitely more accessible and inclusive for everybody to use these apps.
[00:40:56] Eric Le Saux: I think so. I think so. I think there's a number of things there that, uh, people could, [00:41:00] um, Definitely take advantage of for communication.
[00:41:03] Um, yeah. Yeah, definitely.
[00:41:07] Navyata Bawa: So you mentioned that there's the showcases sample and also the hand gameplay sample that people have on GitHub that we can use and right. Play around. So what was, what was the reason that motivated you to put on GitHub or like just open source that project and like yes. Are there other, uh, examples that we can also check out?
[00:41:27] Eric Le Saux: So GitHub, the big advantage is, um, it lowers the time for us to, um, send updates to, uh, to people. So just recently, for instance, um, uh, we, we had some, um, people on getup contacted us by opening issues right now. We just have issues, unfortunately, that they can communicate with us. We're trying to convince some, some people internally to let us open the, uh, discussion groups [00:42:00] on GitHub or something similar, but in any case, uh, the, um, uh, they communicate with us and, uh, we can exchange and we can quickly bring fixes and improvements, uh, to the system.
[00:42:13] So, so that that's, um, that's a big, big plus. Now if somebody has a fix or, uh, they want to send us a pull request. Um, that's also a possibility, right? It's also, uh, we can review it and, and look at it and oh yeah, that's pretty cool. Thank you. And, uh, we share it with the rest of the world or if we don't, uh, other people can still, you know, pulled it down and, uh, did decide to, uh, give it or try themselves.
[00:42:41] So, so that's the beauty of, uh, open source. Um, the other thing is, I think we're also moving, um, with unreal five, that's more of the publishing model that, uh, we want to adopt in general. There will be also the unreal marketplace where, [00:43:00] uh, some stuff will end up there, but, uh, for code that we want to share with people, GitHub is really a nice place.
[00:43:07] So yeah, that's, uh, that's the main reason we'll try to bring also more of the older samples. If we update. We can push them to GitHub and, um, and people can get them from there instead. Uh, but yeah, right now we try to consolidate how we push our samples out before we had the number of different places was still the case.
[00:43:31] It's, it's a bit hard to find the different pieces you need. Uh, so yeah, and that direction is a good one because, uh, it's there to. For sure.
[00:43:41] Navyata Bawa: Yeah. I mean, open source definitely helps create like this healthy community around like every project where people can learn from each other and just grow. So, I mean, that's great.
[00:43:50] So that sounds really exciting. So do you recommend any other resources or like learning materials where developers can learn more about like the shared spaces sample? And's just try it [00:44:00] out.
[00:44:01] Eric Le Saux: Uh, yes, they, they can find it on app lab. First of all, they can try it out without having to build anything. Uh, but, uh, the real interest of the showcase is, uh, well, yeah, running on app lab will let you have a feel of, uh, you know, the, uh, the, the, the quality of replication, you know, any shoes you may have, uh, you can maybe even test, uh, low bandwidth networks.
[00:44:28] You can do all sorts of things just with that. But, uh, really the cool stuff is in the C plus plus and blueprint. That we have on GitHub. So yeah, you, next step is really, uh, you get that and you try to build it yourself. You try to, uh, go through all the, the hoops of creating on the dashboard. Uh, the Oculus dashboard, you create your application.
[00:44:53] Then you go on the platform services and you create your destinations. That's very important because you. [00:45:00] There's a lot of stuff that I've not talked about yet, uh, that, uh, that is right there on the dashboard where, uh, you have, you have a deep link message. In our case, we decided to format it as adjacent, um, uh, uh, message that contains a lot of, uh, key value pairs, uh, that you can use to, um, ship your game in our case destinations map to levels.
[00:45:30] And that mapping is stored on the dashboard where you have a map field that indicates our level. I don't remember, but I think it's map that indicates the name of the level that this corresponds to. So, uh, that's, that's pretty, um, uh, important for people to use, uh, and understand how that works. There's a number of other fields.
[00:45:52] Uh, one indicates whether the level is a lobby or not, or the destination is a lobby or not, whether it's a public match or [00:46:00] not. We didn't talk about that one purple room that we have in shared spaces. It's a, uh, um, so one thing we did not do there's little pan, this one thing we did not do in, uh, in shared spaces was the use of matchmaking.
[00:46:18] Uh, we wanted to didn't have time, but, uh, basically, uh, the purple room kind of represents that it represents a common place that people can join without having been invited before. But you can see that, uh, if 10,000 people are running shared spaces, which I it's unlikely to happen, it is just a simple, simple, but, uh, if, if let's say it happens and, um, People all go to the purple room.
[00:46:45] The purple room is a common room. It's not linked specifically to your lobby. It's linked to all lobbies all lobbies by crossing to that purple door you end up in the purple. And, uh, so you can imagine 10,000 [00:47:00] people connecting to, um, uh, to, to one space on one headset that doesn't work. So , I can, I've not tested it, but I know it's true.
[00:47:12] and, um, uh, so normally you would use matchmaking for this. Oh, you want to go to purple room? Okay. Let's find one that is available that has, uh, that has free space. And then you re you take it, you reserve and you can limit how many people get in that specific instance of the purple room, but it would still be public one, except that it'll be charted as, uh, multiple instances of, uh, rooms depending on the load and so on.
[00:47:41] So you have, uh, you can have many of. Uh, started at once, so that that's where matchmaking should, should, um, uh, enter, uh, and, and see the, the, the light of day for us. We'll probably do that at some point. Uh, we'll we'll do it as an example. Uh, we just didn't have [00:48:00] time yet. I
[00:48:00] Navyata Bawa: think that'll be cool. I mean, that'll be, that'll make it so much fun.
[00:48:04] Eric Le Saux: yeah. And I, I think people need to see that example, basically not that shared spaces is a something you, you are there for a lot. Um, but, um, a real title would have that type of quality where, oh yeah, let's meet in VR. You use fired that up. And, uh, from there you, you can decide what you want to do. Um, Oh, that that brings us, um, uh, I don't know if we have time, but uh, I'd like to talk about VoIP quickly.
[00:48:34] so VoIP VO, so Oculus as a VO solution, right? It was part of the USS you need to be, you know, we could have made it work without, but, uh, uh, we tried to use the photon voice solution as C plus plus library that they provided. And, uh, we, we did not get the quality level we wanted at that point. We know photo and [00:49:00] voice works really well in unity.
[00:49:02] So it's just a question of, uh, okay. So do we go through the effort of adapting this layer and making it work for our specific case? And, uh, we never got to it. Uh, it was too much work at that time. And, um, so. Uh, but for the unity version of shared spaces, we use the photon voice there and that kind of works right.
[00:49:25] It's uh, people can talk and, uh, they can hear each other that that's all good. Um, now we, we are going to add voice to the, uh, the unreal version, but, uh, probably not with a photon version, but with a future version, uh, of the showcase, uh, that will, uh, use. Epic online services. Uh, that's a new project we just started.
[00:49:51] So, uh, it's not coming out yet tomorrow, but, uh, give us some time and we'll, we'll make that work for you. So it will be a new separate project from shared [00:50:00] spaces would be shared spaces us, and it will also be on GI at some point. And that will be an example of using an external OSS, not the Oculus. That we're moving away from because it's unmaintainable, but an external OS with the lib of your platform, plugin that we have created.
[00:50:24] And, um, the two systems working together in and hand with a clean unreal project that shows you how, how those pieces work together. Um, actually, uh, you know, when we started shared spaces, That plugin, I'm talking about the, uh, O VRR platform plugin. We, we started by just handcrafting those calls. Okay. Why are we going to call this and that?
[00:50:48] And we started plugging things together and here's my call back. And, and so on. Look pretty good. It works really well. But then we, we thought, um, as I, I kind of [00:51:00] hinted at the very beginning of our conversation that, um, we need to have full coverage of the overall. and, uh, so that's what we did with cogen.
[00:51:11] Um, now whenever our lab over your platform, uh, APIs change, uh, our cogen system, which is a complex beast of Python and YAML files. Uh, now we produce the C plus plus, um, uh, plugin for unreal. That is targeting unreal, really precisely, both for blueprints and for the C plus plus calls. And all the models.
[00:51:37] Are there everything that is public is in there, hopefully only what is public is in there. and, um, and, and this, this will be in sync with the lib of your platform and will provide to, uh, our developers full access to the li of your platform, tightly bound to it, up to date and, um, free of the OSS constraints.[00:52:00]
[00:52:00] Navyata Bawa: That is super exciting. I mean, I'm sure that that's gone last week.
[00:52:04] Eric Le Saux: Oh, wow. yeah, we just launched that like just, just last week we, we launched it and, um, uh, yeah, very happy with that. It's pretty cool.
[00:52:15] Navyata Bawa: Yeah. I'm sure people are gonna love it. Like when they try it out and see all the things they can build with it, I think it's gonna be great.
[00:52:21] Eric Le Saux: Yes. Yes. Hopefully, hopefully we want them to be successful. Right? That's that's the goal of our team. Our team is basically, Hey, we want you to develop for our platform. Let's make your life easy. You are a bunch of examples. Um, oh, label VR platform is not well exposed to UE four. Let supposed to U E four.
[00:52:41] No, let's do it. And now we have a, uh, really nice plugin that works really well. The way, the way the node work. Is really interesting. Um, I, I, I, I really it's a, it is kind of an advanced topic, but, uh, people interested, please go have a look it's you you'll see, you'll be happier if you use [00:53:00] that. And, um, life, it will be much easier.
[00:53:03] And, uh, in one node you have everything request you, uh, you know, you'll see those little clocks in the corner to show that it's a latent node and things are. You'll be able to do other things, but it'll be called back with the response. You see a success failure for the request. All the payload will be there.
[00:53:20] You can split those output pins into all the different components. It's beautiful. I like it a lot.
[00:53:27] Navyata Bawa: amazing. Um, thank you so much, Eric, for all these help. Full insights and thanks for all the amazing work that you and your team do. But before we end this lot of fun, you know, , I'm sure. So before we end this episode, do you have any suggestions or guidelines for developers who are just getting started with like brand?
[00:53:48] Eric Le Saux: Um, well, we, we talked about all, all the, the concepts of understanding, how, how the social, those three layers work together. [00:54:00] So that's a very, very good starting point. Try to grasp what's going on. Open up those blueprints. Look at what is going on. Go look at your, uh, share spaces. Um, mannequin code. I forgot the exact name of, uh, our character, but, um, uh, there's a lot of code there also.
[00:54:19] Uh, the. The code for spawning players and how we make sure that when you cross the door, you spawn just next to that door. That's a lot of logic in there. Explore all that, try to grasp what we're doing. That's very important. Now on, on the dashboard side, there's some, there's a number of, um, um, uh, things there that you can use that are not necessarily trivial.
[00:54:47] and maybe, maybe one thing that gets me every single time, every single time it gets me. It's the, um, data use checkout checkup. It's it's uh, when you create an application, [00:55:00] you have to go, you have to go in there and indicates that, uh, first of all, that you have a privacy policy for your users and, and that you're going to, um, and that you want to allow your application to be able to see, for instance, The user IDs, it's unique pro title.
[00:55:18] So it's not really, uh, you cannot crosscheck between applications, but, uh, um, that you can see those things that you can see their names that you can invite friends. If you do not check those boxes, you will wonder why, why is this not working? It's just because the over platform will return results that are partial.
[00:55:38] Because, oh, yay. This is not checked. So I will remove this and that and that, and that. And you, you receive empty fields and you wonder what is going on. So it gets me every single time. so yeah. I'm telling you that as a good advice. Um, don't do like me, just, just go check those boxes on, on the dashboard.
[00:55:58] Navyata Bawa: yeah, those are great [00:56:00] suggestions, uh, and great resources for our developers. Yes. So with that, we'd like to conclude this episode of the. And thank you, Eric, for providing us with such great insights into the VR platform. SDK quantum features all its features, the super useful resources, such as the shared space sample and the hand gameplay sample for all at developers to get started.
[00:56:20] It was great having you here. Uh,
[00:56:22] Eric Le Saux: thank you