Hands-On with Tilt Five Augmented Reality Tabletop Gaming!

hey Ron its norm from tested and welcome
back to projections where this week we’re gonna be talking about augmented
reality I had a chance to recently try tilt 5 these are the augmented reality
glasses made by Jerry Ellsworth and her company and based on a technology that
she’s been developing since she was at valve software she took that tech and
launched a product called cast AR about six years ago now and since then has
been iterating on the hardware on the software and on experiences to now
launch a new product called tilt five and I had a chance to chat with her
about what’s changed since Cass a are what’s in this new hardware with type
experiences they’re developing for tabletop AR and all that good stuff
let’s take a listen well Jerry thank you so much for having
us here looking forward to this demo for a long time now I know you’ve been
working on this for years and years and years so the latest iteration of this
technology and what you’re working on it’s an AR headset called tilt five I’ll
say I’m with the company for people who might not know and may have heard
rumblings about like how a our headsets work how does tilt 5 work let’s start
from the basics so our system is kind of unique compared to other systems our
system actually projects out to a special game board which is called a
retroreflector and that allows us to make the headset really high performance
lightweight super wide field-of-view and super comfortable for the user so part
of the the system is it projects out of these silver lenses here down to this
route this retroreflector and it comes back to each user and that allows the
focus of these pixels to be correct so if I put my finger next to a virtual
object it’s in focus and that’s super unique to our system a lot of other
systems have a fixed focus and you can’t focus on real-world objects and the
virtual objects and then also the system is super lightweight so it’s just 85
grams it just weighs nothing it just folds up and slip it on so our main
objective is to have a system you fold the game board out slip the glasses on
and just get right into fun you know no calibrations no complicated setup no
sensors in the room it’s a fundamentally different approach
– AR yeah then what we’ve seen out in the marketplace before where people have
used lenses essentially a projector is on so two waveguides lenses which let
you put objects into real space but have limitations like field of view yeah like
the accomodation vergence problem where things don’t look like they’re in focus
or you have fixed and a plane right there because you’re using the
retro-reflective and essentially projecting out and back end you solve
for a lot of it yeah exactly I mean we like I mean I love the systems that are
like AR anywhere those are really cool but they have limitations we decided to
do AR somewhere it’s kind of our internal way we think of it’s like what
can we do and what can we just nail and make perfect and so with this system
like playing with your friends like head to head or play with your friends
virtually or even solo games like we can nail this and it’s gonna be a great
experience it’s omit the tabletop game board it’s
yeah it’s having this board here which is about like 2 feet by 2 feet where
you’re calling for with the tracking markers around it can be any board game
yeah that you would buy the game store that you were a gen con’ recently people
were flipping out rare or it was awesome you have D&D in there you kept our
defense you can have single-player games multiplayer games action games is so it
doesn’t have to be board games it can be like you know never before have you been
able to sit across the table and play an action like war game with your friends
and that make it really heightens the experience so like if I like pull up
your base I get to see it in your eyes yeah and that’s that’s pretty unique and
all that software is kind of defined and parameterize by the hardware that you’ve
yeah iterated toward we talked about you have this nice yeah I’m here view of it
it really helped helps illustrate I think how this projection system works
they look at the projectors and it’s really novel that looking at here the
projectors actually go toward your eyes bounce yeah yeah so these lenses here
these are actually really special lenses I mean they just look silver and kind of
plain but there’s actually a lot of layers in there this is one of the
inventions I did pat on my back it’s like this actually controls polarization
and the way that the light gets reflected on the inside so that you
get 85% efficiency when the light goes out and back again where if you just had
a silver mirror you’d only get 25% of the light so it’s a very elaborate
high-tech one-way mirror system yeah that doesn’t work the polarization to
make sure that maximum light is transferred to the user but it also
isolates each eye so you get this 3d experience and it also helps isolate
between multiple users so you can have any number of users around there and
then there’s two projectors no the two projectors shine down onto these lenses
and we’ve gone through like five iterations on these and the new ones are
fantastic they’re 110 degree field of view super high fidelity and in really
small and light like even between this iteration in this iteration we shrunk it
down even smaller it’s like two sugar cubes worth of like volume I’m looking
around you know your office you have all these parts everywhere so a projector
like this it’s not something that’s off-the-shelf like you know people
always think that we just get it off the shelf like no this is like four years of
development to make this projector because you can get small projectors but
there’s you know very narrow field of view they have a very shallow depth of
field and so we had all these requirements we had to have like InFocus
over a large depth we had to have a massive field of view had to be super
efficient so that we could run off you know cell phones and right right when
people think of like pico projectors like those would not work great for this
but now a lot of those projectors get too hot thermal was a problem for us too
so we had to come up with solutions so that we get the efficiency of the LEDs
up enough that we weren’t heating the LEDs up and heating the headset so yeah
a lot of work went into the projectors and a projector like that’s one
necessary work well as like no if you point this at the wall it’s only 0.65
lumens or point at your hand you can’t even see right but because of kind of
the amplification of the retroreflector how it just takes all the light and
brings it back to each user you get the super-bright image like we were demoing
at Gen Con and it was probably the most harsh environment we’ve ever convention
lighting these huge like mercury vapour lamps yeah it was super bright and
people are like wow I can’t believe how bright this isn’t
yeah and the objects look opaque and yeah really
are you talk about resolution and these are HP so they’re they’re 720p
all right and then you’re refreshing them at a 180 frames per second 80
frames a second which then needs to talk about how you’re processing all that cuz
games then running at hundred eighty frames no they don’t have to no no
that’s a really cool part of our system and it took us a long time to get here
it’s like I dreamed about this like years ago when I worked at valve I’m
like in the future the game engine is going to run asynchronous from the
actual projection system and in the headset it’s going to do reprojection to
realign the image and do all the prospective transforms that need to be
done and so that’s what we’ve achieved we have a chip on here that receives
images over USB it lands in the chip and then the chip upscales it to 180 frames
per second and does all the reprojection so as you move your head back and forth
or your shift side-to-side the image stays locked to the game board and it’s
updating at this insanely high frame rate in the VR world we hear about
reproduction all the time to compensate for lost frames and it’s very difficult
you know spatial projection for from a lot of movement this is all hardware
base yeah the loop the entire loop is in the
headset which is really unique like VR systems are just brute forcing it by
having giant video cards so what this means is that we can have multiple
headsets running off one PC and as long as the framerate it’s good enough and
you’re happy with the animation right it’s running at 60 frames you could have
two instances of games running at 60 frames or four at 30 frames per second
and for your perspective as long as it’s comfortable as long as I’m used updating
and the reproduction again is updating then you’re usually oh yeah you won’t
have no discomfort yeah you won’t have what’s what’s bad in VR and AR system is
when the tracking goes bad and you drop a frame then awesome the image like goes
flying off somewhere because you had head motion this happens to me all the
time I’ll be sitting on my desk working with unity and I’ll crash unity and I
won’t realize it I’ll be looking at the game board waiting for something to
update it’ll be like come on come on oh you
don’t notice it because the last image is right it’s running basically at zero
frames get the last one frame right but it’s
still smooth it’s just no I should have shown you that we’ll have to fire it
back up I’ll show you that oh it’s neat very cool and then then the tracking
these are the markers oh you’re doing positional tracking of that way say
inside out and what’s unique about our system on the tracking is we have two
cameras that are right here so one of them is 140 degree field of view
infrared camera so you can play at night you can play full daylight so anywhere
in between because we illuminate the game board with infrared light and
that’s on a specific wavelength and we filter the camera so the camera is
another custom-designed piece has a bandpass filter to only respond to that
wavelength of light then we have a second camera in here that runs on a
totally different wavelength of infrared light that’s for tangible so playing
cards and tracking complicated object tracking like tracking miniatures on the
table and that recognize that’s basically computer vision yeah yeah
recognition you can be as simple as QR codes shapes and your your one control
there yeah all that stuff is positionally tracked because you have
that inside out track yep yep man it’s an 8 megapixel camera so it’s super high
res so we can pick out features like it really far distances with it and we
talked about the the tracking field of view being wider than your visual field
with you which so I think you were trying that out you were like clear off
the game board yeah still tracking it yeah and then also you know you did a
thing where you brought a extension to the game yeah you’re also rendering
beyond this because you don’t know the user the system doesn’t know where the
game wardens necessarily except that it’s somewhere within that kind of big
field of view and so I was able to look in and see as far yeah into the world or
even extending the work yeah yeah if you just put another piece of a game board
here like a taught another tile and your your game board just extends and part of
our offering will be will have a base kit which has this fold-out rectangular
board just easy gives you everything you need but then we’ll have the deluxe kit
that has like the tiled system that you can do really fun stuff with like make a
rectangular board if you’re doing D&D right or if you want to have like lots
of vertical you can have a little kickstand so you can tip it up and you
can have all this vertical height out of the game board so that that’s that’s an
interesting thing because extending it I get like you’re having
more surface area which then you can look deeper and more into it but having
an angled game board it doesn’t matter what angle it is because it’s all about
we can’t even tell if you angle this it just it it’s seamless
all you do is just get more vertical it’s it’s a little difficult sometimes
to understand what you’re gonna see when you look at this system a lot of people
are like oh you can never project anything above the game board which is
not true at all you know us sitting here playing a game
side by side I can have objects like this tall off of the game board and it’s
great it’s a great experience all about the angle that you’re looking at the
game board form kind of like you know at a 3d movie things are still coming out
at you both in and out but because it’s wide enough that the angle works and you
can have sterile images that far apart yeah yeah so on this system you can have
like I can’t see anything taller than that but I can see stuff this like in
the center of the game board it’s like this tall easily but if you had a
requirement where I wanted to have something really tall I just took the
the pro board up so there’s a sweet spot in terms of where objects are placed
yeah right in front of my front which tends to be where you’re looking anyway
yeah and what’s funny is people are like well you know it’s gonna clip on the
backside sometimes and it turns out like we play video games all the time that
clip 3d stuff in the back and it’s like it it just works like your brains like
oh yeah that just went off the screen and it feels right well in terms of
sweet spot – one of the things we mentioned earlier is that accommodation
verges I was able to hold a cube up focus on that cube and then focus on
other cubes on the game board and go back look for and go back and forth
immediately you’re nothing that’s changing the rendering what is it about
this projection system the retro reflective material that allows for that
to feel real natural well it’s a combination things it’s kind of
complicated but it’s the aperture of our projectors which send out really tight
ray bundles and then also retroreflector these ray bundles will be going out in
like a cone and it’ll be focusing but since retroreflector brings everything
back again it refocuses them and it brings them through these intersection
points which are correct so it’s like we’re kind of through a clever optical
trick we’re making an entire leg field display without having to compute it and
that works specifically for what you’ve experienced the sweet spot for
tabletop yeah you are at that you know within certain number of meters and like
as close as you want those no minimum focus distance at GenCon we actually had
people like bumping their head into the table like Oh careful wait when you were
first experiment with this I mean people imagine putting this everywhere yeah
right that technically what good work yeah oh yeah yeah we are Maine kid it’s
gonna be like something that you can deploy in the home easy just flip the
game board open slip the glasses on grab your wand and start playing but yeah
there’s some folks that are taking our system and doing really crazy things
like huge domes for location-based experiences so you know all your friends
in this dome you can see each other have a really wide field of you you know and
sharing this virtual experience but again having the parameters of tabletop
that allow the developer to work yeah they all get the same board it’s a
language that exists if you understand your four sides you can do multiple
players so what are some of the game experiences oh well we’re really excited
for our announcement hopefully this is going out after our Kickstarter and
fantasy grounds is coming to our system so for the DeeDee Pathfinder crowd
they’re gonna love this so you can have your whole virtual world laid out in
fantasy grounds the DM gets their own unique view into the space the players
sitting around the table don’t get to see what’s coming until the DM triggers
so the the DM will be able to do things like an eerie mist settles in the valley
and that’s they wave their hand over and eerie mist will come out of their hand
and it’ll be revealed this fog of war will be lifted to the other players and
if your friends can’t login or can’t join you around the table they can login
and you can connect your game board to their game board and so in the DM waves
there and over they would see a virtual representation of the hand and the eerie
fog you know they do a trigger and the monster Springs out because that’s true
asymmetrical play but yeah people are using the same game board so the game
knows there’s gonna be someone in that position it actually doesn’t really
matter where they’re sitting it as long as not rendered properly for them and
some of you can be playing with someone yeah so the DM like could see like well
here’s a trigger point and there could be stats and there could be a monster
sitting here waiting to jump out and the other players are oblivious it until
that moment you trigger it so it’s gonna be great for storytelling so we’re super
excited about that partnership what’s also exciting about that is their system
is truly cross-platform so if your friends don’t have a tilt 5 system yet
you can play on your PC or tablet and still participate there’s the base the
underlying technology and those systems are all the same yeah yeah roll they
bring your ass they have adventure packs that you can download you can bring in
3d assets and work with them yeah that’s super exciting so that’s the D&D crowd
we have a bunch of action games that are going to be shipping with the system so
we have a bunch that we have which are kind of party action games so you have
your friends sitting around the table you have shooting blocks and trying to
grief each other and and do head to head then we have some solo play experience
we have this racing game that’s a very Mario Kart like and then we also it has
a battle arena mode which is really fun so you’re all in this arena it’s got
these spinning discs and stuff and you try to shove your friend’s car it’s kind
of a rocket League like very cool yeah and these are all essentially just like
PC games or a mobile game because you can plug this into yeah Bowl device for
PC or yeah many of them will be on PC and Android devices they just go to
Google Play or you go to steam and you download the game and and you’re good to
go we have a couple developers working on
true card games so locomotives is one of our partners that’s working with our
brand spanking new hardware brave souls and it’s looking really good you got a
little sneak peek of that and so that one’s like a Wild West theme where you
can have up just I think six players they can be either remote it’s truly
cross play also so you can play on your tablet or your PC or on the tilt five
and it’s really up to your developers to choose that they wanted to be really
graphically intensive something with a lot of network play or something that’s
very single-player focus that’s kind of up to up to them exactly
yeah we’re just we’re making a tool you know and we want to put enough you know
tools in the toolbox for you know the developers to make these rich
experiences so that’s why we thought like tracking
physical objects and hands and and cards like super critical you know great
tracking great field of you have them all target a game board so everyone has
the same experience no matter where they’re at you have all the input-output
you have speakers oh yeah we don’t even talk about eleven yeah and this is all
essentially whoever gets one of these can be a developer yeah yes
so each kid is a developer kit so our SDK is super simple like I’m just a chip
designer by trade and I’m writing unity demos every one of those like kind of
cheesy demos that I showed you today was just like a camera test demo that I put
together but our SDK you just take our plug-in drug it drag it on to the camera
and unity or unreal and then that gets your head tracking and then if you want
to use the wand you know the buttons and triggers is
just standard unity input and then for the six degrees of freedom you know it’s
it’s straightforward just hook that to your game object like you can get up and
going in like 10 15 minutes with our system so you got this software side the
Harvard sites being iterated on this is kind of like using production molds now
yeah injection molded plastic you’re getting down to your ID
everything’s getting finalized what’s the roadmap to getting this out
delivering on the cake for starters yeah our timeline is like we already have
some developers working with our kind of pre-production units we’re gonna have a
canary group or kind of an early beta test that’s like q1 yeah yeah it’s gonna
be a smaller group of people that love bug hunting this way goes and then
general availability like the the bulk of the Kickstarter units will go out q2
you know GDC time absolutely or a lot of helpers gonna be interested it really is
one of those technologies that you have to put on and yeah say that about a lot
of the RNA are technologies but like it is don’t know how good it looks until
you put it on so we’re really anxious about our Kickstarter video we’re super
proud about it we’re all but also wringing her hands because we did
compositing in it because to show the experience you kind of have to like
fudge it a little bit and like you have ly slow camp camera angles and you have
like stuff flying off the board which is true for the people that would have been
that are being it’s not necessarily true with him you
have a camera that’s like five degrees off the board or something yeah but what
we’re doing is we’re taking all of the composited shots and we’re doing like
side by sides of like here’s film through the glasses and here’s the
composited shot yeah so hopefully people will give us a little bit of a leeway
because we’re trying to sell the experience because it really is magical
when you try it and if you absolutely and having used a bunch of VR headsets
it really is it’s a different flavor of it and I think the tabletop aspect is
super super cool yeah yeah congratulations with the launch of the
Kickstarter I know it’s been a long time coming yeah and thank you so much for
having here to check it out thank you that was a really enjoyable conversation
for me and I want to thank Jared for being so open about the technology in
these tilt five glasses and what’s changed since they worked on cast a are
and the challenges going forward as they finalize the hardware and also work with
developers on the software when they’re ready to push this out consumers early
next year now did get a chance to use the latest prototype of tilt five for a
variety of tech demos that Jerry showed me and first thing on the hardware it’s
super light they’re pointing this as under 90 grams and while I didn’t have a
scale to to measure it it’s definitely the lightest of any VR or AR headset
I’ve used in recent memory it fit perfectly over my glasses of a plastic
like I said is looks polish and I think in the version I had there was a little
band that could tie so the tighten it to the back of my head and I wanted to do
that but once the glasses were on a couple things I immediately noticed one
it’s a are so I want to see the outside world in the outside world is very
visible not only the field of view to the outside world which some other era
headsets want to narrow that because they want to hide the fact that you
don’t have that much of an AR image either but also the clarity of the
outside world you know yes it is a little bit darker a little bit tinted
then looking through normal glass or plastic but I could interact with people
I was looking at things and I didn’t feel the need to take off those glasses
to read something to look at something you know it is how I think a
should be was also really comfortable was the imagery itself and all the
things that Gerry talked about in terms of the projectors being really bright
even though they’re running on a very low power because the retro-reflective
materials balancing that light directly back into your eye the imagery of the
augmented objects was a vibrant was colorful it was sharp and also had a
really nice frame rate and field of view I did some tests where I moved my head
left and right so I was going beyond the view of the game board and even when I
was looking off to the side of my head I could see the game board
I actually still see the images so that 110 degree field of view for that those
projectors really helps and really is a different type of AR experience if
you’ve used something like a magic leap or a hololens the smoothness of the
motion of the the graphics and the objects that were on the board in the
board or floating above the board also made this really comfortable I know
Jerry quoted about a hundred and eighty frames per second for the projectors and
I’m not sure if that’s alternating frames Nighy on each eye or if they’re
actually 180 on both projectors but regardless the imagery was was super
smooth and I after I interviewed her she did give me that demo again where she
crashed unity and the image was locked in place I almost couldn’t tell that
there weren’t any additional frames I didn’t notice any drop frames at once
that reprojection engine they have inside the hardware to do some
essentially a time warp and space warp and skewing of the images works now of
course if I move my head dramatically a frame or did some crazy lateral movement
did look skewed and it wouldn’t look right so you’re never gonna be running
games at like five frames a second but I can imagine if you’re running at 30fps
or even slightly lower the graphics will still look rock-solid and the images
will still be locked and will never feel like they’re not a part of the real
world even though there’s projected images something I was also truly
impressed by is that seemingly the tilt five technology
solves for the accommodation virgins conflict the idea that these objects
wherever they may look like they’re floating in space will be in focus as
you focus your eyes on them not just because of the stereo overlap I was
looking at these cubes floating above the tabletop surface and as well as
those spread around and really move my head around them and shifting my eyes
focus from one cue to the next and everything looked like they were in the
right place it’s an AR experience that you don’t get out of waveguide bass
headsets right now and I’m still trying to wrap my head around exactly what type
of optical trick allows for this to happen because they’re not generating a
light field they don’t have that type of rendering power on the system but
apparently it’s because of the way the projector it’s shoot all those rays of
light onto this micro surface of this retro reflective material and the way it
bounces back in a way the combine at different planes of focus allows you to
then shift and then see the objects at those variable frames in that sweet spot
which they’ve designed for it was really cool once one of those things that you
really need to try and it’s a thing that you don’t even think about because
things look like the way they’re supposed to in the real world at least
in the right distances as the way they’re supposed to
now the compute stuff is also interesting the demos I saw running off
of a laptop and as Jerry mentioned you could run multiple tilt five headsets
off of a single laptop just based on how much processing power you have on the
laptop what frame rate you want to run the games at and also how many USB ports
it’ll just be USB see right into the headset and so the benefits are of
course you’d only need one computer really to do multiple players it’s gonna
be really open they’re not doing their own version of an app store just
download the games from Steam or the Google Play Store and also the games can
run at different resolutions and then some of the demos I noticed that yes
some of them weren’t necessarily running at native resolution I could see the
tell-tale signs of some anti-alias saying some of the jaggies but again the
movement was so smooth and then some of the games that did look like they’re
running at native resolution that 720p per I was really sharp in terms of the
games the demo I did try were representative of some of
what they’re going for so there was one that was like a marble rolling down a
core so it was really cool to be able to kind of look into the course and even
put some of the retro-reflective material up and see beyond the tabletop
and see how much they’re actually rendering their example of a tower
defense game where I then it was able to throw some tracking markers with QR
codes and then that use that to activate or point with the wand controller and
activate things on that play field and one of the cool things is because there
is computer vision there’s optical tracking of markers they could also do
printing of the markers that’s transparent are not visible to the naked
eye the visible spectrum so it can be something like a CCG car you know a
magic card but underneath it there’s a layer of a QR code and so you’re getting
real true mixed reality with things that play in the real world but also work in
that salt 5 system their partnership with fantasy grounds I think has a lot
of potential because that’s an established platform with licenses for
RPGs like Dungeons & Dragons and so there’s an existing player base so the
existing software stack and it’s cross platform so people with tilt 5 will be
able to play with people who are not in the same room with them who may have
tilt 5 sets or may just be playing on a traditional PC or tablet and so that’s
gonna potentially expand the user base and I think it’s really smart of them
not to design an air system to try to solve for a are everywhere you know with
the complexities of SLAM and mapping and optics that work outside and inside
they’re trying to solve for a our designer a our system with these
parameters of it being a tabletop gaming system or video traditional video gaming
system I would love to play a Diablo like RPG or a gauntlet like RPG on this
2×2 game board and look deep into a spiral staircase and and play with
friends in the room or not in the room I can’t wait for the system to come out
there’s so me a little time before they actually get this out to backers and so
they have time to work on those software experiences working with their dev
partners it’s a great a good launch suite
games and experiences and also to refine that Hardware and further I think
they’re still tweaking I am you and the software related to that but I really am
excited for this and can’t wait to play this when this comes out and I’ll be
backing their Kickstarter and looking forward to that release and if you have
questions about it please feel free to post in the comments below but thanks
for watching we’ll be back next time with more coverage of VR and AR
technologies and I’ll see you next time

100 thoughts on “Hands-On with Tilt Five Augmented Reality Tabletop Gaming!

Leave a Reply

Your email address will not be published. Required fields are marked *