For Halloween 2020, I decided that I wanted to try and scare the life out of the people who pass by my flat. But, how best to do that? With a Halloween hologram featuring yours truly, a Raspberry Pi, and Tensorflow PoseNet.js, of course!
Here's the effect you can create:
So, how does it work?
Making your own hologram is pretty easy. The technique is called "Pepper's Ghost" and it's an effect that the Victorians came up with to create ghostly images in theatres. The same effect is still used today, Disney seem to be big fans of the technology, although I dare say they've added some of their own magic to the effect.
How to build your own
Building your own frame is easy, I managed to construct mine in about an hour with some wood and screws. It looks like this:
To build your own all you need is:
- A television
- Some timber
- A picture frame as big as your television
The general idea is to have a pane of a transparent, reflective surface at a 45 degree angle from the thing you want to create the illusion of. A photo frame is great for this as they come in a variety of sizes and are fairly cheap and easy to get a hold of.
My frame places the glass at a 45 degree angle above a television screen which is showing a video. If you're playing just any video, the reflection will look just like a ghostly television set playing in the sky. But, if you have a black background behind some other object, it'll appear to float in the sky behind the frame!
I actually built a much smaller version of this frame for phones back in 2014, so I had a demo application of a floating companion cube sitting on a server. Once I'd built the frame, I ran the app to test out the effect.
Not bad for a 6 year old WebGL application, eh?
Making the videos
So, now I had the frame I needed to display something on it. Something truly terrifying; something that could make a fearless army on a war march take one look at it and go "You know what, we're good, let's go home.". So naturally, I decided to use my face.
Not just my regular old face of course, this needed to be a ghost face - a face which is the very image of death (I can already hear you saying "but Sean, that is your regular face!"...). I decided disembodies and pale as the moon would be a good look for ghost-me, so I set about filming some footage for the display.
Fortunately, by virtue of my housemate and I both having degrees in media, we have not one, but two(!) green screens. This made life way easier for me. I could set up a green background and wear a green screen around the rest of my body and just be silly/spooky in front of a camera.
Here's what that looks like:
And this is what you end up with:
The process is really simple and didn't require any fancy video editing software, you can just whack it in iMovie like I did and get the effect you need.
And that's pretty much it! At this point you have the frame and the videos, you just need load them onto a computer and play them!
But... videos playing on a loop isn't very scary - how can we time our playback at the best time to put someone an inch closer to a mild coronary?
Perfect timing with Posenet.js
A few years ago, trying to tell a computer when a person was in a video or picture was fairly difficult, but these days modern AI has become so accessible and powerful that it's no more complex then adding a script tag to a webpage. I wanted to be able to start playing a video as a person (not a car or a fox) was passing by my flat, so I started to look into AI models that I could use to trigger the playback of the recorded videos in a web page that would be displayed on the Pepper's Ghost frame.
This is where PoseNet.js comes is. PoseNet is a great little bit of AI that can detect when a person is in a video or image. Perfect for my use case, all I had to do was add it to my web app with all the videos in it and feed it a webcam feed that was pointed out of my window.
Every 5 seconds, PoseNet checks to see if anybody is passing by my window, and if they are it plays one of the 11 ghost videos. All of this runs on a Rapsberry Pi 4, which is incredible when you think about it - I can run AI in a web browser on a $35 computer and use it to scare the heebie-jeebies out of people! You can check out the code here.
And that's how you can make a hologram to scare the life out of anybody who comes within 10 meters of your front door (not that many people are doing that during lockdown. Probably a good idea to not order take out for a while though).