Haunted Museum
Showcase
NYU Tisch Halloween Show, Oct 2018
Tools
Arduino, ml5.js, p5.js, Projection, Photoshop, Webcam
Role
Lead Developer, Interaction Design, Fabrication
Brief
Haunted Museum is an AI installation created for the 2018 Halloween Musical, an immersive musical produced by the Graduate Music Theater Writing Program (GMTWP) at NYU Tisch. It is built for one of the themed rooms (called Twyp University Museum), and it will summon a ghost that can haunt and follow a museum visitor using a machine learning algorithm called PoseNet. It is a project co-created by me, Abi Muñoz and Briana Whyte Harris.
Ideation
The idea of this project originates from Briana, our project coordinator in GMTWP. She played the role as a meseum tour guide, and she hoped that we could build an environment for her tour performance that could give a hint of the evil spirit which served as the center of the Halloween storyline. Relating to museum theme, Abi and I brainstormed and came up with the idea to seal a ghost in one of the museum artifacts, and accitentally released it in the middle of Briana's tour.
Prototype
After the ideation, we built our first prototype to testify the concept regarding the following:
- what type of artifacts should we use to keep/release the ghost;
- what is the best way to visually present a ghost;
- how could we fit the ghost into the space in the meseum room;
- how do we engage visitors with the ghost
To make a projection that could pop up unexpectedly, we made a circuit with a distance sensor that would send a triggering single to the laptop if someone placed an object close enough to the sensor. Parallel to the sensor, we also put four groups of LEDs in series, in order to create an evil blinking eye artifact that would be triggered once the ghost was projected onto the wall.
In light of the overall 1950s aesthetics of the show, we created a ghost that adopts the Cthulhu image with a black and white body of an old-fashioned sculpture. To enage the musuem visitors with the ghost, we used a machine learning algorithm called PoseNet from ml5.js to track the visitor's motion with webcam, and dynamically adjusted the ghost position based on the visitor's movement.
User Testing
With the prototype, we ran several rounds of user testings in the museum room to figure out the optimal way to present the ghost in a way that could both fit into the entire space and yet surprise the audience:
Initially we played the ghost audio from the laptop along with the sketch. But during the user testing, we realized that we could take advantage of the space in the room for audio playback. It turned out to be much more unexpected and creepier when we played the background sound from the other end of the room via a networked computer. According to the audience, it really felt like the room was occupied by the ghost when the sound reverbed.
Based the size of the walls, the position of tables/chairs, the path that was designed for the audience to navigate inside the room, and the positions of the outlets (so that we could plug the projector and laptop to make the project works), we came up with a few combinatons of setup, and eventually decided to place the projector under a sculpture stool in the middle of the room, and casted the ghost to the wall from the sides.
One of the problems identified during the testing was that not many people recognized the Cthulhu (octopus) image, and they thought of it more of an animal rather than a ghost. So we replaced its upper part with a human head. Also, as the blinking eye effect was considered somewhat contradicting the 50s aesthetics, we removed the LEDs and put it under a portrait to make it less noticable.
NYU Tisch Halloween Show
Our piece was presented on the NYU Tisch Halloween Show 2018. It was well received and considered as one of the best installations in the event.
