Hyper Himalayan

Showcase

ITP/IMA Winter Show 2018

Tools

camera, ml5.js, p5.js

Role

Designer, Developer

Brief

Hyper Himalayan is my attempt to visually reconstruct the world - it is an automated image editor that uses a machine learning algorithm called YOLO to assign a personal 5-color palette to any given image.

Ideation

This project originated from a Visual Language class assignment at NYU ITP - we were asked to come up with our own color palette. Bearing this in mind, on a lovely Sunday afternoon of tons of sunshine, I wandered around the Rubin Museum of Arts with a friend and bumped into some magnificent pieces of Himalayan arts. As someone previously having some but not much exposure to Tibetan and Hindu culture, it was quite a fascinating and eye-opening visit. Here are just a few examples of the beautiful pieces:

My favorite among them is this smiling deity portrait shown on the right, which in my opinion, skillfully stroke a balances between peacefulness and holiness. The color palette used here is classic in Himalayan arts: black, red, and orange with two complementary shades of brown.

Though deeply touched, instead of simply basing my palette on this color scheme, I decided to pick something completely different – colors that are nowhere possible to show up in any Himalayan pieces in the reality. After a few experiments, I came up with the following purple-based palette with a flashy green highlight.

And with this palette, I started to explore how the world around me could look differently - for instance, I wished there was something that could help me transform photographs I love into this anti-classic theme. So, I decided to make my own image editor to do the work.

Prototype

My first prototype was based on color comparison - the idea was basically to compare the source image’s color, pixel by pixel, to each color in the palette.

Two color distance algorithms – Euclidean, and CIE76 - were used to determine which palette color is “closest” to a given image color. Then, the editor would replace the color of a source image’s pixel with its closest color in the palette.

Below were six compositions produced by this prototype editor: they were famous graffiti walls I visited in six different cities across the world: Guangzhou, Hongkong, Shanghai, Beijing, Penang, and New York.

While the color scheme of the photos were indeed rendered with the designated palette, a problem was identified: the visual principle of color was violated, as seen in the leakage of the highted green color.

This was caused by the "unbiased" color replacement induced by the algorithm. Each pixel, whether they were part of the prominent or the peripheral features in the image, were replaced using the same method. In this case, replacedments occused outside of the major visual element in an image became distractions, especially when they were replaced with a highlight color. In other words, the editor would need to address the following problem:

How could we restrain the scope of color replacement, so that only the prominent features were highlighted, while leaving the 5-color assignment still functional?

After some research, I found that a machine learning algorithm called YOLO (You Only Look Once) in ml5.js can be helpful under this scenario. It is a network that could divide an image into regions and predict bounding boxes and probabilities for each region. Thus, in my second prototype, I took advantage of these bounding boxes produced by YOLO around detected objects in the image, and only applied the necessary color replacements accordingly.

Used YOLO to identify regions worth highliting.

Performed highlight & primary color replacement within those regions, and performed secondary color replacement on other regions.

In this way, the 5-color palette assignment would be conducted in a much more organized and meaningful way, and thereby fulfilled my intension for visually reconstruct the world as mentioned at the beginning.

ITP/IMA Winter Show 2018

This project with a modified color palette (highlight color changed to yellow) was selected and presented as part of the Visual Language Exibition at ITP/IMA Winter Show 2018. (photo credit: Nianqi Zhang)