Like season 2 of Stranger Things, Coffee and Code is adding some new characters. One of those new characters is me, Chris. I bring some new skills which will help to round out the Coffee and Code team. For example, while completing a masters degree I worked with a computer vision algorithm called seam carving. Seam carving is a great algorithm to demonstrate how the team’s capabilities are growing since what it does is immediately visible. I’m going to tell you a little about how seam carving works and give some examples from the latest season of Stranger Things 2. There should be no spoilers here, but we will be working with images from the latest season, so you may want to turn back now if you have not watched the entire season yet.
Seam carving is a technique for adding or removing portions of an image without changing the main elements of that image. The devil is always in the details, but from a high level the process is pretty basic. The algorithm analyzes an image to determine which areas are less interesting. Then, it carves seams through those areas in ways which are hopefully not noticeable. Easy peasy.
We were thinking of interesting ways to demonstrate seam carving, and then we started watching Stranger Things 2. We love Stranger Things – well, we love most things about it. One of the few things about the current season of the Netflix hit that we would change is the number of blatant product placements. We get that they probably had to come up with extra funds to bring on Sean Astin for such a large role, but it often seems that entire scenes were built around product placements. So, Jon thought that we could fix this little flaw in our favorite show by carving out all of the product placements.
We expect seam carving to do a fairly good job of removing items from basic images, but also expect it to fall down quickly when given more complex scenes. Let’s consider a somewhat straight forward example.
When we run the image above through a seam carving algorithm we can bring the group closer together. The image has some distortions, but all of the important parts have been pretty well preserved.
If, however, we run a more complicated scene through the algorithm, the results are not that great. Let’s consider the scene where Steve and Nancy visit Barbara’s parents.
The algorithm preserved the details of the product placement and it made the people look like they were attacked by Demodogs. That’s the exact opposite of what we want to have happen.
The reason for the poor performance is that the algorithm thinks that all the high contrasting details present in the middle of the scene must be more important than the less detailed areas near the edges of the image. Of course, that was probably a large part of the intention of this scene — to prominently feature the KFC bucket meal. According to our unbiased algorithm, KFC got its money’s worth out of this scene.
The performance can be greatly improved if we give this algorithm some hints. It needs its paddles! We could add an object detection algorithm to find all the known objects in the scene, and then classify each for preservation or removal. But, for now, we just will manually mark which portions of the image to remove and which ones to preserve. Below is the result when we marked the KFC bucket for removal and marked the people to be saved.
While we were thinking of ways we could “improve” Stranger Things 2, we talked about removing some of the new characters.
If we were upset that Max beat Dustin’s Dig Dug record, we could tell the algorithm to scrub her out of the scene.
We didn’t got too much further down this path, especially after the episode where Max … well, we’re not going to spoil that for you. The next steps towards automatic product placement removal would be to add in some sort of object detection, perhaps using a neural net trained on product logos from the 80s. To remove characters we could add facial recognition after the object detection step. But, that will have to wait for another time, we need to go finish watching season 2.
Images copyright of Netflix and the awesome group from Stranger Things.
Originally published on the Coffee and Code blog:
Stranger Things at Coffee and Code
