Where You Are by Dave Klinger

Where You Are by Forgetter, from the 2019 album Parts of Anything

Animation by Ian McDermott and David Klinger
Mixed by Matt Schimelfenig and Brian McTear
Mastered by Paul Hammond.

© 2021 All Rights Reserved

In 2019, David Klinger asked if I would be interested in creating a music video for a song of my choosing from his new album Parts of Everything. I quickly agreed and listened to it several times over, finding myself transported to places both familiar and foreign. The album carries its listener on waves of mellotronic flourishes, warm windy tones, and jazzy interludes that would make Moondog salivate. I eventually landed on the Where you Are. Its slow, dreamy tones made me feel like I was on the floor of an ocean, and after playing it on an endless repeat, I had a vision of Dave floating up, isolated and suspended in the sea.

Dave’s visual work has a hand-made aesthetic that he wanted to maintain, so the challenge for me as a digitally-based artist was to use my tools covertly. He showed me a bunch of watercolors that he had painted, and we decided that instead of me being the sole animator, we would turn the project into a collaboration—Dave would hand-paint all of the watercolor elements in movement cycles and I would pull them into a 3D virtual world in After Effects, poking and prodded them with transformations, distortions, and particle systems. 

The watercolors were there to set the scene, but we still needed the video’s focal point, Dave. Myself having a fondness for rotoscoping, it seemed like the natural choice. The question then became: how to rotoscope the video footage while matching Dave’s watercolor aesthetic? I didn’t want to have him paint about 2500 watercolors for each frame of the video footage, so the solution was to automate it with technology. I started experimenting with PixelFlow, an OpenGL particle system library in Processing that can take video and give it liquid or vapor-like effects. It wasn’t quite working for the footage of Dave floating up throughout the video, coming across as too realistic and detailed to match the watercolors—some of it did, however, get used for the subtle closeups of Dave in the background, overlaid on some watercolor washes. 

After about a year of experimentation and not really getting anywhere satisfactory, I came across the open source rotoscopy application, EbSynth, discovering it through a tutorial by animator and comedian Joel Haver. EbSynth allows an animator to batch rotoscope a video into whatever aesthetic they want, without having to hand paint every frame. Essentially, the user pulls select frames from the video into photoshop, alters or traces them, and then uses them as keyframes guiding EBSynth to apply the aesthetic to every frame. The application using a technique called non-parametric texture synthesis, which is similar to a GAN style transfer, but doesn’t use any AI. After altering the footage in Photoshop and running it through EBSynth, the aesthetic looked right, but the motion was too smooth, making it appear uncanny with the hand-painted elements. 

To overcome this, I created five different renderings in EbSynth using different keyframes, then spliced them together, making every 5th frame of each visible through a custom After Effects expression. With a little more After Effects finesse—posterizing time, subtle transform and distort wiggles, color correction—the footage finally fit.  A summary of the process is illustrated below.

At the outset of the project, I told Dave the video could take a while, he was okay with it knowing the artistic process intimately himself. I don’t think he expected it to take nearly two years, but in doing so proved to be one of the most patient human beings I have ever known. In the end I’m thankful it took a while, because if it hadn’t, the video would not look so good and I would have learned a lot less.