Audio Visualiser

Last June we had another of our famous Coghack™️ hack days here at Cogapp. The theme this time around was “Freeform” which meant we could do whatever we wanted!

Tristan had the idea to create an application which would generate some kind of a visualisation to accompany a piece of audio like poetry or music, and this visualisation should be influenced in some way by the audio itself, with an optional bonus that we could display the lyrics/words to the song/poem, synced up to the music to create a karaoke-style experience as well.

The inspiration from this came from some conversations with staff from the Woodbury Poetry Room at Harvard Library about what sort of thing users would like to see while listening to recordings of poetry. Coupled with dim memories of the wealth of Winamp visualisations that used to play while people listened to music around the turn of the millenium, and how it was high time that we revisited the idea of sound-reactive animations.

We decided that the main app would be built with React, as we could quickly build the app interface with this, and then to try out p5js as the engine for the visualisation.

p5js is an open-source JS library for creating art with code, which aims to make doing so accessible and friendly! It seemed like a good fit for this project as we wanted to create interesting visualisations within a short space of time. [P.S. from Emily: If you’re interested in learning about p5js, I really recommend the YouTube channel The Coding Train where Daniel Shiffman uploads super fun tutorials on making things with p5js. I particularly love their Nature of Code series, which discusses creating code inspired by natural processes (think flocking, fractals, physics simulations, etc.)].

p5js also includes an extension called Sound which adds a bunch of extra functionality on to p5js to allow it to interact with sound files, and particularly of interest to us was the FFT object, which analyses the audio and opens up ways of returning data about it via methods like “getEnergy” or “getCentroid”. This meant we could do both the audio analysis and the visualisation with one library.

With the base React app in place and p5js up-and-running, we were free to experiment with different sound-analysation methods and visualisation ideas we had and we ended up with a few choices. You can test out the app yourself here: https://cogapplabs.github.io/audio-visualiser, while we describe the ideas behind the modes:

Level Meter mode: This was the first visualisation we added, which was copied from this example on the p5.js website, to get a feel for how an FFT visualisation could work and set up the basis to build other visualisations from.

Zen mode: Tristan was keen on a visualisation mode which would just be a solid black screen, which we think is a nice way to focus on the audio and additionally, a nice antidote to the constant barrage of information we receive daily via our screens (if going outside is not your thing perhaps…). This was the first custom mode we were able to complete. Recommended if you feel a little overwhelmed.

Circle mode: Emily had recently watched a film by Jordan Belson called Samadhi from 1967, which features a trippy animation of a circle in the centre of the screen which morphs and changes colour to a somewhat unsettling soundtrack, and was subsequently inspired to create something similar as a visualisation mode. This uses the energy of the audio to inform colour changes, plus some morphing of the shape itself. Honestly, it got to a point where there were so many layers of effects that she may no longer fully understand it herself… NB: May slow your computer down!

Serene mode: Serene mode was Tristan’s next step up from Zen mode, in which the screen is filled with one colour which updates based on the audio’s energy-plus-centroid (or something along those lines…) This creates a simple but effective (and serene!) rainbow visualisation. Goes particularly well with the poetry. (Try the “Gaps” audio file on the app).

Cursor special: In the “hack” spirit, we were trying out Cursor to see if it could help us build some of the app quicker than we would otherwise have been able to, given that we only had one day to complete our project. If you don’t know, Cursor is an IDE with an AI chat built in, which allows it to look at files within your project and give you code suggestions based on the code you already have, which can be very useful. At one point, however, it noticed that the already-complete “Zen mode” didn’t actually do anything (which was the point) and although we’d asked it to do something entirely unrelated, it decided to sneak in an update to Zen mode and created its own visualisation for it (super useful, thanks Cursor! /s). The visualisation Cursor created features a circle which “explodes” along with the audio. We think it’s based on an analysis of the frequencies in the clip. We obviously had to then fix Zen mode again, but we decided to make this its own mode as it had gone to all the trouble of creating it in the first place…

In all cases the visualiser works in full screen mode as we wanted it to be as immersive as possible, and we also added the ability to have a timed caption file, as well as a full text listing, so that people are able to read along with the poems as they listen to them.

You can try out the different interfaces using the online demo of our hack. Or feel free to check out the source code on Github.