P5.js as an Art Medium

This is an artist page by Kristine Fernandez. This project was part of the 2023 CATS+ Computer Club cohort.

My intent during the CATS+ residency was to provide coding support for the residents to support their work. At the end of the day, p5.js was at the root of all of the output that I had a hand in.

p5.js?

p5 is a javascript library used for creative coding, created to make programming art accessible to more people whether you have a coding background or not. I’m drawn to p5 in particular because while the different methods are relatively simple to understand, the output can be really complex and interesting. I like that p5 can feel more like a natural language extension where I can focus on what I’m creating instead of getting bogged down by some of the walls that typical programming can run into (fighting libraries, compiling issues, weird gotchas).

I had read a bit about flow state (e.g. related to the Japanese concept of ikigai or Mihaly Csikszentmihalyi’s work on flow in relation to happiness) and coding is one of the mechanisms where I’m able to feel that blissful feeling of reaching a flow state. Developing in p5 in particular, allows me to find that flow state more frequently because of its ease of use.

Some of the work that I’ve done in my early p5 experimentation mimics this flow state that I feel in the act of coding, typically creating visuals that are pleasant and often tied to sine and cosine functions which give the visuals a wavelike repetition.

Subsequent work has been to use p5 more as a tool to make a concept a reality and that is how I approached the work within this residency.

In Action

At its base, p5 is a drawing library (e.g. draw a square, rotate it, draw a cube and move it based on user input); however, because of its robust tooling, there are many other ways that p5 can be used for an interactive or dynamic piece that ended up being well-suited for everything we needed.

Stop Motion Command Center / Progress Bar

Sara Roma needed a panel that the audience could interact with to affect her stop motion piece. During one of our cohort discussions, I mentioned that progress bars could be programmed at the discretion of the programmer who built it so they can give us a flawed perception of progress. From this thought, Sara conceptualized the idea of a progress bar with a mind of its own so the audience might think that they are starting to understand what impacts the progress percentage but actually it jumps around somewhat arbitrarily.

https://fernakr.github.io/moha-cats/stop-motion/?duration=0.25 (To start, tap the spacebar.)

Progress

The progress bar is the primary p5 portion of this piece. It was surprisingly difficult to create the seemingly random increasing and decreasing of progress. I created some randomized inputs that determined whether the progress bar would increase or decrease; but I also wanted the duration parameter to control how many minutes it took the progress bar to reach completion, which resulted in a mathematical quandary where the probability of increasing, decreasing or staying the same was based off whether the progress bar was greater than or less than where it should be based off the intended duration.

Commands

See Sara Roma’s wiki for more information about how the commands affected their performance. My main intent here was to create a visually engaging piece that invited user interaction while also giving Sara enough context to how long each action had remaining.

Glitch Disconnect

When the progress bar reached 100%, I really wanted to create a visual that surprised the audience so I came up with this idea of the screen glitching out and “disconnecting” to really sell this idea of the panel being functional outside our normal idea of what a web browser is capable of. To do this I used a javascript library to take a screenshot of its current state, inserted that screenshot into the p5 canvas, then used p5 <vertex shaders to distort that image output.

Video

The source is a compressed version of the dance video from a workshop led by Itai Amor. I ran the video through p5 and experimented with shader effects and an AI-generated soundtrack.

https://fernakr.github.io/moha-cats/video/ (To cycle through effects, click the video.)

Shader Effects

Our robotic movements in the videos and the video compression created some really interesting visual artifacts that I thought would be really interesting to run through some shaders, especially ones that could get affected with realtime user input like mouse movement or just the flow of time.

Sound

AI is a very prominent topic in the art world right now, and I won’t get too deep into it, but I did see a website called Melobytes that uses AI to create all sorts of outputs based off all sorts of inputs so I thought why not see what happens when you run the video through the AI. The “music” in the background is the output of running the compressed video through its video-to-song service. I did have to lower its tolerance quite a bit so I’m not sure if it really had an extremely close tie to the visual, but it did produce some eerie sounds.

The Archive

During the course of the residency, we knew we wanted to document our experience so we collected all our notes (both physical and digital) and put them on display during the showcase. Serena had the idea to create a gradient wall that incorporated the physical-to-digital manifestations of our work together. I knew I could do the digital portion in p5 so that became my task.

Giphy: https://fernakr.github.io/moha-cats/archive-giphy

This was the original MVP for creating the general look and feel of the archive wall. I knew there was going to be video as an option, so I figured Giphy API would be a good first pass at testing the concept.

Cats+: https://fernakr.github.io/moha-cats/archive/

Once we collected all of our digital artifacts, I updated the visual a bit to give the artifacts a bit more depth and layering for more visual interest and also incorporated a little easter egg homage to Berry Chair who was an honorary resident (s/o to Kellyn for that idea!).

Appendix

Shaders

Shaders are code that is run on the GPU and used for things like post-processing filters e.g. they can be used to grayscale or distort your image or video. p5 uses OpenGL Shading Language (GLSL) but there are shaders in other platforms like visual effects or in javascript libraries like ThreeJS.

While p5 is code that draws onto the canvas, shader code is run on every pixel of the canvas. So a vertex shader is a shader that affects the position of each pixel. This is good for distortion effects like this slitscan example. Fragment shaders on the other hand affects the color of each pixel. So this is good for any color-based effects like this threshold example.