Pixels and Cellular Automata
Link to Sketch: https://editor.p5js.org/nkumar23/sketches/WMqaK0_ts
Assignment: Create a 1 minute experience of color by manipulating image or video at the pixel level.
Idea and Inspiration:
When I worked at Coursera, I would often take courses from different university partners to “dogfood” the product and get a sense of what was working/not working in my user experience. One of those courses was Model Thinking from Scott Page at the University of Michigan. This course covers a number of approaches to modeling human and system behaviors. One memorable section uses cellular automata to show how individual preferences for how similar or dissimilar they would like their neighbors to be in their home neighborhood could lead to different levels of diversity and segregation in cities. This simulation popped in my mind as we discussed pixels in class; I had a hunch that we could use the outputs of each generation of a cellular automata simulation to control pixel characteristics (like color or brightness.)
After watched Dan Shiffman’s videos on Cellular Automata and Image Processing I learned just how common this idea was— in fact, it’s one of the foundational techniques in image processing!
After getting the Game of Life model working and connecting it to a single image, Lulu and I began experimenting with styling. Lulu found an interesting effect when she plugged in an image of a desert with a blue sky, and drew lines as cells changed state to produce something akin to rain. I plugged in a segregation map of Chicago — I traveled quite a bit to Chicago in the last few years and was struck by just how segregated the city was— and changed the styling to represent movement across the map until the cells all reached a state of stasis.
Descriptive Words:
Our descriptive words for the piece overall were:
Cycles
Nature
Rhythms
Unique
Obscured
Occupation
Our descriptive words for the sections were:
Life
Death
Stasis
The A-Minor Music Machine
So that this doesn’t get buried under the reflections below, here’s the sketch. I will update here when the version that works with Arduino is functional.
——
Within 10 minutes of posting this, I’ll probably be editing code again. This project, more than any up to this point, drove me insane and made me so happy.
At various points in the week, I grunted out loud to myself (and anyone in my general vicinity) as I failed to notice a missing “this.”; I enlisted a former software developer colleague to hop on the phone and talk me through questions late at night when office hours were no longer an option; I thought I’d reached a nice stopping place, then thought of one more feature, then looked up and another hour went by.
And in the process, my empathy for all the devs I know just increased exponentially from an already- empathetic place.
With the preamble aside— what was the assignment? What did I make?
The assignment this week was to use objects, classes, arrays of objects, and interactions between objects. I hadn’t yet used sound in a p5 sketch, so I made that a constraint for myself— do all of the above while using sound.
I got inspiration for this sketch from this example; after looking at it I wanted to see if I could replicate the effect on 4 different drum sounds on one canvas. I started by trying to get it working without any objects/classes, which you can see here. It worked! From there, I started to see a “sound” object coming together— something that had a type for each type of drum sound (kick, snare, etc). It would also be able to display itself and maybe analyze its amplitude. Simple, right?
The process of migrating this code to an object-oriented structure took a lot longer than I’d expected, and was the source of a lot of the grunting and requests for help. Ultimately, there were a couple of important lessons:
Understand variable scoping:
Global variables can be used.. um… globally.
When using something like p5.Amplitude this was a really important concept to internalize. p5.Amplitude measures the entire output audio unless otherwise specified. If p5.Amplitude is set globally, it’s not possible to setInput later on and have that used to, say, draw rectangle height like I do.
this.object is available for all object instances in the class
instantiating a variable inside a function within a class scopes that variable to just be available in the function
know when to use this vs a this.object
While debugging, isolate each action you are trying to perform independent of other actions.
Do this by finding or creating variables to console log in order to see whether the program in functioning as expected
Keep simplifying
I found myself, and still find myself, repeating some code with only small variations for particular cases. Instead of having a bunch of if statements to describe when to play a sound and the same set of if statements describe when to analyze a sound, maybe there could be a common variable that could pass to both .play and .analyze (in this case, this.sound.)
Pieces
I recorded the drum sounds, chords and bass notes in Logic Pro X and exported each sound as an mp3. I loaded those into p5 and used preLoad to make them available to the rest of the sketch.
I used the p5.Amplitude, setInput, and getLevel functions to get the amplitude of any given sound file. I could then pass that amplitude reading (done in the draw function to get it continuously as the file is played) to the rectangle height parameter to create the drum visualizations. Those are triggered by keystrokes.
The chords are stored in an array. The right and left arrows cycle either up or down the array, resetting if they reach the end of the array. When they are created, they get passed a random number between 1 and 6. That number is used to change the background. A few of the numbers change the background color. Others don’t. The number assigned to an instance of an object is random, so there’s no rhyme or reason to when the background changes other than that it only happens when chords are played.
There are some white dots that appear occasionally. These are an array of “Item” objects that get triggered from within the display function. These appear when a particular random number in the chord object appears when a kick drum has been played. In order to make this happen, I created a state variable to measure whether the kick drum has been played (but resets after it gets played). When the kick (and clap, same logic) has played and the random number coincided, these ellipses appear until the condition is no longer true.
The bass notes trigger the triangular visualization in a similar way.
All in all this has been a really rewarding project to really ramp up my understanding of a number of things. As an added bonus, it is a great foundation for working with serial communication in PComp: the variables controlled by keystrokes will be controlled by buttons and other things out in the real world!
Loops and Hoops
This week our ICM assignment focused on using loops to make our code express more with less. I also used the image and preLoad functions to bring in outside images, something I had not yet done. In the process, I got to start building a better intuition for how the different pieces of loops and if statements interact with each other, although I got a bit fixated on the images in my animation and have plenty left to explore about the nature of loops.
Take a look at the sketch here
The NBA season is fast approaching and Rockets fans (me included) are starting to get excited about the new James Harden and Russell Westbrook pairing. I made a silly animation/interaction to have some fun while playing around with loops and animation.
I used For Loops to generate a matrix of Draymond Green defenders on one side of the court while I used the same approach to if statements and incrementing that we used to bounce balls around the screen to animate James Harden and Russel Westbrook images. To add a little interactivity, I used the mousePressed function to both create a toggle state to create conditional “screens” after mouse clicks and to create a click counter, which let me make one screen only trigger after a certain number of clicks.
In the future, I’d like to see how I could have looped over an array of images to create a team of different players to be the defenders instead of 4 Draymonds. I also more than likely got lucky that this program worked— too many variables are named x and y inside of functions without intentionally thinking about variable scoping.
On to objects and functions!
CollaboCoding in P5
Last week we began playing with variables, introduced conditional statements and for loops, and overall added more to our bag of tricks. Our homework asked us to use some of these tools to create buttons or sliders that controlled some other element on the screen (without using buttons that we get for free from the browser or bootstrap-like tools.)
But really, this week was about learning how to make our code organized and commented well enough for another human to understand. So, I’ll reflect a bit about what worked and didn’t work so well while collaborating with a partner.
Our Approach
We decided to start our sketches individually rather than agreeing on a design before we’d had time to process the week’s lessons. Each of us took a couple of days to mock up a sketch of our own, then traded over the weekend to review and add to our partners’ sketch. On Monday/Tuesday before class, we chatted about our changes in-person and helped each other understand our additions or changes to each others’ code.
Drafts
In my initial draft, my intent was to create a set of buttons that mapped to the basic parts of a sentence — subject, verb, adjective, noun — and changed each of those corresponding parts of a sentence with each button press. In order to do this, I knew I’d need to work with an array of strings and use some logic to randomly select from those arrays, then call the results of that logic in the rest of my code. I also knew that I’d need to use the mousePressed() or mouseClicked() functions with conditionals that mapped to each rectangular button.
As I began to figure out the word choice logic, I created variables with different pieces of the process; one variable to create a list of words, another to randomly select an index from that list, another to translate and store that index as an actual word. When I needed to wire that up to buttons, however, I realized that I’d either need— or drastically improve the efficiency of my code— with a function specifically for all of this logic. This took me away from animation, but ultimately was a rewarding process!
My partner created this very cool looking animation. Her goal was to have the background change to the roll-over color of each circle when pressed. However, there was initially some odd behavior that kept the background blue under certain conditions. I made it my goal to debug this issue and get her sketch working as she intended.
Collaboration
Once I got the code in my sketch working the way I wanted it to in a simple, very lightly styled interface, I focused on writing comments that would make the code immediately understandable for my partner, without needing to physically explain anything. For the most part, it worked— my partner was able to add a ton of animation and styling to the next iteration of the sketch. However, I failed to make clear why I created a “quarterW” and “quarterH” variable, so she ended up writing her on logic for rectangle sizing using the built-in width and height variables.
When we met face-to-face to talk through our results, we discovered an unintended behavior of the “frame” in her additions to my sketch, which we were able to debug together. While our comments helped us get 90% of the way, it was helpful to debug together. We were also able to discuss the changes to her sketch that debugged her initial vision (here). The solution was fairly straightforward to implement: at the very end of the code, the part that created toggles for on1, on2, and on3, we needed to ensure that if one button was on, the other 2 were off. By showing her how console logging the states of each “on” variable allowed me to see the unintended behavior in her initial sketch when multiple variables were on at the same time, we were able to quickly understand the roots of the bug.
Overall, the most useful part of this exercise was having to pay close attention to how our code was organized and commented, which will continue to be important as we start adding more and more to our toolkit!
Shoutout to Beste Saylar, my heretofore anonymous partner :)
For clarity: NK Draft, Beste revision // Beste Draft, NK revision
Variables!
Last week, I drew a sketch of a hiker climbing some mountains (vaguely resembling the Grand Tetons) around sunset. As the very first exercise with p5js, we were encouraged to keep everything hard-coded to get a better feel for how different value ranges would express themselves on the screen.
This week, however, we’re having fun with variables!
So- I took last week’s drawing and made it more interactive (click link here to see).
Just like what I imagine so many assignments will be like at ITP, there are a number of elements of this sketch that I would like to do differently, if I had some more time. Here are a few examples:
1) Switch which components the mouse controls
When I initially started the assignment, I knew how to move the sun and sunset color slices, but didn’t yet figure out how to move the hiker— which is really a collection of lines, rather than a single function like fill() or ellipse(). Fill() and ellipse() are easier to parameterize with variables. I wound up using translate() for the hiker after I’d already coded the interactivity with the sun and background. I knew I wanted some elements to move based on time and others to move with mouse interaction. The sun/background moving with time and hiker moving with user control would have made more sense… but this is art school, so let’s just say I meant this as an attempt to invert our relationship with time and the sun. To give the user control over one of the things normally entirely out of our control: time… or something like that :)
2) Make the hiker descend upon hitting the window width + reflect around the y-axis
Currently, the hiker glides backwards down the mountain a good handful of seconds after getting to the width of the window. I couldn’t see immediately why my incrementing structure doesn’t work; with a few more minutes (or hours) I think I can figure this out, but alas, I don’t have them right now. Console.log-ging the x variable shows that the x variable reaches ~430 when the figure appears to leave the window frame, reaches 600 (the current window width) a few seconds later, then starts the descent. The y-variable logs as a negative number the entire time, even though the incrementing function should only reverse the sign when x > width or x <0.
From an aesthetic perspective, it looks strange to see the hiker walk down the mountain backwards. I’d like to have it flip across its y-axis when it turns around, but I’m not quite sure how to do so yet.
While the sketch isn’t perfect, it has definitely given me clear questions that I’d like to learn the answer to. Perhaps the next topics will lead to some more efficient and effective approaches.
On to For loops!
Creating a Static Drawing in p5js
Assignment Prompt: Create your own screen drawing: self-portrait, alien, monster, etc. Use 2D primitive shapes – arc(), curve(), ellipse(), line(), point(), quad(), rect(), triangle() – and basic color functions – background(), colorMode(), fill(), noFill(), noStroke(), stroke(). Remember to use createCanvas() to specify the dimensions of your window and wrap all of your code inside a setup() function
I started this assignment by reflecting about what I’d like to draw. Like so many assignments at ITP, from this early perspective at least, we had a lot of agency to take this in any direction. I recently came back from a trip to the Grand Tetons and Yellowstone National Park, so images of the outdoors are still fresh in my mind. After doing some research into the National Parks’ design system for Visual Language, I decided to make my own outdoor drawing.
I’m not very experienced in drawing, but regardless, with the assignment in mind I tried to keep the shaped in my sketch to those that I knew were possible with 2D primitives in p5. Part of my goal with the assignment was to familiarize myself with the basic drawing functions, including curves, before we moved on to more advanced concepts.
Along the way, my final image transformed. I started with a blue background and bright green grass— a sunny day, with trees and a river taking center stage. But when I looked at my screen after being away for a while, the brightness was a little jarring, and the river just didn’t look right. I tried varying the background to look more like a sunset— that looked better!
Then I started to move the sun around until it look natural. Behind the mountains or fully visible? Closer to the red part of the sunset or higher up? This was not a very scientific process— I looked at a couple sunset pictures but ultimately just went with my intuition.
After getting rid of the rivers (and my use of the curve function,) the screen looked a little too barren.
“Well, the assignment did mention a self-protrait as inspiration… How about I add a stick figure backpacker?”
This did the trick— now the image has a little more movement and a central character. I added some stars to the night sky emerging as the sun fades away.
The final image captured a little taste of the magic that I remember of the Tetons in the evening.
To see the code for this image, go to this link: https://editor.p5js.org/nkumar23/sketches/JujD9a_vr
OK Computing
As a general rule, when I consume a lot of something, I want to know how to make that thing to some baseline level of competence. Whether it’s food, music, literature or film, this principle has generally held. So much of my recent life has been connected to computing, computers, software, and the software industry-- and although I’ve started learning to code a number of times without really going all-in, I’ve never forced myself to sink or swim on a coding project. It’s time to change that.
I’m excited to make progress in ICM towards knowing how to code beyond a conceptual level. I’m excited to make mistakes and debug even when I want to stop. I’m excited to write functions, learn some of the weirdness of JavaScript-- and hopefully to be confident with my ability to learn more complex concepts and languages after this semester ends.
As someone who’s sat on the “non-technical” side of software companies, often in roles that hybridized design, product and business development, I’ve had to learn how to work with engineers and explain technical concepts without actually building them. After ITP, maybe even after this course, I’ll want to cross the line into the “technical.”
But I could have gone to a bootcamp if that’s all I wanted. Being at ITP means that I’ll have the space to wield code as another medium among many. In this course, I want to use code in conjunction with music, video, data and/or text. I want to keep my mind open to the specific projects I might undertake, but regardless of what I do, I’d like to make use of the freedom to be weird and explore the non-practical while outside the confines of the workplace.
I love making music. I love teaching. I love cooking. I’m a fan of music videos, animations, and informative, playful uses of data. Maybe some of these will come together with code this semester. But with each day, as we have stimulating conversations and rapidly add new skills and perspectives, I’m just as excited about creating something I wouldn’t have expected from my September 10, 2019 perspective as I am about anything I’m walking in the door with. Let’s go!