Tearable Dreams

Sketch: https://editor.p5js.org/nkumar23/sketches/GoX7ueD-x

Github: https://github.com/nkumar23/tearcloth2D

Original Plans

I originally entered this final hoping to create a sketch that visualized the quality of my sleep and was controlled by a muscle tension sensor I had already built. I wanted to visualize the entirety of my night’s sleep as a continuous fabric slowly undulating, with holes ripped through it when I clenched my jaw -- detected by the sensor.

I drafted up an initial roadmap that looked like this at a high level (sparing you the detailed steps I created for myself):

1. Create the fabric using ToxicLibs and p5

   1. Adding forces and texture ripping to Shiffman’s example

      1. Start with mouse/key interactions for rip and forces (like wind/gravity)

2. Create a clenched/not-clenched classification model

   1. Collect and label sensor data for training

      1. Create simple interface in p5 to record clench/not-clenched with timestamp so that I can label sensor values

      2. Train ml5 neural net using labeled data

3. Link fabric ripping to sensor data

4. Run this in real time and capture video of animation during sleep

I headed into this project knowing that I’d most likely tackle chunks 1 and pieces of chunk 2, but likely not the entire thing. Along the way, I wound up focusing entirely on the cloth simulation and abandoned the rest of the project for the time being. I added some audio interaction to round out this project in a way that I thought was satisfying. 

I’ll describe the process behind building this cloth simulation, and what the next steps could look like from here.

Inspiration

As difficult as dreams are to remember, I have short, vivid memories of an undulating surface, slowly changing colors, that at times looks like a grid from the matrix, and at other times looks filled in. As I began thinking of how I might want to visualize my sleep, I went immediately to this sort of flowing fabric-like image.

Luckily, as always seems to be the case, there is a Shiffman example for cloth simulation! I also found this example of a tearable cloth in native Javascript.

Process

I started with Shiffman’s 2D cloth with gravity example. He did the legwork of importing the Toxiclibs library and aliasing some important components, like the gravity behavior.

Rip detection

From that starting point, I first wanted to figure out how to rip the cloth. To begin, I calculated the distance between a clicked mouse and a particle and console logged the particle coordinates when the distance was less than 10 (i.e. which particle was clicked?).

https://editor.p5js.org/nkumar23/sketches/4NsLvcVlO

Rip detection, but make it visual

Next, I wanted to see if these particles were where I expected them to be, and nowhere else. To do this, I displayed the particles rather than the springs and made them change color upon click.

https://editor.p5js.org/nkumar23/sketches/cz2XdRYfH

Spring removal

Then, I wanted to remove springs upon click -- which would “tear” the cloth. To do this, I had to add a reference to the spring within the particle class so that specific springs could be identified upon click.

I spliced spring connections and decided to stop displaying the spring upon click, which led to this aesthetic:

https://editor.p5js.org/nkumar23/sketches/-dtg2SCCy

That doesn’t really look like a cloth was torn! Where’s the fraying? This behavior happened because the springs were not removed from the physics environment-- they were just no longer displayed. 

Removing the springs required adding a bit more logic. Instead of adding springs in the draw loop, we now have the framework for adding/removing springs within the Spring class’ logic. 

In the draw loop, we check whether a spring is essentially marked for removal upon click and remove it via the remove() function created in the Spring class. Thanks Shiffman for the help with that logic! 

https://editor.p5js.org/nkumar23/sketches/uykP2GyE2

Adding wind + color

Now that the spring removal creates a more realistic cloth physics, I wanted to add another force to gravity - wind. I wanted to simulate wind blowing through the fabric, creating an undulating, gentle, constant motion. But I did not want the wind to blow at one consistent “speed” -- I wanted it to vary a bit, in a seemingly unpredictable way. Perlin noise could help with this. 

I brought in a “constant force behavior” from Toxiclibs and created an addWind() function that incorporated changes to the parameters of the constant force vector based on Perlin noise. 

Next, I wanted to add a similarly undulating change in color. As I looked at ways to use Perlin noise with color, I came across this tutorial from Gene Kogan that had exactly the kind of surreal effect I wanted. Here’s the implementation of everything up until now + wind and color:

https://editor.p5js.org/nkumar23/sketches/ugZc9Sry7

Adding sound

At this point I had a visualization that was pretty nice to look at and click on, but seemed like it could become even more satisfying with some feedback upon click-- maybe through sound! I added a piece of music I composed and a sound effect I made with Ableton Live -- the finishing touches. Check it out here -- the same as the sketch at the top.

https://editor.p5js.org/nkumar23/sketches/GoX7ueD-x

Next Steps

I would like to add some physical sensors to control parameters for this sketch — things like the way the wind blows, ways to stretch the fabric, rip it in the physical world, etc. I’m not wedded to using the muscle tension sensor anymore, though!

I’d also like to add more thoughtful sound interactions. Perhaps there are different interactions depending on when your rip the hole, where you rip it, how much of the fabric is left.

More broadly, this assignment made me want to explore physics libraries more. It is pretty impressive how nicely this cloth was modeled with Toxiclibs’ help; there’s a whole world of other physics library fun to be had.

Genetic Evolution Simulations and Athletic Performance

This week, we covered Genetic Evolution algorithms and surveyed a few approaches to designing simulations that implement this technique. Below, I’ll lay out a plan for a simulation in p5.js that takes inspiration from the improvement in athletic performance over the last 100+ years.

Scenario

Describe the scenario. What is the population? What is the environment? What is the "problem" or question? What is evolving? What are the goals?

When we watch athletes compete today, it is remarkable just how far they’re able to push the human body to achieve the feats they achieve during the games. Crowds at earlier athletic events were similarly mesmerized by the best athletes in the world during their eras. However, when we watch film of athletic competitions even 30-40 years ago, it often seems like we’re watching an amateur version of today’s sport. When we look at world record times for competitions like the 400m dash, we see a steady improvement over the years; most likely these are due to advances in technology, diet, strategy, training methods and other tools more readily available to modern athletes.

This steady march towards reaching an upper limit on athletic performance reminds me of a genetic evolution simulation. Thousands of athletes have tried to run around a track as fast as they could. Each generation of new athletes learns from the previous ones and finds little ways to improve. Over time, we see world records broken, little by little.

I would like to build a simulation that has objects try to “run” around a “track” as quickly as they can, under a number of constraints that loosely model those of real athletes. The viewer will be able to manually run each generation rather than having the simulation evolve as fast as possible, and we’ll record “world record” for the best individual “athlete” in that generation’s competition. We can display a record book on screen to see how the world records vary and eventually improve over time. We can also display the “stats” for the top n athletes to see how things like “strategy,” “technology,” and “muscle mass” change over time as the athletes improve. 

Phenotype and Genotype

What is the "thing" that is evolving? Describe it's phenotype (the "expression" of its virtual DNA) and describe how you encode its genotype (the data itself) into an array or some other data structure.

Athletes, represented by shapes on the screen, are the “things” that are evolving. The athletes will be physics objects that have a position, velocity and acceleration. They’ll also have other traits like “strategy” and “technology” that loosely model real world factors that can limit or increase their max speed and vary from generation to generation. Instead of thinking of technology/strategy as properties of the object, they could be thought of as forces that are applied to the object’s acceleration or velocity vector and vary from object to object in the system.

Strategy will control the efficiency of the path taken on the track. Each frame, the object will have to decide the angle of its trajectory. If the object zig-zags around the track, it will not be taking the most efficient route. Over time, the objects should learn to take the path with the least amount of wasted movements to get to the finish line. The code will need to be written in a way that the object only goes clockwise, rather than immediately going to the finish line to the objects’ left.

The objects will also have other constraints besides knowing the best route to run. They’ll a “technology” constraint — this will correspond to a max speed or max force property’s ceiling that applies to all athlete objects. Each athlete object’s specific technologies will allow it to reach a percentage of the ceiling — with some having “better” technology, or a higher chance of reaching the ceiling, and some will only reach, say, 50% of the ceiling, which means they will likely complete the race slower than athlete objects with better technology.

Athlete objects will also have different diets that behave similarly to technology. Diet could potentially control a “max force” constraint that affects acceleration, or could be a small velocity multiplication factor that works in conjunction with technology. This factor would apply to all objects, and each individual athlete object’s diet could vary to allow it to have a percentage of the max factor.

These properties encode the genotype and allow the athlete the potential to perform as well as their ceilings will allow them to perform. Over time, those ceilings will go up, which will allow individual athletes to go faster than the fastest in the previous generations.

The athletes’ phenotype, or expressed traits, will be their shapes and speeds of movement around the track. Perhaps diet can modulate size of the circle a bit and tech can change color or control some sort of blur effect.

Fitness Function

What is the fitness function? How do you score each element of the population?

Each generation, 20 athletes will compete. The fitness function will look for the elapsed time for the athlete to reach the finish line. Assume an oval track where the athletes begin at the top of the canvas and run clockwise as shown above. The pseudo-code to calculate the fitness function will be something like:

If athlete.pos.x equals finish.pos.x and  athlete.pos.y is between a range of finish.pos.y, trigger a function that will log the time elapsed since the start of the race and the point at which the if conditions are met..

Mutation and Crossover

Are there any special considerations to mutation and crossover to consider or would the standard approaches work?

Technology and diet ceilings should probably increase on some set interval -- maybe determined by user but set at a default of every 10 generations (like a decade). Crossover can continue in a standard way and so can mutation.

Sound in... My Apartment: Plans for a Quad experiment

The last couple of weeks have been truly bizarre. I was in college in 2008 during the Financial Crisis and thought it was one of the weirder and more momentous experiences I’d lived through. This moment has quickly leapfrogged 2008, and given how terrible this situation is for so many people, I’m very lucky to be in school yet again to participate in/observe how this all unfolds.

Still, it’s been difficult to focus on schoolwork during this first wave of changes to our lives, our world. Prior to going remote, I was very excited to experiment with a quad setup at school; moving from stereo to multichannel was what I signed up for! Going remote, worrying about family and friends, making major adjustments to lifestyle, and just reading the constantly shifting news has drained my motivation tank. So, this week’s assignment is basically a sketch, a set of plans for an experiment, but not what I would have liked to do in a more-normal world.

Idea:

I would like to make a p5 + tone.js sketch that functions like a spatialization test with a twist. I’ve never worked with quad, so this would be a helpful way to start to understand some of the “physics” of different audio setups. In short, the tool will play individual notes in the form of moving balls that, when played together, can form chords. The notes will play through different channels as they collide with the “speakers” on screen.

Here is a rough skeleton of the beginnings:

As I build this out, users will be able to:

• Arrange a multi-channel speaker setup on the canvas

  • The speakers will be rectangles with inputs to indicate the channel number

• Click to create notes visualized as balls that bounce around the screen

  • The balls carry a note that belongs to a particular key

  • The note is randomly chosen from the array of notes when created

    • Default will be to only choose 1,3,5,8 notes from the key to create a major chord

  • As the balls collide with the rectangles, the note associated with the ball plays through the channel associated with the speaker.

• Create many balls, increasing the chance of a chord playing through different speakers

• Use the the right or left arrow to change the key that the notes are drawn from.

• Use the up or down arrows to make balls move faster or slower

• Hit space to clear balls

Execution Plan

I started to wire up the sketch by creating objects for the balls and rectangles. I modified an earlier particle system sketch I made, and kept the system object, which may allow for some more features down the line. I haven’t connected tone yet, which is obviously the meat of this sketch. I plan on working on this over the next 2 weeks to make it work for the binaural assignment. The first set of things I need to to do:

• Finish collision detection between ball and rectangle

• Add tone.js to the html doc

• Hard- assign channels to each speaker

  • Once this work, allow input to assign channel

• When collision is detected, play a sound

  • Make sound draw from an array of possible sounds in a key

  • Create arrays for different keys

  • Allow key to be changed

• Connect slider to ball speed

• Re-configure keys/mouse presses to match a good UX

• Create volume slider and hook up to tone

Particle Systems -- work in progress

Assignment: Build a particle system

Idea

This week, I wanted to make a system of blobs that looked like bacteria moving around the screen. I envisioned a mix of different bacteria with a bunch of common characteristics, but different colors and perhaps shapes or other features. I wanted to use the “extend” functionality of JavaScript to work with inheritance and give these different bacteria unique differences.

To do this, I started by modifying Shiffman’s Coding Challenge to create a blob with bacteria-like qualities, using things like p5 Vectors to get it ready for object-izing. Code here

I knew I would need to create a blob class that I called “Blobject,” which would have its aesthetic characteristics in a “display ()” function and other characteristics, like its ability to update/move, in other functions.

I also knew that I would need to pay special attention to where the blobject started on the screen, likely through its Vertex coordinates.

Finally, I would need to extend the Blobject to create another object (I called it a “Bluebject”) that could get added to the sketch from a “Blobject System” object.

Along the way, however, I got tripped up trying to make all of these things happen. I started with existing code examples and began modifying — but I may have been better off trying to write functions from the ground up, as I think I spent more time trying to backwards engineer code that wasn’t perfectly suited for my use case.

I feel like I’m close, but I have to cut myself off right now for the sake of not staying up all night. In my eventual- particle system sketch, I have this:

I need a bit more time to finish debugging, but I’m going to make it happen!

Fun with Polar Roses

Overview

I set out this week to explore using springs and forces + step up my object-oriented programming game. I ended up doing neither thing. Instead, I started making roses with polar coordinates and wound up getting sucked in to a game of “what does this do?” In the process, I wound up tinkering my way to a better understanding of the relationship between polar and cartesian coordinates + got a better sense of how some interesting visuals that could be incorporated into audiovisual performance can be created.

Final Code 1

Final Code 2

The A-Minor Music Machine

So that this doesn’t get buried under the reflections below, here’s the sketch. I will update here when the version that works with Arduino is functional.

——

Within 10 minutes of posting this, I’ll probably be editing code again. This project, more than any up to this point, drove me insane and made me so happy.

At various points in the week, I grunted out loud to myself (and anyone in my general vicinity) as I failed to notice a missing “this.”; I enlisted a former software developer colleague to hop on the phone and talk me through questions late at night when office hours were no longer an option; I thought I’d reached a nice stopping place, then thought of one more feature, then looked up and another hour went by.

And in the process, my empathy for all the devs I know just increased exponentially from an already- empathetic place.

With the preamble aside— what was the assignment? What did I make?

The assignment this week was to use objects, classes, arrays of objects, and interactions between objects. I hadn’t yet used sound in a p5 sketch, so I made that a constraint for myself— do all of the above while using sound.

I got inspiration for this sketch from this example; after looking at it I wanted to see if I could replicate the effect on 4 different drum sounds on one canvas. I started by trying to get it working without any objects/classes, which you can see here. It worked! From there, I started to see a “sound” object coming together— something that had a type for each type of drum sound (kick, snare, etc). It would also be able to display itself and maybe analyze its amplitude. Simple, right?

The process of migrating this code to an object-oriented structure took a lot longer than I’d expected, and was the source of a lot of the grunting and requests for help. Ultimately, there were a couple of important lessons:

• Understand variable scoping:

  • Global variables can be used.. um… globally.

    • When using something like p5.Amplitude this was a really important concept to internalize. p5.Amplitude measures the entire output audio unless otherwise specified. If p5.Amplitude is set globally, it’s not possible to setInput later on and have that used to, say, draw rectangle height like I do.

  • this.object is available for all object instances in the class

  • instantiating a variable inside a function within a class scopes that variable to just be available in the function

    • know when to use this vs a this.object

• While debugging, isolate each action you are trying to perform independent of other actions.

  • Do this by finding or creating variables to console log in order to see whether the program in functioning as expected

• Keep simplifying

  • I found myself, and still find myself, repeating some code with only small variations for particular cases. Instead of having a bunch of if statements to describe when to play a sound and the same set of if statements describe when to analyze a sound, maybe there could be a common variable that could pass to both .play and .analyze (in this case, this.sound.)

Pieces

I recorded the drum sounds, chords and bass notes in Logic Pro X and exported each sound as an mp3. I loaded those into p5 and used preLoad to make them available to the rest of the sketch.

I used the p5.Amplitude, setInput, and getLevel functions to get the amplitude of any given sound file. I could then pass that amplitude reading (done in the draw function to get it continuously as the file is played) to the rectangle height parameter to create the drum visualizations. Those are triggered by keystrokes.

The chords are stored in an array. The right and left arrows cycle either up or down the array, resetting if they reach the end of the array. When they are created, they get passed a random number between 1 and 6. That number is used to change the background. A few of the numbers change the background color. Others don’t. The number assigned to an instance of an object is random, so there’s no rhyme or reason to when the background changes other than that it only happens when chords are played.

There are some white dots that appear occasionally. These are an array of “Item” objects that get triggered from within the display function. These appear when a particular random number in the chord object appears when a kick drum has been played. In order to make this happen, I created a state variable to measure whether the kick drum has been played (but resets after it gets played). When the kick (and clap, same logic) has played and the random number coincided, these ellipses appear until the condition is no longer true.

The bass notes trigger the triangular visualization in a similar way.

All in all this has been a really rewarding project to really ramp up my understanding of a number of things. As an added bonus, it is a great foundation for working with serial communication in PComp: the variables controlled by keystrokes will be controlled by buttons and other things out in the real world!

CollaboCoding in P5

Last week we began playing with variables, introduced conditional statements and for loops, and overall added more to our bag of tricks. Our homework asked us to use some of these tools to create buttons or sliders that controlled some other element on the screen (without using buttons that we get for free from the browser or bootstrap-like tools.)

But really, this week was about learning how to make our code organized and commented well enough for another human to understand. So, I’ll reflect a bit about what worked and didn’t work so well while collaborating with a partner.

Our Approach

We decided to start our sketches individually rather than agreeing on a design before we’d had time to process the week’s lessons. Each of us took a couple of days to mock up a sketch of our own, then traded over the weekend to review and add to our partners’ sketch. On Monday/Tuesday before class, we chatted about our changes in-person and helped each other understand our additions or changes to each others’ code.

Drafts

In my initial draft, my intent was to create a set of buttons that mapped to the basic parts of a sentence — subject, verb, adjective, noun — and changed each of those corresponding parts of a sentence with each button press. In order to do this, I knew I’d need to work with an array of strings and use some logic to randomly select from those arrays, then call the results of that logic in the rest of my code. I also knew that I’d need to use the mousePressed() or mouseClicked() functions with conditionals that mapped to each rectangular button.

As I began to figure out the word choice logic, I created variables with different pieces of the process; one variable to create a list of words, another to randomly select an index from that list, another to translate and store that index as an actual word. When I needed to wire that up to buttons, however, I realized that I’d either need— or drastically improve the efficiency of my code— with a function specifically for all of this logic. This took me away from animation, but ultimately was a rewarding process!

My partner created this very cool looking animation. Her goal was to have the background change to the roll-over color of each circle when pressed. However, there was initially some odd behavior that kept the background blue under certain conditions. I made it my goal to debug this issue and get her sketch working as she intended.

Collaboration

Once I got the code in my sketch working the way I wanted it to in a simple, very lightly styled interface, I focused on writing comments that would make the code immediately understandable for my partner, without needing to physically explain anything. For the most part, it worked— my partner was able to add a ton of animation and styling to the next iteration of the sketch. However, I failed to make clear why I created a “quarterW” and “quarterH” variable, so she ended up writing her on logic for rectangle sizing using the built-in width and height variables.

When we met face-to-face to talk through our results, we discovered an unintended behavior of the “frame” in her additions to my sketch, which we were able to debug together. While our comments helped us get 90% of the way, it was helpful to debug together. We were also able to discuss the changes to her sketch that debugged her initial vision (here). The solution was fairly straightforward to implement: at the very end of the code, the part that created toggles for on1, on2, and on3, we needed to ensure that if one button was on, the other 2 were off. By showing her how console logging the states of each “on” variable allowed me to see the unintended behavior in her initial sketch when multiple variables were on at the same time, we were able to quickly understand the roots of the bug.

Overall, the most useful part of this exercise was having to pay close attention to how our code was organized and commented, which will continue to be important as we start adding more and more to our toolkit!

Shoutout to Beste Saylar, my heretofore anonymous partner :)

For clarity: NK Draft, Beste revision // Beste Draft, NK revision

Variables!

Last week, I drew a sketch of a hiker climbing some mountains (vaguely resembling the Grand Tetons) around sunset. As the very first exercise with p5js, we were encouraged to keep everything hard-coded to get a better feel for how different value ranges would express themselves on the screen.

This week, however, we’re having fun with variables!

So- I took last week’s drawing and made it more interactive (click link here to see).

Just like what I imagine so many assignments will be like at ITP, there are a number of elements of this sketch that I would like to do differently, if I had some more time. Here are a few examples:

1) Switch which components the mouse controls

When I initially started the assignment, I knew how to move the sun and sunset color slices, but didn’t yet figure out how to move the hiker— which is really a collection of lines, rather than a single function like fill() or ellipse(). Fill() and ellipse() are easier to parameterize with variables. I wound up using translate() for the hiker after I’d already coded the interactivity with the sun and background. I knew I wanted some elements to move based on time and others to move with mouse interaction. The sun/background moving with time and hiker moving with user control would have made more sense… but this is art school, so let’s just say I meant this as an attempt to invert our relationship with time and the sun. To give the user control over one of the things normally entirely out of our control: time… or something like that :)

2) Make the hiker descend upon hitting the window width + reflect around the y-axis

Currently, the hiker glides backwards down the mountain a good handful of seconds after getting to the width of the window. I couldn’t see immediately why my incrementing structure doesn’t work; with a few more minutes (or hours) I think I can figure this out, but alas, I don’t have them right now. Console.log-ging the x variable shows that the x variable reaches ~430 when the figure appears to leave the window frame, reaches 600 (the current window width) a few seconds later, then starts the descent. The y-variable logs as a negative number the entire time, even though the incrementing function should only reverse the sign when x > width or x <0.

From an aesthetic perspective, it looks strange to see the hiker walk down the mountain backwards. I’d like to have it flip across its y-axis when it turns around, but I’m not quite sure how to do so yet.

While the sketch isn’t perfect, it has definitely given me clear questions that I’d like to learn the answer to. Perhaps the next topics will lead to some more efficient and effective approaches.

On to For loops!

Creating a Static Drawing in p5js

Assignment Prompt: Create your own screen drawing: self-portrait, alien, monster, etc. Use 2D primitive shapes – arc(), curve(), ellipse(), line(), point(), quad(), rect(), triangle() – and basic color functions – background(), colorMode(), fill(), noFill(), noStroke(), stroke(). Remember to use createCanvas() to specify the dimensions of your window and wrap all of your code inside a setup() function

I started this assignment by reflecting about what I’d like to draw. Like so many assignments at ITP, from this early perspective at least, we had a lot of agency to take this in any direction. I recently came back from a trip to the Grand Tetons and Yellowstone National Park, so images of the outdoors are still fresh in my mind. After doing some research into the National Parks’ design system for Visual Language, I decided to make my own outdoor drawing.

I’m not very experienced in drawing, but regardless, with the assignment in mind I tried to keep the shaped in my sketch to those that I knew were possible with 2D primitives in p5. Part of my goal with the assignment was to familiarize myself with the basic drawing functions, including curves, before we moved on to more advanced concepts.

Along the way, my final image transformed. I started with a blue background and bright green grass— a sunny day, with trees and a river taking center stage. But when I looked at my screen after being away for a while, the brightness was a little jarring, and the river just didn’t look right. I tried varying the background to look more like a sunset— that looked better!

Then I started to move the sun around until it look natural. Behind the mountains or fully visible? Closer to the red part of the sunset or higher up? This was not a very scientific process— I looked at a couple sunset pictures but ultimately just went with my intuition.

After getting rid of the rivers (and my use of the curve function,) the screen looked a little too barren.

“Well, the assignment did mention a self-protrait as inspiration… How about I add a stick figure backpacker?”

This did the trick— now the image has a little more movement and a central character. I added some stars to the night sky emerging as the sun fades away.

The final image captured a little taste of the magic that I remember of the Tetons in the evening.

To see the code for this image, go to this link: https://editor.p5js.org/nkumar23/sketches/JujD9a_vr