Welcome to Sensability

A music visualization experience to accomodate accessibility and enrich music therapy

Master of Arts in Graphic and Web Design

Capstone Project

August 2022

“For deaf, deafened, or hard of hearing (D/HH) people, the experience of entertainment and enjoyment produced for hearing audiences (as they occupy the larger portion of society) is limited. While it is true that the deaf community creates its own music, people who are deaf experience barriers when attempting to access the music of the larger hearing culture. They do not have access to this shared experience because it is not provided in a modality that is more appropriate to this population of consumers”

- sourced from "Creating Access to Music through Visualization"

How can an accessible, media rich approach to music offer individuals a greater therapeutic impact?

It would be wonderful for Deaf and Hard of Hearing individuals to experience music with visuals and actual representations to connect with the message of the music and understand the instruments which are being played. Some of the responses I received from Deaf individuals was there was no interest or engagement in music. However, there was an appreciation for creating visual music for deaf individuals.

This feeback inspired my design challenge to transform this problem into a creative approach. Creating a music visualization focused on representing the instruments would provide both Deaf and Hard of Hearing individuals with a realized visual experience while maintaining a therapeutic approach. When listening to music, it engages both hemispheres of the brain and multiple areas of the brain bilaterally. For Traumatic Brain Injury patients, music is important as this healing benefit with music helps restore the brain and stimulates neuroplasticity.

Gabby Giffords recovered from a Traumatic Brain injury by using music therapy. Her story inspired this Capstone project as she regained her ability to speak by using neurologic music therapy. She worked with her music therapist to complete the spoken lyrics of songs to regain her speech ability. In addition to “learning to repeat ordinary phrases in a sing-songy voice. A song would gradually become a chant and finally a spoken phrase with the natural rhythm of speech”

- sourced from "How music therapy is helping Rep. Giffords and others recover their ability to speak"

"As a person who lost my hearing later in life, I can certainly see value in creating music in a visual way that is better understood by our deaf eyes. Smile."

- Melissa "echo" Greenlee

Video has no audio

The initial idea

Sensability is a project initially designed for Deaf audiences to visualize and feel music. I started this project in Experimental Interaction, a course as a part of the Master of Arts in Graphic and Web Design curriculum. The idea for my project was to explore different music sources with the movement of the amplitude which incorporates a dimensional visualization.

In Cinema 4D, a three-dimensional modeling software, different sections of the sphere objects are synced to selected areas of the music's amplitude and react by upward and downward movement.

The music visualization correlates with a 3D printed object through an interactive real-world environment or augmented reality experience with the printed object activating the visualization. The shape of the 3D printed tactile aspect in this design was established from a p5.js code file which is a Javascript library for creative coding.

This image shows the Cinema 4D software view of sound connecting to sphere objects.

“I think some of the frustration I've had with listening to music after losing my hearing was that captioning never really seems to capture the heart of the piece — the movement of the audio clip.” - Parker Fox

“What are the lyrics saying? Some music doesn't have lyrics. What is the instrument being played? Maybe there is a video of a person playing the instrument. This would help identify the instrument that is being played and would make an impact." - Amanda Tuite

Parker Fox, is the online learning instructional designer for Minneapolis College of Art and Design. Parker is Hard of Hearing which connects with my audience for Deaf/Hard of Hearing individuals. As an artist, Parker has worked with integrating sound in their projects. Parker has research experience with creating accessible viewing experiences for individuals. The combination of feedback from Parker's experience and design expertise is truly valuable. 

Amanda Tuite is an educator, Deaf community advocate, and mentor as well as CEO and Co-Founder of Access Vine, an inclusion training service. Her experience as a Deaf individual and advocate is a truly valuable asset to my Capstone project. I connected with her along with an American Sign Language (ASL) interpreter through two zoom meetings.

After sharing my initial music visualization to Amanda and Parker, their feedback inspired the idea of an instrumental music visualization with actual instrument representation. The additional suggestion from Parker of possibly “pursuing the therapy aspect of this tool” for the tactile aspect further motivated my idea to expand the audience to music therapy and include haptics with the tactile aspect.

Designing for Deaf, Hard of Hearing, and Music Therapy audiences share this elevated healing connection as music therapy resources could be beneficial for the Deaf and Hard of Hearing community as well.

The Prototype

The prototype of this project started with an initial music visualization focused on connecting the visuals to the instruments in the song. The goal is to establish a visual music experience with instrument representation for Deaf and Hard of Hearing individuals. The song used in this prototype is “A Peaceful Winter” by Scott Holmes from Free Music Archive, which is an online library of royalty-free music. Selecting this song was inspired by responses from Subject Matter Experts, and a desire for more visual representations of instruments in songs.

The prototype design included music instruments that were designed in Cinema 4D, a three-dimensional modeling software, and Adobe Illustrator, a vector graphic digital editing and design software. After the designs were complete, the instrument graphics were integrated into After Effects, a digital motion graphics compositing application, for the production of the motion design.

This image shows the After Effects software view of creating the motion prototype

Initial Prototype

Included in the original design were captions to assist the visuals in the motion design. One of my subject matter experts, Amanda Tuite suggested including captions in the design but making it an option instead of directly integrating them in the video as some individuals might want to see the visuals without captions.

After the initial prototype visualization was created, additional feedback I received from Amanda was: "What is the emotion?", "With the instruments, how is it supposed to inspire me?" I could incorporate winter scene colors and visuals in the music visualization to establish a connection with the song used in this prototype “A Peaceful Winter” by Scott Holmes from Free Music Archive, an online library of royalty-free music. It would help viewers understand the emotion of the music, especially for some viewers watching this without sound.

Additional feedback from one of my subject matter experts, Parker includes: “If someone does not have an idea of what the instrument is or what the notes sound like or has never heard the instrument before; and if you are going to incorporate the instruments, think about how you are going to indicate the highest and lowest string in the violin representing high or low notes.“

Revised Prototype

The instrument visuals continued into this revised prototype with the addition of simulating the live movement representation of the instruments. Winter elements were added to the composition to communicate the feeling and emotion of the music with the instruments. In addition, movement with a camera was specified with keyframes, a position in time within the composition to define the animation transitions.

Final motion prototype

The Final motion prototype from ideation to production is complete.

Sensability App

With designing for music therapy audiences, This app was inspired by a google chrome accessibility profile extension idea, I started at the beginning of this project as a way to accommodate music experiences. It developed into a personalized music visualization where users can select the music visualizations based on the music style, tempo, and if desired accessibility settings. By subscribing to the app, users have the option to receive a Medo device, have access to music therapy resources and connect with music therapists.

Initial thoughts for app design

The main idea for this project was complete, a motion design music visualization. Sometimes when individuals are going through obstacles, there is limited resources or ways to communicate personal feelings or emotions. The original concept for the extension of this music visualization was going to be therapeutic resources for anyone who has endured a surgery, diagnosis, or health complication themselves, or experienced it with a loved one.

Maybe this extension includes: resources, a written journal aspect, shared stories or images, and a comedy or relaxation page for an assisting purpose? With the initial idea of this Sensability project including a 3D printed tactile aspect, I wanted to find a way to successfully connect the tactile aspect with the music visualization. Is rethinking the idea of a subscription aspect possible where the tactile aspect could be included? These ideas are my thoughts on the ways I can progress this main idea into a successful project with a focus on goals to benefit the user.

Original app idea

The main idea for the original app design was to have a selection of music visualizations displayed based on the user's feelings or mood. This would be similar to meditation or relaxing apps in the market such as the Calm app, an app designed for sleep and meditation. The emotional aspect of this idea would be decided by the user. Initially, there were going to be three main styles of visualizations in the app design: dimensional, abstract, and simple. Based on user experience feedback, too many visuals can possibly cause motion sickness for individuals with sensorineural hearing loss. This feedback inspired the simple visualization style.

Revised app design

As the idea for the display of music visualization centered around mood or feelings was a more complicated process, the process was simplified to start with the desired music genre style, tempo, and accessibility settings.  The goal of this updated idea is the music activates a memory response. By communicating the emotion and message of a particular song, the imagery brings the viewer to remember an emotion or memory associated to the music. This serves the goal of visualizing and communicating music for the Deaf, Hard of Hearing audiences and the therapy aspect of music rehabilitation.  The updated visualization styles would consist of actual videos of the instruments being played in addition to 2D and 3D graphics. The app design is prototyped in Figma, a web browser-based interface design tool.

Accessible experience

For music including lyrics and instruments, captions and sign language interpretation are both available to provide a desired, accessible, music experience for Deaf and Hard of Hearing individuals.

Final app design

The Final app design is complete as a prototype in Figma.

Additional music visualization

The video has no audio

In addition to the main motion design visualization, an additional visualization for the app design was created to visualize the trumpet instrument in Cinema 4D, a three-dimensional modeling software.

"Visual imagery associates with episodic memories, where emotions occur as individuals are reminded of memories from listening to the music"

sourced from "Emotional responses to music: The need to consider underlying mechanisms"

Medo

The name Medo is derived from medicine; ritual magic by a medicine man, to promote the desired outcome in healing. Pronounced Medow, it resembles an open field with a peaceful ambiance.

Medo (meadow), is the tactile aspect counterpart of the Sensability app. I utilized a Sindoh 3D WOX 1 3D printer to produce the Medo device. The goal of the Medo device is users will be able to feel the vibrations of the sound along with the corresponding music visualizations with haptics integrated into the design. The web development code to function this Medo device is currently in the testing phase. However, the hardware which will function the Medo device will be included in the design. The hardware consists of a Groove for Arduino base shield, rotary angle sensor, sound sensor, red LED sensor, and vibration motor.

Original designs

Since one of the focuses of this project is a music therapy aspect, the textures on these designs could be similar to occupational therapy objects to relieve stress. The seashell texture on the first idea is inspired by a sensory stimulation guide for individuals with neurocognitive disorders. The texture on the second idea is inspired by textures that can improve the brain through touch. After this first design, I continued the progress by placing objects with texture qualities to relate to the therapy aspect on the Medo device. With the texture idea relating to the therapy aspect, the overall idea of how the Medo device connects to the app visualizations needed to be improved.

Development of Medo design

The seashell texture in the original idea was continued into this updated design as the base of the structure would maintain the same shell shape. I showed my original designs to Parker, a subject matter expert for my project. I asked Parker, how can I make the design have more intention with the texture? Parker suggested, “map unique shapes to the sound I am looking to capture so there is a cohesion between the song, object, and vibration movement."

The demo of Medo will correspond with the instrumental piano song called “Bird Language” by Chad Crouch on Free Music Archive, an online library of royalty-free music. The corresponding visualization is an example that will be displayed in the final app design. This suggestion led to an initial guide to visualize the piano keys which are played along with the number of times each key was played. If it was a high note, it would be shown by a circle, a low note would show a pill shape. Developing the tactile aspect first in Adobe Illustrator, I used the initial music guide for the tactile object and used shapes to represent the piano notation. Once this was complete, I designed the tactile aspect in Cinema 4D.

Paper prototype

I created an example of a paper protype of the tactile aspect which will be 3D printed. Using the paper prototype as a guide, some of the shapes were modeled to achieve an idea of how the actual 3D model will look. After the paper prototype, I continued to develop the shapes to improve the tangible aspect. It also served as a guide for the size of the Medo device, since the haptic hardware needed to be included and connected to a computer through a USB connection.

3D design rendering

The Medo design was created in Cinema 4D, a three-dimensional modeling software. The shell design was developed using a spline path, an outlined path that can be extruded, while sculptural dimension was added by three-dimensional objects.

3D printer software view before printing process

Once the Medo design was complete in Cinema 4D, the stereolithography (STL) file was exported into the Sindoh 3DWOX 1 3D printer app to be produced. The Medo device consists of a top and bottom structure. The bottom section holds the hardware and the top section incorporates the tactile representation which connects to the music visualization. An extruded addition to the top structure holds the Medo device together.

3D printing production

Using a white polylactic acid (PLA) filament for the Medo design the 3D printer composes the design by extruding filament from the nozzle of the printer and a grid structure builds the 3D printed model. The final design was complete after seven hours of printing production for both the top and bottom structures.

Final Medo 3D printed design

The finished Medo device from ideation to production is complete in a 3D printed form. The shapes connecting and representing the visualization is able to be experienced by touch. The Medo device is ready to be used along with the corresponding music visualization.    

Medo demonstration

Even though it cannot be heard, users can literally feel the music. The shapes representing the piano notes on the 3D printed Medo device are included on the piano keys in the visualization. In addition, the colors represent the scale in the octave in which the piano key is played. Since the name of the song is “Bird Language”, the music staff has a double meaning displaying the piano keys and illustrating when a bird sings.

Do you want to see the progress of this Capstone project? Click the link to learn more

Capstone Progress Platform

With this formulated idea for Sensability, I am eager to continue pioneering this design concept. Some ideas I have to integrate into Sensability include: adding more genres for the music visualization, possibly having braille music as the texture on the Medo device, and including therapy resources with an automated intelligence aspect. In the future, one of the goals for the Medo device is to be able to 3D print it with a flexible filament.

One goal for this project is to possibly accompany technology services provided by a music therapist or an organization supporting Deaf and Hard of Hearing audiences. If there was an alternative service or accommodation for music, it could possibly benefit Deaf and Hard of Hearing individuals, to enjoy an adaptive music experience.

Research

Link to the research supporting this Capstone project