Data sonification for scientific exploration & discovery

Data sonification, often used for outreach, education and accessibility, is also an effective tool for scientific exploration and discovery!

Working from the Lindorff-Larsen et al Science (2011) atomic-level molecular dynamics simulation of multiple folding and unfolding events in the WW domain, we heard (and analytically confirmed) correlations between hydrogen bond dynamics and the speed of a protein (un)folding transition.

The results were published this week in The Proceedings of the National Academy of Sciences (PNAS), vol. 121 no. 22, 28 May 2024: “Hydrogen bonding heterogeneity correlates with protein folding transition state passage time as revealed by data sonification”

Congratulations to everyone in the Biophysics Sonification Group:

Carla Scaletti (1), Premila P. Samuel Russell (2), Kurt J. Hebel (1), Meredith M. Rickard (2), Mayank Boob (2), Franz Danksagmüller (9), Stephen A. Taylor (7), Taras V. Pogorelov (2,3,4,5,6), and Martin Gruebele (2,3,5,8)

(1) Symbolic Sound Corporation, Champaign, IL 61820, United States;
(2) Department of Chemistry, University of Illinois Urbana-Champaign, IL 61801, United States;
(3) Center for Biophysics and Quantitative Biology, University of Illinois Urbana-Champaign, IL 61801, United States;
(4) School of Chemical Sciences, University of Illinois Urbana-Champaign, IL 61801, United States;
(5) Beckman Institute for Advanced Science and Technology, University of Illinois Urbana-Champaign, IL 61801, United States;
(6) National Center for Supercomputer Applications, University of Illinois Urbana-Champaign, IL 61801, United States;
(7) School of Music, University of Illinois Urbana-Champaign, IL 61801, United States;
(8) Department of Physics, University of Illinois Urbana-Champaign, IL 61801, United States;
(9) Musikhochschule Lübeck, 23552 Lübeck, Germany

Science of Sound

Nate Butkus sending data from an iPad to control sound parameters in Kyma. Next step: substitute data from a scientific model for the iPad.

Nate and his team from the award-winning “The Show About Science” podcast had a question:

“Can sound help us understand the complex patterns in our universe?”

This question led Nate to Symbolic Sound in Champaign, Illinois on a journey where sound, music, and data intertwine in captivating and thought-provoking ways…

Listen to the podcast or read the transcript at the link below:

101: The Science of Sound with Kimberly Arcand, Martin Gruebele, Carla Scaletti, and Mark Temple

Martin Gruebele, Carla Scaletti, Nate Butkus, Eric Butkus (Photos by Jenny Butkus)

Displacement

Photo by Dr. Javier Alejandro González Ortega

After the 2010 El Mayor Cucapah 7.2 magnitude earthquake in northern Mexico, seismologist Alejandro González Ortega interviewed Don Chayo, a Cucapah native who witnessed the surface rupture. When Don Chayo drew parallels to the origin stories of the Cucapah people, González began to wonder if these stories may have recounted earlier seismic events that had been passed down over the generations.

Over the next several years, González and his colleague, choreographer/physicist Minerva Muñoz, created a performance piece based on 3D seismological data collected by 12 measurement stations during the event. Muñoz enlisted the help of composer Carla Scaletti to map the data to sound using Kyma and artist David Olivares to map the data to video using Unity.

As Muñez and González conducted further research and interviews with the Cucapah elders, a much more disturbing story began to emerge — that of a displaced people whose livelihood was being cut off and whose very language was being forgotten. What had originally been intended as a science/art collaboration about seismic activity began to morph into a deeper metaphor for displacement, disruption and loss.

The result — Wí Shpá, A journey in bare feet — is a poem in movement, images, sounds and words that explores pilgrimage, displacement, change, the relationship of humans with the environment, transformation and resilience.

The sound and visuals were created from seismological data and satellite geodesics of the El Cucapah Mw 7.2 earthquake that occurred on April 4, 2010. Consistent in many details with the cosmogony myths narrated by Don Chayo that had been passed down over generations, El Mayor-Cucapah Mw 7.2 was the most intense earthquake recorded in this region over the last century.

Wí Shpá, A journey in bare feet is an elegy to the ancestors and to the women and men of today; to the people of the river, of the earth, fire and wind. It is a glimpse into a universe in which animals are gods, and meaning is associated with each of the four cardinal directions, colors, the power of nature and of the land.

“Cosmogony of an Event, El Mayor Cucapah Mw 7.2” is an inter and trans disciplinary dialog of artistic creation and research combining the myths of Cucapah cosmogenesis and the scientific studies of El Mayor-Cucapah Mw 7.2, weaving a network of collaboration, tradition, scientific research, knowledge and experiences, but above all, creating a dialog between scientists, artists, native community, collaborators and the general public who participate in this live performance/ritual.

Credits:

Direction, stage creation and interpretation: Minerva Muñoz *
Production: Alejandro González, Minerva Muñoz / La Machina Productions
Scenic Advisor: Jorge Folgueira
Lighting: Minerva Muñoz
Composition and sound design: Carla Scaletti
Visual Art: David Olivares
Video: Marco Meza, Rommel Vázquez
Aerial Video (drone): Alejandro González
Photography: Alfredo Ruiz and Rommel Vázquez
Science: Javier González-García and Alejandro González
Audio Engineer: Rommel Vázquez
Scenography: Leoncio García
Makeup: Rosario Martínez
Lighting technician: Miguel Tamayo
Communication and networks: Stephanie Lozano
Support: Juan Sánchez

The rocket scientist of human hearing

In 1999, astrophysicist/musician David McClain spent an intense three-month period working on The Northern Sky Survey, mapping the sky in the near infrared while getting by on an hour of sleep per night. When he finished the survey, he was suddenly struck by a viral infection that nearly killed him; his doctors were never able to determine the cause and, after three months, the infection dissipated almost as quickly as it had appeared. But afterward David noticed that he could no longer understand his wife when she was speaking to him. He went to an audiologist and discovered he had a sensorineural hearing loss of 60-70 dB in the high frequency range. Hearing aids helped him understand speech, but he was devastated to discover that music never sounded right through the hearing aids. But as a physicist, he was determined to solve the problem.

Motivated by his love of music and informed by his scientific training, McClain has spent the last 16 years developing equations to describe the entire hearing experience – from the cochlea, to the afferent 8th nerve, to processing in the central nervous system, efferent 8th nerve interactions — and developing signal processing algorithms to adapt to and compensate for his hearing loss in a way that preserves the audio experience of music. The result is a collection of signal processing algorithms he calls Crescendo. Kyma is one of the tools David uses for testing out new ideas and prototyping them for Crescendo.

Now he’s blogging about his findings on his web site http://refined-audiometrics.com. In keeping with his motto “Keeping music enjoyable for all!” David hopes that his experiences, research findings, and extensive set of algorithms can benefit others.

 

Composer on a NASA mission

Composer Roland Kuit was recently interviewed on the prime time news program SBS 6 Hart van Nederland to discuss his Kyma sound explorations that will be launched into space on September 8, 2016 on NASA’s OSIRIS-REx mission to the near-earth asteroid Bennu.

While his music is being launched into space on September 8 2016, Kuit will be at the Kyma International Sound Symposium in Leicester, UK presenting his music and ideas along with filmmaker Karin Schomaker so you’ll have an opportunity to meet and talk with him at KISS2016.

Kyma creator on the cover of Computer Music Journal


The Spring 2016 issue of Computer Music Journal (Volume 40 Issue 1) includes a transcript of Carla Scaletti‘s keynote address for the 41st International Computer Music Conference.

In Looking Back, Looking Forward, Scaletti uses mythology, evolutionary anthropology, nostalgia research and a story about the origins of Kyma to illustrate the idea that software is “hardware with cognitive fluidity”.

 

The Listening Back, Listening Forward issue marks the beginning of Computer Music Journal‘s 40th year of publication and, citing recent research showing that nostalgia enhances creativity, CMJ editor Douglas Keislar invites readers to share their own computer music stories for possible publication as letters to the editor throughout this anniversary year.

Also in this issue: Silvia Matheus reviews The Seventh KYMA International Sound Symposium (KISS2015)!

QUANTUM in the desert: art, science, technology & collaboration

In April 2015, QUANTUM, the dance piece Jobin created inspired by his residency at CERN, will be touring northern Mexico including Culiacan, Hermosillo, Tijuana, Ensenada and Mexicali. Technical director Marie Predour will be running the live sound for the piece using a Kyma 7 Timeline.

Choreographer Gilles Jobin took a moment to talk a little about the piece, to explain his ideas on algorithmic choreography and to reflect on collaboration, art, science, and technology.

 

What’s interesting about technology is not so much the technology as a tool, but technology as a new way of thinking — as a different way to organize your work or to think about your work. The same is true when choreographers work with scientists or with musicians. There is a kind of exchange of practices that is enriching for everybody.

14 April Culiacan РFestival Danza Jos̩ Limon

16 April Hermosillo – Un Desierto Para La Danza

19 April Tijuana – Cuerpos in Transito

21 April Ensenada – Espuma Cuantica

23 April Mexicali – Entre Fronteras

27 April to 2 May Torreon – Gilles Jobin will be on the jury for Premio Nacional Guillermo Arriaga

What if the stars made music

Outdoor Culture presents Sounds of the Night Sky featuring Robert Jarvis‘ sound installation: aroundNorth, opening Thursday 20 February 2014 6.30-9.30 pm at the National Trust Stowe New Inn Farm Buckingham MK18 5EQ

This outdoor event will showcase the premier of Robert Jarvis’ new sound art installation, aroundNorth, a piece that was shortlisted for the 2010 PRS New Music Award.  A multi-speaker sound map of the stars driven by the turning of the Earth, aroundNorth uses Kyma to transform the night sky into a celestial music box rotating around Polaris, the North Star. As each star passes a virtual line in the sky, it triggers a musical note whose qualities are determined by the star’s spectrum, mass, brightness and distance from earth, creating a mesmerising sound map of the universe as viewed from our rotating planet.

Visitors will be accompanied down Bell Gate Drive on foot and into the gardens of Stowe after dark, before entering the semi-wilderness of the newly-opened Lamport Garden where the sounds of moving stars will be created like a giant celestial music box! Dress for the outdoors and bring a flashlight!  Click here for more information and tickets.

Effects of pitch-shifting on stuttering

 

Torrey Loucks, professor of Speech and Hearing Science & researcher with Beckman Institute’s Cognitive Neuroscience group & Illinois International Stuttering Research Program

When you listen over headphones to your own voice through a pitch-shifter, do you find yourself mimicking the pitch shift? Or do you automatically counter the pitch shift by shifting in the opposite direction?  According to speech and hearing science professor Torrey Loucks, most people compensate by shifting in the opposite direction to correct the deviation. But there is another group who follow the pitch changes.  We don’t yet understand why these individual differences in auditory vocal responses occur.

In a recently published study, Professor Loucks and his graduate students HeeCheong Chon and Woojae Han at the University of Illinois in the department of Speech and Hearing Science utilized Kyma for real-time pitch shifting in an experiment that attempts to increase understanding of the neural mechanisms underlying stuttering.

In this study, a group of adults who stutter and typically fluent adults were asked to produce the same vowel sound while monitoring their own voices through headphones.  Kyma was used to shift the pitch of the subject’s voice up or down by as much as 200 cents (2 half-steps) for a duration of 500 ms.

In most cases, the participants adapted by lowering or raising the pitch of their voices to counter (rather than mirror) the pitch-shift imposed by Kyma. In the stuttering participants, the adaptive response to the pitch-shift was significantly delayed as compared to the responses of non-stutterers. The stuttering participants also tended to have a lower magnitude of pitch shift responses.

In their report, “Audiovocal integration in adults who stutter” recently published in the International Journal of Language & Communication Disorders, the authors analyze and discuss some of the implications of this result.

One theoretical prediction is that persons who stutter rely more strongly on auditory feedback to produce speech, whereas typically fluent speakers use a more robust internal predictive model of the expected result and are less reliant on audio feedback. On the other hand, there are also studies suggesting that persons who stutter tend to have slower auditory reaction times and are less adept at pitch tracking than typically fluent speakers. Loucks et al conclude that, although their results “do not negate arguments that adults who stutter are more dependent on feedback, their dependence is not expressed through a more reactive pitch-shift response”.

In an article describing his research with the Beckman Institute’s Cognitive Science group, Professor Loucks explains some of the wide-ranging implications of research on stuttering:

Stuttering is very interesting because it appears to occur at the junction point between formulating what you want to say and actually being able to express it.

He also corrects some outdated preconceptions on the phenomenon of stuttering:

There are no predisposing events that make a person stutter that could be prevented either by being a better parent or being a different sort of child. No one is to blame for the occurrence of stuttering because it is a biological disorder.

Nonetheless, the stigma of stuttering can be reduced considerably by realizing that we need to accept communication disorders as occurring in the population. There’s nothing negative about having a communication disorder and people should know that stuttering does not affect a person’s ability to learn, to succeed academically, all of those things. It has nothing to do with intelligence and should not be in any way a barrier to a person realizing their full potential. It is a neurobiological disorder that no one could prevent, but which we can find better ways to treat and possibly cure in the future.

Professor Loucks on University of Illinois campus

 

 

Einstein’s bicycle

Einstein taught his uncle how to ride a bicycle, and, soon, choreographer Gilles Jobin will be able to return the favor by teaching a group of present-day physicists how to dance in his new role as the first Collide@CERN-Dance and Performance Artist in Residence.  Jobin wonders whether there is a parallel between the scientist’s totally focused thought in pursuit of knowledge and the artist’s total focus in the moment of performance. The wickedly playful Jobin promises to stir things up around the ring with several interventions across campus in unexpected places, all investigating the connection between thinking and moving, all certain to be stimulating and fun as learning to ride a bike for the very first time.

 

Jobin is an internationally acclaimed choreographer based in Geneva and well-known for commissioning new electronic music for his dance scores (several of which have employed Kyma)!

For periodic updates on this playful collision of art and science, follow ArtsAtCERN.