StateSong: Converting human data to music using the E4

How effective is music created with data collected from our bodies on our different states and moods? Engineers at Medimuse created an app, StateSong, which in combination with the E4, records and converts biometric data from the wrist as music to help improve peoples’ states over time.

This article has been guest-authored by Robert Joseph, Medimuse Chief Executive and Inventor of StateSong, and Peter J. Slack, Medimuse Chief Engineer, and Software Architect.

Music deeply affects our states and emotions.

Interestingly, nearly all the music you have ever heard has been created by someone or something outside of you. What if you could reverse this process and turn your moods and emotions into music? If music from outside of you (exogenous) is good at changing your states, how much more effective might music from inside of you (endogenous) be in helping to reach desired, better, or different moods?

To answer these questions, we at MediMuse, are working on StateSong, a software that will convert and record biometric data streams as music. StateSong presents a new set of tools for studying and understanding biometric signals in a new way based on a technology patent awarded to Robert Joseph, a Stanford graduate in the field of neuropsychology, and which has been developed as software by Peter Slack, an engineer who has previously built inventive music applications.

Converting human data to StateSongs

Empatica provides high-quality software libraries for developers and researchers like us to explore and develop new applications. StateSong’s conversion process is informed by the biometric data it accepts from Empatica’s E4 device via their software libraries, translating real-time inputs into midi signals, which are then used to create sounds via our integrated synthesizer.

We currently use the E4’s accelerometer, heart rate, and BVP readings to formulate music patterns but will be leveraging other forms of data as we develop our product further. The app is currently in Beta testing, and can already be used to create StateSong songs with the E4.

Music for a variety of activities

Additionally, we have developed an emulation application that artists and music makers can import and convert previously captured E4 data files into midi for creating or remixing tracks into StateSongs using their Digital Audio Workstations (listen to a DJ Endogenous session).

This setting would provide a unique venue for creating scores for movies, television, games, VR, and even the Metaverse based on real human states and emotions.

StateSong can be used for wellness, sports, or creative activities, for example:

• In Health & Wellness: calm a Post-traumatic stress disorder (PTSD) sufferer; coax a baby to sleep, or guide meditators into focus & relaxation.
• In Sports & Fitness: push runners to reach second wind sooner, for longer; or to enhance endurance
• In Art & Entertainment: enhance artistry and creativity; make mood music for movies, TV, gaming, or VR

How it works

With the E4, we have the distinct advantage of a biometric device that (1) is research and development worthy at the highest level; (2) provides an array of sensors that allows for an array of data types to be converted and combined into complex music compositions and soundscapes; (3) all provided seamlessly on a secure, stable, and reliable platform that protects user data according to HIPPA requirements.

We at MediMuse feel fortunate and are grateful to the Empatica team for all their diligence and hard work that paved the way for the years of effort that have gone into creating StateSong.

Owners of E4 devices have the opportunity to test the app with their Android devices and record their own state music while involved in everyday activities, thanks to the wrist-worn design of the E4. Activities such as meditating, studying, daydreaming, gardening, running, walking, laughing, and crying can be recorded with the E4, and converted to music through the Statesong app, giving the opportunity to hear what their different states sound like.

A guide on how to download and install the app can be found here. Right now, StateSong is only downloadable from the Google Play Store as a beta Android application with the E4, but will soon be available for iOS as well as integrated with an array of devices that capture brainwaves, biochemicals, and movements.

MediMuse is also building application components as ‘Open Source’ so that users of the app can use the available toolsets to create new ideas for the development of the StateSong app or plugin. The work on the app and software is an ongoing one, with a lot envisioned for the development and product pipelines, including EEG (brainwave) and Metaverse integrations. Announcements regarding future releases will be made on www.medimuse.net.

It is said, “Music is what feelings sound like”. Now, this is true, both figuratively and literally.

***A graduate of Neuro-Psychology from Stanford University, *Robert Joseph* worked in the Pediatrics Endocrinology Lab of Stanford Medical Schoo, exploring the hormone systems that control stress, and reproductive and metabolic balance. He became involved in research on Anorexia Nervosa, and it was within this context that he observed hormonal activity patterns within patterns and patterns that repeated in phases. Noting that these were the very definition of musical canons and fugues, Robert envisioned that if we could convert hormonal data into sound, might not endogenous music (created from the inside) be as complex and beautiful as compositions created by Bach? Thus, the notion of StateSong was born, encapsulated in the phrase, “Listen, you’re made of music!”. As wearables that captured biometric data began to appear, Robert formulated a startup company, MediMuse, and subsequently acquired the patent for the technology upon which the StateSong platform and applications are built.
Peter J. Slack, P.Eng., on the other hand, is a licensed professional Engineer based outside of Ottawa, Canada. Aside from practicing engineering, Peter has been studying and playing music since six years of age, and has been actively coding computer software for more than 40 years. In addition to his role as Chief Engineer at MediMuse, Peter also serves as a senior software engineer at WaveDNA.com, which conducts research, develops, and publishes music software tools. As Peter began looking for new opportunities to advance his music and coding research, he joined the MediMuse™ team to support, materialize and implement a shared vision with Robert Joseph for StateSong® Technology.

---

If you are interested in joining the growing number of researchers using the E4 wristband to collect real-time physiological data safely and continuously, you can reach out to us at research@empatica.com. And to learn more, simply visit our dedicated webpage.