The Project
This project is made possible by a grant from the American Orchestras’ Futures Fund, a program of the League of American Orchestras made possible by the generous support of the Ann and Gordon Getty Foundation.
Our vision for the generous grant provided by the League of American Orchestras’ Futures Fund is a two-year partnership with Nokia Bell Labs in which we research and experiment with bidirectional communication at live concerts.
Traditionally, concerts provide one-way communication from musicians to the audience. With the help of the Experiments in Art and Technology (E.A.T.) lab at Nokia, we are exploring new ways for performers to sense the response of the audience in real-time through technology. Through clarifying the bridge between music and technology, we hope to involve audiences already passionate for live music, attract new audiences less familiar with orchestral work, and engage student performers in the creative aspects of the performance.
C4C is a technology developed by E.A.T. that converts audiences’ to speakers, enabling the audience to become “instrumentalists” of the performance. The performers can play sounds through the audience’s phones in synchrony with the ongoing performance. By establishing this flow of music communication through C4C, we aim to promote what E.A.T. describes as “empathetic communication,” a shared immersive experience for audience members and performers in classical concerts.
The Pieces
Written for string orchestra, Dr. Patricio Molina’s Homage to Khalil Gibran pays tribute to his middle-eastern heritage by honoring Lebanese-American writer and poet Kahlil Gibran, author of famous prose work The Prophet. The Prophet deals philosophically with the human condition, which Dr. Molina’s piece captures and reimagines. Employing C4C, Dr. Molina incorporated various middle-eastern instruments such as the darbuka and kanun, vocalizations, and even a reading of the passage “On Talking” from Gibran’s The Prophet, which were then layered alongside the ongoing concert.
“There’s this one line from Khalil Gibran’s The Prophet, and it reads “And there are those who have the truth within them, but they tell it not in words / In the bosom of such as these the spirit dwells in rhythmic silence” and we thought gosh, that’s exactly what we’re trying to prove, so we just took this idea of talking and meaningful conversation and tried to have this dialogue going on between the audience and performers.”
Homage to Nicanor Parra by Dr. Patricio Molina reflects his Chilean heritage by commemorating the famous Chilean poet Nicanor Parra. This piece draws upon Parra’s poem “El Hombre Imaginario (The Imaginary Man)”, as well as the traditional Chilean rhythm and Cueca style. With C4C, we provided auditory, visual, and haptic representations of Parra’s poetry through media of percussion sound samples, Philips Hue lights, and sensory feedback motors, all of which sought to mimic the musical traditions of Chile and provide the audience an immersive experience.
“To us, this piece emulated the vibrant, energetic nature of Chilean music and culture more than anything else. We wanted to reinforce the moving rhythms and dynamic changes by using visual and haptic aids, which came to us in the form of Hue lights as well as little haptic feedback motors to provide the audience with more than just music. We’re super happy with how we were able to accentuate this amazing piece!”
Spiral, written Dr. Mesia Austin, NJYS’s very own percussion director, embodies her contemplation of the chaos—and peace—of the COVID-19 lockdown. Taking root in the organic, meditative sounds of clarinets and percussion instruments, Dr. Austin’s piece is supplemented with “soundwaves” of a natural, forest-based soundscape. The noise of waterfalls, raindrops, and other nature sounds can be heard moving throughout the room via speakers and the audience’s phone speakers. As an additional supplement to the performer-audience connection, biofeedback stations were set up at the performance to analyze audience members’ heart rates as the music swelled and diminished, visually measured by a change in the lights’ color schemes of shifting blue to green hues as measured heart rate patterns rose and fell throughout the shifting tempi of the piece.
“We wanted to achieve a transformation of the space audiences were listening in, which is why we wanted to emulate as much as possible a natural ‘soundscape’ with realistic and calming nature audio clips, as well as callbacks to a forest or grotto sort of area with calming tones of blue and green achieved through our biofeedback-centered lights. We’re personally super happy with what we managed to accomplish in the technology aspect of this stunning piece!”
The Team
Dr. Patricio Molina, the composer of Homage to Kahlil Gibran and Homage to Nicanor Parra, is a composer and musician born in Chile to Chilean and Syrian parents. He is a composer and a teacher at Rutgers University, where he has received two doctorates in piano performance and music composition.
Dr. Mesia Austin, the composer of Spiral, is the Director of Percussion and Conductor of the Percussion Ensemble at the New Jersey Youth Symphony. She is also an adjunct professor of percussion at Rutgers University from which she received her Doctor of Musical Arts in 2016. She is currently a percussion teacher, director, and composer.
Maestro Helen Cha Pyo is the conductor of the New Jersey Youth Symphony. We have her and the NJYS team to thank for receiving such a generous grant from the League of American Orchestras, bringing together Bell Labs and NJYS, and performing our music to a live audience.
Danielle McPhatter is an interdisciplinary programmer, musician and artist that works as a researcher in the Experiments in Art and Technology lab at Nokia Bell Labs. In her work, she is interested in alternative modes of interactive storytelling that explore the intersections and various modes of connectivity between media, humankind, and technology.
Ethan Edwards is a Researcher in Experiments in Art and Technology (E.A.T.) at Nokia Bell Labs. He works directly with scientists and artists to help facilitate collaboration and builds technology which spans both worlds. He is an artist himself graduating with an MFA in Sound Art from Columbia University and has had work featured in various museums, galleries, and performances around the world.
Jimmy Chen is a graduating senior at Bridgewater-Raritan High School. “As someone who’s always wanted to explore new opportunities in live concert performances, I'm so excited to be working on this project with E.A.T. researchers. Exploring the intersections between music and technology has been so insightful in deepening human connections and exploring emotions, and I hope to continue pursuing these passions in the future.”
Abhinav Datla is a graduating senior at Sayreville War Memorial High School. “As an enthusiast of orchestral music, it feels absolutely thrilling to be taking the genre into the future and exploring ways to change the audience/performer dynamic. It is also an inspiring experience to combine my lifelong passions in technology and innovation, as well as music as a percussionist.”
Ryoma Takenaga is currently a junior The Academy for Information Technology. “This NJYS/Nokia project is the culmination of all my music studies since age 5 and my current focus at the Academy for Information Technology. Music and technology are the two things that I am most passionate about and plan to pursue as a career. Being a part of this innovative project is exciting to me because it provides a wonderful opportunity to integrate both my tech skills and musical creativity. Here is a unique chance to collaborate, and contribute to the creation of music that will enhance the experience for both performers and audience.“
Brian Zhang is a graduating senior at The Academy for Mathematics, Science, and Engineering. “I’m Brian, one of the two lead student technologists for NJYS’ collaboration with Nokia Bell Labs. So far, music and technology have been two very separate influences to me, and I can’t wait to see how we can bring these two disciplines together in a way that will make NJYS concerts even better!”
Samantha Liu is in her junior year at Ridge High School. As a violinist from NJYS’ Youth Symphony, she believes strongly in the communicatory nature and emotional benefits of music. She found the Experiments in Arts and Technology at Nokia extraordinarily cool for exploring music through a discipline most consider contradictory: engineering. She hopes this intersection holds a way to retain empathetic connection in a technology-driven future.
