In its second year: NJYS Exploration of Music,
Technology & Communication
This project is made possible by a grant from the American Orchestras’ Futures Fund, a program of the League of American Orchestras made possible by the generous support of the Ann and Gordon Getty Foundation.
Our Mission
This project chiefly addresses the disconnect which exists in a classical concert between audience members and the orchestral musicians. Our team mentioned that, as musicians, they felt the communication at a performance only flows in one direction: the orchestra delivers a performance, and the audience quietly receives it. We wanted to create instead an outlet for the audience members to engage with the performers throughout the piece, facilitated by various technological enhancements on the audience’s phones and throughout the concert hall.
OUR PIECE: Dream a New World
by Dr. Patricio Molina
Our Team
Samantha Liu
Samantha Liu is a violinist with Youth Symphony and a senior at Ridge High School. She joined the NJYS Exploration of Music, Technology & Communication Project after experiencing firsthand the capacity for music to bring individuals together in an era of digitizing human communication. She appreciates the perspectives the engineering work has offered her, as well as the chance to work with an intelligent and wonderful team.
Ryoma Takenaga
Ryoma Takenaga is a bassist in the New Jersey Youth Symphony Jazz Orchestra and a senior at the Academy for Information Technology. He joined NJYS Exploration of Music, Technology & Communication Project because he was interested in the intersection of music and technology. Ryoma enjoyed the creative nature of this project. Brainstorming new ideas that may impact the way musicians and audiences consume/interpret music in the future compels him to approach music from a new perspective.
Samhita Tatavarty
Samhita Tatavarty is a bassoonist with Youth Symphony, a senior at Ridge High School, and in her second year working on the NJYS Exploration of Music, Technology & Communication Project. She is passionate about its interdisciplinary facets, and appreciates the freedom they give her to explore under-researched overlaps between technology and the arts. Overall, she has greatly enjoyed her time working on all aspects of the project with her phenomenal group of peers and mentors, and is grateful for the knowledge the experience has given her.
Lukas Kebuladze
Lukas Kebuladze, a junior at Franklin High School and cellist with Youth Symphony, was inspired to become a part of the NJYS Exploration of Music, Technology, and Communication project through prior study of music composition. His interest in this subject led to an exciting opportunity for him to work as a team with other like-minded individuals in order to further deepen the connection between performers and the audience.
Benjamin Chee
Benjamin Chee is a junior at Ridge High School and Bb Clarinet player with the Youth Symphony. He was eager to participate in this Exploration of Music, Technology, and Communication project to explore the connection between musicians and the audience—in a typical concert setting, the musicians produces the music that the audience then receives, but he wondered if modern technology could be applied such that the audience could produce something that the musicians receive.
Carmen Dai
Carmen Dai is violinist with the Youth Symphony and a junior at Newark Academy. She joined the NJYS Exploration of Music, Technology & Communication Project to show the audience what music means to her through a combination of her passions, such as art and technology. Along with her wonderful team members, Carmen aspires to bring their joy and enthusiasm towards this project to the members of YS, the audience members, and contribute to the ever growing bond between musician and listener.
Samvit Singhal
Samvit Singhal is a senior at New Providence High School and a pianist for the NJYS Jazz Orchestra. After missing the live music experience for two years, Samvit wanted to see how technology could be used to create an even better experience for musicians and audiences alike. He joined this project to explore how humans engage with live music and how differnet elements can augment or diminish the emotional response to music.
Dr. Patricio Molina
Dr. Patricio Fadel Molina, Ph.D., D.M.A. is a Chilean-Syrian composer, pianist, and educator based in New Jersey, USA. He is known for his virtuosic performances, and culturally competent and dynamic compositions. He is a leader in arts education in New Jersey, and an ardent advocate for the arts in the NYC area and around the world. Patricio is committed to serving marginalized communities and working toward decolonizing academic research. He is in demand as a lecturer on a wide variety of topics including composition, Arabic music, and teaching composition for performers. His accomplishments in the field of arts education have been recognized by the New Jersey Education Association with their Award for Excellence. Patricio holds a Master’s Degree with distinction from the Manhattan School of Music, a D.M.A in Piano Performance, and a Ph.D. in Music Composition from Rutgers University. He serves as Associate Director of the Conservatory for Newark School of the Arts, and Music Director at St. Thomas the Apostle Church of Bloomfield, New Jersey. He serves as Adjunct Professor of Music at Passaic Valley Community College. He is co-founder and President of Notes for Growth Foundation.
Ms. Danielle McPhatter
Danielle McPhatter is an interdisciplinary programmer, musician and artist interested in alternative modes of interactive storytelling that explore the intersections and various modes of connectivity between media and technology. Danielle expresses her passions for the sonic and visual arts via the creation of interactive narrative experiences, games and installations. Her experience with virtual and augmented realities, motion tracking, creative programming and physical computing in both the visual and audio domains combine with her knowledge of a variety of programming languages and software environments to forge new forms of interaction with technology.
Ms. Helen Cha-Pyo
Helen H. Cha-Pyo is currently in her fourth season as the Artistic Director and Principal Conductor of the Wharton Institute for the Performing Arts (WIPA). Under her artistic leadership, WIPA serves over 1,400 students from 13 different counties of New Jersey through a range of classes at Wharton Performing Arts School, 15 performing ensembles at New Jersey Youth Symphony and Paterson Music Project, an El Sistema-inspired program serving over 400 children in the city of Paterson, NJ. As principal conductor of NJYS Youth Symphony, she has led the orchestra in prestigious concert halls including Richardson Auditorium, NJ State Theatre, Union County Performing Arts Center, Kasser Theater and New Jersey Performing Arts Center, including the 40th anniversary performance of Carmina Burana with six local high school choruses.
Our Approach
More specifically, our approach is built around C4C (Compositions for the Collective), a mobile web-app which syncs together the audience members’ phones and converts them into live speakers during the concert. We programmed into this app moments where the audience is prompted to answer a question gauging their reaction to the piece so far: bold, reverent, melancholy, etc. Based on their answers, different sounds would be cued for the musicians to hear and improvise upon, as well as a visual that reflected the reaction that they chose. Through this call-response system, we hoped to “flip the switch” on a typical concert, giving the listeners a chance to affect and shape the ongoing music.
We also employed C4C to play aloud different voice samples on the phones of the audience members, featuring our student musicians describing what their new world looked like with responses varying from a place where speed limits don’t exist, to places of self-acceptance and empathy. Since the musicians normally only function as members of a larger ensemble, we hoped that, through this enhancement, student musicians would be able to express themselves personally and let the audience better glimpse their lives beyond the music, further strengthening the bond between the participants of a concert.
Components
Lights
We had Philips Hue bulbs arranged around the concert hall, which changed color to match the piece and cue the audience to interact with their phones.
Voices
The musician’s voices were played aloud from the audience’s phone, each describing a unique personal vision of a new world.
Improvisation
Based upon the audience’s reaction to the piece, a sound sample is played on speakers for the musicians to improvise upon, thus creating a call-response conversation between the listeners and players.
A.I. Art
The audience will be prompted to select one of six emotions from their phones. An AI-generated image will appear based on the emotion selected.
Impact
This was an enriching experience for both the team and the audience. We were able to enhance the connection between the audience and musicians through the music, and the audience gained a new perspective of receiving music through an innovative, modernized approach. As one of the interviewers mentioned, the technology made this piece seem to have a new kind of style, increasing the novelty and impact of this project.
For the members of the orchestra, it was rewarding to have their recorded voice memos be played on speakers during the concert since their voices are rarely heard in a normal music concert. They felt more “heard” as individuals and created a more personal connection in being able to share their stories.
Results
Here are some of the responses that we received from the audience members when they were asked what their favorite parts of the performance were:
- “The change in colors with the intensity of the music”
- “I liked being able to interact with the performance”
- “I enjoyed having the audience polled about how they felt.”
- “My favorite part was when the crescendo hit the climax and the drums came in. It gave me goosebumps every time. It was special because it made a clear image in my mind. The improvised parts were so beautifully and added so much emotion to the piece.”
- “I found it very impactful listening to everyone’s “new world” it was very emotional and real to hear everyone’s hopes for a better future.”
What was your favorite technological component?
94 responsese
Lighting 23.4%
Sounds (voices) 60.6%
Challenges
Due to challenges imposed by the COVID-19 pandemic, meeting and working together as a team became a challenge, as we had to meet online over Zoom for most of the project. It was a challenge to coordinate everything asynchronously in an online and remote format since it was hard to communicate with other team members.
As members of the technology group, we learned how to utilize new interfaces, such as C4C technology, in our project. For many of the ideas we had planned, we had to wait until the last few orchestra rehearsals to test what we had worked on, such as the stage lights. The lights in the back were not bright enough, so we moved white curtains to amplify the lights in the back of the stage. In the end, everything came together wonderfully.
Final Take-Aways
It is seldom that band, orchestra, and jazz musicians get to work so closely on a project. However, this music and technology project made this collaboration a reality. We used our different backgrounds in music and our different technical expertise to create a project with multiple components that engaged the audience. Because of this, we unanimously found that this was an incredibly rewarding (and challenging) hands-on experience.
We also realized the importance of bidirectionality in music, which is the relationship between the musicians and the audience. Members of the audience raved about the improvisations that the musicians had in response to the most popular emotion that the audience selected. They also thoroughly enjoyed hearing the voices of the musicians, something that rarely ever happens in a normal concert. At the same time, the voices also added a personal touch to the musicians, who are often more ‘working parts’ than individuals within an orchestra.
Lastly, through our experience working directly with Dr. Patricio Molina’s composition, we came to appreciate technology as a storytelling medium–not only enhancing a piece’s melodies and rhythms, but also the cultures, backgrounds, and perspectives which influenced the composer. Through this, we hoped that we gave all members of the performance a more lasting, more resonant concert experience, bringing everyone closer through music.
Looking Forward
Members of the team want to continue research on this topic by looking into how various demographics are affected differently by technological additions. For instance, would our enhancements be taken positively by someone who grew up in the digital age, but not an older generation individual? Alternately, could the presence of classical music help acclimatize someone more unfamiliar with technology? The team is also curious about how we can condense our project to make it simpler and what new technologies, such as psychoacoustics, we can use to make the process more intricate yet simple.