AI and Opera: Giving Voice to the Voiceless

The stage lights dim, a hush falls over the audience, and then, a voice emerges – rich, nuanced, full of operatic emotion.

Yet, this is no ordinary performance.

The voice belongs to a non-verbal performer, his every inflection, every subtle change in pitch and cadence, sculpted not by vocal cords alone, but by the intricate dance of his hands, translated into sound through a revolutionary blend of artificial intelligence and 3D sensing.

This is the world of Sensorium Ex, an opera that premiered in Omaha, Nebraska, in May 2025.

In doing so, it didn’t just break new ground for the arts, but fundamentally redefined what it means to give voice to the voiceless.

For seven years, librettist Brenda Shaughnessy and composer Paola Prestini, under the direction of Jerron Herman and Jay Scheib, meticulously crafted this extraordinary production.

It tells the deeply moving story of a mother and her non-verbal, non-ambulatory son, brought to life by a predominantly disabled cast, many with impaired or limited speech.

The journey was less a typical operatic production and more a monumental act of collaborative innovation, demanding dedication, time, focus, and, crucially, a technological leap that seemed, until recently, confined to the realm of science fiction.

At the heart of this technological marvel is Dr. Luke Dubois, Associate Professor of Integrated Design & Media at NYU’s Tandon School of Engineering and Co-Director of the Ability Project.

His laboratory, described as a creative hub where one could imagine rock legends recording, is precisely where the fusion of art and engineering takes on a tangible form.

Dubois, with a Ph.D. in music and a deep background in electronics and speech synthesizers, was tasked by Prestini five years ago with an audacious challenge: to enable speech-challenged artists to sing opera.

The history of assistive communication devices, or AAC, dates back to Ray Kurzweil’s invention in 1975, which used Optical Character Recognition to help the visually impaired “listen” to text. Later, Intel, in collaboration with Stephen Hawking, developed a system where text input and eye-gaze movement translated into synthesized speech.

While groundbreaking for its time, the output was, by all accounts, robotic – a significant hurdle for an art form like opera, where nuance, emotion, and artistic personalization are paramount.

Dr. Dubois and his team devised a two-step strategy to overcome this limitation. First, they tackled the personalization of speech using AI.

For performers with conditions like apraxia, where jumbled speech is vocalized despite the mental intent of speaking normally, a combination of speech editing software (ProMoNet) and a neural vocoder transformed these sounds into coherent, albeit initially robotic, speech. The critical humanizing element came from NJIT Professor Mark Cartwright, who employed machine learning and AI to inject the individual’s unique sound and speech characteristics, transforming the sterile robotic output into something recognizably human.

The second, and perhaps most revolutionary, step involved injecting individual creativity and emotion into the performance.

How do you convey operatic emotion – the crescendo of passion, the whisper of sorrow – when traditional vocal control is limited?

The answer emerged from an unlikely source: hand movements.

Eric Singer, a Brooklyn-based musician, engineer, and self-proclaimed “mad scientist,” designed and built the “Ther’minator.” A clever spin on the Theremin, an instrument played without physical contact, the Ther’minator uses low-cost Time of Flight (ToF) sensors – the same technology found in car airbags to detect human presence – to transform LiDAR sensing signals from hand movements into modular voltages.

These signals, captured by three Ther’minator units integrated with a custom-designed board by production designer Andrea Lauer, are fed into a computer. There, they merge with the performer’s personalized, AI-trained speech.

A crucial calibration process ensures that each performer’s unique arm movements are accurately correlated with their vocal intent, allowing for real-time modulation of cadence, emotions, and pitch.

Watching Jakob Jordan, who plays the lead role of Kitsune, the son, in rehearsals, one witnesses a profound transformation: a gestural language becoming a vocal one, a bridge between physical expression and operatic artistry.

The Omaha premiere of Sensorium Ex was more than just a series of performances; it was a revelation.

Paola Prestini reflected on the seven-year journey, emphasizing that while the pursuit of uncompromising quality and excellence in performance was no different from her other productions – a point insisted upon by the opera’s Disability Advisory Board – the process itself was an unprecedented journey of learning and listening. There was no existing template for an opera featuring non-verbal performers.

Logistical hurdles abounded, from integrating live caregivers or voice coaches on stage to convincing opera venues to embrace such a pioneering production. Even sourcing talent required ensuring the performers’ families were deeply invested in the success of the endeavor.

Prestini sees Sensorium Ex as a monumental milestone, not just for her personally and for the cast, but for the future of opera itself.

By making all the underlying technology open-source, the production aims to create a global environment for inclusive art performances. This isn’t merely about enabling a few artists; it’s about setting a new standard, a blueprint for other productions and opera houses to follow, fostering a world where artistic expression knows no physical bounds.

The implications of this artistic and technological fusion extend far beyond the stage. This effort represents a monumental disruption, hinting at a future where speech-challenged artists could grace mainstream productions like “Phantom of the Opera” or “Hamilton,” enhanced by assistive performance technologies.

Furthermore, the technology demonstrated here holds immense promise for broader quality-of-life enhancements, from physical therapy and rehabilitation to daily communication.

The very LiDAR technology now critical for autonomous vehicles is finding applications that profoundly impact human lives in the most dramatic and unexpected ways.

Future advancements, such as Frequency Modulated Continuous Wave (FMCW) LiDAR for detecting axial velocity of hand movements, event-based cameras for low-latency tracking, or smart glasses for eye-tracking, promise to further revolutionize this field.

The accelerating trend of technology assisting physically challenged individuals, integrating robotics, 3D sensing, AI, speech enhancement, and movement automation, is not just a niche development; it is a profound movement towards elevating the quality of life for all humanity.

“Sensorium Ex” stands as a powerful testament to this truth, a stirring reminder that when human creativity merges with technological ingenuity, the boundaries of possibility simply cease to exist.