Sensorium Ex: Redefining Opera

The curtain rises not just on a stage, but on a revolution.

In a world increasingly shaped by algorithms and sensors, the human spirit, in its most expressive form, finds a new voice.

Imagine the grandeur of opera, its soaring melodies and dramatic narratives, now performed by artists previously deemed unable to participate due to physical challenges.

This isn’t a speculative dream; it is the breathtaking reality of Sensorium Ex, an audacious new opera that has redefined the very essence of performance, proving that innovation, compassion, and sheer human will can shatter the most formidable barriers.

For seven arduous years, librettist Brenda Shaughnessy and composer Paola Prestini, under the astute direction of Jerron Herman and Jay Scheib, poured their vision into Sensorium Ex.

Their mission: to tell the poignant story of a mother and her non-verbal, non-ambulatory son, with a cast predominantly composed of disabled performers, many with impaired or limited speech.

The sheer audacity of the premise alone would give most producers pause, but for this team, it was a call to innovation.

The traditional operatic mold, predictable and linear for centuries, was about to be reshaped by dedication, creativity, profound philanthropic support, and, crucially, cutting-edge technology.

At the heart of this technological marvel is Dr. Luke Dubois, Associate Professor of Integrated Design & Media at NYU Tandon School of Engineering, and Co-Director of the Ability Project.

His laboratory, a vibrant nexus of art and engineering, feels less like an academic space and more like the kind of studio where musical legends once forged their sonic masterpieces.

It is here that the seemingly insurmountable challenge of giving nuanced, emotional voice to non-verbal performers was tackled.

The existing landscape of Augmentative & Alternative Communication (AAC) devices, pioneered by visionaries like Ray Kurzweil and famously utilized by Stephen Hawking, offered functional communication.

Yet, their output, while revolutionary for its time, remained undeniably robotic – a stark limitation for a medium like opera, where every cadence, every inflection, every breath conveys a universe of emotion.

Dr. Dubois’s collaboration on Sensorium Ex began five years ago, sparked by Prestini’s request for a technological bridge between intent and operatic expression.

The solution devised by Dubois and his team is a two-pronged attack on the problem of vocalization.

First, leveraging artificial intelligence to personalize speech.

For performers with conditions like apraxia, where the brain intends clear speech but the vocalization emerges jumbled, a combination of speech editing software like ProMoNet and neural vocoders transforms these sounds into coherent, albeit initially robotic, speech.

The crucial humanizing element is then injected by Professor Mark Cartwright of NJIT, who uses machine learning and AI to imbue the synthesized voice with the individual’s unique sound and speech characteristics, making it truly their own.

The second, and perhaps most captivating, innovation lies in enabling performers to inject emotion, tempo, emphasis, and vocal stretch in real-time.

This is where the magic of 3D sensing comes into play.

Enter Eric Singer, a Brooklyn-based musician, engineer, and self-proclaimed “mad scientist,” whose company, Singertronics, birthed the “Ther’minator.”

A clever nod to the Theremin, an instrument played without physical contact, the Ther’minator transforms LiDAR (Light Detection and Ranging) signals into modular voltages.

Using remarkably low-cost Time of Flight (ToF) sensors, typically found in everyday applications like automobile airbags, the Ther’minator captures the subtle and macro movements of a performer’s hands.

These hand gestures, translated into electrical signals via a board ingeniously designed by production designer Andrea Lauer, are then fed into a computer.

Here, the LiDAR output voltages are seamlessly integrated with the performer’s AI-personalized speech, calibrated uniquely for each individual to match their arm movements with their vocal intent.

The spectacle of Jakob Jordan, who plays the lead role of Kitsune, the son, practicing with the sensor board, is a testament to this fusion of art and engineering.

His hand movements, once perhaps a silent language, now conduct a symphony of digitally enhanced vocal expression, bringing his character to life with unprecedented depth.

The Omaha premiere in May 2025, with four sold-out shows accommodating 150-200 attendees each, and seating thoughtfully configured for a spectrum of disability conditions, was more than just an opera; it was a societal statement.

Paola Prestini, a driving force behind Sensorium Ex and leader of non-profits National Sawdust and Vision Into Art, reflected on the seven-year odyssey.

Her insight was profound: the project was both unlike and precisely like her other productions.

Unlike, because there was no template; every logistical and technical hurdle, from integrating live caregivers on stage to convincing skeptical opera venues, had to be navigated for the very first time.

Like, because the paramount focus remained on uncompromising quality and excellence in performance.

The advice from their Disability Advisory Board was clear: no compromises on artistic merit.

This commitment, Prestini believes, is critical for future success and audience acceptance, ensuring that Sensorium Ex is seen not as a novelty, but as a legitimate and powerful artistic endeavor.

The impact of Sensorium Ex reverberates far beyond the stage.

By making all its technology open-source, the production has laid down a gauntlet, creating a blueprint for all-inclusive art performances globally.

This monumental disruption in the fusion of art and technology begs a compelling question: will mainstream productions like Phantom of the Opera or Hamilton soon embrace such talent, enhanced by assistive performance technologies?

The potential is immense, not just for the arts, but for broader quality-of-life enhancements, from physical therapy to rehabilitation, where similar sensing and AI technologies could unlock new levels of human potential.

The accelerating trend of robotics, 3D sensing, AI, speech enhancement, and movement automation is not merely about convenience; it is about elevating the quality of life for all humanity.

Sensorium Ex stands as a powerful testament to this truth, a symphony of human ingenuity and artistic courage that has not only given voice to the voiceless but has opened a new, vibrant chapter in the ongoing narrative of human expression.

It is a reminder that when technology serves empathy, the possibilities are truly limitless.