WORLD FIRST NEUROMORPHIC IN-CABIN SENSING
ENABLED BY PROPHESEE METAVISION® TECHNOLOGIES
Advanced drowsiness & microsleep
prediction
Early neural degenerative disease
detection
Greater accuracies
on saccadic motion of the eye
Micro expressions
monitoring
Equal reliability
as existing near infrared based systems
Eye analytics
with focus on blink frequency and duration
Video Courtesy : Xperi Corporation
World first Neuromorphic Driver Monitoring Solution in action
DTS, a wholly owned subsidiary of Xperi Holding Corp., has collaborated with Prophesee on bringing the power of event-based vision to in-cabin sensing in vehicles. The efficiency and accuracy of the Metavision technology helps DTS create the world’s first neuromorphic driver monitoring solution (DMS).
With a focus on privacy by definition, the system complements standard vision systems in cars, and enhances occupant safety and increases personalization opportunities to improve the overall user experience.
The DTS system takes advantage of the highly efficient data processing capabilities of the HD Event-Based Vision sensor realized in collaboration between Sony and PROPHESEE, using the raw feed to enable specific driver monitoring features such as gaze tracking, head pose, identification, and eyelid opening.
The neuromorphic sensors capture information at an equivalent frame rate of 10,000 frames per second without requiring active illumination. This enables better low light performance for driver monitoring features.
Training of neuromorphic models with driving simulator data as well existing frame-based data
10,000 frames per second
without requiring active illumination
Excellent low-light
performance
Designed with privacy
at the core
Motion blur
absent
Extremely accurate
eyelid movement
Ultra fast
response times
The innovation uses event cameras to enable in-cabin sensing solutions since event-based vision has some key advantages over traditional approaches. The reliability is equal to existing near infrared based systems where traditional methods perform well while outperforming them where traditional systems perform poorly.
By taking advantage of the rich temporal information and high dynamic range provided by event-based vision sensing, the system achieves greater accuracies on saccadic motion of the eye, extremely accurate eye lid movement, pupil analytics, and very precise eye closing intervals. These inputs are used for prediction of drowsiness and impairment providing increased reliability and accuracy.
Using existing datasets for core DMS features
Advantages of using event simulators
Time event-based vision solutions enable faster response times, do not suffer from motion blur, provide higher dynamic range, and near analog continuous representations of the events. The custom neural architectures developed permit dynamic pseudo-frame rates, performing inference only when needed and thus can adapt to the changing cabin environment.
In addition to typical in-cabin monitoring use cases, time event includes advanced drowsiness and microsleep prediction, early neural degenerative disease detection, sudden sickness, and impairment detection.
Near analog continuous
representations
of events
Adaptation to changing cabin
environment
Sudden sickness, and impairment
detection
“Being able to instantaneously detect the subtlest, almost imperceptible, face and eye motions can be lifesaving,” “The low light powerful vision processing capabilities, fast response times and low power operation that this pioneering neuromorphic-based technology application enables makes it ideal for the safety-critical applications of driver monitoring systems, while also broadening the ability to further enhance and personalize the vehicle cabin.”
Petronel Bigioi, CTO Product, Xperi
In addition to excellent low light performance and robustness to various illumination conditions, neuromorphic sensing addresses head-on the privacy aspect specific to traditional imaging-based sensing (always-on cameras). Xperi’s implementation has 12 patents pending and relies on an end-to-end data generation and training system built to address the particularities of the sensor.
The training data set was generated based on Xperi’s extensive computer vision infrastructure, reusing ground-truth from the visible and near infrared spectrums, synthesizing a completely novel approach in neuromorphic based sensing.
Data acquisition of drowsiness, cognitive load, emotional state and other important data for driver safety
Deep dive into the neuromorphic Driver Monitoring Solution by Xperi
ABOUT XPERI HOLDING CORPORATION
Xperi invents, develops, and delivers technologies that enable extraordinary experiences. Xperi technologies, delivered via its brands (DTS, HD Radio, IMAX Enhanced, Invensas, TiVo), and by its startup, Perceive, make entertainment more entertaining, and smart devices smarter. Xperi technologies are integrated into billions of consumer devices, media platforms, and semiconductors worldwide, driving increased value for partners, customers and consumers.
Xperi, DTS, IMAX Enhanced, Invensas, HD Radio, Perceive, TiVo, DTS AutoSense, and their respective logos are trademarks or registered trademarks of affiliated companies of Xperi Holding Corporation in the United States and other countries. All other company, brand and product names may be trademarks or registered trademarks of their respective companies.
ABOUT PROPHESEE INVENTORS COMMUNITY
Since 2014, a network of researchers, start-ups and companies have shown incredible imagination and innovation using Prophesee’s neuromorphic vision technologies.
This has created an Event-Based Vision ecosystem of inventors sharing their work and ideas. Their creativity with Prophesee’s technologies inspires us.
We are gathering them here to inspire future inventors in turn, in the hope that, like the projects here, they create something new together and reveal the invisible.
