VR experience
Enter the Autism Experience
Immersive walk-through that approximates auditory and visual overload from a first-person perspective.
Augmented Human Lab · Labothon · Team 02
WhatItFeelsLike
Seeing the world through the lens of others.
A research prototype that takes everyday video of a candidate environment, surfaces the sensory triggers it contains, and renders a sensory-modulated version of the same clip — so families and practitioners can make informed choices about autism-friendly spaces before a site visit.
01 — Motivation
Imagine choosing a place for a group of friends.
A restaurant. An AirBnB. The space looks fine in photos — but for an autistic person, the same room may be saturated with sensory triggers that turn a casual visit into a draining experience.
- Echoey rooms amplify chatter into a wall of noise.
- Crowds layer overlapping voices that are hard to filter.
- Vibrant, cluttered scenes make it hard to maintain focus.
02 — Research question
The question we set out to answer.
How can we generate sensory-modulated videos from everyday video footage to help make informed decisions about autism-friendly environments?
03 — Prior work
Existing approaches.
Prior efforts invite neurotypical viewers to step into a VR simulation. The thesis is sound — the best way to learn something is to experience it — but every viewer has to gear up.
VR experience
Sensory overload in VR
Generative bursts of colour and motion staged in everyday spaces — malls, classrooms, public concourses.
We wanted to use everyday video — recorded with a phone — as the substrate, so a parent or therapist can run an assessment in two minutes.
04 — Our system
From candid video to a sensory readout.
Input
Real-world video footage of any place of your choice — restaurant, AirBnB, classroom, lobby.
Output
A modified video that identifies common stressors and visually modulates the sensory effects of the original, plus a structured JSON report.
Frame
Cognitive Amplification. We augment the viewer's ability to understand the perception of others — surfacing what is already in the scene rather than taking anything away.
05 — Sensory manifestations in autism
What we modulate, and why.
Hypersensitivity to visual effects
- Artificial lighting (brightness, flicker)
- Increased perceived brightness; visible flicker.
- Specific colours
- Saturated colours appear brighter than they actually are.
- Visual motion
- Different scenes and colours can merge in perception.
Hypersensitivity to audio effects
- Background noise (hyperacusis)
- Ambient noise is amplified; silence is rare.
- Sharp, transient sounds
- High-pitched onsets stand out painfully.
- High-ceiling rooms
- Long reverberation tails make speech hard to follow.
Parmar KR, Porter CS, Dickinson CM, Pelham J, Baimbridge P, Gowen E. Visual Sensory Experiences From the Viewpoint of Autistic Adults. Front Psychol. 2021 Jun 8;12:633037. doi: 10.3389/fpsyg.2021.633037
06 — Pipeline
How a clip becomes a report.
upload→
demux (ffmpeg)→
video_analyzer (OpenCV)→
audio_analyzer (librosa)→
processed render→
mux + report
Orchestrated by
analyzers/main_processor.py; HTTP surface in
routers/analyze.py; entry point main.py.
Per-job artefacts are written to results/<job_id>/.
07 — What we detect
Visual and audio triggers, mapped to research.
Visual triggers
| Light flicker | Frame-brightness variance — proxy for fluorescent / PWM flicker |
|---|---|
| Brightness | Mean luminance, scored 0–10 |
| Colour saturation | HSV saturation distribution + dominant-colour palette |
| Visual clutter | Edge density & pattern repetition |
| Motion | Inter-frame difference magnitude |
Audio triggers
| Background level | RMS → estimated dB SPL, mean & peak |
|---|---|
| Sudden events | Energy-spike onsets with timestamps |
| Low-frequency hum | Spectral mass below ~120 Hz (HVAC / mechanical) |
| High-frequency content | Spectral mass above ~4 kHz (beeps / alarms) |
| Speech intelligibility | SNR-based estimate |
08 — Output schema
What lands on disk per job.
{job_id}_video.mp4— demuxed video stream{job_id}_audio.wav— extracted audio{job_id}_video_processed.mp4— overlays / annotations applied{job_id}_audio_processed.wav— re-rendered audio{job_id}_final.mp4— muxed deliverable, shown as the right-hand player on the Results tab{job_id}_report.json— structured report (visual / audio / timeline / concerns / recommendations)
09 — Scoring
Sensory load on a 0–10 scale.
Each modality (visual, audio) yields a 0–10 sub-score, combined into an overall sensory score on the same scale. Lower is calmer; higher indicates more potential triggers. The report also exposes a per-second sensory_timeline so peaks within a clip can be inspected.
10 — Limitations
What we don't yet capture.
- Visual motion effects beyond inter-frame difference are not modulated — sub-frame motion artefacts are missed.
- Other sensory channels — smell and touch — are out of scope.
- Many autistic individuals report physical reactions (a felt pain, sometimes likened to being punched) alongside sensory manifestations. We surface the visual / auditory cues; we don't simulate the somatic response.
11 — Future contributions
Where this is going.
- Inclusive design standards. Parents, therapists, and architects can use these readouts to avoid environments that exceed a person's threshold — fewer sensory overloads, fewer aborted visits.
- Smart-wearable integration. Move beyond uploaded clips to a live camera feed or real-time monitoring of an environment.
12 — Caveats for researchers
Read this before citing the numbers.
- dB SPL is estimated from recording RMS without calibration — useful for relative comparison only.
- Flicker detection is bounded by the source frame rate; sub-frame flicker is invisible to this method.
- Sensory profiles vary across individuals; the score is an environmental aggregate, not a personalised prediction.
- Job IDs are 8-char UUID prefixes (not collision-proof for very large sample sets).
13 — Reproducibility
Where the data lives.
Source uploads live under uploads/, processed artefacts under
results/{job_id}/. The
Results
tab pairs every input with its processed output and surfaces the
parameters extracted from the report for visual comparison.
Loading jobs…