Reality Synthesizer is an open-source app that uses iOS 15.4's LiDAR + a synth's MIDI signals to visualize music as it's played.
iOS 15.4 opened up the ability to tap into the LiDAR sensor using just AVFoundation. The result is more accurate depth maps & depth data without having to use ARKit and all the overhead that entails. Faster performance. Simpler code.
Enabling (simplistic) augmented reality experiences while simultaneously achieving better performance and incurring less overhead would be important for any potential head-mounted hardware where, my guess is, battery drain will be of the utmost concern.
For many years I've been thinking about ways people can interact with augmented reality – or actually augment their reality. One way is through interactive audio storytelling (I'll have more on that soon over at [ Ссылка ]).
Another way is with music.
I've been to thousands of concerts. Well executed live music visualizations can really elevate an experience. I recall Nine Inch Nails, Smashing Pumpkins, Muse, Digitalism, and deadmau5
all having excellent visualizations. I'm sure there are another 100 I'm forgetting.
With Reality Synthesizer anyone who has an iPhone/iPad with a LiDAR sensor + a device that outputs MIDI (i.e. most synthesizers) can see the music as they play it. The code is open-source primarily because I want any interested people to hack it and come up with better shaders for more impressive visualizations. I hope a shader novice/pro finds the repo, has fun with it, and improves on my absolute-beginner-level shaders.
Longer term: I hope ~5-15 years from now there are entire industries of creatives dedicated to custom AR glasses visualizations. Snap Inc. provides the closest thing we have to that today – but I think all AR-related hardware has a long way to go. I'd love to be wrong on that. I extra-hope (however unrealistic) some of the custom future AR visualizations are driven by synth-y hardware. I slapped together Reality Synthesizer after taking Atlas Obscura's experimental music course – and I would love to tweak what I'm seeing using some crazy hardware reminiscent of modular synths. If that comes to pass: Watch out for blown retinas.
The code is here: [ Ссылка ]
