Screen shot of desktop running xmms and mrv plugin, and me looking at my hand.
Mediated reality is to "augment, diminish, or otherwise alter your visual
perception of reality." Most readers will be familiar with the concept of
virtual reality. Mediated reality is a similar to virtual reality in that both
alter the user's percepction of their environment, typically by replacing their
vision with the output from a head-mounted display. However, virtual and
mediated reality differ in that virtual reality seeks to create an entirely new
environment for the user, whereas mediated reality alters what you would
normally see - i.e. it alters your perception of your actual
environment.
An eyetap is a device which allows all or a portion of a user's vision to be
mediated by computer control providing a mediated reality. Typically, this
incorporates a camera and display, configured such that the computer is able to
tap into the user's vision, alter the images, and then present them to the user
again, in real time.
Some examples of mediated reality currently being developed include advertisement replacement or filtering (visual 'spam' filters), or computer assisted construction where the computer presents visual cues to aid the user in construction tasks.
MRV explores the concept of a muscially mediated reality. MRV alters a user's visual perception based on music to which the user is listening. To understand what MRV does, consider the typical spectrum bar graph analyzer found on many commercial stereo equipment. These are the bars that rise and fall with the music, typically with vertical bars representing the lower freqencies (base, or lower tones) to the left, and increasing frequencies plotted to the right. The image below depicts a typical spectrum analyzer.
In the same way that the bar rise and fall to the beat of the music, MRV alters incoming video from an eyetap and redisplays it to the user. For the wearer, the world appears to pulse to the beat of the music.
MRV is currently being developed as a plug-in to XMMS, an open source MP3 player. Thus, MRV will analyze and react to any MP3 xmms plays.
In time with the beat, MRV applies the GXand and GXor function operators for XImages. This was a simple way to create a cool "trailing" visual effect. Specular highlights in the scene work especially well to create bright trails, as can be seen in the screenshots below. The GXand function is periodically replaced by the GXor (to prevent the image from saturating to pure white after a while).
