Academy Xi Blog

Everyday examples of Mixed Reality

By Academy Xi

Share on facebook
Share on linkedin
Share on twitter

examples of mixed reality

Virtual Reality (VR) and Augmented Reality (AR) are rapidly changing the way in which we perceive the world around us. Mixed Reality is set to be even more immersive and will likely define our digital experiences in the future. Find out what Mixed reality is, how it works and how it’s already being used in everyday life.  

While the experiences offered by Virtual Reality and Augmented Reality are informational and consumed on screens, Mixed Reality promises to be far more immersive. 

Instead of bringing more of the physical world into our digital screens, technology is primed to use Mixed Reality to take digital data and fuse it with our actual environment.

What is Mixed Reality?

Mixed Reality (MR) is a combination of multiple advanced technologies, primarily Virtual Reality and Augmented Reality.

What is Mixed Reality MR

To understand Mixed Reality, let’s differentiate VR and AR:

Virtual Reality:

Using VR devices, the user’s current reality is replaced within the virtual world. It doesn’t account for the user’s immediate surroundings, thus it is usually recommended to use VR technology in spacious environments where the user won’t accidentally bump into something in the physical space. By design, it isolates the user from their context in order to provide an immersive experience. Current examples of VR devices in the market are the Oculus Rift and HTC Vive.

Augmented Reality:

this technology usually just roughly overlays digital content on top of the real world. It doesn’t account for your surroundings, nor interacts with your space. Simply AR rests on top of any surface, with the physical world acting as a static background for it. Examples of AR devices recently developed were the Google Glasses and Snapchat’s Spectacles.

Unlike AR that simply overlays digital elements on the physical space without considering its unique and changing composition, MR devices constantly gather information about the surroundings. This information will then be used in order to seamlessly place digital content and information on the physical space and allows the user to interact with it.

Unlike VR, you don’t disappear into the digital world, the digital world goes to you.

Mixed Reality may sound like brand new, futuristic technology that is still decades away from being a reality. In fact, MR has been around since the 1990s as a term coined to explore the potential of combining both virtual and augmented reality together.

Even augmented and virtual reality are not brand new, despite their commercial use and applications as of recent.  In fact, both VR and AR technology can be traced back to the 1960s.

Today, Mixed Reality is a technology that is quickly making its way into mainstream use. Microsoft’s Hololens is a well-known example of an existing, commercially available Mixed Reality device. It is a holographic computer you wear around your head, with lenses over your eyes that project holograms you can manipulate and interact with as though they existed in your physical surroundings.

It features 3 sensors and 5 cameras that constantly learn your 3D surroundings, and can even remember where you place digital elements in your physical space so that when you use the system again, your windows and apps will be right where you left them.

While Hololens operates on the Windows system and offers applications and software you can interact with as though you would on a touch screen, Meta 2 is an MR device that allows users to interact with virtual objects as if they were physical objects.

These two are just examples of MR technology that currently exists, and gives us insight on what the technology can do now, and what it can potentially do in the future.

How does mixed reality work?

Imagine sitting in an office, where you can see all the material objects that surround you, like walls, desks and chairs, but you can also interact with digital content, like a shared document which updates to the cloud, or a prototype that you can experiment with using a digital interface. These are the possibilities of mixed reality. How does it work?

Cutting-edge technology is able to create mixed reality by:

  • Using input devices – Mixed reality software takes its commands from a series of input devices which collect data in the real world. Input devices include orientation trackers, magnetic trackers, optical trackers, eye tracking devices and ultrasonic trackers, to name just a few.  
  • Detecting physical objects – Mixed reality relies on object recognition, also known as object detection. This term refers to the ability of the tech to identify the form, shape and position of different objects in the space captured by a device’s camera.
  • Real environment perception – Mixed reality software uses input devices to sense the environment around it. It maps the space and then superimposes information onto that space in a way that merges virtual and real-world content.

Mixed Reality in the real world

Mixed Reality in the real world

The possibilities and implications of Mixed Reality’s continuous growth demonstrates that the future of MR is both realistic and limitless.

While we can only predict where MR can take us next, its benefits and uses in various industries today and in the near future are astounding:

Mixed Reality in Healthcare and medicine

Mixed Reality simulations are already helping medical students understand their patients better, particularly those who are hearing and visually-impaired. Through MR technology, how these patients experience the real world is simulated, leading to greater empathy and understanding from medical practitioners.

Doctors — and surgeons, in particular — are able to learn complicated surgeries and develop cutting-edge procedures. They can run scenarios and simulate the same conditions in the operating room, allowing them to plan and predict outcomes.

Through MR, first responders can also better prepare for work scenarios safely and without risk, making them better prepared for these stressful situations. MR also helps PTSD patients through controlled exposure in a safe environment and at their own pace.

Mixed Reality in Education

Experiential education is one of the most effective learning and teaching tools. Through Mixed Reality, students are able to interact with what they’re learning unlike ever before. It’s not just visual, audio, or traditional learning methods that will enable students to learn, but actual experiences that promote deeper, immersive learning.

Mixed Reality may also be used in the future to expand on Virtual Reality’s work with immersing people in different cultures, raising awareness to causes in a way that bolsters greater empathy. People will be able to learn about other cultures, social, political, and economic concerns in near first-hand experience through Mixed Reality.

Mixed Reality in Gaming

There is great potential for Mixed Reality in gaming, unlocking an experience that combines the intense and impressive worlds in video games into the actual environment. It has the potential to gamify fitness, as Augmented Reality did with the Pokemon Go phenomenon.

With digital elements being embedded in the actual environment, people will be motivated to physically move to access digital content and experience them hands-free, instead of simply looking at screens on handheld devices.

Mixed Reality in Retail and business

Many possibilities await businesses when they make use of Mixed Reality technologies. Companies are now maximising VR and AR, like providing augmented reality maps to customers for better understanding and access to their stores.

With MR, stores can give customers unprecedented information as they walk through the aisles, empowering customers to make informed and confident purchases without the need for assistance. They can also experience a product or a service before purchasing.

Mixed Reality will unlock a whole new way for people to experience the world, which could mean new products and goods for companies to develop and offer. Just like today’s trend of microtransactions and in-app purchases, businesses can sell virtual wares to populate this mixed reality world.

Examples of Augmented Reality in Everyday Life

Augmented Reality is used in all kinds of everyday apps and software to give us all more immersive and creative experiences. Some of the most widely used examples of everyday AR include:

  • Snapchat – The filters that the Snapchat app offers use AR to overlay digital content on the user’s face during a video call. 
  • Photo editing software – Even amateur photographers can add artistic special effects to their photos with AR. As an example, Tipit has launched an automated photo editing mobile app that uses AR called Model.  
  • Google ARCore – Google’s platform for building augmented reality experiences uses different APIs and enables your phone to sense its environment, understand the world and interact with information. Some of the APIs are available across Android and iOS, allowing you to share AR experiences.
  • Interior design apps – Taking the guesswork out of redesigning a room, the latest interior design apps can use AR to show you what a space will look like with a certain choice of paint, wallpaper or furniture. Intrigued? Check out Ikea’s Place app

The future of MR

Imagine being able to try out clothes before shopping for them online, testing out if a wallpaper will look good in your environment before applying it, or virtual labels around your home reminding you of where you left your keys.

Tech giants like Apple, Google, Intel, and Microsoft are already pouring resources on developing Mixed Reality technologies. Apple has patented many MR technologies, with Samsung and Facebook getting in on the ground floor. There’s a lot of buzz and excitement about Magic Leap’s MR application of projecting light straight into the retina.

The future of Mixed Reality may mean that we will only need a single device to replace our screens. This will bring forth new ways to create content, as well as new ways of consuming them. As with other great technological disruptions that have changed our current way of life, Mixed Reality will create more industries and more jobs.

Mixed Reality will unlock a future where our natural world and powerful digital information meets. In a few years, we will see it mature and be nearly invisible to the user, blending seamlessly into the human experience.

Interested in future-proofing your career by learning cutting-edge tech skills? Check out our online courses in Australia to find out more. 

Academy Xi Blog

Virtual Reality: The future of music tech in 2023

By Academy Xi

Share on facebook
Share on linkedin
Share on twitter

Recently, I attended my first Virtual Reality music concert.

Standing on my bed in my small city apartment, I found myself in a psychedelic, virtual environment, dancing and cheering a DJ on a virtual dancefloor. The performers were HEAVYGRINDER, and the application, TheWaveVR on the HTC Vive.

This was the third flagship show to debut on TheWaveVR — and it was seriously unreal. After removing the headset, I had the leftover feeling of being at a live music. I had formed a memory of attending a music concert with people.

This sense of presence and accessibility — being able to feel like you’re really experiencing live music, without leaving your home — is reason enough to be optimistic about the future of music in VR.

In this article, I will explore additional reasons to why VR is set to impact the music industry in a massive way.

Before we dive in…

Here’s a quick differentiation between the two main types of VR music concerts:

  • 360 VR Video: A real-world concert is filmed using a 360-degree camera. This type of VR concert has gained popularity on mobile-based headsets like the Samsung Gear VR and Google Daydream through applications like Next VR. These applications offer live VR concerts, featuring bands like Imagine Dragons and Third Eye Blind.
  • Fully-simulated VR: TheWaveVR is an example of a fully-simulated VR music concert. Performers embody avatars and perform solely within a virtual space (i.e. there is no real-world event). This type of experience is better suited to high-end VR headsets like the HTC Vive as interactive elements like dance are important.

I wrote this article with fully-simulated VR in mind, but it effectively applies to both. In the future, we may see overlaps between the two types, and they may ultimately blend into a single platform.

A new wave of social

Today’s music streaming services, like Spotify, focus on the listening habits of the individual to drive user engagement. Song selections, albums, and playlists are all curated towards YOU based on your previous listening habits. This encourages you to treat music as a personal experience, something that entertains you in isolation.

This is where VR has the potential to offer something incredible to music tech: social experience.

You’d be forgiven for thinking that VR itself is an isolating technology. The truth is, VR has the potential to be an extremely social platform, allowing users to connect, interact, and share experiences with each other from anywhere in the world.

Voice, body language, and interaction help users to feel the presence of other people, making them feel like they’re together and that they’re occupying the same space, regardless of where they are in the real world.

Adding social to the VR concert experience means that it becomes something you do with people — something you do with your friends. When you’re with people you’re more likely to have a profound experience and develop richer memories. These will become memories that you and your friends will talk about for years as something that actually happened: “Remember that time we saw Daft Punk in Virtual Reality?”

On a grander level, there’s something about being part of a crowd at a concert that gives you a powerful sense of unity. Being one in a crowd of people vibing and dancing in sync, heightens the elation you feel when listening to live music. You feel like you’re a part of something larger than yourself, which is an awesome feeling. VR could be the tool that allows people from around the world to come together and unify, by experiencing the same music together.

Gamifying dance

Audioshield first look, VR mixed-reality

It’s like Guitar Hero, but re-imagined for Virtual Reality.

This is how I often describe Audioshield, a VR rhythm game that has you blocking incoming orbs to the beat of a song.

Here’s Dylan Fitterer, the game’s creator:

“Brandon Laatsch (Hover Junkers, Duck Season) had an idea that music works so well in VR because it can synchronize three senses: sight, sound, and touch. I think he’s onto something there. VR’s main strength as a new medium is accurate body inputs, so any game that leans into that is on the right track,” says Dylan.

Accurate body inputs are at the crux of what makes this game fun. The orbs are flying towards the player to the beat of the song and instinctively the player defends themselves with their handheld shields. Through playing the game, the user has been encouraged to move their body to the beat of the music. They’ve been tricked into dancing. You could say that Audioshield gamifies dance.

Once the player is dancing (whether they’re aware of it or not), they’re almost guaranteed to have a good time. The movement of dance encourages the release of feel-good hormones, which makes the player feel elated.

It’s no wonder that rhythm-based experiences like Audioshield and SoundBoxing are some of my most regularly returned to VR games.

TheWaveVR also utilises the gamification of dance through the use of ‘challenges’ — simple but effective tasks that encourage users to dance and interact within the scene (like Jump, Hand Ups, Make Some Noise, and Move It).

During the ‘group hug’ challenge, the crowd huddles together into a giant mess of avatars, awkwardly holding hug poses until the progress bar fills up. Challenges like this bring crowds together and help channel a sense of unity.

Subscribe to the experience

So, let’s say Virtual Reality is going to be the future of music tech. It’s going to be a cultural revolution that will have music-lovers worldwide rejoicing together through shared music experiences.

That’s great, but how will it be monetised?

The most obvious form of monetisation, drawing parallels with the real thing, is ticket sales. Steven Hancock, COO of Melody VR said at Midem 2017:

“The live streaming works in the essence that the second a gig or a festival or a concert sells out, you can switch to a live VR ticket with a limitless capacity, so wherever you might only be able to hold 10,000 people…, 100,000, 200,000 people could buy into that live stream as it’s happening.”

Melody VR has taken a slightly different approach to VR concerts. While TheWaveVR is a completely simulated virtual environment, Melody VR is more concerned with capturing real-world events, using 360-degree filming technologies.

What Hancock is speaking about here is using VR as a complementary experience to the real thing. Missed out on buying a ticket to an upcoming concert? No worries, here’s a cheaper ticket to the Virtual Reality alternative that you can experience in real-time. Of course, it won’t be as exciting as the real thing, but hey, it’s something.

Ticket sales could become the primary source of monetisation when VR concerts start to gain popularity.

The Weeknd, Coldplay, and Katy Perry have an upcoming performance that’s only available as a fully simulated Virtual Reality experience. Would you buy a $5 ticket, even just to see what it’s like? Now imagine this on a worldwide scale. Everyone tuning into the same concert experience from across the globe, each paying $5. And the artist only has to perform once, and they don’t even need to travel!

Another likely form of monetisation could be monthly subscriptions: $10 a month for access to unlimited Virtual Reality nightclubs, concerts, and music festivals. It’s not hard to imagine the Spotify of the future integrating VR concerts into their existing subscription packages in an attempt to outdo competitors.

Whether ticket or subscription based, VR music will be profitable through the leveraging of its worldwide accessibility.

Shooting cats out of cannons

‘Chocolate’, a VR music experience from the mind of Tyler Hurd, could be described as an audio-visual sugar high, or probably more accurately as a bite-sized acid-trip.

This psychedelic, otherworldly experience has you jazzing out to music, shooting cats out of your canon hands, and watching them fall in slow-motion. It’s pure bliss.

Like Chocolate, some things are simply impossible to experience without the use of a VR headset. And of course, some things would be frowned upon by society if enacted in real-life (like shooting cats out of canons).

The potential for boundless and uninhibited experiences is one of strongest draw cards for Virtual Reality, allowing people to exist in ways far beyond their wildest imaginations. This is especially true for music experiences, as exemplified by Chocolate and TheWaveVR.

I reached out to Adam Arrigo, CEO of TheWaveVR, to get his perspective on the potential of music in VR. He wrote back the following:

“When we started working on the project about a year ago, the problem we said we were solving was ‘live music is expensive and inefficient.’ Like other companies, we saw the main value of VR as providing greater access to live music experiences. When we actually dug into what was engaging and unique about VR, though, we found that the real value was creating experiences that are only possible in — and designed specifically for — VR.”

“We stopped thinking of how to retrofit existing experiences and media types to VR and really asked ourselves what type of music content should exist in this medium. How do we amplify what’s good about concerts, give the audience and performer superpowers, create totally new artistic experiences that could transport fans inside artists’ minds, using 3D graphics and interaction design? How do we make it feel like a dream and not a recreation of something that will always be better in real life?”

Arrigo goes on to write about how the affordances of VR open up limitless possibilities for both the performer and the audience:

“The performer can be 10 times the size of the audience, be made of swirling particles that explode to the beat — they can transform the venue with the click of a button. And the audience? Instead of clapping, they can throw a giant pulsing star into the sky, or they can create their own light show that others can contribute to.”

Live music in VR should not aim to replicate the existing concert experience, but should instead aim to enhance it. Each performance could be something unique — something magical and foreign that users could not experience otherwise.”

Arrigo rounded off the email with this:

“We all just need to keep an open mind and keep pushing the envelope of what’s imaginable, because there’s so much incredible stuff coming in the years ahead.”

An actual real-life nightclub

We’ve talked about sociability, gamification, monetisation, and unimaginable experiences, but we’ve overlooked the most important facet of this investigation, and the most important consideration of any new product: What do the users think?

I reached out to TheWaveVR community to gauge user opinion. I received a bunch of comments, but this line from TheWaveVR user Lavender stood out:

“Every time I’ve been to a main headline event, I’ve come away with new friends and memories, and I get the same energy and groove as I do going to an actual IRL nightclub.”

This opinion, widely shared amongst the community, is the main reason I believe that Virtual Reality will be the future of music tech. If our relatively basic VR technology can replicate the feeling of going to a nightclub or concert now, can you imagine what the future of live music experiences could hold? It will be something truly incredible!

Search our website

Find the right course