Mixed reality is a technology that allows the overlaying of holograms onto the real world. Someone might say, “but isn't that augmented reality?”, yes, except it frees your hands, in a way. This is because you put on a pair of glasses and this device projects holograms onto the real world. Furthermore, the most advanced models, such as Microsoft's Hololens 2, will recognize your hands, meaning you can rotate, enlarge, and drag 3D holograms in “mid-air”, just like the character Tony Stark in Iron Man.
Mixed Reality for Remote Assistance
This application aims to assist people in the fields of science, medicine, engineering, training, entertainment — in short, in anything where overlaying holograms onto real-world images serves a purpose.
This could revolutionize healthcare by greatly improving the training available to physicians, allowing them to make diagnoses or learn to perform procedures on a digital model. Even during surgery, Mixed Reality could provide surgeons with potentially life-saving information. It could also enable remote consultations with doctors, allowing patients to be examined without leaving their homes and eliminating travel time to appointments — with the possibility of being seen by specialists in other states or countries. Another example would be a surgery in Brazil guided by a renowned surgeon in the United States. The Brazilian doctor would put on the glasses (the remote expert sees everything the wearer sees) and be guided by the instructor from a distance. Or a newly graduated doctor in a small town guided by an experienced physician from a major medical center. Or an ordinary person having to administer first aid under the guidance of a remote paramedic. Imagine the life-saving potential this could have.
Imagine a German machine breaking down in Brazil. Today, depending on the case, a technician would have to travel from Germany to carry out the repair. In the near future, a trained employee could put on a mixed reality headset, such as the Hololens 2, and be guided by the German technician remotely. Imagine the savings in time and resources.
Engineers and architects also have reason to be excited. Complex structures, such as engines, will soon be able to be tested and shared as holograms with other professionals anywhere in the world before expensive prototypes are built. Similarly, architectural blueprints will likely soon be obsolete, replaced by digital models simulating construction at full scale. Clients will be able to walk through the empty space of a completed apartment and see the entire interior design created by the architect. Or a production engineer could walk through an empty warehouse and see all the factory installations before a single real fixture has been put in place.
The U.S. Armed Forces already use Mixed Reality in soldier training, as in the STC program, which in part prepares combatants for warfare. Incidentally, many of the technologies we know today were originally developed for the U.S. military.
MR could soon become an incredibly powerful and revolutionary tool in education as well, as it would mean adding to a classroom a degree of realism and a volume of information never before seen in human history. Fully immersive lessons would certainly greatly increase student engagement and facilitate information retention. Imagine a history class in which children can walk through the streets of ancient Rome or Greece, witness medieval battles as if they were there, watch the D-Day landing at Normandy; geography lessons where, instead of textbooks about the Himalayas, you are “transported” to the mountains and experience firsthand the grandeur of the place; English classes where you are “transported” to a real classroom in the US or England and interact with students and ask the teacher questions in real time; or a science class that gives you the opportunity to “be” in a real, world-renowned laboratory observing scientists at work. We would have the chance to launch in a rocket, land on the moon, and look back at Earth; we could come face to face with a dinosaur reproduced at full scale through holography — in short, any imaginable situation can be recreated with this technology.
MediView XR Raises $4.5 Million to Give Surgeons Radiological Vision with Mixed Reality
MediView XR recently raised $4.5 million to further develop its Mixed Reality Surgical Navigation system. Accessed via the Microsoft Hololens, MediView's product grants surgeons a form of “X-ray vision” when performing cancer ablations and biopsies. The system generates a personalized 3D holographic model for each patient based on CT scans and MRIs. Ultrasound then updates the holographic display throughout the entire procedure. This process not only reduces the harmful X-ray radiation used in today's standard procedures, but also improves visual acuity by translating 2D data into three dimensions. Surgeons can even rotate around the body while the Mixed Reality overlays remain accurately mapped to the patient. Meanwhile, voice commands and hand tracking allow surgeons to access all necessary information on-site. In its first set of human trials, MediView had already used its system on five patients with live tumors and initiated a nine-patient trial in August. Leveraging its newly acquired capital, the company also aims to obtain FDA approval by 2021.
Surgeons worldwide are forced to interpret 2D images for 3D applications. MediView's technology would eliminate this obstacle and reduce surgical error. Personalized 3D visualizations can also be used to educate patients about their conditions in a more intuitive way. The educational applications of AR also extend to medical schools, where mapping real data onto hands-on procedures could increase student engagement and learning. The success of tumor removal surgeries depends largely on the precision with which surgeons can incise them, ensuring no trace of cancer is left behind. As AR headsets become increasingly sophisticated, accurate 3D models (combined with biomarkers injected into the bloodstream to tag tumor cells) could greatly improve patient outcomes. MediView's CEO, John Black, who has performed over 2,000 surgeries, aims to transform the way surgeons interact with real-time data visualizations.
Apple's Mixed Reality Glasses
Apple is working on a product that would completely change our lives. I confess this is something I am very eager to see and experience. I believe this product will be even bigger and more impactful than the iPhone.
Unfortunately, they are currently being shipped only to the United Kingdom, but stay tuned — in the future, they will certainly be made available in other regions.
What does it do? In principle, the same as other mixed reality glasses such as Microsoft's Hololens 2 (information added to the real world in real time, such as traffic, weather, routes, product information, “holographic” project presentations for other people with the ability to manipulate them by hand, and so on) — however, the major draw is, as would be expected from an Apple product, the design, practicality, lightness, and comfort. It has so many aesthetic qualities that you could wear it around without drawing negative attention — that is, it would draw negative attention if it were a bulky headset, but quite the opposite: it will most likely attract attention for its elegance. And when you look at the glasses from the outside, they look like ordinary glasses. But on the inside is where the magic happens. That is where you will see the display of text, graphics, and images such as maps.
We all know that Apple is really investing heavily in Mixed Reality right now. It has released a new framework that allows developers to easily create apps that take advantage of all the sensors in their devices. All you need is an iPhone from 2015 onwards or an iPad from 2017 onwards, or anything with a sleek apple logo on its surface with an A9 processor or newer, and then simply download any AR app from the App Store, study the protocols, and, voilà, your app will be interacting with the real world. In addition, there are reports that since 2017, the company has had a massive team of over 1,000 engineers working on a secret project involving a car in Israel. Furthermore, Apple has also acquired a large number of companies related to Augmented, Virtual, and Mixed Reality.
You might be wondering: lightweight glasses with a high-resolution lens and a large volume of information will need a powerful CPU and a large battery to last all day. How will this be reconciled? Well, of course they are working on this, and besides counting on the significant increase in internet connection speeds — most likely by 2020, when they intend to launch — they will be on the Apple 14 generation of five-nanometer processors.
According to CNN reports, Apple's plan is for its glasses to feature an 8K display for each eye. That is practically a 16K screen. It seems insane to think about that resolution at first, but remember that resolutions double within increasingly short periods of time.
Some doubt that the Apple 14 processor will be capable of handling the task, but if it isn't, we have no doubt that this will be resolved.
It has also been reported that Apple is developing a new operating system called rOS, or Reality Operating System. The glasses will be controlled via touch panels on the side of the device, voice activation, and even gestures to interact with them. Now, Tim Cook himself has said in many interviews that he genuinely believes Mixed Reality is the future, because it will allow the insertion of certain virtual elements into the real world and that this could significantly enhance your experience and access to information, placing you at a monumental advantage over people who do not have that information at that exact moment. In other words, it will greatly help improve the quality of real-time decision-making.
One of the most accurate analysts when it comes to Apple leaks reported that the company will begin mass-producing its glasses as early as the 4th quarter of 2019 and that they will launch in 2020, most likely toward the end of the year. He also said they would be marketed as an iPhone accessory and would stream data from the smartphone rather than housing all the necessary components internally, which would make them heavier and more expensive while consuming more battery — a problem already discussed above. Apple will likely include a streaming chip that would take all the data from the iPhone in your pocket and transmit it to the device.
Now, when it comes to quality of information when you are placed within an environment, a minimum number of cameras is required — ideally covering all directions. This way, the glasses will be able to track everything happening 360 degrees around you and alert you accordingly. Say you are crossing the street and a car suddenly approaches very quickly from the side or from behind — the glasses can be programmed to alert and warn you quickly so you can move out of the way and avoid an accident.
I would like to make it clear that what has been written here is solely our opinion on how we believe Apple might build the glasses, based on all the patents we have reviewed.
Mixed Reality Combined with Artificial Intelligence
Mixed reality, when combined with artificial intelligence, will add “superpowers” to human beings in a way never seen before. One could almost say that a person wearing mixed reality glasses belongs to a different species than one without the device — such is the gap in terms of real-time access to data. We will have real-time access to information that would otherwise take minutes to retrieve through a search on our phones. Imagine your glasses recognizing a friend you haven't seen in years who walks past you — and neither of you notices — in some city in the world that, coincidentally, you are both visiting at the same time. Or police glasses recognizing a wanted criminal through facial recognition. Or the glasses detecting that someone is about to become ill because their vital signs are transmitting warning signals via sensors to those nearby. Seeing a monument and having information about its history appear instantly on your lens. In short, this new era is only just beginning, and I am very glad to be a part of it. What about you? What do you think?
LYNX Unveils New Mixed Reality Headset
Earlier this week, French startup Lynx launched the R-1, a standalone mixed reality headset built for enterprise users. It features an opaque display (no external light enters), but films the entire surrounding environment to give the sensation of overlaying holograms onto the real world.
It is one of the first products to use Qualcomm's new Snapdragon XR2 chip and features six cameras for positional tracking. It has no physical hand controls, but recognizes hands, and therefore uses a digital controller that follows hand movements.
The R-1's new prism-type lenses also make it more compact than traditional XR lenses. Claiming a resolution of 1,600 × 1,600 per eye and a 90-degree FOV, Lynx is on track to compete with Magic Leap and Microsoft Hololens.
The price is expected to be around $1,500, with a launch date scheduled for summer 2020.
Mixed reality is about to add a layer of digital intelligence to all of our surroundings, transforming retail, manufacturing, education, tourism, real estate, and nearly every major sector underpinning our economy today.
Last year, the global VR/AR/MR market reached a value of $814.7 billion, and is expected to continue growing at a rate close to 63% through 2025. In one of the greatest technological revolutions of this century, smartphones dematerialized cameras, sound systems, video game consoles, TVs, GPS systems, calculators, and paper.
Mixed reality glasses are poised to continue this trend, dematerializing the smartphone itself. We will no longer look down at tiny two-dimensional screens, but instead see through a fully immersive 3D interface that adds valuable information to our daily lives in real time.