Cleveland got its first taste of virtual reality immersive gameplay in March 2022 when the technology strutted into Boss VR Arena featuring Zero Latency Technology. Nearly eight months later, Sandbox VR, a company founded by Steven Zhao in Hong Kong in 2016, planted its flag in Westlake’s Crocker Park. But VR isn’t just for gamers, adrenaline junkies or high-tech enthusiasts. It’s not locked into an entertainment venue. The language continues to develop—VR’s cousins, Mixed Reality and Augmented Reality, have been used interchangeably and Meta CEO Mark Zuckerberg is still trying to wrangle what the Metaverse is (along with everyone else)—but Apple, Google and Microsoft have all poured billions into VR-related projects. Cleveland, in many ways, has capitalized on the technology; the city’s integration of VR has placed it at the forefront of the anticipated VR-era. So far it’s focused in medicine and education.
VR in Medicine
It may not be as entertaining as slaying zombies in immersive gameplay, but the Cleveland Clinic Virtual Reality Shopping platform makes unique use of VR to help patients. The program captures user data in relation to the performance of instrumental activities of daily living, or iADL.
In the Cleveland Clinic’s Virtual Grocery Store Task, the user faces a mix of cognitive and motor tasks as they walk around a simulated grocery store. Every moment—freeze of gait, turning speed, referral to a shopping list and shopping cart avoidance—is recorded, feeding real-time data to physicians.
According to the Netherland’s ongoing Rotterdam Study, iADL decline occurs five to seven years before Parkinson’s or Alzheimer’s disease is detected. As a result, Dr. Jay Alberts, vice chair of innovation at the Cleveland Clinic’s Neurological Institute, hopes the technology could be used to raise a “red flag” if warning signs occur in a patient’s dataset.
“We’re able to actually record from the subthalamic nucleus, that’s one of the structures with the basal ganglia that’s affected with Parkinson’s,” Alberts says, referring to an ongoing study of 15 Parkinson’s patients from Cleveland Clinic’s Center for Neurological Restoration, which received a three-year $2 million grant from the Michael J. Fox Foundation for Parkinson’s Research last year. “We’re getting the neural signature associated with this freezing of gait. Now we can treat it better either through medication or through different methods of deep brain stimulation.”
Beginning November 2022, Sheffield Family Health Center agreed to help the clinic with its study. Over the course of one year, a total of 100 patients, ages 55-90, will volunteer to complete the virtual grocery store task during their annual physical.
The virtual world helps break patients away from clinical settings, which can have limitations.
“It’s very difficult to identify freezing of gait in a clinical environment,” Alberts says. “Everything is clean lines, [there are] no distractions. And so that’s not the real world. It’s difficult then to induce freezing. If you can’t observe it, it’s difficult to treat.”
But with VR mimicking the real world, headsets transmitting data to a physician’s screen and an Infinadeck omnidirectional treadmill allowing patients to enact daily motion (the same kind featured in the 2018 movie Ready Player One) accuracy is finally snared.
Dr. Alberts is also working with Augmented Reality or AR. AR allows a user to see virtual images superposed over actual reality. The popular 2016 mobile game Pokemon GO is an example of AR.
In 2019, 50 participants—25 in each group—volunteered to compare the Cleveland Clinic’s developing AR program using HoloLens. Guided by a Physical Therapy Assistant avatar named after Alberts’ late mother, Donna, the patients received one-on-one physical therapy for eight weeks. The study proved the care received was equivalent in-person therapist visits. It also showed the technology’s interface was user-friendly.
“When we’re designing medical applications, we absolutely have to have the end user in mind. We designed this with the end user, the individual with Parkinson’s [as well as older adults], in mind,” Alberts says.
Courtesy Cleveland Clinic
Currently, Cleveland Clinic’s AR program is under review, though the next steps will be seeking FDA approval. Alberts would like to see the technology eventually used in rural areas with limited medical access, a hope carved from his upbringing in Iowa. These prospects—and the user-friendly interface—are the most rewarding aspects for him.
“There’s a big need for us to be thinking about, how do we reach underserved and rural individuals? And I think technology can do that,” Alberts says.
“We can potentially push this technology out to people who live beyond the Northeast Ohio area. Or even those who are in Northeast Ohio, but maybe they have issues with access to care.”
The Cleveland Clinic’s use of VR and AR applications are not limited to studying Parkinson’s disease. Neurological data for concussions and epilepsy may also be on the horizon.
On a side note, Cleveland Clinic is also working with the Department of Defense to develop a Readiness Evaluation with Augmented Reality Return-to-Duty platform. This past September, 157 service members in Fort Benning, Georgia, underwent AR simulations aimed to determine when troops are ready to return to active duty. The AR replicated military-related scenarios, from gun jams to room breaches. Though this study, Troop READY, is being analyzed, the service members reviewed it as “excellent.”
VR in the Operating Room
Dr. Ben Roitberg moved from Chicago to Cleveland to become MetroHealth’s Department of Neurological Surgery chair in 2017. That year, Cleveland Clinic neurosurgical residents at MetroHeath began to practice procedures with VR headsets.
Drills into a virtual skull gave haptic feedback; rare cases of separating conjoined twins could be walked through in a simulation. This was the first-generation system ImmersiveTouch—a VR platform headquartered in Chicago—offered. ImmersiveTouch’s ability to create 3D renderings from 2D CT and MRI scans came under Roitberg’s radar while working in the Windy City.
“There’s a lot of value from an education perspective, particularly early in their [residents’] training,” says MetroHealth neurosurgeon Dr. Deven Reddy. “A lot of these things were traditionally taught through research papers [that] we spent hours going through.”
Since 2019, Reddy has used ImmersiveTouch himself to rehearse complicated and rare surgeries, including skull base surgery and vascular neurosurgery. Donning an Oculus lens and taking up controllers, Reddy maneuvers the virtual skull before him, mapped from a patient’s actual scan. Reddy can plan his surgical approach in real-life dimensions, rather than triangulating thousands of planar images in his mind.
Reddy remarks the technology is especially apt for planning facial reconstruction, since bone and vascular anatomy show up particularly clearly.
Though ImmersiveTouch is still evolving, more than 16 schools and medical centers across the country have since joined MetroHealth in adopting the platform. Reddy eventually sees the technology being used within the operating room, and points to ImmersiveTouch’s demonstration of a new AR system at the Congress of Neurological Surgeons in San Francisco in October 2022 as an indicator.
Reddy imagines other surgical departments, such as ENT, trauma and spinal, will begin to use the technology within the next two years.
“As it gets better and it’s able to show a greater degree of anatomical detail with a high degree of accuracy, I think it becomes a common sense extension of what we normally do,” Reddy says. “That we have an opportunity to be at the front end of it is a privilege.”
MR in Education
In 2013, Case Western Reserve University and Cleveland Clinic announced plans for their shared $515 million Health Education Campus. But there was a catch: no cadaver lab would be built in the new 485,000 square-foot space.
The medical program would still use—and continues to use—cadavers as a core component on its main campus, which all medical students interact with during a two-week bootcamp, but students’ primary way of learning was to change.
“The thought was that technology is advanced far enough that we shouldn’t have to invest in a [new] cadaver lab,” says Dr. Mark Griswold, Pavey family professor of radiology and faculty director of the Interactive Commons at CWRU. “They basically made the charge to us to prove that.”
Formed in 2014, the Interactive Commons consists of only about 10-14 consistent team members. They’re tasked with answering the impossible: What will a university look like in 2050? When this target merged with the ambitions of an anatomy building without a cadaver lab, Griswold and the Interactive Commons started looking for their answer.
Griswold pulled from his radiology background, looking at MRI and CT images that could equal cadaver lab training. They considered large, human-size touchscreen devices and VR headsets, but the tech wasn’t up to par with what they needed.
Then-Chief Research and Strategy Officer for Microsoft, Craig Mundie, tapped the Interactive Commons’ shoulder. Cleveland-born, Mundie was a welcome force for the new Health Education Campus already, after then-Cleveland Clinic CEO Toby Cosgrove had invited him to become a partner.
“’We think we have something that might help you, but it’s not public knowledge yet,’” Griswold recalls Mundie saying. “’Why don’t you come and visit us in Redmond?’”
In December 2014, Griswold, along with CWRU electrical engineering and computer science associate professor Dr. Marc Buchner, assistant dean for education, informatics and technology at the Cleveland Clinic Lerner College of Medicine Dr. Neil Mehta and then-CWRU chief information officer, Sue Workman, flew out to Microsoft’s headquarters in Washington.
“We saw the prototype for what became the Microsoft HoloLens and immediately knew that this was going to change the way that we do everything,” Griswold says. “And I don’t say that lightly.”
Courtesy Case Western Reserve University
Mixed Reality, like Augmented Reality, transposes virtual images in a real space. While the language is evolving, MR differs from AR in how it accomplishes the effect. Rather than the viewer looking through a screen to see the virtual objects, MR projects virtual images onto the surface of a user’s eyes. They can see the virtual, as well as the physical world, simultaneously.
Mixed Reality is what Microsoft HoloLens uses, and it was Griswold’s solution.
While the Interactive Commons had considered earlier headsets, full VR-immersion made teaching classes of 50 or more pupils impractical. VR is a complete dive into a digital atmosphere: Each student would be confined to their own private lesson, unable to interact with peers.
With MR, classroom interaction, teacher instruction and virtual 3D models, could coexist.
That February, Interactive Commons returned to Redmond to explore HoloLens programming. Although 30 organizations partnered with Microsoft to pilot test HoloLens, including NASA, Volvo and Japan Airlines, CWRU was the only university. This would prove instrumental to Cleveland’s technological lead in the field.
Using HoloLens, CWRU’s Interactive Commons developed HoloAnatomy Software Suite. The Jackson Hole Science Media Award-winning software launched just months after the Health Education Campus opened in 2019. And results were promising.
In a study published in Medical Science Educator in November 2019, students were shown to grasp concepts twice as fast using HoloAnatomy as they would with 2D models. Students also performed as well as traditional cadaver-based lab students.
“We took one of the oldest classes that you can imagine, human anatomy, and in a period of about five years, we went from hundreds of years of history to teaching in this new way,” Griswold says.
The pandemic further proved HoloAnatomy’s mettle. In March 2020, every CWRU School of Medicine student in their first year received a HoloLens by mail from the university. Professors, donning a HoloLens themselves, instructed in real-time to students at home. Class continued with 3D models of lungs, vascular systems and muscles hovering over coffee tables and nightstands states away. Of the 177 students who answered CWRU’s proceeding survey, 81% felt the MR classroom was just as effective as in-person learning, if not better.
In the time since, 17 schools, including one as far as Poland as well as the United Kingdom’s esteemed Oxford University, have adopted curriculums featuring the Interactive Commons’ HoloAnatomy Software Suite.
But Griswold isn’t stopping there. His mind falls on other topics that have been taught the same way for decades. He wonders how they can be enhanced with HoloLens.
Take the Red Monastery for example. The 1,700-year-old, Coptic Orthodox monastery in Egypt his colleague, professor and chair of art history Dr. Elizabeth Bolman, teaches about. Though he’s seen exquisite photographs from Bolman’s textbook and heard her describe the monastery’s splendor, what would it be like to actually take a class inside?
“We had to knock it down a little bit to have it run in HoloLens,” Griswold says, as he pulls up a millimeter-precise laser scan of the monastery on his computer. What he shows me is like a virtual tour. The colors are vivid. I can only imagine what the space would look like under a HoloLens visor, enveloped in a graphically accurate depiction of the real place; it’s walls, ceiling and floor.
Red Monastery, courtesy Interactive Commons
“I had no idea how big the space was until I saw it in HoloLens, until I was in there with Betsy,” Griswold says, noting they’ve taught around 100 students on the structure by this method.
“How many of them could we afford to take to the real place?” Griswold continues. “Very few of them could actually understand it in the way that they have now.”
And even if they went to the real place, Griswold adds, the women wouldn’t be allowed inside.
The Interactive Commons’ use of HoloLens is so much more than a niche feat. Their technological application breaks cultural and economic barriers, as well as time and distance. Griswold admits cost remains a challenge, but believes, as with any technology that becomes integral to daily life, the expense will deteriorate rapidly. He lists MR’s wonders for companies up and down the chain. Think manufacturing training, remote factory maintenance and 3D programming.
In the realm of the performing arts, consider unbridled imagination. In November 2017, CWRU’s Department of Dance presented Imagined Odyssey. Using HoloLens, 80 audience members cohesively witnessed 3D MR visuals alongside actual dancers on stage. The technique and scale of technological integration has yet to be replicated.
“That was five years ago,” Griswold reminds me.
As artistic director and professor of dance Gary Galbraith put it in a promotional video about the event on CWRU’s The Daily, “When you’re the one in front, you just have to keep going.”
“We’ve been given this opportunity to lead the world in this area. As a region, we can adopt this as our future,” Griswold says. “We should be amazingly proud as a city.”
Courtesy the Cleveland Museum of Art
MR in Exhibition
To move forward, we must now move backward. Roughly 1,500 years back, to a mountain located in southern Cambodia called Phnoma Da.
Within Phnoma Da is a cave temple. Inside stands Krishna Lifting Mount Govardhan, fully intact. Water drips from the polished sculpture’s bejeweled sandstone as worshipers pay homage to the Cambodian boy-god. Legend has it he saved the floodplain area from a devastating rainstorm, shielding his people beneath the mountain itself.
Present-day Cleveland, Krishna Lifting Mount Govardhan is a fragmented work within the Cleveland Museum of Art’s nationally renowned Indian and Southeast Asian Art Collection. His left shin is missing, along with both of his feet. His face is weathered, but close-up, you can still see the ringlets carved into his hair. His frozen smile remains a friendly gesture.
Though Cleveland’s Krishna was restored to the best it could be in 1970, recent evidence surfaced in 2014 that what were believed to be the figure’s legs actually belonged to another Krishna statue, still stationed in Cambodia.
After undergoing laser scans within the museum’s own conservation department, CWRU and an MRI at the Cleveland Clinic, the pieces were traded between CMA and the National Museum of Cambodia.
Unveiling the newly restored masterwork was in order. George P. Bickford Curator of Indian and Southeast Asian Art at the Cleveland Museum of Art, Sonya Rhie Mace, was placed in charge of the exhibition. She had a truly daunting task.
How Cleveland’s Krishna came to be so dispersed—across three continents—was just one chapter of the story. Mace also had to introduce the legend of Krishna, Phnoma Da, the initial restoration and latest discoveries to the public, who likely would be hearing about it for the first time.
Who would be patient enough to read walls of text?
“‘Could digital help us tell this story better?’” Chief Digital Information Officer for The Cleveland Museum of Art, Jane Alexander, remembers Mace asking her.
“I thought this was a great idea,” Alexander says. “We needed some sort of immersion to really make people understand this story.”
The project was approved in 2018, with Alexander sketching concepts and sending them over to digital partners, DOME. In total, there would be four digital components to Revealing Krishna: Journey to Cambodia’s Sacred Mountain. In addition to projection walls, a narrated timeline and digital reunion of Krishna and Phnom Da’s other eight gods, a MR HoloLens tour through the Cleveland Krishna’s journey was scripted. For the HoloLens, CMA reached out across the field, to CWRU’s Interactive Commons.
“We didn’t want something like VR where you’re in your own world,” Alexander explains. “A lot of people come to the museum as an experience or to be with family. [With] HoloLens, you can see each other.”
Sonya Rhie Mace and Mark Griswold view a test model of Krishna statue fragments through HoloLens 2, courtesy the Cleveland Museum of Art
As the CMA worked with Griswold’s team, logistics were discussed. They timed the tour so that groups of six would start the MR portion every two minutes, allowing for their goal of 60,000 to 80,000 visitors over two months, akin to an amusement park ride.
A Cambodian flute and whimsical sparks guided HoloLens wearers to where the next segment took place. Fifteen seconds transpired between each station to ensure there was time to adjust to the HoloLens goggles themselves.
Like the Red Monastery, the actual temple in which Cleveland’s Krishna had been housed was scanned to laser precision on a trip in 2019. Re-rendered indoors, it was the grand finale of the tour: Krishna Lifting Mount Govardhan, in one piece, within the temple, as the people of Phnom Da would have seen him in 600 A.D.—probably the closest we will ever come to time travel.
Holographic recreation of cave temple, courtesy the Cleveland Museum of Art
Though the exhibit was delayed an entire year due to the COVID-19 pandemic (resulting in only 22,000 visitors) it earned one of the highest approval ratings in the museum’s history. Ninety-seven percent of visitors noted the exhibition exceeded expectations, Alexander says. World-wide, museums are contacting the CMA, asking what it takes to put on an MR display of this scale—including those in New York and Paris.
“Cleveland Museum of Art is leading in digital innovation in art museums in the world,” says Alexander. “People are always worried that digital will take away the credibility, but if you build things in support of understanding—it’s for storytelling, it’s for closer looking, it’s for deeper understanding.”
Probing for deeper understanding is a key part of the museum’s goal to make art more accessible to the public, something they have been making strides for since their opening of Gallery One in 2010, now ArtLens Gallery.
In regards to when we can expect another exhibit like Revealing Krishna, Alexander says, “I don’t believe in digital for digital sake—it depends on what the goal is. [But] we’ll always have innovation in our exhibitions, our goal is really transparency and making things available to more audiences.”
Though the full exhibition is no longer installed, the CMA has a recording of the MR sequence online, along with 3D laser scans of all the Phnom Da gods in their online collection.
Waking the Futureverse
Cleveland’s VR scene continues to amplify. Cleveland State University is applying the technology in its own curriculum. Dr. Chansu Yu, professor of Electrical Engineering and Computer Science, highlights the use of VR in their Engineering MakerSpace, a 3D VR driving simulator and cybersecurity visualization. Montre Augmentaverse, an app co-founded by three Clevelanders, allows the user to drop digital, AR talismans within 150 square feet from their iPhone others with the app can see (or hear). It’s the first AR app without preset objects. In October 2022, Montre powered FutureLAND’s A.RT Show. Whether Clevelanders realize it, their city is literally inventing the future. “That’s a rare opportunity,” Griswold says. “These kinds of opportunities to lead the world don’t come around every day."
Get ahead of the weekend by signing up for our free weekly “In the CLE” newsletter — your guide to fun throughout The Land. Arriving in your inbox every Wednesday, this weekend to-do list fills you in on everything from concerts to museum exhibits — and more. Click here to subscribe.
