Virtual Reality Comes of Age

Aswe see improvements in technology performance and assessment there is also theopportunity to significantly reduce cost of medical education and training.Paul Pribaz reports.

Natalia Grudzien, a second-year medical student at the University of Illinois College of Medicine at Peoria (UICOMP), lifts two controllers up to her waist, squeezes the triggers, and extends her arms to her sides. “This is like a 360-degree computer screen,” she says, tilting her head and lifting her chin. The large head-mounted display covering the top half of her face doesn’t seem to faze her at all. Natalia is clearly immersed in the virtual reality experience; she pauses the lecture with a tap of her thumb and reaches into her pocket to mute her buzzing cell phone. Another tap and she is again staring ahead intently, slowly craning her neck and moving sideways, as if trying to get a better view. “I’ve never been in virtual reality before. I am completely blown away.” Natalia is watching a pre-recorded lecture about the soft tissues of the knee. She’s able to pause, rewind, and, by using a more exaggerated version of an iPad gesture, she’s able to manipulate the digital images projected in front of her eyes. “This is amazing for understanding the spatial relationship between the different structures. I can literally go into the anatomy to see how it all fits together.”

Instructors are enthusiastic about the teaching opportunities in VR.
All images: Jump Simulation at OSF Health.

Natalia is using a commercially availableheadset, in this case the HTC Vive, in combination with custom software thathas been developed at the University of Illinois and OSF HealthCare. Thesoftware allows access to pre-recorded content in a highly immersive format. Anassessment module at the end of a lecture lets her confirm that she understandsthe content. “I really think this is going to be amazing for medical education.You can’t spend hours and hours in there, but versus a textbook it’s much moreengaging.”

The HTC Vive head mounted display.

Interaction

The evolution of the custom softwarefollowed a common constraint in simulation and clinical education, namely, thedifficulty of quantifying the value of expensive tools. For Dr. MatthewBramlet, Director of the Advanced Imaging and Modeling program at JumpSimulation at OSF HealthCare, the lightbulb moment came when he was watching asurgeon interact with a virtual model of a heart. “We had previously printed in3D our complex cases for surgical planning. While we have been extremelyfortunate to cover the expenses of that with a generous philanthropic gift, ourgoal has been to try and create a sustainable way for other centers toparticipate in this work. Once we dropped the digital file into a visual 3Denvironment and let the surgeons play with it, they started telling us whatthey saw and what they actually planned to do for the case. I realized that ifwe could capture what he was saying at that moment, we had educational goldand, more importantly, a sustainable path.”

Dr. Bramlet subsequently founded a company,called Enduvo, which has taken the software license and is working tocommercialize it.

“Our first deployment outside our owninstitution is at the NIH’s National Institute of Allergy and InfectiousDiseases, where they are using the software platform to train biovisualizationscientists in Africa about the structure of tiny pathogens, which they say ishelping them design better studies to understand the etiology of disease.They’ve told us that they love not having to stay up so late at night to talkto another continent over Skype – they just record what they want to say withthe 3D image and email the entire file.”

Dr. Bramlet cautions that while thesoftware platform makes it easy to teach in a VR environment, there is asignificant amount of work required to create anatomically accurate 3D models.“There is definitely some skill and practice required to turn a DICOM datasetinto an accurate 3D model. But once that’s done, you can load it and any otherdigital files, like an image or a movie, into the environment, hit record, andstart teaching.”

Not all users of Enduvo are medicalstudents or PhD scientists. Dr. David “Trey” Jantzen, a pediatric cardiologistat the Children’s Hospital of Illinois, has developed a fetal cardiac screeningcourse for sonographers. One challenge in training allied health professionalsis that their educational and clinical exposure may vary widely depending onwhere they trained and where they now work.

“This tool easily helped me to standardizemy educational content. Now that I’ve developed exactly what I want to teach, Ican more efficiently deliver this content to a broad audience in an immersivevirtual environment. I think this saves time for the educator and the learnerand should be a lot more fun for the learner as well.”

Dr. Jantzen is conducting a research studyto compare effectiveness, cost, and time spent with traditional versus virtualreality education formats.

“It makes intuitive sense that if you don’tneed the instructor, you are going to save money. We’re also looking at thelearner time spent in the course, since these are practicing professionals whoare paid when they go to class. We hypothesize that their actual ‘in-seat’ timeis going to be reduced significantly, which will provide an ongoing return.”

A DICOM image must be segmented at each layer to create an accurate 3D digital model.

ImprovedTraining

Dr. Teresa Riech, an emergency medicinephysician at OSF HealthCare, is a former combat medic, Flight Surgeon, andChief of Clinical Services for the 183rd Fighter Wing, who has witnessedfirst-hand the need for improved training in the principles of triage and fieldmedicine for medical officers and enlisted medics.

“We know that training in mass casualtytriage helps, but we don’t know what method works best or how realistic thescenarios need to be,” Riech says. (Bhalla, 2015) Riech has planned andexecuted many large-scale Mass Casualty Incident trainings, but notes that notraining exercise simulates the stress experienced in the real world.

“Currently we spend a lot of resources onlarge-scale live simulations, but we don’t know how well this kind of trainingtranslates to performance in a real situation.” (Kilner, 2011) She has createda triage training course in virtual reality, and has started deploying it forpre-hospital responders at all levels.

“I believe this kind of training is verygeneralizable: hospitals, municipalities, and military operations, from noviceto expert learners, would all benefit from exposure to this tool.”

One of Dr. Riech’s primary motivators isthe significant time savings that a VR course offers versus traditionaltraining modalities.

“Planning this course took a fraction ofthe time compared to a live mass casualty exercise. And now I can run aninfinite number of learners through it, without the normal expenses of travel,actors, and moulage. Once you have collected all your digital files andimported them into the virtual world, the actual teaching process is incrediblyeasy.”

Dr. Riech is excited about otheropportunities for training in VR. Pre-hospital response for chemical,biological, radiological, nuclear and explosive (CBRNE) situations, orpediatric mass casualty scenarios, are two areas where there is very little inthe literature about the best ways to train. “As you can imagine, the stress ofresponding to pediatric emergencies is much higher, and the requirements forcare are more nuanced than adults.”

David Dominguese, PhD, is a ResearchAssistant Professor of Anatomy and Director of Technology-Based MedicalEducation at the University of Illinois College of Medicine at Peoria. Earlierthis year, he participated in the school’s renovation of its anatomy lab, whichincluded construction of a new classroom for technology.

“A difficult decision to make was on thetype of hardware to use for virtual reality, because we know that it’s onlygoing to improve over time. Having access to the Enduvo software platform hasmade it very easy for me and other instructors, without any software developersupport, to quickly build interactive content.”

Professor Dominguese has publishedpreliminary results and presented early findings that both faculty andstudents’ have very positive reactions to using VR (Dominguese, 2018). Hiscurrent research is evaluating the effectiveness of VR on learning performanceversus traditional methods, and he hopes to publish his findings next year. “Wedecided that we wanted to investigate teaching and learning using VR andimplement into the curriculum early, so that we could pioneer technology andadd to scholarship in the field that is vastly needed.”

In the meantime, he and his colleagues areworking hard to develop more content that can augment or even replace some oftheir existing lectures. “We are implementing an enhanced flipped classroommodel, where students often spend approximately 30 to 60 minutes in the labduring a schedule week with each block of content. Currently, the biggestchallenge is logistical tracking each user’s time spent in VR and what is themost applicable content to deliver in VR.”

Even with these challenges, he and other instructors are enthusiastic about the teaching opportunities in VR. “What we know is that best practices for actual learning are supported with this tool. The learner has the freedom to interact with the anatomy in the most immersive way possible. For me as an instructor, I can receive instant feedback whether or not they are learning.”

Pre-surgical review of patient specific anatomy.

Challenges

As more institutions evaluate theeducational promise of augmented and virtual reality, they will have to grapplewith the challenge of understanding the rapidly evolving state of the hardwaretechnology, the growing number of software and content options, as well as thephysical space requirements for providing a lab. For UICOMP and JumpSimulation, the capital expenses for renovating space and purchasing equipmentfor a six-person VR lab were significantly lower than the cost of addingadditional high-fidelity simulation rooms. Yet even with access to the Enduvoplatform, there is still a significant challenge in coaxing educators to trymoving their content into VR.

Dr. Bramlet observes that “the biggestenergy gradient right now for our faculty is that they need to thinkdifferently about how they present their material in VR instead of PowerPoint.Even if I can show them that their one-hour lecture can be reduced to ten orfifteen minutes in VR, they still need to get comfortable with the environmentitself and also think about how they are going to present their content anddesign mini-assessments for each of their learning objectives.”

As arguments that VR is just anotherpassing fad are slowly losing steam with the many improvements in hardware performanceand user experience, it remains to be seen whether or not learners andinstitutions will embrace the financial and intellectual leap of faith requiredby early adopters. Many believe that there is an entirely new VR ecosystememerging just as with mobile device app development a decade ago, and GoldmanSachs estimates that the potential market size in healthcare and education forVR and AR software will be $5.8B by 2025. (Bellini, 2016) But VR developers arein high demand and their skills are not easy to acquire. In addition to atraditional software engineering background, they need to be fluent in 3Dtools, like Unity and Unreal, platform specific software development kits(SDKs), user interface and user experience (UI/UX) principles, as well as sounddesign.

In the meantime, Dr. Bramlet is excited about new collaborative opportunities that his Enduvo software platform can provide. “We’re finding that many institutions are much more willing to share curricular content than they were before. Our early customers are honestly just as excited about a marketplace for content as they are about the education and research that they can develop by themselves in VR.”

Mass casualty triage training in VR.

Aboutthe Author

Paul Pribaz, MS, serves as the Chair of theTechnology Committee for the Society for Simulation in Healthcare and teaches acourse on Simulation Program Administration at Drexel University’s Master ofScience in Medical and Healthcare Simulation (MSMS) program. Paul was the VicePresident of Simulation Administration for Jump Simulation, responsible forfinancial and operational oversight of simulation-based activities in researchand education. He previously worked as the Executive Director of SIMnext and ofJump Innovation and facilitated the collaboration of clinicians and engineersto develop next generation educational technologies.

REFERENCES

1. Bellini H, Chen W, Sugiyama M, Shin M, Alam S, Takayama D. Virtual & Augmented Reality: Understanding the Race for the Next Computing Platform. The Goldman Sachs Group, Inc. 2016.
2. Bhalla MC, Frey J, Rider C, Nord M, Hegerhorst M. Simple Triage Algorithm and Rapid Treatment and Sort, Assess, Lifesaving, Interventions, Treatment, and Transportation mass casualty triage methods for sensitivity, specificity, and predictive values. Am J Emerg Med. 2015;33(11):1687-91.
3. Dominguese D, Pan S-C, Harris J, Bramlet MA. Virtual reality teaching and learning platform for medical school education. In: Langran E, Borup J, editors. Society for IT & Teacher Education International Conference 2018; Washington, D.C., United States: Association for the Adv. of Computing in Education (AACE); 2018. p. 1109-12.
4. Kilner T, Brace SJ, Cooke MW, Stallard N, Bleetman A, Perkins GD. In "Big Bang" Major Incidents do Triage Tools Accurately Predict Clinical Priority? A Systematic Review. Injury. 2011;42(5):460-8.

Originally published in Issue 3, 2018 of MTM Magazine.