COMRADE: Methods for Adaptive Competency Management and Just-in-Time Clinical Acumen Augmentation

Dr. Thomas Talbot shares ideas for enhancing the Electronic Medical Record to act as a didactic tool to support physician competency.

Although we have fantastic lifelike manikins, virtual reality surgeries, artificial intelligence models, engaging software simulators and even medical games, very little innovation has gone to the medical practice environment where the vast majority of patient encounters are: the clinic.

After graduating residency, physicians are often left to their own devices in regards to the nature of their ongoing continuing medical education (CME), typically taking a number of self-selected CME credits per year and following a recertification process periodically. Such a system has proven helpful in maintaining an overall level global demonstrable knowledge within the physician population. Current CME and maintenance of certification (MOC) regimen currently do not target maintaining a physician’s breadth of competency nor do they target an individual’s weaknesses because they do not assess the physician under practice.

To maintain a full-breadth capability for a physician in a particular specialty, there is interest in developing adaptive competency management systems such as the Continuous Observation of Medical Records for Advanced Doctor Education (COMRADE) concept. This concept constructs a model of physician encounters by collating diagnostic codes from clinical visits and comparing the data to a list of expected competencies for that specialty. When combined with predictive models of clinical skills decay, it should be possible to statistically determine an actual or impending degradation and counter it.

The Need for Adaptive Competency Management

To understand how such a concept can work, let’s examine three physicians and how their situations can be improved through adaptive competency management. Dr. Allen is a successful family practice and aerospace medicine physician. He primarily sees pilots and their families. His practice is light on women’s medicine and chronic diseases of the elderly. He worries that he is not seeing enough women to handle patient complaints about conditions he rarely sees. Major Brown is a military pediatrician who has just returned from a one year deployment to Afghanistan which involved administrative duties and sporadic adult medical encounters. He is now at a medium size hospital where he is responsible for complex patients and a fairly large number of neonatal intensive care graduates that he has rarely encountered since residency. Dr. Cohen is a busy internist who keeps up her CME, though she is unaware that she will be seeing one patient with unusual complaints of vertigo and another with Klippel-Trenaunay-Weber syndrome the next week and she will be unprepared to comfortably manage these encounters.

Each of these three are good doctors, but can benefit from adaptive competency management. Dr. Allen needs to maintain broad competency as a family physician. Dr. Brown has been out of practice and needs to be reintegrated; Dr. Cohen sees a variety of patients, but will be unprepared for two unfamiliar cases in her upcoming schedule.

How Does Adaptive Competency Management Work?

My COMRADE concept addresses these issues. The theory is that physicians will tend to be familiar with patient conditions they see on a regular basis, but over time, will become increasingly uncomfortable with conditions rarely or infrequently seen. It is important to note that board examination tests may easily miss this unfamiliarity. Physicians are astute test takers who can leverage scaffolding within multiple choice questions to find associations that make them appear knowledgeable. When a physician encounters a symptom or diagnoses that have become unfamiliar, they struggle to come up with interview questions and feel uncomfortable trying to recall the proper diagnostic approach and management of that condition.

Since military physicians have been using a unified electronic medical record (EMR) for about 10 years, it is a trivial matter to construct a map of the diagnoses that they see and compare that against that physician’s expected range of knowledge for their board specialty. Using mathematical models to predict clinical skill decay, the COMRADE system will determine areas of likely weakness and schedule appropriate virtual standardized patient (VSP) encounters into their schedules.

The COMRADE concept requires use of four technologies which are either currently available or can be made available in the near future: 1) EMR Data Mining, 2) A Skill Decay Model, 3) A Training Intervention Scheduler and 4) Virtual Standardized Patients (Figure 1).

The first COMRADE technology is electronic medical record (EMR) data mining. This is necessary to construct a map of a particular physician’s encounters. The easiest method will probably be to poll the ICD-9/10 codes of all encounters under that physician and aggregate them. A running map will be generated of encounters over the last 3-5 years, the last year and the last three months. It may be possible to delve deeper into medical records for additional data to answer questions about quality of care. An example would be aggregation of the physician’s ADHD, Diabetes and Asthma cases; the system could determine if they appear to meet clinical practice guidelines. The initial COMRADE concept, however, is designed to use only ICD-9/10 codes as a frequency of exposure indicator. If a practice record is not available, it may be possible to substitute a screening test or even specialty board scores for baseline data.

The second COMRADE technology is skill decay determination (Figure 2). The clinical exposure map produced by EMR data mining and other measures will provide a frequency of exposure estimate across a range of conditions. This can be compared against a physician’s expected clinical knowledge set that is determined by their specialty board organization’s list of expected competencies. The ability for an automated system to do this depends on having an accurate skill decay model that can predict areas of weakness. The Defense Medical Research Development Program (DMRDP) and TATRC have funded work to construct these skill decay models. Currently, the Nemours Foundation is constructing a clinical skill decay model using their pediatricians and Design Interactive is constructing a model in the general medicine arena. These groups are also investigating the intervention level necessary to maintain skills. Additionally, research is underway at the University of Maryland and the University of Wisconsin which are independently constructing procedural skill decay models.

The third COMRADE technology is a training intervention scheduler. The training scheduler takes a training priority list from the skills decay model. With access to the physician’s patient schedule, the training scheduler looks for appointment openings and schedules training. It is important to note that this training is not scheduled ad hoc, after hours, during lunch or on the physician’s free time – it is scheduled during the clinic day like any other clinical encounter, as it is a substitute patient experience.

The amount of time needed for training will vary considerably. For Dr. Allen, who needs to maintain competency and breadth of practice, the scheduler might put in 1-2 training appointments per week. For Dr. Brown, who is reintegrating into practice, the scheduler may reserve half his time for simulated patients, then gradually ramp down the percentage of training time. Dr. Cohen’s case represents a particular opportunity, that of just-in-time training. A training scheduler can look ahead in her booking schedule for unusual conditions. If an unusual patient is encountered, the scheduler can poll the skills decay model to determine the likelihood the physician is familiar with that condition. If it appears unfamiliar, the scheduler can preemptively provide a simulated encounter or can even review materials in advance so that she is prepared.

The fourth COMRADE technology is the Virtual Standardized Patient (VSP). The VSP (Figure 3), a near future technology, is an interactive virtual human avatar with a number of advanced features. VSPs are capable of understanding natural language questions from the physician and responding by verbally answering both open and closed ended questions with either an evolving narrative or direct responses. VSPs are non-verbally and emotionally expressive. VSPs have very sophisticated assessment systems that evaluate interviewing skills and case specific diagnostic skills. VSPs can be bundled with review material.

The creation of VSPs is the focus of my work at USC Institute for Creative Technologies (ICT) and will result in the USC Standard Patient to be beta tested in early 2015. Standard Patient Hospital will be freeware and include an authoring system in the attempt to achieve the critical mass of patient variety that will be needed by a system such as COMRADE. The VSP represents a considerable advance in simulated patient interaction, standardization and assessment (Talbot 2012a).

It is important to note that VSP encounters are employed instead of review questions, case presentations, or multimedia Virtual Patients (VP). VPs are effective for clinical decision making, but provide a lot of scaffolding to the learner that impairs our ability to assess their current level of interview skills which are also case knowledge (Talbot 2012b). VSPs are interview skill specific and will, in my opinion, give an excellent approximation of how well a physician is likely to perform in an actual patient encounter.

Obtaining un-scaffolded physician performance data is critical because accurate assessment closes the COMRADE feedback loop and provides retargeting information to the skills decay model. In a typical intervention, the physician will have their ‘appointment’ with the VSP. After the initial VSP encounter, the physician’s performance is assessed and they are provided with specific feedback about what they got right and where they can improve for a case interview. The physician is then given the opportunity to repeat the encounter and assess their improvement. The encounter may even be rescheduled later to take advantage of the ‘graduated interval recall’ effect (Pavlik 2008, May 2014).

What would this intervention look like to our model doctors? Dr. Allen is likely to see cases like a VSP with dysmenorrhea. This VSP will assess his ability to take a proper gynecological history, followed by an assessment and a review of the management of various causes. Dr. Brown will see lots of VSPs to ramp up his skill levels, but will be given extra materials targeting late management of preemies. Dr. Cohen, our just-in-time candidate, will rehearse with a VSP to review how to investigate vertigo symptoms. She also might receive a Medscape, Wikipedia or CDC link to review material for a rare condition like Klippel-Trenaunay-Weber which has no VSP case in the COMRADE library.

Conclusion

For a COMRADE concept to work, the assessment, statistical evaluation, and interventions will need to be closely integrated into the physician’s electronic health record interface and have a low burden on the user. Adaptive knowledge management has the potential to keep physicians keen on a broad basis and allow primary care physicians to feel comfortable managing more complex diagnoses without constant specialist input. The potential for improved physician satisfaction, patient safety and improved efficiency for the medical system is great. Current work is at the stage of developing virtual standardized patient technologies and proving the breadth/exposure relationship as well as constructing the mathematical models that can predict likely clinical skills decay and the optimal refresher intervals.

Although work has been done to enable a concept like COMRADE, there is no guarantee that such a system will be created. The success of adaptive competency management will depend on political will, organizational commitment, technology development, testing & evaluation, and significant funding. The current state of this concept, as of the date of this publication, is that the virtual standardized patient technology is functioning and about to be fielded with an eye toward extending capabilities to a high-fidelity virtual physical examination. Research to validate skills decay model employing EMR data is underway through two independent efforts by Nemours Foundation & Design Interactive.

If such an adaptive competency management system is created, the benefits will be enormous. Physicians will have guidance and support during their clinical career and primary care physicians will maintain masteries that allow them to handle a great number of medical complaints and conditions that have been moving, often unnecessarily, to expensive specialists. This will save time and money. It will improve physician career satisfaction, especially in primary care, where the work is devolving into the role of super nurse practitioner and gatekeeper for any complaint requiring a bit of thought. This system will also largely automate the execution and documentation maintenance of competency efforts.

The EMR must not be a cross to bear for the physician, it must be a tool that empowers them, and makes their job easier. It must serve to improve the lives of our patients. The priorities we set today will determine our course.

About the Author

Dr. Talbot is the Principal Medical Expert & Associate Professor of Medical Education, University of Southern California Institute for Creative Technologies (ICT) Chief Scientist, Armed Forces Simulation Institute for Medicine at the US Army Telemedicine & Advanced Technology Research Center (TATRC).

References

Talbot TB, Sagae K, John B, Rizzo AA. Designing useful virtual standardized patient encounters. Interservice/Industry Training, Simulation and Education Conference Proceedings, 2012.

Talbot TB, Sagae K, John B, Rizzo AA. Sorting out the Virtual Patient: How to exploit artificial intelligence, game technology and sound educational practices to create engaging role-playing simulations. International Journal of Gaming and Computer Mediated Simulations. 2012:4(4).

Pavlik PI, Anderson JR. Using a model to compute the optimal schedule of practice. Journal of Experimental Psychology 2008:14(2): 101–117.

May W, Talbot TB, Woerhle T, Souder D. Behind the Scenes of a Virtual Standardized Patient Authoring System Development. International Meeting on Simulation in Healthcare, January 2014.