Melissa Mathews identifies common mistakes and describes necessary steps to conduct a ROI evaluation.

For simulationists and administrative leaders of simulation programs, understanding how to show and report the simulation return on investment is critical to the growth and sustainment of the program. Leading programs have a continuous uptake of new simulators, task trainers and program models as well as a reporting process to financial stakeholders. This article will identify the common missed elements in most simulation investment proposals, review key measures within a sound program evaluation and the steps taken to conducting a return on investment evaluation.

Advancing simulation to the next level requires making the case for investment.
Advancing simulation to the next level requires making the case for investment.

Many decisions are made in healthcare that impact investments made to improve the education of future healthcare providers, prevent injury and harm to patients as well as those interventions which improve the efficiency and quality of care. Financial and administrative decision makers in healthcare organizations must make intelligent investments to reduce risk to organizations within the healthcare system already stressed to reduce healthcare costs and improve quality. Educational leaders are under pressures to improve achievement scores and pass rates, meet the demands of the clinical skill requirements, increase throughput, decrease cost, improve instruction and deliver quality trained healthcare professionals.

Making the case for investments in simulation as a tool to assess risk, test systems, and train clinicians is often difficult for simulationists who often enter the profession as clinicians or educators with little experience in business and finance. Empowering simulationists with information on how to present proposals for simulation programs and simulation technology investment is critical to improving healthcare and reducing healthcare costs.

Several things are often missed in making the return on investment argument:

  • Simulationists are not clearly articulating return on investment in proposals when asking for simulation technology. Proposals should contain two or more approaches to training (ie computer-based training versus simulation based.)
  • Simulationists are not measuring return on investment alongside program evaluations and reporting out to financial stakeholders.
  • Making the objective argument for simulation technology remains economically challenging due to intangible benefits. Nonetheless, proposals should factor in and state intangible benefits as part of the ROI. (Asche et al. 2017)

The first step to begin the simulation investment proposal with a return on investment is to complete a plan for a full circle program evaluation if the technology will be used for professional development. Collecting any preliminary or pilot information, even if anecdotal is key to the planning and measurement reporting to the stakeholders making decisions. Capturing the current state and gaps here is critical. While Kirkpatrick is often the model used for program evaluation, I recommend a closer look at Don Moore in his article “Achieving desired results and improved outcomes: integrating planning and assessment throughout learning activities”. Moore captures the relevant key aspects of program evaluation for education geared towards healthcare professional development. Most importantly, Moore drives us to measuring to the level of improving healthcare quality and community health as the ultimate measure of success in the use of simulation technology. This outcome is the sharp end of the spear in terms of improving patient safety and quality care.

Chart A - Program evaluation Moore versus Kirkpatrick.
Chart A - Program evaluation Moore versus Kirkpatrick.

Next it is important to determine two or more approaches to closing the performance gap or solving the problem including the current approach with or without simulation or different simulation approaches. Comparing methods shows the stakeholders and financial decision maker’s current approaches and possible alternatives. It is also good to show the potential benefit and yield for low cost effective interventions versus more expensive effective interventions.

The return on investment should be conducted for all methods. The standard ROI is calculated by beginning to collect the costs of training. The standard ROI is calculated as follows:

ROI = (net benefits/costs) X100

where net benefits = benefits-costs.

For a decision maker, a program is a good investment if ROI > 0. The chart below demonstrates and provides examples of measures in a step by step process. Notwithstanding, also providing measures of intangible benefits.

Tangible benefits are those measured in costs and intangible benefits cannot be measured directly but they do have a very significant business impact. Tangible benefits: Making the processes and personnel more efficient and effective while reducing the cost of care and services. Intangible benefits are those things that are difficult to measure directly, such as factors related to:

Patients: personalized healthcare leads to improved outcomes and HCAHPS scores.

Professionals: increase engagement and increase experiential learning opportunities.

Families: enhanced understanding, trust, and confidence.

Proceduralists: simulation rehearsal, preplanning and preparation leads to greater confidence and understanding. Opportunities to improve processes and/or techniques through innovation. Improved communication among multi-disciplinary team.

Students / Residents / Clinical Staff: improved procedure, technique, processes, and disease education.

Researchers / Innovators: providing the tools and biomedical engineering that allows the “bench to bedside” discovery.

Educators: providing the instructional scaffoldings that lead to improved educational outcomes.

Marketing: increase the college or hospital’s brand awareness and image.

Financial Stakeholders: decreasing medical errors, decreasing safety events, reducing malpractice lawsuits. Build case for malpractice insurance reduction and insurance reimbursement and common standards.

Public branding and relations: Simulation would indirectly impact a college or hospital’s reputation by adding another structural resource that leads to an improved process of delivering care which leads to improved outcomes. Patient Safety is also improved as it is related to the process of healthcare delivery.

Chart B - Steps to Measuring Return on Investment.

In conclusion, providing robust simulation proposals involves comparing choices, reporting and including intangible benefits. An approach to reporting the education effectiveness by an in-depth program evaluation is critical to proving the educational success. Finally following the steps to conducting a return on investment will contribute to producing a case for using simulation technology. Taking the time to write and produce proposals that lay out what will be measured and a plan for reporting is critical to improving the chances for receiving financial support for simulation investments from financial leaders at your organizations. Advancing simulation to the next level will take a commitment to producing proposals that are robust and make the case for investment. 

About the Author
Melissa Mathews, BSN, MHPE, RN-BC holds a Master’s degree in Health Professional Education from Vanderbilt University. She is a Certified Healthcare Simulation Educator. She is an experienced educator and innovator with 12 years’ experience in simulation-based education programs; designing, developing and evaluating programs. Her focus is on improving performance, patient safety and quality through simulation. She is a member of the Society for Simulation in Healthcare, sits on several technology advisory boards and is the Co-Chair of the Florida Healthcare Simulation Alliance. She is an Advanced Education Sales Executive at SynDaver.

Asche, C. V., Kim, M., Brown, A., Golden, A., Laack, T. A., Rosario, J., ... & Okuda, Y. (2018). Communicating Value in Simulation: Cost–Benefit Analysis and Return on Investment. Academic Emergency Medicine, 25(2), 230-237.
Steven Boggs, M. D., & Okuda, Y. (2014). Cutting costs while maintaining quality: how the VA has leveraged simulation. Physician executive, 40(2), 38.
Moore Jr, D. E., Green, J. S., & Gallis, H. A. (2009). Achieving desired results and improved outcomes: integrating planning and assessment throughout learning activities. Journal of continuing education in the health professions, 29(1), 1-15.
Twigg, D., & McCullough, K. (2014). Nurse retention: a review of strategies to create and enhance positive practice environments in clinical settings. International journal of nursing studies, 51(1), 85-92.
Kirkpatrick, D. L. (1977). Evaluating training programs: Evidence vs. proof. Training Dev J.

Originally published in Issue 4, 2018 of MT Magazine.