Abstract: This article discusses the role that virtual and augmented reality-based learning is starting to play in the education of future healthcare professionals through the medium of simulation scenarios/technologies and how usability testing (a technique designed to evaluate a product by repeatedly testing it on users) can be incorporated to provide optimal experiences for end users. Virtual and augmented reality technologies are becoming more advanced and are being adapted from their video game and entertainment origins into healthcare learning technologies. These new technologies are no exception to the need for usability testing. One such technology is called BodyExplorer© by Lumis Corporation. BodyExplorerã is an augmented reality learning system that projects images of anatomy and physiology onto the surface of any type of manikin. Users can interact with projected human anatomy and physiology, for example, see a functioning heart and lungs, use a light pen to strip away layers of the chest and pericardium and even perform guided simulated medication administrations. Through the use of The System Usability Scale (SUS) and individual surveys, an overall usability rating was produced for BodyExplorer© . In addition to a usability rating, insights for additional uses and alternate ways that the product could be adapted to better facilitate learning in an educational setting were discovered. By examining the data produced from usability tests on these types of education technologies, it is possible to identify and control use-related risks and hazards, minimize instances of technology induced error, and increase patient safety.
Introduction
In the United States, and many developed countries around the world, medical error induced deaths, also considered as types of Preventable Adverse Event (PAEs), are a common occurrence with over 400,000 PAE deaths per year as of 2013 (James, 2013). As defined by Martin and Daniel (2016), medical error is “an act (either of omission or commission) or one that does not achieve its intended outcome, the failure of a planned action to be completed as intended (an error of execution), the use of a wrong plan to achieve an aim (an error of planning), or a deviation from the process of care that may or may not cause harm to the patient.” Preventable medical errors, currently the third leading cause of death in the United States behind heart disease and cancer, can prematurely end an otherwise long life or accelerate an imminent death (Martin & Daniel, 2016; Kiernan, 2018). While medical errors are caused by many different situations (including but not limited to: negligence of healthcare staff, overconfidence in healthcare staff by family members, over/under confidence of a clinicians or nurses in their own judgements, and post-surgical complications), there are also ways to combat instances of this type of error. The training of healthcare professionals is an important factor that can contribute to the elimination of medical errors. The ability of nurses and clinicians to be appropriately confident in their clinical judgments is an essential part of safe and effective healthcare (Yang, Thompson & Bland, 2012). The more experienced and confident a nurse/medical professional is with using medical technologies, performing medical procedures, administering crisis interventions, and identifying symptoms quickly and accurately, the less likely a deadly error is to occur (Kahriman, et al., 2018). With this in mind, it can be inferred that the thorough training of nurses and health care professionals is a necessary, and worthwhile, investment.
Different methods are currently utilized to train upcoming nurses and healthcare professionals who are enrolled in collegiate and vocational programs. Many of these institutions have implemented training programs that focus on exposing the students to realistic scenarios and other situations that could arise in a clinical setting. These training programs utilize healthcare simulation (SIM) of various types, as well as medical simulations/simulators that incorporate virtual reality (VR) and augmented reality (AR) components. Some programs even offer technological training on the medical devices (both in and outside of SIM) that will be used to help acclimate their students to the reality of clinical practice ("Edinboro Simulation Lab," n.d.; McMacken, 2019; "Medical Simulation," n.d.; "Research and Innovation," n.d.; "Simulation Lab," n.d.).
The simulation of real-world situations in healthcare is a proven aid in healthcare education. Similar to how the usefulness of simulation has been appreciated, the need for usability evaluation at each stage of product development is also starting to be realized. As innovative products are developed for health care that are new hybrids of medical training technologies, it will be necessary to ensure that they are actually usable. One such hybrid medical training technology is the BodyExplorer© Augmented Reality Learning System. BodyExplorer© projects images of anatomy and physiology onto the surface of a plastic/latex mannequin from a movable adjustable pole with a device bolted onto the post that is suspended above the manikin’s torso (see Figure 1). Users can interact with the projected anatomy and physiology by using a light pen to strip away virtual layers of the chest and pericardium in order to see the heart and lungs working, as well as experience the results of specific simulated drug injections by progressing through a drug recognition Medication Administration module (see Figures 2 & 3). When fully functional, the BodyExplorer© system also has the capability to perform simulated cricoid pressure and rapid sequence intubation learning modules (D. Nelson, personal communication, November 28, 2018).
While advancing technologies have the potential to increase efficiency in simulation learning, factors such as a user’s age, technological experience, and comfort level with technological advancements influence the technologies’ rate of success. Because potential users of these technologies can have such varied uses for and levels of interactions with the same products, usability testing and usability studies are necessary tools in the assessment of these types of systems/devices. Usability tests and studies are helpful in identifying and controlling use-related risks and hazards, minimizing instances of technology induced error (TIE), and ultimately in improving education and increasing patient safety. Since the BodyExplorer© is an AR system, it can be used on virtually any surface and can be programed to fit the needs of virtually any end-user. With a wide range of uses, and an even wider range of potential users, it is important that this technology is as user friendly as possible. The purpose of this BodyExplorer© usability study was to help make the system more user-friendly in the future and an overall better learning tool for the students utilizing it.
Background
Specific usability problems and occurrences of technology-induced medical error share a close and statistical relationship (Koutkias, Niès, Jensen, Maglaveras, Beuscart, Kushniruk, and Borycki, 2011). Koutkias et al. also noted that such errors may be prevented through application of different methods including usability testing. This research shows that the main approach to usability engineering is known as usability testing, which is a practical and scientific approach towards evaluating how usable systems are (Koutkias et al., 2011). Many usability issues only appear when the real-world context of use is replicated in the laboratory setting, making testing that much more important (Dahl, Alsos, & Svanæs, 2010). Usability testing can also provide feedback to designers about ways of improving the usability, safety, and degree of integration of a device/system with areas such as education.
Usability testing relies heavily on the use of usability studies. Usability studies, like most other forms of testing, are employed when initially developing a system or technology or after a system or technology has been found to function less than ideally for its intended purpose. Usability tests are multi-step procedures with a main goal of observing people (e.g. how they approached the device, if they seemed hesitant, what they touched first, where they stood, if there were any universal questions or concerns about the device, if the device attracted people of the same ages, races, and genders etc.) while using a device or system that is being investigated.
Usability studies are especially necessary for medical and healthcare devices. Many healthcare devices that are utilized within a customer’s home (e.g. glucometers) require relatively long series of sequential tasks to be performed, are sensitive to user procedural errors, and may lead to health risks if used incorrectly. These devices can be more difficult for older adults to use, not only because these users are less familiar and comfortable with such devices, but also because they may have to remember sequences of steps to operate them properly (Mykityshyn, Fisk, & Rogers, 2002). More people are living longer and as a result are relying more heavily on medical equipment to monitor their health, or to provide home treatment, making the usability of these devices an imperative feature. Because these devices are so necessary and directly impact people’s health, good design is especially critical. These same concepts can be applied to technologies and systems that are developed for the training of healthcare students and professionals.
Usability studies are an important part of the design and success of many different technologies. When properly utilized, usability studies can identify problems from user feedback. Usability studies allow engineers and researchers to easily correct these problems as long as they are willing to work closely with end users and take their advice. Usability studies will be needed greatly as medical simulation training technologies continue to develop and merge with other modalities such as Virtual Reality, Augmented Reality, and Mixed Reality (MR). Products similar to BodyExplorer© are being developed that also require usability testing to ensure that they will provide the user with the best learning experience. One such product is a neurosurgical training tool that allows images of dissected cattle brains to be projected onto a physical specimen to guide students and professionals through various trainings and surgical procedures (Gokyar & Cokluk, 2018). Another similar study was conducted on the “Cognitive Augmented Reality Cubes (CogARC) product. CogARC is an AR game that was developed to help students and caregivers screen elderly patients for the early detection of dementia (Boletsis & McCallum, 2016). Many other devices exist, such as the SleeveAR, MirrARbilitation, and AR-REHAB, that are designed to help students, professionals and patients progress through rehabilitation by utilizing motion tracking and specifically designed rehabilitation-based game challenges (Cavalcanti, Santana, Gama, & Walter, 2018). In each of these cases, the usability of the device was examined in order to determine its usefulness and its potential to be used as a learning tool for students and professionals alike.
In addition to these AR based products, some VR training tools are also being created and tested for usability. One such tool is a VR sterile catheterization game made using the Occulus Rift system (Kardong-Edgren, Breitkreuz, Werb, Forman, & Ellertson, 2019). Other VR training products include a simulation that helps students learn clinical strategies when performing mental health assessments (Verkuyl, Romaniuk, & Mastrilli, 2018) and a game used to develop a student’s pediatric nursing and care skills (Verkuyl, Atack, Mastrilli, & Romaniuk, 2016). These products are among the many innovative products that the healthcare training industry can expect to see in the coming years. This new era of simulation and training technologies will ultimately pose unique challenges to designers and manufactures as they attempt to take all of the factors that were previously discussed into consideration and turn them into successful products. With the fields of simulation education and VR/AR/MR rapidly expanding, usability studies will be necessary if these technologies are to be successful in their intended purposes. The BodyExplorer© Study performed by the author of this paper is one such example of a usability study conducted on a piece of nursing education technology that produced a usability rating and recommendations for other system applications. This paper is a report and discussion of the BodyExplorer study which is provided below in its entirety.
BodyExplorer© Study
Demographics
After IRB approval, this mixed methods study was conducted at a small private university in a mid-Atlantic state. Snowball sampling, a recruitment method by which research participants recruit others for a test or study by word of mouth referral, was used to find participants. Any student, faculty/professor, or professional in a health profession (including biomedical engineering) or other persons who would either use physiological modeling in their courses or professional endeavors were invited to test the BodyExplorer© system and give their input. Sixty-five participants were recruited for the study. Of the sixty-five participants, six were used for initial pilot testing of the study’s protocol. The remaining 59 participants were men and women of varying ages, education levels, and areas of study.
Pilot Study
A pilot study was performed at a skills lab in an accredited simulation center with six subjects: all faculty members. The six participants were exposed to the BodyExplorer© system for no longer than thirty minutes during which a moderator was present to introduce the system and demonstrate how to interact with it. After interacting with the system, each of the six participants were then asked to complete a survey about their perception of the system’s usability. When answering the Demographics section of the survey, three of the subjects self-reported an attained education level of graduate school, two self-reported an attainment of both an undergraduate and a graduate level education, while one subject chose the “other” option without specification. When asked about their field of study, four of the subjects in the pilot study reported “Nursing”, one subject reported “Health Services Administration”, and one subject cited “Nuclear Medicine Technology.” Five of these subjects reported their age as being between fifty-one and sixty years, while one reported an age of sixty-one years or older. As a result of the pilot study participants’ survey responses, the demographic questions were adjusted before re-administering the survey.
Instrument
The System Usability Scale (SUS) is a tool for measuring usability of a wide variety of products and services, including hardware, software, mobile devices, websites, and applications (see Figure 4). In this fashion, the SUS has the advantage of being “technology agnostic” (Kortum & Acemyan, 2013; Brooke, 2013). The SUS is not diagnostic, but solely for use in classifying the ease of use of the site, application, or environment being tested. The SUS consists of a ten-item questionnaire with five response options for respondents; Strongly Disagree, Disagree, Neutral, Agree, Strongly Agree (Brooke, 1996; Kortum & Acemyan, 2013). In the SUS tool, the word “Website” was changed to “BodyExplorer” to better reflect the purpose of this survey and the response options were slightly altered: Agree, Somewhat Agree, Neutral, Somewhat Disagree, Disagree (see Figure 5). Each question in this ten-item survey can be answered by selecting one of five responses ranging from “Disagree” on the left to “Agree” on the right. Each of the five responses is assigned a point value with “Disagree” corresponding to 0 and “Agree” corresponding to 5. The SUS has been shown to be a reliable and valid instrument that is freely available for use with a Cronbach’s alpha of 0.92 (Bangor, Kortum, & Miller, 2008; Kirakowski, 2019). The SUS has references in over 1,300 articles and publications and has become an industry standard ("System Usability Scale (SUS)", 2013).
Procedure
The study was conducted at a skills lab in an accredited simulation center. Each learning session took no longer than thirty minutes. During the sessions, consents were obtained from each participant before they were introduced to the BodyExplorer© system. One moderator was present for each session. The moderator introduced the BodyExplorer© and demonstrated how to interact with it. To interact with the system, the user held an infrared light pen that allowed them to manipulate the virtual anatomy being displayed, as well as start and stop various features including simulated heart and lung sounds and an electrocardiogram (ECG) monitor window. Users could also administer a simulated drug through the use of a syringe and a simulated intravenous injection site. While participants interacted with the system, the moderator described some of the functions of, and ideas behind, the system. The moderator also initiated any necessary system-resets, prepared for the next testing session, and extended session times if the participant wanted to share more feedback on their experience.
Data Collection
At the end of their session with the BodyExplorer©, each participant was asked to fill out a ten-item usability study questionnaire on an iPad designed to further elicit their impressions of BodyExplorer©. Participants were asked to provide answers to questions about how the BodyExplorer© system could be used in other educational settings. Throughout the entire session, the moderator assisted the participants with technical issues as needed and took notes on the participants’ experiences and comments.
Data Analysis
After the data collection phase of the study was completed, the data from the survey was processed and analyzed. Once participants’ scores were obtained for each of the ten questions in the survey, initially ranged from 0 to 5, the values were converted to numbers that corresponded to those of the SUS scale. According to the SUS scale’s process requirements, rescoring was completed in order to normalize the scores and produce a percentile ranking with associated adjective rating scale ("System Usability Scale (SUS)", 2013). Per SUS tool instructions, the rescoring is done differently for even and odd numbered questions. The initial scores from even numbered questions (2, 4, 6, 8, and 10) were each subtracted from 5. The initial scores from the odd numbered questions (1, 3, 5, 7, and 9) each had 1 subtracted from them. Each participant’s converted scores were then added together across the 10 questions and the sum was multiplied by 2.5. This converted the sum of the original scores, initially 0-40, to post-conversion scores ranging from 0 to 100. These numbers represented percentiles and not percentages. Adjectives associated with scores were “average” = 50.9 (SD, 13.8), a “good” score was 71.4 (SD, 11.6), and an “excellent” score was 85.5 (SD, 10.4). SUS scores and the adjective rating scale represented a measure of the perceived usability of the BodyExplorer© (Bangor et al., 2009).
Results
For the sixty-five participants who responded to the survey, the SUS tool provided insight on the usability of the BodyExplorer© system. The average SUS score reported across various technologies is 68 out of 100 (these scaled scores are not percentages) placing the average score in the 50th percentile (Bangor, Kortum, & Miller, 2008; Brooke, 2013; Kirakowski, 2019; Tullis, & Albert, 2008; Sauro, 2011). The average SUS score for BodyExplorer© was 73.38 (SD: 14.87, range: 58.51-88.25). According to the percentile ranking of SUS scores, this places BodyExplorer© slightly above the 60th percentile (Sauro, 2011). This indicates that BodyExplorer© has a higher usability score than a majority of all applications tested.
A review of digital video recordings of the participants interacting with BodyExplorer© indicated that there was an overall satisfaction with the system. Participants overwhelmingly reported that they enjoyed using the system. Participants thought that the current functionality (simulated breathing, heart and lung noises, and audio and visual feedback) was impressive and helpful. Both the anatomy windowing and the drug recognition modules were met with curiosity and approval. Some participants even thought that the technology was intriguing and questioned how it was programmed. Many users speculated that BodyExplorer© would be a feasible replacement for the expensive Anatomage Tables currently on the market. A majority of the participants also stated that they believe the BodyExplorer© would be more useful as a learning tool than any of their anatomy textbooks.
A few major themes emerged from the recorded data. Participants reported multiple different ideas for future curriculums and other uses for the system outside of nursing and health care training (see Table 1 below).
While a vast majority of the comments and feedback about BodyExplorer© was positive, there were some aspects that participants were skeptical about. Many of the negative comments concerned confusion about how the light pen and Wii remote function together. Participants as a whole struggled to use the light pen to navigate the system. While a majority of these participants were able to adapt to and figure out how to best use the light pen, some were unable to completely grasp the technique. Another concern was about the calibration process used for the system. Participants with knowledge of this topic expressed concerns about the inaccuracies in the calibration process being comparable to that of the Smart Board white board’s calibration system. Another main concern was about the complexity of the setup procedure. After having the procedure briefly explained to them, participants commented that they did not believe they could set it up without extensive help from a moderator or professor.
Discussion
The direct interaction with the system combined with the SUS survey and audio and visual recordings of all sessions allowed for the evaluation of the overall usability experience of the BodyExplorer© Learning System both qualitatively and quantitatively. Based on the findings from the 65 total responses to the SUS, the BodyExplorer’s usability rating can be considered “good” in accordance with the adjective scoring system. This result suggests that the BodyExplorer© system is perceived to work well but has room for definite improvements despite scoring above average according to the SUS scoring system. When the qualitative data gathered from recordings and direct interactions with participants is considered along with the SUS score, it can be concluded that BodyExplorer© could potentially be marketed as a viable product in its current state, although it is not as polished as it could be.
General reactions of participants to the system show that the technology is still so new and interesting that most participants are willing to ignore some to the minor inconveniences and technical glitches present in the BodyExplorer’s system. While potential users are enthralled by the current technology and its vast range of potential uses, improvements are necessary if the system is to improve enough to be commercially marketable. The identification and collection of other potential uses for the system were another factor in the organizational layout of future improvement and product development. Usability, while an indicator of overall ease of use, is not necessarily indicative of the product’s future market success. Due to constant developments in the medical field, devices must be capable of change to ensure that the best possible standard of care is provided to patients without compromising the availability and profitability of the device itself. In addition to usability, the system should also be evaluated for its efficacy of learning. A huge consideration for a product like this is if people are actually learning more efficiently using this AR learning tool than they are with other traditional methods. Once these questions have been answered then one of the last factors that must be considered is the return on investment that can be expected for institutions that would purchase this system. Further studies are recommended for the BodyExplorer© Learning System to answer these questions.
Conclusions & Recommendations
This study evaluated the usability of the BodyExplorer© Learning System. BodyExplorer© was found to be a viable learning tool for nursing students with a usability rating of “Good” according to the SUS scale. The system is fairly intuitive but still requires the presence of a trained operator for all use. Most users found that they could adapt to the system and eventually learn with it. From the interactions had with BodyExplorer©, participants of this study produced many suggestions for potential alterations to the system. Many participants could see multiple uses for the technology as it is developed further and suggested both improvements and alternate applications of the system. Based on the results of this study, these suggestions can be sorted into two main categories: recommendations for future curriculums and recommendations for uses outside of the healthcare field.
Recommendations for future curriculums that could be developed for the BodyExplorer© include modules that cover:
Anesthesia Practice
Anatomy & Physiology of other bodily structures
Isolation & Manipulation of specific organs/organ structures
Real-time Drug Recognition
Medication Administration
Long-term Medication Effects
Blood Pressure Practice
Auscultation Practice – Bowel, Heart, Lungs
Effects of Disease on Internal Bodily Structures
Intubation Practice
Vein Occlusion
These suggestions provide a comprehensive potential suite of training modules for students and professionals who would use BodyExplorer© as a training tool. The creation of these customized training modules would provide a wider client base for this product. With added functionality and a wider client base, the potential for increase in revenue once BodyExplorer© is commercialized could also be expected to increase. In addition to expanding BodyExplorer’s functionality in the healthcare training realm, creating alternate versions of this technology to be applicable to other industries and training programs could have a similar effect on the expected client base and revenue streams. With this in mind, recommendations for uses of BodyExplorer© outside of the healthcare field include modules that cover:
Crime Scene Analysis
Forensics – Decomposition Rates & Body Exhumations
Drug and Alcohol Education
Health Classes – Sex Education
Mortuary Services – Autopsies of Virtual Cadavers & Embalming Procedures
Veterinarian Training
Before these future curriculums and applications should be considered, the BodyExplorer© system will need updates to certain physical components. The BodyExplorer© system is still in its Beta Version and as such various improvements are being made. Suggestions for updates to the BodyExplorer© system are made based off of observations made during the course of this study. One recommended improvement to the system would be the inclusion of small table that would attach to the projector pole to reduce the need for recalibration due to accidental shifting of external components. Another recommendation would be a simpler infrared light pen than the one currently being used. The number of buttons on the pen confused many of the study participants. A simpler infrared pen with one button would make the system more intuitive to use. As an alternative to the infrared light pen, a complete overhaul of the motion tracking/interaction aspect of the BodyExplorer© is recommended. The Wii remote and light pen combo was shown by the results of the SUS tool to work well, but improvement is still needed. Many participants struggled with the gap that was created between the tip of the infrared pen and the cursor seen in the projection. The gap between the user’s hand and the actual position of the cursor due to calibration inaccuracies was slightly disorienting and made operating BodyExplorer© difficult. This gap problem could be solved by utilizing a different method of motion tracking similar to that of the Kinect motion sensor.
Limitations
The findings should be interpreted in the context of the limitations of the study. One substantial limitation of this study was related to the inability of every participant to spend the same amount of time directly interacting with the BodyExplorer© system. Some participants were able to interact with BodyExplorer© one-on-one with input from the moderator while other participants experienced it as a group. For those students that interacted with the system in a group setting, not all got to interact with BodyExplorer© for the same length of time. Some student groups were larger than others as a result of time constraints and scheduling conflicts with the day to day activities that took place in the same accredited simulation center room that the system was set up in. Also, because the participants in this study have included students who were asked to participate on behalf of a university teacher, the sample may include students who had higher or lower expectations of the BodyExplorer©, or no insights on how it could be utilized as a learning technology for nursing or other students. The audio/video recordings taken of the sessions were also problematic. The audio/video recordings were used to collect user impressions as they were interacting with the system that may not have been expressed through the survey alone. Due to technology issues, some video feeds did not pick up audio as well. The pilot study yielded some interesting ideas about uses for the technology due to the experience and occupations of the participants, but no audio feed exists so it is impossible to transcribe those ideas.
Another limitation of the study was the demographics survey. The demographics survey was amended after the pilot study so that the questions about education level and age range were clearer, however, the survey needed a few other small tweaks due to the fact that the age range was extended upward and not down. This resulted in at least one participant to have to select an age range higher than their actual age. Despite the slight inaccuracy in reported ages, this should not affect the overall conclusions drawn from the study. The sample did not include a breakdown of male versus female responders; therefore, any gender differences in perceptions of the BodyExplorer© system could not be explored. Additionally, this study was only performed at one school and in one state. Due to this fact it is impossible to know if any reactions or feedback obtained about the BodyExplorer© gathered at other schools, or in different states, would be similar to the results of this study.
Moving Forward
Simulation technology has come a long way since it was first introduced. High-fidelity simulation manikins have become more streamlined and are now able to internally house many of the bulky components that made them hard to use and store. Current high-fidelity manikins are also more functional and easily programmable than their early counterparts as they are able to connect to all computers, cameras, and recording systems through WIFI connections. Today’s highest-fidelity manikins cost thousands of dollars even though they are now easier to operate, move, and repair. Despite the cost, it is still possible for many to find a version of the technology that falls within a price range or budget. This is because simulations and simulators now exist in all shapes, sizes, and fidelities. Because of this fact, simulation has actually turned out to be a flexible and durable form of medical education and training (Lateef, 2010). This will only continue to be true as more affordable simulation technologies in the VR/AR realm are developed. Simulation technologies show great potential as teaching and learning tools for aspiring nurses and healthcare workers with it being widely acknowledged that SIM provides opportunities for students to get hands-on practice without the consequences of hurting a living patient. This type of training is valuable because it allows students to make mistakes, reset the scenario, and try again. For students with no prior experience working as a nurse, these first practice sessions without an actual human patient provide a unique opportunity to learn in a safe environment (Damewood, 2016).
Providing nursing/healthcare students with access to better, more user-friendly educational tools will only help to lower their chances of making mistakes in the field due to inexperience or technology induced error. TIE occurs when employees are unfamiliar with how to use/operate hospital technologies and systems. Kushniruk, Borycki, Anderson, & Anderson (2009) explain that when a nurse cannot operate a piece of equipment or confidently navigate a health information technology due to inexperience, lack of training, or poorly designed user interfaces, it may result in medication errors, and ultimately injury or death of that patient. Many existing technologies that nurses and other healthcare workers interact with are poorly designed and hard for someone with little experience to operate, potentially increasing the occurrence of TIE. It is possible that the new forms of healthcare SIM training being developed will be able to better acquaint students with in-hospital systems, procedures, and technologies. While these new methods of training are important, it will only help address the problem if the training technologies are also user friendly, well designed, and intuitive. Through the implementation of usability studies, such as the one performed on BodyExplorer©, it will be possible to create these effective training technologies and to identify and control these new technologies’ use-related risks and hazards, minimize instances of technology induced error, and increase patient safety around the globe.
References
Ahmet Gokyar, & Cengiz Cokluk. (2018). Evaluation of the usability of image projecting augmented reality technique: An experimental study on fresh cadaveric cattle brain. Archives of Clinical and Experimental Surgery, (2), 65. https://doi.org/10.5455/aces.20170825113026
Bangor, A., Kortum, P., Miller, J. "An Empirical Evaluation of the System Usability Scale." International Journal of Human-Computer Interaction 24, no. 6 (2008): 574-594. http://dx.doi.org/10.1080/10447310802205776
Bangor, A., Kortum, P., & Miller, J. (2009). Determining what individual SUS scores mean: adding an adjective rating scale. Journal of Usability Studies, 4(3). 114-123.
Boletsis, C., & McCallum, S. (2016). Augmented Reality Cubes for Cognitive Gaming: Preliminary Usability and Game Experience Testing. International Journal of Serious Games, 3(1), 3. Retrieved from http://search.ebscohost.com/login.aspx?direct=true&db=edb&AN=113857277&site=eds-live&scope=site
Brooke, J. "Sus: A Retrospective." Journal of Usability Studies 8, no. 2 (2013): 29-40.
Brooke, J. "Sus: A Quick and Dirty Usability Scale." In Usability Evaluation in Industry, 189-194: Taylor and Francis, 1996.
Dahl, Y., Alsos, O. A., & Svanæs, D. (2010). Fidelity Considerations for Simulation-Based Usability Assessments of Mobile ICT for Hospitals. International Journal of Human-Computer Interaction, 26(5), 445–476. https://doi.org/10.1080/10447311003719938
Damewood, A. (2016). Current Trends in Higher Education Technology: Simulation. TechTrends: Linking Research & Practice to Improve Learning, 60(3), 268–271. https://doi.org/10.1007/s11528-016-0048-1
Huiqin Yang, Thompson, C., & Bland, M. (2012). The effect of clinical experience, judgment task difficulty and time pressure on nurses’ confidence calibration in a high fidelity clinical simulation. BMC Medical Informatics & Decision Making, 12(1), 113–121. https://doi.org/10.1186/1472-6947-12-113
Kahriman, I., Öztürk, H., Bahcecik, N., Sökmen, S., Kücük, S., Calbayram, N., & Altundag, S. (2018). The effect of theoretical and simulation training on medical errors of nurse students in Karadeniz Technical University, Turkey. Journal of the Pakistan Medical Association, 68(11), 1636-1643. doi:10.19070/2332-2780-1600051
Kardong-Edgren, S., Breitkreuz, K., Werb, M., Foreman, S., & Ellertson, A. (2019). Evaluating the Usability of a Second-Generation Virtual Reality Game for Refreshing Sterile Urinary Catheterization Skills. Nurse Educator, 44(3), 137–141. https://doi.org/10.1097/NNE.0000000000000570
Kiernan, L. C. (2018). Evaluating competence and confidence using simulation technology. Nursing, 48(10), 45–52. https://doi.org/10.1097/01.NURSE.0000545022.36908.f3
Kirakowski, J. "The Use of Questionnaire Methods for Usability Assessment." University College Cork, http://sumi.ucc.ie/sumipapp.html (accessed: 6 May 2019).
Kortum, P., Acemyan, C. "How Low Can You Go?: Is the System Usability Scale Range Restricted?". Journal of Usability Studies 9, no. 1 (2013): 14-24.
Koutkias, V., Niès, J., Jensen, S., Maglaveras, N., Beuscart, R., Kushniruk, A., & Borycki, E. (2011). Exploring the Relationship between Usability and Technology-Induced Error: Unraveling a Complex Interaction. Studies in Health Technology & Informatics, 166, 48. Retrieved from https://reddog.rmu.edu/login?url=http://search.ebscohost.com/login.aspx?direct=true&db=edb&AN=60622546&site=eds-live&scope=site
Kushniruk, A. W., Borycki, E. M., Anderson, J. G., & Anderson, M. M. (2009). Preventing technology-induced errors in healthcare: the role of simulation. Studies In Health Technology And Informatics, 143, 273–276. Retrieved from https://reddog.rmu.edu/login?url=http://search.ebscohost.com/login.aspx?direct=true&db=mnh&AN=19380947&site=eds-live&scope=site
Lateef, F. (2010). Simulation-based learning: Just like the real thing. Journal of Emergencies, Trauma and Shock, 3(4), 348–352.
Martin, M., & Daniel, M. (2016, May 03). Medical error-the third leading cause of death in the US. Retrieved from https://doi.org/10.1136/bmj.i2139
Mykityshyn, A. L., Fisk, A. D., & Rogers, W. A. (2002). Learning to use a home medical device: mediating age-related differences with training. Human Factors, 44(3), 354. Retrieved from https://reddog.rmu.edu/login?url=http://search.ebscohost.com/login.aspx?direct=true&db=edb&AN=8865195&site=eds-live&scope=site
Sauro, J. A Practical Guide to the System Usability Scale: Background, Benchmarks & Best Practices. Create Space Independent Publishing Platform, 2011
System Usability Scale (SUS). (2013, September 06). Retrieved from https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html
Tullis, T., Albert, W. Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics. Morgan Kaufmann Publishers Inc., 2008.
Verkuyl, M., Atack, L., Mastrilli, P., & Romaniuk, D. (2016). Virtual gaming to develop students’ pediatric nursing skills: A usability test. Nurse Education Today, 46, 81–85. https://doi.org/10.1016/j.nedt.2016.08.024
Verkuyl, M., Romaniuk, D., & Mastrilli, P. (2018). Virtual gaming simulation of a mental health assessment: A usability study. Nurse Education in Practice, 31, 83–87. https://doi.org/10.1016/j.nepr.2018.05.007
FIGVirgínia C. Cavalcanti, Maria I. de Santana, Alana E. F. Da Gama, & Walter F. M. Correia. (2018). Usability Assessments for Augmented Reality Motor Rehabilitation Solutions: A Systematic Review. International Journal of Computer Games Technology. https://doi.org/10.1155/2018/5387896