M*Modal is a medical speech understanding software company. Doctors dictate their notes into a microphone and the software extracts key structured medical data.
We wanted to understand how doctors enter patient data so that we could design a notification system for them.
- Help doctors capture more complete and accurate patient information
- Help doctors capture information needed for billing purposes (not always the same as information needed for care)
Field research: 8 hospitals, 3 private practices, 18 total participants across a range of specialties
Competitive analysis and literature review: 20 companies, 18 academic studies
- Doctors experience alert fatigue, which may make them miss a truly important alert
- Opportunity to help doctors monitor and interact with their EMR (electronic medical records) without taking away their attention
- Opportunity to increase review and self-editing by doctors
- Either use speech understanding software or use transcription services, both need review by the doctor
- Opportunity to eliminate the need for doctors to enter irrelevant information just for billing reasons
We brainstormed many ideas, then plotted them on a quad to evaluate them. On one axis is how disruptive or annoying an idea is, and on the other is how well an idea gets a doctor’s attention. This helped us see the tradeoffs between our ideas and narrow them down.
Usability testing, round 1
We went to 2 hospital cafeterias and approached medical professionals for feedback. Each person was asked to rate 12 ideas on a scale from “annoying” to “helpful”.
Usability testing, round 2
- Context is essential
- How to address a notification needs to be very clear
- Pop up notifications were totally ineffective
HTML prototype, analogous domain
We were confident in what we knew so far, bu we wanted to test our ideas with more people. Because doctors are very hard to recruit, we used an analogous system to test our second round of prototypes. We created 2 systems to handle complicated coffee orders and observed people. Each person was given a few scenarios of complicated coffee orders to put into the systems, and each scenario was missing key information. The system prompted each person to find the mission information.
Usability testing, round 3
- Important to make clear how you can interact with the system
- People get frustrated if they feel like they are doing extra work without understanding why
- Important that people’s mental model of the system is accurate
Adobe Flex prototype
Usability testing, round 4
- There was minor confusion about the system at first, but after the very short learning curve, people completely understand how to use it.
- Helping doctors understand the purpose of a notification increased their motivation to respond to them.
- It’s helpful and convenient to have multiple ways to respond to notifications.
- Follow up testing needed due to the system’s purpose as long-term use.
Measurement of success
- Doctors’ time spent on health records (creating, editing notes), before and after
- Billing departments’ time spent chasing doctors down for clarification, before and after
- Number of errors found and clarifications needed by hospital administration, before and after
- Number of notifications closed per doctor over time
- Time between notification popup and closure
- Doctors’ satisfaction with the system
- Hospital administrations’ satisfaction with the system
- Quality of patients’ care
- Filter notifications, especially if records are very long and many notifications are generated
- Billing level indicator
- Reduce the number of dialogs
- Education portion: teach doctors about documentation requirements for billing