Meaningful Use of Meaningless data - Lesson from the Teachers in Atlanta.

People are shocked by the story that teachers who are supposed to be setting good examples for school kids were involved in cheating.  The teachers were erasing incorrect answers on students standardized tests and entering the correct answers.

From CBS evening news:

 " It was to show phony progress at often troubled schools, what the report calls "the pressure to meet targets in the data-driven environment." Educator Diane Ravitch, author of "The Death and Life of the Great American School System: How Testing and Choice Are Undermining Education," blames it on a federal law that links funding with test performance.

"We have a terrible federal law called No Child Left Behind that says that all schools have to have 100 percent of their students proficient in reading in math by the year 2014 or their schools will be shut down," Ravitch said.

From CNN

"I think the overall conclusion was that testing and results and targets being reached became more important than actual learning for children," (Gov. Nathan) Deal said. 

So is anyone surprised that this would happen?  When a single number becomes more important that the process, in high stakes situations, people will do what it takes to improve that target number rather than improve the process that the number is supposed to measure.  



Why is this an important lesson for Medicine?

1.  We have for long struggled with the USMLE scores as being a single number that is used to measure the quality of a medical student.  This is what is used as one of the most important criterion for selection to residency training programs.  This is in spite of there being no good data that scoring high on the USMLE equates to potential to be a good doctor.  Recently there was a an article in the leading journal on medical education highlighting this problem.  The USMLE has 3 steps.  Students who score well on Step 1 will postpone taking step 2 till they have finished interviewing for residency programs - just in case they get a lower score on step 2 and jeopardize their chances!  Many medical schools give students dedicated time to prepare for the USMLE.  So students who have been getting a curriculum that medical educators believe is appropriate to make them good doctors for 2 years take a couple weeks off to read for a test!  USMLE is a test of medical knowledge and it also tests whether a student can apply this to the clinical environment.  But it cannot measure a lot of other important competencies like communication skills, professionalism, ability to be lifelong learners etc.



2.  The recent mandate for Meaningful Use (HITECH Act) of EHR data has the risk of falling into the same trap.  While well-intentioned, it has several measures of very important clinical processes.  If done right, these processes have the potential of improving healthcare.  The problem is that these measures are linked to incentive payments from the Center for Medicare and Medicaid Services which make them high stake for doctors working on razor thin margins.  Doctors who fail to use EHRs meaningfully with their Medicare patients are at risk of penalties. Let us take an example:

One of the 25 measures of meaningful use is to show that doctors are doing reconciliation of medications (checking what patients are taking and where there are discrepancies, updating the medical record or educating the patient appropriately).  One popular EHR system has a system for putting a check mark against each medication as you confirm it or to delete it if the patient is not taking it.  Doctors and nurses go over this exercise with the patients and it can be extremely time consuming.  One would assume that after you have gone through the entire list of medications, you should have complied with the meaningful use requirement.  Unfortunately, after doing all this, to record in the database that you reviewed the medications, you have to click a button that marks that you reviewed the medications.  As doctors get pressured to see more patients in less time, guess what is going to happen?  Yes, they may be tempted to click that button whether or not they went through the process of reconciling the medications.  This is an example of how poorly designed software will lead to physicians targeting a number (% of office visits where reconciliation occurred) rather than do what is a critical part of a visit - making sure the patient is taking the right medication.  

The teachers in Atlanta has taught us a valuable lesson.  Even people who are expected to set examples of high moral values, when forced to meet requirements that are measured by numbers without actually getting the support they need to do their job right, will take shortcuts to make the numbers look better.  In today's data-driven society, this is a valuable lesson.

Making EHRs More Meaningful for Physicians - Part II

Peer rating of EHR notes!



One of the biggest problems with the profusion of meaningless data in EHRs is separating the wheat from the chaff, finding the proverbial needle in a haystack.

Something that the Web 2.0 has taught us is that the users who create data can also help to make it more meaningful.  Thus we have tags in Flickr and "likes" in Facebook.  How can we apply that to EHRs?

Every physician who has seen a patient for follow up after a long hospital admission knows what a relief it is to see a meaningful note that succinctly summarizes the hospital course.  For every useful note providing meaningful information there are tens of notes with meaningless words and phrases that don't add to patient care in any substantial way.

What if we have a thumbs up and thumbs down option next to every note?  What if anyone reading the note could give it an anonymous rating?  What if we let physicians see statistics of how many people had opened their note and how many of those rated it up or down?

Then users could filter the notes in the EHR by their ratings.  Also we could recognize the excellent work done by our infectious disease docs who document the best notes among all clinicians.  Maybe we could link this with a bonus just like we do for the meaningful use of EHRs!