Meaningful Use of Meaningless data - Lesson from the Teachers in Atlanta.

People are shocked by the story that teachers who are supposed to be setting good examples for school kids were involved in cheating.  The teachers were erasing incorrect answers on students standardized tests and entering the correct answers.

From CBS evening news:

 " It was to show phony progress at often troubled schools, what the report calls "the pressure to meet targets in the data-driven environment." Educator Diane Ravitch, author of "The Death and Life of the Great American School System: How Testing and Choice Are Undermining Education," blames it on a federal law that links funding with test performance.

"We have a terrible federal law called No Child Left Behind that says that all schools have to have 100 percent of their students proficient in reading in math by the year 2014 or their schools will be shut down," Ravitch said.

From CNN

"I think the overall conclusion was that testing and results and targets being reached became more important than actual learning for children," (Gov. Nathan) Deal said. 

So is anyone surprised that this would happen?  When a single number becomes more important that the process, in high stakes situations, people will do what it takes to improve that target number rather than improve the process that the number is supposed to measure.  



Why is this an important lesson for Medicine?

1.  We have for long struggled with the USMLE scores as being a single number that is used to measure the quality of a medical student.  This is what is used as one of the most important criterion for selection to residency training programs.  This is in spite of there being no good data that scoring high on the USMLE equates to potential to be a good doctor.  Recently there was a an article in the leading journal on medical education highlighting this problem.  The USMLE has 3 steps.  Students who score well on Step 1 will postpone taking step 2 till they have finished interviewing for residency programs - just in case they get a lower score on step 2 and jeopardize their chances!  Many medical schools give students dedicated time to prepare for the USMLE.  So students who have been getting a curriculum that medical educators believe is appropriate to make them good doctors for 2 years take a couple weeks off to read for a test!  USMLE is a test of medical knowledge and it also tests whether a student can apply this to the clinical environment.  But it cannot measure a lot of other important competencies like communication skills, professionalism, ability to be lifelong learners etc.



2.  The recent mandate for Meaningful Use (HITECH Act) of EHR data has the risk of falling into the same trap.  While well-intentioned, it has several measures of very important clinical processes.  If done right, these processes have the potential of improving healthcare.  The problem is that these measures are linked to incentive payments from the Center for Medicare and Medicaid Services which make them high stake for doctors working on razor thin margins.  Doctors who fail to use EHRs meaningfully with their Medicare patients are at risk of penalties. Let us take an example:

One of the 25 measures of meaningful use is to show that doctors are doing reconciliation of medications (checking what patients are taking and where there are discrepancies, updating the medical record or educating the patient appropriately).  One popular EHR system has a system for putting a check mark against each medication as you confirm it or to delete it if the patient is not taking it.  Doctors and nurses go over this exercise with the patients and it can be extremely time consuming.  One would assume that after you have gone through the entire list of medications, you should have complied with the meaningful use requirement.  Unfortunately, after doing all this, to record in the database that you reviewed the medications, you have to click a button that marks that you reviewed the medications.  As doctors get pressured to see more patients in less time, guess what is going to happen?  Yes, they may be tempted to click that button whether or not they went through the process of reconciling the medications.  This is an example of how poorly designed software will lead to physicians targeting a number (% of office visits where reconciliation occurred) rather than do what is a critical part of a visit - making sure the patient is taking the right medication.  

The teachers in Atlanta has taught us a valuable lesson.  Even people who are expected to set examples of high moral values, when forced to meet requirements that are measured by numbers without actually getting the support they need to do their job right, will take shortcuts to make the numbers look better.  In today's data-driven society, this is a valuable lesson.