How Good Are Doctor Rating Sites?

November 10, 2008 · Filed Under e-pts resources, general, news & gossip, reforming hc · 16 Comments

Ruth Given has written a paper entitled, MD Rating Websites: Current State of the Space and Future Prospects (PDF), that was recently published on THCB. It’s a 39-page informal analysis (with an emphasis placed on informal) that takes a fairly good and comprehensive look at the space of doctor rating sites as they exist today.

I think that the main issue Given hit upon in the report (but I’m not sure she recognizes as the primary challenge of doctor rating sites) is the numbers issue. With over 700,000 physicians in the U.S., a ratings database of 10,000 or even 20,000 is pitifully and woefully small.

And not only is such a number small, it is statistically troublesome, likely painting an inaccurate picture of providers listed. If most sites only have one or two ratings for any given physician (and such data is statistically useless, if the average family physician is covering between 2,000 to 2,500 patients), then the sites themselves provide little added value outside of the directory listing. Which, as Given notes, is inaccurate or nonexistent more often than not.

Tags: doctors, md, physician, rating, ratings, review, reviewing, reviews

CNN Takes on Doctor Ratings… And Gets it Wrong

April 17, 2008 · Filed Under trends & principles · 5 Comments

CNN has recently published an article about what to look for in a doctor rating website. Unfortunately, they repeat some misconceptions and errors about these services.

The most serious error is the claim that the greater volume a website has of doctor ratings, the more reliable or statistically valid it will become.

Tags: doctors, health 2.0, ratings

Patients Rating Hospitals? What Next!?

April 15, 2008 · Filed Under e-pts resources, policy issues, reforming hc, trends & principles · 2 Comments

Dr. Robert Wachter has an interesting essay over at THCB entitled, Should Patient Satisfaction Scores Be Adjusted for Where Patients Shop? As health care in the U.S. continues to move in the direction of tailoring itself to patient satisfaction, the question becomes — how do we make such ratings more reliable and fair? The answer is, “Not easily.”

Patient satisfaction in a hospital is a many-varied thing. You could have the rudest doctor in the world (ala TV’s “Dr. House”), but if he ends up saving your life, how badly could you rate the rest of the hospital? You could end up dying of a preventable infection from the hospital, and yet, who’s going to capture your unhappiness once you’re in the grave? You could have the nicest, most kindest surgeon in the world, but if he leaves a sponge in you and has to re-open you back up to retrieve it (but tells you it’s just “a routine check to ensure everything is healing properly”), is your hospital rating really going to help others understand that hospital may not be the best one to visit?

Wachter’s point is a valid one — that we need to adjust hospital ratings, and even ratings within the same hospital, based upon the patient’s experience. Outpatient is usually a very different experience than inpatient. Staying in a psychiatric bed is usually nothing like staying in the ICU. Dealing with the madness of the E.R. is nothing like going in overnight for a routine colonoscopy. A hospital in a poor urban area is generally going to be more poorly rated than one in a rich suburb.

Researchers have long recognized the importance of rating “apples to apples.” If you try and compare the efficacy of an antidepressant, for instance, with say, an asthma inhaler, I’m sure you’ll get two very different results. But they shouldn’t have been compared in the first place, because they have virtually nothing to do with one another.

The same is true as we experiment with new ways to provide consumers with more information about the hospitals in their community. These rating systems should be carefully and scientifically devised, normed, validated, and then used only for “apples to apples” comparisons.

And the same can be emphatically said, too, for online ratings of virtually anything. Almost no online ratings’ systems have been normed, empirically validated, or have any sort of random selection occurring. This means that virtually every rating system you come across online — whether it’s for medications, or doctors, or, well, even a TV — is not really a scientific measure on which you should be basing your decision. The population of people submitting these ratings are not a random sample, and so what you see and read online should be taken with a grain of salt.

Eventually, all of these things will be sorted out and we’ll find some happy medium. Until then, we’re left with a lot of pseudo-science and questionable data.

Tags: doctors, hospitals, rating systems, ratings

What Would Your Ideal Doctor Rating System Look Like?

December 4, 2007 · Filed Under positive patterns · 1 Comment

Dr. Bob Wachter over at the Health 2.0 blog has an interesting entry about doctor rating sites, which are popping up like dandelions in summer.

He reviews a lot of the current efforts underway, including Zagat’s recent announcement they would get into the doctor rating business. But perhaps the most interesting part of his article is where he describes his own ideal rating system for physicians:

The stakes are immense, and a balanced approach is more likely to bear fruit than any single peephole. Ultimately, if I’m choosing a doc for me or a loved one, I’d like to know it all: bedside manner (4 stars from Zagat), structural measures (is the doctor’s office computerized?), process measures (are diabetics getting statins appropriately?), surrogate outcomes (what’s the average hemoglobin A1c?), and hard outcomes (what are the risk-adjusted mortality or hospitalization rates?). And then I’d like the appropriate specialty board (ABIM, American Board of Surgery, etc.) to tell me whether the physician is meaningfully engaged in quality improvement activities, and how well he or she did on the certifying exam – the best measure we have of knowledge and clinical judgment. Yes, you heard me right: I’d like the Board to tell me whether the doc was in 5th percentile on the certifying exam or the 87th. It doesn’t pass the smell test to say that we consider both these board certified docs to be undifferentiate-able. In this new era of transparency, if we physicians would want that information before choosing a doc for ourselves (and I sure would), then I believe that patients should have access to it as well.

What do you think of this suggested system? How would your ideal rating system differ?

Tags: doctor, doctors, ratings

A project of the