I think that the main issue Given hit upon in the report (but I’m not sure she recognizes as the primary challenge of doctor rating sites) is the numbers issue. With over 700,000 physicians in the U.S., a ratings database of 10,000 or even 20,000 is pitifully and woefully small.

And not only is such a number small, it is statistically troublesome, likely painting an inaccurate picture of providers listed. If most sites only have one or two ratings for any given physician (and such data is statistically useless, if the average family physician is covering between 2,000 to 2,500 patients), then the sites themselves provide little added value outside of the directory listing. Which, as Given notes, is inaccurate or nonexistent more often than not.

For a number to have meaning in research, one needs to look at where the number comes from. In terms of rating sites (any rating site), ratings are more likely to be valid and reliable if:

• They come from a significant number of people compared to the number of people using the service or product
• They come from a randomized set of people who have used the product or service
• They represent both positive and negative opinions
• They objectively assess certain characteristics of the product or service to arrive at an objective rating number

You’ll notice most ratings sites don’t have much concept of statistics or statistical significance and virtually every physician rating site fails this simple set of minimal requirements for a rating site to have even minimal face validity.

An example of a rating site that meets most (but still not all) of the above requirements is Amazon.com. Because of the sheer number of consumers who use the site to buy stuff from, virtually every popular product has a significant number of reviews. Because it attracts such large numbers, it is likely that it is a fairly randomized sample (as randomized as one can hope for online). And virtually all popular products carry both positive and negative reviews. The one thing Amazon.com doesn’t do well is trying to rate products objectively, instead relying mainly upon a well-written narrative by the reviewer.

So it doesn’t really matter what a physician rating site’s business model looks like in terms of whether it will succeed or fail. Most user-generated content sites online today have an unrealistic, non-existent or unachievable business model that goes something like this (with apologies to South Park’s underwear gnomes):

1. Gather together a massive amount of users and get them to generate content for our for-profit, commercial site
2. ???
3. Profit!

It matters only if under Step 1, these physician ratings sites can generate enough ratings to make their data valuable and worthwhile to most users who visit their site. Because if they only satisfy 1 or 2 percent of their users (or present users with inaccurate, invalid data based 2 or 3 user ratings), people will simply not revisit the site in the future. And more importantly, they’ll fail to recommend the site to others (where most successful social networking sites succeed).

There are other validity problems with the data generated by these sites, which I’ve covered previously last year. None of these problems have been solved, and the fact remains that rating a person on their entire livelihood based on your usually brief interaction with them once is rife with concerns.

Given’s report is a valuable addition to the e-Health landscape, but it also has a few flaws. I think that with a good editor, the report could’ve been half as long and twice as good. And frustratingly, sometimes I found the author seemingly contradicting herself. For instance, on page 14 she writes,

Search ads are indeed cheap. The MD rating websites using them to promote themselves appreciate this. But their low cost means these ads are not particularly lucrative for the sites displaying them.

And then in the next paragraph, she notes the premium that pharmaceutical advertising garners — the main source of revenue for virtually any consumer-focused medical or health website:

One source I came across indicated that pharmaceutical display ads typically pay between $50 and $100 per 1,000 page views, while other advertisers pay at most $15. Unfortunately, we haven’t seen many drug ads on these sites thus far.

Which is fine as far as an observation goes while randomly looking at a particular site a few times over a 2 or 3 month period. But it’s not really contributing to the understanding of the underlying business model in this space unless you’ve done an actual, objective analysis (or simply asked them in the survey conducted, which wasn’t done).

And then some opinions are just in contradiction to other professional forecasts and data:

And the opportunity for profiting from generous pharmaceutical industry spending may decline significantly in the future as more brand name drugs go off patent, since brand competition is what drives advertising outlays.

Yet eMarketer, which tracks trends such as online pharmaceutical and health spending, actually reports that 2008 ad spending will again rise by approx. 20% from 2007 (when it also rose about 19%), and is expected to grow another 20 to 25% in 2009. This objective data doesn’t jive with such opinion.

Overall, however, I found the report a must-read for anyone who wants to understand how these kinds of sites are doing and whether they provide any real value to e-Patients and other consumers. I have my doubts as they exist today, because they still lack the volumes necessary to make their data of value (and valid).

The most serious error is the claim that the greater volume a website has of doctor ratings, the more reliable or statistically valid it will become.

It’s a matter of statistics: The more reviews you read, the more likely you are to get an accurate assessment. “I would check a lot of different Web sites,” says Carol Cronin, executive director of the Informed Patient Institute. “Look across them, not just within one.”

Speaking of volume, a common concern about doctor rating sites is that one angry patient can make multiple nasty comments, using a different name each time (or, conversely, that the physician herself could go on and make multiple glowing comments).

But Martin Schneider, chairman of the Informed Patient Institute, says these sites have ways of detecting when one person is making several comments under different names. Back in the 1990s, Schneider was president of a now-defunct doctor rating site called thehealthpages.com. “Even back then, we had to the technology to stop that from happening,” he says.

These claims are commonly made, but they are largely incorrect. Here’s why…

In survey research (which is basically what a doctor rating site is trying to be), you need a sample that is both large and randomized. That is, you do not go out and post an announcement saying, “Take our survey if you think you have depression” if you’re looking for an unbiased data sample on depression in the general population. You need to have a group of people that both have and don’t have depression in order to obtain generalizable results.

The same is true with ratings sites. They may get the volumes needed, but none of these sites have any way of addressing the biased sample problem. People who rate their doctors are likely to fall into one of two categories — they either had a horrible experience with them and want others to know, or they had a wonderful experience with them and want others to know. But most people who fall in between these two extremes and have run-of-the-mill experiences with the doctor will likely never rate, because they have little incentive to do so.

You will also need a humongous number of patients rating each doctor — at least 20 to 30% of their entire patient list — in order to for the ratings to start gaining enough power to be reliable and valid (notwithstanding the population sample bias issue).

CNN admits as much later on in the same article quoting Dr. Robert Wachter:

While patient reviews might be useful, they have several clear drawbacks, our experts say. First, many doctors have just a few reviews or none at all. Second, even if a doctor has 20, 30, 50 or 100 reviews, that’s still only a small fraction of his entire patient population — and a warped fraction at that.

“The person most likely to write is the one who’s most enthralled with the doctor, or the one who’s most pissed,” Wachter says. “You’re getting a skewed view.”

The other advice — decide what’s important to you, look for patterns in the ratings, look for specifics in people’s ratings of their doctor and put more weight onto detailed reviews rather than general comments, and consult objective data already available — is generally solid, but still doesn’t address the foundational statistical problems with these types of online ratings systems. All the business people gloss over these problems, but if a rating isn’t scientific, its value is diminished substantially.

And honestly, Martin Schneider is a bit naive if he thinks it isn’t a simple thing to rate one doctor multiple times on all of these sites. Simply by clearing one’s cookies, using a few webmail addresses and using a Web proxy, you can register as many accounts as you would like on any of these services in a matter of minutes.

Patient satisfaction in a hospital is a many-varied thing. You could have the rudest doctor in the world (ala TV’s “Dr. House”), but if he ends up saving your life, how badly could you rate the rest of the hospital? You could end up dying of a preventable infection from the hospital, and yet, who’s going to capture your unhappiness once you’re in the grave? You could have the nicest, most kindest surgeon in the world, but if he leaves a sponge in you and has to re-open you back up to retrieve it (but tells you it’s just “a routine check to ensure everything is healing properly”), is your hospital rating really going to help others understand that hospital may not be the best one to visit?

Wachter’s point is a valid one — that we need to adjust hospital ratings, and even ratings within the same hospital, based upon the patient’s experience. Outpatient is usually a very different experience than inpatient. Staying in a psychiatric bed is usually nothing like staying in the ICU. Dealing with the madness of the E.R. is nothing like going in overnight for a routine colonoscopy. A hospital in a poor urban area is generally going to be more poorly rated than one in a rich suburb.

Researchers have long recognized the importance of rating “apples to apples.” If you try and compare the efficacy of an antidepressant, for instance, with say, an asthma inhaler, I’m sure you’ll get two very different results. But they shouldn’t have been compared in the first place, because they have virtually nothing to do with one another.

The same is true as we experiment with new ways to provide consumers with more information about the hospitals in their community. These rating systems should be carefully and scientifically devised, normed, validated, and then used only for “apples to apples” comparisons.

And the same can be emphatically said, too, for online ratings of virtually anything. Almost no online ratings’ systems have been normed, empirically validated, or have any sort of random selection occurring. This means that virtually every rating system you come across online — whether it’s for medications, or doctors, or, well, even a TV — is not really a scientific measure on which you should be basing your decision. The population of people submitting these ratings are not a random sample, and so what you see and read online should be taken with a grain of salt.

Eventually, all of these things will be sorted out and we’ll find some happy medium. Until then, we’re left with a lot of pseudo-science and questionable data.

He reviews a lot of the current efforts underway, including Zagat’s recent announcement they would get into the doctor rating business. But perhaps the most interesting part of his article is where he describes his own ideal rating system for physicians:

The stakes are immense, and a balanced approach is more likely to bear fruit than any single peephole. Ultimately, if I’m choosing a doc for me or a loved one, I’d like to know it all: bedside manner (4 stars from Zagat), structural measures (is the doctor’s office computerized?), process measures (are diabetics getting statins appropriately?), surrogate outcomes (what’s the average hemoglobin A1c?), and hard outcomes (what are the risk-adjusted mortality or hospitalization rates?). And then I’d like the appropriate specialty board (ABIM, American Board of Surgery, etc.) to tell me whether the physician is meaningfully engaged in quality improvement activities, and how well he or she did on the certifying exam – the best measure we have of knowledge and clinical judgment. Yes, you heard me right: I’d like the Board to tell me whether the doc was in 5th percentile on the certifying exam or the 87th. It doesn’t pass the smell test to say that we consider both these board certified docs to be undifferentiate-able. In this new era of transparency, if we physicians would want that information before choosing a doc for ourselves (and I sure would), then I believe that patients should have access to it as well.

What do you think of this suggested system? How would your ideal rating system differ?