Feeling Good With Four Stars On Yelp

The other day, a friend asked if I knew anything about Aqua, a restaurant in San Francisco.

My response: “Don’t know it, but I’ll bet it gets 4 stars on Yelp.”

Sure enough, it does. How did I guess that? Because nearly everything I’ve searched for gets four stars on Yelp.

Of course, individual reviews vary. But I’ve always been curious about why listings in many star systems end up with nearly the same average rating over time. With Yelp, I became curious as to why these averages are so high. Looking first at the individual reviews, I saw some psychosocial reasons for this. The site encourages positivity by allowing you to tag other reviews as only Useful, Funny, or Cool. (More than once I’ve been tempted to write a review just to call someone else’s Really Dumb.) Reviewers sometimes compensate for a lackluster review with a higher number of stars. Some examples:

- “I’d easily give this place a 3 star [sic], but it gains one star for being the only place to get Sushi in Lincoln Square.” Four stars.

-”Not sure how the hostess sleeps at night with that gigantic stick lodged up her ass.” Four stars.

-”[T]he “scone” was so dry you could sand paint off the walls.” Four stars.

-”I have to say that the drinks I ordered were BAD. My Margarita was so sour and bitter that I had to return it. The vodka tonics must have been made with grade Z tonic water as it tasted like dirty soda water. I won’t even get into the dirty martini.” Four stars.

(Forgive me for taking these clips out of context, but it’s more fun that way.)

The prevalence of positive reviews might also be due to the site’s social networking element, which displays your (ostensibly) real name next to your reviews. Some of these people also get together in person. Do you really want to be the jerk who got all negative over an overcooked burger at the struggling mom-and-pop?

I initially surmised that many in the Yelp community had had those empowered childhoods where criticism was considered demoralizing.* But as I dug further into the reviews, I became impressed by their thoughtfulness. Which suggests another bias: People on Yelp — and elsewhere — tend to review places they like. Farhad Manjoo provides some supporting evidence so I don’t have to.

And yet, individual reviews are not the only cause of high average ratings: Yelp has built the bias into its search engine. At the category search level (e.g. sushi, bars, or salons), “best match” is a weak concept. Lots of results will be highly relevant to a search for sushi. So, what’s the secondary sorting logic? A combination of most reviewed, highest rated, and other special sauce criteria alluded to by a Yelp exec I once spoke with.

When you privilege the most reviewed, highest rated businesses, what happens? Logic indicates that the more reviews there are, the more likely things will average out… and in a community that evaluates matters of high subjectivity with a skew towards positivity, four stars is where the average will land. (Interestingly, All Songs Considered’s now-defunct Open Mic area had anonymous ratings. The song ratings all migrated to a similar average, but they landed more in the middle of the scale.)

In addition, the most reviewed will become more reviewed because they appear more often in the top search results, while the less reviewed will continue to lag. Weirdly, the result of this power law distribution is that Yelp falls behind the coolhunters. If Acme Grill had a moment in the spotlight 6 months ago and got tons of reviews, even when its popularity dies down, it will appear higher in Yelp search results than a newer, hipper thing. People will be more inclined to review it, and the situation is perpetuated.

I’ve long relied on word of mouth and online reviews to make purchase and entertainment decisions. When review communities first reached critical mass on Amazon, they paralyzed me. I treated any bad review, even when among other good ones, as a veto. By now, however, many of us have learned how to extract what’s useful. I’ve also come to understand that reviews are not just useful for consumers, but fun (and cathartic) for the reviewers to write. That said, Yelp does have a lot of influence, for better and worse. So it’s important to remember that a Yelp star is no Michelin (that’s not entirely a bad thing). And that all stars should be taken with a grain of salt.

—————————————————-

*There are likely other variances as well. Geographic, for example: People in the Washington, DC, area seem to me more faux polite than those in New England.