Monday, November 05, 2007

Movie critics: Not what they say, but how many of them say something

To clear up any potential confusion, in playing around with movie releases from 2005, unless otherwise noted I've been using total box office receipts rather than absolute profitability in judging a movie's success, as that is what the media care about. It is the metric used in determining a film's popularity.

Although an adjustment for budget is a necessity from the perspective of the studio producing the movie, it is effectively meaningless to the viewing public. My aim is to make a little sense out of how to determine which movies will be likable before dropping ten bucks and giving up three hours in going to see them, not how to optimize the amount Miramax spends on a movie to maximize its profitability.

It doesn't matter to me, as a potential moviegoer, what a movie's production costs are. When I go to see it in theater, my expenses are nearly the same irrespective of whether I'm going to see The Ring or King Kong. To the extent that there is a difference in cost, it is usually slightly cheaper for me to see more expensive movies, as they tend to play in more theatres and with greater frequency, thus cutting down on schedule disruptions and transportation costs.

That said, absolute profit and box office performance are fungible when it comes to average viewer scores, both from professional critics (from Rotten Tomatoes, whose critic base consistently outperform the reviewers used by other sites like MetaCritic) and Yahoo users' ('mavens') movie reviews. For the critics, profit and total receipts correlate with ratings at .32 and .30, respectively. For mavens, it is .41 in both cases. They are virtually the same.

Those modest relationships are not satisfying. While critical review correlates well in a couple of specific genres like action and science fiction, on the whole it does a poor job of 'explaining' a movie's box office success, accounting for only 9% of the variation in performance. The mavens do a bit better, about doubling that explanatory power. But previously I'd wondered if perhaps it wasn't what reviewers had to say, but just how many decided to take the time to say something, that mattered:

Movies that illicit a response in the reviewer, be he professional or otherwise, are the ones he'll most likely rate and write about. More likely than not, even if he isn't particularly impressed, he probably tends to get more from the movie than the next guy who decided not to write about the film. If this is the case, movies rated by the greatest number of people should be the top performers, whereas the ones receiving sporadic comment languish in the face of uninterest.
So how do the number of reviews a movie receives fare against the amount of money it brings in? Quite well, actually.

For critics, the relationship with total receipts is a solid .54, consistently stronger on the whole and across budget levels than the correlation (.31) with actual average critical rating is. Parenthetically, Rotten Tomatoes has an established, static base of "Approved Tomatometer Critics" who may or may not choose to submit a review (either on their personal websites or directly to RT) for a specific movie.

If the entire lot has a professional opinion to give, it's because there is something relevant in the movie, be it positive, negative, or somewhere in between, that speaks to them and in turn compels them to speak to the consuming public. That's not the case when only a handful of reviewers have something to say.

Those who choose to remark on the less remarked upon film usually are doing so for some particular reason--an ideological motivation, a like or dislike for a director or starring actor, etc, that does not translate well to a larger audience. Herein lies the value of specific reviewers, of course. If you find one who exemplifies your own thoughts and tastes in an edifying way, his reviews often correlate well with a movie's worth from a personal perspective. But for the most part, your neighbors won't get a whole lot of use out those reviews.

Though it might cause umbrage, the message to professional critics is this: You're not individually unique, but merely a quantitative cog. We don't need to know the specifics of the clever profundities you've discovered, only whether you've decided to blather on this particular movie or have instead elected to withhold your words for another film.

For mavens, the relationship is nearly perfect. The correlation is .95, meaning the number of Yahoo users who weigh in on a movie explains 90% of how well that movie ends up doing at the box office.

At first it may seem intuitively obvious that more reviewers means more popularity, but even in this most democratic of reviewer forums with tens of thousands of people grading movies, we're looking at the opinions of less than one-half of one percent of the people who saw the thing. More importantly, they may just as easily give a scathing review of a movie as they might sing its paens.

That a movie induces a response shows that it it, quite literally, remarkable. Though the taste it leaves in the mouth may be bitter, it is not so bland as to be immediately forgotten. Thus, it is worth watching. And so people go to see it.

The numbers via Swivel are here.

Digg!


3 comments:

Steve Sailer said...

A lot of professional reviewers, especially daily newspaper reviewers, pretty much have to review every movie that comes to town (or have someone else on the staff do it). It's the bane of their existence. It deforms the reviewing profession since the people who prosper are the ones who can churn out zingy one-liners about any piece of dreck they get assigned to see.

Steve Sailer said...

What might be interesting is to look at movies where the Rotten Tomatoes consensus is the opposite of the box office numbers. And you might restrict it to just pictures above a certain budget, such as $20 million (you can get budgets from IMDB or from BoxOfficeMojo).

It would be interesting to make up a list of critical favorites that had enough star power going for them that they could have been hits that instead tanked at the box office, like Angelina Jolie's "A Mighty Heart" or (to a lesser extent) George Clooney's "Michael Clayton."

By the way, IMDB ratings come with some demographic breakdowns if you click through, although they aren't that interesting.

Audacious Epigone said...

Steve,

I wasn't aware that many reviewers had to tackle everything. The number of reviews counted varies pretty widely though, from 43 at the low end to 247 at the high end, in 2005. And Rotten Tomatoes uses a consistent reviewer base. There is variance even for the most widely released movies. King Kong was looked at by 166 critics, while Batman Begins had 245.

Thanks for pointing out the demographic breakdown on IMBD. As you say, though, it's pretty mundane, by age range and gender only.