Thursday, August 02, 2007

Movie critics vs movie mavens

++Addition++Steve Sailer weighs in.


As a follow-up to a post on movie box office performance, I grouped films from 2005 by their respective genres to see if criticism became more useful this way. Or at least I tried to group them by genre. Nearly every movie is billed as belonging to multiple genres. How do you classify The Chronicles of Narnia? As adventure, drama, science fiction, for children, film adaptation, or something else? How irritating!

I shouldn't complain, though. In trying not to neglect a reader's possible interest, I'm pretty liberal with how I label each post.

Arbitrariness is unavoidable in categorizing exclusively. I had to give my best effort based on description, word of mouth, and the trailer/cover presentation. If a movie was geared toward children, it was assigned to the "kids" category. The greatest challenge is in choosing romance or comedy, as the former rarely comes without the latter. Anything that struck me as something I'd never see unless coerced by my better half is classified as romance.

In a previous post delving into movie revenues and 'expert' criticism, I found almost no relationship between the two. A far better predictor is found by simply looking at how much was spent on a movie. Steve Sailer, who plays a movie critic when he's not revealing a surefire strategy for Republican political dominance or explaining the most significant obstacles to functioning liberalism in the Middle East, wrote:
There is a fair amount of agreement between critics and the public on movies within various classes: e.g., that Saving Private Ryan was better than Flags of Our Fathers, or that Gladiator was better than Alexander. Movies tend to "work" or not work, rather like a good band within a musical genre is pretty clearly better than a bad band.

Steve's comment is borne out for the most part--critics do as well as better by genre, except for horror (which they despise) and romance (which, not surprisingly, they just don't seem to get). Criticism and cynicism are often mistakenly thought to be synonyms, and this provides some justification for that confusion, as these genres are the two most susceptible to it. In action and science fiction, they do very well (.55 and .59 r-values, respectively). The disdain for horror extends across the board, as it received the lowest average score of all of the genres considered. For drama and childrens' movies, they perform about as well as movies at large.

Drama films are most likely to be heavily politically or culturally ideological, which is probably why critics don't do so well in this serious genre. It's similarly a tough one for the general public, as the characters and the actions they take are often judged in many gradations, open to more interpretation than say, concluding that Scar is a bad dude.

Fat Knowledge, in wondering how the general public fared compared to the critics, pointed me to Yahoo, which constructs an average user score based on thousands of online ratings. The folks obliterate the critics. Rotten Tomatoes' critic scores, the most reliable relative to box office receipts of the different services looked at, correlate with revenue at .295 to the Yahoo users' .415.

While the critics do a little better by genre than by all movies in general, Yahoo users beat the critics in every genre, excepting action and science fiction, by slim margins. Romance is the most glaring. Critic scores and box office performance correlate at a meaningless .06 to users' .77. That blossoming could never happen in real life! But two cowboys... Horror was similarly divergent. Apparently stuffy critics cannot degrade themselves enough to review horror movies with any seriousness--they all belong in the garbage bin!

The genre-to-revenue relationships (r-values) for professional critics, Yahoo users:

Action: .55, .48
Comedy: .41, .58
Drama: .28, .38
Horror: .09, .61
Kids: .27, .79
Romance: .06, .77
Sci-Fi: .59, .55
Thriller: .31, .42

Simply put, if you want to know how a movie will do, ask the moviegoing public that will go to see it. Not only do the movie-maven plebians better than the putative experts at predicting actual box office performance, they're a lot more stable across the board, always providing at least a moderate amount of insight. Of course, uppity critics would hate to be amalgamated as a group, so unique are their individual opinions! Find a critic that you feel to be insightful, and his criticism becomes valuable.

The data, via Swivel, is available here.


Fat Knowledge said...


I am not surprised that Yahoo users have a higher correlation than the critics.

I took a quick look at the numbers and I wonder if there is really enough data to support very strong conclusions. For example, in romance, I bet if you took out The New World and replaced it with The Notebook that the critics score would jump up a bunch. And looking at the box office receipts of romance, they all seem pretty dinky. Don't know if that would impact the correlation or not.

Oh, and I have been messing around with Google Docs for their spreadsheet application. Pretty cool how it works, and how you can share spreadsheet data on a webpage or embed it in your blog. I think it is better than swivel for sharing data. I think you might like it.

agnostic said...

But critics aren't trying to predict how much money a movie will make, just what they thought of the movie. And unless the general public has rarefied tastes, sales and artistic worth may not correlate that strongly. Don't worry, there's no "John Q Public is a moron for not liking modern art" argument coming...

Take your favorite Woody Allen movie and release it nationwide, competing with a good Hollywood comedy like The 40 Year Old Virgin. As funny as the latter is, it won't stand up to the best Woody Allen movie merit-wise, yet it will garner much more at the box office.

And it's not because Woody is too cerebral or depressing: he makes comedies that involve lots of gags and one-liners, are rarely slow-paced, and so on. They focus almost exclusively on well-to-do neurotic Manhattanites, but why should that detract from its worth?

The scenario would be worse if you put 8 1/2 up against a good Hollywood drama. The general public just doesn't have very refined tastes.

Now, a lack of concern for blockbuster status may be a necessary condition for being a serious critic, but it's not sufficient -- that's why lots of critics are almost as stupid as the public.

They're stuck in the adolescent mentality that holds that going against the grain is cool in itself, and that rising above the level of Independence Day makes something bold and intriguing. They're like critics who think that Pink Floyd or Jasper Johns is deep.

Sure, these critics have cleared the "don't care about money" hurdle, but that's the easiest one, nothing to brag about. What should come after that is the hard part of being a critic, yet many stop there.

As in so many other areas of cultural life, we have lost the aristocratic tradition of dispassionate analysis and debate. The masses will always be less refined than the elite, but the elite have stopped caring about having good values and behavior rubbing off on their social inferiors, at least in America. And most of the elite is so self-complacent in its immature haughtiness that it remains stunted.

The internet could be a way out, though, since people like Steve Sailer or Udolpho would face trouble getting jobs at either mainstream or elite bodies to review movies.

agnostic said...

or Michael Blowhard!

Audacious Epigone said...


Removing that movie does better the correlation to .34, but it's not statistically significant.

Nine is a small N, and going over a couple of years would shore it up, but the p-value for yahoo users is .01 (1% 'chance' that the apparent relationship is a fluke) even with the limited size. What stands out is how rigorous the users' ratings are across the board, even with the small sample sizes.

You're always at the head of the tech curve. I'm a laggard. Most of the stuff you've suggested I've adopted, so I'll fiddle around with Google Docs. Thanks for pointing it out.


Makes sense in that a genre like action, free of anything like that Woody Allen effect, there is a lot of agreement between the general public and the critics.

Outside of comedy or drama, though, I don't an artistic or aesthetic value that is lost on the rabble. In the kids genre, for instance, I see a puerility worse than the kind you mention coloring the critics: The Pacifier is butchered by the critics, no doubt because it stars a BAMF action hero they universally detest.

Relative to the general public, critics also seem to swoon over melancholy themes and disspirited characters.

My insight is limited, as I rarely watch movies, in fact I don't even have a DVD player! Hehe, kidding.

"They're stuck in the adolescent mentality that holds that going against the grain is cool in itself, and that rising above the level of Independence Day makes something bold and intriguing. They're like critics who think that Pink Floyd or Jasper Johns is deep."

Great quote. How often I've grasped for such a descriptor in more social situations than I can count, but I've never been able to offer such a succinct delivery.