Thursday, December 27, 2007

Ranking obfuscation, reading-level style

Since I spent half an hour playing with this after being introduced to it by Ilkka Kokkarinen, I might as well try to make something out of nothing. The tool purports to classify the level of educational attainment required to understand whatever blog or other site you offer it.

Taking a run through my blogroll, GNXP ("High school") and Agnostic ("Junior high school") both fall short of my "College (undergrad)" rating. That in spite of the fact that the entire GNXP crew (including Agnostic) have at least two years more schooling under their belts than do I, and have IQs that probably average around a standard deviation higher than mine (if not more). Perhaps even more absurdly, my unserious facebook profile earns a "Genius" rating. The thing is obviously bunk (although it does give criticsrant.com, its hoster, a "Junior high school" rating).

But if you use a normative scale to rank things, people tend to pay attention, even when the rankings are tripe.

In the comments of Ilkka's post, Markku suggests that the tool probably just counts rare words. The three links I have to opinionaters outside of North America all come in as "Genius", and are the only three on the list that do (not that it's undeserved!). So he seems to be near the mark.

Hmm, "Genius" might be seen in a negative light as it requires the reader to make sense of gibberish and infer what is trying to be said but is being so poorly expressed. Meanwhile "Elementary school" is optimal, being written so clearly and succinctly that it takes little work on the part of the reader to make sense of what is being said... It is this kind of empty conjecture about dopey 'rankings' that is so useless.

Forget how suspect the methodology is. Let's use the results to broadcast something controversial! Why not start with a political slap, in the spirit of the IQ hoax borne out of results from the '00 Presidential election?

The nation's top two 'conservative' newspapers, the WSJ and the Washington Times, earn "College (undergrad)" and "High school" ratings, respectively. The top two 'liberal' papers, the NYT and LAT, both earn "Junior high school" ratings. So much for the haughtiness of stuffy coastal elites. They're revealed to be unfledged indeed!

This tool is silly, but it serves as a useful example of how rankings and indices that are arrived at without disclosing their methodologies should always be greeted with skepticism. Often, if an inordinate amount of digging is required to get some idea of what methods were used, or they are not available at all, the presenter is either tendentious, so lazy that even in granting him good faith his data entry is likely so sloppy that his results are probably erroneous, or he's disguising personal preference as quantitative data so that his output is about as useful as the information obtained in reading a daily horoscope.

8 comments:

Anonymous said...

Mine says college postgrad. I know I have readability issues; but some ideas can't be presented on a high school level without insulting everyone. JSBolton

MensaRefugee said...

Big whoopie. Im a Post-Grad.

I dont have to care about money anymore. I got status baby! STATUS!

Audacious Epigone said...

Yeah, yeah, both of you rub it in. Anyway, I'm comfortable with who I am and what this blog is!

Anonymous said...

I didn't intend to. This service could be ranking you on whether you have quotes with footnotes, or odd terms that only scholars use.JSB

Audacious Epigone said...

John,

I know, I was just kidding. The top ranking it gave to all of those on my sidebar that are outside of North America (and hence use different spellings and other English idioms that aren't common in the US) the 'genius' rating leads me to assume you're about right.

Soap said...

I used a reading level calculator built into WordPerfect to judge the reading level of a children's book I wrote once. Sure enough, it ranked as being a 3rd grade reading level (Flesch-Kincaid), which was about the age of my target audience.

This one is probably less complicated than that, judging by how broad the categories are. But Im pretty sure that reading level calculation in general is more complex than just counting rare words. There's the Flesch-Kincaid scale (http://en.wikipedia.org/wiki/Flesch-Kincaid) and a few others.

Audacious Epigone said...

Soap,

Maybe. But the sentence-length, syllables-per-word Flesch formula cannot be the tool in its entirety. None of the three non-North American sites in my sidebar can 'compete' on that scale with John S Bolton, yet each of them are 'genius' and he is 'college-post grad'. Probably some combination of each, although I suspect there is something silly lurking in there as well.

undergroundman said...

Surprisingly, mine rates Genius?