It's a question I plan to examine more thoroughly later in the week, but for now consider the absolute (in thousands) and percentage increases (decreases) in voters from '04 to '08, by race (using national exit polling numbers for now, and adjusting "Other" for '08 down to 2%, which is where it should be, not 3%, which comes to a total of 101% by race. I will later refine it by aggregating state level totals for '04 as was done for '08):
The white population has grown, not shrunk, since 2004. Neither the black nor the Hispanic population grew at those respective rates over the last four years. Whites were not enthused about this election. Non-whites were.
Two consecutive Presidential election cycles with exit polling data available at the state level allows for some interesting insights to be gleaned. One recurring media narrative open borders supporters like to emphasize is that while McCain held his ground among whites, he lost it not only among blacks, which was unavoidable, but also in the crucially important Hispanic vote, and consequently he went down in flames.
That's not the case. Even though national exit polling shows McCain receiving 55% of the white vote to Bush's 58%, that 3 point shift represents a net change of 5.5 million votes. His combined loss of support among blacks (7 points), Hispanics (9 points), Asians (9 points), and others (9 points) resulted in a net swap of 4.8 million votes. Whites comprise three-fourths of the electorate, so the percentage shift among non-whites must be three times the magnitude of the white shift just to break even. It wasn't, due to the diminishing marginal returns a black Democrat is able to reap given how overwhelmingly supportive of the Democratic party blacks already are.
McCain's drop in white support relative to how Bush performed in '04 is what resulted in an electoral landslide. Because the most competitive states are whiter and blacker (and less Hispanic and Asian) than the nation as a whole is, the 5.5 million net shift is even more important than it might initially appear to be.
Leaving the non-white vote from November 4th alone, if McCain had merely retained the level of white support Bush did in '04, he would've won Colorado, Indiana, North Carolina, Ohio, and Virginia. It wouldn't have been enough to keep Nevada, New Mexico, Florida, or Iowa (barely) from flipping to the Democrats, so Obama would've still won the electoral college, 296-242. But the election wouldn't have been the 365-173 blowout it ended up being.
What if, leaving the non-Hispanic vote unchanged, McCain had enjoyed the same support among Hispanics Bush did in '04? Using the inaccurate Hispanic numbers reported from that election's exit polling that overstated the GOP's performance among Hispanics, and recalling that much of the Hispanic goodwill towards Republicans was bought via the housing bubble that provided years of explosive construction work and astronomical increases in Southwestern housing valuations, he would've won Florida*. That's it. Obama's electoral victory would've remained overwhelming, at 337-201.
* It could've been enough for him to squeak by in Indiana as well. For that to be the case, McCain's support among Hispanics would have to have been 13 points lower than what Bush garnered in '04. However, EMR estimated that Hispanics comprised 3% of Indiana's electorate in '04 and 4% in '08. For both elections, that represents too small a sample for reporting voting patterns, so there's no way to say conclusively whether or not Bush's Hispanic support in the state would've been sufficient for McCain to have held on to it. Granting it to McCain, Obama wins 326-212, a margin still 60 electoral votes worse than retaining Bush's white support would've meant.