“It isn’t as astonishing the number of things I can remember, as the number of things I can remember that aren’t so” – Paine, Mark Twain: A Biography, 1912
This Sunday, the Vikings are playing the Lions. I’m confident they’ll be keeping score throughout the game. Yard by yard, play by play. Rather than having the refs remember who played better and declaring an arbitrary score on Monday morning,
There’s a huge, often un-acknowledged difference between what people say they do and what they do. This delta widens with time. If you’re looking for accuracy – metrics need to be captured in context. If you’re looking for fiction – then making up what someone did two days ago is as valuable as asking them to remember (making it up just might be more interesting).
PodcastAlley, Podcasts.Yahoo.com, and a number of the other podcast directories offer ranking and voting for “your favorite podcast”. There are three problems with this;
- The metrics at different sites don’t know about each other – diluting the value of each of them. For example, what does it mean to be #8 at PodcastAlley and simultaneously #23 at Podcasts.Yahoo.com?
- All these systems rank unrelated podcasts against each other – just like Arbitron or Nielsen Ratings. The only people interested in how, for example, meatloaf ranks higher than lawn mowers are advertisers. This ranking doesn’t help the listeners’ enjoyment or the producers improve (in fact it could be detrimental to both).
- I still have to remember to vote, at one site or multiple (just like Arbitron). The action of voting and ranking is separate from listening, so I don’t.
I’ll let Mark Ramsey wrap this one up for us:
“If we want the advertising community to place any credence whatsoever in our measurements, then we are obliged to use measurement methodology which inspires credence.”