Centerfield

Long Tail Controversy and Explanations

Dr. Elberse of Harvard caused a ruckus in an article in the current Harvard Business Journal issue about an interesting new study she's been involved in ( Wall Street Journal summary here), as far as I know the only one to look at how online buying patterns are changing. She also raises a ruckus with her title, "Should You Invest in the Long Tail," which is chosen to discredit Chris Anderson's Long Tail book.

I see her title as attacking a straw man, personally. My sense of the Long Tail isn't about ignoring what's popular, as she implies, but rather understanding that stocking the top ten of anything is getting you less market share than it did pre-Internet. But, Chris Anderson is perfectly able to take care of himself, as he shows. As is Anita in response. They're also more fun than my verson.

The Long Tail is of particular interest to me, because I studied it personally, and was involved in a way to overcome it somewhat in one arena, web caching. The Long Tail is a newfangled network-researchy name for an oldfangled problem: why libraries need to be big to keep more than a handful of their clients happy, and why you can never keep all your customers happy with any finite-sized library. Even in the Library of Congress, some people will want books only published abroad.

A mathematician named Zipf came up with a mathematical model that he designed to model the frequency of linguistic word use. It decays slowly, to reflect the fact that, although "the" and "and" are way popular, you also see "arthroscopic" and "quantitative" in real use as well. The law turns out to apply to libraries and web caches as well.

A cache is a bunch of commonly used stuff kept close by for quick access. They've traditionally done well indeed in computers, speeding up processors and disks alot. A web cache is that idea applied to the Web, except it doesn't actually work as well. And that's because what people look at on the web follows Zipf's Law. You need to continuouly prefetch the entire Web onto your web cache to get the kind of improvement you get with disk caches.

I was involved with a couple of measures to work around this a bit. One measure was to support efficient push and look at how much different policies for pushing pages accessed by cache users would help. Another was, observing that caches work better the more people they have going through them, to implement a scalable way of having big cache clouds work together. Our work showed each of those could've been a big improvement, but would still not bring you to the domain of disk cache improvements. So, *I* certainly saw fit to invest in the Long Tail. The answer to Elberse's question, of course, depends on what you're doing.

So what did Dr. Elberse et al see over time? That the long tail on the overall curve tails off at the end and becomes more concentrated at the moat interesting stuff. The highest 10%, for example, becomes more popular and the lowest 10% becomes less popular. That's very interesting stuff, and it's something to think about and research more. The work raises one additional question for me: if you have a certain number of marketing or cache slots to fill, will they become more or less effective over time? Notice that's a different question.

Another thing to think about is that books' title popularity is still long-tailed, despite having had centuries to concentrate.

Hat tip and good call on the Elberse/Anderson controversy at Marginal Revolution.