THE tracking of top-seller lists is a national obsession. Our culture is a massive popularity contest. We are consumed by hits — making them, choosing them, talking about them, and following their rise and fall.

Every weekend is a box- office horse race, and every Thursday night is a Darwinian struggle to find the fittest TV show and let it live to see another week. A few hit songs play in heavy rotation on the radio dials, while entertainment executives in all these industries sweat as they search for the next big thing.

This is the world the blockbuster built. The massive media and entertainment industries grew up over the past half- century on the back of box- office rockets, gold records and double-digit TV ratings. No surprise that hits have become the lens through which we observe our own culture. We define our age by our celebrities and mass-market products — they are the connective tissue of our common experience.

The star-making system that Hollywood began eight decades ago has now spun out into every corner of commerce, from shoes to chefs. Our media is obsessed with what’s hot and what’s not. Hits, in short, rule.

Yet look a little closer and you’ll see that this picture, which first emerged with the postwar broadcast era of radio and television, is now starting to become tattered at the edges. Hits are starting to — gasp — rule less. No 1 is still No 1, but the sales that go with that are not what they once were.

Most of the top 50 bestselling albums of all time were recorded in the Seventies and Eighties (the Eagles, Michael Jackson), and none of them was made in the past five years. Hollywood box-office revenue was down by more than 6 per cent in 2005, reflecting the reality that the cinema-going audience is falling even as the population grows.

Every year network TV loses more of its audience to hundreds of niche cable channels. Males aged 18 to 34, the most desirable audience for advertisers, are starting to turn off the TV altogether, shifting more and more of their screen time to the internet and video games. The ratings of top TV shows have been falling for decades, and the No 1 show today wouldn’t have made the top ten in 1970.

In short, although we still obsess over hits, they are not quite the economic force they once were. Where are those fickle consumers going instead? No single place. They are scattered to the winds as markets fragment into countless niches.

The one big growth area is the web, but it is an uncategorisable sea of a million destinations, each defying in its own way the conventional logic of media and marketing.

This shattering of the mainstream into a zillion different cultural shards is something that upsets traditional media and entertainment no end. After decades of executives refining their skill in creating, picking and promoting hits, those hits are suddenly not enough.

The audience is shifting to something else, a muddy and indistinct proliferation of . . . well, we don’t have a good term for such non-hits. They’re certainly not “misses”, because most weren’t aimed at world domination in the first place. They’re “everything else”.

It’s odd that this should be an overlooked category. We are, after all, talking about the vast majority of everything. Most movies aren’t hits, most music recordings don’t make the top 100, most books aren’t bestsellers, and most video programmes don’t even get measured by Nielsen, much less clean up in prime time. Many of them nevertheless record audiences in the millions worldwide. They just don’t count as hits, and are therefore not counted.

But they’re where the formerly compliant mass market is scattering to. The simple picture of the few hits that mattered and the everything else that didn’t is now becoming a confusing mosaic of a million mini-markets and micro-stars. Increasingly, the mass market is turning into a mass of niches.

How technology is turning mass markets into millions of niches

IN 1988, a British mountain climber named Joe Simpson wrote a book called Touching the Void, a harrowing account of near-death in the Peruvian Andes. Though reviews for the book were good, it was only a modest success, and soon was largely forgotten.

Then, a decade later, a strange thing happened. Jon Krakauer’s Into Thin Air, another book about a mountain-climbing tragedy, became a publishing sensation. Suddenly, Touching the Void started to sell again.

Booksellers began promoting it next to their Into Thin Air displays, and sales continued to rise. In early 2004, IFC Films released a docudrama of the story, to good reviews. Shortly thereafter, HarperCollins released a revised paperback, which spent 14 weeks on The New York Times bestseller list. By mid-2004, Touching the Void was outselling Into Thin Air more than two to one.

What happened? Online word of mouth. When Into Thin Air first came out, a few readers wrote reviews on Amazon.com that pointed out the similarities with the then lesser-known Touching the Void, which they praised effusively. Other shoppers read those reviews, checked out the older book, and added it to their shopping carts.

Pretty soon the online bookseller’s software noted the patterns in buying behaviour — “Readers who bought Into Thin Air also bought Touching the Void” — and started recommending the two as a pair. People took the suggestion, agreed wholeheartedly, wrote more rhapsodic reviews. More sales, more algorithm-fuelled recommendations, and a powerful positive feedback loop kicked in.

Particularly notable is that when Krakauer’s book hit shelves, Simpson’s was nearly out of print. A decade ago readers of Krakauer would never even have learnt about Simpson’s book — and if they had, they wouldn’t have been able to find it. Online booksellers changed that. By combining infinite shelf space with real-time information about buying trends and public opinion, they created the entire Touching the Void phenomenon. The result: rising demand for an obscure book.

Oh sure, there’s also a lot of crap here in the Long Tail. But then again, there’s an awful lot of crap hiding between the radio tracks on hit albums, too. People have to skip over it on CDs, but they can more easily avoid it online, where the best individual songs can be cherrypicked (with the help of personalised recommendations) from those whole albums. So, unlike the CD — where each crap track costs perhaps one-twelfth of a $15 album price — all of the crap tracks online just sit harmlessly on some server, ignored by a marketplace that evaluates songs on their own merit.

What’s truly amazing about the Long Tail is the sheer size of it. Again, if you combine enough of the non-hits, you’ve actually established a market that rivals the hits. Take books: the average Borders carries around 100,000 titles. Yet about a quarter of Amazon’s book sales come from outside its top 100,000 titles.

Consider the implication: if the Amazon statistics are any guide, the market for books that are not even sold in the average bookstore is already a third the size of the existing market — and what’s more, it’s growing quickly. If these growth trends continue, the potential book market may actually be half as big again as it appears to be, if only we can get over the economics of scarcity.

Venture capitalist and former music industry consultant Kevin Laws puts it this way: “The biggest money is in the smallest sales.”

Who killed the hit album?

WHAT caused a generation of the industry’s best customers — fans in their teens and twenties — to abandon the record store? The industry’s answer was simply “piracy”: the combined effects of Napster and other online file trading and CD burning and trading gave rise to an underground economy of any song, anytime, for free. And there’s something to that. Despite countless record industry lawsuits, the traffic on the peer-to-peer (P2P) file-trading networks has continued to grow, with about ten million users now sharing music files each day.

But while technology was indeed behind the customer flight, it didn’t just allow fans to sidestep the cash register. It also offered massive, unprecedented choice in terms of what they could hear. The average file-trading network has more music than any music store. Given that choice, music fans took it. Today, not only have listeners stopped buying as many CDs, they’re also losing their taste for the blockbuster hits that used to make them throng those stores on release day.

Given the option to pick a boy band or find something new, more and more people are opting for exploration, and are typically more satisfied with what they find.

How long tails emerge

NONE of the aforementioned happens without one big economic trigger: reducing the costs of reaching niches. What causes those costs to fall? Although the answer varies from market to market, the explanation usually involves one or more of three powerful forces coming into play.

The first force is democratising the tools of production. The best example of this is the personal computer, which has put everything from the printing press to the film and music studios in the hands of anyone. The power of the PC means that the ranks of “producers” — individuals who can now do what just a few years ago only professionals could do — have swelled a thousandfold.

Millions of people now have the capacity to make a short film or album, or publish their thoughts to the world — and a surprisingly large number of them do. Talent is not universal, but it’s widely spread: give enough people the capacity to create, and inevitably gems will emerge.

The second force is cutting the costs of consumption by democratising distribution. The fact that anyone can make content is only meaningful if others can enjoy it. The PC made everyone a producer or publisher, but it was the internet that made everyone a distributor.

At its most dramatic, this is the economics of bits versus atoms, the difference between fractions of pennies to deliver content online and the dollars it takes to do it with trucks, warehouses and shelves. Still, even for physical goods, the internet has dramatically lowered the costs of reaching consumers. Over decades and billions of dollars, Wal-Mart set up the world’s most sophisticated supply chain to offer massive variety at low prices to tens of millions of customers around the world. Today anybody can reach a market every bit as big with a listing on eBay.

The third force is connecting supply and demand, introducing consumers to these new and newly available goods and driving demand down the Tail. This can take the form of anything from Google’s wisdom-of-crowds search to iTunes’ recommendations, along with word-of-mouth, from blogs to customer reviews. The effect of all this for consumers is to lower the “search costs” of finding niche content.

In economics, search costs refer to anything that gets in the way of finding what you want. Some of those costs are non-monetary, such as wasted time, hassle, wrong turns and confusion. Other costs actually have a dollar figure, such as mistaken purchases or paying too much for something because you couldn’t find a cheaper alternative. Anything that makes it easier to find what you want at the price you want lowers your search costs.

Self-publishing without shame

WE THINK of books through a commercial lens, assuming that most authors want to write a bestseller and get rich. But the reality is that the vast majority of authors not only won’t become bestsellers, but also aren’t even trying to write a hugely popular book. Each year, nearly 200,000 books are published in English. Fewer than 20,000 will make it into the average book superstore. Most won’t sell.

In 2004, 950,000 books out of the 1.2 million tracked by Nielsen BookScan sold fewer than 99 copies. Another 200,000 sold fewer than 1,000 copies. Only 25,000 sold more than 5,000 copies. The average book in America sells about 500 copies. In other words, about 98 per cent of books are non-commercial, whether they were intended that way or not.

The quest for mass-market acceptance requires compromise — a willingness to pick topics of broad rather than narrow interest, and to write in conversational rather than academic style. Most writers can’t do that and many others won’t. Instead, the vast majority of authors choose to follow their passions and assume they won’t make money. Many want no more than to be read by some group that matters to them — from their peers to like-minded souls.

Such profitless publishing can be lucrative all the same. The book becomes not the product of value but the advertisement for the product of value — the authors themselves. Many such non-commercial books are best seen as marketing vehicles meant to enhance the academic reputation of their authors, market their consultancy, earn them speaking fees, or just leave their mark on the world. Seen that way, self-publishing is not a way to make money; it’s a way to distribute your message.