Net, Blogs and Rock'n'Roll

What makes a good recommender system, and how do you validate it?

If you talk to the people who design systems to produce recommendations tailored to user preferences, you'll see lots of impressively daunting mathematics, with formulae for measuring things like "Mean Absolute Error".

If you talk to the people who sell these systems, you'll hear stories of uncannily perceptive and prescient suggestions that anticipated interests that Jo Consumer didn't even know she had yet.

The information that neither tend to volunteer until asked is that there are no agreed criteria for a perfect, or even a good, recommendation.

Hence you will see reviews of different recommender systems where people put the same starting point into different systems and then compare the results to judge which is the best. But the judgements, like this one, are based on the subjective views of experts: they have no way of articulating what's behind those judgements in a way that could be replicated in other studies. And how reliable are those judgements? Well, when Paul Lamere asked the readers of his blog to compare two sets of recommendations and identify which was done by an expert reviewer and which by machine, over three quarters of the respondents got it wrong. That means if you'd tossed a coin you could have done better than asking these people. (I was one of the few who got it right, but I had already said that there was no way to determine which was which with any confidence, therefore my answer was a guess, and it turned out to be a lucky one.)

So what is a good recommendation, and can we measure value or 'accuracy' at all?

Further reading: notes and bibliography

At university, my supervisor used to say that he would always read the references at the end of a research paper first. His reasoning was that this gave you an easily skimmable indication of both the coverage of the paper and its influences. I'm not sure that many people share this attitude, but if you do, you'll be interested to see that I've put both of the Notes and the Bibliography of Net, Blogs and Rock'n'Roll online.

It's probably more likely that you'll find these useful if you're one of the growing masses who have read the book, and would like easy connections to the articles and books that I mentioned in it. The notes are all hyperlinked, so you don't have to copy out URLs from the back of the book, just click the link to go there directly. Ditto with the bibliography, which has links to further details of each book on LibraryThing and Amazon.com.

Money making scheme?

In a review of Net, Blogs and Rock'n'Roll (the book) in this week's Spectator, Christopher Howse writes

Jennings, a full-time creature of the net, does not deny the popular delusions and madness of crowds that give some music releases undeserved success, but he comes up on page 64 with a practicable scheme for aggregating trusted criticism that should make him money, if someone takes him up on it.

What's the scheme? Well I could be a tease and say that you have to buy the book to find out. But that's not really my style. If I let you in on the secret, perhaps you can tell me something useful? Here's what I assume must be the paragraph in question:

What I’d like to do is compile my own chart. I’d like to be able to select a handful of my friends with whom I share different tastes plus a few other acquaintances, critics or DJs whom I count as tastemakers, and then see a chart aggregating what they have been listening to in the last week. Such a chart would give me useful, trusted pointers to the kind of music I might find interesting. It would also let me know when my friends and I have been listening to the same thing, which could spark further discussions. It is surely technically possible, and I hope someone will build it before long.

Now my question to you is: how do I make money off this?

iPods and identity: a film and a book

What's on your mind when you use your iPod? Are you sealing yourself off from an inhospitable environment, or are you creating a soundtrack to add an imaginary gloss to that environment? These are the sorts of questions being posed by a new film project, Between Ears, by two Dutch filmmakers.

The film is in production and you can contribute to the research via a questionnaire. From the press release,

Between Ears will be a cinematic trip around the world, that leads the viewer along to a surprising diversity of people who shut off from reality around them. With the help of an MP3 player they try to move into a place between their ears. The focus of the film is mainly on the intense experiencing of image and sound, in a way which enables the viewer to identify with the people in the film.

Much of the field of sociological study of people listening to personal music has been staked out by Michael Bull at University of Sussex, going back before the iPod to the days of the original Sony Walkman. The Between Ears site has a video interview with Michael, and his new book, Sound Moves: iPod Culture and Urban Experience, was published just a few weeks ago. Here's my review of his last book, Sounding Out the City (2000).