A question I’ve been asked from time to time about my web site DeepSID is how the SID composers in the MUSICIANS folders have been divided according to the quality of their songs.
All of the composer folders in each letter folder inside MUSICIANS have received a rating from 1-5 stars. I then decided that a certain amount of stars, say 3, determines that the composer is at least okay, maybe even good. If the rating is 4 or more, it’s a great composer.
This was done because I needed this division for at least three features of DeepSID:
- The list of recommendations, available from a link in the top line.
- The “All”, “Decent” and “Good” sort options for every MUSICIANS letter folder.
- Jumping to a random composer of a decent quality from the front page.
I knew that judging the composers to fit these three features would be a sensitive area. Even a minefield. I thought for a while about how I wanted to proceed doing this. I really wanted to involve a lot of people, but there were complications. Let’s go through each of these.
Wisdom of the crowd
There is a process called the wisdom of the crowd where the collective opinion of a group of people makes for a fair judgment, especially if the group is large. To quote Wikipedia, the idiosyncratic noise associated with each individual judgment cancels out. This would have been the perfect way to judge the composers in the two major collections used by DeepSID.
I also knew it was impossible. Not only when I created DeepSID, but probably always will be.
At the time of typing this, the High Voltage SID Collection contains more than 53000 tunes. Out of these, the MUSICIANS folder alone take up more then 49000 tunes. But to judge a composer, I wouldn’t have to rely on the ratings of all of his or her tunes. A rating for his entire folder would suffice. That’s a fair point, but that amounts to more than 1700 composers. It’s still a big number.
Also, you still have to listen to a certain number of tunes to make that decision.
For this process to work, it needs to have enough people voting for a song to work – and that’s exactly the problem. The C64 scene is a niche. Sure, there a lot of fans all around the world, but the problem is that you have to filter them. A lot are only interested in collecting and playing the games in a casual way. I’m just guessing here, but the fans interested in voting for SID tunes is probably only a small subset of this. I could probably add more filter steps here, but the really big one in the end is DeepSID itself. The biggest visitor numbers I have seen on a daily basis is approximately 40-50 users. It’s good, but not a lot.
Now, imagine that each visitor of DeepSID wants to rate tunes and folders. First, the visitor would have to register and login. Typically only about a third of all visitors do that. Now the visitor starts rating by clicking stars. Since there are more than 1700 composers and 53000 tunes, most visitors are only doing this for an extremely small subset of that. It will mostly be the popular composers, the popular tunes. Rob Hubbard’s classics will of course get a lot of ratings. Jeroen Tel, sure. But as soon as we move past these legends, the ratings are definitely going to scatter like dust in the wind. The wisdom of the crowd is not of much use when you have hundreds of ratings for Rob Hubbard while there are none for the underdogs.
But even in spite of this, at least the list of recommendations would be somewhat fair because after all, these are the legends. They are what people would probably rate for.
It’s just not good enough for the other two features. Sure, jumping to a random composer is not that important, but it was added because of the other feature where that was important – the sort options for MUSICIANS. For this to work in each folder letter, I needed to have a rating for each and every single composer folder or it wouldn’t be as useful as I wanted this feature to be.
This required me to have several user ratings for all of those folders. It was certainly not the case when I created the feature, and I’m pretty sure it’s not even the case today.
Asking for opinions
All right, so the wisdom of the crowd – tons of visitors rating tunes and folders – is clearly not feasible, at least not for many years to come. But how about just talking to other SID fans? Enthusiastic composers, dedicated listeners that have been through most of the collection already?
While I would have loved that, I wanted to make the new features happen while I was coding them. Striking while the iron was hot. I could have asked around, but again, taking the more than 53000 tunes and 1700 composers into account, it would have been a really big question to ask. I was afraid that it would be a mix of spotty answers and long response times. Family dads that only had time at certain hours or days.
I would give composer X and Y this many stars. Who? Composer Z? Uh… never listened to him. I can’t say.
Sure, I could have done that and used the little information I received to help me get off the ground. But there would still have been a lot of composers that no one, or at least very few, would know about.
All by myself
Since I knew I needed 100% coverage of ratings for all composers, I had to do this myself. It simply looked like I had no other choice at the time. Not if I wanted it to get it done within a reasonable time frame. But given the fact that DeepSID was already gaining in popularity and becoming a cherished online SID player, I also knew I had to try to be a fair judge as I parsed through the many composer folders:
- No bias was allowed. Maybe I didn’t like the genre, but the composer might still know his art.
- I had to listen to at least 6 tunes in each composer folder before making a judgment.
- Only exception to the above rule would be famous composers I had already heard a lot from.
- The choice of music player should not tilt my bias. Maybe the composer didn’t have a choice.
- Be aware of the years. He might have been bad to begin with, but listen to him now.
As you might imagine, going through the more than 1700 composer folders took quite a while. I began rating composers in the end of November 2018, and I had reached the last one in the end of March 2019. So that took about four months to do. Lots of afternoons spent listening and rating.
Four months without involving anyone but myself. And with no wife and no kids, I had the time.
During the process of listening to each composer, I noticed a lot of patterns repeating themselves. To be honest, there was a lot of junk. Piles of pure crap. Remember, one of the philosophies of the High Voltage SID Collection is that it aims to contain as close to everything ever made, no matter the quality.
To be fair, even the HVSC team have their limits too – but it is very light. The most fascinating fact is that there are many folders with crap that are also quite massive. What drove this guy to make so much music, yet not really learn anything in the process? It really is Sturgeon’s law.
I tried to be as fair and objective as possible. Perhaps I didn’t always succeed, but I really tried.
If I could hear crap in the beginning but then a promising evolution towards better tunes, it influenced the rating. If I could hear that although the music itself was mediocre but at least the composer had a grasp at how to make instruments, it also influenced the rating. And of course vice versa, if I could hear this is a good musician in spite of the awful player and the simple beeps, it too influenced the rating. There were many things to consider in addition to the above list and I took the job very seriously.
Let’s come together
I know I have been walking on thin ice by evaluating the composer folders myself, but I also hope you understand that I did it to deliver a valuable feature much faster than would have been possible had many people been involved. It’s not that I didn’t want anyone to be involved. I would have loved that.
And it’s not too late!
During the years since DeepSID was launched, there have been a few persons that have contacted me, questioning a rating I have given a composer folder, then suggesting why they deserved better. Until now I have always accommodated this and tweaked the rating accordingly. They all had a good point.
If you have looked at the list of recommendations, or you have been using the quality sort feature in the MUSICIANS letter folders, then wondering why that underdog is missing or why that tone deaf clot is recommended, you are most welcome to contact me and suggest otherwise.