The Story of DeepSID

It’s funny how it can sometimes feel like you’ve had your proudest creation behind you.

Maybe it was the time in the 80’s with the editor and the C64 tunes. The couple of maps I created for Half-Life. Or the years where I became overly obsessed with PC games and I fell off the face of the Earth for a few years. Then I created a games checklist-and-database called GameDeed and thought, I might actually have a web site that might steal some of the thunder from The BackLoggery and HowLongToBeat. Was I on the way to my proudest creation yet? No, not really.

GameDeed turned out to be a resounding fiasco.

GameDeed

Nevertheless, GameDeed was still very important and I don’t regret the time I spent coding it. I learned so much about web development from it that later benefit DeepSID. Before GameDeed I made a few static web sites and even my own theme for WordPress, but none of that would have prepared me for the monster site that DeepSID would eventually evolve into.

But GameDeed sure was a frustrating site. I started coding it in February 2013 and had a lot of fun setting up the table design. Until I stumbled upon the name GameDeed and bought the domain for it, it was just called “Project Belt” for a while. I remember that when I released it in August 2013, Huseyin Kilic (Wisdom) commended me and believed the site would be a great success. I then tried to spread the word in various forums and social sites, but it was surprisingly difficult to get any traction. Sites like e.g. Reddit didn’t take kindly to self promotion and I had to learn all that the hard way on the internet.

For some reason I just couldn’t get any luck with GameDeed.

As the years went by, I naively kept thinking that if I add this or that feature to GameDeed, surely the gamers on the internet would start appreciating it. So I added Steam synchronization. I scraped a ton of games from Steam using their API. In fact, I scraped Steam so thoroughly that I’m pretty sure I’m the one responsible for the nerf they made to the API so that multiple games could no longer be requested through the API. Only one at a time. Luckily I was almost done scraping when they did that.

I added a lot of other features to GameDeed, many of which took quite some effort and coding to do. I added Xbox synchronization. Bulk editing. A games database of C64 games with Zzap!64 review scores that no one gave a damn about. Description lines in your checklist deeds. An expandable space with Wikipedia articles and screenshots. A backlog feature (which I still think is probably the best thing about the web site today) and a ton of smaller features to boot. But no matter how hard I tried, I just couldn’t for the life of me make the site take off. The traffic always remained low.

So, in October 2017, I had finally had enough and froze all development of GameDeed. I even thought about taking the site down, but I couldn’t. Even though the traffic was low, I still had a few loyal visitors. The exceptions. They kept coming back, adding their checklist achievements. I wouldn’t dare take down the site and piss them off, so I let the site sit there. Maybe sometimes fix a bug, but nothing more than that.

Getting retro with C64 again

In all that time, all those many years, the retro activities of C64 didn’t interest me much. I believed it was all in the past and that I had finished my very last production for it in 1992. But now that I had lost most of my interest in playing games anymore, and obviously also didn’t have a coding project, I suddenly opened my eyes for what was happening in the scene. All those demos, the music, the graphics. It was better than ever before, and it just kept coming, party after party, for one competition after another.

I got bitten by the C64 bug once again.

No doubt the instigator of it all was the five-part blog series I wrote about my computer chronicles in the 80’s and 90’s. The reception I got from it was astounding. If only GameDeed had received just a fraction of that attention. The next month I made a comeback SID tune in CheeseCutter. My first in 25 years. Soon after that I even bought a small keyboard for testing chords and ideas.

As part of the renewed interest of music on the C64, I soon stumbled upon a SID visualizer by tamat that I found quite interesting. It could show the inner workings of the SID chip while playing a tune. I decided to expand upon it with a few improvements of my own, and it turned into an Improved SID Visualizer. It had new pulse width boxes, widened frequency boxes, and I also added various other minor tweaks.

One of the tweaks I did was to expand upon the list of tunes to choose from in the left side. While doing this, I sat back and pondered how cool it might be to have a proper browser here, instead of the short and static list. However, I knew that creating a good browser would require some effort. I wanted it to be worth my time, and the SID player in tamat’s visualizer had too many limitations. So I decided to forget about it.

But the thought had squeezed into my mind and wouldn’t let go.

A SID player for the web

I kept realizing more good reasons for creating my own SID player for the web. The app market for iOS had just one player for SID tunes, but it had an abysmal user interface that was just infuriatingly bad. If I had a web site, I wouldn’t have to deal with that app anymore. And then there was the problem with the High Voltage SID Collection. It had more than 50000 tunes. Apart from it being a hassle to download and extract, it also required a separate SID player. And that brought up the next can of worms.

Most offline SID players didn’t impress me much. There were a lot of simple players that barely showed the title, author and release fields along with a few drop-down menus for setting things up. One thing that kept confounding me was how almost all of them shoved the STIL text information too far away. These are accompanying texts that may describe the source of a cover or the composer’s own comments about the tune. Why would you want to hide that away, only to be popped up in an annoying window through an obscure menu or hotkey command? It should always be there, ready for reading!

Sure, there were a few okay SID players to be found that showed some quality, but I could still see a lot of room for improvements. My own takes on how to show information about the SID tune you’re listening to right now. How about players, for example? I always wanted to know what C64 player a tune was written in. None of the offline players I examined bothered to show this up front either. I kept getting ideas of how to do things better, or at least present them in a refreshingly different way.

So I decided to put together a proof of concept.

The cut of the first sod barely contained control buttons, and it only used Hermit’s emulator. His emulator was good for playing simple SID files, but it couldn’t do any of the advanced stuff such as digi or RSID. So many files failed to play in HVSC, and this brought me right back to the reason I didn’t want to go on from the Improved SID Visualizer. The SID emulator just wasn’t as good as I wanted it to be for a project like this. I needed it to at least play digi tunes. I started searching around for other SID emulators for the web, but I kept finding subpar alternatives. I was “this close” to aborting the project completely.

That’s when I found Jürgen Wothke’s WebSid.

Finally an emulator that could also play digi tunes. It wasn’t perfect. It didn’t run cycle-by-cycle and it didn’t emulate nearly as well as reSID, but that emulator didn’t exist for the web and it would probably have been too heavy to run smoothly on most computers anyway. WebSid was good enough for my project and it finally cemented the feasibility of my project. It was time to knuckle down.

Launching DeepSID

I started coding DeepSID in March 2018. WebSid became its main emulator. I just used the WebSid code from the GitHub repositories without Jürgen Wothke knowing anything about it. Of course I wanted to be sure that I could make it all work first, but perhaps I was also thinking of the good old quote, “It’s easier to ask forgiveness than it is to get permission.”

The same day I decided to launch DeepSID, I wrote an e-mail to Jürgen Wothke. I wanted him to have one last look at it right before I opened the gates. After all, he had written the main emulator for the site, so I wanted him to have that honor. Luckily, Jürgen was impressed. He spent several hours playing songs in it and liked the idea of users being able to rate them. He was even wishing for personal playlists.

When the web site was launched in May 2018, it had enough features to leave a good first impression. SID rows showed year and the player used. You could rate songs. Folders could be sorted. It had fast forward. An information box with name, author, copyright as well as PAL/NTSC and SID chip model flags. The STIL box was visible. It was even possible to play tunes from SOASC instead of using the emulators.

Adding a ton of features

Since the launch in May 2018, I have continuously expanded the site with additional features. I was quite driven for a long time and spent most of my free time updating the site. Especially 2019 was crazy. In fact, I managed to add so many features in a row that one guy on Facebook was asking me if I was doing time. I wasn’t – I was just being a nerd that had obsessions instead of normal hobbies.

Let’s take a look at the most prominent ones.

2018

  • A decent search feature
  • SOASC playback (as SID handlers)
  • Compute’s Gazette SID Collection
  • A page area with tabs in the right side
  • A tab page for Disqus comments
  • A tab page for STIL text
  • Main volume slider
  • A tab page for CSDb
  • A tab page for composer profile
  • Graph for years on profile page
  • A tab page for GameBase64
  • Lots of profile pictures
  • Personal and public playlists
  • Brands for composer profiles
  • Competition results in CSDb page
  • A piano view
  • Quality filtering of letter folders in HVSC
  • A graph view
  • Top 10 lists in the front page

The site was fresh and I was on a roll. It was a good thing I was this driven to begin with as comments and commending about new features were very limited. Even as late as August 2018, I had my doubts that DeepSID was visited much. I was afraid it was turning into GameDeed all over again.

More. I needed more features.

2019

  • Improved experience on mobile devices
  • Support for Open Graph
  • Recommendation boxes (and later an overview page)
  • A tab page for personal settings
  • Search “here” feature
  • A sundry tab for oscilloscope
  • Competitions folders from CSDb
  • A tab page for players and editors
  • A black color scheme
  • Jürgen Wothke upgraded WebSid with cycle-by-cycle support
  • 2SID and 3SID supported by graph and piano views
  • A tab page for Remix64 entries
  • Bought DeepSID.com (only used for redirection)
  • Support for tags
  • Added a ton of tags
  • Support for exotic tunes such as 4SID, 8SID and 10SID
  • A memo view
  • Legacy WebSid option for slower computers
  • An auto-play prevention overlay

The list above may not look like it, but some of those features were immense. Lots of entries to parse for competition folders and the page with players and editors. The black color scheme required that I went through every single color change everywhere and changed them into meticulously controlled CSS variables. And I added a ton of tags for literally tens of thousands of tunes.

I even experimented with the MobyGames API for game and image data other than from GameBase64, but alas the API had too tight limitations on calls per hour. That just wasn’t good enough for DeepSID.

I also started tracking online visitors at any given time. In September 2019, the numbers were not exactly encouraging. About half a dozen to a dozen users online. Luckily, the word started spreading and the numbers rose. In the end of September, I often had 30-40 users online.

2020

  • SID Happens folder for uploading new tunes
  • A ton of tags for CGSC tunes
  • Support for extended lyrics in CGSC
  • Removed SOASC playback (too many problems)
  • Avatar images for non-composers
  • Removed Disqus (it affected the performance of DeepSID)
  • Download playlists as ZIP-files

The site became more familiar to C64 sceners, especially when sites such as Remix.Kwed.Org and Remix64 started linking to its SID tunes. However, this also became the year where I looked at other projects too. I started helping Thomas “Laxity” Petersen with the development of his promising C64 music editor SID Factory II, and I began developing a visual overlay site for CSDb fittingly called CShellDB.

2021

  • Search shortcuts
  • Three columns with shortcut lists in the front page
  • Added a SID handler for YouTube videos
  • Color strips for a few specific players
  • An annex box with tips and composer links
  • Filter controls (6581 only)

I was starting to step down development of DeepSID – at least a little bit. Too many of my weekends had been absorbed by this and it was probably time for a much needed change.

I even had one feature I somewhat begrudgingly decided to abandon. It was a new comment system to replace the “hole” still left behind by Disqus. I had a lot of good ideas, like how a pane could lift up from the bottom to allow for comments. However, I decided to discard the idea because of how little the Disqus comment system was actually used. Sometimes weeks would go by without a single comment. I was quite confident history would repeat itself, and this feature was just too massive to be worth that.

It was my experience with so many unused features in GameDeed finally speaking.

2022 has barely started at the time of typing this, but so far I have added an unofficial groups folder as well as a new stats view. And that’s also where I’m starting to wonder if enough is enough. There really is such a thing as too much, and I’m also aware that I’ve probably crossed that line a long time ago. It has started to look a lot like a bloated Christmas tree.

The irony is, I actually despised ICQ back in the day because of that.

Stone Oakvalley’s Authentic SID Collection

As far back as in the first year of DeepSID, in 2018, I decided to add support for SOASC as additional SID handlers to choose in addition to WebSid and Hermit’s emulator. It made a lot of sense. Those that didn’t like emulators could listen to FLAC or MP3 recordings of SID tunes recorded from a real C64. And SOASC not only had this covered, they even did it for a wide range of SID chip revisions.

I made sure to add SID handlers for each of these revisions.

Sure, visuals like the piano and the graph view would be off limits, but it was made up for by being able to jump to any point in the song just by clicking the progress bar anywhere. It all felt promising to begin with. Then the problems began queuing up. I had CORS problems in Chrome and Edge and the Howler plugin I used wanted HTML5 playback to be enabled, but eventually I managed to solve those problems.

Then I decided to contact the SOASC guys to show them my implementation of their hard work in DeepSID. They were always nice and helpful people, but one thing they did insist upon was that I used one of their PHP scripts to request the songs. This script would automatically figure out the proper server to stream the music from. SOASC had gigabytes of recorded data, so logically it was spread out among several mirrors.

It made a lot of sense to use this script, but I could also feel that it took longer to start a tune when clicking it in DeepSID. There was also a major problem with HVSC updates. These changes some of their paths and it didn’t fit well together with the way I had designed my own system. This meant that a few songs here and there just refused to play. Together with the delay, it didn’t make for a great experience.

Sometimes the SOASC servers were also down for whatever reason. In fact, it happened so often that at one point I even added a dynamic status label in DeepSID showing whether it was online or not.

The last straw came in 2020. The SOASC guys decided to move some servers around, and it made the start delay even worse in DeepSID. I tried many things to see if I could speed things up, but none of it worked. Eventually I decided that the lag was now so severe that the user experience was just too terrible. So in September 2020, I decided to remove the SID handlers altogether.

I never heard a peep about this from the SOASC guys. I guess they didn’t see the value in this.

The Chrome crisis

In March 2021, the developers of the Blink web engine decided to patch a serious security bug. The bug made it possible to exploit something called the data race in the audio code. The need to fix this bug was made especially urgent because the developers learned that some attackers were already using it to gain control of a device or steal cross-origin data.

Normally the developers would have made sure to make the fix more stable, but because of its severity, it had to be rolled out as soon as possible. This happened in Blink version 89, and thus it soon became part of web browsers such as Chrome and Edge. Unfortunately the fix affected the ScriptProcessorNode API by introducing an audio stutter that was impossible to get rid of, even with large buffer sizes.

Since WebSid used this API, it also affected DeepSID.

Due to the nature of the problem happening inside the guts of the web browser, there wasn’t much I could do about it myself. All I could do was to recommend using Firefox instead, which was sort of a safe haven as they were using their own web engine. Luckily for me, Jürgen Wothke (the coder of the WebSid engine) would have none of this nonsense and put on his war gloves.

Jürgen immediately started a Chromium discussion thread read by the Blink developers. In the beginning they tried dismissing him with words about the ScriptProcessorNode being deprecated and how he really should be using the newer Audio Worklet instead. But Jürgen was relentless. He kept writing post after post, constantly keeping them on their toes.

The debate went on, all while DeepSID sounded like crap for everyone but the few Firefox users.

In the meantime, Jürgen translated 90% of Hermit’s emulator to the suggested Audio Worklet method as a test. However, Hermit’s emulator was relatively simple while WebSid was considerably more complex. It wasn’t just a SID emulator – it contained ~20 different emulators, it used WASM, supported asynchronous loading of additional resources from within emulators, had separate scope data streams, and many other features. Converting all this to Audio Worklet would have been an exorbitant undertaking.

Audio Worklet also required separate audio threads, and only HTTPS was supported.

Eventually the developers did manage to fix the bug, but it took quite some time. In the end of May 2021, Chrome and Edge updated to version 91 with the fix in it, and finally DeepSID was playing smoothly again. Nevertheless, that was more than two months where DeepSID was at the mercy of web engine developers claiming the use of a deprecated audio API was unwise.

It only proves how fragile DeepSID really is. And it’s not just the web engine. Think of all the resources it depends upon. CSDb, GameBase64, Remix64, updates from the CGSC and HVSC teams. There are a lot of strings attached and a chef is no better than his ingredients.

The future of DeepSID

As mentioned earlier, I’m trying to step down the once so intense development of DeepSID. It already has an abundance of features and it’s difficult to find space for more. But it’s not due to a lack of ideas. I still have a mile long wish list, and it’s there for all to see in the GitHub source codes of DeepSID. As both the developer and the administrator, I also have a responsibility of keeping it afloat and up-to-date.

You wouldn’t believe the steps I have to go through each time an HVSC update lands.

In late 2021 and the beginning of 2022, the visitor numbers used to be around 40-50 users at any given time. For some reason it’s been down to 20-30 for most of March 2022. I have no explanation for that. The record for the most visitors so far came on 28th December 2021. It was 78 concurrent users. But the problem is, I’m not actually sure if that’s really good or still ludicrous. I bet it is compared to CSDb.

But compared to GameDeed, it’s in an entirely different league of its own.

10 comments on “The Story of DeepSID

  1. I’d be lost without DeepSid, I can’t easily set up C64 these days (still have them in the loft though). I enjoy reading the comments people leave about mine and other composers songs, it is my go to for this. CSDB is great but I listen on Deepsid so it’s nice I can read and comment from there as well. Fantastic job JCH, probably one of my most visited and used websites.

  2. To say that I have really liked DeepSID from the day I first used would be a severe understatement. I always wanted an offline SID player that had many of the bells-and-whistles that DeepSID has now, but the ideas in my head just never got turned into actual code. As soon as I first glanced at the DeepSID site, I was like “THiS IS IT! This is what I wanted a SID player to be all along!” And on top of it, it was a web-based app, so no need for cross-compiling or coding it up on a cross-platform scripting language or something like that.

    I don’t know about anybody else, but I am HUGELY appreciative of the work you’ve put into the DeepSID code and into keeping the DeepSID site running. You should really have a tipjar on the DeepSID site – I’m sure I wouldn’t be the only one who would be willing to donate to you to keep it going.

    As for the SOASC story (of which I wasn’t aware until now) – that’s sad. They have built up a really valuable database of recordings, and DeepSID would be a far superior UI front-end for that than what they have on their own site.

    Thanks for all your efforts throughout the decades (not just for DeepSID, but for everything else you’ve done for the C64 scene in the past, too), and I hope you’ll keep DeepSID running in the future, too!

  3. Wow, thanks. It was also great to finally have a site that was really used after the fiasco that was GameDeed. But as I wrote in my history here, I learned a lot of making GameDeed so it was sort of a necessary evil.

    I hope to be supporting it for many years to come.

  4. Well, what a fantastic insight. I did wonder what happened to the SOASC links and remember the Chrome SNAFU vividly 😉 Clearly DeepSID and CShellDB are labours of love, and the quality just shines through. Both are first class resources and visited fairly regularly by me, and I imagine many more silent lurkers. Thanks for all your hard work!

    A personal feature request for CShellDB would be quick shortcuts to filter on main release categories: cracks/music/graphics/demos. Sometimes it would be nice to browse only demo releases or just look for new games 🙂

  5. I blame DeepSID for slowly sucking me back into the C64. I swore retro was an impractical waste of time. Now I visit more days than not. I probably found it via Facebook, but can’t recall if it was before or after the pandemic. And while I think VICE is crap due to it’s awful keyboard support (to say nothing of how it bogs the system down), I have started to noodle around with programming again. Because of DeepSID.

    I have forgotten so much. I assumed it would all come back, but no. Programming is hard now. It used to be like breathing for me. Ah well, I will keep trying. Because DeepSID reminded me of a time when computers were fun.

    I recall the panic when the Google Chromium dicks broke the audio. Thanks to JW for knowing how to get those dickholes to fix things. Those dickholes obviously grew up on Apples or IBMs.

    Anyway thank you again for this amazing DeepSID which lets people explore and search for C64 songs based on various criteria. I do miss the old comment system. I bet more people would engage more if they could just insta-login with their Google or FB acct. Or if no login was needed. Probably a downside with that.

    But I digress. DeepSID has let me discover music and artists I never heard of. Thanks for that too. That Jerusalem Spider guy and Jammer are impressive.

    Since someone else requested a feature, I would like to have a filter a search by multiple criteria, kind of like how Dyxum.com lets you filter as you search for zoom lenses.

  6. Thanks, it’s always nice to hear about how DeepSID is received.

    Logging in with Google or Facebook? Hmm! Believe it or not, I didn’t actually think of that. I need to investigate this further. I’ve written it down on my ToDo list.

    I’m in a bit of an off period these days regarding DeepSID coding, but eventually I’ll get back in the mood again. When that happens I’ll seriously consider adding advanced searching.

Leave a Reply