Downloads Server Outage
Incident Report for BlissRoms
Postmortem

What happened?

As part of the cleanup operation regarding our backend codebase, we were migrating to a new format of fetching the backend from our repository hosting service GitHub. Unfortunately, the new backend code depended on a newer version of the database program and as a result we could not finish the migration in a timely manner.

What is the potential impact to users/maintainers?

Nothing, apart from the downtime. The migration was slow to complete since we had to perform it manually, but it was successful and no data was lost. We keep backups to ensure that we can roll back immediately if we detect any problems or instability in the system.

Future steps

In the future we plan on making automated scripts to ensure that the migration process goes smoothly.

Posted May 21, 2022 - 06:59 UTC

Resolved
This incident has been resolved.
Posted May 21, 2022 - 06:56 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted May 21, 2022 - 06:50 UTC
Update
The servers are down as we work on the permanent fix.
Posted May 21, 2022 - 05:52 UTC
Update
The servers are temporarily back up. We are working on a more permanent fix.
Posted May 20, 2022 - 14:54 UTC
Identified
The downloads server is currently unavailable due to unexpected maintenance. We are working on a fix.
Posted May 20, 2022 - 14:45 UTC
This incident affected: Backend (API) and Websites (BlissRom Download Servers).