Ever since we got set up with bank accounts, one of our priorities for our new funding has been to migrate our servers. For context, all Codidact communities are currently running on a single server provided by one of our volunteers at their own expense, and have been since the start. Now that we have funding of our own, it's time to address that and take the opportunity to upgrade as well.

This week, that project has got underway. Our sysadmin team have planned out what we want our architecture to look like, we've bought the server capacity, and we're on the way to setting it up.

Next steps

We're not quite ready to make the move yet — there's a bit more work to do before that point. First and foremost, we want to test-run the new architecture: we'll take a copy of the current live database and load that onto the new servers, then expose them to the Internet and test that they work as expected. We'll probably need some volunteers to help us try to break it — keep an eye on Discord for that coming up.

Once that's done we'll be ready to start the migration process. We'll announce a maintenance window once we have dates and timings, during which the network will be read-only while we move all the data. At the moment we expect this to take a few hours, but that could be longer or shorter — we'll know more once we've done the testing and have a better idea of how long things are likely to take.

What to expect

While we're making this move, we're taking the chance to upgrade our servers as well. This should mean that you see improved response times and reduced likelihood of failures. We're aware that we've had a number of outages recently, and while most of these self-recover fairly quickly with the automated processes we have in place, some have lasted longer. They've mostly been down to one cause: we're outgrowing the server we're on and running out of capacity. This upgrade should help solve that, as we're splitting out services across multiple servers.

What this upgrade doesn't do is introduce any code changes. That means no UI updates or behaviour changes. In terms of how your communities look and work, you should see no change at all once we're back out of read-only mode. We're holding the code at the same point while we migrate to make sure that the migration is as simple as possible: the more stuff we try to do at once, the more there is to go wrong. Once the migration is complete and working we've got a few changes in the pipeline waiting to go, mainly maintenance and display updates.

We'll update this post as we progress with this project. If you have any comments or questions, or you notice anything odd as we work on this, please let us know!

Update 2022-12-09

We've now done all the preliminary work necessary to move servers completely. With that done, the final steps are to copy the live data over to our new server and point all our URLs over there. Doing that means we need to go read-only (and potentially down completely for a short while) so we don't miss data. To that end:

Codidact will be read-only on 2022-12-11, UTC evening. We don't anticipate significant downtime as the work left to do is fairly minimal, but you may see some disruption.

We'll update here again once work has been finished.

Update 2022-12-11

For reasons unrelated to Codidact (work emergency), we've had to delay this move. We are now targeting 2022-12-18 (one week from now).

Update 2022-12-12

We caught a window in which we could make the move without much disruption, and this is now complete. More details about the new setup to come (this edit is being made by someone who doesn't know all the nitty-gritty).

It looks like this move once again reset the "last read" state for categories. We're sorry about that and since it's now happened twice, we'll take a deeper look. (We previously thought it was an isolated case.)

If you see anything else that doesn't look right, please let us know.

posted over 2 years ago

CC BY-SA 4.0

2y ago by Monica Cellio‭

ArtOfCode‭ staff

9 151 811 244