Incident postmortem: 25 June 2026

Incident

At around 03:22 UTC on 25 June 2026, the Rails server which runs the QPixel app that powers our communities began returning an unhandled low-level error in response to HTTP requests, which prevented access to Codidact communities. Initial attempts to connect to the server to resolve the error failed. A sysadmin came online at around 07:15 UTC; the immediate cause was quickly resolved and Codidact communities came back online at 07:26 UTC, with full resolution completed at 08:08 UTC.

Root cause

The immediate cause of this outage was that the QPixel server and associated services were unable to run due to a lack of disk space on the server. There were two underlying causal factors, which were:

(a) an excess of log files were using more disk space than had been accounted for;
(b) some old database backups were not deleted after creation and upload by the automated backup process.

(a) Log files

There are innumerable log files present on any computer system, but in this particular case the files of concern were the system journal logs, and the QPixel server logs.

The system journal log is a running log kept by the journald service, which keeps logs for systemd. systemd is the initialisation system for the operating system the server runs on and is responsible for system and service management. This includes core operating system services as well as services for installed applications and user-defined services, including the service which manages the QPixel server. As a result, systemd produces a significant amount of output, which is kept in log files by journald. By default, journald is configured to keep no more than 4 GB of log data. No additional configuration was applied to journald, so the system journal log files occupied roughly 4 GB at the time of the incident.

The QPixel server separately keeps its own logs, which record request data, information, warnings, and error details. A standard log management tool, logrotate, was configured to manage these log files, which was set to start a new log file each day, compress the old log file, and keep only the most recent 7 days' logs. This is the same configuration which was initially set several years ago and has not changed since. This was sufficient at the time, but because this log scales with the amount of web traffic the server receives and as there has been a significant increase in traffic over that period, these log files occupied roughly 3.5 GB at the time of the incident.

(b) Database backups

There is an automated process in place which takes a daily backup of the QPixel database and saves it to long-term, off-server storage for resilience and recovery purposes. This process is currently managed automatically by Amazon Relational Database Service (RDS), where our database is held. Previously, this process was managed by a scheduled job on the web server running a script to achieve the same outcome.

It became apparent during the investigation that when this process was changed, the previous approach was not fully cleaned up. The script was still present on the server and was still scheduled to run daily, but because the storage location had changed, the script was failing to upload the backups it created. This resulted in some backups not being deleted automatically. This was not the case every time the script ran, although it was not possible to determine why. At the time of the incident, there were four old backup files present occupying around 2.5 GB.

Response & Remediation

The incident was initially responded to almost immediately by two of our team who were online at the time - one member of our community team and one member of our sysadmin team. Initial attempts to connect to the server to investigate the incident failed, which was likely due to the lack of disk space, although a later attempt succeeded. An initial review of relevant logs quickly identified the issue, and some further investigation was able to narrow down the files concerned to the journal log and QPixel server log.

Due to the nature of how the Foundation operates, the community manager was the only person at the time who had access to the production server, and was not in a position to start remediation. The sysadmin available at the time did not have access to the production server, so although they were able to provide advice on investigation which helped to investigate the cause, they were also not able to start remediation.

An additional sysadmin came online at around 07:15 UTC, and was able to access the production server. The majority of the investigation work had already been completed at this point and the direct cause of the incident was known, so this sysadmin was quickly able to remove some unnecessary log files, freeing up enough space to restart the QPixel server and associated services, and bringing Codidact communities back online at 07:26 UTC.

Remediation work continued for a further 42 minutes, including cleaning up other unnecessary files and reconfiguring log management services, and the incident was finally resolved at 08:08 UTC.

Learning & Next Steps

Several learning points were identified from this incident, including:

(a) organisational bus factor;
(b) old configurations were not reviewed for an extended period of time;
(c) an old process was not fully cleaned up after changes

(a) bus factor

In this case, the relevant bus factor was “people who both have access to the production server and the necessary knowledge to take remedial actions”. This was a single point of failure.

This has been identified before as a risk following previous incidents and has caused extended downtime before. Unfortunately, due to the volunteer, non-profit nature of the Foundation, our resources are not vast and this is likely to continue to be a risk, although the Foundation is exploring options to increase this bus factor. As a result, mitigations against downtime through other actions are especially important.

(b) old configurations were not reviewed

This is related to a similar risk that has been identified following another recent incident involving the loss of our development server and services hosted there. This has highlighted that we have a lack of a standardised server setup, image, or configuration set, which has resulted in each new server being set up independently, inevitably resulting in some differences and oversights between setups.

As a result, we have now set up a private repository under our Codeberg organisation to hold standardised and server-specific configurations. Configuration files will be shared between servers where possible, and recorded separately if not. This will result in better consistency between setups, and better access to and reviewability of configurations over time.

(c) an old process was not fully cleaned up

Changes of the scale that changed our automated backup process are fortunately rare, and as such it is difficult to put in place a single standard process to manage them. This incident has, however, highlighted the importance of forward planning for significant changes. It is likely that future changes of this scale will involve a detailed infrastructure or system administration plan before starting to help ensure that each part of the process is completed and cleaned up in full.

The process involved in this incident has now been cleaned up and stopped from running, which will resolve this particular issue of old database backups being kept.