Deploying to dev broke production

I sent the team a message asking them to test a change on dev. The application had a new way to control which products each user could access. They could try it, check the result, and tell me what needed to change.

What I did not know was that I had pressed a killer switch in production and started a chain of disasters.

Only Dev

The application creates product quotes. Before this change, it used a shared catalog setting to control product access. A product had an is_standard_quotable flag, and some permission rules set per-user used that flag to decide who could quote it.

The new requirement was more specific. An administrator needed to select which users could access a product family. A user could also receive access to one product without access to its family.

What happened if those two settings were different? Either assignment was enough. Family access covered the products in that family. A direct product assignment granted access to that product alone.

Here, "No" meant no assignment, not an explicit denial. A product assignment could grant access when the family did not. But an empty product assignment list could not cancel family access.

That also affected removal. Removing a direct product assignment did not block a user who still had family access. To remove access, an administrator had to remove both applicable assignments. Products without a family could still receive direct user assignments.

These rules applied to non-Admin users. Members of the Admin group and Django superusers retained access to all products. Inactive products still could not be quoted, regardless of access.

So, the implementation added user assignments at both levels. It converted the existing settings into assignments, removed the old permission, and removed the product flag.

That last part will become important.

I deployed the application to dev for testing, sent an email to the team to test the new deployment, and went to rest. Later, Sentry sent an alert about a database query error.

I had an explanation for it. Since I had only deployed the application to dev, the error must come from dev. Perhaps the backend change had not fully taken effect yet. The code and database could be temporarily out of step while the deployment completed.

That was what I thought. I did not open the alert to confirm it. It was not simply that I saw a dev error and decided it did not matter. I had classified it as a temporary deployment error without checking the environment or the full error details.

The Second Alert

The same error arrived again. This time I knew it was worth looking into because Sentry would only send repeated alerts for the same error if and only if there was a rise in occurence.

But then why is a temporary deployment error still happening?

I opened Sentry. The environment was production. Damn...

The error was also quite specific. I have omitted the application's table prefix in the examples below:

ProgrammingError: column product.is_standard_quotable does not exist

It must have been due to some database thing, but then it hit me: An existing quote request was loading its related products. The generated SQL still selected is_standard_quotable. PostgreSQL could not return a column that no longer existed.

This was not the backend taking a little longer to update. The running application expected the old schema. The database already had the new one.

But then I had only deployed the application to dev. How did the production database get involved?

Two Separate Changes

Our GitHub Actions workflow did more than build and publish Docker images. After publishing an image, it ran the database migrations from that image. The branch determined which database it used: staging for dev, production for main.

Before I rested, I asked my agent to merge the change from dev into main just so that when it was time to promote the changes, I could do click ops instead of waiting for an image to be built.

The change had reached main, so the workflow built the production image and migrated the production database. The running production application still used the previous image.

Building an image and running that image were separate actions. Updating the database was attached to the first one.

That arrangement came from how we normally worked. The application had a small user base. Changes went through dev testing and then moved to production promptly. We had not usually left a change waiting between those stages.

This time, the delay was the problem.

The source version had been increased, and the new production image existed. What had not changed was the image used by the running production application. That distinction explained why the database could be ahead of the app.

When I checked the sequence of events, the order was clear. The migration completed before production errors began. My email asking the team to test dev followed those errors.

By the time I sent that email, production requests were already failing. Almost no one noticed because the query would only affect a specific part of the application dashboard. Most people worked on a different part.

Just Roll Back

I chose to roll back the database so that it would work with the application already running in production.

I had Django migrations. I knew which migration introduced the change. Could we not just reverse that migration and continue?

I inspected the migration to work through a plan for recovery. The inspection exposed another problem: reversing the schema changes would not restore the old access settings.

The migration had added the user assignment tables, converted access, deleted the old permission, and removed the old product field. Its data conversion used this operation:

migrations.RunPython(convert_access, migrations.RunPython.noop)

The forward function converted access. The reverse function did nothing. That is what RunPython.noop means. Django documents this behavior.

A normal reverse migration would recreate the old product field with its default value, False. However, in this case, a migration would not recover which products previously had True because migrations don't store data. In addition, it would also remove the new assignment tables without restoring the deleted permission and its grants.

The missing-column error could disappear while product access remained wrong.

So, there were two things to recover: the structure the old application expected, and the values that made its access rules work.

Can we restore from a backup?

Turns out I had a backup from before the migration. I had set up auto backup and retention for 7 days. Which meant if I could check carefully, I might just find a db snapshot of the state just before my migration. I also had a backup of the current state.

That sounded useful, but replacing the current database with the older backup would introduce another problem. Users had created more data since that backup. The current database contained quotes, revisions, and line items that the older copy did not have. I needed the old access settings. I did not need to remove the work done since the backup.

There was also a more immediate question. In the Azure portal, the available action was called Fast Restore. What exactly would it restore, and where?

I was concerned that selecting it would replace the current database. At first, I thought I could have either the current copy or the restored copy, but not both at once.

Happily, that was not how it worked: Fast Restore creates a separate PostgreSQL server from the selected backup. It does not replace the existing server. Azure's restore documentation describes the new-server step.

That changed what was possible. I could leave the current database in place and use the restored database to read the missing values.

I restored the backup and added its connection URL to the local environment as DATABASE_BACKUP_URL. The current database remained under DATABASE_URL.

Now I had both copies. I still had to establish what was safe to take from the older one.

Which Values?

The restored backup preceded the migration, but it was not taken immediately before the release. This meant a value in the backup was not automatically the correct value for the moment before a failure.

I used a private recovery script to inspect both databases. The inspection command had this form, with the local script path shortened:

# From the backend directory; RECOVERY_DIR contains the private recovery script.
uv run --frozen python "$RECOVERY_DIR/recovery.py" inspect

Its inspection mode used read-only database transactions. It saved the results as current-access.json and backup-access.json. Here, "current" means the production state before the rollback, not the database after recovery.

For example, these queries supplied the migration history and record counts for each database:

SELECT app, name FROM django_migrations ORDER BY app, name;
SELECT count(*) FROM quote;
SELECT count(*) FROM quoterevision;
SELECT count(*) FROM quotelineitem;

The saved comparison showed that the current database contained newer quotes, revisions, and line items.

Those counts did not capture possible edits to existing records. But they already showed that the older database could not replace the current one without losing newer records.

Next, we compared the inputs to the access conversion. Were the product IDs the same? Had products moved between families? Had users changed groups or status? Those checks matched. The migration histories differed by only the access migration we wanted to reverse.

We then checked a more specific question: would the old settings in the backup produce the assignments now present in production?

The Python below is a shortened reconstruction of those checks for this article. It reads the saved files, not either database. It accepts the snapshot directory as its command-line argument. Product rows contain the ID, active status, family ID, and old flag. User rows contain the ID, active status, and superuser status. The current product rows use None for the removed flag.

# Compare saved access snapshots. No database connection is required.
import json
import sys
from pathlib import Path

directory = Path(sys.argv[1])
current = json.loads((directory / "current-access.json").read_text())
backup = json.loads((directory / "backup-access.json").read_text())

def rows(snapshot, key):
    return {tuple(row) for row in snapshot[key]}

for key in ("families", "users", "groups", "memberships"):
    assert rows(current, key) == rows(backup, key), key
assert {tuple(p[:3]) for p in current["products"]} == {
    tuple(p[:3]) for p in backup["products"]
}
print("Access input comparisons: match")

print("Migrations only in current:", sorted(
    name for _app, name in rows(current, "migrations") - rows(backup, "migrations")
))
print("Migrations only in backup:", sorted(
    name for _app, name in rows(backup, "migrations") - rows(current, "migrations")
))

groups = dict(backup["groups"])
memberships = rows(backup, "memberships")
admins = {user for user, group in memberships if groups[group] == "admin"}
permission_groups = {group for group, in backup["permission_groups"]}
privileged = {user for user, in backup["permission_users"]}
privileged.update(
    user for user, group in memberships if group in permission_groups
)

active_by_family = {}
for product, active, family, enabled in backup["products"]:
    if active and family is not None:
        active_by_family.setdefault(family, []).append(enabled)
fully_enabled = {
    family for family, flags in active_by_family.items() if all(flags)
}

expected_families = set()
expected_products = set()
for user, _active, superuser in backup["users"]:
    if superuser or user in admins:
        continue
    for family, in backup["families"]:
        if user in privileged or family in fully_enabled:
            expected_families.add((family, user))
    for product, active, family, enabled in backup["products"]:
        if user in privileged:
            grant = family is None
        else:
            grant = active and enabled and family not in fully_enabled
        if grant:
            expected_products.add((product, user))

# Remove the application prefix from exported assignment table names.
assignments = {
    name.split("_", 1)[1]: values
    for name, values in current["assignments"].items()
}
actual_families = set(map(tuple, assignments["productfamily_allowed_users"]))
actual_products = set(map(tuple, assignments["product_allowed_users"]))
print("Family assignment differences:", len(expected_families ^ actual_families))
print("Product assignment differences:", len(expected_products ^ actual_products))

This follows the original conversion rules. Non-Admin users with the old permission received every family and direct access to products without a family. Other users received whole-family access only where every active product was enabled. Otherwise, they received direct access to the enabled active products. Empty families did not qualify as fully enabled. The conversion also retained assignments for inactive users; it did not enable their accounts.

The ^ operator finds entries present in only one set. It detects both missing and extra assignments. Equal counts alone would not establish that the same users had access to the same products.

Running this reconstruction against the saved inspection files gives:

Access input comparisons: match
Migrations only in current: ['0030_user_catalog_access']
Migrations only in backup: []
Family assignment differences: 0
Product assignment differences: 0

The zero assignment differences also match the report saved during the recovery. All family assignments and direct product assignments matched the backup conversion.

This gave us evidence that the backup was a suitable source for this recovery. It did not prove that nothing had changed anywhere in the application since the backup was taken.

The recovery could therefore stay narrow. Reverse the one migration. Restore the old product flags. Recreate the old permission and its group assignments. Leave the current quotes and other records in place.

There was no need to copy whole backup tables over production tables.

Try It Locally

Before changing production, we took a full logical dump of the current database and restored the application schema into a temporary local PostgreSQL instance.

The first rehearsal reversed the migration, restored the settings, and then rolled back the entire transaction. We checked that this returned the local database to its starting state.

The next rehearsal committed the recovery. We then checked it with the previous application version, not just the new code in the working directory. That was the version we needed to support.

It passed the Django system checks and had no pending migrations for that version. The catalog and user access checks also matched the old policy.

There was another question to answer: had the recovery changed anything it was supposed to leave alone?

Record counts were useful, but they were not enough. A table can contain the same number of rows while its values change. We compared content hashes for the retained application tables, excluding the fields and records that the recovery was meant to change.

The rehearsal passed those checks too.

The Live Recovery

I stopped writes before the live recovery. We took a fresh backup of the current database and checked that the full backup archive could be read.

The recovery script then ran the changes in one database transaction. It blocked competing writes to the application tables and checked that the access state still matched the state we had inspected.

It also checked the migration plan. Only the intended migration could run backward.

Then it reversed that migration, restored the product flags, recreated the old permission and its group grants, and checked the retained records before committing. A failure in those checks would roll back the transaction.

The transaction committed successfully.

All quotes, revisions, and line items remained. Independent checks confirmed the old catalog behavior.

Those were the database checks. I still needed to check the running application. I restarted it and opened the dashboard. It worked. I checked the error again. It was no longer appearing, and Sentry was no longer receiving that error during my checks.

The recovery was complete.

What Must Change?

The immediate problem was fixed, but I do not want another release to depend on the same assumption.

First, I want migrations to remain compatible with the previous application version wherever possible. In this case, removing a field that the old application still queried made the migration incompatible.

A safer change would keep the old field while introducing the new access tables. The old permission and its assignments would also need to remain available. Removing them would be a later step, after no running application version needed them and the rollback requirements were clear.

For access rules, keeping a column is only part of the work. Its values must still make sense to any version allowed to use it. If old and new versions can both write access settings, the transition also needs rules for keeping those settings consistent.

Second, I need to distinguish backward compatibility from reversibility. Can the old application continue to work after the migration? That is one question. Can I reverse the migration and recover the required state? That is another.

In this incident, Django could restore the old field's structure. The backup supplied its missing values. Future changes that remove data need a recovery plan before they run, not after an alert arrives.

Third, I no longer want the image release workflow to run database migrations automatically. I should be able to build and publish a production image without changing the database used by the current application.

The migration should be a separate, controlled step coordinated with deployment. A manual step alone would not make an incompatible migration safe. It would, however, stop image creation from changing production before I intended to deploy the application.

And there is one smaller change that does not need a new workflow: just open the stupid alert and check it out, you installed it for a reason

My explanation for the first error was plausible. The problem was that I treated my setup with overfamiliarity.