DEV Community

FROWNINGdev
FROWNINGdev

Posted on Edited on Originally published at frowningdev.github.io

My Django migration linter raised 407 alarms on six big codebases. 405 were wrong

I maintain orm-lens (until recently django-orm-lens), a static analyzer for Django models and migrations. One of its checks, migration-risk, reads every migration file and flags operations that break on a table that already has rows: a NOT NULL column with no default, a UNIQUE column that existing duplicates will violate, an index build that locks the table.

Its test suite was green, so I pointed it at real projects:

Project GitHub stars Critical findings
NetBox 21.8k 212
PostHog 40.5k 79
authentik 26.2k 58
Healthchecks 10.5k 27
Plane 61.7k 25
paperless-ngx 47.4k 6

That is 407 critical findings. I read every one, and two were real.

The other 405 were not edge cases. Each came from something Django does when it migrates a populated table, which I had modelled wrongly. Here they are, ordered by how many false alarms each caused.

1. Squashed migrations never touch a populated table (208)

class Migration(migrations.Migration):
    replaces = [("dcim", "0002_auto_..."), ..., ("dcim", "0059_...")]
Enter fullscreen mode Exit fullscreen mode

A squashed migration only runs on a fresh database. On a database where the replaced migrations are already applied, Django marks the squash as applied without running it. On a database where only some of them are applied, it keeps running the originals. Either way, every AddField inside a squash lands on an empty table. Squashes alone made up more than half of all the alarms.

2. blank=True text fields get "" (83)

migrations.AddField("virtualmachine", "serial",
                    models.CharField(blank=True, max_length=50))
Enter fullscreen mode Exit fullscreen mode

No default= and no null=True, so the column is NOT NULL, yet this is safe. BaseDatabaseSchemaEditor._effective_default() fills existing rows with "" for any blank=True field whose class allows empty strings. Projects also subclass CharField: NetBox's ColorField(blank=True, max_length=6) gets the same treatment. The linter now treats a max_length argument as a sign of a text field.

3. Many-to-many fields add no column (37)

ManyToManyField, django-taggit's TaggableManager and GenericRelation create a join table, or nothing at all. They are never a NOT NULL risk on the model's own table.

4. makemigrations splits foreign keys out of CreateModel (25)

migrations.CreateModel(name="ContactGroupMembership", fields=[...]),
...
migrations.AddField("contactgroupmembership", "group",
                    models.ForeignKey(..., on_delete=models.CASCADE)),
Enter fullscreen mode Exit fullscreen mode

To break dependency cycles, the autodetector creates the table first and adds some foreign keys afterwards, in the same migration. That table was created a few lines earlier, so it is empty.

5. A squash and the migrations it replaces are not two leaves (17)

Django refuses to migrate an app whose migration graph has two leaf nodes. The check flagged authentik's apps as conflicting because the squash and the last migration it replaces both looked like leaves. The loader drops the replaced nodes and points their dependents at the squash, so the linter now does the same before it counts leaves.

6. auto_now, db_default and GeneratedField fill themselves (16)

auto_now_add=True gets the current time on ADD COLUMN, and db_default is a database-side default. A GeneratedField is computed by the database. None of them needs default=.

7. NULLs don't collide (12)

models.CharField(blank=True, max_length=128, null=True, unique=True)
Enter fullscreen mode Exit fullscreen mode

Existing rows get NULL. On PostgreSQL, MySQL and SQLite a unique constraint never treats two NULLs as equal, so this is safe. The same holds for default=None, which authentik uses on four fields across three migrations.

8. Names that come back (5)

A RemoveField followed by an AddField of the same name, or a RenameField onto it, leaves the field in place. So does a DeleteModel followed by a CreateModel. The linter compared each historical removal with today's models.py, saw the field still there, and called it drift.

9. Data migrations that run first (2)

authentik adds a NOT NULL column right after a RunPython that, its comment says, deletes every row. A static tool cannot prove what a RunPython does, so this is now a warning that says why, rather than a certain critical.

The two that were real

PostHog's 0308_add_indirect_person_override_constraints removes two foreign-key columns and adds them back as NOT NULL with no default:

migrations.RemoveField(model_name="personoverride", name="old_person_id"),
migrations.AddField(model_name="personoverride", name="old_person_id",
                    field=models.ForeignKey(on_delete=..., to=...)),
Enter fullscreen mode Exit fullscreen mode

On a populated personoverride table this fails. The table may well have been empty when it shipped. The linter cannot know that, and it does not pretend to.

The same exercise for N+1

nplusone had its own share of noise: writes through a relation (interface.vdcs.set(...)) reported as reads, one field suggested in both select_related and prefetch_related, and loops in test files. After those fixes I drew a random sample of the remaining high-confidence findings across the six projects, and all six were real N+1s. One example is installation.organization read inside a loop over installations, which is fixed with select_related("organization").

What I took from it

A static analyzer is only as good as its precision on code it has never seen. Synthetic test fixtures confirmed that the rules did what I wrote. They could not tell me whether what I wrote was true of Django. Big open-source codebases can, and running against them is now part of how the rules are tested. Each of the nine causes above is now its own test case, so a later refactor that forgets how Django migrates a populated table fails the suite instead of shipping.

The other rule is the one from cause 9: severity follows certainty. When the linter cannot prove a risk, because a RunPython it cannot see through runs first, it says "warning, and here is why I can't be sure" instead of "critical". A check that is wrong 405 times out of 407 teaches people to skim the whole report, and then the two real findings get skimmed too.

All of the above shipped in py-1.14.0. Try it on your own history:

uvx orm-lens migration-risk      # migration findings, critical by default
uvx orm-lens nplusone            # N+1 loops, with the select_related to add
Enter fullscreen mode Exit fullscreen mode

It reads source only: no database, no settings module, no django.setup(). It is free and MIT. If it is wrong about your project, open an issue. That is how this whole list came about.


Updated 10 November 2026 after a comment below. Originally published on the orm-lens site.

Top comments (6)

Collapse
 
launchgatecheck profile image
Launch Gate •

The number that makes this post credible isn't the 2 real findings - it's that you read all 407. A checker with a 99.5% false-positive rate trains its users to dismiss it, and an ignored check is worse than none, because it teaches the whole report to be skimmed. The fixes all move in the right direction, but #9 is the one worth generalizing: when the tool can't prove the thing (a RunPython it can't see through), severity should track certainty. "Warning, and here is why I can't be sure" keeps the finding readable; a certain-critical it can't defend is what got the other 405 dismissed.

One suggestion: keep the corpus. Six codebases with per-finding verdicts is a labeled regression suite most linters never get. Each false-positive class - squashes, blank=True defaults, M2M, NULL uniqueness - becomes a fixture, so the next refactor has to prove it still reads Django the way Django actually behaves. Synthetic tests taught you the tool was green; labeled reality taught it the framework.

Collapse
 
frowningdev profile image
FROWNINGdev •

Agreed on both. "Severity follows certainty" says what #9 turned into better than I did, so I added it to the post as a rule rather than a one-off fix.

On the corpus: the nine false-alarm classes are already cut down into test cases in test_migration_risk.py (squashes, blank=True, M2M, NULL uniqueness and the rest), so each one has to keep reading Django the way Django actually behaves. The full six-codebase run with a verdict per finding isn't checked in as a labelled set yet. That's a good next step, since it's the only thing that would catch a tenth class I haven't seen.

Collapse
 
launchgatecheck profile image
Launch Gate •

The tenth class is exactly the right reason to keep it - a labeled corpus only earns its keep if it's open-ended, and the class you haven't seen is the one the synthetic tests are guaranteed to miss.

The corpus has a second use past regression, too: a checker that publishes its own false-positive history is easier to trust than one that publishes a detection count. "Here are the nine ways I've been wrong, and the fixtures proving I'm not wrong that way anymore" is a stronger credibility document than any accuracy number, because the reader can verify it against their own migrations.

Thread Thread
 
frowningdev profile image
FROWNINGdev •

yeah fair point, I was only thinking of it as test fixtures, not as something to actually show people. but you're right, "here's every way it was wrong + the fixture that pins it" sells it way better than me saying it's 99% accurate. opened an issue for it (#239 on the repo), the plan is to keep the six-project run in there with a verdict per finding, so a tenth class shows up as a diff instead of me finding out from someone's angry comment :)

Thread Thread
 
launchgatecheck profile image
Launch Gate •

"a diff instead of someone's angry comment" is the whole argument in one line - every monitoring system eventually comes down to whether you'd rather hear it from your own tests or from a user.

One bonus you get for free once it's in the repo: the corpus doubles as a contribution guide. "New rule PRs name the class they'd have caught, or add a fixture showing a new one" is a bar reviewers can actually apply, and it keeps the ninth-class problem from coming back through someone else's well-meaning rule.

Thread Thread
 
frowningdev profile image
FROWNINGdev •

ha, that's a good one, stealing it. "name the class your rule would've caught or add a fixture for a new one" is way easier to review against than "looks reasonable to me". dropped it into #239 so it doesn't get lost, it'll go into CONTRIBUTING once the corpus is in