Alembic/Flask-migrate doesn't recognise database structure

Question

I have a Flask-based application and I'm trying to enable migrations to be able to manage database structural changes. My database is not empty and contains tables and records before I started with alembic and flask-migrate.

First, I tried to use alembic stand-alone, just pip install alembic and then configure everything and use autogenerate for generating the diff. All works fine but the output was not exactly what I expected:

INFO  [alembic.autogenerate.compare] Detected removed table u'user'
  Generating /Users/me/Documents/Project/app/migrations/versions/3de1151ce5de_.py ... done

So the table user actually is present in my database (I think it's a reserved table in postgresql) and I have never removed it. Also, I have a bunch of other tables which seem to be invisible to alembic. At first I thought I failed to configure alembic properly, so I started over and decided to try this nice flask extension by Miguel Grinberg. This was a lot easier to setup, but the result was exactly the same message.

Any ideas what am I doing wrong?

EDIT: I decided to add some important information on how I edited my original alembic env.py:

....
import sys,os sys.path.append(os.getcwd())
import app from app import db
target_metadata = db.metadata

....

def run_migrations_online():
    """Run migrations in 'online' mode.

    In this scenario we need to create an Engine
    and associate a connection with the context.

    """
    # all the models need to be imported so they are visible when migration runs
    from app.models import Model1, Model2, Model3, Model4, Model5

...

This changes were necessary for the alembic to see my declarative base and the models before any migration starts. After dropping the standalone approach I just followed the flask-migrate tutorial and didn't touch any files there.

EDIT: Ok, so we've got the deleted 'user' table explained. Still, my other models don't appear in the scripts. I think what happens is since the app is started before the migration scripts are created, the sqlalechmy db.create_all() call creates the tables and this is why alembic doesn't see any difference in the colums vs models. Perhaps I'm calling create_all() in the wrong place?

EDIT: Yes! That was it, I moved db.create_all() below the manager.run() and it detects the changes properly now. Thanks for all the help!

Answer 1

Your database has a user table, which is not represented in any of your SQLAlchemy models. Correct?

Alembic will then assume that you have removed the table, since all the auto-migration script does is show the differences between the models defined in your application and the actual database. Alembic has no way to know that you have a table in your database that is to be considered exempt from migration scripts.

In any case, the automatic migration feature is great, but it is not supposed to be fully "automatic". You always have to review the generated scripts and fix any errors or inaccuracies before you use them.

The way to move forward is to edit the migration script to correct this. Just take that remove table away. Migration scripts are meant to be hand edited, don't be afraid to do that.

The unfortunate problem is that every time you issue a new migration that table remove will show up again. I don't think Alembic has a way to provide exceptions.