Skip to main content
Version: 0.5

Model migrations

Migrations ensure that any changes made to model definitions are applied at the database level. Marten's migrations mechanism is designed to be mostly automatic: migrations are generated from your model definitions when you run a dedicated command, after having introduced some changes to your model definitions.

Overview​

Marten is able to automatically create migrations when you introduce changes to your models.

For example, let's assume you just added a hometown string field to an existing Author model:

class Author < Marten::Model
  field :id, :big_int, primary_key: true, auto: true
  field :first_name, :string, max_size: 255
  field :last_name, :string, max_size: 255
  field :hometown, :string, max_size: 255, blank: true, null: true
end

After introducing this change, you can run the genmigrations management command as follows:

$ marten genmigrations
Generating migrations for app 'blog':
  › Creating [src/blog/migrations/202206221856241_add_hometown_to_blog_author_table.cr]... DONE
      ○ Add hometown to blog_author table

When running this command, the table definition corresponding to your current model will be analyzed and compared to the equivalent table that is defined by your migration files. Depending on the result of this analysis, a new set of migration files will be generated in order to account for the changes that were made to the models.

Generated migrations are saved under a migrations folder that lives in the local structure of every application. As a general rule of thumb, you should always look at what is actually generated by the genmigrations command. Indeed, the introspection capabilities of this command could be limited depending on what you are trying to achieve.

In the above example, the generated migration file looks something like this:

# Generated by Marten 0.1.0 on 2022-06-22 18:56:24 -04:00

class Migration::Blog::V202206221856241 < Marten::Migration
  depends_on :blog, "202205290942181_previous_migration_name"

  def plan
    add_column :blog_author, :hometown, :string, max_size: 255, null: true
  end
end

As you see, the migration explicitly depends on the previous migration for the considered application (blog in this case) and it defines a migration "plan" that involves adding a hometown string column to the blog_author table.

We can then apply this migration to the database by running the migrate Marten command:

$ marten migrate
Running migrations:
  › Applying blog_202206221856241_add_hometown_to_blog_author_table... DONE

Available commands​

Marten provides a set of four commands allowing interacting with model migrations:

  • genmigrations allows generating migration files by looking for changes in your model definitions
  • migrate allows applying (or unapplying) migrations to your databases
  • listmigrations allows listing all the migrations for all your installed applications, and whether they have already been applied or not
  • resetmigrations allows resetting multiple migrations into a single one

DB connections specificities​

Migrations can be used with all the built-in database backends provided by Marten: PostgreSQL, MySQL, and SQlite. That being said, there are some key differences among those backends that you should be aware of when it comes to migrations. These differences stem from the fact that not all of these databases support schema alteration operations or DDL (Data Definition Language) transactions.

PostgreSQL and SQLite support DDL transactions, but MySQL does not support them. As such, if a migration fails when being applied to a MySQL database, you might have to manually undo some of the operations in order to try again.

Finally, it should be noted that SQLite does not support most schema alteration operations. Because of this, Marten will perform the following set of operations when applying model changes to a SQLite database:

  1. create a new table corresponding to the new model definition
  2. copy the data from the old table to the new one
  3. delete the old table
  4. rename the new one so that it matches the model's table name

Migration files​

As presented in the overview section above, migration files are automatically generated by Marten by identifying changes in your model definitions (although migrations could be created and defined manually if needed). These files are persisted in a migrations folder inside each application's main directory.

Migrations always inherit from the Marten::Migration base class. A basic migration will look something like this:

# Generated by Marten 0.1.0 on 2022-03-13 16:08:37 -04:00

class Migration::Main::V202203131608371 < Marten::Migration
  depends_on :users, "202203131607261_create_users_user_table"
  depends_on :main, "202203131604261_add_content_to_main_article_table"

  def plan
    add_column :main_article, :author_id, :reference, to_table: :users_user, to_column: :id, null: true
  end
end

Each migration must define the following mandatory information: the migration's dependencies and the migration's operations.

Dependencies​

Marten migrations can depend upon one another. As such, each migration generally defines one or many dependencies through the use of the #depends_on class method, which takes an application label and a migration name as positional arguments.

Migration dependencies are used to ensure that changes to a database are applied in the right order. For example, the previous migration is part of the main app and depends on two other migrations: first, it depends on the previous migration of the main app (202203131604261_add_content_to_main_article_table). Secondly, it depends on the 202203131607261_create_users_user_table migration of the users app. This makes sense considering that the migration's only operation is adding an author_id foreign key targetting the users_user table to the main_article table: the dependency instructs Marten to first apply the 202203131607261_create_users_user_table migration (which creates the users_user table) before applying the migration adding the new foreign key (since this foreign key requires the targetted table to exist first in order to be created properly).

Again, migration dependencies are automatically identified and generated by Marten when you create migrations through the use of the genmigrations command. These dependencies are identified by looking at the DB relationships between the application for which migrations are generated for and the other installed applications.

Operations​

Migrations must define operations to apply (or unapply) at the database level. Unless instructed otherwise, these operations are all executed within a single transaction for the backends that support DDL transactions (PostgreSQL and SQLite).

When they are generated automatically, migrations will define a set of operations to execute as part of a #plan method. This method can define one or more operations. For example:

# Generated by Marten 0.1.0 on 2022-03-30 22:13:06 -04:00

class Migration::Press::V202203302213061 < Marten::Migration
  depends_on :press, "202203111822091_initial"

  def plan
    add_column :press_article, :rating, :int, null: true
    remove_column :press_article, :old_status
  end
end

Migrations can be applied and unapplied. This means that when a migration is applied, all the operations defined in the #plan method will be executed in the order they were defined. But if the migration is unapplied, the exact same operation will be "reversed" in reverse order.

In the previous example, the "plan" of the migration involves two operations: first, we are adding a new rating column to the press_article table and then we are removing an old_status column from the same table. If we were applying this migration, these two operations would be executed in this order. But if we were unapplying the migration, this means that we would first re-create the old_status column in the press_article table, and then we would remove the rating column from the press_article table.

The bidirectional aspect of the #plan method can be leveraged for most migration use cases, but there could be situations where the operations involved when applying the migration differ from the operations involved when unapplying the migration. In such situations, forward operations can be defined in a #forward method while backward operations can be defined in a #backward method:

# Generated by Marten 0.1.0 on 2022-03-30 22:13:06 -04:00

class Migration::Press::V202203302213061 < Marten::Migration
  depends_on :press, "202203111822091_initial"

  def forward
    add_column :press_article, :rating, :int, null: true
    remove_column :press_article, :old_status
  end

  def backward
    # do something else
  end
end

Generating migrations​

Generating migrations is possible through the use of the genmigrations management command. This command will scan the table definition corresponding to your current models and will compare it to the equivalent table that is defined by your migration files. Based on the result of this analysis, a new set of migrations will be created and persisted in your applications' migrations folders.

info

The genmigrations command can only create migrations for installed applications. If the command is not detecting the intended changes, make sure that your models are part of an installed application.

Running marten genmigrations will generate migrations for all of the models provided by your installed applications, but it is possible to restrict the generation to a specific application label by specifying an additional argument as follows:

$ marten genmigrations my_app

Another usefull command option is the --empty one, which allows generating an empty migration file (without any defined operations):

$ marten genmigrations my_app --empty

Applying and unapplying migrations​

As mentioned previously, migrations can be applied by running the migrate management command.

Running marten migrate will execute non-applied migrations for your installed applications. That being said, it is possible to ensure that only the migrations of a specific application are applied by specifying an additional argument as follows:

$ marten migrate my_app

To unapply certain migrations (or to apply some of them up to a certain version only), it is possible to specify another argument corresponding to the version of a targetted migration. For example, we could unapply all the migrations after the 202203111821451 version for the my_app application by running:

$ marten migrate my_app 202203111821451

If you wish to unapply all the migrations of a specific application, you can do so by targeting the zero version:

$ marten migrate my_app zero

Finally, it should be noted that it is possible to "fake" the fact that migrations are applied or unapplied by using the --fake command option. When doing so, only the fact that the migration was applied (or unapplied) will be registered, and no migration operations will be executed:

$ marten migrate my_app 202203111821451 --fake

Transactions​

The operations of a migrations will be executed inside a single transaction by default unless this capability is not supported by the database backend (which is the case for MySQL).

It is possible to disable this default behavior by using the #atomic method, in the migration class:

# Generated by Marten 0.1.0 on 2022-03-30 22:13:06 -04:00

class Migration::Press::V202203302213061 < Marten::Migration
  depends_on :press, "202203111822091_initial"

  atomic false

  def plan
    add_column :press_article, :rating, :int, null: true
    remove_column :press_article, :old_status
  end
end

Data migrations​

Sometimes, it is necessary to write migrations that don't change the database schema but that actually write data to the database. This is often the case when backfilling column values for example. In order to do so, Marten provides the ability to run arbitrary code as part of migrations through the use of a special #run_code operation.

For example, the following migration will run the #run_forward_code method when applying the migration, and it will run the #run_backward_code method when unapplying it:

# Generated by Marten 0.1.0 on 2022-03-30 22:13:06 -04:00

class Migration::Press::V202203302213061 < Marten::Migration
  depends_on :press, "202203111822091_initial"

  def plan
    run_code :run_forward_code, :run_backward_code
  end

  def run_forward_code
    # do something
  end

  def run_backward_code
    # do something
  end
end

#run_code can be called with a single argument if you don't want to specify a "backward" method. The following migration is technically equivalent to the one in the previous example:

# Generated by Marten 0.1.0 on 2022-03-30 22:13:06 -04:00

class Migration::Press::V202203302213061 < Marten::Migration
  depends_on :press, "202203111822091_initial"

  def forward
    run_code :run_forward_code
  end

  def backward
    run_code :run_backward_code
  end

  def run_forward_code
    # do something
  end

  def run_backward_code
    # do something
  end
end
info

Data migration logics can't be defined as part of the #forward and #backward methods directly. Indeed, the #plan, #forward, and #backard methods don't actually do anything to the database: they simply build a plan of operations that Marten will use when applying or unapplying the migrations. That's why it is necessary to use the #run_code operation when defining data migrations.

It should be noted that one convenient way to start writing a data migration is to generate an empty migration skeleton for the targetted application:

$ marten genmigrations my_app --empty

Executing custom SQL statements​

Sometimes the built-in migration operations that are provided by Marten might not be enough and you may want to run arbitrary SQL statements on the database as part of migrations.

To do so, you can leverage the #execute migration operation. This operation must be called with a forward statement string as first positional argument, and it can also take a second positional (and optional) argument in order to specify the statement to execute when unapplying the migration (which is only useful when defining a bidirectional operation inside the #plan method).

For example:

# Generated by Marten 0.1.0 on 2022-03-30 22:13:06 -04:00

class Migration::Press::V202203302213061 < Marten::Migration
  depends_on :press, "202203111822091_initial"

  def plan
    execute("CREATE EXTENSION 'uuid-ossp';")
  end
end

Resetting migrations​

When your application starts having a certain amount of migrations, it can become interesting to reset them. This means reducing all the existing migrations to a unique migration file. This can be achieved through the use of the resetmigrations management command.

For example, let's consider the following situation where we have three migrations under the my_app application:

$ marten listmigrations
[my_app]
  [✔] my_app_202203111821451_create_my_app_article_table
  [✔] my_app_202203111822091_add_status_to_my_app_article_table
  [✔] my_app_202204051755101_add_rating_to_my_app_article_table

Running the resetmigrations command will produce the following output:

$ marten resetmigrations my_app
Generating migrations for app 'my_app':
  › Creating [src/my_app/migrations/202206261646061_auto.cr]... DONE
      ○ Create my_app_article table

If we look at the generated migration, you will notice that it includes multiple calls to the #replaces class method:

# Generated by Marten 0.1.0 on 2022-06-26 16:46:06 -04:00

class Migration::MyApp::V202206261646061 < Marten::Migration
  replaces :my_app, "202203111821451_create_my_app_article_table"
  replaces :my_app, "202203111822091_add_status_to_my_app_article_table"
  replaces :my_app, "202204051755101_add_rating_to_my_app_article_table"

  def plan
    create_table :my_app_article do
      column :id, :big_int, primary_key: true, auto: true
      column :title, :string, max_size: 155
      column :body, :text
      column :status, :string, max_size: 64, null: true
      column :rating, :int, null: true
    end
  end
end

These #replaces method calls indicate the previous migrations that the current migration is replacing. This new migration can be committed to your project's repository. Then it can be applied like any other migration (even if the underlying database was already up to date with the latest migrations). The following migrations will use it as a dependency and will disregard all the migrations that were replaced.

Later on, you can decide to remove the old migrations (the ones that were replaced by the new migration). But obviously, you should only do so after a certain while in order to give a chance to developers to apply all the latest migrations.

warning

The resetmigrations management command will not carry on run_code operations nor any other "manually" added operations. As a matter of fact, resetmigrations will simply look at your model definitions and try to recreate a new migration file from the beginning (without considering your old migration files). As such, if your database requires special run_code or run_sql operations, you should make sure that those are added as following migrations.