One of the biggest challenges with blue-green deployments is migrating the database. While it’s easy to run two different versions of your software at the same time, you can only run one database (i.e. schema) at a time. For example, if you change a table to use a different column name, and then try to run both versions of your software at the same time (as you’d want to with a blue-green deployment) then the old software will break because it doesn’t know about the new column name.
This situation presents a large but not insurmountable challenge. We can resolve this challenge by following some basic principles.
- The most basic principle is that each version N of the software MUST work with both version N and version N+1 of the database.
- Version N and version N+1 of the software must both be able to safely run at the same time on version N+1 of the database.
- Additionally, migrating the database and migrating the code are independent. The database usually migrates first because version N+1 of the software is not required to work with version N of the database.
This diagram illustrates the directions in which compatibility must be maintained between the code and the database.
The consequence of these conditions is that we can do any database refactoring that we like with zero downtime, the refactorings just need to be broken down into smaller refactorings and spread across multiple deployments. Also the individual refactorings don’t need to be in consecutive deployments, they can be spread across more deployments as long as they are done in order.
One complication is that this approach becomes very difficult if your releases are very far apart. Imagine a release schedule of once every three months, and imagine a database refactoring spread over three releases… It becomes increasingly likely that things will get forgotten or de-prioritized, and you will end up with a lot of half-finished database refactorings in your system.
The closer together your refactorings are in time, they easier they are to manage. In fact, if you are practicing continuous deployment, the database cleanup steps in subsequent deployments can happen almost immediately after the release once you are confident with the release and the old code has been retired. Blue Green deployment operates in a feedback loop: it enables you to do smaller and more frequent deployments, and it works much better and is much easier if you do smaller and more frequent deployments.
In the table below we see some sample refactorings and how they can be reduced to a set of compatible changes. Note that each column represents changes you would make for an individual release N, and the refactorings (rows) are split by database change and software change. Imagine overlaying the diagram above onto the table below and seeing in which directions compatibility needs to be maintained. Finally, sets of database changes for a single release should be done in a single transaction of course and the changes should be atomic.
This should always be tested against production-scale data and while the database is in use in a perf environment. You can see how long the schema migration takes, the performance impact of the migration to a system under load, and find solutions to these impacts if necessary.
We’ve outlined here just one technical component of zero downtime deployments. There are many other important non-technical components of zero downtime deployments, including considering the culture of your organization, the maturity of your devops practice, and the release cadence expected by customers or mandated by your industry. There are even other possible technical components such as how to migrate non-database data stores, collaborating service migration, and code that depends on data not on schema.
However, with the basic principles above, I am confident that one of the more difficult components of zero downtime deployment (database migrations) can be solved, and that the other components that apply to your own situation can similarly be solved if you are willing to do the work.