docs: add db migration troubleshooting section
Add some documentation around migration failures.
@@ -131,3 +131,40 @@ be able to use Determined as long as you use Docker images based on CUDA
docker run --gpus all --rm nvidia/cuda:9.0-runtime nvidia-smi
docker run --gpus all --rm nvidia/cuda:10.0-runtime nvidia-smi
+
+*****************************
+ Database Migration Failures
+*****************************
+
+.. code::
+
+ Dirty database version <a long number>. Fix and force version.
+
+If you see the above error message, a database migration was likely
+interrupted while running and the database is now in a dirty state.
+
+Make sure you back up the database and temporarily shut down the master
+before proceeding further.
+
+To fix this error message, locate the up migration with a suffix of
+``.up.sql`` and a prefix matching the long number in the error message
+in `this directory
+<https://github.com/determined-ai/determined/tree/master/master/static/migrations>_`
+and carefully run the SQL within the file manually against the database
+used by Determined. For convenience, all the information needed to
+connect except the password can be found with:
+
+.. code::
+
+ det master config | jq .db
+
+If this proceeds successfully, then mark the migration as successful by
+running the following SQL:
+
+.. code::
+
+ UPDATE schema_migrations SET dirty = false;
+
+And restart the master. Otherwise, please seek assistance in the
+community `Slack
+<https://join.slack.com/t/determined-community/shared_invite/zt-cnj7802v-KcVbaUrIzQOwmkmY7gP0Ew>`__.
