# GitLab database incident

### For your information:

* [Gitlab Database incident](https://docs.google.com/document/d/1GCK53YDcBWQveod9kfzW-VCxIABGiryG7_z_6jHdVik/pub)
* [Discussion on HackerNews](https://news.ycombinator.com/item?id=13537052)

![](https://1559079918-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LVq_rgcAGCHUmTcjeaQ%2F-LXOteHRyVxMiz_DWBFD%2F-LXOtz3u53HPerQgP5Zj%2Fthisisfine.jpg?alt=media\&token=7ee1f2ad-3053-4649-8eaf-c40f760e57bb)

### Lessons learned:

* Engineers should get more sleeps
* Restore strategy is more important than backup strategy
* Testing backup plans would not be a bad idea. If we don't test backups, we don't have them. We must rechecking backup/restore plans monthly, quarterly or yearly
* Always careful, anything with `sudo` command, we need to **double/triple check**
* Change terminal PS1 format/colors to make it clear whether you’re using production or staging
  * **RED** for production
  * **Blue/green** for staging
* Show the full hostname in the bash prompt for all users by default (e.g: `db1.staging.gitlab.com` instead of just `db1`)