> For the complete documentation index, see [llms.txt](https://knowledge.tracelog.in/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://knowledge.tracelog.in/mindset/technical-based/gitlab-database-incident.md).

# GitLab database incident

### For your information:

* [Gitlab Database incident](https://docs.google.com/document/d/1GCK53YDcBWQveod9kfzW-VCxIABGiryG7_z_6jHdVik/pub)
* [Discussion on HackerNews](https://news.ycombinator.com/item?id=13537052)

![](/files/-LXOtz3u53HPerQgP5Zj)

### Lessons learned:

* Engineers should get more sleeps
* Restore strategy is more important than backup strategy
* Testing backup plans would not be a bad idea. If we don't test backups, we don't have them. We must rechecking backup/restore plans monthly, quarterly or yearly
* Always careful, anything with `sudo` command, we need to **double/triple check**
* Change terminal PS1 format/colors to make it clear whether you’re using production or staging
  * **RED** for production
  * **Blue/green** for staging
* Show the full hostname in the bash prompt for all users by default (e.g: `db1.staging.gitlab.com` instead of just `db1`)