# GitLab database incident

### For your information:

* [Gitlab Database incident](https://docs.google.com/document/d/1GCK53YDcBWQveod9kfzW-VCxIABGiryG7_z_6jHdVik/pub)
* [Discussion on HackerNews](https://news.ycombinator.com/item?id=13537052)

![](/files/-LXOtz3u53HPerQgP5Zj)

### Lessons learned:

* Engineers should get more sleeps
* Restore strategy is more important than backup strategy
* Testing backup plans would not be a bad idea. If we don't test backups, we don't have them. We must rechecking backup/restore plans monthly, quarterly or yearly
* Always careful, anything with `sudo` command, we need to **double/triple check**
* Change terminal PS1 format/colors to make it clear whether you’re using production or staging
  * **RED** for production
  * **Blue/green** for staging
* Show the full hostname in the bash prompt for all users by default (e.g: `db1.staging.gitlab.com` instead of just `db1`)


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://knowledge.tracelog.in/mindset/technical-based/gitlab-database-incident.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.