Knowledge base
  • Goal of knowledge base
  • Linux & core
    • Linux
      • Record SSH session for reporting
      • Compress / Decompress files
      • Colorize logs
      • Cron output & logging
      • Signal
      • Break out and escape SSH session
      • Mount volume permanently
      • Show processes most consuming CPU & MEM
      • Improve and optimize battery life on Linux
      • File ownership & groups in linux
      • Automatic security update/patch on Ubuntu
      • Clean buffers and cached on linux
      • Bash completion on Linux/Mac
    • Core services
      • Nginx reload
      • OpenVPN Split tunneling
      • Nmap commands
    • Hardware
      • CPU Architecture fundamental
  • Database
    • MySQL
      • InnoDB - innodb_file_per_table parameter
      • MySQL - enable slow query log
      • MySQL - export large tables
    • MongoDB
  • Container
    • Docker
      • ADD or COPY in Dockerfile
        • Clean data of docker completely
    • Podman
  • Automation
    • Ansible
      • Output format
  • Build & Deployment
    • Jenkins
      • Jenkins - force exit pipeline when failure
  • Language & Toolset
    • PHP
      • Composer
      • php-redis & php-igbinary
  • Mindset
    • Technical based
      • Writing well
      • Reinvent The Wheel
      • Approach a new system
      • Backup philosophy
      • Mindset for building HA and scalable system
      • GitLab database incident
    • Non-technical based
      • How to read news efficiency?
      • How long should you nap?
      • Assume good faith
  • Reference & learning source
    • Books
      • Sysadmin/SRE
      • Mindsets
      • Software fundamentals
    • English
Powered by GitBook
On this page
  • For your information:
  • Lessons learned:
  1. Mindset
  2. Technical based

GitLab database incident

PreviousMindset for building HA and scalable systemNextNon-technical based

Last updated 6 years ago

For your information:

Lessons learned:

  • Engineers should get more sleeps

  • Restore strategy is more important than backup strategy

  • Testing backup plans would not be a bad idea. If we don't test backups, we don't have them. We must rechecking backup/restore plans monthly, quarterly or yearly

  • Always careful, anything with sudo command, we need to double/triple check

  • Change terminal PS1 format/colors to make it clear whether you’re using production or staging

    • RED for production

    • Blue/green for staging

  • Show the full hostname in the bash prompt for all users by default (e.g: db1.staging.gitlab.com instead of just db1)

Gitlab Database incident
Discussion on HackerNews