SLOs, Error windows and alerts are complicated. Here an attempt to make it easy

Overview

slo-computer

SLOs, Error windows and alerts are complicated. Here's an attempt to make it easy

SLO, burn_rate, error_rate, budget_spend are convoluted terms that can throw one off. Even the SRE workbook by Google can leave you with a lot of open questions.

Goal

The goal of this command (has an importable lib too) is to factor in some "bare minimum" input to

  • Is this a Low traffic service in which case it makes little sense to use an SLO approach
  • Compute the actual alert values and condition to set alerts on

Examples

Q: What alerts should I set for my service to achieve 99.9 % availability over 30 days

➜  slo-computer git:(master) ✗ ./slo-computer suggest --throughput=4200 --slo=99.9 --duration=720

		Alert if error_rate > 0.002 for last [24h0m0s] and also last [2h0m0s]
		This alert will trigger once 6.67% of error budget is consumed,
		and leaves 360h0m0s before the SLO is defeated.
		

		Alert if error_rate > 0.010 for last [1h0m0s] and also last [5m0s]
		This alert will trigger once 1.39% of error budget is consumed,
		and leaves 72h0m0s before the SLO is defeated.

Q: What alerts should I set for my service with throughpput 100rpm to achieve 90 % availability over 7 days

➜  slo-computer git:(master) ✗ ./slo-computer suggest --throughput=100 --slo=99.9 --duration=168
slo-computer: error: 
	If this service reported 10.000 errors for a duration of 5m0s
	SLO (for the entire duration) will be defeated wihin 1h40m47s

	Probably
	- Use ONLY spike alert model, and not SLOs (easiest)
	- Reduce the MTTR for this service (toughest)
	- SLO is too aggressive and can be lowerd (business decision)
	- Combine multiple services into one single service (teamwide)
, try --help
You might also like...
Errhandle - A golang error handling framework

errhandle A golang error handling framework About errhandle: An error handling f

A docker image and a launcher to run sasm on Windows and MacOS
A docker image and a launcher to run sasm on Windows and MacOS

Sasm-docker Sasm-docker simplifies the setup and use of SASM by running it inside a docker container and using x11 (X Window System) in order to displ

Raspberry Pi Archlinux Automated Offline Installer with Wi-Fi. Windows, Mac and more features coming.
Raspberry Pi Archlinux Automated Offline Installer with Wi-Fi. Windows, Mac and more features coming.

Raspberry Pi Archlinux Automated Installer with Wi-Fi. Windows, Mac and more features coming. Download Go to releases page and download the zip file f

Command-line tool to remotely execute commands on Windows machines through WinRM

winrm-cli This is a Go command-line executable to execute remote commands on Windows machines through the use of WinRM/WinRS. Note: this tool doesn't

KissMP Installer for Windows
KissMP Installer for Windows

I'm using colours in the terminal which only Powershell supports so if you see weird characters in command prompt then that's why.

App for VMware Workstartion to auto start VM's on windows reboot

VMware Workstation AutoStart This is an auto start app for VMware Workstation to auto start VM's on windows reboot with VMware Workstation installed.

Installs containerd on Windows, optionally with default CNI plugins

containerd-installer Installs containerd on Windows, optionally with default CNI plugins Usage NAME: containerd-installer.exe - Install containerd

self-made curl because windows doesn't have one 😡

Go-Curl Basic Feature of Curl made on go Installation go mod tidy then go build . Usage go-curl get "https://jsonplaceholder.typicode.com/posts/1" -b=

How to get a Go / Golang app using the Gin web framework running natively on Windows Azure App Service WITHOUT using a Docker container

Go on Azure App Service View the running app - https://go-azure-appservice.azurewebsites.net 😎 This is an example repo of how to get a Go / Golang a

Owner
last9
SRE Platform for Distributed Systems
last9
Generate Prometheus rules for your SLOs

prometheus-slo Generates Prometheus rules for alerting on SLOs. Based on https://developers.soundcloud.com/blog/alerting-on-slos. Usage Build and Run

Ganesh Vernekar 16 Nov 27, 2022
HSDP Metrics alerts webhook broker and CF events forwarder for Microsoft Teams

hsdp-events Microservice helper to translate HSDP Metrics webhooks to Microsoft Teams webhooks Configuration Environment Description EVENTS_TOKEN Rand

Philips Labs 2 Mar 18, 2022
Translate Prometheus Alerts into Kubernetes pod readiness

prometheus-alert-readiness Translates firing Prometheus alerts into a Kubernetes readiness path. Why? By running this container in a singleton deploym

Coralogix 20 Oct 31, 2022
AWS Cloudtrail event alerting lambda function. Send alerts to Slack, Email, or SNS.

Cloudtrail-Tattletail is a Lambda based Cloudtrail alerting tool. It allows you to write simple rules for interesting Cloudtrail events and forward those events to a number of different systems.

Peter Sanford 19 Jan 6, 2023
A Terraform module that creates AWS alerts billing for your resources.

terraform-aws-billing-alarms terraform-aws-billing-alarms for project Replace name project to New Project agr 'terraform-aws-billing-alarms' 'new-pr

hadenlabs 1 Oct 20, 2021
Alertmanager-cli is a cli writtin in golang to silence alerts in AlertManager

Alertmanager-cli is a cli writtin in golang to silence alerts in AlertManager

Snigdha Sambit Aryakumar 4 Aug 27, 2022
Go-indexeddb - This expirement will attempt to implement IndexedDB features for Go

go-indexeddb This expirement will attempt to implement IndexedDB features for Go

null 3 May 29, 2022
library to make building things with bitclout easy

cloutcli library to make building things with bitclout easy quick start demo cmd $ ./clout demo clout demo visualizegraph # make clout.gv graph fi

Andrew Arrow 18 Jul 27, 2022
go-awssh is a developer tool to make your SSH to AWS EC2 instances easy.

Describing Instances/VPCs data, select one or multiple instances, and make connection(s) to selected instances. Caching the response of API calls for 1day using Tmpfs.

kenju 5 Oct 11, 2021
A very simple utility that allows you to run the desired command or script as soon as a certain process with a known PID completes correctly or with an error.

go-monkill A very simple utility that allows you to run the desired command or script as soon as a certain process with a known PID completes correctl

Michael Savin 7 Dec 17, 2022