⛑ Gatus - Automated service health dashboard

As mentioned in my previous issue #61 i'm currently using healthchecks, and the last feature i would need is some generic content regex / string presence check. now, i might be missing something everybody already knows, since you have this BODY placeholder, and maybe that's just it.

anyway, it seems that the process basically would mean gatus curls the site content and runs a regex or literal string against it, and when it finds it, test passes.

what do you think?

in terms of performance, i think that this is actually the most heavy that healthchecks is doing, but already now i'm running all the tests in parallel, and gatus outperforms healthchecks by magnitudes. so nice.

Describe the bug

When using ICMP and [CONNECTED] == true, whenever an endpoint fails, the Google Chat/Spaces App only sends a blank message with no data.

What do you see?

Nothing is displayed, it sends an empty message.

What do you expect to see?

Full alert, like with HTTPS, which works just fine. This is ICMP only.

List the steps that must be taken to reproduce this issue

Setup googlechat alert, have an ICMP endpoint fail.

Version

latest branch

Additional information

No response

First of all, thanks for such a simple tool and for making it open source :) I tried to see if the following issue was posted already, but I couldn't find anything.

All my 301 and 302 checks are failing, and it seems to be because when making the HTTP requests, it's following redirects by default.

Here's an example. https://inbox.google.com does a 301 to https://mail.google.com/, but the status code in the image below is 200.

Is this by design?

I'm monitoring around 50-55 services with gatus, most are HTTP and 29 of them are using the pat keyword (with wildcards, so about as expensive a query as it can get). All using the default poll interval of 60s.

I am starting to see some responses of context deadline exceeded (Client.Timeout exceeded while awaiting headers) in the body check. I have manually checked the services in question and they're healthy. Restarting gatus, or just waiting a few minutes seems to resolve this. This does not occur continuously, but have seen it twice in the space of a few hours.

I can only assume that this is due to a concurrency issue, as it's more than possible that the combination of service times takes longer than 60s to respond. I do not know enough of the gatus architecture to know if this is a problem or not.

I am running v2.3.0 with #100 changed locally (as I've not updated since I tested it). I will repeat the test with 2.4.0 and report the results here.

Supporting service groups could allow a cuter front end experience.

i.e.

services:
  - name: k8s-cluster-watch-dog
    url: http://k8s-cluster-watch-dog-v1.tools-${ENVIRONMENT}:8080/health
    group: core         <-------
    interval: 1m
    conditions:
      - "[STATUS] == 200"
      - "[BODY].status == UP"
  - name: prometheus
    url: http://prometheus-operator-prometheus.kube-system:9090/-/healthy
    group: core         <-------
    interval: 1m
    conditions:
      - "[STATUS] == 200"
      - "[BODY] == Prometheus is Healthy."

would generate a dashboard that puts both k8s-cluster-watch-dog and prometheus under the "core" folder.

Could also support tags, and allow filtering by tags instead

i.e.

services:
  - name: k8s-cluster-watch-dog
    url: http://k8s-cluster-watch-dog-v1.tools-${ENVIRONMENT}:8080/health
    tags:              <-------
      - core
    interval: 1m
    conditions:
      - "[STATUS] == 200"
      - "[BODY].status == UP"
  - name: prometheus
    url: http://prometheus-operator-prometheus.kube-system:9090/-/healthy
    tags:              <-------
      - core
      - metrics
    interval: 1m
    conditions:
      - "[STATUS] == 200"
      - "[BODY] == Prometheus is Healthy."

Forgive the terrible drafts, just thought of this on the fly.

MAC OS Big Sur version 11.0.1

Everything works perfectly ok except icmp in MAC OS.

Config:

• name: node-1 url: "icmp://104.237.x.x" group: Uptime (Ping) conditions:
  • "[CONNECTED] == true"
  • "[RESPONSE_TIME] > 0"

the x.x is in ip is set intentionally.

Is this wrong config?

Hello, I'm a Chinese and my team use Wechat Company (Wecom) to watch status of servers.

I'm not a developer, I try to write wecom alerting code and it work's fine on my machine. But because of my Go language level, there may be many exceptions that have not been handled. If possible, I hope it can be merge into the official code. Thanks.

Push to Wecom, we need 3 necessary parameters, Aid, Cid and Secret.

package wecom

import (
	"encoding/json"
	"fmt"
	"io/ioutil"
	"net/http"

	"github.com/TwinProduction/gatus/alerting/alert"
	"github.com/TwinProduction/gatus/alerting/provider/custom"
	"github.com/TwinProduction/gatus/core"
)

// AlertProvider is the configuration necessary for sending an alert using Wecom
type AlertProvider struct {
	Aid    string `yaml:"aid"`
	Cid    string `yaml:"cid"`
	Secret string `yaml:"secret"`
	ToUser string `yaml:"touser"` // Not necessary
	ToParty string `yaml:"toparty"` // Not necessary

	// DefaultAlert is the default alert configuration to use for services with an alert of the appropriate type
	DefaultAlert *alert.Alert `yaml:"default-alert"`
}

Then, we need use Cid and Secret to get accessToken.

	var wetoken string
	wetoken = fmt.Sprintf("%s%s%s%s", "https://qyapi.weixin.qq.com/cgi-bin/gettoken?corpid=", provider.Cid, "&corpsecret=", provider.Secret)
	client := &http.Client{}
	req, err := http.NewRequest("GET", wetoken, nil)
	resp, err := client.Do(req)
	if err != nil {
		fmt.Println("Failure : ", err)
	}
	respBody, _ := ioutil.ReadAll(resp.Body)
	r := make(map[string]interface{})
	json.Unmarshal([]byte(respBody), &r)
	accessToken := r["access_token"]

Last, use accessToken to push.

	return &custom.AlertProvider{
		URL:    fmt.Sprintf("https://qyapi.weixin.qq.com/cgi-bin/message/send?access_token=%s", accessToken),
		Method: http.MethodPost,
		Body: fmt.Sprintf( TODO:BODY ),
		Headers: map[string]string{"Content-Type": "application/json"},
	}

If ToUser and ToParty are empty, the BODY will be:

{
	"agentid": %s,
	"msgtype": "text",
	"text": {
		"content": "%s"
	},
	"duplicate_check_interval": 180
}

If one of the two is empty, the BODY will be:

{
	"toparty": %s,
	"agentid": %s,
	"msgtype": "text",
	"text": {
		"content": "%s"
	},
	"duplicate_check_interval": 180
}

or

{
	"touser": %s,
	"agentid": %s,
	"msgtype": "text",
	"text": {
		"content": "%s"
	},
	"duplicate_check_interval": 180
}

If both parameters are not empty, the BODY will be:

{
	"touser": %s,
	"toparty": %s,
	"agentid": %s,
	"msgtype": "text",
	"text": {
		"content": "%s"
	},
	"duplicate_check_interval": 180
}

I stumbled on this neat status page solution the other day and am trying it out, cool stuff!

I see Store is an interface, and that there is one implementation (in-memory cache with a file used to persist over restarts). However I think it's missing some way to persist the observed results which is more usable in a stateless container, where you don't want to write to a file system.

Is there perhaps already some extension out there that provides a Store backed by e.g. Redis or Postgres? I would not mind contributing, but if the effort had already been made then that would be unnecessary.

The point would not be to enable some high availability configuration (I think that would add quite a bit more synchronization work), but simply being able to restart or reconfigure gatus without losing the previous results would be nice.

Also, the auto save interval could be made configurable.

Describe the feature request

In this project https://github.com/prometheus/blackbox_exporter, many metrics like probe_success, probe_duration etc... provided. And in dns probe https://github.com/prometheus/blackbox_exporter/blob/master/prober/dns.go#L129 also has metrics for dns.

Why do you personally want this feature to be implemented?

Maybe we can add these metrics in gatus, and user can scrpe to promethus and make dashboard on Grafana.

How long have you been using this project?

3 month

Additional information

If the team agrees to add this feature, I can submit PR to contribute gatus

Call me indecisive if you want, since I've done this once before (see 70c9c4b87c8e595f4374b895853312cefc2152f2), but after thing about the pros and cons, I decided that Apache 2 offers better protection for Gatus.

Long story short:

• v0.0.1 to v1.3.0: Apache 2
• v1.3.1 to v3.3.5: MIT
• v3.3.6 and up: Apache 2

This PR is here for traceability.

Step to reproduce:

$ cat config/config.yaml
services:
  - name: example
    url: "https://example.org/"
    interval: 30s
    conditions:
      - "[STATUS] == 200"

$ docker run \
    --rm \
    -p 8080:8080 \
    -v "$(pwd)"/config/config.yaml:/config/config.yaml \
    twinproduction/gatus

Change the name field in the config/config.yaml.

Observation:

In the HasLoadedConfigurationFileBeenModified function, the result of the comparison of config.lastFileModTime.Unix() != fileInfo.ModTime().Unix() is always false.

Describe the feature request

I'd like Gatus to have an API that allows managing its configuration so I can create a Kubernetes controller (or maybe extend https://github.com/stakater/IngressMonitorController) to watch Ingress resources and automatically create checks in Gatus for them.

An alternative could be to leverage something like https://github.com/TwiN/gatus/issues/326 and have a sidecar like the https://github.com/kiwigrid/k8s-sidecar to pass the configuration to Gatus but that would have more moving parts.

Why do you personally want this feature to be implemented?

To be able to programatically setup checks for services running in a Kuberntes cluster.

How long have you been using this project?

No response

Additional information

No response

Summary

This PR allows configuration to split across multiple files using the common config.d/-path style. The approach reads all *.yml and *.yaml files and merges them in memory into a single file before parsing. This seemed easier und required fewer changes than parsing each file individually and merging the objects afterwards.

Fixes #326

As I did not work with golang before, feel free to remark on obvious errors and bad patterns.

I envisaged to write tests, but did not figure out yet, how invasive tests are intended to be structured for this project. I hope the chosen way is in you interest.

Checklist

• [x] Tested and/or added tests to validate that the changes work as intended, if applicable.
• [x] Added the documentation in README.md, if applicable.

Describe the bug

Sometimes, the TestStore_InsertCleansUpOldUptimeEntriesProperly test will fail

What do you see?

https://github.com/TwiN/gatus/actions/runs/3723336201/attempts/1

 --- FAIL: TestStore_InsertCleansUpOldUptimeEntriesProperly (0.04s)
    sql_test.go:109: oldest endpoint uptime entry should've been ~5 hours old, was 4h59m40.064240509s
    sql_test.go:119: oldest endpoint uptime entry should've been ~5 hours old, was 4h59m40.068831372s
    sql_test.go:129: oldest endpoint uptime entry should've been ~8 hours old, was 7h59m40.073425836s
    sql_test.go:139: oldest endpoint uptime entry should've been ~239h0m0s hours old, was 238h59m40.0780609s
    sql_test.go:150: oldest endpoint uptime entry should've been ~8 hours old, was 7h59m40.08307097s

What do you expect to see?

Test should pass

List the steps that must be taken to reproduce this issue

1. Execute TestStore_InsertCleansUpOldUptimeEntriesProperly between hh:59:30 and hh:59:59
2. Note how the test will fail

Version

latest (master)

Additional information

I suspect this has something to do with https://github.com/TwiN/gatus/blob/f6a621da285dd43b3207ec21e4101b4b11eff6c5/storage/store/sql/sql_test.go#L100

Describe the feature request

Ability to send repeated notifications every x min/hr if the endpoint is still offline/dead.

Why do you personally want this feature to be implemented?

We now had over 300 nodes monitoring via Gatus and we received the notifications via Slack and Telegram. One scenario is when some nodes are offline during the weekend. We always forget to fix them in the next workday.

I've tried some other services and seems like it's a common feature to alert repeatedly if something still goes wrong.

How long have you been using this project?

No response

Additional information

No response

Describe the bug

A JSON API endpoint I'm monitoring briefly fails from time to time with the error message

invalid character '<' looking for beginning of value

The failing condition is len([BODY].25447.bill) >= 1.

I don't really understand what the error message is supposed to mean. No JSON payload present in the [BODY]?

What do you see?

An opaque error message.

What do you expect to see?

A clear/understandable error message.

List the steps that must be taken to reproduce this issue

Set up an endpoint with

- name: "TEST"
  enabled: true
  url: "https://api.votelog.ch/api/v1/bill/380b49bf-8044-4f65-a2e5-ac3a00e3e083/votes?lang=de"
  interval: 60s
  conditions:
    - "[STATUS] == 200"
    - "len([BODY].25447.bill) >= 1"

Version

4.3.2

Additional information

No response

Describe the feature request

When using a black-and-white or dark logo with transparent background[*], it becomes barely visible when dark mode is toggled on. The same is true for light mode with a white/light logo with transparent background. Thus it would be nice if we could configure two seperate logo files/URLs, one for the (default) light mode and one for the dark mode.

Config could become ui.logo.light and ui.logo.dark, maybe with a fallback to the current behaviour when ui.logo is directly set to a string instead of the two subkeys.

Why do you personally want this feature to be implemented?

It would improve Gatus' web frontend aesthetically. 💅

How long have you been using this project?

3 days (awesome project! ❤️)

Additional information

[*] Example Gatus instance with such a logo is found here: https://status.votelog.ch/