AWS Cloudtrail event alerting lambda function. Send alerts to Slack, Email, or SNS.

Overview

Cloudtail-Tattletail

Cloudtrail-Tattletail is a Lambda based Cloudtrail alerting tool. It allows you to write simple rules for interesting Cloudtrail events and forward those events to a number of different systems.

Cloudtrail-Tattletail is designed to run with a minimal set of dependencies to make it easy to get up and alerting without needing to setup a lot of different AWS services. The only hard requirement is that you enable S3 events to trigger the Tattletail lambda function from the S3 bucket that your Cloudtrail logs are going to.

Currently Cloudtrail-Tattletail supports the following destinations to forward alerts to:

  • SNS Topic
  • Email Address (via SES)
  • Slack Channel (via slack_webhook)

Forwarding to an SNS Topic allows for easy extensibility.

Configuration

There are two basic things you need to configure to start using Cloudtrail-Tattletail: Rules and Destinations.

Rules describe which events should be forwarded and alerted on. Rules are written in the jq query language. This makes it easy to write both simple and complex matching logic.

Each rule that matches an event will be forwarded to all the destinations listed in the rule.

Destinations are the upstream service that the alert should be forwarded to.

Setup

  1. Create a configuration file with your alert rules and destinations.
  2. Enable cloudtrail logging to an S3 bucket.
  3. Create a Go lambda function
  4. Grant the lambda function access to the cloudtrail s3 bucket
  5. Add an s3 trigger to invoke the lambda function for new files
  6. Add permissions for SNS and SES if you are using those destinations

Configuration example

[[rule]]
name = "Create User"
jq_match = 'select(.eventName == "CreateUser") | "username: \(.responseElements.user.userName)"'
description = "A new IAM user has been created"
destinations = ["Default SNS", "Slack Warnings", "Email"]

[[rule]]
name = "Create AccessKey"
jq_match = 'select(.eventName == "CreateAccessKey")'
description = "A new Access Key has been created"
destinations = ["Default SNS", "Slack Warnings", "Email"]

[[rule]]
name = "Modifications"
# match any event that doesn't begin with List,Get,Describe,etc.
jq_match = 'select(.eventName|test("^(List|Get|Describe|AssumeRole|Decrypt|CreateLog|ConsoleLogin)")|not)'
description = 'A config change occurred'
# just send this to Slack
destinations = ["Slack Warnings"]

[[destination]]
id = "Default SNS"
type = "sns"
sns_arn = "arn:aws:sns:us-east-1:1234567890:cloudtail_alert"

[[destination]]
id = "Slack Warnings"
type = "slack_webhook"
webhook_url = "https://foo.slack.com/some/webhook/url"

[[destination]]
id = "Email"
type = "ses"
to_emails = ["[email protected]", "[email protected]"]
from_email = "[email protected]"

The configuration file can either be bundled directly in lambda function, or it can be uploaded to an S3 bucket and the lambda function will fetch it when it is invoked. Bundling the configuration file directly is simpler but you have to reupload the whole lambda function any time you want to make configuration changes.

To include the configuration file directly in the lambda function simply create a file named tattletail.toml in the cloudtrail-tattletail working directory. Running make cloudtrail-tattletail.zip will include the configuration in the zip bundle file if it is present.

To load the configuration from an S3 bucket set the following environment variables on the Lambda function S3_CONFIG_BUCKET and S3_CONFIG_PATH. Make sure the Lambda function has permission to GetObject for that bucket + path.

If the s3 config environment variables are set they will take precedence over any bundled config file.

Lambda Function

To build the Lambda function code bundle run make cloudtrail-tattletail.zip.

Create a Go Lambda function. Upload the cloudtrail-tattletail.zip file as the code bundle.

Add an s3 trigger from your CloudTrail s3 bucket.

Add any permissions for invoking SNS or SES if you are using those destination types.

Writing jq_match queries

Each cloud trail event is tested against jq_match individually. This means your jq should not include a top level .records[]. If you want to test your jq_match query against a cloudtrail log you can do it like this:

cloudtrail.stream.json # run the jq_match query against the stream file $ jq 'select(.eventName == "CreateAccessKey")' cloudtrail.stream.json { "eventVersion": "1.08", "userIdentity": { "type": "AssumedRole", "arn": "arn:aws:sts::123456789:assumed-role/admin_read_write/AWSCLI-Session", "accountId": "123456789", "sessionContext": { "sessionIssuer": { "type": "Role", "arn": "arn:aws:iam::123456789:role/admin", "accountId": "123456789", "userName": "admin" }, "webIdFederationData": {}, "attributes": { "mfaAuthenticated": "true", "creationDate": "2021-07-13T14:59:19Z" } } }, "eventTime": "2021-07-13T15:30:43Z", "eventSource": "iam.amazonaws.com", "eventName": "CreateAccessKey", "awsRegion": "us-east-1", "sourceIPAddress": "1.1.1.1", "userAgent": "console.amazonaws.com", "requestParameters": { "userName": "nopermuser" }, "responseElements": { "accessKey": { "userName": "nopermuser", "status": "Active", "createDate": "Jul 13, 2021 3:30:43 PM" } }, "requestID": "c6325cf0-ea6e-48ea-b2bd-2df2630ce790", "eventID": "7f234c0f-61d9-4d9e-add6-f767474d9be6", "readOnly": false, "eventType": "AwsApiCall", "managementEvent": true, "eventCategory": "Management", "recipientAccountId": "123456789" } ">
# assuming your jq_match = 'select(.eventName == "CreateAccessKey")'

# get the stream of cloudtrail events:
$ jq '.Records[]' 364679954851_CloudTrail_us-east-1_20210713T1540Z_DWktkdMpEvhK04Eq.json > cloudtrail.stream.json

# run the jq_match query against the stream file
$ jq 'select(.eventName == "CreateAccessKey")' cloudtrail.stream.json
{
  "eventVersion": "1.08",
  "userIdentity": {
    "type": "AssumedRole",
    "arn": "arn:aws:sts::123456789:assumed-role/admin_read_write/AWSCLI-Session",
    "accountId": "123456789",
    "sessionContext": {
      "sessionIssuer": {
        "type": "Role",
        "arn": "arn:aws:iam::123456789:role/admin",
        "accountId": "123456789",
        "userName": "admin"
      },
      "webIdFederationData": {},
      "attributes": {
        "mfaAuthenticated": "true",
        "creationDate": "2021-07-13T14:59:19Z"
      }
    }
  },
  "eventTime": "2021-07-13T15:30:43Z",
  "eventSource": "iam.amazonaws.com",
  "eventName": "CreateAccessKey",
  "awsRegion": "us-east-1",
  "sourceIPAddress": "1.1.1.1",
  "userAgent": "console.amazonaws.com",
  "requestParameters": {
    "userName": "nopermuser"
  },
  "responseElements": {
    "accessKey": {
      "userName": "nopermuser",
      "status": "Active",
      "createDate": "Jul 13, 2021 3:30:43 PM"
    }
  },
  "requestID": "c6325cf0-ea6e-48ea-b2bd-2df2630ce790",
  "eventID": "7f234c0f-61d9-4d9e-add6-f767474d9be6",
  "readOnly": false,
  "eventType": "AwsApiCall",
  "managementEvent": true,
  "eventCategory": "Management",
  "recipientAccountId": "123456789"
}

Cloudtrail Tattletail will only alert on queries that return a non-null, non-false value.

Advanced jq_matching

Cloudtrail Tattletail will annotate alerts with custom formatted output of any jq expression that evaluates to something other than the full cloudtrail record. For example. to include the username of a CreateUser event as the annotation you could use the following jq_match:

jq_match = 'select(.eventName == "CreateUser") | "username: \(.responseElements.user.userName)"'

# this outputs:
"username: hacker1"

If you do not want to include any match metadata with the alerts use select():

jq_match = 'select(.eventName == "CreateUser")

Screenshots

Slack Webhook

Slack Webhook

Email (SES)

Email

Issues
  • Filter slack webhookUrl to log

    Filter slack webhookUrl to log

    Thank you for the great product 👍 This is what I wanted. 😄

    As a feature I need. I want to filter and output the WebHook URL in the Lambda execution log.

    image

    opened by Sho2010 1
Owner
Peter Sanford
Peter Sanford
A plugin for running Open Policy Agent (OPA) in AWS Lambda as a Lambda Extension.

opa-lambda-extension-plugin A custom plugin for running Open Policy Agent (OPA) in AWS Lambda as a Lambda Extension. To learn more about how Lambda Ex

GoDaddy 23 May 10, 2022
A Terraform module that creates AWS alerts billing for your resources.

terraform-aws-billing-alarms terraform-aws-billing-alarms for project Replace name project to New Project agr 'terraform-aws-billing-alarms' 'new-pr

hadenlabs 1 Oct 20, 2021
Serviço de consulta de CEP Serverless usando Lambda function em Golang

Consulta CEP Serverless Consulta CEP foi desenvolvido com o objetivo de facilitar a vida do desenvolvedor que precisa de um serviço de consulta de CEP

Otavio Baldan 1 Oct 26, 2021
This is for managing Slack App Manifests, it is no use if you are not developing an App for Slack.

Terraform Provider Slack App This is for managing Slack App Manifests, it is no use if you are not developing an App for Slack. Requirements Terraform

ChangeEngine 2 May 23, 2022
Stream, Mutate and Sign Images with AWS Lambda and ECR

ocistow About How it works Try it yourself Prerequisites CLI (cmd/ocistow) Lambda (cmd/ocistow-lambda) Deploy Invoke Verify signatures with =cosign= I

Martin Baillie 17 May 12, 2022
Deploy 2 golang aws lambda functions using serverless framework.

Deploy 2 golang aws lambda functions using serverless framework.

NguyenTheAnh 0 Jan 20, 2022
KEDA is a Kubernetes-based Event Driven Autoscaling component. It provides event driven scale for any container running in Kubernetes

Kubernetes-based Event Driven Autoscaling KEDA allows for fine-grained autoscaling (including to/from zero) for event driven Kubernetes workloads. KED

KEDA 5.3k Aug 11, 2022
Translate Prometheus Alerts into Kubernetes pod readiness

prometheus-alert-readiness Translates firing Prometheus alerts into a Kubernetes readiness path. Why? By running this container in a singleton deploym

Coralogix 19 Mar 7, 2021
SLOs, Error windows and alerts are complicated. Here an attempt to make it easy

slo-computer SLOs, Error windows and alerts are complicated. Here's an attempt to make it easy SLO, burn_rate, error_rate, budget_spend are convoluted

last9 56 Jul 14, 2022
HSDP Metrics alerts webhook broker and CF events forwarder for Microsoft Teams

hsdp-events Microservice helper to translate HSDP Metrics webhooks to Microsoft Teams webhooks Configuration Environment Description EVENTS_TOKEN Rand

Philips Labs 2 Mar 18, 2022
Alertmanager-cli is a cli writtin in golang to silence alerts in AlertManager

Alertmanager-cli is a cli writtin in golang to silence alerts in AlertManager

Snigdha Sambit Aryakumar 3 Jan 25, 2022
Time Series Alerting Framework

Bosun Bosun is a time series alerting framework developed by Stack Exchange. Scollector is a metric collection agent. Learn more at bosun.org. Buildin

Bosun 3.3k Aug 3, 2022
This plugin helps you to use the AWS Command Line Interface (AWS CLI) to start and end sessions to your managed instances

Session Manager Plugin This plugin helps you to use the AWS Command Line Interface (AWS CLI) to start and end sessions to your managed instances. Sess

Amazon Web Services 156 Aug 5, 2022
Terraform provider to help with various AWS automation tasks (mostly all that stuff we cannot accomplish with the official AWS terraform provider)

terraform-provider-awsutils Terraform provider for performing various tasks that cannot be performed with the official AWS Terraform Provider from Has

Cloud Posse 20 Jul 29, 2022
Infrastructure testing helper for AWS Resources that uses AWS SSM to remotely execute commands on EC2 machines.

Infrastructure testing helper for AWS Resources that uses AWS SSM to remotely execute commands on EC2 machines, to enable infrastructure engineering teams to write tests that validate behaviour.

Ankit Wal 18 Jul 28, 2022
Amazon Web Services (AWS) providerAmazon Web Services (AWS) provider

Amazon Web Services (AWS) provider The Amazon Web Services (AWS) resource provider for Pulumi lets you use AWS resources in your cloud programs. To us

William Garcia Jacobo 0 Nov 10, 2021
Aws asg updater - Update AMIs within AWS Auto Scaling groups automatically.

AWS Auto Scaling Groups Updater AWS Auto Scaling group is a great way of managing Amazon EC2 instances. AWS Auto Scaling group watches the correspondi

Alexey Shagraev 1 Jan 6, 2022
Aws-secretsmanager-caching-extension - Cache server for AWS Secrets Manager

AWS Lambda Extension / Sidecar Container Cache Server The cache server is writte

CustomerGauge 4 Apr 4, 2022