Biscuit is a multi-region HA key-value store for your AWS infrastructure secrets.

Overview

Biscuit

Build Status

Biscuit is a simple key-value store for your infrastructure secrets.

Is Biscuit right for me?

Biscuit is most useful to teams already using AWS and IAM to manage their infrastructure. If that describes your team, then Biscuit might be useful to you if you answer "yes" to any of these questions:

  • Do you live in constant fear of accidentally committing infrastructure secrets to source control?
  • Do you commit private keys to your source repository?
  • Do you share passwords with other developers?
  • Do you want to manage secrets securely across multiple regions?

Features

  • Provides a simple key/value CLI to secure storage.
  • Secrets can live alongside with your code in source control.
  • Operates with KMS keys across multiple regions.
  • Facilitates management of AWS IAM Policies, KMS Policies, and KMS Grants across multiple regions.
  • Local encryption using AES-GCM-256 or Secretbox (NaCL).
  • Offline mode: Using the "testing" key manager, you can use Biscuit in test environments without changing your code and without network dependencies.

Feature Comparison

Package Requires a server? Multi-region HA Rotation Storage AWS KMS Principals Web UI
Biscuit No Yes Yes No File Required AWS Only No
Credstash No No Yes No DynamoDB Required AWS Only No
Lyft Confidant Yes No No No DynamoDB Required AWS Only Yes
Hashicorp Vault Yes Yes Yes Yes Varied Optional Multiple No

Quick Start

Installing

Downloading

See releases for the latest release.

Building from Source

If you have Golang 1.16+ installed, you can install with:

go get -v github.com/dcoker/biscuit

Setup

# Verify that your AWS credentials are readable.
biscuit kms get-caller-identity

# Provision a KMS Key w/useful defaults in us-east-1, us-west-1, 
# and us-west-2 and create a secrets.yml file.
biscuit kms init -f secrets.yml

# Store the launch codes.
biscuit put -f secrets.yml -- launch_codes 0000

# Decrypt the launch codes.
biscuit get -f secrets.yml launch_codes

Next steps: examine secrets.yml in your favorite text editor, and run biscuit --help to learn about additional commands.

Uninstalling

Done already?

The biscuit kms init step above may have created a KMS Key and some associated policies using CloudFormation. You can remove those by running:

biscuit kms deprovision
rm secrets.yml

Note: any biscuit files you created before deprovisioning will no longer be readable.

Glossary

The secret is the plaintext value which you wish to protect.

A label is a short alphanumeric string that identifies a set of keys across multiple AWS regions. It is present in CloudFormation stack names (biscuit-label) and in KMS key aliases (alias/biscuit-label).

The key manager is responsible for the provisioning of encryption keys. The encryption keys generated by the key manager are used to encrypt the secret. An encrypted version of the encryption key -- decryptable only by KMS -- is stored as the key ciphertext.

Secret values consist of the information necessary for the key manager to provide the plaintext encryption key to decrypt a ciphertext for a named secret. Values consist of a Key ID (a string, meaningful to the key manager), an indicator of which key manager is in use (string), an algorithm (string), the key ciphertext (the encrypted key, base64), and the ciphertext (base 64). Here is an example of a value named api_key:

api_key:
- key_id: arn:aws:kms:us-west-1:123456789012:key/37793df5-ad32-4d06-b19f-bfb95cee4a35
  key_manager: kms
  algorithm: secretbox
  key_ciphertext: CiA3edlKfUWXVgiDDuzbz95S/pkM8grwRsYkjRoURv0LGhKnAQEBAQB4N3nZSn1Fl1YIgw7s28/eUv6ZDPIK8EbGJI0aFEb9CxoAAAB+MHwGCSqGSIb3DQEHBqBvMG0CAQAwaAYJKoZIhvcNAQcBMB4GCWCGSAFlAwQBLjARBAw4OEtFZrisfC3xJHACARCAO+HJpH4bWD/MF9BYjBvl5ztcezTNxo5SPeAOKJ3Z8Pff2vh1uCZhEEjxnF7t1tqTma8oeESuu2vpPiZp
  ciphertext: YsI/4Qnzpu+Vm+JP4LhnO8Y3dSoz61/vKHBXGVI1pVAUCjMhvjb9ohcdjA==

Names identify a value. When using AWS KMS, names are encoded into the encryption context and must be provided by the process that decrypts the value.

A key template is a special entry in the .yml file which tells the biscuit tool how to handle values that are added to that file. This is an example template:

_keys:
- key_id: arn:aws:kms:us-west-1:123456789012:key/37793df5-ad32-4d06-b19f-bfb95cee4a35
  key_manager: kms
  algorithm: secretbox
- key_id: arn:aws:kms:us-west-2:123456789012:key/c0045b15-9880-4b17-84da-a35760e8a16f
  key_manager: kms
  algorithm: secretbox
- key_id: arn:aws:kms:us-east-1:123456789012:key/d1c5a8e3-adfb-4f79-af0b-cde9f1a31292
  key_manager: kms
  algorithm: secretbox

IAQ

How much does this cost?

Biscuit requires one key in each region you wish to use. Most users can expect to pay ~$1/mo per region per label, and additional usage charges for heavy users. See AWS Key Management Service Pricing for pricing.

Can I use it in a single AWS region?

Yes. Use the -r flag or the BISCUIT_REGIONS environment variable to specify the region.

Example:

biscuit kms init -r us-west-2 -f secrets.yml

What do I do with the .yml file after it is created?

The .yml file is safe for committing to version control, your CI system, embedding in native binaries, copying to S3, publishing in a newspaper, etc. You can deploy these to your production servers by whatever mechanism is most appropriate for your environment, but by far the easiest way to start is to simply include it in your deployments in the same way you would a configuration file.

Once I've created a value, how do I let AWS resources decrypt it?

You can use KMS Grants, KMS Key Policies, or IAM Policies to manage access to the secrets.

KMS Grants

KMS Grants enable you to delegate access to specific KMS operations to some AWS principal. Often your AWS resources will be running with an IAM Role, and thus often the easiest thing to do is to use KMS Grants to allow your IAM Roles to decrypt the appropriate values.

Biscuit will create and retire those grants for you.

Here's how to grant role/webserver and user/gordon the ability to decrypt the launch codes:

biscuit kms grants create --grantee-principal role/webserver -f secrets.yml launch_codes
biscuit kms grants create --grantee-principal user/gordon -f secrets.yml launch_codes
biscuit kms grants list -f secrets.yml launch_codes

If you wish to allow a principal to decrypt all values encrypted under the same set of keys as the launch codes, you can pass the --all-names flag:

biscuit kms grants create -g role/webserver -f secrets.yml --all-names launch_codes

You can also retire grants when they are no longer useful:

biscuit kms grants list -f secrets.yml launch_codes
biscuit kms grants retire -f secrets.yml --grant-name biscuit-ff8102edc8 launch_codes

Biscuit manages grants using the KMS CreateGrant, ListGrants, and RetireGrant APIs.

KMS Key Policies

KMS Keys have their own policies. See Key Policies for more details. If you have just a few users, this is possibly the easiest mechanism to use to control access. You can run biscuit kms edit-key-policy to edit the policy document across all of your regions at once.

Biscuit manages Key Policies using the KMS GetKeyPolicy and SetKeyPolicy APIs.

IAM Policies

IAM Policies are attached to a myriad of AWS entities, and they can also be used to enable access to KMS operations. See Key Policies for more details.

Example: You have a server running with a static AWS access key and secret key. You can give that server the ability to decrypt all values encrypted under a set of keys by attaching a standard user policy, specifying kms:Decrypt as the Action and the full key ARNs in the Resource field.

Note: IAM Policies are global entities, whereas KMS Keys are unique per region. Thus if you have a 3-region configuration, any IAM Policies that explicitly grant access to KMS Keys will need to list all 3 region-specific key ARNs.

Biscuit does not manage IAM Policies for you.

If you wish to disallow IAM Policies from controlling access to your keys, you can do so by passing --disable-iam-policies to kms init. When IAM Policies are disabled, the only way to control access to keys is via Grants and KMS Key Policies. For more information on how this works, see the CloudFormation template in the source repository and the Key Policies doc linked above.

How do I control which AWS region is used to decrypt the values?

Each AWS region has its own isolated KMS instance. This means that KMS keys are per-region resources, and using a KMS key requires communicating with the KMS service in the region that holds that key.

When encrypting, Biscuit will attempt to encrypt the secrets by using all of the KMS instances corresponding to the regions of the keys it is told to use. This can incur cross-datacenter traffic and is slower than using only the closest region.

When decrypting, Biscuit only needs to decrypt under one of the keys. By default, Biscuit will prioritize using keys that are in the same region that the caller is in. This is determined by the AWS_REGION environment variable. If you do not have an AWS_REGION variable set, Biscuit will process keys in the order that they appear in the .yaml file.

You can override this behavior by passing a --aws-region-priority flag to the get or export operations. Here is an example invocation which prioritizes keys in ap-north-1 and us-west-2:

biscuit get --aws-region-priority ap-north-1,us-west-2 -f secrets.yml launch_codes

We recommend you make arrangements on your EC2 instances to either set the AWS_REGION flag, or pass a latency-ordered list of regions via the --aws-region-priority flag.

How do I keep my development and production keys separate?

Biscuit tracks keys across regions by using a label. Labels are embedded into the name of the CloudFormation stack and a KMS Key Alias in each region. The default behavior is to use the default label, but you can change this by passing the -l flag. Example: biscuit kms init -l development

Labels are not persisted with the values and are not visible in the .yml files. They are an organizational tool to facilitate managing keys across regions, and are passed as parameters to various commands.

Here are some common scenarios:

# Create key in a single-region, and allow developers to do whatever they want. 
biscuit kms init -r us-west-2 -l development -f development.yml --administrators role/developers --users role/developers
biscuit put -f development.yml ssl_key -i selfsigned.key

# Create keys in three regions, and allow a limited set of people to administer it,
# and allow developers (and gordon) to read and write the secrets.
biscuit kms init -l production -f production.yml --administrators role/prod-keymaster --users role/developers,user/gordon
biscuit put -f production.yml ssl_key -i wildcard.key

# Create a file that doesn't use encryption at all.
biscuit put -f unittest.yml -a none -- database_password testing

What's the difference between an "administrator" and a "user"?

Biscuit installs a KMS Key Policy similar to the default policy recommended by AWS. This creates a distinction between "administrators" and "users".

Administrators can administer the key but not necessarily encrypt or decrypt with that key. They could, if they wished, replace the key policy with one that does allow them encrypt and decrypt, but that would be unusual.

Users can encrypt, decrypt, and generate ephemeral data keys. In most cases, you'll want to make your development team a "user" and possibly also "administrators".

If you use biscuit kms init to create your keys, you can use the --administrators and --users flag to set membership. If you have already created the keys but want to change the policy, use the interactive biscuit kms edit-key-policy to apply changes to all regions simultaneously.

What is the minimum IAM Policy needed to run kms init?

The IAM Policy below is the smallest set of permissions needed to get started with Biscuit using kms init. Be sure to replace the account number 123456789012 with your own.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowKmsListAliasesAndCreateKey",
            "Effect": "Allow",
            "Action": [
                "kms:ListAliases",
                "kms:CreateKey"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "AllowKmsCreateAlias",
            "Effect": "Allow",
            "Action": [
                "kms:CreateAlias"
            ],
            "Resource": [
                "arn:aws:kms:*:123456789012:alias/biscuit-*"
            ]
        },
        {
            "Sid": "AllowKmsDeleteAlias",
            "Effect": "Allow",
            "Action": [
                "kms:DeleteAlias"
            ],
            "Resource": [
                "arn:aws:kms:*:123456789012:alias/biscuit-*"
            ]
        },
        {
            "Sid": "AllowCloudFormationCreate",
            "Effect": "Allow",
            "Action": [
                "cloudformation:DescribeStacks",
                "cloudformation:CreateStack",
                "cloudformation:DeleteStack"
            ],
            "Resource": [
                "arn:aws:cloudformation:*:123456789012:stack/biscuit-*"
            ]
        },
        {
            "Sid": "AllowCloudFormationDelete",
            "Effect": "Allow",
            "Action": [
                "cloudformation:DeleteStack"
            ],
            "Resource": [
                "arn:aws:cloudformation:*:123456789012:stack/biscuit-*"
            ]
        }
    ]
}

Does Biscuit support multi-factor authentication?

Yes. IAM Policies and the KMS Key Policies support the aws:MultiFactorAuthPresent condition.

Biscuit does not enforce any access control on its own. All enforcement is implemented by AWS. Any features that are available in the IAM Policies, KMS Key Policy documents, or KMS Grants are available to you.

Biscuit does not expose command line flags to use all of the features, but you can edit the various policies after they are created, or override the default CloudFormation template.

Biscuit is known to work well with awsmfa.

I manually edited the .yaml file and changed the name of a value and now it won't decrypt. What's wrong?

The kms key manager annotates the ciphertext with an EncryptionContext containing the name of the value. If you change the name of a value, then the decrypting process will provide the wrong name to the decrypt operation and the decrypt will fail. If you wish to change the name of a secret, re-encrypt it using the new name instead.

I want to change something about the CloudFormation template. What do I do?

The biscuit kms init command allows you to override the built-in CloudFormation template with your own via the --cloudformation-template-url parameter.

My account administrator does not let me create CloudFormation stacks, help!

The only expectations that Biscuit has about your KMS configuration is that the keys have aliases of the form alias/biscuit-{label} and that you have sufficient permissions to operate on the KMS keys. You can create the keys using whatever process is compatible with your organization's policies.

How do I rotate the values?

Biscuit considers the rotation of secrets (such as database passwords) to be application-specific features and thus does not have any native support for it. However, you can implement a rotation scheme appropriate for your situation simply by serializing that state as the secret and then read it with your application-specific rotation behaviors.

Here is an example using JSON and jq:

biscuit put -f secrets.yml -- database_passwords '{"1": "pass1", "2": "pass2"}'
biscuit get -f secrets.yml database_passwords | jq -r '.[to_entries | map(.key) | map(tonumber) | max | tostring]'
Issues
  • Couple proposed refactors

    Couple proposed refactors

    These suggested refactors should bring it into a little more easily manageable state. They are all individual commits, so we can pick and choose what we want.

    opened by frankywahl 0
  • Allow `doc.go` to be created with go generate

    Allow `doc.go` to be created with go generate

    This will also give us information if somebody tries to edit this file, the go tooling indicates that this file is auto generated.

    This also allows other architectures that treat echo -e command differently to not impede on how doc.go is generated.

    opened by frankywahl 0
  • Local Development

    Local Development

    Update binary to allow custom AWS endpoint

    Use localstack as a default for testing

    Update testing framework to use Github Workflows instead

    Following in up on https://github.com/dcoker/biscuit/pull/15, this enables all branches to run tests and allows test to run against localstack

    opened by frankywahl 0
  • Goreleaser

    Goreleaser

    Switch to using GoReleaser

    TODO (potentially in follow up PRs)

    • [x] Fix /tests folder to work with alpine image (done in https://github.com/dcoker/biscuit/pull/16)
    • [x] Write Tests Workflow (Unit & Integration) (done in https://github.com/dcoker/biscuit/pull/16)
    opened by frankywahl 0
  • Maintain for newer tools and version of Go

    Maintain for newer tools and version of Go

    This PR brings biscuit up to date with newer tools.

    • glock => go modules
    • go tools vet => go vet
    • go-bindata => embed.FS
    • No more GOPATH when using modules.
    opened by frankywahl 0
Releases(v0.1.4)
Owner
Doug
Doug
HA LDAP based key/value solution for projects configuration storing with multi master replication support

Recon is the simple solution for storing configs of you application. There are no specified instruments, no specified data protocols. For the full power of Recon you only need curl.

Mikhail Panfilov 12 Jun 15, 2022
A disk-backed key-value store.

What is diskv? Diskv (disk-vee) is a simple, persistent key-value store written in the Go language. It starts with an incredibly simple API for storin

Peter Bourgon 1.2k Aug 8, 2022
Distributed reliable key-value store for the most critical data of a distributed system

etcd Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in order

etcd-io 40.7k Aug 4, 2022
Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service.

Olric Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service. With

Burak Sezer 2.3k Aug 1, 2022
a persistent real-time key-value store, with the same redis protocol with powerful features

a fast NoSQL DB, that uses the same RESP protocol and capable to store terabytes of data, also it integrates with your mobile/web apps to add real-time features, soon you can use it as a document store cause it should become a multi-model db. Redix is used in production, you can use it in your apps with no worries.

Mohammed Al Ashaal 1.1k Jul 29, 2022
GhostDB is a distributed, in-memory, general purpose key-value data store that delivers microsecond performance at any scale.

GhostDB is designed to speed up dynamic database or API driven websites by storing data in RAM in order to reduce the number of times an external data source such as a database or API must be read. GhostDB provides a very large hash table that is distributed across multiple machines and stores large numbers of key-value pairs within the hash table.

Jake Grogan 724 Jul 28, 2022
Pogreb is an embedded key-value store for read-heavy workloads written in Go.

Embedded key-value store for read-heavy workloads written in Go

Artem Krylysov 914 Aug 2, 2022
CrankDB is an ultra fast and very lightweight Key Value based Document Store.

CrankDB is a ultra fast, extreme lightweight Key Value based Document Store.

Shrey Batra 30 Apr 12, 2022
yakv is a simple, in-memory, concurrency-safe key-value store for hobbyists.

yakv (yak-v. (originally intended to be "yet-another-key-value store")) is a simple, in-memory, concurrency-safe key-value store for hobbyists. yakv provides persistence by appending transactions to a transaction log and restoring data from the transaction log on startup.

Aadhav Vignesh 5 Feb 24, 2022
Multithreaded key value pair store using thread safe locking mechanism allowing concurrent reads

Project Amnesia A Multi-threaded key-value pair store using thread safe locking mechanism allowing concurrent reads. Curious to Try it out?? Check out

Nikhil Nayak 6 Apr 7, 2022
ShockV is a simple key-value store with RESTful API

ShockV is a simple key-value store based on badgerDB with RESTful API. It's best suited for experimental project which you need a lightweight data store.

delihiros 2 Sep 26, 2021
A rest-api that works with golang as an in-memory key value store

In Store A rest-api that works with golang as an in-memory key value store Usage Fist of all, clone the repo with the command below. You must have gol

Eyüp Arslan 0 Oct 24, 2021
Distributed key-value store

Keva Distributed key-value store General Demo Start the server docker-compose up --build Insert data curl -XPOST http://localhost:5555/storage/test1

Yaroslav Gaponov 0 Nov 15, 2021
Simple in memory key-value store.

Simple in memory key-value store. Development This project is written in Go. Make sure you have Go installed (download). Version 1.17 or higher is req

Mustafa Navruz 0 Nov 6, 2021
A simple in-memory key-value store application

vtec vtec, is a simple in-memory key-value store application. vtec provides persistence by appending transactions to a json file and restoring data fr

Ahmet Tek 3 Jun 22, 2022
An in-memory key:value store/cache (similar to Memcached) library for Go, suitable for single-machine applications.

go-cache go-cache is an in-memory key:value store/cache similar to memcached that is suitable for applications running on a single machine. Its major

Patrick Mylund Nielsen 6.4k Aug 6, 2022
NutsDB a simple, fast, embeddable and persistent key/value store written in pure Go.

A simple, fast, embeddable, persistent key/value store written in pure Go. It supports fully serializable transactions and many data structures such as list, set, sorted set.

徐佳军 2.3k Aug 6, 2022
KV - a toy in-memory key value store built primarily in an effort to write more go and check out grpc

KV KV is a toy in-memory key value store built primarily in an effort to write more go and check out grpc. This is still a work in progress. // downlo

Ali Mir 0 Dec 30, 2021
PrimeKV is a Secure, REST API driven Key/Value store.

PrimeKV PrimeKV is a Secure, REST API driven Key/Value store. Features 100% In-memory. Encrypted by default, All stored values are bi-directionally en

Josh Burns 0 Jan 10, 2022