Go implementation of the Data At Rest Encryption (DARE) format.

Overview

Godoc Reference Travis CI Go Report Card

Secure IO

Go implementation of the Data At Rest Encryption (DARE) format.

Introduction

It is a common problem to store data securely - especially on untrusted remote storage. One solution to this problem is cryptography. Before data is stored it is encrypted to ensure that the data is confidential. Unfortunately encrypting data is not enough to prevent more sophisticated attacks. Anyone who has access to the stored data can try to manipulate the data - even if the data is encrypted.

To prevent these kinds of attacks the data must be encrypted in a tamper-resistant way. This means an attacker should not be able to:

  • Read the stored data - this is achieved by modern encryption algorithms.
  • Modify the data by changing parts of the encrypted data.
  • Rearrange or reorder parts of the encrypted data.

Authenticated encryption schemes (AE) - like AES-GCM or ChaCha20-Poly1305 - encrypt and authenticate data. Any modification to the encrypted data (ciphertext) is detected while decrypting the data. But even an AE scheme alone is not sufficiently enough to prevent all kinds of data manipulation.

All modern AE schemes produce an authentication tag which is verified after the ciphertext is decrypted. If a large amount of data is decrypted it is not always possible to buffer all decrypted data until the authentication tag is verified. Returning unauthenticated data has the same issues like encrypting data without authentication.

Splitting the data into small chunks fixes the problem of deferred authentication checks but introduces a new one. The chunks can be reordered - e.g. exchanging chunk 1 and 2 - because every chunk is encrypted separately. Therefore the order of the chunks must be encoded somehow into the chunks itself to be able to detect rearranging any number of chunks.

This project specifies a format for en/decrypting an arbitrary data stream and gives some recommendations about how to use and implement data at rest encryption (DARE). Additionally this project provides a reference implementation in Go.

Applications

DARE is designed with simplicity and efficiency in mind. It combines modern AE schemes with a very simple reorder protection mechanism to build a tamper-resistant encryption scheme. DARE can be used to encrypt files, backups and even large object storage systems.

Its main properties are:

  • Security and high performance by relying on modern AEAD ciphers
  • Small overhead - encryption increases the amount of data by ~0.05%
  • Support for long data streams - up to 256 TB under the same key
  • Random access - arbitrary sequences / ranges can be decrypted independently

Install: go get -u github.com/minio/sio

DARE and github.com/minio/sio are finalized and can be used in production.

We also provide a CLI tool to en/decrypt arbitrary data streams directly from your command line:

Install ncrypt: go get -u github.com/minio/sio/cmd/ncrypt && ncrypt -h

Performance

Cipher 8 KB 64 KB 512 KB 1 MB
AES_256_GCM 90 MB/s 1.96 GB/s 2.64 GB/s 2.83 GB/s
CHACHA20_POLY1305 97 MB/s 1.23 GB/s 1.54 GB/s 1.57 GB/s

On i7-6500U 2 x 2.5 GHz | Linux 4.10.0-32-generic | Go 1.8.3 | AES-NI & AVX2

Comments
  • Implement DARE 2.0

    Implement DARE 2.0

    This change adds a DARE 2.0 implementation for en/decrypting io.Reader and io.Writer. Further it adds a generic decrypted reader/writer to handle 1.0/2.0 compatibility.

    • The default for encrypting io.Reader/io.Writer is DARE 2.0
    • The default for decrypting io.Reader/io.Writer is DARE 1.0 and 2.0. (backward compatible)

    This change also separates the DARE implementations from the io.Reader/io.Writer implementations to make reasoning about the code easier. As part of this separation this change adds new test vectors for DARE 1.0 and 2.0.

    Fixes #16

    opened by aead 9
  • ncrypt: read password from file and fix return code when failed

    ncrypt: read password from file and fix return code when failed

    This PR contains two commits:

    1. The first one fixes the return code when fail to read the password
    2. ~~The second one adds the -password-file flag to support reading the password from a file.~~
    3. The second one adds the -p flag to support specifying the password.
    blocked 
    opened by pjw91 5
  • Add CodeQL security scanning

    Add CodeQL security scanning

    Hi, I'm a PM on the GitHub security team. This repository is eligible to try the new GitHub Advanced Security code scanning beta.

    Code scanning runs a static analysis tool called CodeQL which scans your code at build time to find any potential security issues. We've tuned the set of queries to be only the most severe, most precise issues. We'll show alerts in the security tab, and we'll show alerts for any net new vulnerabilities on pull requests as well. We've tried to make this super developer friendly, but we'd love your feedback as we work through the beta.

    If you're interested in trying it out, you can merge this pull request to set up the Actions workflow.

    opened by jhutchings1 1
  • add support for variable payload size

    add support for variable payload size

    This change adds support for custom payload size for encrypted packages. It adds a PayloadSize field to the Config structure which specifies the size of the encrypted payload.

    Fixes #9

    opened by aead 1
  • Add buffer pool and stateful errors

    Add buffer pool and stateful errors

    • Make Reader and Close on Writers errors stateful, so they can't be silently ignored.
    • Add buffer pool, to help small operations.
    • Make benchmarks parallel, so they can be controlled with -cpu param.

    Stateful errors are required for buffers and are good practice anyway.

    benchmark                             old ns/op     new ns/op     delta
    BenchmarkEncryptReader_8KB            7040          1754          -75.09%
    BenchmarkEncryptReader_8KB-32         5711          564           -90.13%
    BenchmarkEncryptReader_64KB           23739         17406         -26.68%
    BenchmarkEncryptReader_64KB-32        8358          1680          -79.90%
    BenchmarkEncryptReader_512KB          143045        135148        -5.52%
    BenchmarkEncryptReader_512KB-32       17855         9144          -48.79%
    BenchmarkEncryptReader_1MB            274863        266411        -3.07%
    BenchmarkEncryptReader_1MB-32         28966         17709         -38.86%
    BenchmarkDecryptReader_8KB            8390          2036          -75.73%
    BenchmarkDecryptReader_8KB-32         5852          700           -88.03%
    BenchmarkDecryptReader_64KB           24062         17358         -27.86%
    BenchmarkDecryptReader_64KB-32        6742          2038          -69.77%
    BenchmarkDecryptReader_512KB          136199        128197        -5.88%
    BenchmarkDecryptReader_512KB-32       17214         9722          -43.52%
    BenchmarkDecryptReader_1MB            261589        256942        -1.78%
    BenchmarkDecryptReader_1MB-32         29842         18367         -38.45%
    BenchmarkDecryptReaderAt_8KB          16163         3108          -80.77%
    BenchmarkDecryptReaderAt_8KB-32       11139         1011          -90.92%
    BenchmarkDecryptReaderAt_64KB         48030         33390         -30.48%
    BenchmarkDecryptReaderAt_64KB-32      12682         2859          -77.46%
    BenchmarkDecryptReaderAt_512KB        135685        122129        -9.99%
    BenchmarkDecryptReaderAt_512KB-32     21756         8825          -59.44%
    BenchmarkDecryptReaderAt_1MB          253436        241940        -4.54%
    BenchmarkDecryptReaderAt_1MB-32       29090         17011         -41.52%
    BenchmarkEncryptWriter_8KB            8110          1721          -78.78%
    BenchmarkEncryptWriter_8KB-32         5590          460           -91.77%
    BenchmarkEncryptWriter_64KB           23016         17731         -22.96%
    BenchmarkEncryptWriter_64KB-32        6519          1212          -81.41%
    BenchmarkEncryptWriter_512KB          128791        125068        -2.89%
    BenchmarkEncryptWriter_512KB-32       9761          8861          -9.22%
    BenchmarkEncryptWriter_1MB            248845        244758        -1.64%
    BenchmarkEncryptWriter_1MB-32         18768         17645         -5.98%
    BenchmarkDecryptWriter_8KB            10045         1721          -82.87%
    BenchmarkDecryptWriter_8KB-32         10071         612           -93.93%
    BenchmarkDecryptWriter_64KB           25237         16298         -35.42%
    BenchmarkDecryptWriter_64KB-32        7284          1506          -79.32%
    BenchmarkDecryptWriter_512KB          130734        122230        -6.50%
    BenchmarkDecryptWriter_512KB-32       14006         8581          -38.73%
    BenchmarkDecryptWriter_1MB            249146        240821        -3.34%
    BenchmarkDecryptWriter_1MB-32         22672         17055         -24.78%
    
    benchmark                             old MB/s     new MB/s     speedup
    BenchmarkEncryptReader_8KB            145.45       583.65       4.01x
    BenchmarkEncryptReader_8KB-32         179.29       1816.31      10.13x
    BenchmarkEncryptReader_64KB           2760.74      3765.09      1.36x
    BenchmarkEncryptReader_64KB-32        7841.21      38999.69     4.97x
    BenchmarkEncryptReader_512KB          3665.21      3879.36      1.06x
    BenchmarkEncryptReader_512KB-32       29363.77     57335.81     1.95x
    BenchmarkEncryptReader_1MB            3814.91      3935.93      1.03x
    BenchmarkEncryptReader_1MB-32         36200.60     59211.25     1.64x
    BenchmarkDecryptReader_8KB            122.05       503.07       4.12x
    BenchmarkDecryptReader_8KB-32         174.98       1461.79      8.35x
    BenchmarkDecryptReader_64KB           2723.60      3775.56      1.39x
    BenchmarkDecryptReader_64KB-32        9720.77      32154.53     3.31x
    BenchmarkDecryptReader_512KB          3849.43      4089.72      1.06x
    BenchmarkDecryptReader_512KB-32       30457.05     53925.25     1.77x
    BenchmarkDecryptReader_1MB            4008.49      4080.99      1.02x
    BenchmarkDecryptReader_1MB-32         35137.48     57088.75     1.62x
    BenchmarkDecryptReaderAt_8KB          63.36        329.47       5.20x
    BenchmarkDecryptReaderAt_8KB-32       91.93        1012.87      11.02x
    BenchmarkDecryptReaderAt_64KB         1364.47      1962.73      1.44x
    BenchmarkDecryptReaderAt_64KB-32      5167.56      22921.02     4.44x
    BenchmarkDecryptReaderAt_512KB        3864.01      4292.92      1.11x
    BenchmarkDecryptReaderAt_512KB-32     24098.26     59407.79     2.47x
    BenchmarkDecryptReaderAt_1MB          4137.44      4334.04      1.05x
    BenchmarkDecryptReaderAt_1MB-32       36046.33     61642.61     1.71x
    BenchmarkEncryptWriter_8KB            126.27       594.86       4.71x
    BenchmarkEncryptWriter_8KB-32         183.17       2226.33      12.15x
    BenchmarkEncryptWriter_64KB           2847.45      3696.05      1.30x
    BenchmarkEncryptWriter_64KB-32        10052.84     54070.81     5.38x
    BenchmarkEncryptWriter_512KB          4070.85      4192.01      1.03x
    BenchmarkEncryptWriter_512KB-32       53709.97     59168.98     1.10x
    BenchmarkEncryptWriter_1MB            4213.78      4284.14      1.02x
    BenchmarkEncryptWriter_1MB-32         55871.68     59426.78     1.06x
    BenchmarkDecryptWriter_8KB            101.94       594.91       5.84x
    BenchmarkDecryptWriter_8KB-32         101.68       1673.98      16.46x
    BenchmarkDecryptWriter_64KB           2596.84      4021.05      1.55x
    BenchmarkDecryptWriter_64KB-32        8996.95      43507.52     4.84x
    BenchmarkDecryptWriter_512KB          4010.34      4289.34      1.07x
    BenchmarkDecryptWriter_512KB-32       37431.90     61100.09     1.63x
    BenchmarkDecryptWriter_1MB            4208.68      4354.17      1.03x
    BenchmarkDecryptWriter_1MB-32         46249.03     61480.32     1.33x
    
    benchmark                             old allocs     new allocs     delta
    BenchmarkEncryptReader_8KB            12             11             -8.33%
    BenchmarkEncryptReader_8KB-32         12             11             -8.33%
    BenchmarkEncryptReader_64KB           12             11             -8.33%
    BenchmarkEncryptReader_64KB-32        12             11             -8.33%
    BenchmarkEncryptReader_512KB          19             18             -5.26%
    BenchmarkEncryptReader_512KB-32       19             18             -5.26%
    BenchmarkEncryptReader_1MB            27             26             -3.70%
    BenchmarkEncryptReader_1MB-32         27             26             -3.70%
    BenchmarkDecryptReader_8KB            18             17             -5.56%
    BenchmarkDecryptReader_8KB-32         18             17             -5.56%
    BenchmarkDecryptReader_64KB           18             17             -5.56%
    BenchmarkDecryptReader_64KB-32        18             17             -5.56%
    BenchmarkDecryptReader_512KB          25             24             -4.00%
    BenchmarkDecryptReader_512KB-32       25             24             -4.00%
    BenchmarkDecryptReader_1MB            33             32             -3.03%
    BenchmarkDecryptReader_1MB-32         33             32             -3.03%
    BenchmarkDecryptReaderAt_8KB          24             22             -8.33%
    BenchmarkDecryptReaderAt_8KB-32       24             22             -8.33%
    BenchmarkDecryptReaderAt_64KB         24             22             -8.33%
    BenchmarkDecryptReaderAt_64KB-32      24             22             -8.33%
    BenchmarkDecryptReaderAt_512KB        29             27             -6.90%
    BenchmarkDecryptReaderAt_512KB-32     29             27             -6.90%
    BenchmarkDecryptReaderAt_1MB          37             35             -5.41%
    BenchmarkDecryptReaderAt_1MB-32       37             35             -5.41%
    BenchmarkEncryptWriter_8KB            12             11             -8.33%
    BenchmarkEncryptWriter_8KB-32         12             11             -8.33%
    BenchmarkEncryptWriter_64KB           12             11             -8.33%
    BenchmarkEncryptWriter_64KB-32        12             11             -8.33%
    BenchmarkEncryptWriter_512KB          19             18             -5.26%
    BenchmarkEncryptWriter_512KB-32       19             18             -5.26%
    BenchmarkEncryptWriter_1MB            27             26             -3.70%
    BenchmarkEncryptWriter_1MB-32         27             26             -3.70%
    BenchmarkDecryptWriter_8KB            14             13             -7.14%
    BenchmarkDecryptWriter_8KB-32         14             13             -7.14%
    BenchmarkDecryptWriter_64KB           14             13             -7.14%
    BenchmarkDecryptWriter_64KB-32        14             13             -7.14%
    BenchmarkDecryptWriter_512KB          21             20             -4.76%
    BenchmarkDecryptWriter_512KB-32       21             20             -4.76%
    BenchmarkDecryptWriter_1MB            29             28             -3.45%
    BenchmarkDecryptWriter_1MB-32         29             28             -3.45%
    
    benchmark                             old bytes     new bytes     delta
    BenchmarkEncryptReader_8KB            74877         1144          -98.47%
    BenchmarkEncryptReader_8KB-32         75206         1549          -97.94%
    BenchmarkEncryptReader_64KB           74877         1144          -98.47%
    BenchmarkEncryptReader_64KB-32        75646         1705          -97.75%
    BenchmarkEncryptReader_512KB          74990         1256          -98.33%
    BenchmarkEncryptReader_512KB-32       75902         1764          -97.68%
    BenchmarkEncryptReader_1MB            75117         1384          -98.16%
    BenchmarkEncryptReader_1MB-32         75868         1712          -97.74%
    BenchmarkDecryptReader_8KB            75125         1392          -98.15%
    BenchmarkDecryptReader_8KB-32         75171         1908          -97.46%
    BenchmarkDecryptReader_64KB           75126         1392          -98.15%
    BenchmarkDecryptReader_64KB-32        75182         2012          -97.32%
    BenchmarkDecryptReader_512KB          75237         1504          -98.00%
    BenchmarkDecryptReader_512KB-32       75347         1709          -97.73%
    BenchmarkDecryptReader_1MB            75366         1633          -97.83%
    BenchmarkDecryptReader_1MB-32         75505         1809          -97.60%
    BenchmarkDecryptReaderAt_8KB          149123        1656          -98.89%
    BenchmarkDecryptReaderAt_8KB-32       149729        2544          -98.30%
    BenchmarkDecryptReaderAt_64KB         149126        1657          -98.89%
    BenchmarkDecryptReaderAt_64KB-32      149804        2221          -98.52%
    BenchmarkDecryptReaderAt_512KB        149255        1784          -98.80%
    BenchmarkDecryptReaderAt_512KB-32     149511        1935          -98.71%
    BenchmarkDecryptReaderAt_1MB          149542        2074          -98.61%
    BenchmarkDecryptReaderAt_1MB-32       150141        2381          -98.41%
    BenchmarkEncryptWriter_8KB            74843         1112          -98.51%
    BenchmarkEncryptWriter_8KB-32         74854         1354          -98.19%
    BenchmarkEncryptWriter_64KB           74883         1144          -98.47%
    BenchmarkEncryptWriter_64KB-32        75237         1206          -98.40%
    BenchmarkEncryptWriter_512KB          77223         3423          -95.57%
    BenchmarkEncryptWriter_512KB-32       80640         6186          -92.33%
    BenchmarkEncryptWriter_1MB            85695         11542         -86.53%
    BenchmarkEncryptWriter_1MB-32         101491        25308         -75.06%
    BenchmarkDecryptWriter_8KB            75002         1272          -98.30%
    BenchmarkDecryptWriter_8KB-32         75029         1720          -97.71%
    BenchmarkDecryptWriter_64KB           75004         1272          -98.30%
    BenchmarkDecryptWriter_64KB-32        75028         1662          -97.78%
    BenchmarkDecryptWriter_512KB          75175         1438          -98.09%
    BenchmarkDecryptWriter_512KB-32       75304         1561          -97.93%
    BenchmarkDecryptWriter_1MB            75468         1730          -97.71%
    BenchmarkDecryptWriter_1MB-32         75857         2029          -97.33%
    
    opened by klauspost 0
  • Add DecryptBuffer for small payloads

    Add DecryptBuffer for small payloads

    Adds a DecryptBuffer which will allow to decrypt smaller buffers without too much overhead.

    There is fallback for the legacy DARE1 which doesn't benefit too much from this, but that can be added later if needed.

    opened by klauspost 0
  • add `DecryptReaderAt` implementation

    add `DecryptReaderAt` implementation

    This commit adds a ReaderAt implementation that allows decrypting a data using random access patterns.

    With DecryptReaderAt it is possible to decrypt an io.ReaderAt and reading from arbitrary (plaintext) offsets.

    opened by aead 0
  • select AES-GCM by default on s390x

    select AES-GCM by default on s390x

    This commit removes the internal cpu package and uses x/sys/cpu instead. Additionally AES-GCM is now selected by default on s390x.

    This commit also introduces go module support.

    opened by aead 0
  • add an error type representing decryption failure

    add an error type representing decryption failure

    This commit adds an error type: sio.Error. It is returned by an io.Reader / io.Writer if the decryption failed. A type check can be used to distingush a decryption error from other reading/writing errors like network errors, short buffer errors, ...

    Fixes #24

    opened by aead 0
  • prepare v1.0 codebase for v1.1/v2.0 migration

    prepare v1.0 codebase for v1.1/v2.0 migration

    This change prepares the current code for v1.1/v2.0 migration. Therefore the current reader/writer code is moved to properly named files and the existing code is tweaked to support multiple (non-backward compatible) versions.

    Updates #16

    opened by aead 0
  • add CLI tool ncrypt

    add CLI tool ncrypt

    This change adds a CLI encryption tool: ncrypt. ncrypt uses the sio library to encrypt arbitrary streams. This change adds just basic support for the core features.

    Update README.md mentioning ncrypt.

    feature 
    opened by aead 0
Owner
Object Storage for the Era of the Hybrid Cloud
MinIO’s high performance, Kubernetes-native object storage suite is built for the demands of the hybrid cloud.
Object Storage for the Era of the Hybrid Cloud
The minilock file encryption system, ported to pure Golang. Includes CLI utilities.

Go-miniLock A pure-Go reimplementation of the miniLock asymmetric encryption system. by Cathal Garvey, Copyright Oct. 2015, proudly licensed under the

Cathal Garvey 172 Sep 21, 2022
An easy-to-use XChaCha20-encryption wrapper for io.ReadWriteCloser (even lossy UDP) using ECDH key exchange algorithm, ED25519 signatures and Blake3+Poly1305 checksums/message-authentication for Go (golang). Also a multiplexer.

Quick start Prepare keys (on both sides): [ -f ~/.ssh/id_ed25519 ] && [ -f ~/.ssh/id_ed25519.pub ] || ssh-keygen -t ed25519 scp ~/.ssh/id_ed25519.pub

null 26 Sep 27, 2022
DERO Homomorphic Encryption Blockchain Protocol

Homomorphic encryption is a form of encryption allowing one to perform calculations on encrypted data without decrypting it first. The result of the computation is in an encrypted form, when decrypted the output is the same as if the operations had been performed on the unencrypted data.

null 104 Sep 22, 2022
Sekura is an Encryption tool that's heavily inspired by the Rubberhose file system.

It allows for multiple, independent file systems on a single disk whose existence can only be verified if you posses the correct password.

null 51 Feb 1, 2022
A document encryption solution for the reMarkable 2 ePaper tablet.

Remarkable 2 Encryption This repository contains multiple tools to encrypt the home folder of the reMarkable 2 epaper tablet using gocryptfs. Detailed

RedTeam Pentesting GmbH 30 Sep 24, 2022
A super easy file encryption utility written in go and under 800kb

filecrypt A super easy to use file encryption utility written in golang ⚠ Help Wanted on porting filecrypt to other programing languages NOTE: if you

Flew Software 79 Aug 27, 2022
Encryption Abstraction Layer and Utilities for ratnet

What is Bencrypt? Bencrypt is an abstraction layer for cryptosystems in Go, that lets applications use hybrid cryptosystems without being coupled to t

null 17 Jul 23, 2022
A simple, semantic and developer-friendly golang package for encoding&decoding and encryption&decryption

A simple, semantic and developer-friendly golang package for encoding&decoding and encryption&decryption

null 282 Sep 26, 2022
Encryption & Decryption package for golang

encdec Encryption & Decryption package for golang func main() { startingTime := time.Now() privKey, pubKey := GenerateRsaKeyPair() fmt.Println("Priva

MD MOSTAIN BILLAH 3 Feb 11, 2022
A simple, modern and secure encryption tool (and Go library) with small explicit keys, no config options, and UNIX-style composability.

A simple, modern and secure encryption tool (and Go library) with small explicit keys, no config options, and UNIX-style composability.

Filippo Valsorda 11.7k Oct 2, 2022
Easy to use encryption library for Go

encryptedbox EncryptedBox is an easy to use module for Go that can encrypt or sign any type of data. It is especially useful when you must serialize y

Jesse Swidler 17 Jul 20, 2022
A tool for secrets management, encryption as a service, and privileged access management

Deploy HCP Vault & AWS Transit Gateways via Terraform https://medium.com/hashicorp-engineering/deploying-hcp-vault-using-the-hcp-terraform-provider-5e

Temur Yunusov 0 Nov 23, 2021
TTAK.KO-12.0223 Lightweight Encryption Algorithm with Galois/Counter Mode (LEA-GCM)

LEACrypt The Lightweight Encryption Algorithm (also known as LEA) is a 128-bit block cipher developed by South Korea in 2013 to provide confidentialit

Pedro F. Albanese 0 Dec 28, 2021
Functional encryption for images

ImageFE Functional encryption for images. Introduction In the traditional cryptography framework, a decryptor either recovers the entire plaintext fro

null 3 Mar 8, 2022
Attempts to make attribute based encryption work, particularly trying out bn256 pairing curve

EC Pairings over bn256 This is an attempt to solve the core problem of attribute based encryption, where the goal is to be able to use CA-issued attri

Robert Fielding 1 Jan 5, 2022
Go Encrypt! Is a simple command-line encryption and decryption application using AES-256 GCM.

Go Encrypt! Go Encrypt! is a command-line application used to easily encrypt and decrypt files with the AES-256 GCM encryption algorithm. Usage Usage

Peter Georgas 0 Jan 5, 2022
Lattigo: lattice-based multiparty homomorphic encryption library in Go

Lattigo: lattice-based multiparty homomorphic encryption library in Go Lattigo i

null 1 Aug 17, 2022
Length-preserving encryption algorithm

hctr2 Length-preserving encryption algorithm https://eprint.iacr.org/2021/1441.pdf Security Disclosure This project uses full disclosure. If you find

Eric Lagergren 2 Feb 28, 2022
Card-encrypt - The encryption code necessary to enroll debit cards in the Palla API

?? Card RSA Encryption Thank you for choosing Palla! ?? In this repository you'l

palla 0 Jul 15, 2022