Stack Up is a simple deployment tool that performs given set of commands on multiple hosts in parallel.

Related tags

Command Line sup
Overview

Stack Up

Stack Up is a simple deployment tool that performs given set of commands on multiple hosts in parallel. It reads Supfile, a YAML configuration file, which defines networks (groups of hosts), commands and targets.

Demo

Sup

Note: Demo is based on this example Supfile.

Installation

$ go get -u github.com/pressly/sup/cmd/sup

Usage

$ sup [OPTIONS] NETWORK COMMAND [...]

Options

Option Description
-f Supfile Custom path to Supfile
-e, --env=[] Set environment variables
--only REGEXP Filter hosts matching regexp
--except REGEXP Filter out hosts matching regexp
--debug, -D Enable debug/verbose mode
--disable-prefix Disable hostname prefix
--help, -h Show help/usage
--version, -v Print version

Network

A group of hosts.

# Supfile

networks:
    production:
        hosts:
            - api1.example.com
            - api2.example.com
            - api3.example.com
    staging:
        # fetch dynamic list of hosts
        inventory: curl http://example.com/latest/meta-data/hostname

$ sup production COMMAND will run COMMAND on api1, api2 and api3 hosts in parallel.

Command

A shell command(s) to be run remotely.

# Supfile

commands:
    restart:
        desc: Restart example Docker container
        run: sudo docker restart example
    tail-logs:
        desc: Watch tail of Docker logs from all hosts
        run: sudo docker logs --tail=20 -f example

$ sup staging restart will restart all staging Docker containers in parallel.

$ sup production tail-logs will tail Docker logs from all production containers in parallel.

Serial command (a.k.a. Rolling Update)

serial: N constraints a command to be run on N hosts at a time at maximum. Rolling Update for free!

# Supfile

commands:
    restart:
        desc: Restart example Docker container
        run: sudo docker restart example
        serial: 2

$ sup production restart will restart all Docker containers, two at a time at maximum.

Once command (one host only)

once: true constraints a command to be run only on one host. Useful for one-time tasks.

# Supfile

commands:
    build:
        desc: Build Docker image and push to registry
        run: sudo docker build -t image:latest . && sudo docker push image:latest
        once: true # one host only
    pull:
        desc: Pull latest Docker image from registry
        run: sudo docker pull image:latest

$ sup production build pull will build Docker image on one production host only and spread it to all hosts.

Local command

Runs command always on localhost.

# Supfile

commands:
    prepare:
        desc: Prepare to upload
        local: npm run build

Upload command

Uploads files/directories to all remote hosts. Uses tar under the hood.

# Supfile

commands:
    upload:
        desc: Upload dist files to all hosts
        upload:
          - src: ./dist
            dst: /tmp/

Interactive Bash on all hosts

Do you want to interact with multiple hosts at once? Sure!

# Supfile

commands:
    bash:
        desc: Interactive Bash on all hosts
        stdin: true
        run: bash
$ sup production bash
#
# type in commands and see output from all hosts!
# ^C

Passing prepared commands to all hosts:

$ echo 'sudo apt-get update -y' | sup production bash

# or:
$ sup production bash <<< 'sudo apt-get update -y'

# or:
$ cat <<EOF | sup production bash
sudo apt-get update -y
date
uname -a
EOF

Interactive Docker Exec on all hosts

# Supfile

commands:
    exec:
        desc: Exec into Docker container on all hosts
        stdin: true
        run: sudo docker exec -i $CONTAINER bash
$ sup production exec
ps aux
strace -p 1 # trace system calls and signals on all your production hosts

Target

Target is an alias for multiple commands. Each command will be run on all hosts in parallel, sup will check return status from all hosts, and run subsequent commands on success only (thus any error on any host will interrupt the process).

# Supfile

targets:
    deploy:
        - build
        - pull
        - migrate-db-up
        - stop-rm-run
        - health
        - slack-notify
        - airbrake-notify

$ sup production deploy

is equivalent to

$ sup production build pull migrate-db-up stop-rm-run health slack-notify airbrake-notify

Supfile

See example Supfile.

Basic structure

# Supfile
---
version: 0.4

# Global environment variables
env:
  NAME: api
  IMAGE: example/api

networks:
  local:
    hosts:
      - localhost
  staging:
    hosts:
      - stg1.example.com
  production:
    hosts:
      - api1.example.com
      - api2.example.com

commands:
  echo:
    desc: Print some env vars
    run: echo $NAME $IMAGE $SUP_NETWORK
  date:
    desc: Print OS name and current date/time
    run: uname -a; date

targets:
  all:
    - echo
    - date

Default environment variables available in Supfile

  • $SUP_HOST - Current host.
  • $SUP_NETWORK - Current network.
  • $SUP_USER - User who invoked sup command.
  • $SUP_TIME - Date/time of sup command invocation.
  • $SUP_ENV - Environment variables provided on sup command invocation. You can pass $SUP_ENV to another sup or docker commands in your Supfile.

Running sup from Supfile

Supfile doesn't let you import another Supfile. Instead, it lets you run sup sub-process from inside your Supfile. This is how you can structure larger projects:

./Supfile
./database/Supfile
./services/scheduler/Supfile

Top-level Supfile calls sup with Supfiles from sub-projects:

 restart-scheduler:
    desc: Restart scheduler
    local: >
      sup -f ./services/scheduler/Supfile $SUP_ENV $SUP_NETWORK restart
 db-up:
    desc: Migrate database
    local: >
      sup -f ./database/Supfile $SUP_ENV $SUP_NETWORK up

Common SSH Problem

if for some reason sup doesn't connect and you get the following error,

connecting to clients failed: connecting to remote host failed: Connect("[email protected]"): ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

it means that your ssh-agent dosen't have access to your public and private keys. in order to fix this issue, follow the below instructions:

  • run the following command and make sure you have a key register with ssh-agent
ssh-add -l

if you see something like The agent has no identities. it means that you need to manually add your key to ssh-agent. in order to do that, run the following command

ssh-add ~/.ssh/id_rsa

you should now be able to use sup with your ssh key.

Development

fork it, hack it..

$ make build

create new Pull Request

We'll be happy to review & accept new Pull Requests!

License

Licensed under the MIT License.

Issues
  • Bad file descriptor on localhost

    Bad file descriptor on localhost

    Config:

    env:
      NAME: sup
    
    networks: # Groups of hosts
      local:
        hosts:
          - localhost
      production:
        hosts:
          - [email protected]
    commands: # Named set of commands to be run remotely
      ping:
        desc: Print current date/time.
        run: date
    

    Error on localhost :

    $  sup local ping
    [email protected] | + date
    2015/12/01 12:09:23 [email protected]: read |0: bad file descriptor
    

    On production host works fine:

    $ sup production ping
    [email protected]:22 | + date
    [email protected]:22 | Tue Dec  1 12:09:37 MSK 2015
    
    opened by lexa-uw 32
  • Run a local command

    Run a local command

    Could I run a local command as part of my set of commands ? For instance: I need to build a distribution package of my frontend in local machine and after that i need to upload to start the deploy.

    Something like:

      prepare:
        desc: Prepare to upload
        run: npm run build
        local: true
    
      upload:
        desc: Upload this repository
        upload:
          - src: ./
            dst: /tmp/$IMAGE
    
    targets:
      deploy:
        - prepare
        - upload
        .
        .
        .
    

    So, after passing the local : true to my task, I'm sure that will be executed locally before the upload.

    opened by eduardonunesp 23
  • Deploy multiples Sups

    Deploy multiples Sups

    Would be cool if sup have an option to work with "sub sups", so you can organize multiple deploys with only one command, for instance:

    .
    ├── Supfile
    ├── project1
    │   ├── Dockerfile
    │   ├── Supfile
    │   ├── example.dev.cfg
    │   ├── example.go
    │   ├── example.local.cfg
    │   ├── example.prod.cfg
    │   ├── example.stg.cfg
    │   └── scripts
    ├── project2
    │   ├── Dockerfile
    │   ├── Supfile
    │   ├── example.dev.cfg
    │   ├── example.go
    │   ├── example.local.cfg
    │   ├── example.prod.cfg
    │   ├── example.stg.cfg
    │   └── scripts
    └── project3
        ├── Dockerfile
        ├── Supfile
        ├── example.dev.cfg
        ├── example.go
        ├── example.local.cfg
        ├── example.prod.cfg
        ├── example.stg.cfg
        └── scripts
    
    opened by eduardonunesp 15
  • Add default Env variable SUP_RELEASE

    Add default Env variable SUP_RELEASE

    The problem

    Naming a release consistently across hosts is hard (since time might be off between hosts).

    The proposed solution

    Introduce SUP_RELEASE as a "global" variable like SUP_NETWORK. The idea is stolen from Capistrano.

    SUP_RELEASE would be assumed to be unique and be time ordered.

    opened by stengaard 14
  • Print usage items (networks/targets/commands) in order

    Print usage items (networks/targets/commands) in order

    When printing usage, networkUsage and cmdUsage iterate the maps directly, which may lead to different outputs (not in a same order) every time.

    1485307471

    This PR just sorts the names of available networks/targets/commands before iterating and printing them so that they are listed by alphabetical order.

    opened by zyguan 9
  • [WIP] Add sudo option

    [WIP] Add sudo option

    With this

    commands:
      whoami:
        sudo: true
        run:  whoami
    

    would output root.

    Makes sup prompt for a sudo password and wrap the command in sudo -S. Will only prompt for password once.

    If you can do passwordless sudo in your stack you should still use run: sudo my-cmd.sh instead of this feature.

    Inspiration from fabric and ansible

    opened by stengaard 9
  • Dynamic hosts list from command or http endpoint

    Dynamic hosts list from command or http endpoint

    ie..

    # Some Supfile
    
    ---
    
    env:
      NAME: x
    
    networks:
      production:
        hosts: http://discovery.etcd.io/asdfasdf/hosts
    
    commands:
      .....
    

    the hosts expects hosts on newlines, and will use that list for the present time. This is useful to automated environments. We could also allow hosts: $ command

    opened by pkieltyka 8
  • Upload issues

    Upload issues

    Hello thanks for your awesome project.

    I had an issues with upload:

    1. if src is not present I had no errors
    2. -debug key didn't change this behaviour
    3. there should be a message that something was(or not) copied
    4. currently I see only tar (not so usefull) messages: tar: .config/fish/fundle/edc/bass/.git: time stamp 2017-06-20 15:47:40 is 199.705320196 s in the future
    5. upload doesn't resolve local home dir using tilde ~ (As I found $HOME works, but...)
    6. Relative path issue
        upload:
          - src: ../.config/fish
            dst: /tmp
    
    #will copy into /tmp/.config/fish
    #but:
    cp -r ../.config/fish /tmp
    #will copy contents into /tmp/fish which is default behavior for unix
    

    sup -v                                                                  
    # 0.5
    uname -a
    # Linux war 4.11.5-1-ARCH #1 SMP PREEMPT Wed Jun 14 16:19:27 CEST 2017 x86_64 GNU/Linux
    go version
    # go1.8.3 linux/amd64
    

    Thanks

    opened by slavaGanzin 7
  • Installation failing on Ubuntu 14.04 machine

    Installation failing on Ubuntu 14.04 machine

    When I try to install following the instructions on the project page, the installation fails with:

    # github.com/pressly/sup/vendor/golang.org/x/crypto/curve25519
    go/src/github.com/pressly/sup/vendor/golang.org/x/crypto/curve25519/doc.go:15: undefined: scalarMult
    

    My go version is 1.6.2

    opened by lym 7
  • [WIP] Added support to RSYNC upload

    [WIP] Added support to RSYNC upload

    Now you are able to upload your files using rsync, is similar to the upload method, but uses rsync command instead. The flags at command are: -a for archive mode and -c to check files that need to be sync by checking the hash of the file.

    Example of config:

    version: 0.3
    
    env:
      TMP_DIR: /tmp/mydir
    
    networks:
      local:
        hosts:
          - localhost
    
      dev:
        hosts:
          - [email protected]
          - [email protected]
    
    commands:
      pre-build:
        run: mkdir $TMP_DIR; exit 0
    
      build:
        rsync:
          - src: ./src/
            dst: $TMP_DIR
    
      test:
        run: $TMP_DIR/test.sh
    
    targets:
      deploy:
        - pre-build
        - build
        - test
    
    opened by eduardonunesp 7
  • Error

    Error "panic: runtime error: slice bounds out of range" when using option --sshconfig

    Hello,

    when I tried the --sshconfig option I got an error as below:

    $ sup --sshconfig=~/.ssh/config vps ping     
    panic: runtime error: slice bounds out of range
    
    goroutine 1 [running]:
    main.resolvePath(0x0, 0x0, 0xc4200165b0, 0xc)
    	/Users/vojtechvitek/go/src/github.com/pressly/sup/cmd/sup/main.go:191 +0x105
    main.main()
    	/Users/vojtechvitek/go/src/github.com/pressly/sup/cmd/sup/main.go:304 +0xd1d
    

    Am I missing something, or this is a bug?

    btw, sup --help show this:

      -sshconfig string
        	Read SSH Config file, ie. ~/.ssh/config file
    

    Note that there are only 1 hyphen. I tried with both 1 & 2 hyphens and the result is the same.

    opened by hth2 6
  • Add SUPFILE_DIR as default environment variable

    Add SUPFILE_DIR as default environment variable

    This will allow Supfiles to be run different directories and consistently resolve files relative to the Supfile.

    This CL contains refactors the resolution order of environment variables such that it's a bit easier to reason about.

    Fixes #99

    opened by stengaard 1
  • ssh: Specify HostKeyCallback in ClientConfig

    ssh: Specify HostKeyCallback in ClientConfig

    Since golang/go#19767 SSH library requires this field on each connection.

    Currently all new installations of sup via go get doesn't work at all. This commit fixes it.

    opened by zimnx 1
  • Provide user feedback for common SSH configuration issues

    Provide user feedback for common SSH configuration issues

    This PR introduces error messages to clarify some regular issues I encounter:

    • Sup always tries to connect to a SSH_AUTH_SOCK even if one is not defined
    • Sup silently continues if a defined SSH_AUTH_SOCK is unreachable
    • Sup doesn't report any permissions issues preventing keys from being read
    • Sup can't currently handle passphrase-encrypted keys but produces no warning about this

    Dependencies needed to be updated to accomplish this last task, and a small change was required to maintain the existing (lack of) SSH hostname check behavior.

    opened by qguv 0
Releases(v0.5.3)
A CLI tool for working with CloudWatch logs. It performs functions that I need at work.

CloudWatch Logs Utility A simple utility for working with CloudWatch Logs. AWS should probably build this themselves, but since they won't, I am here

J. Alexander Curtis 0 Dec 31, 2021
Inspect-descriptor-set - Example protobuf descriptor set inspector CLI tool

Quick little example of parsing a protobuf descriptor file. ❯ go run main.go -f

Daniel Selans 0 Jan 25, 2022
painless task queue manager for shell commands with an intuitive cli interface (execute shell commands in distributed cloud-native queue manager).

EXEQ DOCS STILL IN PROGRESS. Execute shell commands in queues via cli or http interface. Features Simple intuitive tiny cli app. Modular queue backend

Mohammed Al Ashaal 12 Jan 29, 2022
Hosty is a command-line utility that allows for fast inspection and editing of /etc/hosts-like files

Hosty Description Hosty is a command-line utility that allows for fast inspection and editing of /etc/hosts-like files. It is written in golang and us

null 11 Sep 3, 2021
Go library and CLI utility for /etc/hosts management.

Etc Hosts Management Utility & Go Library /etc/hosts Management It is easy to open your /etc/hosts file in text editor and add or remove entries. Howe

txn2: Kubernetes apps and system utilities. 265 Jun 21, 2022
Hostover is a hosts file poisoning script written in Go

Hostover - Hosts file Poisoner Hostover is a hosts file poisoning script written in Go It will poison the hosts file with fake IP-Domain connections,

null 3 Jan 1, 2022
Go-test-app - Test application to verify environment deployment and reachability over HTTP

Test app Test application to verify environment deployment and reachability over

Universal Development 2 May 23, 2022
Tool for shell commands execution, visualization and alerting. Configured with a simple YAML file.

Sampler. Visualization for any shell command. Sampler is a tool for shell commands execution, visualization and alerting. Configured with a simple YAM

Alexander Lukyanchikov 10.3k Jun 26, 2022
Dependency-free replacement for GNU parallel, perfect fit for usage in an initramfs.

coshell v0.2.5 A no-frills dependency-free replacement for GNU parallel, perfect for initramfs usage. Licensed under GNU/GPL v2. How it works An sh -c

gdm85 41 May 11, 2022
A parallel downloader with resume capability

Grozilla The Grozilla is a simple implementation that allows downloading of video,audio,package or zip files parallely and efficiently using light wei

Prashant Agarwala 39 May 5, 2022
miscellaneous useful commands, including 'gosh' the Go scripting tool

utilities Miscellaneous useful commands. gosh This is a tool for running Go code from the command line. See here. findCmpRm This finds files with copi

Nick Wells 23 Jun 15, 2022
ag is a tool for defining an alias for a group of commands

AG Introduction ag is a command line tool that similar to Makefile. with ag you can make an alias for group of commands with custom flags. This tool i

Mostafa Hosseini 4 Nov 30, 2021
Command-line tool to load csv and excel (xlsx) files and run sql commands

csv-sql supports loading and saving results as CSV and XLSX files with data processing with SQLite compatible sql commands including joins.

Dhamith Hewamullage 22 May 6, 2022
A CLI tool for running Go commands with colorized output

Goli Goli is a CLI Tool for running Go commands with colorized output. Note: Goli is still a WIP. It has very basic commands and limitations. Feel fre

Arthur Diniz 14 May 3, 2022
Cobra CLI tool to generate applications and commands

Cobra Generator Cobra provides its own program that will create your application and add any commands you want. It's the easiest way to incorporate Co

Steve Francia 146 Jun 21, 2022
🔍 Find stack overflow questions from your favorite terminal!

Stack Overflow Cli (Stovc) Stack Overflow Cli is a CLI tool that helps you to search a question in stack overflow in your terminal. Installation Downl

Kedi 3 Nov 20, 2021
This is an concurrent-queue and concurrent-stack lib for the go.

This is an concurrent-queue and concurrent-stack lib for the go. Getting Started Pull in the dependency go get github.com/boobusy/vector Add the impor

白衣画甲 0 Jan 2, 2022
Alex 1 Feb 12, 2022
CLI tool to upload object to s3-compatible storage backend and set download policy for it.

typora-s3 CLI tool to upload object to s3-compatible storage backend and set download policy for it. Build $ git clone https://github.com/fengxsong/ty

fengxsong 0 Dec 29, 2021