Test your command line interfaces on windows, linux and osx and nodes viá ssh and docker

Overview

Build Status GoDoc Go Report Card Maintainability Test Coverage Github All Releases

Commander

Define language independent tests for your command line scripts and programs in simple yaml files.

  • It runs on windows, osx and linux
  • It can validate local machines, ssh hosts and docker containers
  • It is a self-contained binary - no need to install a heavy lib or language
  • It is easy and fast to write

For more information take a look at the quick start, the examples or the integration tests.

Table of contents

Installation

Any system with Go installed

Probably the easiest way to install commander is by using go get to download and install it in one simple command:

go get github.com/commander-cli/commander/cmd/commander

This works on any OS, as long as go is installed. If go is not installed on your system, follow one of the methods below.

Linux & osx

Visit the release page to get the binary for you system.

curl -L https://github.com/commander-cli/commander/releases/download/v2.3.0/commander-linux-amd64 -o commander
chmod +x commander

Windows

  • Download the current release
  • Add the path to your path environment variable
  • Test it: commander --version

Quick start

A commander test suite consists of a config and tests root element. To start quickly you can use the following examples.

# You can even let commander add tests for you!
$ ./commander add --stdout --file=/tmp/commander.yaml echo hello
tests:
  echo hello:
    exit-code: 0
    stdout: hello

written to /tmp/commander.yaml

# ... and execute!
$ ./commander test /tmp/commander.yaml
Starting test file /tmp/commander.yaml...

✓ echo hello

Duration: 0.002s
Count: 1, Failed: 0

Complete YAML file

Here you can see an example with all features for a quick reference

nodes:
  ssh-host1:
    type: ssh
    addr: 192.168.0.1:22
    user: root
    pass: pass
  ssh-host2:
    type: ssh
    addr: 192.168.0.1:22
    user: root
    identity-file: /home/user/id_rsa.pub
  docker-host1:
    type: docker
    image: alpine:2.4
  docker-host2:
    type: docker
    instance: alpine_instance_1

config: # Config for all executed tests
    dir: /tmp #Set working directory
    env: # Environment variables
        KEY: global
    timeout: 50s # Define a timeout for a command under test
    retries: 2 # Define retries for each test
    nodes:
    - ssh-host1 # define default hosts
    
tests:
    echo hello: # Define command as title
        stdout: hello # Default is to check if it contains the given characters
        exit-code: 0 # Assert exit-code
        
    it should skip:
        command: echo "I should be skipped"
        stdout: I should be skipped
        skip: true
        
    it should fail:
        command: invalid
        stderr:
            contains: 
                - invalid # Assert only contain work
            not-contains:
                - not in there # Validate that a string does not occur in stdout
            exactly: "/bin/sh: 1: invalid: not found"
            line-count: 1 # Assert amount of lines
            lines: # Assert specific lines
                1: "/bin/sh: 1: invalid: not found"
            json:
                object.attr: hello # Make assertions on json objects
            xml:
                "//book//auhtor": Steven King # Make assertions on xml documents
            file: correct-output.txt
        exit-code: 127
        skip: false

    it has configs:
        command: echo hello
        stdout:
            contains: 
              - hello #See test "it should fail"
            exactly: hello
            line-count: 1
        config:
            inherit-env: true # You can inherit the parent shells env variables
            dir: /home/user # Overwrite working dir
            env:
                KEY: local # Overwrite env variable
                ANOTHER: yeah # Add another env variable
            timeout: 1s # Overwrite timeout
            retries: 5
            nodes: # overwrite default nodes
              - docker-host1
              - docker-host2
        exit-code: 0

Executing

# Execute file commander.yaml in current directory
$ ./commander test 

# Execute a specific suite
$ ./commander test /tmp/test.yaml

# Execute a single test
$ ./commander test /tmp/test.yaml --filter "my test"

# Execute suite from stdin
$ cat /tmp/test.yaml | ./commander test -

# Execute suite from url
$ ./commander test https://your-url/commander_test.yaml

# Execute suites within a test directory
$ ./commander test --dir /tmp

# Execute suites in a different working directory 
$ ./commander test --workdir /examples minimal_test.yaml

Adding tests

You can use the add argument if you want to commander to create your tests.

# Add a test to the default commander.yaml
$ ./commander add echo hello
written to /tmp/commander.yaml

# Write to a given file
$ ./commander add --file=test.yaml echo hello
written to test.yaml

# Write to stdout and file
$ ./commander add --stdout echo hello
tests:
  echo hello:
    exit-code: 0
    stdout: hello

written to /tmp/commander.yaml

# Only to stdout
$ ./commander add --stdout --no-file echo hello
tests:
  echo hello:
    exit-code: 0
    stdout: hello

Documentation

Usage

NAME:
   Commander - CLI app testing

USAGE:
   commander [global options] command [command options] [arguments...]

COMMANDS:
     test     Execute the test suite
     add      Automatically add a test to your test suite
     help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --help, -h     show help
   --version, -v  print the version

Tests

Tests are defined in the tests root element. Every test consists of a command and an expected result, i.e. an exit-code.

tests: # root element
  echo test: # test case - can either be the command or a given title
    stdout: test
    exit-code: 0

A test is a map which configures the test. The key (echo test in the example above) of the test can either be the command itself or the title of the test which will be displayed in the test execution.

If the same command is tested multiple times it is useful to set the title of the test manually and use the command property. Further the title can be useful to describe tests better. See the commander test suite as an example.

  • name: title or command under test
  • type: map
  • default: {}

Examples:

tests:
  echo test: # command and title will be the same
    stdout: test
    exit-code: 0
    
  my title: # custom title
    command: exit 1 # set command manually
    exit-code: 1

command

command is a string containing the command to be tested. Further the command property is automatically parsed from the key if no command property was given.

  • name: command
  • type: string
  • default: can't be empty
  • notes: Will be parsed from the key if no command property was provided and used as the title too
echo test: # use command as key and title
  exit-code: 0
  
it should print hello world: # use a more descriptive title...
  command: echo hello world  # ... and set the command in the property manually
  stdout: hello world
  exit-code: 0

config

config sets configuration for the test. config can overwrite global configurations.

  • name: config
  • type: map
  • default: {}
  • notes:
    • for more information look at config
echo test:
  config:
    timeout: 5s

exit-code

exit-code is an int type and compares the given code to the exit-code of the given command.

  • name: exit-code
  • type: int
  • default: 0
exit 1: # will pass
  exit-code: 1
exit 0: # will fail
  exit-code: 1

stdout

stdout and stderr allow to make assertions on the output of the command. The type can either be a string or a map of different assertions.

If only a string is provided it will check if the given string is contained in the output.

  • name: stdout
  • type: string or map
  • default:
  • notes: stderr works the same way
echo test:
  stdout: test # make a contains assertion
  
echo hello world:
  stdout:
    line-count: 1 # assert the amount of lines and use stdout as a map
contains

contains is an array or string. It checks if a string is contained in the output. It is the default if a string is directly assigned to stdout or stderr.

  • name: contains
  • type: string or array
  • default: []
  • notes: default assertion if directly assigned to stdout or stderr
echo hello world:
  stdout: hello # Default is a contains assertion

echo more output:
  stdout:
    contains:
      - more
      - output
exactly

exactly is a string type which matches the exact output.

  • name: exactly
  • type: string
  • default:
echo test:
  stdout:
    exactly: test
json

json is a map type and allows to parse json documents with a given GJSON syntax to query for specific data. The key represents the query, the value the expected value.

  • name: json
  • type: map
  • default: {}
  • notes: Syntax taken from GJSON
cat some.json: # print json file to stdout
  name.last: Anderson # assert on name.last, see document below

some.json file:

{
  "name": {"first": "Tom", "last": "Anderson"},
  "age":37,
  "children": ["Sara","Alex","Jack"],
  "fav.movie": "Deer Hunter",
  "friends": [
    {"first": "Dale", "last": "Murphy", "age": 44, "nets": ["ig", "fb", "tw"]},
    {"first": "Roger", "last": "Craig", "age": 68, "nets": ["fb", "tw"]},
    {"first": "Jane", "last": "Murphy", "age": 47, "nets": ["ig", "tw"]}
  ]
}

More examples queries:

"name.last"          >> "Anderson"
"age"                >> 37
"children"           >> ["Sara","Alex","Jack"]
"children.#"         >> 3
"children.1"         >> "Alex"
"child*.2"           >> "Jack"
"c?ildren.0"         >> "Sara"
"fav\.movie"         >> "Deer Hunter"
"friends.#.first"    >> ["Dale","Roger","Jane"]
"friends.1.last"     >> "Craig"
lines

lines is a map which makes exact assertions on a given line by line number.

  • name: lines
  • type: map
  • default: {}
  • note: starts counting at 1 ;-)
echo test\nline 2:
  stdout:
    lines:
      2: line 2 # asserts only the second line
line-count

line-count asserts the amount of lines printed to the output. If set to 0 this property is ignored.

  • name: line-count
  • type: int
  • default: 0
echo test\nline 2:
  stdout:
    line-count: 2
not-contains

not-contains is a array of elements which are not allowed to be contained in the output. It is the inversion of contains.

  • name: not-contains
  • type: array
  • default: []
echo hello:
  stdout:
    not-contains: bonjour # test passes because bonjour does not occur in the output
 
echo bonjour:
  stdout:
    not-contains: bonjour # test fails because bonjour occurs in the output
xml

xml is a map which allows to query xml documents viá xpath queries. Like the [json][#json] assertion this uses the key of the map as the query parameter to, the value is the expected value.

  • name: xml
  • type: map
  • default: {}
  • notes: Used library xmlquery
cat some.xml:
  stdout:
    xml:
      //book//author: J. R. R. Tolkien

some.xml file:

<book>
    <author>J. R. R. Tolkien</author>
</book>
file

file is a file path, relative to the working directory that will have its entire contents matched against the command output. Other than reading from a file this works the same as exactly.

The example below will always pass.

output should match file:
  command: cat output.txt
  stdout:
    file: output.txt

stderr

See stdout for more information.

  • name: stderr
  • type: string or map
  • default:
  • notes: is identical to stdout
# >&2 echos directly to stderr
">&2 echo error": 
  stderr: error
  exit-code: 0

">&2 echo more errors":
  stderr:
    line-count: 1

skip

skip is a boolean type, setting this field to true will skip the test case.

  • name: skip
  • type: bool
  • default: false
echo test:
  stdout: test
  skip: true

Config

You can add configs which will be applied globally to all tests or just for a specific test case, i.e.:

config:
  dir: /home/root # Set working directory for all tests

tests:
  echo hello:
    config: # Define test specific configs which overwrite global configs
      timeout: 5s
  exit-code: 0

dir

dir is a string which sets the current working directory for the command under test. The test will fail if the given directory does not exist.

  • name: dir
  • type: string
  • default: current working dir
dir: /home/root

env

env is a hash-map which is used to set custom env variables. The key represents the variable name and the value setting the value of the env variable.

  • name: env
  • type: hash-map
  • default: {}
  • notes:
    • read env variables with ${PATH}
    • overwrites inherited variables, see #inherit-env
env:
  VAR_NAME: my value # Set custom env var
  CURRENT_USER: ${USER} # Set env var and read from current env

inherit-env

inherit-env is a boolean type which allows you to inherit all environment variables from your active shell.

  • name: inherit-env
  • type: bool
  • default: false
  • notes: If this config is set to true in the global configuration it will be applied for all tests and ignores local test configs.
inherit-env: true

interval

interval is a string type and sets the interval between retries.

  • name: interval
  • type: string
  • default: 0ns
  • notes:
    • valid time units: ns, us, µs, ms, s, m, h
    • time string will be evaluated by golang's time package, further reading time/#ParseDuration
interval: 5s # Waits 5 seconds until the next try after a failed test is started

retries

retries is an int type and configures how often a test is allowed to fail until it will be marked as failed for the whole test run.

  • name: retries
  • type: int
  • default: 0
  • notes: interval can be defined between retry executions
retries: 3 # Test will be executed 3 times or until it succeeds

timeout

timeout is a string type and sets the time a test is allowed to run. The time is parsed from a duration string like 300ms. If a tests exceeds the given timeout the test will fail.

  • name: timeout
  • type: string
  • default: no limit
  • notes:
    • valid time units: ns, us, µs, ms, s, m, h
    • time string will be evaluated by golang's time package, further reading time/#ParseDuration
timeout: 600s

Nodes

Commander has the option to execute tests against other hosts, i.e. via ssh.

Available node types are currently:

  • local, execute tests locally
  • ssh, execute tests viá ssh
  • docker, execute tests inside a docker container
nodes: # define nodes in the node section
  ssh-host:
    type: ssh # define the type of the connection 
    user: root # set the user which is used by the connection
    pass: password # set password for authentication
    addr: 192.168.0.100:2222 # target host address
    identity-file: ~/.ssh/id_rsa # auth with private key
tests:
  echo hello:
    config:
      nodes: # define on which host the test should be executed
        - ssh-host
    stdout: hello
    exit-code: 0

You can identify on which node a test failed by inspecting the test output. The [local] and [ssh-host] represent the node name on which the test were executed.

✗ [local] it should test ssh host
✗ [ssh-host] it should fail if env could not be set

local

The local node is the default execution and will be applied if nothing else was configured. It is always pre-configured and available, i.e. if you want to execute tests on a node and locally.

nodes:
  ssh-host:
    addr: 192.168.1.100
    user: ...
tests:
  echo hello:
    config:
      nodes: # will be executed on local and ssh-host
        - ssh-host
        - local
    exit-code: 0

ssh

The ssh node type will execute tests against a configured node using ssh.

Limitations:

  • The inhereit-env config is disabled for ssh hosts, nevertheless it is possible to set env variables
  • Private registries are not supported at the moment
nodes: # define nodes in the node section
  ssh-host:
    type: ssh # define the type of the connection 
    user: root # set the user which is used by the connection
    pass: password # set password for authentication
    addr: 192.168.0.100:2222 # target host address
    identity-file: ~/.ssh/id_rsa # auth with private key
tests:
  echo hello:
    config:
      nodes: # define on which host the test should be executed
        - ssh-host
    stdout: hello
    exit-code: 0

docker

The docker node type executes the given command inside a docker container.

Notes: If the default docker registry should be used prefix the container with the registry docker.io/library/

nodes:
  docker-host:
    type: docker
    image: docker.io/library/alpine:3.11.3
    docker-exec-user: 1000 # define the owner of the executed command
    user: user # registry user
    pass: password # registry password, it is recommended to use env variables like $REGISTRY_PASS
config:
  nodes:
    - docker-host
    
tests:
  "id -u":
     stdout: "1001"

Development

See the documentation at development.md

Misc

Heavily inspired by goss.

Similar projects:

Comments
  • Add test against directory

    Add test against directory

    closes #85. I thought I would take a stab at this. I am mainly using this as a learning experience and would love feedback. I thought this would be a good issue to learn the codebase.

    This change will allow a user to execute./commander test examples/ or ./commander test --file-order examples/. Here are the following outputs respectively:

    [email protected] commander (add-test-against-directory) $ ./commander test examples/ 
    Starting test against directory: examples/...
    
    ✓ [minimal_test.yaml] [local] echo hello
    ✓ [environment.yaml] [local] echo $KEY
    ✓ [commander.yaml] [local] echo hello
    ✓ [environment.yaml] [local] it should overwrite env
    ✗ [commander.yaml] [local] it should fail
    ✓ [commander.yaml] [local] it should print hello world
    ✓ [commander.yaml] [local] it should validate exit code
    
    Results
    
    ✗ [commander.yaml] [local] 'it should fail', on property 'Stderr'
    
    Expected
    
    /bin/sh: invalid: command not found
    
    to contain
    
    /bin/sh: 1: invalid: not found
    
    
    Duration: 0.041s
    Count: 7, Failed: 1
    
    Test suite failed, use --verbose for more detailed output
    
    [email protected] commander (add-test-against-directory) $ ./commander test --file-order examples/ 
    Starting test against directory: examples/...
    
    ✓ [commander.yaml] [local] it should print hello world
    ✓ [commander.yaml] [local] it should validate exit code
    ✓ [commander.yaml] [local] echo hello
    ✗ [commander.yaml] [local] it should fail
    ✓ [environment.yaml] [local] echo $KEY
    ✓ [environment.yaml] [local] it should overwrite env
    ✓ [minimal_test.yaml] [local] echo hello
    
    Results
    
    ✗ [commander.yaml] [local] 'it should fail', on property 'Stderr'
    
    Expected
    
    /bin/sh: invalid: command not found
    
    to contain
    
    /bin/sh: 1: invalid: not found
    
    
    Duration: 0.036s
    Count: 7, Failed: 1
    
    Test suite failed, use --verbose for more detailed output
    

    The change also supports testing multiple files for a test that has the same title. Here is an example:

    [email protected] commander (add-test-against-directory) $ ./commander test examples/ "echo hello" 
    Starting test against directory: examples/...
    
    ✓ [commander.yaml] [local] echo hello
    ✓ [minimal_test.yaml] [local] echo hello
    
    Duration: 0.012s
    Count: 2, Failed: 0
    
    File Errors:
    [environment.yaml] could not find test echo hello!
    

    If it's something you would consider merging let me know and I can finish up writing the unit test and integration tests for the change as well as the docs. Just opening this now to get the ball rolling (:

    Thanks!

    Checklist

    • [x] Added unit / integration tests for windows, macOS and Linux?
    • [x] Added a changelog entry in CHANGELOG.md?
    • [x] Updated the documentation (README.md, docs)?
    • [x] Does your change work on Linux, Windows and macOS?
    opened by dylanhitt 18
  • Add --config flag to provide default config to all suites being ran

    Add --config flag to provide default config to all suites being ran

    Closes #136

    Checklist

    • [x] Added unit / integration tests for windows, macOS and Linux?
    • [x] Added a changelog entry in CHANGELOG.md?
    • [x] Updated the documentation (README.md, docs)?
    • [x] Does your change work on Linux, Windows and macOS?
    opened by dylanhitt 15
  • Add file assertion

    Add file assertion

    Fixes #105

    Happy to discuss and make changes!

    Checklist

    • [x] Added unit / integration tests for windows, macOS and Linux?
    • [x] Added a changelog entry in CHANGELOG.md?
    • [x] Updated the documentation (README.md, docs)?
    • [x] Does your change work on Linux, Windows and macOS?
    opened by glesica 13
  • Unable to run docker via kind cluster

    Unable to run docker via kind cluster

    Prerequisites

    • [x] Can you reproduce the problem?
    • [x] Are you running the latest version?
    • [x] Did you perform a cursory search?

    Description

    I am trying to run commander to run custom tool test via docker and kind, its failing on docker path. I tried searching on documentation but not able to resolve the test yet. If I run same docker commands from CLI it does work and lists the clusters successfully docker ps -a --filter label=io.x-k8s.kind.cluster --format '{{.Label "io.x-k8s.kind.cluster"}}'"

    Is there any method to add docker executable to test path?

    Steps to Reproduce

    step 1: ./commander test --verbose Fails with error:

    Error:failed to list kind clusters when bootstrapping failed to list clusters: command "docker ps -a --filter label=io.x-k8s.kind.cluster --format '{{.Label "io.x-k8s.kind.cluster"}}'" failed with error: exec: "docker": executable file not found in $PATH

    Expected behavior: list the kind clusters

    Actual behavior: Stderr throws the error above

    Specifications

    • Version: commander version 2.4.0
    • Platform: darwin arm64
    • Subsystem:

    You can get the version information from executing commander --version.

    opened by d2iqpchavan 11
  • --dir bug

    --dir bug

    Prerequisites

    • [x] Can you reproduce the problem?
    • [x] Are you running the latest version?
    • [x] Did you perform a cursory search?

    Description

    --dir flag doesn't work with commands with shell scripts

    Steps to Reproduce

    1. Unpack the attachment
    2. Execute ./commander test tests/test.yaml Expected behavior: the test should pass Actual behavior: the test passed
    3. Execute ./commander test --dir tests/ Expected behavior: the test should pass Actual behavior:
    Starting test against directory: tests/...
    
    panic: yaml: unmarshal errors:
      line 3: cannot unmarshal !!str `echo "H...` into struct { Tests map[string]suite.YAMLTest "yaml:\"tests\""; Config suite.YAMLTestConfigConf "yaml:\"config\""; Nodes map[string]suite.YAMLNodeConf "yaml:\"nodes\"" }
    
    goroutine 18 [running]:
    github.com/SimonBaeumer/commander/pkg/suite.ParseYAML(0xc00032e000, 0x1a, 0x21a, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    	/home/travis/gopath/src/github.com/SimonBaeumer/commander/pkg/suite/yaml_suite.go:57 +0x2cc
    github.com/SimonBaeumer/commander/pkg/app.testFile(0xc0000260c0, 0xd, 0x0, 0x0, 0xd, 0x0, 0x0)
    	/home/travis/gopath/src/github.com/SimonBaeumer/commander/pkg/app/test_command.go:102 +0xa7
    github.com/SimonBaeumer/commander/pkg/app.testDir.func1(0xc0002e0c00, 0xc000215fc0, 0x2, 0x2, 0x7ffeefbff97f, 0x6, 0xc000080960)
    	/home/travis/gopath/src/github.com/SimonBaeumer/commander/pkg/app/test_command.go:75 +0x19b
    created by github.com/SimonBaeumer/commander/pkg/app.testDir
    	/home/travis/gopath/src/github.com/SimonBaeumer/commander/pkg/app/test_command.go:67 +0x182
    

    dir-bug.zip

    Specifications

    • Version: 2.1.0
    • Platform: Darwin
    • Subsystem: ?
    opened by billychow68 11
  • Output package has too much responsibilities

    Output package has too much responsibilities

    The output/cli.go has more responsibilities than formatting the output.

    • It measures the execution time
    • Counts succeeded and failed tests

    The runtime should be responsible for that, the output should only receive the data and process it.

    opened by SimonBaeumer 10
  • Add --filter flag

    Add --filter flag

    Fixes #137 #77

    Checklist

    • [x] Added unit / integration tests for windows, macOS and Linux?
    • [x] Added a changelog entry in CHANGELOG.md?
    • [x] Updated the documentation (README.md, docs)?
    • [x] Does your change work on Linux, Windows and macOS?
    • [x] Add regex support
    • [ ] (?) Add wildcard support
    • [x] Fix directory support
      • [ ] Maybe print warning that no test was found in file
    opened by SimonBaeumer 9
  • Skip test case

    Skip test case

    closes #127

    Checklist

    • [x] Added unit / integration tests for windows, macOS and Linux?
    • [x] Added a changelog entry in CHANGELOG.md?
    • [x] Updated the documentation (README.md, docs)?
    • [x] Does your change work on Linux, Windows and macOS?
    opened by dylanhitt 8
  • Preserve execution order alphabetically by test title

    Preserve execution order alphabetically by test title

    Fixes #

    Checklist

    • [x] Added unit / integration tests for windows, macOS and Linux?
    • [x] Added a changelog entry in CHANGELOG.md?
    • [x] Updated the documentation (README.md, docs)?
    • [x] Does your change work on Linux, Windows and macOS?
    opened by SimonBaeumer 8
  • Add ability to run tests in sequential order

    Add ability to run tests in sequential order

    While running the tests in parallel speeds up the execution overall, there are times when executing tests in sequential order is needed. Currently, I'm managing this in a shell script calling "Commader YAML TEST" individually.

    I could be wrong but I believe I read somewhere that the --dir will run the test files (YAML) in sequential order? This request is specifically for running the tests sequentially within the YAML.

    opened by billychow68 8
  • Move test execution to runner file, refactor output

    Move test execution to runner file, refactor output

    Closes #121.

    TODO:

    • [x] Migrate runtime.Start to it's own file runner.go
    • [x] Stream TestResults to output in runtime
    • [x] Aggregate TestResults to a single struct Result
    • [x] Add FileName to TestCase struct when executing against a dir
    • [x] Define a single OutputWriter to be used in both Runtime and for out.PrintSummary
    • [x] Test runner.go
    • [x] Converge struct Results in Test_Command when executing against a dir
    • [x] Switch output to go/template

    Checklist

    • [x] Added unit / integration tests for windows, macOS and Linux?
    • [ ] Added a changelog entry in CHANGELOG.md?
    • [ ] Updated the documentation (README.md, docs)?
    • [x] Does your change work on Linux, Windows and macOS?
    refactor 
    opened by dylanhitt 8
  • Support bash and awk verifiers

    Support bash and awk verifiers

    To process and validate a result with custom logic bash and awk verifiers could be an interesting option to write smaller assertions which are more complex.

    opened by SimonBaeumer 1
  • Add proper error handling and coroutine cancellation

    Add proper error handling and coroutine cancellation

    To exit early on errors (i.e. missing addresses) go routines should be cancelled if an error occured. Also all routines should be cancelled gracefully on SIGTERM and SIGINT signals.

    opened by SimonBaeumer 0
  • Fix nil-pointer exception in SSH authentication

    Fix nil-pointer exception in SSH authentication

    ToDo

    • [x] Reproduced issue
    • [ ] Investigate

    Checklist

    • [ ] Added unit / integration tests for windows, macOS and Linux?
    • [ ] Added a changelog entry in CHANGELOG.md?
    • [ ] Updated the documentation (README.md, docs)?
    • [ ] Does your change work on Linux, Windows and macOS?
    opened by SimonBaeumer 4
  • Enable regular expression for stdout, stderr assertion

    Enable regular expression for stdout, stderr assertion

    Prerequisites

    • [x] Are you running the latest version?
    • [x] Did you perform a cursory search?

    Description

    I think it would be nice if we can use regular expression in stdout and stderr assertion, for example:

    tests:
        echo 'There are 30 dogs': 
            stdout:
                lines:
                    1: There (is|are) \d+ dog[s]?
    

    Specifications

    • Version: 2.4
    • Platform: Linux
    • Subsystem: Centos
    opened by pham-anh 3
  • SSH public key authentication broken in commander 2.4.0 on CentOS 8.4.2105

    SSH public key authentication broken in commander 2.4.0 on CentOS 8.4.2105

    Prerequisites

    • [x] Can you reproduce the problem?
    • [x] Are you running the latest version?
    • [x] Did you perform a cursory search?

    Description

    When using SSH public key authentication, commander crashes:

    # commander test test.yml --verbose
    Starting test file test.yml...
    
    panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x79ea53]
    
    goroutine 23 [running]:
    golang.org/x/crypto/ssh.publicKeyCallback.auth(0xc00030a960, 0xc000024240, 0x20, 0x20, 0x0, 0x0, 0xa80aa0, 0xc00034c000, 0xa76980, 0xc0000721b0, ...)
    	/home/travis/gopath/pkg/mod/golang.org/x/[email protected]/ssh/client_auth.go:204 +0xd3
    golang.org/x/crypto/ssh.(*connection).clientAuthenticate(0xc000340080, 0xc00033e000, 0x0, 0xa)
    	/home/travis/gopath/pkg/mod/golang.org/x/[email protected]/ssh/client_auth.go:44 +0x332
    golang.org/x/crypto/ssh.(*connection).clientHandshake(0xc000340080, 0xc0002c8d58, 0x8, 0xc00033e000, 0x0, 0x0)
    	/home/travis/gopath/pkg/mod/golang.org/x/[email protected]/ssh/client.go:113 +0x2b6
    golang.org/x/crypto/ssh.NewClientConn(0xa865a0, 0xc00000e010, 0xc0002c8d58, 0x8, 0xc000140968, 0xa865a0, 0xc00000e010, 0x0, 0x0, 0xc00030a980, ...)
    	/home/travis/gopath/pkg/mod/golang.org/x/[email protected]/ssh/client.go:83 +0xf8
    golang.org/x/crypto/ssh.Dial(0x9b9aeb, 0x3, 0xc0002c8d58, 0x8, 0xc000140968, 0xc00030a980, 0x1, 0x2)
    	/home/travis/gopath/pkg/mod/golang.org/x/[email protected]/ssh/client.go:177 +0xb3
    github.com/commander-cli/commander/pkg/runtime.SSHExecutor.Execute(0xc0002c8d58, 0x8, 0x0, 0x0, 0x0, 0x0, 0xc000308620, 0x15, 0xc0002c8db0, 0xf, ...)
    	/home/travis/gopath/src/github.com/commander-cli/commander/pkg/runtime/ssh_executor.go:76 +0x1d0
    github.com/commander-cli/commander/pkg/runtime.(*Runner).Run.func2(0xc0002c8f40, 0xc00030a8c0, 0xc00006eae0, 0xc00006ea80)
    	/home/travis/gopath/src/github.com/commander-cli/commander/pkg/runtime/runner.go:49 +0x231
    created by github.com/commander-cli/commander/pkg/runtime.(*Runner).Run
    	/home/travis/gopath/src/github.com/commander-cli/commander/pkg/runtime/runner.go:30 +0x113
    

    Using plain username/password authentication works.

    Steps to Reproduce

    1. Create a YML document as seen below
    2. Run commander test test.yml
    3. Watch the beautiful stacktrace :)
    nodes:
      node1:
        type: ssh
        addr: node1:22
        identity-file: /root/.ssh/id_rsa.pub
        #user: root
        #pass: trustno1
    
    config:
      nodes:
        - node1
    
    tests:
      check kurt user:
        command: id kurt
        stdout:
          contains:
            - kurt
          not-contains:
            - no such user
        exit-code: 0
    
    

    Expected behavior: [What you expected to happen]

    SSH login succeeds and test is executed.

    Actual behavior: [What actually happened]

    Application crashes.

    Specifications

    • Version: v2.4.0
    • Platform: CentOS Linux 8.4.2105 (x86_64)
    • Subsystem: ?

    You can get the version information from executing commander --version.

    opened by stdevel 1
Releases(v2.5.0)
Owner
null
Test your code without writing mocks with ephemeral Docker containers 📦 Setup popular services with just a couple lines of code ⏱️ No bash, no yaml, only code 💻

Gnomock – tests without mocks ??️ Spin up entire dependency stack ?? Setup initial dependency state – easily! ?? Test against actual, close to product

Yury Fedorov 935 Sep 17, 2022
Flugel Test Documentation for steps to run and test the automatio

Flugel Test Documentation Documentation for steps to run and test the automation #Test-01 1 - Local Test Using Terratest (End To End) 1- By runing " t

Suc0_Unix 0 Dec 30, 2021
This repository includes consumer driven contract test for provider, unit test and counter api.

This repository includes consumer driven contract test for provider, unit test and counter api.

Ahmet Zümberoğlu 0 Feb 1, 2022
go-test-trace is like go test but it also generates distributed traces.

go-test-trace go-test-trace is like go test but it also generates distributed traces. Generated traces are exported in OTLP to a OpenTelemetry collect

JBD 376 Sep 22, 2022
Test-assignment - Test assignment with golang

test-assignment We have a two steam of data and we need to save it in the map: I

null 0 Jan 19, 2022
Cloud Spanner load generator to load test your application and pre-warm the database before launch

GCSB GCSB Quickstart Create a test table Load data into table Run a load test Operations Load Single table load Multiple table load Loading into inter

Cloud Spanner Ecosystem 21 Jul 13, 2022
Automatically generate Go test boilerplate from your source code.

gotests gotests makes writing Go tests easy. It's a Golang commandline tool that generates table driven tests based on its target source files' functi

Charles Weill 4.2k Sep 24, 2022
Generate a test coverage badge like this one for your go projects.

coverage-badge-go ?? Generate a test coverage badge like this one for your go projects. Usage on: pull_request: branches: -

tj-actions 24 Sep 23, 2022
Fortio load testing library, command line tool, advanced echo server and web UI in go (golang). Allows to specify a set query-per-second load and record latency histograms and other useful stats.

Fortio Fortio (Φορτίο) started as, and is, Istio's load testing tool and now graduated to be its own project. Fortio is also used by, among others, Me

Fortio (Φορτίο) 2.7k Sep 21, 2022
Load generator for measuring overhead generated by EDRs and other logging tools on Linux

Simple load generator for stress-testing EDR software The purpose of this tool is to measure CPU overhead incurred by having active or passive securit

Hilko Bengen 6 Jun 14, 2022
Learning go with tests! Also setting up automated versioning via SemVer.

learn-go-with-tests The purpose of this repo is to complete the learn-go-with-test course located here. I am also interested in learning how to automa

Derek Willingham 0 Nov 23, 2021
A simple mock server configurable via JSON, built using GoLang.

GoMock Server A simple mock server configurable via JSON, built using GoLang. How To A file name endpoint.json must be placed in the context root, wit

Miguel Alexandre 1 Jul 28, 2022
End to end functional test and automation framework

Declarative end to end functional testing (endly) This library is compatible with Go 1.12+ Please refer to CHANGELOG.md if you encounter breaking chan

Viant, Inc 220 Sep 13, 2022
POC for test the idea of Phoenix LiveView in Go and Echo

go-echo-live-view Little POC for test the idea of Phoenix LiveView in Go and Echo (https://github.com/labstack/echo) The idea was stolen from https://

Arturo Elias 11 Jul 10, 2022
Full-featured test framework for Go! Assertions, mocking, input testing, output capturing, and much more! 🍕

testza ?? Testza is like pizza for Go - you could life without it, but why should you? Get The Module | Documentation | Contributing | Code of Conduct

Marvin Wendt 394 Sep 2, 2022
Quick and dirty test to compare MySQL perf between ARM64 & Rosetta MySQL on M1Pro mac

Quick and dirty test to compare MySQL perf between ARM64 & Rosetta MySQL on M1Pro mac

Chris Atkins 0 Nov 5, 2021
Benchmarking deferent Fibonacci functions and algorithms with running unit test

GoFibonacciBench Benchmarking deferent Fibonacci functions and algorithms with running unit test ... Introduction: Fibonacci numbers are special kinds

null 2 Feb 27, 2022
This testing tool surrounds go-ethereum with cannon to catch the blocks of retesteth going into go-ethereum and test cannon with them

Siege This testing tool surrounds go-ethereum with cannon to catch the blocks of retesteth going into go-ethereum and test cannon with them. Usage Sta

Diederik Loerakker 8 Mar 15, 2022
go-carpet - show test coverage in terminal for Go source files

go-carpet - show test coverage for Go source files To view the test coverage in the terminal, just run go-carpet. It works outside of the GOPATH direc

Sergey Mudrik 228 Sep 2, 2022