Go package for running Linux distributed shell commands via SSH.

Related tags

hacktoberfest
Overview

logo

Go Test Go Report Card Go Report Card Go Doc

Description

Go package for running Linux distributed shell commands via SSH.

Contribute

A lot of my focus so far as been on the streaming aspect of this library, which means some updates to Config were done with haste, and as an extension associated checks may have suffered. Therefore, any contributions/improvements are welcome within the scope of the package. Open an issue and let's discuss!

Example:

package main

import (
	"fmt"
	"github.com/discoriver/massh"
	"golang.org/x/crypto/ssh"
)

func main() {
	// Create pointers to config & job
	config := massh.NewConfig()

	job := &massh.Job{
		Command: "echo hello world",
	}

	config.SetHosts([]string{"192.168.1.130", "192.168.1.125"})

	// Password auth
	config.SetPasswordAuth("u01", "password")

	// Key auth in same config. Auth will try all methods provided before failing.
	err := config.SetPrivateKeyAuth("~/.ssh/id_rsa", "")
	if err != nil {
		panic(err)
	}

	config.SetJob(job)
	config.SetWorkerPool(2)
	config.SetSSHHostKeyCallback(ssh.InsecureIgnoreHostKey())

	// Make sure config will run
	config.CheckSanity()

	res, err := config.Run()
	if err != nil {
		panic(err)
	}

	for i := range res {
		fmt.Printf("%s:\n \t OUT: %s \t ERR: %s\n", res[i].Host, res[i].Output, res[i].Error)
	}
}

More examples available in the examples directory.

Usage:

Get the massh package;

go get github.com/DiscoRiver/massh

Documentation

Other

Bastion Host

Specify a bastion host and config with BastionHost and BastionHostSSHConfig in your massh.Config. You may leave BastionHostSSHConfig as nil, in which case SSHConfig will be used instead. The process is automatic, and if BastionHost is not nil, it will be used.

Streaming output

There is an example of streaming output in the direcotry _examples/example_streaming, which contains one method of reading from the results channel, and processing the output.

Running config.Stream() will populate the provided channel with results. Within this, there are two channels within each Result, StdOutStream and StdErrStream, which hold the stdout and stderr pipes respectively. Reading from these channels will give you the host's output/errors.

When a host has completed it's work and has exited, Result.DoneChannel will receive an empty struct. In my example, I use the following function to monitor this and report that a host has finished (see _examples/example_streaming for full program);

func readStream(res Result, wg *sync.WaitGroup) error {
	for {
		select {
		case d := <-res.StdOutStream:
			fmt.Printf("%s: %s", res.Host, d)
		case <-res.DoneChannel:
			fmt.Printf("%s: Finished\n", res.Host)
			wg.Done()
		}
	}
}

Unlike with Config.Run(), which returns a slice of Results when all hosts have exited, Config.Stream() requires some additional values to monitor host completion. For each individual host we have Result.DoneChannel, as explained above, but to detect when all hosts have finished, we have the variable NumberOfStreamingHostsCompleted, which will equal the length of Config.Hosts once everything has completed. Here is an example of what I'm using in _examples/example_streaming;

if NumberOfStreamingHostsCompleted == len(cfg.Hosts) {
		// We want to wait for all goroutines to complete before we declare that the work is finished, as
		// it's possible for us to execute this code before we've finished reading/processing all host output
		wg.Wait()

		fmt.Println("Everything returned.")
		return
}

Right now, the concurrency model used to read from the results channel is the responsibility of those using this package. An example of how this might be achieved can be found in the https://github.com/DiscoRiver/omnivore/tree/main/internal/ossh package, which is currently in development.

Issues
  • Executing instructions in a declaritive manner

    Executing instructions in a declaritive manner

    Amazing work on this project !

    Addressing problem

    If a user wants to execute large set of shell commands. It would annoying to always create a Go program every single time and setting the instructions in an array.

    Solution

    Based on the inspiration of Ansibles. We can set a simple yaml or json file which consists of an array of strings of shell commands to be executed.

    Reason behind this

    I am working a p2p network for executing custom scripts. Currently I am using Ansibles as a plugin system. But it's extremely cluttered and requires external dependencies. If I can implement the following issue. I can use your project as a plugin system to my project. Project link: https://github.com/Akilan1999/p2p-rendering-computation

    opened by Akilan1999 8
  • Adjust how command output is returned

    Adjust how command output is returned

    Currently, command output is only returned after every host has finished executing;

    for r := 0; r < len(c.Hosts); r++ {
    		res = append(res, <-results)
    	}
    
    return res
    

    After implementing this package into a production app, for an environment of around 200 servers, it became clear that this can cause an almost invisible bottleneck, especially if the worker pool is low compared to the number of hosts being touched. I wanted to refer to it as a perceived delay, but it can actually hang up the caller because it is waiting for every host to finish the work and return something.

    There are two requirements I have to improve this;

    • Be able to track how many hosts haven't returned anything, and give some indication on what work is left.
    • Have the caller be able to process command output as a host finishes executing, rather than waiting.
    enhancement 
    opened by DiscoRiver 4
  • When Stream() returns many lines, sometimes chunks of lines are not captured.

    When Stream() returns many lines, sometimes chunks of lines are not captured.

    Intermittently, running stream to capture a reasonable amount of text sometimes misses chunks of lines.

    You can see an example of this here in "Go Test": https://github.com/DiscoRiver/massh/runs/3832011162?check_suite_focus=true

    I was using the command cat /var/log/auth.log in this test, and you'll see the following on line 48;

    do: pam_unix(sudo:session): session opened for user root by (uid=0)
    

    This should be something like what is on line 44;

    Oct  7 20:45:48 fv-az224-778 sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
    

    There doesn't immediately seem to be a pattern, but more investigation is necessary into the concurrency model.

    bug help wanted 
    opened by DiscoRiver 3
  • Ability to run multiple jobs

    Ability to run multiple jobs

    opened by DiscoRiver 2
  • Pre-processing

    Pre-processing

    Issue

    One of the use cases for Massh is querying log files on a distributed environment, to consolidate them into a single filtered log file.

    The way I use this right now is to run a grep command on each remote server, which gathers output and sticks it all in a timestamp-sorted local file. The problem is that some historical logs are archived in tar.gz format, which severely limits simple/easy access to the files within.

    Simplified, the problem is that we need a way to prepare files to be worked on.

    Possible Solutions

    • The obvious solution is to delcare that this isn't a responsibility of the Massh package, and should be handled by providing a shell script as the massh job.
    • Would it make sense to add a pre-processing script function that configures an environment on the target system, and then a separate command that is run on that directory? Having the actual work hidden within a script, rather than set as a massh job explicitly, may be undesireable and reduce the overall readability of the surrounding application. The current thoughts that occur to me of this approach are;
      • Separate exit codes for pre-processing script and command, which may aid in debugging.
      • Improved readability of the application by offloading less work to a shell script.
      • Would increasing the complexity of the massh package be beneficial, over the previous solution of containing all the work within the shell script?
      • Having pre-processing scripts return expected values, such as workingDirectory, may improve interactivity within the application and improve overall readability.

    Most of my hesitation is around how much we're offsetting to shell scripts and how we maintain readability and efficiency within the Go program. Being able to debug and follow exactly what commands are doing is critical to building with massh. I'm also aware that we need to avoid the package scope getting out of control. I'm unsure what the best approach is here right now, but will continue to think.

    enhancement research 
    opened by DiscoRiver 1
  • Example in Documentation not building.

    Example in Documentation not building.

    ./main.go:14:32: not enough arguments in call to config.SetPublicKeyAuth
            have (string)
            want (string, string)
    
    opened by JoePython1 1
  • Update README.md

    Update README.md

    code highlighting for your good repo 👍

    opened by ilyabrin 1
  • Run jobstack bug

    Run jobstack bug

    Fixes #18

    opened by DiscoRiver 0
  • Comment cleanup

    Comment cleanup

    null

    opened by DiscoRiver 0
  • Run() does not return JobStack results correctly.

    Run() does not return JobStack results correctly.

    run (see unexported), does not account for Config.JobStack, and therefore only returns a single job result from each host. Function needs to be updated so that the results channel has the correctly length when JobStack is present. Similar to how runStream functions. 

    bug 
    opened by DiscoRiver 0
Releases(v1.12.0)
Owner
Disco
Go programmer, and functional recluse.
Disco
Tool for shell commands execution, visualization and alerting. Configured with a simple YAML file.

Sampler. Visualization for any shell command. Sampler is a tool for shell commands execution, visualization and alerting. Configured with a simple YAM

Alexander Lukyanchikov 9.1k Oct 24, 2021
An open-source GitLab command line tool bringing GitLab's cool features to your command line

GLab is an open source GitLab CLI tool bringing GitLab to your terminal next to where you are already working with git and your code without switching

Clement Sam 1.5k Oct 25, 2021
A Mighty CLI for AWS

awless is a powerful, innovative and small surface command line interface (CLI) to manage Amazon Web Services. Twitter | Wiki | Changelog Why awless a

WALLIX 4.8k Oct 13, 2021
:cherry_blossom: A command-line fuzzy finder

fzf is a general-purpose command-line fuzzy finder. It's an interactive Unix filter for command-line that can be used with any list; files, command hi

Junegunn Choi 39.7k Oct 15, 2021
A Commander for modern Go CLI interactions

Cobra is both a library for creating powerful modern CLI applications as well as a program to generate applications and command files. Cobra is used i

Steve Francia 23.5k Oct 18, 2021
Go package for running Linux distributed shell commands via SSH.

Go package for running Linux distributed shell commands via SSH.

Disco 112 Oct 22, 2021
A small CLI tool to check connection from a local machine to a remote target in various protocols.

CHK chk is a small CLI tool to check connection from a local machine to a remote target in various protocols.

null 26 Apr 22, 2021
yq is a portable command-line YAML processor

yq a lightweight and portable command-line YAML processor. yq uses jq like syntax but works with yaml files as well as json. It doesn't yet support ev

Mike Farah 4.4k Oct 16, 2021
Commando - run commands against networking devices in batch mode

Commando is a tiny tool that enables users to collect command outputs from a single or a multiple networking devices defined in an inventory file.

Roman Dodin 25 Sep 14, 2021
Command Line Interface for Terraform Enterprise/Cloud ( tecli )

In a world where everything is Terraform, teams use Terraform Cloud API to manage their workloads. TECLI increases teams productivity by facilitating such interaction and by providing easy commands that can be executed on a terminal or on CI/CD systems.

Amazon Web Services - Labs 193 Oct 23, 2021
git-xargs is a command-line tool (CLI) for making updates across multiple Github repositories with a single command.

Table of contents Introduction Reference Contributing Introduction Overview git-xargs is a command-line tool (CLI) for making updates across multiple

Gruntwork 496 Oct 15, 2021
Integrated console application library, using Go structs as commands, with menus, completions, hints, history, Vim mode, $EDITOR usage, and more ...

Gonsole - Integrated Console Application library This package rests on a readline console library, (giving advanced completion, hint, input and histor

null 13 Sep 17, 2021
Kong is a command-line parser for Go

Kong is a command-line parser for Go Introduction Help Help as a user of a Kong application Defining help in Kong Command handling Switch on the comma

Alec Thomas 682 Oct 15, 2021
Source code editor in pure Go.

Editor Source code editor in pure Go. About This is a simple but advanced source code editor As the editor is being developed, the rules of how the UI

Jorge Miguel Pinto 229 Sep 14, 2021
Query, update and convert data structures from the command line. Comparable to jq/yq but supports JSON, TOML, YAML, XML and CSV with zero runtime dependencies.

dasel Dasel (short for data-selector) allows you to query and modify data structures using selector strings. Comparable to jq / yq, but supports JSON,

Tom Wright 1.6k Oct 18, 2021
Bucket-ssh. A fuzzy ssh manager for managing and categorizing ssh connections.

Bssh is an ssh bucket for categorizing and automating ssh connections. Also, with parallel command execution and connection checks(pings) over categories (namespaces).

Furkan Aksoy 13 Sep 13, 2021
📡 ssh into browser tab.

tabssh idk Uses TabFS and gilderlabs/ssh. Set your TabFS mount path in tabssh.go. $ go run tabssh.go and $ ssh -o StrictHostKeyChecking=no localhost

Omar Rizwan 59 Oct 7, 2021
CONTRIBUTIONS ONLY: A Go (golang) command line and flag parser

CONTRIBUTIONS ONLY What does this mean? I do not have time to fix issues myself. The only way fixes or new features will be added is by people submitt

Alec Thomas 3.1k Oct 18, 2021
Query git repositories with SQL. Generate reports, perform status checks, analyze codebases. 🔍 📊

askgit is a command-line tool for running SQL queries on git repositories. It's meant for ad-hoc querying of git repositories on disk through a common interface (SQL), as an alternative to patching together various shell commands.

AskGit 2.6k Oct 15, 2021