A shell parser, formatter, and interpreter with bash support; includes shfmt

Overview

sh

Go Reference

A shell parser, formatter, and interpreter. Supports POSIX Shell, Bash, and mksh. Requires Go 1.14 or later.

Quick start

To parse shell scripts, inspect them, and print them out, see the syntax examples.

For high-level operations like performing shell expansions on strings, see the shell examples.

shfmt

GO111MODULE=on go get mvdan.cc/sh/v3/cmd/shfmt

shfmt formats shell programs. See canonical.sh for a quick look at its default style. For example:

shfmt -l -w script.sh

For more information, see its manpage, which can be viewed directly as Markdown or rendered with scdoc.

Packages are available on Alpine, Arch, Docker, FreeBSD, Homebrew, MacPorts, NixOS, Scoop, Snapcraft, Void and webi.

gosh

GO111MODULE=on go get mvdan.cc/sh/v3/cmd/gosh

Proof of concept shell that uses interp. Note that it's not meant to replace a POSIX shell at the moment, and its options are intentionally minimalistic.

Fuzzing

This project makes use of go-fuzz to find crashes and hangs in both the parser and the printer. The fuzz-corpus branch contains a corpus to get you started. For example:

git checkout fuzz-corpus
./fuzz

Caveats

  • When indexing Bash associative arrays, always use quotes. The static parser will otherwise have to assume that the index is an arithmetic expression.
$ echo '${array[spaced string]}' | shfmt
1:16: not a valid arithmetic operator: string
$ echo '${array[dash-string]}' | shfmt
${array[dash - string]}
  • $(( and (( ambiguity is not supported. Backtracking would complicate the parser and make streaming support via io.Reader impossible. The POSIX spec recommends to space the operands if $( ( is meant.
$ echo '$((foo); (bar))' | shfmt
1:1: reached ) without matching $(( with ))
  • Some builtins like export and let are parsed as keywords. This allows statically building their syntax tree, as opposed to keeping the arguments as a slice of words. Note that this means expansions like declare {a,b}=c are not supported.

JavaScript

A subset of the Go packages are available as an npm package called mvdan-sh. See the _js directory for more information.

Docker

To build a Docker image, checkout a specific version of the repository and run:

docker build -t my:tag -f cmd/shfmt/Dockerfile .

This creates an image that only includes shfmt. Alternatively, if you want an image that includes alpine, add --target alpine. To use the Docker image, run:

docker run --rm -v $PWD:/mnt -w /mnt my:tag <shfmt arguments>

pre-commit

It is possible to use shfmt with pre-commit and a local repo configuration like:

  - repo: local
    hooks:
      - id: shfmt
        name: shfmt
        minimum_pre_commit_version: 2.4.0
        language: golang
        additional_dependencies: [mvdan.cc/sh/v3/cmd/[email protected]]
        entry: shfmt
        args: [-w]
        types: [shell]

Related projects

The following editor integrations wrap shfmt:

Other noteworthy integrations include:

Issues
  • syntax: add mksh support

    syntax: add mksh support

    Initially suggested by @D630 in #77.

    Open questions needed to be answered before we decide whether or not to support it:

    • Where is the formal specification of its syntax? The man page (https://www.mirbsd.org/htman/i386/man1/false.htm) seems to be the only reference.
    • What language features does it have that we don't yet support via either POSIX Shell or Bash?
    • Is it compatible with the POSIX Shell syntax? i.e., do they conflict?
    • Is it compatible with the Bash syntax? i.e., do they conflict?

    And of course, whether or not this is worth the effort. I myself haven't seen any mksh scripts in the wild, whereas sh and bash are both very common.

    enhancement mksh 
    opened by mvdan 45
  • cmd/shfmt: official docker images

    cmd/shfmt: official docker images

    I'd be great if you could provide a docker image on docker hub. My use case is to use it with drone.io, there are other CIs that can use docker images natively. Something like:

    FROM alpine:latest
    ENTRYPOINT ["/usr/bin/shfmt"]
    ADD https://github.com/mvdan/sh/releases/download/v1.2.0/shfmt_v1.2.0_linux_amd64 /usr/bin/shfmt
    RUN chmod +x /usr/bin/shfmt
    

    would work for me.

    And maybe the following for automated builds:

    FROM golang:alpine
    ENTRYPOINT ["shfmt"]
    COPY . /go/src/app
    WORKDIR /go/src/app
    RUN set -ex \
           && apk add --no-cache --virtual .build-deps git \
           && go-wrapper download ./... \
           && go-wrapper install ./... \
           && apk del .build-deps 
    
    help wanted 
    opened by mmlb 42
  • syntax: newline style around &&, || and | configurable in the printer

    syntax: newline style around &&, || and | configurable in the printer

    Right now the code:

    [ -z "$HOME" ] && foo || bar
    [ -z "$HOME" ] &&
        foo ||
        bar
    

    will be formatted:

    [ -z "$HOME" ] && foo || bar
    [ -z "$HOME" ] \
        && foo \
        || bar
    

    Can you add an option or something to control the formatting of the line breaks ?

    enhancement 
    opened by legionus 39
  • syntax: configurable formatting of opening braces

    syntax: configurable formatting of opening braces

    Thanks for the great tool. I have started using this tool with the VS Code. I would like to provide one feedback w.r.t formatting functions (which would at least help couple of us).

    The opening brace of the function definition should be in next line.

    Currently it's as below...

    function fun() {
    }
    

    Suggested way...

    function fun()
    {
    }
    

    Refer examples here: https://www.shellscript.sh/functions.html

    One advantage of following the suggested format is, in the terminal editors like (vim), you can jump between functions by pressing [ and ] character twice. This is very important feature for specially scripts - as once you are done developing the script on your desktop, you would copy/run it on a system where there may not be graphical interfaces or IDEs.

    help wanted 
    opened by abhijithda 32
  • syntax: add option to configure how many statements in a list can fit in a single line

    syntax: add option to configure how many statements in a list can fit in a single line

    Thanks for this tool. In our shell script, we have lines like

     	local cmd="docker build -t $DOCKER_REGISTRY/$IMG:$TAG ."
    	echo $cmd; $cmd
    

    We would like to not break lines like echo $cmd; $cmd into 2 separate lines. I did not see any flag to support this. Can this be supported?

    enhancement help wanted 
    opened by tamalsaha 28
  • cmd/shfmt: add a way to skip custom directories when walking directories

    cmd/shfmt: add a way to skip custom directories when walking directories

    When recursing it would be nice to be able to ignore certain directories or files (for example, node_modules).

    Either a --exclude=node_modules, or --ignore=node_modules, or a .shfmtignore-type functionality would be nice.

    opened by paulbellamy 26
  • Introduce an option to disable changing white space within source lines

    Introduce an option to disable changing white space within source lines

    I'm starting to use shfmt (via Visual Studio Code editor), but one behaviour that prevents me from using it wholeheartedly is the way shfmt changes the whitespace inside source lines. I wonder if there can be an option to disable this behaviour, but it would continue to change the white spaces at the beginning or end of the line.

    The motivation is to not mess up custom alignment of code, creating layout that might be used to help the code read better.

    opened by stevenyvr987 26
  • interp: builtins skip the exec handler

    interp: builtins skip the exec handler

    im sure that this is definitely intentional, but this is actually undesirable in my use case im developing hilbish and use this module to run sh of course, but used to only use this as a last option fallback instead of running my custom builtins or lua code. but now im considering actually using an exec handler for the runner but realized cd wasnt behaving how i expected it to be since it's this module's cd and not mine

    so is there a way for me to change how the builtin works from outside the package? i havent found anything in docs like that

    opened by TorchedSammy 24
  • interp: TestRunnerTerminalExec/Pseudo flaky on Mac

    interp: TestRunnerTerminalExec/Pseudo flaky on Mac

    --- FAIL: TestRunnerTerminalExec (0.00s)
        --- FAIL: TestRunnerTerminalExec/Pseudo (0.14s)
    ##[error]        terminal_test.go:129: signal: hangup
    FAIL
    FAIL	mvdan.cc/sh/v3/interp	4.548s
    

    @theclapp perhaps you could help me reproduce this again? Do you have any ideas?

    Worst case scenario, we can temporarily skip the pty tests on Mac.

    help wanted 
    opened by mvdan 22
  • interp: calling a program with >> on Windows does not append

    interp: calling a program with >> on Windows does not append

    For simple echo it's working, but:

    $ ls >> out
    ls: write error: Bad file descriptor
    

    ls is from Windows Git (C:\Program Files\Git\usr\bin\ls.exe).

    Replace in interp\module.go, fix it, but it's bad implementation:

    var DefaultOpen = ModuleOpen(func(ctx context.Context, path string, flag int, perm os.FileMode) (io.ReadWriteCloser, error) {
    	if flag&os.O_APPEND != 0 {
    		flag = flag & (^os.O_APPEND)
    		f, err := os.OpenFile(path, flag, perm)
    		f.Seek(0, 2)
    		return f, err
    	}
    	return os.OpenFile(path, flag, perm)
    })
    
    help wanted needs info 
    opened by antylama 22
  • Use official Docker GitHub actions

    Use official Docker GitHub actions

    Use the following actions instead of manual Docker buildx setup:

    • https://github.com/docker/login-action
    • https://github.com/docker/setup-qemu-action
    • https://github.com/docker/build-push-action

    Notes:

    • It would be good to merge "Build and push" and "Build and push (alpine)" steps but I have no solution to propose for now;
    • there are some linting changes (using prettier).

    WIP because CI needs to be tested against upstream repo:

    • create a DOCKER_USER secret so that forks can test the CI (push);
    • I am using token and 2FA on DockerHub so the update description step could not be tested.

    Regarding the last step (Update DockerHub description), we may consider using this https://github.com/marketplace/actions/docker-hub-description.

    opened by fauust 19
Releases(v3.5.1)
Owner
Daniel Martí
I work on stuff in Go.
Daniel Martí
Mdfmt - A Markdown formatter that follow the CommonMark. Like gofmt, but for Markdown

Introduction A Markdown formatter that follow the CommonMark. Like gofmt, but fo

杨英明 13 Jun 6, 2022
Parse line as shell words

go-shellwords Parse line as shell words. Usage args, err := shellwords.Parse("./foo --bar=baz") // args should be ["./foo", "--bar=baz"] envs, args, e

mattn 431 Aug 8, 2022
Pryrite, interactively execute shell code blocks in a markdown file

Pryrite Pryrite is a command line tool that interactively runs executable blocks in a markdown file. One can think of pryrite as a console REPL/debugg

Rama Shenai 167 Aug 9, 2022
omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.

omniparser Omniparser is a native Golang ETL parser that ingests input data of various formats (CSV, txt, fixed length/width, XML, EDI/X12/EDIFACT, JS

JF Technology 482 Aug 1, 2022
A simple CSS parser and inliner in Go

douceur A simple CSS parser and inliner in Golang. Parser is vaguely inspired by CSS Syntax Module Level 3 and corresponding JS parser. Inliner only p

Aymerick 219 Jul 15, 2022
Unified diff parser and printer for Go

go-diff Diff parser and printer for Go. Installing go get -u github.com/sourcegraph/go-diff/diff Usage It doesn't actually compute a diff. It only rea

Sourcegraph 367 Aug 5, 2022
Quick and simple parser for PFSense XML configuration files, good for auditing firewall rules

pfcfg-parser version 0.0.1 : 13 January 2022 A quick and simple parser for PFSense XML configuration files to generate a plain text file of the main c

Rory Campbell-Lange 0 Jan 13, 2022
A NMEA parser library in pure Go

go-nmea This is a NMEA library for the Go programming language (Golang). Features Parse individual NMEA 0183 sentences Support for sentences with NMEA

Adrián Moreno 180 Jul 9, 2022
TOML parser for Golang with reflection.

THIS PROJECT IS UNMAINTAINED The last commit to this repo before writing this message occurred over two years ago. While it was never my intention to

Andrew Gallant 3.9k Aug 6, 2022
User agent string parser in golang

User agent parsing useragent is a library written in golang to parse user agent strings. Usage First install the library with: go get xojoc.pw/userage

Alexandru Cojocaru 71 Aug 2, 2021
Simple HCL (HashiCorp Configuration Language) parser for your vars.

HCL to Markdown About To write a good documentation for terraform module, quite often we just need to print all our input variables as a fancy table.

Dmytro Shamenko 15 Dec 14, 2021
A markdown parser written in Go. Easy to extend, standard(CommonMark) compliant, well structured.

goldmark A Markdown parser written in Go. Easy to extend, standards-compliant, well-structured. goldmark is compliant with CommonMark 0.29. Motivation

Yusuke Inuzuka 2.3k Aug 9, 2022
A PDF renderer for the goldmark markdown parser.

goldmark-pdf goldmark-pdf is a renderer for goldmark that allows rendering to PDF. Reference See https://pkg.go.dev/github.com/stephenafamo/goldmark-p

Stephen Afam-Osemene 86 Jul 6, 2022
Experimental parser Angular template

Experimental parser Angular template This repository only shows what a parser on the Go might look like Benchmark 100k line of template Parser ms @ang

Rustam 8 Dec 15, 2021
A dead simple parser package for Go

A dead simple parser package for Go V2 Introduction Tutorial Tag syntax Overview Grammar syntax Capturing Capturing boolean value Streaming Lexing Sta

Alec Thomas 2.5k Aug 11, 2022
Freestyle xml parser with golang

fxml - FreeStyle XML Parser This package provides a simple parser which reads a XML document and output a tree structure, which does not need a pre-de

null 8 Jul 1, 2022
An extension to the Goldmark Markdown Parser

Goldmark-Highlight An extension to the Goldmark Markdown Parser which adds parsing / rendering capabilities for rendering highlighted text. Highlighte

Kevin Zuern 1 May 25, 2022
A parser combinator library for Go.

Takenoco A parser combinator library for Go. Examples CSV parser Dust - toy scripting language Usage Define the parser: package csv import ( "err

shellyln 3 Jul 1, 2022
A simple json parser built using golang

jsonparser A simple json parser built using golang Installation: go get -u githu

Krisna Pranav 1 Dec 29, 2021