Package for downloading things from a string URL using a variety of protocols.

Related tags

Network go-getter


CircleCI Go Documentation

go-getter is a library for Go (golang) for downloading files or directories from various sources using a URL as the primary form of input.

The power of this library is being flexible in being able to download from a number of different sources (file paths, Git, HTTP, Mercurial, etc.) using a single string as input. This removes the burden of knowing how to download from a variety of sources from the implementer.

The concept of a detector automatically turns invalid URLs into proper URLs. For example: "" would turn into a Git URL. Or "./foo" would turn into a file URL. These are extensible.

This library is used by Terraform for downloading modules and Nomad for downloading binaries.

Installation and Usage

Package documentation can be found on GoDoc.

Installation can be done with a normal go get:

$ go get

go-getter also has a command you can use to test URL strings:

$ go install

$ go-getter ./foo

The command is useful for verifying URL structures.

URL Format

go-getter uses a single string URL as input to download from a variety of protocols. go-getter has various "tricks" with this URL to do certain things. This section documents the URL format.

Supported Protocols and Detectors

Protocols are used to download files/directories using a specific mechanism. Example protocols are Git and HTTP.

Detectors are used to transform a valid or invalid URL into another URL if it matches a certain pattern. Example: "" is automatically transformed into a fully valid Git URL. This allows go-getter to be very user friendly.

go-getter out of the box supports the following protocols. Additional protocols can be augmented at runtime by implementing the Getter interface.

  • Local files
  • Git
  • Mercurial
  • HTTP
  • Amazon S3
  • Google GCP

In addition to the above protocols, go-getter has what are called "detectors." These take a URL and attempt to automatically choose the best protocol for it, which might involve even changing the protocol. The following detection is built-in by default:

  • File paths such as "./foo" are automatically changed to absolute file URLs.
  • GitHub URLs, such as "" are automatically changed to Git protocol over HTTP.
  • GitLab URLs, such as "" are automatically changed to Git protocol over HTTP.
  • BitBucket URLs, such as "" are automatically changed to a Git or mercurial protocol using the BitBucket API.

Forced Protocol

In some cases, the protocol to use is ambiguous depending on the source URL. For example, "" could reference an HTTP URL or a Git URL. Forced protocol syntax is used to disambiguate this URL.

Forced protocol can be done by prefixing the URL with the protocol followed by double colons. For example: git:: would download the given HTTP URL using the Git protocol.

Forced protocols will also override any detectors.

In the absence of a forced protocol, detectors may be run on the URL, transforming the protocol anyways. The above example would've used the Git protocol either way since the Git detector would've detected it was a GitHub URL.

Protocol-Specific Options

Each protocol can support protocol-specific options to configure that protocol. For example, the git protocol supports specifying a ref query parameter that tells it what ref to checkout for that Git repository.

The options are specified as query parameters on the URL (or URL-like string) given to go-getter. Using the Git example above, the URL below is a valid input to go-getter:

The protocol-specific options are documented below the URL format section. But because they are part of the URL, we point it out here so you know they exist.


If you want to download only a specific subdirectory from a downloaded directory, you can specify a subdirectory after a double-slash //. go-getter will first download the URL specified before the double-slash (as if you didn't specify a double-slash), but will then copy the path after the double slash into the target directory.

For example, if you're downloading this GitHub repository, but you only want to download the testdata directory, you can do the following:

If you downloaded this to the /tmp directory, then the file /tmp/archive.gz would exist. Notice that this file is in the testdata directory in this repository, but because we specified a subdirectory, go-getter automatically copied only that directory contents.

Subdirectory paths may also use filesystem glob patterns. The path must match exactly one entry or go-getter will return an error. This is useful if you're not sure the exact directory name but it follows a predictable naming structure.

For example, the following URL would also work:*


For file downloads of any protocol, go-getter can automatically verify a checksum for you. Note that checksumming only works for downloading files, not directories, but checksumming will work for any protocol.

To checksum a file, append a checksum query parameter to the URL. go-getter will parse out this query parameter automatically and use it to verify the checksum. The parameter value can be in the format of type:value or just value, where type is "md5", "sha1", "sha256", "sha512" or "file" . The "value" should be the actual checksum value or download URL for "file". When type part is omitted, type will be guessed based on the length of the checksum string. Examples:


When checksumming from a file - ex: with checksum=file:url - go-getter will get the file linked in the URL after file: using the same configuration. For example, in file: go-getter will download a checksum file under the aforementioned url using the http protocol. All protocols supported by go-getter can be used. The checksum file will be downloaded in a temporary file then parsed. The destination of the temporary file can be changed by setting system specific environment variables: TMPDIR for unix; TMP, TEMP or USERPROFILE on windows. Read godoc of os.TempDir for more information on the temporary directory selection. Content of files are expected to be BSD or GNU style. Once go-getter is done with the checksum file; it is deleted.

The checksum query parameter is never sent to the backend protocol implementation. It is used at a higher level by go-getter itself.

If the destination file exists and the checksums match: download will be skipped.


go-getter will automatically unarchive files into a file or directory based on the extension of the file being requested (over any protocol). This works for both file and directory downloads.

go-getter looks for an archive query parameter to specify the format of the archive. If this isn't specified, go-getter will use the extension of the path to see if it appears archived. Unarchiving can be explicitly disabled by setting the archive query parameter to false.

The following archive formats are supported:

  • tar.gz and tgz
  • tar.bz2 and tbz2
  • tar.xz and txz
  • zip
  • gz
  • bz2
  • xz

For example, an example URL is shown below:


This will automatically be inferred to be a ZIP file and will be extracted. You can also be explicit about the archive type:


And finally, you can disable archiving completely:


You can combine unarchiving with the other features of go-getter such as checksumming. The special archive query parameter will be removed from the URL before going to the final protocol downloader.

Protocol-Specific Options

This section documents the protocol-specific options that can be specified for go-getter. These options should be appended to the input as normal query parameters (HTTP headers are an exception to this, however). Depending on the usage of go-getter, applications may provide alternate ways of inputting options. For example, Nomad provides a nice options block for specifying options rather than in the URL.

General (All Protocols)

The options below are available to all protocols:

  • archive - The archive format to use to unarchive this file, or "" (empty string) to disable unarchiving. For more details, see the complete section on archive support above.

  • checksum - Checksum to verify the downloaded file or archive. See the entire section on checksumming above for format and more details.

  • filename - When in file download mode, allows specifying the name of the downloaded file on disk. Has no effect in directory mode.

Local Files (file)


Git (git)

  • ref - The Git ref to checkout. This is a ref, so it can point to a commit SHA, a branch name, etc. If it is a named ref such as a branch name, go-getter will update it to the latest on each get.

  • sshkey - An SSH private key to use during clones. The provided key must be a base64-encoded string. For example, to generate a suitable sshkey from a private key file on disk, you would run base64 -w0 .

    Note: Git 2.3+ is required to use this feature.

  • depth - The Git clone depth. The provided number specifies the last n revisions to clone from the repository.

The git getter accepts both URL-style SSH addresses like git::ssh://[email protected]/foo/bar, and "scp-style" addresses like git::[email protected]/foo/bar. In the latter case, omitting the git:: force prefix is allowed if the username prefix is exactly [email protected].

The "scp-style" addresses cannot be used in conjunction with the ssh:// scheme prefix, because in that case the colon is used to mark an optional port number to connect on, rather than to delimit the path from the host.

Mercurial (hg)

  • rev - The Mercurial revision to checkout.

HTTP (http)

Basic Authentication

To use HTTP basic authentication with go-getter, simply prepend username:[email protected] to the hostname in the URL such as https://Aladdin:[email protected]/index.html. All special characters, including the username and password, must be URL encoded.


Optional request headers can be added by supplying them in a custom HttpGetter (not as query parameters like most other options). These headers will be sent out on every request the getter in question makes.

S3 (s3)

S3 takes various access configurations in the URL. Note that it will also read these from standard AWS environment variables if they're set. S3 compliant servers like Minio are also supported. If the query parameters are present, these take priority.

  • aws_access_key_id - AWS access key.
  • aws_access_key_secret - AWS access key secret.
  • aws_access_token - AWS access token if this is being used.
  • aws_profile - Use this profile from local ~/.aws/ config. Takes priority over the other three.

Using IAM Instance Profiles with S3

If you use go-getter and want to use an EC2 IAM Instance Profile to avoid using credentials, then just omit these and the profile, if available will be used automatically.

Using S3 with Minio

If you use go-gitter for Minio support, you must consider the following:

  • aws_access_key_id (required) - Minio access key.
  • aws_access_key_secret (required) - Minio access key secret.
  • region (optional - defaults to us-east-1) - Region identifier to use.
  • version (optional - defaults to Minio default) - Configuration file format.

S3 Bucket Examples

S3 has several addressing schemes used to reference your bucket. These are listed here:

Some examples for these addressing schemes:

GCS (gcs)

GCS Authentication

In order to access to GCS, authentication credentials should be provided. More information can be found here

GCS Bucket Examples

GCS Testing

The tests for get_gcs.go require you to have GCP credentials set in your environment. These credentials can have any level of permissions to any project, they just need to exist. This means setting GOOGLE_APPLICATION_CREDENTIALS="~/path/to/credentials.json" or GOOGLE_CREDENTIALS="{stringified-credentials-json}". Due to this configuration, get_gcs_test.go will fail for external contributors in CircleCI.

  • Client options + http progress + http byte range requets

    Client options + http progress + http byte range requets


    This PR introduces a non breaking change that allows to configure a client by changing Get/GetAny/GetFile api:

    func Get(dst, src string, opts ...ClientOption) error {

    This pr also introduces http download continue to save some bandwidth.

    This pr also allows to request for a progress bar display. ( which only works for http downloads for now )

    In a future PR I would like to add context support ~& race protections~.

    opened by azr 11
  • Git

    Git "ref" parameter described as taking a ref, but is a tag or branch name only

    The documentation for the "ref" parameter is

    The Git ref to checkout. This is a ref, so it can point to a commit SHA, a branch name, etc. If it is a named ref such as a branch name, go-getter will update it to the latest on each get.

    However the implementation is to call the "--branch" option on the git command line, meaning that it has to be either a branch or tag name. The implementation of --branch in git is to list through the refs, finding either "refs/heads/" (for priority) or "refs/tags/".

    Experimentation also shows that you can't pass a SHA, or string such as "refs/heads/master" etc, you can only pass a branch name or tag name. There are tests in go-getter that test the ref parameter being a branch name or a tag name, but nothing further also.

    So I think the documentation needs clarifying here.

    opened by tomqwpl 10
  • context is not respected for HTTP requests

    context is not respected for HTTP requests

    It seems the passed Ctx to getter.Client is not passed to the http.Request:

    I wrote a test for this below. I receive the context.Canceled error, but it still waits for both requests (HEAD + GET) to finish, thus this test takes 6 seconds instead of <1sec.

    package cache
    import (
    func TestHashicorpGoGetter__RespectsContextCanceled(t *testing.T) {
    	ctx, cancel := context.WithCancel(context.Background())
    	cancel() // cancel immediately
    	srv := GivenHTTPServer(t, http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
    		t.Logf("Received %s %s", req.Method, req.URL)
    		time.Sleep(3 * time.Second)
    	tmpdir := t.TempDir()
    	g := getter.Client{
    		Ctx:  ctx,
    		Src:  srv.URL,
    		Pwd:  tmpdir,
    		Dst:  "testfile",
    		Mode: getter.ClientModeFile,
    	t1 := time.Now()
    	err := g.Get()
    	dur := time.Since(t1)
    	if !errors.Is(err, context.Canceled) {
    		t.Log("error is not context.Canceled")
    	if dur > 1*time.Second {
    		t.Log("Get() took too long")
    func GivenHTTPServer(t testing.TB, h http.Handler) *httptest.Server {
    	srv := httptest.NewServer(h)
    	t.Cleanup(func() {
    	return srv

    Obviously it is more realistic to cancel the context during the request. This could be achieved by moving the cancel() call into the http handler.

    opened by zeisss 9
  • Use default AWS credential handling under normal circumstances

    Use default AWS credential handling under normal circumstances

    Fixes #191 (and #185, #157, #152, #22)

    This PR changes go-getter to default to the AWS SDK's built in credential resolution. This makes it possible for go-getter to download from S3 using an assumed role (e.g. when setting environment variables AWS_SDK_LOAD_CONFIG=true AWS_PROFILE=..., go-getter will assume that role before attempting any operations). It should also add support for various other edge cases relating to AWS credentials.

    The one case where go-getter will continue to construct its own credential chain is the one with an override for the instance metadata URL. I'm not familiar with the use case of overriding the instance metadata URL, so I left that code as it was.

    I'd like to add a test to protect against regressions in assume-role behavior, but I'd appreciate some guidance there. I see the hardcoded read-only IAM access key/secret. Do those credentials also have the ability to assume some role?

    opened by jbergknoff-rival 9
  • Add support for tar archives

    Add support for tar archives

    Some projects distribute with .zip or .tgz, others have only a .tar (example: distributes

    It seems that the go-getter library only supports .tgz or .bz2, but not uncompressed .tar by itself.

    enhancement Getters 
    opened by ketzacoatl 9
  • Add support for S3

    Add support for S3

    As discussed in #2 I have worked on adding support for S3, I have also included the features requested in the mentioned ticket, to summarise this adds the following features:

    • Detect S3 URLS in the following formats (this means that any S3 urls will no longer be fetched using the HTTP getter)
    • Also supported is the ability to override the default authentication method and the version being "getted" (this is only supported by GetFile). For example:
    • Using Get all files with the given prefix are fetched and stored in the destination path relative to the given prefix
    • Using GetFile fetches the given file from S3 and is stored at the destination path. As mentioned it is also possible to fetch a versioned S3 object.

    If you have any further questions/feature suggestions let me know, thanks.

    opened by dancannon 9
  • Let shallow clone work with any ref

    Let shallow clone work with any ref

    Currently the shallow clone only works if ref is either not set, or set to the default branch. This is becuase go-getter does not set the ref to checkout during a shallow clone so Git assumes the default branch, and then when go-getter tries to checkout a ref other than the default branch it fails because the shallow clone did not fetch that ref.

    To fix this, I have used --branch to explicitly set the ref to fetch during a clone so that when a shallow clone is done go-getter will clone the ref given and not the default branch. This will also make clones of non-default branches just a tad bit more optimized since the clone will not waste time checking out a branch it ultimately will not use.


    opened by zachwhaley 8
  • Add failing cases for SCP style git

    Add failing cases for SCP style git "URLs"

    Some additional detection cases:

    1. SSH URL syntax this case passes
    2. SCP style with forced getter and pwd, this fails with detection as file URL
    3. SCP style with forced getter (no pwd), this fails with no PWD error in file detector
    opened by paultyng 8
  • FIX: Private enterprise github references

    FIX: Private enterprise github references

    I was redirected from here

    Added a few unit test cases for github detector Removed the hardcoded domain ‘’ in detectSSH() and set it from the source Modified the initial testing logic for HTTPS in Detect() such that handles domain names where github is a subdomain

    PS: The test cases was added by looking at the code and behavior before my code changes and not from knowledge of use cases/expected values. My main objective for the test cases was to ensure I don’t break the behavior regular URLs

    opened by johnthedev97 8
  • Migrate to GHA

    Migrate to GHA

    This PR officially migrates go-getter to GHA from CircleCI. It's intention is to work the same as the CircleCI tests but in GHA format.

    Note: After working on updating the AWS/GCP tests, it was concluded that the older go versions (1.14.15 and 1.15.3) were blocking the GCP tests from passing. The issue was that go1.17 is required for the dependent GCS libraries and we concluded that the removal of the older go versions was the logical path to move forward.

    opened by claire-labry 7
  • go-get checksum file & guess checksum

    go-get checksum file & guess checksum


    This pr allows to checksum from a file; this should be the last needed feature for packer to be able to use go-getter 🙂


    opened by azr 7
  • go get failure

    go get failure

    go get fails to install dependencies correctly.

    go version go1.18 darwin/arm64

    go get imports
   cannot find module providing package

    Was able to find a solution here (apparently new versions of grpc have removed the naming package).

    opened by p-real 0
  • v2: Url field isn't exported in Request type

    v2: Url field isn't exported in Request type

    Field u should be exported in order to set the url when building a request. The current tests only function because it's executing from the same package:

    type Request struct {
        Src string
        Dst string
        Pwd string
        Forced string
        Umask os.FileMode
        GetMode Mode
        Copy bool
        Inplace bool
        ProgressListener ProgressTracker
        DisableSymlinks bool
        u               *url.URL
        subDir, realDst string
    opened by Jukie 0
  • Setting host header in HttpGetter doesn't work

    Setting host header in HttpGetter doesn't work

    According to golang/go#29865, the Host header is ignored and should be set instead using req.Host

    Nomad uses go-getter and supports setting HTTP headers on artifact, but setting host header also invalid for this reason. It needs a small change for the particular case of the header Host

    opened by WeiKangJian 1
  • How to download a folder via http

    How to download a folder via http

    I try to download like it, but failed and the error is no source URL was returned

    	var httpGetter = &getter.HttpGetter{
    		// Disable pre-fetch HEAD requests
    		DoNotCheckHeadFirst: true,
    		// As an alternative to the above setting, you can
    		// set a reasonable timeout for HEAD requests
    		// HeadFirstTimeout: 10 * time.Second,
    		// Read timeout for HTTP operations
    		ReadTimeout: 30 * time.Second,
    		// Set the maximum number of bytes
    		// that can be read by the getter
    		MaxBytes: 500000000, // 500 MB
    	dest, err := url.Parse("")
    	err = httpGetter.Get("./hack", dest)
    	if err != nil {
    opened by jiajun-c 0
  • question/feature; downloading things from docker image data source

    question/feature; downloading things from docker image data source

    Would it be a scope of this repository to include docker data source? Downloading things from docker image like:

    opened by bukowa 0
Consistent workflows to provision, secure, connect, and run any infrastructure for any application.
🚀Gev is a lightweight, fast non-blocking TCP network library based on Reactor mode. Support custom protocols to quickly and easily build high-performance servers.

gev 中文 | English gev is a lightweight, fast non-blocking TCP network library based on Reactor mode. Support custom protocols to quickly and easily bui

徐旭 1.5k Jan 6, 2023
Encode and Decode Message Length Indicators for TCP/IP socket based protocols

SimpleMLI A Message Length Indicator Encoder/Decoder Message Length Indicators (MLI) are commonly used in communications over raw TCP/IP sockets. This

American Express 24 Nov 24, 2022
Bee is a tool to scan ports by TCP and UDP protocols

Bee - Port scan tool ?? Bee is a tool to scan ports by TCP and UDP protocols Building from Source Code First, we compile the source code with the ligh

kraken 2 Oct 10, 2021
Implementation of Minecraft protocols : ping, query and icon.

mcutils - Implementation of Minecraft protocols in Go Informations General All protocols are implemented in Go, without any external dependency. All p

null 13 Dec 19, 2022
Golang pow implementation client <-> server over UDP and TCP protocols

Client <-> server over UDP and TCP pow protocol Denial-of-Service-attacks are a typical situation when providing services over a network. A method for

Nikita Kryuchkov 0 Jan 13, 2022
Client - Server TCP Chat For String Messages And Random Files

GoChat Client - Server TCP Chat For String Messages And Random Files GoChat is a chat for string messages and random files using Golorem by Derek A. R

Brian GL 0 Sep 29, 2021
generate Wireguard keypairs with a given prefix string

wireguard-vanity-address Generate Wireguard keypairs with a given prefix string. The Wireguard VPN uses Curve25519 keypairs, and displays the Base64-e

yinheli 1 Nov 9, 2022
Cf-cli-find-app-plugin - CF CLI plugin to find applications containing a search string

Overview This cf cli plugin allows users to search for application names that co

null 0 Jan 3, 2022
An URL shortener service written in Golang

ggz An URL shortener service written in Golang. Features Support MySQL, Postgres or SQLite Database. Support RESTful or GraphQL API. Support Auth0 or

null 182 Dec 26, 2022
A tool for connect url write by Golang

goURL A tool for connect url write by Golang To-Do In this month (2021.11), our development plan is adding the following functions: -X, a flag to choo

null 0 Nov 8, 2021
A tool that makes http requests and outputs the url and the content (optionally to file)

BKK Basic Crawler A tool that makes http requests and outputs the url and the content (optionally to file) How to run.. the tests go test the compiler

Jero Berlin 0 Nov 8, 2021
🍔 Product-storage service, work on gRPC. Client sends the URL to download products, and requests the result.

?? Product-storage service, work on gRPC. Client sends the URL to download products, and requests the result. The server transfer request to a third-party resource for .csv-file uploading and saves the products to own database.

Pavel V 9 Dec 16, 2021
Kick dropper is a very simple and leightweight demonstration of SQL querying, and injection by parsing URl's

__ __ __ __ _____ ______ | |/ |__|.----.| |--.______| \.----.| |.-----.-----.-----.----.

RE43P3R 2 Feb 6, 2022
Fetch-npm-package - A small utility that can be used to fetch a given version of a NPM package

Use fetch-npm-package <package> <version> <output-dir> E.g. fetch-npm-package is

Bjørn Erik Pedersen 1 May 21, 2022
Examples using the stomp package from git://

stompngo_examples - A collection of examples for package stompngo Features Full demonstration of support for STOMP protocols: Protocol Level 1.0 Proto

Guy M. Allard 16 Jan 22, 2021
This service is intented to collect data using grpc using Go lang backend and cassandra DB as storage

Go Data Collection This service is intented to collect data using grpc using Go lang backend and cassandra DB as storage. Dev Setup make test_env make

Daiem Ali 1 Apr 13, 2022
Package arp implements the ARP protocol, as described in RFC 826. MIT Licensed.

arp Package arp implements the ARP protocol, as described in RFC 826. MIT Licensed. Portions of this code are taken from the Go standard library. The

Matt Layher 305 Dec 20, 2022
Package dhcp6 implements a DHCPv6 server, as described in RFC 3315. MIT Licensed.

dhcp6 Package dhcp6 implements a DHCPv6 server, as described in IETF RFC 3315. MIT Licensed. At this time, the API is not stable, and may change over

Matt Layher 76 Sep 27, 2022
A Go package for sending and receiving ethernet frames. Currently supporting Linux, Freebsd, and OS X.

ether ether is a go package for sending and receiving ethernet frames. Currently supported platform: BPF based OS X FreeBSD AF_PACKET based Linux Docu

Song Gao 78 Sep 27, 2022