Enable your Golang applications to self update with S3

Overview

s3update

Enable your Golang applications to self update with S3. Requires Go 1.8+

This package enables our internal tools to be updated when new commits to their master branch are pushed to Github.

Latest binaries are hosted on S3 under a specific bucket along its current version. When ran locally, the binary will fetch the version and if its local version is older than the remote one, the new binary will get fetched and will exit, stating to the user that it got updated and need to be ran again.

In our case, we're only shipping Linux and Darwin, targeting amd64 platform.

Bucket will have the following structure:

mybucket/
  mytool/
	VERSION
	mytool-linux-amd64
	mytool-darwin-amd64

Usage

Updates are easier to deal with when done through a continuous integration platform. We're using CircleCI but the following excerpt can easily be adapted to whichever solution being used.

CircleCI

xgo is being used to easily cross-compile code.

Adding the following at the end of the build script will push the binaries and its version to S3.

xgo --targets="linux/amd64,darwin/amd64" -ldflags="-X main.Version=$CIRCLE_BUILD_NUM" .

if [[ "$CIRCLE_BRANCH" = "master" ]]; then
	aws s3 cp mytool-darwin-10.6-amd64 s3://mybucket/mytool/mytool-darwin-amd64 --acl authenticated-read
	aws s3 cp mytool-linux-amd64 s3://mybucket/mytool/mytool-linux-amd64 --acl authenticated-read
	echo -n $CIRCLE_BUILD_NUM > VERSION && aws s3 cp VERSION  s3://mybucket/mytool/VERSION --acl authenticated-read
fi

Example

package main

import (
	"github.com/heetch/s3update"
)

var (
	// This gets set during the compilation. See below.
	Version = ""
)

func main() {
	err := s3update.AutoUpdate(s3update.Updater{
		CurrentVersion: Version,
		S3Bucket:       "mybucket",
		S3Region:       "eu-west-1",
		S3ReleaseKey:   "mytool/mytool-{{OS}}-{{ARCH}}",
		S3VersionKey:   "mytool/VERSION",
	})

  if err != nil {
    // ...
  }

  ...
}

Binary must be compiled with a flag specifying the new version:

go build -ldflags "-X main.Version=111" main.go

Contributions

Any contribution is welcomed!

  • Open an issue if you want to discuss bugs/features
  • Open Pull Requests if you want to contribute to the code

Copyright

Copyright © 2016 Heetch

See the LICENSE (MIT) file for more details.

Comments
  • Use os.Executable in favor of osext.Executable

    Use os.Executable in favor of osext.Executable

    Since Go 1.8 os.Executable is part of the stdlib, so osext can be dropped from the dependencies.

    That change is not tested, because a test requires to create a s3 bucket and a fake binary. That said I'm confident because a recent change in osext invokes os.Executable if the library is compiled under Go 1.8.

    opened by tbruyelle 3
  • Fix upgrade on linux systems

    Fix upgrade on linux systems

    Previously the still open executable file was directly written to, which caused the program to abort with a text file busy message on some systems.

    Now the following approach is taken:

    • Move the old file to a backup path
    • Try to write to the original path
    • If opening/writing the original path fails: Move backup to old location
    • If upgrade is successful: Remove backup file

    Based on https://github.com/exercism/cli/pull/290

    opened by arkan 2
  • Reduce memory consumption

    Reduce memory consumption

    Instead of copying the downloaded update into memory and then writing from memory to the final file, the reader for the download is used for buffered writing to the final file.


    I tested it with a "hello world" program that compiles to a 10 MB executable and then using pprof to create a memory profile. For creating the profile I imported github.com/pkg/profile into the s3updater package and called profile.Start(profile.MemProfile).Stop() right before the os.Exit(0). I profiled the code multiple times and the resulting profiles weren't always the same, but each time the original s3updater code lead to at least 8 MB memory usage, while the code after my changes always lead to less than 110 KB. For example:

    Original s3updater | After my changes -----------------------|------------------- memory profile | memory profile

    I only used Go profiling a few times before and only CPU profiling, so maybe there's something wrong about this or it doesn't show the full picture of what's happening. But I also checked my PC's memory usage and the difference can be seen in the same way.

    Now, 10 MB RAM is not a big deal. But some Go programs use packages like github.com/gobuffalo/packr (or one of its many alternatives that are compared here), so the file to update could be 100 MB or more. When running on a memory-constrained device like a Raspberry Pi Zero W (with only 512 MB RAM), this would be bad.

    Additionally, in the original implementation the garbage collector doesn't know that it can release the allocated memory, because when s3udpater downloads the update and runs it, it blocks in cmd.Run() and data still references the downloaded bytes and the GC doesn't know if it will still be accessed or not. This means the memory usage isn't just high during the update, but for all remaining time the updated program is running. Maybe this can already be mitigated by moving the part with the download and file operations into a function, so the GC sees that there's no reference to the slice of bytes any more after the function returned. But that would still lead to the temporary high memory usage during the update. With the buffered write both is covered.

    Let me know what you think :)

    opened by philippgille 1
  • Replace exec.Command with syscall.Exec

    Replace exec.Command with syscall.Exec

    Instead of spawning child processes you should replace the parent process. You could do this by replacing the exec.Command block on line 156-177 with this line: syscall.Exec(os.Args[0], os.Args, os.Environ())

    Related to exec: https://groob.io/posts/golang-execve/

    opened by nicklasring 1
  • Save download to temporary file

    Save download to temporary file

    Instead of moving current to a backup file, save download to temporary file.

    This will prevent issue where update is cancelled by external factor, where we would not be able to move .bak to its original location.

    opened by tomahh 1
  • DELIVERY-225 fix file not found

    DELIVERY-225 fix file not found

    Following DELIVERY-213 we didn't fixed the error

    The argument passed to syscall.Exec (os.Args[0]) suppose relative binary will be found/resolved properly

    It succeeded with ./releases/aww-darwin-amd64 and /usr/local/bin/aww but failed with path call aww

    Using absolute destination enforce the absolute resolution and hence resolves the error for further releases

    opened by luphaz 0
  • Add autorestart mechanism after an update

    Add autorestart mechanism after an update

    With the current implementation each update requiers the user to re-run the command.

    This PR solves this by restarting the command automatically, so the user doesn't need to do anything except waiting for the process to complete 🤘

    opened by arkan 0
  • Prevent broken binary if download process is Interrupted

    Prevent broken binary if download process is Interrupted

    Currently, if something happen during the download process, the binary file is corrupted and need to be manually restored.

    I propose doing a very simple verification by checking the downloaded file size after downloading has finished, then proceed.

    opened by theamazinghari 0
  • Add tests

    Add tests

    Currently only tests for errors.

    A test for a working update via S3 would require a mocked S3 that's being started and stopped from the test. I think that can be done, but maybe in a later PR.

    opened by philippgille 1
Releases(v0.1.0)
Owner
Heetch
Enjoy your night out
Heetch
Simple plugin to enable the /flip command inside of Mattermost

Plugin Starter Template This plugin serves as a starting point for writing a Mattermost plugin. Feel free to base your own plugin off this repository.

null 0 Nov 4, 2021
Demo on how an executable can respawn after an update

Auto respawn on update demo Demo on how an executable can respawn after an update How to build go build updatedemo.go How to run ./updatedemo Rebuil

Natanael Copa 1 Nov 2, 2021
An easy way to add useful startup banners into your Go applications

Try browsing the code on Sourcegraph! Banner Add beautiful banners into your Go applications Table of Contents Motivation Usage API Command line flags

Claudemiro 409 Oct 20, 2022
Embedded, self-hosted swagger-ui for go servers

swaggerui Embedded, self-hosted Swagger Ui for go servers This module provides swaggerui.Handler, which you can use to serve an embedded copy of Swagg

Andy Walker 54 Nov 20, 2022
Universal code search (self-hosted)

Sourcegraph OSS edition is a fast, open-source, fully-featured code search and navigation engine. Enterprise editions are available. Features Fast glo

Sourcegraph 7.1k Dec 6, 2022
Self hosted search engine for data leaks and password dumps

Self hosted search engine for data leaks and password dumps. Upload and parse multiple files, then quickly search through all stored items with the power of Elasticsearch.

Davide Pataracchia 22 Aug 2, 2021
A framework for constructing self-spreading binaries

A framework that aids in creation of self-spreading software Requirements go get -u github.com/redcode-labs/Coldfire go get -u github.com/yelinaung/go

Red Code Labs 926 Dec 1, 2022
Listmonk - a standalone, self-hosted, newsletter and mailing list manager

listmonk is a standalone, self-hosted, newsletter and mailing list manager. It is fast, feature-rich, and packed into a single binary. It uses a Postg

null 0 Jan 13, 2022
An easy to use, extensible health check library for Go applications.

Try browsing the code on Sourcegraph! Go Health Check An easy to use, extensible health check library for Go applications. Table of Contents Example M

Claudemiro 438 Nov 24, 2022
A simple wrapper to daemonize Go applications.

daemonigo A simple library to daemonize Go programming language applications. Installing $ go get github.com/tyranron/daemonigo After this command da

Kai Ren 38 Jul 15, 2022
Prometheus instrumentation library for Go applications

Prometheus Go client library This is the Go client library for Prometheus. It has two separate parts, one for instrumenting application code, and one

Prometheus 4.3k Dec 2, 2022
A simple Cron library for go that can execute closures or functions at varying intervals, from once a second to once a year on a specific date and time. Primarily for web applications and long running daemons.

Cron.go This is a simple library to handle scheduled tasks. Tasks can be run in a minimum delay of once a second--for which Cron isn't actually design

Robert K 215 Nov 19, 2022
A BPMN engine, meant to be embedded in Go applications with minim hurdles, and a pleasant developer experience using it.

A BPMN engine, meant to be embedded in Go applications with minim hurdles, and a pleasant developer experience using it. This approach can increase transparency for non-developers.

Martin W. Kirst 83 Nov 30, 2022
A simple package to daemonize Go applications.

A simple package to daemonize Go applications.

Henrique Dias 0 Nov 13, 2021
Chaosblade executor for chaos experiments on Java applications

Chaosblade-exec-jvm: Chaosblade executor for chaos experiments on Java applications Introduction The project is a chaosblade executor based on jvm-san

null 349 Dec 5, 2022
James is your butler and helps you to create, build, debug, test and run your Go projects

go-james James is your butler and helps you to create, build, debug, test and run your Go projects. When you often create new apps using Go, it quickl

Pieter Claerhout 56 Oct 8, 2022
:guardsman: A teeny tiny and somewhat opinionated generator for your next golang project

A Yeoman Golang Generator We are very sorry Gophers, but other names for the generator where taken, so we choose go-lang. But we have gocreate as an a

Axel Springer SE 25 Sep 27, 2022
Auto-evaluate your Golang code.

Ginker Ginker is a GUI application for auto-evaluating your Golang code. It allows you to write and run Golang code on the fly and it will help you to

nkoporec 8 Jun 24, 2021
:mailbox_closed: Your own local SMS gateway in Go

gosms Your own local SMS gateway What's the use ? Can be used to send SMS, where you don't have access to internet or cannot use Web SMS gateways or w

null 1.4k Nov 20, 2022