A simple tool to help apply changes across many GitHub repositories simultaneously

Related tags

Git turbolift
Overview

turbolift

A simple tool to help apply changes across many GitHub repositories simultaneously.

Philosophy

Anyone who has had to manually make changes to many GitHub repositories knows that it's hard to beat the simplicity of just cloning the repositories and updating them locally. You can use any tools necessary to make the change, and there's a degree of immediacy in having local files to inspect, tweak or run validation.

It's dumb but it works. It doesn't scale well, though. Manually cloning and raising PRs against tens/hundreds of repositories is painful and boring.

Turbolift essentially automates the boring parts and stays out of the way when it comes to actually making the changes. It automates forking, cloning, committing, and raising PRs en-masse, so that you can focus on the substance of the change.

Historical note: Turbolift supersedes an internal system at Skyscanner named Codelift. Codelift was a centralised batch system, requiring changes to be scripted upfront and run overnight. While Codelift was useful, we have found that a decentralised, interactive tool is far easier and quicker for people to use in practice.

Demo

This demo shows Turbolift in action, creating a simple PR in two repositories:

Screencast demo of turbolift in use

Installation

Downloading binaries

Pre-built binary archives can be downloaded from the Releases page.

  • Download, extract the archive, and move it onto your PATH.
  • Note that the binaries are not currently notarized for MacOS Gatekeeper. If errors are displayed, use xattr -c PATH_TO_TURBOLIFT_BINARY to un-quarantine the binary, or right-click on the binary in Finder and choose 'Open' once to allow future execution. Distribution will be improved under https://github.com/Skyscanner/turbolift/issues/43.

You must also have the GitHub CLI, gh, installed:

  • Install using brew install gh
  • Before using Turbolift, run gh auth login once and follow the prompts, to authenticate against github.com and/or your GitHub Enterprise server.

Basic usage:

Making changes with turbolift is split into six main phases:

  1. init - getting set up
  2. Identifying the repos to operate upon
  3. Running a mass clone of the repos (which automatically creates a fork in your user space)
  4. Making changes to every repo
  5. Committing changes to every repo
  6. Creating a PR for every repo

It is expected that you'll go through these phases in series.

The turbolift tool automates as much of the Git/GitHub heavy lifting as possible, but leaves you to use whichever tools are appropriate for making the actual changes.

Caveats

With great power comes great responsibility. We encourage Turbolift users to consider the following guidelines:

  • Don't use Turbolift to raise pointless PRs. If a reviewer might think the change is trivial or unimportant, think about whether it's actually needed.
  • If you need to make a change to a large number of repositories, we've found that it's generally better to raise PRs to a small subset at first and collect feedback. Simply comment out repositories in repos.txt to make Turbolift temporarily ignore them.
  • For complicated or potentially contentious changes, think about ways to validate them before raising PRs. This could range from working in a pair, through writing a peer-reviewed script, all the way to preparing a design document for the planned changes.
  • If you can run automated tests locally, then do (e.g. turbolift foreach to run linting and tests for each repository).
  • Raising draft PRs can be a good way to collect feedback, especially CI test results, with less pressure on reviewers (TODO: relies on https://github.com/Skyscanner/turbolift/issues/11 being implemented)
  • In an organisation with shared infrastructure (e.g. CI), raising many PRs in a short timeframe can cause a lot of load. Consider spacing out PR creation using the --sleep option or by commenting out chunks of repositories in repos.txt.

Detailed usage

init - getting set up

As per the installation instructions above, make sure gh is installed and authenticated before starting.

If working with repositories on a GitHub Enterprise server, ensure that you have the environment variable GH_HOST set to point to that server.

To begin working with Turbolift and create a 'campaign' to hold settings and working copies of repositories:

turbolift init --name CAMPAIGN_NAME

This creates a new turbolift 'campaign' directory ready for you to work in. Note that CAMPAIGN_NAME will be used as the branch name for any changes that are created

Next, please run:

cd CAMPAIGN_NAME

Identifying the repos to operate upon

Update repos.txt with the names of the repos that need changing (either manually or using a tool to identify the repos).

gh-search is an excellent tool for performing GitHub code searches, and can output a list of repositories in a format that turbolift understands:

e.g.

$ gh-search --repos-with-matches YOUR_GITHUB_CODE_SEARCH_QUERY > repos.txt

Running a mass clone

turbolift clone

This clones all repositories listed in the repos.txt file into the work directory.

Making changes

Now, make changes to the checked-out repos under the work directory. You can do this manually using an editor, using sed and similar commands, or using codemod/comby, etc.

You are free to use any tools that help get the job done.

If you wish to, you can run the same command against every repo using turbolift foreach ... (where ... is the shell command you want to run).

For example, you might choose to:

  • turbolift foreach rm somefile - to delete a particular file
  • turbolift foreach sed -i '' 's/foo/bar/g' somefile - to find/replace in a common file
  • turbolift foreach make test - for example, to run tests (using any appropriate command to invoke the tests)
  • turbolift foreach git add somefile - to stage a file that you have created

At any time, if you need to update your working copy branches from the upstream, you can run turbolift foreach git pull upstream master.

It is highly recommended that you run tests against affected repos, if it will help validate the changes you have made.

Committing changes

When ready to commit changes across all repos, run:

turbolift commit --message "Your commit message"

This command is a no-op on any repos that do not have any changes. Note that the commit will be run with the --all flag set, meaning that it is not necessary to stage changes using git add/rm for changed files. Newly created files will still need to be staged using git add.

Repeat if you want to make multiple commits.

Creating PRs

Edit the PR title and description in README.md.

Next, to push and raise PRs against changed repos, run:

turbolift create-prs

Use turbolift create-prs --sleep 30s to, for example, force a 30s pause between creation of each PR. This can be helpful in reducing load on shared infrastructure.

Important: if raising many PRs, you may generate load on shared infrastucture such as CI. It is highly recommended that you:

  • slow the rate of PR creation by making Turbolift sleep in between PRs
  • create PRs in batches, for example by commenting out repositories in repos.txt

If you need to mass-close PRs, it is easy to do using turbolift foreach and the gh GitHub CLI (docs):

For example:

turbolift foreach gh pr close --delete-branch YOUR_USERNAME:CAMPAIGN_NAME

Status: Preview

This tool is fully functional, but we have improvements that we'd like to make, and would appreciate feedback.

Contributing

See CONTRIBUTING.md

Local development

To build locally:

make build

To run tests locally:

make test
Comments
  • Do not create forks if not necessary

    Do not create forks if not necessary

    Turbolift creates a fork for each project it needs to interact with. Being Turbolift an excellent tool to handle maintenance tasks in the projects where you work usually it doesn't make much sense to fork these repositories where you already have write access. It would be better if Turbolift could use the original repository if possible and fallback to a fork if not.

    opened by arnaugm 12
  • Replace emoji with jest-style colors

    Replace emoji with jest-style colors

    I've experimented with some true-console experience and tried to implement some jest-like badges. That's not so distracting as more or less fits into terminal color palette and resolution. I would say it also feels a bit more like a professional tool, but that's for sure IMO only.

    Just take a look and please tell me WDYT?

    image image

    under review 
    opened by lanwen 7
  • foreach sed fails with 'unknown shorthand flag' error

    foreach sed fails with 'unknown shorthand flag' error

    I get the following error when running foreach sed with turbolift 2:

    `$ turbolift foreach sed -i '' 's/foo/bar/g' README.md

    Error: unknown shorthand flag: 'i' in -i Usage: turbolift foreach -- SHELL_COMMAND [flags] Flags: -h, --help help for foreach Global Flags: -v, --verbose verbose output 2021/05/21 11:49:08 unknown shorthand flag: 'i' in -i`

    A temporary workaround is to encapsulate the sed command in double quotes, but I think this is a bug in the way parameters are handled. (also some of the README examples do not work either)

    opened by danielt998 7
  • Add git clone command

    Add git clone command

    Partially addresses #48

    This makes no actual changes to any commands, but adds the ability to call Clone function.

    As there is still some ambiguity around how we should address the actual turbolift clone command, I will address that in a followup PR.

    @sledigabel @rnorth would you prefer to preserve current behaviour in clone.go to fork by default? I think not forking by default is more user-friendly in general, but it sounds like the Skyscanner use case requires bulk forking.

    My suggestion would be to make the new Clone command default, and place ForkAndClone behind a different cobra command.

    opened by Macavirus 7
  • turbolift clone/create-prs does not work in windows

    turbolift clone/create-prs does not work in windows

    When doing a turbolift clone in a windows environment, it reports the following error:

    Executing: git [checkout -b C:\src\turbolift\campaignx] in work/xxx/xxxxxx fatal: 'C:\src\turbolift\campaignx' is not a valid branch name. It does actually manage to clone the repositories, but when then trying turbolift create-prs it reports the following error:

    Executing: git [push -u origin C:\src\turbolift\campaign] in work/xxx/xxxxxxx fatal: invalid refspec 'C:\src\turbolift\campaign'

    opened by kathrynallan 6
  • Disabling all arguments after foreach

    Disabling all arguments after foreach

    Fixes #50. Uses DisableFlagParsing as per the documentation to disable the subsequent flag parsing in the command and leaving it as arguments to the shell command.

    Added TraverseChildren to rootCmd to allow -v and -h to be parsed.

    https://pkg.go.dev/github.com/spf13/cobra#Command

    Tested on Mac OS.

    I'm unable to write tests as we're not testing the flag parsing at the moment.

    opened by sledigabel 4
  • Adds the ability to clone only and not fork

    Adds the ability to clone only and not fork

    This is a follow up of #49.

    With this commit, the clone command is defaulting to cloning and forking only when the user has not push rights onto the repository.

    If the --fork flag is specified, it will fork and clone systematically.

    opened by sledigabel 3
  • New files are ignored by the commit command

    New files are ignored by the commit command

    That's actually a feature of a git commit -a, however it's unexpected for a turbolift user, as it states that you don't have to do anything. Currently workaround is to turbolift foreach -- git add -A, but maybe that could be part of the commit command by default?

    opened by lanwen 3
  • Turbolift can't start when there's an empty line in repos.txt

    Turbolift can't start when there's an empty line in repos.txt

    image

    I had a txt file that had an empty line in the end. I think the tools should be able to filter empty lines so that you could group repositories by org or something.

    opened by kevinrobayna 2
  • Add schedule to CodeQL Scan

    Add schedule to CodeQL Scan

    Running CodeQL on regularly whether there are changes or not is beneficial. If new queries or vulnerabilities are found and the project is inactive at the time this run is used to make sure that they are flagged.

    I chose a random day and time to run weekly. Cron is every Monday at 09:09 UTC

    opened by logan-porelle 1
  • Adds `update-prs` with `close` feature

    Adds `update-prs` with `close` feature

    Relates to #65 Fixes #60 Relates to #67

    This PR implements a global PR closing command. It introduces a new update-prs verb which has only one option at the moment: --close.

    It uses the same mechanism as in #67, looking at the PRs and identifying the one related to the campain, then closes it.

    It handles the case where there is no open PR.

    opened by sledigabel 1
  • Auto-merge flag during PR creation

    Auto-merge flag during PR creation

    Description

    It would be very helpful to add --auto-merge flag to turbolift create-prs command.

    Motivation

    In case of huge amount of repos it would be very convenient to enable auto-merge (if the repo has such function of course), so if all of the checks are green, then PR would be merged automatically.

    opened by bbednarek 0
  • Option for shallow clone for repos with large commit histories

    Option for shallow clone for repos with large commit histories

    It would be nice to have an option to do a shallow clone for quicker clones. This would allow us to grab for example, the last 30 commits per repo instead of the entire commit history

    # maybe --shallow-clone which could default to 50 last commits
    turbolift clone --shallow-clone
    # provide the clone depth which could default to 50 or -1 to account for the default deep clone
    turbolift clone --depth=50
    

    https://www.perforce.com/blog/vcs/git-beyond-basics-using-shallow-clones https://github.blog/2020-12-21-get-up-to-speed-with-partial-clone-and-shallow-clone/

    Example

    git clone -–depth [depth] [remote-url]
    
    opened by nitrocode 0
  • feat: support for dry-run mode

    feat: support for dry-run mode

    What does this PR do?

    It does a few things to achieve dry-run to work properly:

    1. a way to display in a clear manner that dry-run is enabled. For that, we are adding an Infof function to the logger, which uses a new Info color. This color will display the background in Cyan and the foreground in White.
    2. a new persistent, DryRun flag, attached to the root command. This flag will be available to all subcommands.
    3. a helper function for all the cobra commands and subcommands, that will be run at the PreRun stage. This function will check if the DryRun flag is enabled, and print a message built with the previous blocks.
    4. an evaluation of the DryRun flag at all the implementations of the executors, so that the first thing that happens in there is checking if DryRun is true. If so, it will just print the command, arguments and workDir, returning without error. For the case of the ExecuteAndCapture function, it will return the very same string than the above implementation.
    5. Finally it add tests for git/github packages, which use the new code

    Why is it important?

    It will support testing locally changes before submitting real pull requests.

    Other considerations

    I probably missed some other places where the dry-run mode should be applied. Please point them out to me so that I can update the code.

    Related issues

    • Closes #38
    opened by mdelapenya 2
  • [Question] What is the user account of the submitted PRs?

    [Question] What is the user account of the submitted PRs?

    Reading the code, it seems it's the current user in the terminal session, but I wonder if it's possible to configuring turbolift to use a machine account instead.

    opened by mdelapenya 0
  • Apparent incompatibility with `gh` 2.18.0

    Apparent incompatibility with `gh` 2.18.0

    https://github.com/cli/cli/releases/tag/v2.18.0, released yesterday, seems to behave slightly differently and breaks turbolift create-prs.

    I get:

    ...
      OK   Pushing changes in myorg/myrepo to origin
     FAIL  Creating PR in myorg/myrepo: Error: exit status 1. Stderr: aborted: you must first push the current branch to a remote, or use the --head flag
    

    Downgrading gh to 2.17.0 seems to be a temporary workaround and works flawlessly.

    bug 
    opened by rnorth 3
Releases(2.1.1)
Owner
Skyscanner
Skyscanner
Find trending repositories on GitHub

octotrends.com A niftly little tool I wrote to try and find repos and languages that are rapidly growing on GitHub. Growth rates are based on % growth

Dominik Dabrowski 7 Jun 14, 2022
A Simple and Comprehensive Vulnerability Scanner for Container Images, Git Repositories and Filesystems. Suitable for CI

A Simple and Comprehensive Vulnerability Scanner for Containers and other Artifacts, Suitable for CI. Abstract Trivy (tri pronounced like trigger, vy

Aqua Security 15.6k Jan 9, 2023
SQL interface to git repositories, written in Go. https://docs.sourced.tech/gitbase

gitbase gitbase, is a SQL database interface to Git repositories. This project is now part of source{d} Community Edition, which provides the simplest

source{d} 2k Dec 25, 2022
Quickly clone git repositories into a nested folders like GOPATH.

cl cl clones git repositories into nested folders like GOPATH and outputs the path of the cloned directory. Example: cl https://github.com/foo/bar Is

Felix Geisendörfer 13 Nov 30, 2022
A command-line tool that makes git easier to use with GitHub.

hub is a command line tool that wraps git in order to extend it with extra features and commands that make working with GitHub easier. For an official

GitHub 22.2k Jan 1, 2023
:octocat: lazyhub - Terminal UI Client for GitHub using gocui.

lazyhub lazyhub - Terminal UI Client for GitHub using gocui. Demo Features ?? Check the trending repositories on GitHub today ?? Search repositories ?

ryo-ma 170 Dec 14, 2022
Write Github actions in Go

goaction Package goaction enables writing Github Actions in Go. The idea is: write a standard Go script, one that works with go run, and use it as Git

Eyal Posener 191 Dec 20, 2022
backup data (code, comments, issues) from github

github-backup Backup your GitHub repositories (including issues and comments). The backup will include a copy of the git repository itself (a bare, mi

Mario Finelli 1 Nov 21, 2021
This github action find the Issues linked in a Pull Request.

Linked Issue (Github Action) This action find the Issues linked in a Pull Request. It parses the HTML of the PR page to find the linked issues. Inputs

Go Actions 5 Jun 30, 2022
A Github Action that verify if your README.md has broken links

A GitHub Action that automatically check if some link in your README.md is broken or not responding.

Leandro Lima 7 Nov 10, 2022
A simple cli tool for switching git user easily inspired by Git-User-Switch

gitsu A simple cli tool for switching git user easily inspired by Git-User-Switch Installation Binary releases are here. Homebrew brew install matsuyo

Masaya Watanabe 207 Dec 31, 2022
chglog is a changelog management library and tool

chglog chglog is a changelog management library and tool Why While there are other tool out there that will create a changelog output as part of their

GoReleaser 184 Nov 16, 2022
A GitLab API client enabling Go programs to interact with GitLab in a simple and uniform way

A GitLab API client enabling Go programs to interact with GitLab in a simple and uniform way

Xanzy 1.9k Dec 28, 2022
Simple git hooks written in go that installs globally to your machine

Go-hooks Simple git hooks written in go that installs globally to your machine Install curl -fsSL

Vadim Makerov 2 Oct 19, 2022
🥄A simple generator for semantic git messages.

?? Tablespoon EXPERIMENTAL PREVIEW A simple generator for semantic git messages. Installation | Contributing Tablespoon is a simple generator which ca

Matt 6 Jul 22, 2022
Go-commitlinter - simple commit message linter

go-commitlinter go-commitlinter is simple commit message linter. Quick Start go

Masahiro331 15 Oct 8, 2022
DepCharge is a tool designed to help orchestrate the execution of commands across many directories at once.

DepCharge DepCharge is a tool that helps orchestrate the execution of commands across the many dependencies and directories in larger projects. It als

Andrew LeTourneau 23 Sep 27, 2022
ChangeTower is intended to help you watch changes in webpages and get notified of any changes written in Go

ChangeTower is intended to help you watch changes in webpages and get notified of any changes written in Go

The Cats 34 Nov 17, 2022
git-xargs is a command-line tool (CLI) for making updates across multiple Github repositories with a single command.

Table of contents Introduction Reference Contributing Introduction Overview git-xargs is a command-line tool (CLI) for making updates across multiple

Gruntwork 713 Dec 31, 2022
CLI tool for manipulating GitHub Labels across multiple repositories

takolabel Installation Mac $ brew install tommy6073/tap/takolabel Other platforms Download from Releases page in this repository. Usage Set variables

Takayuki NAGATOMI 10 Nov 3, 2022