git-xargs is a command-line tool (CLI) for making updates across multiple Github repositories with a single command.

Overview

Go Report Card gruntwork-io

Table of contents

Introduction

Overview

git-xargs CLI

git-xargs is a command-line tool (CLI) for making updates across multiple Github repositories with a single command. You give git-xargs:

  1. a script or a command to run
  2. a list of repos

and git-xargs will:

  1. clone each repo
  2. run your specified script or command against it
  3. commit any changes
  4. open pull requests
  5. provide a detailed report of everything that happened

For example, have you ever needed to add a particular file across many repos at once? Or to run a search and replace to change your company or product name across 150 repos with one command? What about upgrading Terraform modules to all use the latest syntax? How about adding a CI/CD configuration file, if it doesn't already exist, or modifying it in place if it does, but only on a subset of repositories you select? You can handle these use cases and many more with a single git-xargs command.

Example: writing a new file to every repo in your github organization

As an example, let's use git-xargs to create a new file in every repo:

git-xargs \
  --branch-name test-branch \
  --github-org <your-github-org> \
  --commit-message "Create hello-world.txt" \
  touch hello-world.txt

Here's what it looks like in action:

git-xargs to the rescue!

In this example, every repo in your org will have a new file named hello-world.txt written to it with the contents "Hello, World!". You'll then receive an easy-to-read printout of exactly what happened on STDOUT:

*****************************************************************
  GIT-XARGS RUN SUMMARY @ 2021-04-12 23:05:18.478435534 +0000 UTC
  Runtime in seconds: 4
*****************************************************************


COMMAND SUPPLIED

[touch hello-world.txt]

 REPOS SUPPLIED VIA --repos FILE FLAG
│────────────────────────│────────────────────────│
│ ORGANIZATION NAME (5)  │ URL                    │
│────────────────────────│────────────────────────│
│ zack-test-org          │ terraform-aws-asg      │
│ zack-test-org          │ terraform-aws-vpc      │
│ zack-test-org          │ terraform-aws-security │
│ zack-test-org          │ terraform-aws-eks      │
│ zack-test-org          │ circleci-test-1        │
│────────────────────────│────────────────────────│

 ALL REPOS THAT WERE TARGETED FOR PROCESSING AFTER FILTERING MISSING / MALFORMED REPOS
│───────────────────│────────────────────────────────────────────────────│
│ REPO NAME         │ REPO URL                                           │
│───────────────────│────────────────────────────────────────────────────│
│ terraform-aws-vpc │ https://github.com/zack-test-org/terraform-aws-vpc │
│ terraform-aws-eks │ https://github.com/zack-test-org/terraform-aws-eks │
│ circleci-test-1   │ https://github.com/zack-test-org/circleci-test-1   │
│───────────────────│────────────────────────────────────────────────────│


 REPOS THAT WERE SUCCESSFULLY CLONED TO THE LOCAL FILESYSTEM
│───────────────────│────────────────────────────────────────────────────│
│ REPO NAME         │ REPO URL                                           │
│───────────────────│────────────────────────────────────────────────────│
│ terraform-aws-eks │ https://github.com/zack-test-org/terraform-aws-eks │
│ circleci-test-1   │ https://github.com/zack-test-org/circleci-test-1   │
│ terraform-aws-vpc │ https://github.com/zack-test-org/terraform-aws-vpc │
│───────────────────│────────────────────────────────────────────────────│


 REPOS THAT SHOWED FILE CHANGES TO THEIR WORKING DIRECTORY FOLLOWING COMMAND EXECUTION
│───────────────────│────────────────────────────────────────────────────│
│ REPO NAME         │ REPO URL                                           │
│───────────────────│────────────────────────────────────────────────────│
│ terraform-aws-eks │ https://github.com/zack-test-org/terraform-aws-eks │
│ terraform-aws-vpc │ https://github.com/zack-test-org/terraform-aws-vpc │
│ circleci-test-1   │ https://github.com/zack-test-org/circleci-test-1   │
│───────────────────│────────────────────────────────────────────────────│


 REPOS THAT WERE SUPPLIED BY USER BUT DON'T EXIST (404'D) VIA GITHUB API
│────────────────────────│──────────│
│ REPO NAME              │ REPO URL │
│────────────────────────│──────────│
│ terraform-aws-asg      │          │
│ terraform-aws-security │          │
│────────────────────────│──────────│


 REPOS WHOSE SPECIFIED BRANCHES DID NOT EXIST ON THE REMOTE, AND SO WERE FIRST CREATED LOCALLY
│───────────────────│────────────────────────────────────────────────────│
│ REPO NAME         │ REPO URL                                           │
│───────────────────│────────────────────────────────────────────────────│
│ terraform-aws-eks │ https://github.com/zack-test-org/terraform-aws-eks │
│ terraform-aws-vpc │ https://github.com/zack-test-org/terraform-aws-vpc │
│ circleci-test-1   │ https://github.com/zack-test-org/circleci-test-1   │
│───────────────────│────────────────────────────────────────────────────│


*****************************************************
  PULL REQUESTS OPENED
*****************************************************
│───────────────────│────────────────────────────────────────────────────────────│
│ REPO NAME         │ PR URL                                                     │
│───────────────────│────────────────────────────────────────────────────────────│
│ circleci-test-1   │ https://github.com/zack-test-org/circleci-test-1/pull/82   │
│ terraform-aws-eks │ https://github.com/zack-test-org/terraform-aws-eks/pull/81 │
│ terraform-aws-vpc │ https://github.com/zack-test-org/terraform-aws-vpc/pull/77 │
│───────────────────│────────────────────────────────────────────────────────────│

Getting started

  1. Export a valid Github token. See the guide on Github personal access tokens for information on how to generate one. For example, on Linux or Mac, you'd run:

    export GITHUB_OAUTH_TOKEN=<your-secret-github-oauth-token>
  2. Download the correct binary for your platform. Visit the releases page and download the correct binary depending on your system. Save it to somewhere on your PATH, such as /usr/local/bin/git-xargs.

  3. Set execute permissions. For example, on Linux or Mac, you'd run:

    chmod u+x /usr/local/bin/git-xargs
  4. Check it's working. Run the version command to ensure everything is working properly:

    git-xargs --version
  5. Provide a script or command and target some repos. Here's a simple example of running the touch command in every repo in your Github organization. Follow the same pattern to start running your own scripts and commands against your own repos!

    git-xargs \
      --branch-name "test-branch" \
      --commit-message "Testing git-xargs" \
      --github-org <enter-your-github-org-name> \
      touch git-xargs-is-awesome.txt

Reference

How to supply commands or scripts to run

The API for git-xargs is:

git-xargs [-flags] <CMD>

Where CMD is either the full path to a (Bash, Python, Ruby) etc script on your local system or a binary. Note that, because the tool supports Bash scripts, Ruby scripts, Python scripts, etc, you must include the full filename for any given script, including its file extension.

In other words, all the following usages are valid:

git-xargs --repo --repo gruntwork-io/cloud-nuke \
   --repo gruntwork-io/terraform-aws-eks \
   --branch-name my-branch \
   /usr/local/bin/my-bash-script.sh
git-xargs --repos ./my-repos.txt \
  --branch-name my-other-branch \
  touch file1.txt file2.txt
git-xargs --github-org my-github-org \ 
  --branch-name my-new-branch \
  "$(pwd)/scripts/my-ruby-script.rb"

Branch behavior

Passing the --branch-name (-b) flag is required when running git-xargs. If you specify the name of a branch that exists on your remote, its latest changes will be pulled locally prior to your command or script being run. If you specify the name of a new branch that does not yet exist on your remote, it will be created locally and pushed once your changes are committed.

Git file staging behavior

Currently, git-xargs will find and add any and all new files, as well as any existing files that were modified, within your repo and stage them prior to committing. If your script or command creates a new file, it will be committed. If your script or command edits an existing file, that change will also be committed.

Paths and script locations

Scripts may be placed anywhere on your system, but you are responsible for providing absolute paths to your scripts when invoking git-xargs:

git-xargs \
  --branch-name upgrade-tf-14 \
  --commit-message "Update modules to Terraform 0.14" \
  --repos data/batch3.txt \
  $(pwd)/scripts/my-ruby-script.rb

or

git-xargs \
  --branch-name upgrade-tf-14 \
  --commit-message "Update modules to Terraform 0.14" \
  --repos data/batch3.txt \
  /usr/local/bin/my-ruby-script.rb

If you need to compose more complex behavior into a single pull request, write a wrapper script that executes all your commands, or place all your logic into one script.

How to target repos to run your scripts against

git-xargs supports four methods of targeting repos to run your selected scripts against. They are processed in the order listed below, with whichever option is found first being used, and all others after it being ignored.

Option #1: Github organization lookup

If you want the tool to find and select every repo in your Github organization, you can pass the name of your organization via the --github-org flag:

git-xargs \
  --commit-message "Update copyright year" \
  --github-org <your-github-org> \ 
  "$(pwd)/scripts/update-copyright-year.sh"

This will signal the tool to look up, and page through, every repository in your Github organization and execute the scripts you passed via the --scripts flag.

Option #2: Flat file of repository names

Oftentimes, you want finer-grained control over the exact repos you are going to run your script against. In this case, you can use the --repos flag and supply the path to a file defining the exact repos you want the tool to run your selected scripts against, like so:

git-xargs \
  --commit-mesage "Update copyright year" \
  --repos data/batch2.txt \
  "$(pwd)/scripts/update-copyright-year.sh"

In this example, batch2.txt looks like this:

gruntwork-io/infrastructure-as-code-training
gruntwork-io/infrastructure-live-acme
gruntwork-io/infrastructure-live-multi-account-acme
gruntwork-io/infrastructure-modules-acme
gruntwork-io/infrastructure-modules-multi-account-acme

Flat files contain one repo per line, each repository in the format of <github-organization>/<repo-name>. Commas, trailing or preceding spaces, and quotes are all filtered out at runtime. This is done in case you end up copying your repo list from a JSON list or CSV file.

Option #3: Pass in repos via command line args

Another way to get fine-grained control is to pass in the individual repos you want to use via one or more --repo arguments:

git-xargs \
  --commit-mesage "Update copyright year" \
  --repo gruntwork-io/terragrunt \
  --repo gruntwork-io/terratest \
  --repo gruntwork-io/cloud-nuke \
  "$(pwd)/scripts/update-copyright-year.sh"

Option #4: Pass in repos via stdin

And one more (Unix-philosophy friendly) way to get fine-grained control is to pass in the individual repos you want to use by piping them in via stdin, separating repo names with whitespace or newlines:

echo "gruntwork-io/terragrunt gruntwork-io/terratest" | git-xargs \
  --commit-mesage "Update copyright year" \
  "$(pwd)/scripts/update-copyright-year.sh"

Notable flags

git-xargs exposes several flags that allow you to customize its behavior to better suit your needs. For the latest info on flags, you should run git-xargs --help. However, a couple of the flags are worth explaining more in depth here:

Flag Description Type Required
--branch-name You must specify the name of the branch to make your local and remote changes on. You can further control branching behavior via --skip-pull-requests as explained below String Yes
--repos If you want to specify many repos and manage them in files (which makes batching and testing easier) then use this flag to pass the filepath to a repos file. See the repos file format for more information String No
--repo Use this flag to specify a single repo, e.g., --repo gruntwork-io/cloud-nuke. Can be passed multiple times to target several repos String No
--github-org If you want to target every repo in a Github org that your GITHUB_OAUTH_TOKEN has access to, pass the name of the Organization with this flag, to page through every repo via the Github API and target it String No
--commit-message The commit message to use when creating commits. If you supply this flag, but neither the optional --pull-request-title or --pull-request-description flags, then the commit message value will be used for all three. String No
--skip-pull-requests If you don't want any pull requests opened, but would rather have your changes committed directly to your specified branch, pass this flag. Note that it won't work if your Github repo is configured with branch protections on the branch you're trying to commit directly to! Boolean No
--dry-run If you are in the process of testing out git-xargs or your intial set of targeted repos, but you don't want to make any changes via the Github API (pushing your local changes or opening pull requests) you can pass the dry-run branch. This is useful because the output report will still tell you which repos would have been affected, without actually making changes via the Github API to your remote repositories. Boolean No

Best practices, tips and tricks

Write your script to run against a single repo

Write your script as if its operating on a single repo, then target many repos with git-xargs. Remember that at runtime, each of the scripts you select will be run, in the order you specify, once per repo that you've targeted.

Handling prerequisites and third party binaries

It is currently assumed that bash script authors will be responsible for checking for prequisites within their own scripts. If you are adding a new bash script to accomplish some new task across repos, consider using the Gruntwork bash-commons assert_is_installed pattern to ensure the operator has any required binaries installed.

Grouping your repos into separate batches

This is a pattern that ended up working out well for us as we wrote and executed more and more ambitious scripts across our many repos as a team: By breaking your target repos into separate batches, (batch1.txt, batch2.txt, batch3.txt) and starting with a few repos (or even one repo!) in the initial batches, and then gradually expanding the batches in size, you can easily test your new scripts against a few repos and double check the generated pull requests for any issues prior to widening your target batches.

How git-xargs works

This section provides a more in-depth look at how the git-xargs tool works under the hood.

  1. git-xargs will clone each of your selected repos to your machine to the /tmp/ directory of your local machine. The name of each repo, plus a random number each run, are concatenated together to form the local clone name to make the local repo easier to find in case you need to debug your script locally, e.g., terraform-aws-module-security3978298.
  2. it will checkout a local branch (whose name you can optionally specify with the --branch-name flag)
  3. it will run all your selected scripts against your selected repos 
  4. it will commit any changes in each of the repos (with a commit message you can optionally specify via the --commit-message flag)
  5. it will push your local branch with your new commits to your repo's remote 
  6. it will call the Github API to open a pull request with a title and description that you can optionally specify via the --pull-request-title and --pull-request-description flags, respectively), unless you pass the --skip-pull-requests flag
  7. it will print out a detailed run summary to STDOUT that explains exactly what happened with each repo and provide links to successfully opened pull requests that you can quickly follow from your terminal. If any repos encountered errors at runtime (whether they weren't able to be cloned, or script errors were encountered during processing, etc) all of this will be spelled out in detail in the final report so you know exactly what succeeded and what went wrong.

Tasks this tool is well-suited for

The following is a non-exhaustive list of potential use cases for git-xargs:

  • Add a LICENSE file to all of your GitHub repos, interpolating the correct year and company name into the file
  • For every existing LICENSE file across all your repos, update their copyright date to the current year
  • Update the CI build configuration in all of your repos by modifying the .circleci/config.yml file in each repo using a tool such as yq
  • Run sed commands to update or replace information across README files
  • Add new files to repos
  • Delete specific files, when present, from repos
  • Modify package.json files in-place across repos to bump a node.js dependency using jq https://stedolan.github.io/jq/
  • Update your Terraform module library from Terraform 0.13 to 0.14 . 
  • Remove stray files of any kind, when found, across repos using find and its exec option
  • Add baseline tests to repos that are missing them by copying over a common local folder where they are defined
  • Refactor multiple Golang tools to use new libraries by executing go get to install and uninstall packages, and modify the source code files' import references

If you can instrument the logic in a script, you can use git-xargs to run it across your repos!

Contributing

Contributing scripts to this project

We hope that this tool will help save you some time as you apply it to your own automations and maintenance tasks. We also welcome the community to contribute back scripts that everyone can use and benefit from.

Initially, we'll add these scripts to the ./scripts directory in this repository and will eventually organize them into sub-folders depending on their purposes / use cases. If you would like to have your script considered for inclusion in this repo, please first ensure that it is:

  • High quality: meaning free of typos, any obvious bugs or security issues
  • Generic: meaning that it is likely to be of general use to many different people and organizations, and free of any proprietary tooling or secrets

Once you've done this, please feel free to open a pull request adding your script to the ./scripts directory for consideration.

Thanks for contributing back! Our hope is that eventually this repo will contain many useful generic scripts for common maintenance and upgrading tasks that everyone can leverage to save time.

Running the tool without building the binary

Alternatively, you can run the tool directly without building the binary, like so:

./go run main.go \
  --branch-name test-branch \
  --commit-message "Add MIT License" \
  --repos data/test-repos.txt \
  $(pwd)/scripts/add-license.sh

This is especially helpful if you are developing against the tool and want to quickly verify your changes.

Running tests

Tests are included under the /cmd directory. There is a mixture of unit tests as well as CLI interface tests that verify certain flags are required, error messages are returned as expected when invalid input is provided, etc.

go test -v ./cmd

License

This code is released under the Apache 2.0 License. See LICENSE.txt

Issues
  • Implement rate-limit-aware http transport

    Implement rate-limit-aware http transport

    Description

    In the excellent bug report #59, @mkowaliszyn-coursera points out that git-xargs should respect the Github API's rate-limit headers, and I agree. This approach was very heavily "inspired" by howterraform-provider-github handles the exact same issue.

    These changes implement a rate-limit-aware HTTP transport that:

    1. Sleeps 1 second in between API calls made on behalf of the same client, thereby following the guidelines specified in Github's best practices for integrators doc
    2. Inspects Github API responses' Retry-After header when rate-limited, sleeping for the number of seconds specified by the same

    These changes also configure git-xarg's Github API client to use the rate-limit-aware http transport.

    Fixes #59

    Documentation

    TODOs

    Please ensure all of these TODOs are completed before asking for a review.

    • [x] Ensure the branch is named correctly with the issue number. e.g: feature/new-vpc-endpoints-955 or bug/missing-count-param-434.
    • [x] Update the docs.
    • [x] Keep the changes backward compatible where possible.
    • [x] Run the pre-commit checks successfully.
    • [x] Run the relevant tests successfully.
    • [x] Ensure any 3rd party code adheres with our license policy or delete this line if its not applicable.

    Related Issues

    opened by zackproser 19
  • Refactor project layout for better go build and get ergonomics

    Refactor project layout for better go build and get ergonomics

    Fixes https://github.com/gruntwork-io/git-xargs/issues/18 Fixes #7

    @mmlb Please have a look and let me know if this is what you had in mind. Thanks!

    Updates project layout which will allow for installing git-xargs with a single call to go get github.com/gruntwork-io/git-xargs.

    opened by zackproser 14
  • feat: Add --draft flag for draft pull requests

    feat: Add --draft flag for draft pull requests

    Hey folks, thanks for the neat tool!

    My team is interested in using the GitHub drafts feature so I've implemented a --draft flag here.

    I played around with a few tests but I didn't find one that felt like it provided meaningful coverage. What we want I think is a test where we check that the PR submitted to client.PullRequests.Create has the has Draft: true, which would require some storage structure on the mock used in perhaps another test like TestProcessRepo. This did seem like a lot for a flag, so wanted to see what you thought before adding it. Open to any suggestions for testing approaches!

    Also looks like my editor removed a couple extra spaces, please let me know if you'd like them reverted.

    closes #47

    opened by ryanwholey 8
  • Pull request body validation error

    Pull request body validation error

    DEBU[0010] Successfully pushed local branch to remote origin  Repo=terraform-aws-architecture-catalog
    DEBU[0011] Error opening Pull request                    Base=master Body="_**This PR was created by git-xargs. See https://github.com/gruntwork-io/prototypes/pull/152 for context.**_ We recently renamed all of our repos to follow the Terraform registry format of  (e.g., ). This PR does a search & replace to update all references to these repos to the new names. GitHub can handle redirects, so in theory, everything should work fine with the old names, but there were so many renames, that to reduce confusion, this will update most of our references to the proper values." Error="POST https://api.github.com/repos/gruntwork-io/terraform-aws-architecture-catalog/pulls: 422 Validation Failed [{Resource:PullRequest Field:base Code:invalid Message:}]" Head=refs/heads/rename-repos
    DEBU[0011] Error encountered while processing repo       Error="POST https://api.github.com/repos/gruntwork-io/terraform-aws-architecture-catalog/pulls: 422 Validation Failed [{Resource:PullRequest Field:base Code:invalid Message:}]" Repo name=terraform-aws-architecture-catalog
    
    
    bug needs design 
    opened by zackproser 7
  • Implement buffered channel for pull requests

    Implement buffered channel for pull requests

    Description

    Fixes #59

    This PR addresses #59 by implementing several new features and flags to help us respect GitHub's API rate limits when using git-xargs:

    • Distinct processing channels for expensive and non-expensive work
    • Automatic, and configurable, pull request retries when rate limited
    • Automatic, and configurable, backoff when rate limiting is detected
    • Making the pause between pull requests configurable

    In addition, three new flags to assist with configuring your git-xargs jobs so that they do not trip GitHub rate limits:

    • --seconds-between-prs The number of seconds to wait between pull requests when serially opening them at the end of a run. In local testing, setting this value to 12 seconds resolves the secondary rate limiting issues seen reliably on the same test data set prior to making these changes.
    • --seconds-to-wait-when-ratelmited The number of seconds to wait when git-xargs detects it has been rate limited
    • --max-pr-retries The number of times to retry a failed pull request before abandoning it

    TODOs

    Please ensure all of these TODOs are completed before asking for a review.

    • [x] Ensure the branch is named correctly with the issue number. e.g: feature/new-vpc-endpoints-955 or bug/missing-count-param-434.
    • [x] Update the docs.
    • [x] Keep the changes backward compatible where possible.
    • [x] Run the pre-commit checks successfully.
    • [x] Run the relevant tests successfully.

    Related Issues

    opened by zackproser 6
  • Don't try to open PR if one is already open

    Don't try to open PR if one is already open

    Fixes #32.

    Key changes:

    1. Check if a PR already exists for a given branch before trying to open up a new one.
    2. Refactor the code a bit into a updateRepo method so that if the working tree reports itself as clean, we don't try to commit, push, or open a PR at all.
    opened by brikis98 5
  • No PR opened, if branch is up to date

    No PR opened, if branch is up to date

    Hello,

    found something which is probably a bug or at least unexpected behavior.

    If, for some reason the changes I made with git-xargs exists already in the branch on github, the tool prints the following errors and does not open a PR.

    [git-xargs] DEBU[2021-04-27T08:45:41+02:00] Error pushing new branch to remote origin     Error="already up-to-date" Repo=room-management-example
    [git-xargs] DEBU[2021-04-27T08:45:41+02:00] Error encountered while processing repo       Error="already up-to-date" Repo name=room-management-example
    

    I run into this error, because I run my git-xargs command two times. The first time I run into an error on opening the PR. After fixing this, I run the command a second time and failed on the update. I had to delete the remote branch first. I think it the tool should not break on an "already up-to-date" message, because the state of the repository is like it should be, only the way to achived was different.

    Think this is an edge case and nothing urgent.

    bug 
    opened by radicarl 5
  • Support for overriding base branch names

    Support for overriding base branch names

    Support overriding base branch name (Follow up to https://github.com/gruntwork-io/git-xargs/issues/22)

    Description

    Since our git workflow uses multiple branches of the same repository in some places, I thought it would make sense to support an optional argument for overriding the base branch name globally (might be an addition to #22).

    Documentation

    • README
    • --base-branch-name=<xxx>

    TODOs

    Please ensure all of these TODOs are completed before asking for a review.

    • [x] Ensure the branch is named correctly with the issue number. e.g: feature/new-vpc-endpoints-955 or bug/missing-count-param-434.
    • [x] Update the docs.
    • [x] Keep the changes backward compatible where possible.
    • [ ] Run the pre-commit checks successfully.
    • [ ] Run the relevant tests successfully.

    Related Issues

    #22

    opened by nirnanaaa 4
  • Homebrew suport

    Homebrew suport

    Is your feature request related to a problem? Please describe.

    Simple install experience via brew install git-xargs.

    Describe the solution you'd like

    Using goreleaser this should be easy to configure.

    Example: https://github.com/estahn/cloudping/blob/master/.goreleaser.yml

    opened by estahn 4
  • Respect GitHub Secondary Rate Limits

    Respect GitHub Secondary Rate Limits

    Is your feature request related to a problem? Please describe. GitHub's secondary rate limits prevent the creation of many resources in a short time frame. For example, creating PRs is capped in a period and GitHub's limits kick in. git-xargs will report an error and failure to create a PR when this happens with a HTTP 503 error.

    git-xargs should respect the rate-limiting headers that GitHub returns and pause/sleep for that amount of time before continuing. The secondary rate limiting prevents the use of git-xargs against large amounts of repos (dozens to hundreds) because of these limits. The update scripts need to add a long pause that kills any performance (i.e. 30s) and concurrency when dealing with many repos.

    https://docs.github.com/en/rest/guides/best-practices-for-integrators#dealing-with-secondary-rate-limits

    Describe the solution you'd like It would be great if git-xargs respected the GitHub X-RateLimit-* headers and automatically slept for that duration before continuing with the API requests.

    https://docs.github.com/en/rest/overview/resources-in-the-rest-api#secondary-rate-limits

    Describe alternatives you've considered The workaround is to put a fixed pause in the update script, up to 30s to avoid being throttled and having the batch job fail.

    Additional context

    opened by mkowaliszyn-coursera 3
  • Using quotes to construct more complex commands doesn't work

    Using quotes to construct more complex commands doesn't work

    git-xargs version v0.0.12

    To Reproduce This fails: git-xargs --loglevel=DEBUG --dry-run --branch-name test --repo gruntwork-io/git-xargs "touch a && touch b"

    and so does even this: git-xargs --loglevel=DEBUG --dry-run --branch-name test --repo gruntwork-io/git-xargs "touch a"

    while this works git-xargs --loglevel=DEBUG --dry-run --branch-name test --repo gruntwork-io/git-xargs touch a

    So it seems that quotes are the problem, but they are necessary to construct complex commands.

    bug 
    opened by infraredgirl 3
  • Make printed table markdown-friendly

    Make printed table markdown-friendly

    Description

    This change uses different characters for the pipe and hyphen when printing tables and removes the top and bottom borders.

    Not sure if this makes sense to all users. The table may not look as nice in the CLI, but it can be more easily copy-pasted into GitHub issues/PRs and other markdown needs. It doesn't seem worth having a separate --render-markdown flag.

    TODOs

    Read the Gruntwork contribution guidelines.

    • [ ] Update the docs.
    • [x] Run the relevant tests successfully, including pre-commit checks.
    • [ ] Include release notes. If this PR is backward incompatible, include a migration guide.

    Release Notes (draft)

    Updated the table outputs to be markdown friendly.

    Migration Guide

    (I don't think this warrants a migration guide even though changes are made to outputs.)

    opened by rhoboat 0
  • 0.0.15 no longer creates PRs.

    0.0.15 no longer creates PRs.

    Describe the bug The latest version, 0.0.15, no longer creates PRs. I tried same command with 0.0.14 binary and it works fine.

    To Reproduce Steps to reproduce the behavior including the relevant Terraform/Terragrunt/Packer version number and any code snippets and module inputs you used.

    	./git-xargs   --branch-name mkow-test4  --repos ./java-apps.txt   --commit-message "My message"   --pull-request-title "My title"   --loglevel DEBUG   `pwd`/scripts/myScript.sh
    

    Expected behavior The PR should be created.

    Nice to have

    • [x] Terminal output
    • [ ] Screenshots

    Additional context Add any other context about the problem here.

    Teminal output:

    $ git-xargs   --branch-name mkow-test2  --repos ./java-apps.txt   --commit-message "My message"   --pull-request-title "My title"   --loglevel DEBUG   ***/updateServiceInfra.sh
    [git-xargs] INFO[2022-06-23T14:48:59-04:00] git-xargs running...
    [git-xargs] DEBU[2022-06-23T14:48:59-04:00] Looking up filename provided repo             Name=cplus-messaging-application Organization=****
    [git-xargs] DEBU[2022-06-23T14:48:59-04:00] Successfully fetched repo                     Name=cplus-messaging-application Organization=***
    [git-xargs] DEBU[2022-06-23T14:48:59-04:00] Repo will have all targeted scripts run against it  Repository=cplus-messaging-application
    [git-xargs] DEBU[2022-06-23T14:48:59-04:00] Attempting to clone repository using GITHUB_OAUTH_TOKEN  Repo=cplus-messaging-application
    [git-xargs] DEBU[2022-06-23T14:49:00-04:00] Enumerating objects: 951, done.
    Counting objects: 100% (130/130), done.
    Compressing objects: 100% (78/78), done.
    Total 951 (delta 44), reused 106 (delta 28), pack-reused 821  Repo=cplus-messaging-application
    [git-xargs] DEBU[2022-06-23T14:49:00-04:00] Created branch                                Branch Name=refs/heads/mkow-test2 Repo=cplus-messaging-application
    [git-xargs] DEBU[2022-06-23T14:49:00-04:00]                                               Repo=cplus-messaging-application
    [git-xargs] DEBU[2022-06-23T14:49:01-04:00] Executing command against local clone of repo...  Command="[***updateServiceInfra.sh]" Directory=***/workspace/git-xargs-cplus-messaging-application1237588432 Repo=cplus-messaging-application
    [git-xargs] DEBU[2022-06-23T14:49:31-04:00] Output of command [***/scripts/updateServiceInfra.sh] for repo cplus-messaging-application in directory ***/workspace/git-xargs-cplus-messaging-application1237588432:
    [git-xargs] DEBU[2022-06-23T14:49:31-04:00] Local repository worktree no longer clean, will stage and add new files and commit changes  Repo=cplus-messaging-application
    [git-xargs] DEBU[2022-06-23T14:49:32-04:00] Successfully pushed local branch to remote origin  Repo=cplus-messaging-application
    [git-xargs] INFO[2022-06-23T14:49:32-04:00] Repository successfully processed             Repo name=cplus-messaging-application
    
    
    *****************************************************************
      GIT-XARGS RUN SUMMARY @ 2022-06-23 18:49:32.233789 +0000 UTC
      Runtime in seconds: 32
    *****************************************************************
    
    
    COMMAND SUPPLIED
    
    [/***/updateServiceInfra.sh]
    
    REPO SELECTION METHOD USED FOR THIS RUN - (see README.md for more information)
    
    repos-file
     REPOS SUPPLIED VIA --repos FILE FLAG
    │───────────────────│─────────────────────────────│
    │ ORGANIZATION NAME │ URL                         │
    │───────────────────│─────────────────────────────│
    │ ***      │ cplus-messaging-application │
    │───────────────────│──────────────────[git-xargs] DEBU[2022-06-23T14:49:32-04:00] pullRequestWorker received pull request job. Delay: 0 seconds. Retries: 0 for repo: cplus-messaging-application on branch: refs/heads/mkow-test2
    [git-xargs] DEBU[2022-06-23T14:49:32-04:00] openPullRequest received job with retries: 0. Config max retries for this run: 3
    ───────────│
    
     ALL REPOS THAT WERE TARGETED FOR PROCESSING AFTER FILTERING MISSING / MALFORMED REPOS
    │─────────────────────────────│─────────────────────────────────────────────────────────────│
    │ REPO NAME                   │ REPO URL                                                    │
    │─────────────────────────────│─────────────────────────────────────────────────────────────│
    │ cplus-messaging-application │ https://***/cplus-messaging-application │
    │─────────────────────────────│─────────────────────────────────────────────────────────────│
    
    
     REPOS THAT WERE SUCCESSFULLY CLONED TO THE LOCAL FILESYSTEM
    │─────────────────────────────│─────────────────────────────────────────────────────────────│
    │ REPO NAME                   │ REPO URL                                                    │
    │─────────────────────────────│─────────────────────────────────────────────────────────────│
    │ cplus-messaging-application │ https://***/cplus-messaging-application │
    │─────────────────────────────│─────────────────────────────────────────────────────────────│
    
    
     REPOS THAT SHOWED FILE CHANGES TO THEIR WORKING DIRECTORY FOLLOWING COMMAND EXECUTION
    │─────────────────────────────│─────────────────────────────────────────────────────────────│
    │ REPO NAME                   │ REPO URL                                                    │
    │─────────────────────────────│─────────────────────────────────────────────────────────────│
    │ cplus-messaging-application │ https://***/cplus-messaging-application │
    │─────────────────────────────│─────────────────────────────────────────────────────────────│
    
    
     REPOS WHOSE SPECIFIED BRANCHES DID NOT EXIST ON THE REMOTE, AND SO WERE FIRST CREATED LOCALLY
    │─────────────────────────────│─────────────────────────────────────────────────────────────│
    │ REPO NAME                   │ REPO URL                                                    │
    │─────────────────────────────│─────────────────────────────────────────────────────────────│
    │ cplus-messaging-application │ https://***/cplus-messaging-application │
    │─────────────────────────────│─────────────────────────────────────────────────────────────│
    

    Note the async log output from the PR creation thread in the report.

    bug 
    opened by mkowaliszyn-coursera 1
  • Possible error when running repeated attempts on single repo

    Possible error when running repeated attempts on single repo

    The PR was opened successfully: https://github.com/gruntwork-io/terraform-aws-ci/pull/448, but this error was still thrown. Also the report didn't get generated correctly.

    git-xargs \
    --loglevel debug \
    --repo gruntwork-io/terraform-aws-ci \
    --branch-name update-pr-template9 \
    --seconds-between-prs 3 \
    ~/Development/repos/prototypes/git-xargs-assets/scripts/pull-request-template.sh
    [git-xargs] INFO[2022-05-25T10:40:03-07:00] git-xargs running...
    [git-xargs] DEBU[2022-05-25T10:40:03-07:00] Looking up filename provided repo             Name=terraform-aws-ci Organization=gruntwork-io
    [git-xargs] DEBU[2022-05-25T10:40:03-07:00] Successfully fetched repo                     Name=terraform-aws-ci Organization=gruntwork-io
    [git-xargs] DEBU[2022-05-25T10:40:03-07:00] Repo will have all targeted scripts run against it  Repository=terraform-aws-ci
    [git-xargs] DEBU[2022-05-25T10:40:03-07:00] Attempting to clone repository using GITHUB_OAUTH_TOKEN  Repo=terraform-aws-ci
    [git-xargs] DEBU[2022-05-25T10:40:05-07:00] Enumerating objects: 10181, done.
    Counting objects: 100% (231/231), done.
    Compressing objects: 100% (136/136), done.
    Total 10181 (delta 105), reused 199 (delta 89), pack-reused 9950  Repo=terraform-aws-ci
    [git-xargs] DEBU[2022-05-25T10:40:05-07:00] Created branch                                Branch Name=refs/heads/update-pr-template9 Repo=terraform-aws-ci
    [git-xargs] DEBU[2022-05-25T10:40:05-07:00]                                               Repo=terraform-aws-ci
    [git-xargs] DEBU[2022-05-25T10:40:05-07:00] Executing command against local clone of repo...  Command="[/Users/rhozen/Development/repos/prototypes/git-xargs-assets/scripts/pull-request-template.sh]" Directory=/var/folders/l0/nr3fgm0j39zchyhxy4gpsyz80000gn/T/git-xargs-terraform-aws-ci385613384 Repo=terraform-aws-ci
    [git-xargs] DEBU[2022-05-25T10:40:05-07:00] Output of command [/Users/rhozen/Development/repos/prototypes/git-xargs-assets/scripts/pull-request-template.sh] for repo terraform-aws-ci in directory /var/folders/l0/nr3fgm0j39zchyhxy4gpsyz80000gn/T/git-xargs-terraform-aws-ci385613384:
    [git-xargs] DEBU[2022-05-25T10:40:05-07:00] Local repository worktree no longer clean, will stage and add new files and commit changes  Repo=terraform-aws-ci
    [git-xargs] DEBU[2022-05-25T10:40:06-07:00] Successfully pushed local branch to remote origin  Repo=terraform-aws-ci
    [git-xargs] INFO[2022-05-25T10:40:06-07:00] Repository successfully processed             Repo name=terraform-aws-ci
    ERROR: sync: WaitGroup is reused before previous Wait has returned
    

    Command that was run:

    #!/usr/bin/env bash 
    
    set -e
    
    readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
    
    cp -r "$SCRIPT_DIR/../data/github-templates/pr-template.md" "./.github/pull_request_template.md"
    
    
    opened by rhoboat 3
  • Consider using patcher comments to bump versions

    Consider using patcher comments to bump versions

    So that these dependencies don't go out of date too quickly: https://github.com/gruntwork-io/git-xargs/blob/3dc1d1a726cad67a83da88eeb064950442920863/.circleci/config.yml#L9-L10

    Also worth bumping Go 1.17 now.

    opened by rhoboat 0
  • --skip-pull-requests not working

    --skip-pull-requests not working

    Docs seem to suggest that you can skip pull request and commit straight to master. --skip-pull-requests with branch name as master give the following error

    Error="a branch named \"refs/heads/master\" already exists"

    bug 
    opened by KarenTrinnyLondon 0
  • Does git-xargs support only 10 PRs?

    Does git-xargs support only 10 PRs?

    Describe the bug Using git-xargs with 10+ repositories, I end up with only 10 PRs (consistently between runs, I tried running the command multiple times)

    To Reproduce Steps to reproduce the behavior including the relevant Terraform/Terragrunt/Packer version number and any code snippets and module inputs you used. Running the git-xargs command with default settings produces only 10 PRs. For example I ran the following:

    git-xargs \
      --repos $PWD/tools/git-xargs/dbt_v1/repos \
      --branch-name branch-name-here \
      --loglevel DEBUG \
      --commit-message "My Commit Message" \
      --pull-request-description "DESCRIPTION HERE" \
      --draft \
      --dry-run \
      -- $PWD/tools/git-xargs/dbt_v1/script.sh
    

    The output looks like this:

    REPOS AGAINST WHICH PULL REQUESTS FAILED TO BE OPENED │──────────────────────────────│────────────────────────────────────────────────────────────│ │ REPO NAME │ REPO URL │ │──────────────────────────────│────────────────────────────────────────────────────────────│ │ my-11th-repo │ my-11th-repo-url │ │──────────────────────────────│────────────────────────────────────────────────────────────│

    REPOS WHOSE SPECIFIED BRANCHES DID NOT EXIST ON THE REMOTE, AND SO WERE FIRST CREATED LOCALLY │──────────────────────────────────────│────────────────────────────────────────────────────────────────────│ │ REPO NAME (11) │ REPO URL │ │──────────────────────────────────────│────────────────────────────────────────────────────────────────────│ ...I omitted content here... │──────────────────────────────────────│────────────────────────────────────────────────────────────────────│


    PULL REQUESTS OPENED


    │──────────────────────────────────────│────────────────────────────────────────────────────────────────────────────│ │ REPO NAME (10) │ PR URL │ │──────────────────────────────────────│────────────────────────────────────────────────────────────────────────────│ ...I omitted content here... │──────────────────────────────────────│────────────────────────────────────────────────────────────────────────────│

    // paste code snippets here
    

    Expected behavior I'm expecting 10+ PRs to be created, however only 10 were created. I can see that on the 11th repository a branch was created with the right content, however no PR.

    Can it be related to the tool itself or a policy I have in my organization?

    Nice to have

    • [ ] Terminal output
    • [ ] Screenshots

    Additional context Add any other context about the problem here.

    THANKS!!! Eyal

    bug 
    opened by peleyal 6
Releases(v0.0.15)
CLI tool for manipulating GitHub Labels across multiple repositories

takolabel Installation Mac $ brew install tommy6073/tap/takolabel Other platforms Download from Releases page in this repository. Usage Set variables

Takayuki NAGATOMI 9 Feb 16, 2022
Run your MapReduce workloads as a single binary on a single machine with multiple CPUs and high memory. Pricing of a lot of small machines vs heavy machines is the same on most cloud providers.

gomap Run your MapReduce workloads as a single binary on a single machine with multiple CPUs and high memory. Pricing of a lot of small machines vs he

null 20 May 1, 2022
A better way to clone, organize and manage multiple git repositories

git-get git-get is a better way to clone, organize and manage multiple git repositories. git-get Description Installation macOS Linux Windows Usage gi

Greg Dlugoszewski 58 Jun 10, 2022
git-glimpse is a command-line tool that is aimed at generating a git prompt like the one from zsh-vcs-prompt.

Git GoGlimpse git-glimpse is a command-line tool that is aimed at generating a git prompt like the one from zsh-vcs-prompt. The particularity of this

Corentin de Boisset 0 Jan 27, 2022
A CLI to replace your git commit command, so your git message can partially follow the Conventional Changelog ecosystem

COMMIT CLI A CLI to replace your git commit command, so your git message can partially follow the Conventional Changelog ecosystem. And yes, it is bui

Hisam Fahri 1 Feb 9, 2022
GitHub CLI extension to clone (or update) all repositories in an Organization, with the ability to filter via search queries.

gh-org-repo-sync GitHub CLI extension to clone all repositories in an Organization, with the ability to filter via search queries. If a local clone al

Armel Soro 8 May 16, 2022
Query git repositories with SQL. Generate reports, perform status checks, analyze codebases. 🔍 📊

askgit is a command-line tool for running SQL queries on git repositories. It's meant for ad-hoc querying of git repositories on disk through a common interface (SQL), as an alternative to patching together various shell commands.

AskGit 3.1k Jun 30, 2022
A command line tool that builds and (re)starts your web application everytime you save a Go or template fileA command line tool that builds and (re)starts your web application everytime you save a Go or template file

# Fresh Fresh is a command line tool that builds and (re)starts your web application everytime you save a Go or template file. If the web framework yo

null 0 Nov 22, 2021
A simple single-file executable to pull a git-ssh repository and serve the web app found to a self-contained browser window

go-git-serve A simple single-file executable to pull a git-ssh repository (using go-git library) and serve the web app found to a self-contained brows

Justin Searle 0 Jan 19, 2022
A command line http test tool. Maintain the case via git and pure text

httptest A command line http test tool Maintain the api test cases via git and pure text We want to test the APIs via http requests and assert the res

wklken 11 Jan 6, 2022
A minimal CLI tool to enable (or disable) dependabot for all your repositories

Enable Dependabot A minimal CLI tool to enable (or disable) dependabot for all your repositories. Installation Install via Go go get -v github.com/RiR

Rick Rackow 1 Feb 10, 2022
An open-source GitLab command line tool bringing GitLab's cool features to your command line

GLab is an open source GitLab CLI tool bringing GitLab to your terminal next to where you are already working with git and your code without switching

Clement Sam 2k Jun 25, 2022
A command line tool to prompt for a value to be included in another command line.

readval is a command line tool which is designed for one specific purpose—to prompt for a value to be included in another command line. readval prints

Venky 0 Dec 22, 2021
Go terminal app listing open pull requests in chosen GitHub repositories

go-pr-watcher About Shows open pull requests on configured GitHub repositories. Getting started Create GitHub personal token with read permissions Cre

Oleg 0 Oct 29, 2021
Mass download all github repositories(public & private) of an organization, ideally in a few seconds.

Git Mass Mass download all github repositories(public & private) of an organization, ideally in a few seconds. Writing this as a simple bash script wo

Nithin Jois 0 Dec 27, 2021
Self-host your GitHub repositories.

self-forge One day, I'd like to write a lightweight clone of GitHub. For now, here's ~100 lines of Go that host your source files. Clones all of a Git

Andrew Healey 12 Jan 6, 2022
Contextual information about your git projects, right on the command-line

gitty gitty is a smart little CLI helper for git projects, that shows you all the relevant issues, pull requests and changes at a quick glance. It cur

Christian Muehlhaeuser 367 Jun 24, 2022
A command-line driven git server

GitGo GitGo is split into three parts: The API server The GIT server The CLI client We need a couple of certificates before setting up the application

null 0 Dec 14, 2021
ghcv-cli makes it easy to view the user-created issues, pull requests, and repositories in the terminal.

ghcv-cli ghcv-cli makes it easy to view the user-created issues, pull requests, and repositories in the terminal. About Show a list of pull requests c

Kyosuke Fujimoto 14 Mar 13, 2022