manipulate and inspect VCS repositories in Go

Overview

go-vcs - manipulate and inspect VCS repositories

go-vcs is a library for manipulating and inspecting VCS repositories in Go. It currently supports Git and Mercurial (hg).

Note: the public API is experimental and subject to change until further notice.

Build Status GoDoc

Resolving dependencies

For hg blame, you need to install hglib: pip install python-hglib.

Installing

go get -u sourcegraph.com/sourcegraph/go-vcs/vcs

Implementation differences

The goal is to have all supported backends at feature parity, but until then, consult this table for implementation differences.

Feature git gitcmd hg hgcmd
vcs.CommitsOptions.Path
vcs.BranchesOptions.MergedInto
vcs.BranchesOptions.IncludeCommit
vcs.BranchesOptions.BehindAheadBranch
vcs.Repository.Committers
vcs.FileLister
vcs.UpdateResult

Contributions that fill in the gaps are welcome!

Development

First-time installation of protobuf and other codegen tools

You need to install and run the protobuf compiler before you can regenerate Go code after you change the vcs.proto file.

  1. Install protoc, the protobuf compiler. Find more details in the protobuf README.

    On OS X, you can install it with Homebrew by running:

    brew install --devel protobuf
    

    Then make sure the protoc binary is in your $PATH.

  2. Install gogo/protobuf.

    go get -u github.com/gogo/protobuf/...
    
  3. Install gopathexec:

    go get -u sourcegraph.com/sourcegraph/gopathexec
    

Regenerating Go code after changing vcs.proto

go generate sourcegraph.com/sourcegraph/go-vcs/vcs/...

Running tests

Run go test ./vcs/....

Contributors

See all contributors here.

Comments
  • Proposal: Add UpdateResult to UpdateEverything method.

    Proposal: Add UpdateResult to UpdateEverything method.

    This is an early first draft of a proposal. It comes with a sample implementation, but it's very easy to refactor and change it.

    Background

    There are situations where just updating the remote with UpdateEverything method and know if it succeeded is not enough, and it's desirable to know a few additional things:

    • Has anything changed at all?
    • More specifically, have any branches been added, deleted, or updated?

    That can enable, e.g., updating the UI when repo updates/changes are available.

    Proposal

    I propose having a way to perform a remote update and returning the results of the operation, which contains the changes to branches, if any.

    There are 2 ways I see of doing so:

    1. Add another interface and method, e.g., UpdateEverythingResult, which returns (UpdateResult, error). This is a non-API-breaking change, but it results in two very similar methods.
      • It's possible to then deprecate UpdateEverything method and remove it after some time.
    2. Change UpdateEverything method (and therefore RemoteUpdater interface) to have an additional return value, so it returns (UpdateResult, error) instead of just error as it currently does.
    3. (Implicit extra option.) This proposal can be rejected altogether.

    I think I would prefer going with the 2nd approach, because this library's API is not yet frozen, and I see this as an overall improvement of the API.

    Thoughts?

    Current Implementation

    See https://github.com/sourcegraph/go-vcs/pull/80.

    opened by dmitshur 17
  • vcs/gitcmd: ResolveRevision: Use only stdout, ignore stderr for predictable command output parsing.

    vcs/gitcmd: ResolveRevision: Use only stdout, ignore stderr for predictable command output parsing.

    • This makes it so that warnings and other non-critical messages potentially printed to stderr do not mess up parsing of the expected stdout output.
    • Command exit code being 0 is used to test if the command was successful (returned as err value from Output).
    • Fixes #30.
    opened by dmitshur 14
  • Add UpdateResult to UpdateEverything method.

    Add UpdateResult to UpdateEverything method.

    This is an early draft of a sample implementation for proposal #79. It's very easy to refactor and change it.

    • Currently implemented in gitcmd backend only, the new functionality parses the output of git remote update operation and returns it to the caller.
    • This is a breaking API change because it adds an extra return parameter to a method of RemoteUpdater interface.
    • Add tests for the parser.

    Resolves #79.

    opened by dmitshur 10
  • Add BranchesOptions for Repository.Branches method, support getting behind/ahead counts.

    Add BranchesOptions for Repository.Branches method, support getting behind/ahead counts.

    This is a breaking API change for Branches method of Repository interface. As any breaking API change, it is not done lightly, but needs to be done in order to support additional features for Branches method call. By using an options struct, future enhancements and pagination support will not incur additional API breakage.

    • It changes the signature of Branches to be more like Commits, which accepts an Options struct where parameters may be specified that describe details of the Branches query.
    • ~~It also adds a total uint return value, which will always be the total number of branches, even if fewer are returned (due to pagination, for example). Note that there is no pagination support at this time, but since it's likely to be added in the future, it's hopefully better to avoid a second API breakage at that time.~~
    • Implement support for BehindAheadBranch in vcs/gitcmd implementation. If set to a non-empty string, it uses that branch name as the base branch, and gets relative behind/ahead commit counts for each branch returned.

    TODO.

    • [ ] Add BehindAheadBranch support to vcs/git implementation.
    • [x] Add tests.
    opened by dmitshur 8
  • Use vfs.FileSystem instead of own interface.

    Use vfs.FileSystem instead of own interface.

    Closes #11.

    Getting the ball rolling on this one. It seems to be okay so far. I had to update vcsstore accordingly and will make a separate PR for that. It should work in the interim due to vendoring.

    I didn't test the native path, only the cmd-based stuff (via this and this compiling and working) so far.

    Just wanted to get this in front of your eyes so you can look it over and give feedback.

    opened by dmitshur 8
  • Windows support

    Windows support

    • replaced "git rev-parse SPEC^{commit}" with short-hand "git rev-parse SPEC^0" because {} has a special meaning in Windows
    • git client expects path arguments to use slashes, so forcing them
    • ssh expects path to key file (-i KEY) to use slashes (checked with Cygwin's git and MSYS's git)
    • forcing using slashes instead of OS-specific file separator
    • making OS-specific GIT_SSH and GIT_ASKPASS helpers (bat files on Windows vs shell scripts on Unix)
    • fixed some unit tests (actually everything but "native git" work now on Windows)
    • os.Chmod is platform-independent while os.File.Chmod is not implemented on Windows
    • On Windows, commit hash may differ when there are symlinks made by different tool (Cygwin/Msys/...) thus computing proper hash before test (affects unit tests)
    • code refactoring: extracted method to write a file and set permissions after
    • fixed "native hg" stat() method on Windows

    TODO list:

    • is there a easy way to make "native git" compile on Windows?
    • enhance TestRepository_FileSystem_Symlinks test on Windows - I made some pieces to execute on Unix/Darwin only
    opened by alexsaveliev 7
  • Breaking change: Remove total count from Commits method.

    Breaking change: Remove total count from Commits method.

    The total commit count did not respect CommitOptions. In many backends, it performed completely separate, potentially expensive logic.

    Many users of Commits discard the total count, resulting in wasted potentially expensive calculations. It makes most sense to have it available in a separate endpoint (with separate options) for interested parties. Begin by removing it from Commits method.

    This is a breaking change, but I believe it leads to a better, simpler future API. Comments welcome.

    opened by dmitshur 7
  • WIP - Move VCS impls to subpackages and create central Opener/Cloner registries

    WIP - Move VCS impls to subpackages and create central Opener/Cloner registries

    The API for go-vcs is very messy. This PR fixes it by:

    1. Moving VCS implementations (git, gitcmd, hg, hgcmd) to separate subpackages.
    2. Handling open/clone operations using a registry that VCS impl packages register opener/cloner funcs in.
    3. Removing extraneous types (HgRepository, GitRepository).

    @shurcooL, if you have any feedback, let me know. I don't believe this will break Go-Package-Store. (Sourcegraph would know for sure but our build queue is a bit deep right now. :cry:)

    opened by sqs 7
  • Using Mercurial Command Server

    Using Mercurial Command Server

    It would be much better to use Mercurial's Command Server instead of executing hg commands. There would be much less overhead (only one hg process per repository would be started… ever) which would be better for performance. Communication would happen via pipes.

    opened by brejoc 6
  • libgit2/git2go ?

    libgit2/git2go ?

    Currently, go-vcs uses exec, which is slow (process startup). Using libgit2/git2go can make this much faster (relevant in servers handling thousands of requests).

    Downside: introduces libgit2 as a dependency.

    opened by jbenet 6
  • Update git2go API.

    Update git2go API.

    Resolves #60.

    This PR updates the API. However, it doesn't work now because there seems to be an issue in latest git2go on next branch with Clone method, see https://github.com/libgit2/git2go/commit/4b9cbd78fd266767f6bdf55257c4ee2b1611bbe0#commitcomment-11919367. Edit: I've made https://github.com/libgit2/git2go/issues/217 to track that issue.

    Until that is resolved, this PR will fail.

    opened by dmitshur 5
  • Update Dependencies (Renovate Bot)

    Update Dependencies (Renovate Bot)

    This master issue contains a list of Renovate updates and their statuses.

    Open

    These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

    • [ ] Pin ubuntu Docker tag
    • [ ] Update ubuntu Docker tag to v12.10

    Advanced
    • [ ] Check this box to trigger a request for Renovate to run again on this repository
    opened by renovate[bot] 0
  • Travis: Run tests on Windows.

    Travis: Run tests on Windows.

    There has been work done to the library over time to support Windows. There's a chance for Windows support to deteriorate over time, since it's not tested often. Fix that by running tests on Windows in CI. Travis CI has recently added support for Windows, making this possible.

    References:

    • https://blog.travis-ci.com/2018-10-11-windows-early-release
    • https://twitter.com/travisci/status/1050777251862720513
    opened by dmitshur 2
  • Configure Renovate

    Configure Renovate

    Mend Renovate

    Welcome to Renovate! This is an onboarding PR to help you understand and configure settings before regular Pull Requests begin.

    🚦 To activate Renovate, merge this Pull Request. To disable Renovate, simply close this Pull Request unmerged.


    Detected Package Files

    • Dockerfile (dockerfile)

    Configuration Summary

    Based on the default config's presets, Renovate will:

    • Start dependency updates only once this onboarding PR is merged
    • Enable Renovate Dependency Dashboard creation.
    • If Renovate detects semantic commits, it will use semantic commit type fix for dependencies and chore for all others.
    • Ignore node_modules, bower_components, vendor and various test/tests directories.
    • Autodetect whether to pin dependencies or maintain ranges.
    • Rate limit PR creation to a maximum of two per hour.
    • Limit to maximum 10 open PRs at any time.
    • Group known monorepo packages together.
    • Use curated list of recommended non-monorepo package groupings.
    • A collection of workarounds for known problems with packages.
    • Run Renovate on following schedule: on the 1st through 7th day of the month

    🔡 Would you like to change the way Renovate is upgrading your dependencies? Simply edit the renovate.json in this branch with your custom config and the list of Pull Requests in the "What to Expect" section below will be updated the next time Renovate runs.


    What to Expect

    With your current configuration, Renovate will create 3 Pull Requests:

    Pin ubuntu Docker tag to 1830542
    • Schedule: ["on the 1st through 7th day of the month"]
    • Branch name: renovate/pin-dependencies
    • Merge into: master
    • Upgrade ubuntu to sha256:18305429afa14ea462f810146ba44d4363ae76e4c8dfc38288cf73aa07485005
    Update ubuntu Docker tag to v12.04.5
    • Schedule: ["on the 1st through 7th day of the month"]
    • Branch name: renovate/ubuntu-12.x
    • Merge into: master
    • Upgrade ubuntu to sha256:18305429afa14ea462f810146ba44d4363ae76e4c8dfc38288cf73aa07485005
    Update ubuntu Docker tag to v22
    • Schedule: ["on the 1st through 7th day of the month"]
    • Branch name: renovate/ubuntu-22.x
    • Merge into: master
    • Upgrade ubuntu to sha256:20fa2d7bb4de7723f542be5923b06c4d704370f0390e4ae9e1c833c8785644c1

    ❓ Got questions? Check out Renovate's Docs, particularly the Getting Started section. If you need any further assistance then you can also request help here.


    This PR has been generated by Mend Renovate. View repository job log here.

    opened by renovate[bot] 0
  • hg implementation broken.

    hg implementation broken.

    If you look at the current CI build 559 of the latest master (https://travis-ci.org/sourcegraph/go-vcs/builds/365273607), it's failing mostly because of hg.

    Go 1.4 build is failing to build the golang.org/x/crypto/poly1305 package. This is not a bug, because golang.org/x repos only maintain support for current stable and previous version of Go, and 1.4 is very old.

    Go 1.5 build is successfully installing the hglib module:

    $ sudo easy_install python-hglib
    Searching for python-hglib
    Reading https://pypi.python.org/simple/python-hglib/
    Downloading https://pypi.python.org/packages/5a/c8/b8c95d3ee5d57c7961d918c16a09ebb8b3122a87e373bbda9f1f35769ff3/python-hglib-2.5.tar.gz#md5=1a0b708eec9275028ea91068c3bd2401
    Best match: python-hglib 2.5
    Processing python-hglib-2.5.tar.gz
    Writing /tmp/easy_install-0BigtL/python-hglib-2.5/setup.cfg
    Running python-hglib-2.5/setup.py -q bdist_egg --dist-dir /tmp/easy_install-0BigtL/python-hglib-2.5/egg-dist-tmp-4LPXFX
    zip_safe flag not set; analyzing archive contents...
    Moving python_hglib-2.5-py2.7.egg to /usr/local/lib/python2.7/dist-packages
    Adding python-hglib 2.5 to easy-install.pth file
    
    Installed /usr/local/lib/python2.7/dist-packages/python_hglib-2.5-py2.7.egg
    Processing dependencies for python-hglib
    Finished processing dependencies for python-hglib
    

    But it's failing to use it:

    --- FAIL: TestRepository_BlameFile (1.34s)
    	blame_test.go:97: hg cmd: BlameFile(f, &{NewestCommit:63e47acf80095270f4e2b81e8cc01a89416c0cf3 OldestCommit: StartLine:0 EndLine:0}): EOF (stderr: Traceback (most recent call last):
    		  File "<stdin>", line 92, in <module>
    		  File "build/bdist.linux-x86_64/egg/hglib/client.py", line 390, in annotate
    		  File "build/bdist.linux-x86_64/egg/hglib/client.py", line 261, in rawcommand
    		hglib.error.CommandError: (255, 'f1f126ec4cf9: line1\n63e47acf8009: line2', 'abort: --: no such file in rev 63e47acf8009')
    		)
    --- FAIL: TestRepository_ResolveBranch (0.43s)
    	repository_test.go:1846: hg.Open("/tmp/go-vcs-test/hg920022753") failed: .hg/requires: unknown requirement: generaldelta
    --- FAIL: TestRepository_ResolveBranch_error (0.34s)
    	repository_test.go:1846: hg.Open("/tmp/go-vcs-test/hg399928286") failed: .hg/requires: unknown requirement: generaldelta
    --- FAIL: TestRepository_ResolveRevision (0.35s)
    	repository_test.go:1846: hg.Open("/tmp/go-vcs-test/hg117032946") failed: .hg/requires: unknown requirement: generaldelta
    --- FAIL: TestRepository_ResolveRevision_error (0.35s)
    	repository_test.go:1846: hg.Open("/tmp/go-vcs-test/hg811635395") failed: .hg/requires: unknown requirement: generaldelta
    --- FAIL: TestRepository_ResolveTag_error (0.35s)
    	repository_test.go:1846: hg.Open("/tmp/go-vcs-test/hg842049905") failed: .hg/requires: unknown requirement: generaldelta
    --- FAIL: TestRepository_ResolveTag (0.50s)
    	repository_test.go:1846: hg.Open("/tmp/go-vcs-test/hg028952680") failed: .hg/requires: unknown requirement: generaldelta
    --- FAIL: TestRepository_Branches (0.81s)
    	repository_test.go:1846: hg.Open("/tmp/go-vcs-test/hg753804206") failed: .hg/requires: unknown requirement: generaldelta
    --- FAIL: TestRepository_Tags (0.64s)
    	repository_test.go:1846: hg.Open("/tmp/go-vcs-test/hg820735671") failed: .hg/requires: unknown requirement: generaldelta
    --- FAIL: TestRepository_GetCommit (0.59s)
    	repository_test.go:1846: hg.Open("/tmp/go-vcs-test/hg759776108") failed: .hg/requires: unknown requirement: generaldelta
    --- FAIL: TestRepository_Commits (0.59s)
    	repository_test.go:1846: hg.Open("/tmp/go-vcs-test/hg400518341") failed: .hg/requires: unknown requirement: generaldelta
    --- FAIL: TestRepository_FileSystem_Symlinks (0.36s)
    	repository_test.go:1846: hg.Open("/tmp/go-vcs-test/hg913714375") failed: .hg/requires: unknown requirement: generaldelta
    --- FAIL: TestRepository_Commits_options (0.88s)
    	repository_test.go:1846: hg.Open("/tmp/go-vcs-test/hg192693908") failed: .hg/requires: unknown requirement: generaldelta
    --- FAIL: TestRepository_FileSystem (0.62s)
    	repository_test.go:1846: hg.Open("/tmp/go-vcs-test/hg110844604") failed: .hg/requires: unknown requirement: generaldelta
    --- FAIL: TestOpen (0.31s)
    	repository_test.go:1631: Open("hg", "/tmp/go-vcs-test/hg929900531"): .hg/requires: unknown requirement: generaldelta
    2018/04/11 19:11:06 parsing time "" as "Mon Jan 02 15:04:05 2006 -0700": cannot parse "" as "Mon"
    FAIL
    FAIL	sourcegraph.com/sourcegraph/go-vcs/vcs	9.546s
    

    tip build is having the same issues as 1.5.

    I'm going to send PR #103 that disables hg tests in CI because they're not passing (and the fix isn't trivial), so CI can pass for git which is working correctly.

    opened by dmitshur 0
  • vcs/gitcmd: need a way to UpdateEverything and propogate error/stderr

    vcs/gitcmd: need a way to UpdateEverything and propogate error/stderr

    Hello,

    We use http://gitolite.com/gitolite/ as our Git host, and an issue we have is that if a "git push" encounters error with one of the mirroring server (e.g. fail to push to mirror A, but succeed to mirror B), the following "git fetch" command will put a warning message in stderr, even if the "git fetch" works correctly.

    This breaks the usage of "gitcmd.UpdateEverything" as it tries to parse the stderr. Could I work with your team to come up with a fix so that we could optionally skip the parsing stderr part?

    I've come up with a diff https://github.com/devastating/go-vcs/commit/c42b5bcbca473c4a7a279d7a228c6b898f688b61 to attempt to address the issue. It should not break anything, but does create a new public API. Do you have any suggestions? Thank you so much.

    Nick

    opened by devastating 5
Owner
Sourcegraph
Code search and navigation for teams (self-hosted, OSS)
Sourcegraph
Tool to manage multiple git repositories

go-many-git Tool to manage multiple git repositories Requirements Proper go installation with GOPATH set git >= 1.8.5 Installation go get -u github.co

Adrien Brochard 28 Aug 27, 2022
Installs git repos onto your system and keeps them up-to-date

Gitfile Installs git repos onto your system and keeps them up-to-date. It's a lightweight package manager for things that haven't been published to a

Brad Urani 18 Jan 16, 2021
ide-gen is a tool for development workspace prepare automation by automatic VCS repositories discovery and clone and project generation for supported IDEs.

ide-gen is a tool for development workspace prepare automation by automatic VCS repositories discovery and clone and project generation for supported IDEs.

null 8 May 8, 2022
CLI for exploring AWS EC2 Spot inventory. Inspect AWS Spot instance types, saving, price, and interruption frequency.

spotinfo The spotinfo is a command-line tool that helps you determine AWS Spot instance types with the least chance of interruption and provides the s

Alexei Ledenev 73 Sep 18, 2022
List, find and inspect operating system processes in Go

ps Package ps provides functionality to find, list and inspect operating system processes, without using cgo or external binaries. Supported operating

Tobias Klauser 14 Aug 5, 2022
A tool allows you to inspect in-bound and out-bound dns messages

This tool allows you to inspect in-bound and out-bound dns messages. You can use the tool to poison your own cache. Call ./dns-mitm to start the appli

Maximilian 2 Dec 11, 2021
Lenient VCS repository URL parsing library for Go

============================================= go-vcsurl - Lenient VCS repository URL parser go-vcsurl parses VCS repository URLs in many common format

Sourcegraph 14 Dec 23, 2019
🌍 📋 A web dashboard to inspect Terraform States

?? ?? A web dashboard to inspect Terraform States

Camptocamp 1.6k Sep 27, 2022
Gopher2600 is an Atari 2600/VCS Emulator.

Gopher2600 Gopher2600 is an emulator for the Atari 2600 written in the Go language. The accuracy of the emulation is very high and the 6507, TIA and R

Stephen Illingworth 148 Sep 17, 2022
Inspect-descriptor-set - Example protobuf descriptor set inspector CLI tool

Quick little example of parsing a protobuf descriptor file. ❯ go run main.go -f

Daniel Selans 0 Jan 25, 2022
git-glimpse is a command-line tool that is aimed at generating a git prompt like the one from zsh-vcs-prompt.

Git GoGlimpse git-glimpse is a command-line tool that is aimed at generating a git prompt like the one from zsh-vcs-prompt. The particularity of this

Corentin de Boisset 0 Jan 27, 2022
ffcommander - An easy frontend to FFmpeg and Imagemagick to automatically process video and manipulate subtitles.

% FFCOMMANDER(1) ffcommander 2.39 % Mikael Hartzell (C) 2018 % 2021 Name ffcommander - An easy frontend to FFmpeg and Imagemagick to automatically pro

Mikael Hartzell 2 May 9, 2022
Explore Docker registries and manipulate Docker images!

L/S tags Utility and API to manipulate (analyze, synchronize and aggregate) images across different Docker registries. Example invocation $ lstags alp

Ivan Ilves 293 Sep 12, 2022
QueryCSV enables you to load CSV files and manipulate them using SQL queries then after you finish you can export the new values to a CSV file

QueryCSV enable you to load CSV files and manipulate them using SQL queries then after you finish you can export the new values to CSV file

Mohamed Shapan 100 Dec 22, 2021
Groupie Trackers consists on receiving a given API and manipulate the data contained in it, in order to create a site, displaying the information.

groupie-tracker Objectives Groupie Trackers consists on receiving a given API and manipulate the data contained in it, in order to create a site, disp

Mustafa Ustaz 0 Jan 13, 2022
🏃‍♂️ A new way to execute commands and manipulate command output in Go

??‍♂️ A new way to execute commands and manipulate command output in Go

Sourcegraph 17 Jul 18, 2022
Golang package to manipulate time intervals.

timespan timespan is a Go library for interacting with intervals of time, defined as a start time and a duration. Documentation API Installation Insta

null 81 Aug 27, 2022
A Golang library to manipulate strings according to the word parsing rules of the UNIX Bourne shell.

shellwords A Golang library to manipulate strings according to the word parsing rules of the UNIX Bourne shell. Installation go get github.com/Wing924

Wei He 17 Mar 15, 2022
Manipulate subtitles in GO (.srt, .ssa/.ass, .stl, .ttml, .vtt (webvtt), teletext, etc.)

This is a Golang library to manipulate subtitles. It allows you to manipulate srt, stl, ttml, ssa/ass, webvtt and teletext files for now. Available op

Quentin Renard 407 Sep 15, 2022
Handy tools to manipulate korean character.

About hangul hangul is a set of handy tools for manipulate korean character in Go language. Example package main import ( "fmt" hangu

Homin Lee 39 Jul 23, 2022