Query git repositories with SQL. Generate reports, perform status checks, analyze codebases. πŸ” πŸ“Š

Overview

GoDev BuildStatus Go Report Card TODOs codecov

askgit

askgit is a command-line tool for running SQL queries on git repositories. It's meant for ad-hoc querying of git repositories on disk through a common interface (SQL), as an alternative to patching together various shell commands. It can execute queries that look like:

-- how many commits have been authored by [email protected]?
SELECT count(*) FROM commits WHERE author_email = '[email protected]'

You can try queries on public git repositories without installing anything at https://try.askgit.com/

There's also preliminary support for executing queries against the GitHub API.

More in-depth examples and documentation can be found below.

Installation

Homebrew

brew tap augmentable-dev/askgit
brew install askgit

Go

go get -v -tags=sqlite_vtable github.com/augmentable-dev/askgit

Will use the go tool chain to install a binary to $GOBIN.

GOBIN=$(pwd) go get -v -tags=sqlite_vtable github.com/augmentable-dev/askgit

Will produce a binary in your current directory.

Using Docker

Build an image locally using docker

docker build -t askgit:latest .

Or use an official image from docker hub

docker pull augmentable/askgit:latest

Running commands

askgit operates on a git repository. This repository needs to be attached as a volume. This example uses the (bash) built-in command pwd for the current working directory

[pwd] Print the absolute pathname of the current working directory.

docker run --rm -v `pwd`:/repo:ro augmentable/askgit "SELECT * FROM commits"

Running commands from STDIN

For piping commands via STDIN, the docker command needs to be told to run non-interactively, as well as attaching the repository at /repo.

cat query.sql | docker run --rm -i -v `pwd`:/repo:ro augmentable/askgit

Usage

askgit -h

Will output the most up to date usage instructions for your version of the CLI. Typically the first argument is a SQL query string:

askgit "SELECT * FROM commits"

Your current working directory will be used as the path to the git repository to query by default. Use the --repo flag to specify an alternate path, or even a remote repository reference (http(s) or ssh). askgit will clone the remote repository to a temporary directory before executing a query.

You can also pass a query in via stdin:

cat query.sql | askgit

By default, output will be an ASCII table. Use --format json or --format csv for alternatives. See -h for all the options.

Tables

Local Git Repository

When a repo is specified (either by the --repo flag or from the current directory), the following tables are available to query.

commits

Similar to git log, the commits table includes all commits in the history of the currently checked out commit.

Column Type
id TEXT
message TEXT
summary TEXT
author_name TEXT
author_email TEXT
author_when DATETIME
committer_name TEXT
committer_email TEXT
committer_when DATETIME
parent_id TEXT
parent_count INT
blame

Similar to git blame, the blame table includes blame information for all files in the current HEAD.

Column Type
line_no INT
file_path TEXT
commit_id TEXT
line_content TEXT
stats
Column Type
commit_id TEXT
file_path TEXT
additions INT
deletions INT
files

The files table iterates over ALL the files in a commit history, by default from what's checked out in the repository. The full table is every file in every tree of a commit history. Use the commit_id column to filter for files that belong to the work tree of a specific commit.

Column Type
commit_id TEXT
path TEXT
contents TEXT
executable BOOL
branches
Column Type
name TEXT
remote BOOL
target TEXT
head BOOL
tags
Column Type
full_name TEXT
name TEXT
lightweight BOOL
target TEXT
tagger_name TEXT
tagger_email TEXT
message TEXT
target_type TEXT

GitHub Tables

This functionality is under development and likely to change

The following tables make GitHub API requests to retrieve data during query execution. As such, you should ensure the GITHUB_TOKEN environment variable is set so that API requests are authenticated. Unauthenticated API requests (no GITHUB_TOKEN) are subject to a stricter rate limit by GitHub, and may take longer to execute (query execution will try to respect the applicable rate limit).

github_org_repos and github_user_repos

These tables can be queried as table-valued functions expecting a single parameter, like so:

-- return all repos from a github *org*
SELECT * FROM github_org_repos('augmentable-dev')

-- return all repos from a github *user*
SELECT * FROM github_user_repos('augmentable-dev')
Column Type
id INT
node_id TEXT
name TEXT
full_name TEXT
owner TEXT
private BOOL
description TEXT
fork BOOL
homepage TEXT
language TEXT
forks_count INT
stargazers_count INT
watchers_count INT
size INT
default_branch TEXT
open_issues_count INT
topics TEXT
has_issues BOOL
has_projects BOOL
has_wiki BOOL
has_pages BOOL
has_downloads BOOL
archived BOOL
pushed_at DATETIME
created_at DATETIME
updated_at DATETIME
permissions TEXT
github_pull_requests

This table expects 2 parameters, github_pull_requests('augmentable-dev', 'askgit'):

SELECT count(*) FROM github_pull_requests('augmentable-dev', 'askgit') WHERE state = 'open'
Column Type
id INT
node_id TEXT
number INT
state TEXT
locked BOOL
title TEXT
user_login TEXT
body TEXT
labels TEXT
active_lock_reason TEXT
created_at DATETIME
updated_at DATETIME
closed_at DATETIME
merged_at DATETIME
merge_commit_sha TEXT
assignee_login TEXT
assignees TEXT
requested_reviewer_logins TEXT
head_label TEXT
head_ref TEXT
head_sha TEXT
head_repo_owner TEXT
head_repo_name TEXT
base_label TEXT
base_ref TEXT
base_sha TEXT
base_repo_owner TEXT
base_repo_name TEXT
author_association TEXT
merged BOOL
mergeable BOOL
mergeable_state BOOL
merged_by_login TEXT
comments INT
maintainer_can_modify BOOL
commits INT
additions INT
deletions INT
changed_files INT
github_issues

This table expects 2 parameters, github_issues('augmentable-dev', 'askgit'):

SELECT count(*) FROM github_issues('augmentable-dev', 'askgit') WHERE state = 'open'
Column Type
id INT
node_id TEXT
number INT
state TEXT
locked BOOL
title TEXT
user_login TEXT
body TEXT
labels TEXT
active_lock_reason TEXT
created_at DATETIME
updated_at DATETIME
closed_at DATETIME
merged_at DATETIME
merge_commit_sha TEXT
assignee_login TEXT
assignees TEXT
url TEXT
html_url TEXT
comments_url TEXT
events_url TEXT
repository_url TEXT
comments INT
milestone TEXT
reactions INT

Example Queries

This will return all commits in the history of the currently checked out branch/commit of the repo.

SELECT * FROM commits

Return the (de-duplicated) email addresses of commit authors:

SELECT DISTINCT author_email FROM commits

Return the commit counts of every author (by email):

SELECT author_email, count(*) FROM commits GROUP BY author_email ORDER BY count(*) DESC

Same as above, but excluding merge commits:

SELECT author_email, count(*) FROM commits WHERE parent_count < 2 GROUP BY author_email ORDER BY count(*) DESC

This is an expensive query. It will iterate over every file in every tree of every commit in the current history:

SELECT * FROM files

Outputs the set of files in the tree of a certain commit:

SELECT * FROM files WHERE commit_id='some_commit_id'

Same as above if you just have the commit short id:

SELECT * FROM files WHERE commit_id LIKE 'shortened_commit_id%'

Returns author emails with lines added/removed, ordered by total number of commits in the history (excluding merges):

SELECT count(DISTINCT commits.id) AS commits, SUM(additions) AS additions, SUM(deletions) AS deletions, author_email
FROM commits LEFT JOIN stats ON commits.id = stats.commit_id
WHERE commits.parent_count < 2
GROUP BY author_email ORDER BY commits

Returns commit counts by author, broken out by day of the week:

SELECT
    count(*) AS commits,
    count(CASE WHEN strftime('%w',author_when)='0' THEN 1 END) AS sunday,
    count(CASE WHEN strftime('%w',author_when)='1' THEN 1 END) AS monday,
    count(CASE WHEN strftime('%w',author_when)='2' THEN 1 END) AS tuesday,
    count(CASE WHEN strftime('%w',author_when)='3' THEN 1 END) AS wednesday,
    count(CASE WHEN strftime('%w',author_when)='4' THEN 1 END) AS thursday,
    count(CASE WHEN strftime('%w',author_when)='5' THEN 1 END) AS friday,
    count(CASE WHEN strftime('%w',author_when)='6' THEN 1 END) AS saturday,
    author_email
FROM commits GROUP BY author_email ORDER BY commits

Interactive mode

askgit --interactive

Will display a basic terminal UI for composing and executing queries, powered by gocui.

Exporting

You can use the askgit export sub command to save the output of queries into a sqlite database file. The command expects a path to a db file (which will be created if it doesn't already exist) and a variable number of "export pairs," specified by the -e flag. Each pair represents the name of a table to create and a query to generate its contents.

askgit export my-export-file -e commits -e "SELECT * FROM commits" -e files -e "SELECT * FROM files"

This can be useful if you're looking to use another tool to examine the data emitted by askgit. Since the exported file is a plain SQLite database, queries should be much faster (as the original git repository is no longer traversed) and you should be able to use any tool that supports querying SQLite database files.

Issues
  • Following installation instructions doesn't work?

    Following installation instructions doesn't work?

    I'm not very familiar with Go, so perhaps I'm doing something wrong?

    $ go install -v -tags=sqlite_vtable github.com/augmentable-dev/gitqlite
    can't load package: package github.com/augmentable-dev/gitqlite: cannot find package "github.com/augmentable-dev/gitqlite" in any of:
            /usr/lib/go-1.10/src/github.com/augmentable-dev/gitqlite (from $GOROOT)
            /home/erez/go/src/github.com/augmentable-dev/gitqlite (from $GOPATH)
    
    opened by erezsh 12
  • Is there a query to extract file content on a specific date?

    Is there a query to extract file content on a specific date?

    Hi to all, imagine I have repo in which I update a txt file day by day.

    Is there a way to have the version of this file on 2021-12-21?

    A query like SELECT * FROM myFile.txt AS OF TIMESTAMP('2021-12-21'); that give me in output that file at that date?

    Thank you

    opened by aborruso 8
  • Took very long time on the first run

    Took very long time on the first run

    Is it expected that the basic command from README is so heavy? I initially thought that there's something wrong with my invocation, I did this:

    docker run --rm -v `pwd`:/repo:ro augmentable/askgit "SELECT * FROM commits"
    

    Then my computer just seemed to be stuck. I was seeing resource usage like this for over a minute:

    Screenshot 2020-08-30 at 10 02 57

    Then it eventually finished after about 2.5 minutes but I was seriously worried that I'm doing something wrong, e.g., not escaping the SQL query correctly.

    What does it do on the first run? Is it building some sort of database behind the scenes? Would even "simpler" queries like SELECT count(*) FROM commits take similarly long?

    opened by borekb 6
  • Installation is broken with Homebrew

    Installation is broken with Homebrew

    Upgrading askgitdev/askgit/askgit with Homebrew on Linux (Ubuntu 20.04) is failing:

    > uname -srm
    Linux 5.14.11-051411-generic x86_64
    
    > lsb_release -d
    Description:    Ubuntu 20.04.3 LTS
    
    > brew outdated
    askgitdev/askgit/askgit (v0.4.7) < v0.4.8
    
    > brew upgrade
    ==> Auto-updated Homebrew!
    Updated 1 tap (homebrew/cask).
    ==> Updated Casks
    Updated 1 cask.
    
    Updating Homebrew...
    ==> Upgrading 1 outdated package:
    askgitdev/askgit/askgit v0.4.7 -> v0.4.8
    ==> Downloading https://github.com/askgitdev/askgit/archive/v0.4.8.tar.gz
    Already downloaded: /home/giermulnik/.cache/Homebrew/downloads/bc83f30eb7ec1aa03e0e8e020c5cd9006e5ccb1da98eb05d36d61777e2d14864--askgit-0.4.8.tar.gz
    ==> Upgrading askgitdev/askgit/askgit
      v0.4.7 -> v0.4.8
    
    ==> make
    Last 15 lines from /home/giermulnik/.cache/Homebrew/Logs/askgit/01.make:
    
    -- nuking .build/
    -- building .build/libaskgit.so
    -- building .build/askgit
    # github.com/libgit2/git2go/v32
    /home/giermulnik/.cache/Homebrew/go_mod_cache/pkg/mod/github.com/libgit2/git2go/[email protected]/Build_system_dynamic.go:12:3: error: #error "Invalid libgit2 version; this git2go supports libgit2 between v1.2.0 and v1.2.0"
       12 | # error "Invalid libgit2 version; this git2go supports libgit2 between v1.2.0 and v1.2.0"
          |   ^~~~~
    # github.com/libgit2/git2go/v32
    /home/giermulnik/.cache/Homebrew/go_mod_cache/pkg/mod/github.com/libgit2/git2go/[email protected]/Build_system_static.go:12:3: error: #error "Invalid libgit2 version; this git2go supports libgit2 between v1.2.0 and v1.2.0"
       12 | # error "Invalid libgit2 version; this git2go supports libgit2 between v1.2.0 and v1.2.0"
          |   ^~~~~
    make: *** [Makefile:17: .build/libaskgit.so] Error 2
    make: *** Waiting for unfinished jobs....
    make: *** [Makefile:23: .build/askgit] Error 2
    
    If reporting this issue please do so at (not Homebrew/brew or Homebrew/core):
      https://github.com/askgitdev/homebrew-askgit/issues
    
    > brew info libgit2
    libgit2: stable 1.3.0 (bottled), HEAD
    C library of Git core methods that is re-entrant and linkable
    https://libgit2.github.com/
    /home/linuxbrew/.linuxbrew/Cellar/libgit2/1.3.0 (102 files, 4.8MB) *
      Poured from bottle on 2021-10-01 at 00:16:35
    From: https://github.com/Homebrew/linuxbrew-core/blob/HEAD/Formula/libgit2.rb
    License: GPL-2.0-only
    ==> Dependencies
    Build: cmake βœ”, pkg-config βœ”
    Required: libssh2 βœ”
    ==> Options
    --HEAD
            Install HEAD version
    ==> Analytics
    install: 1,052 (30 days), 1,994 (90 days), 6,492 (365 days)
    install-on-request: 140 (30 days), 193 (90 days), 598 (365 days)
    build-error: 0 (30 days)
    
    opened by yermulnik 5
  • Issues with repository directories with special characters

    Issues with repository directories with special characters

    askgit has trouble with repository directories that contain characters that are either special to Go's %q string encoding or special to sqlite. Some characters cause askgit to exit with "unrecognized token" trying to create the virtual table, while others make it further and fail (or produce no data) when executing SQL statements.

    Some special characters do not render well on github, so I've included equivalent shell commands for making these directories.

    The following directories behave the same. select count(*) from commits returns no results (not the number 0 - it returns an empty resultset), while select count(*) from files panics with "panic: invalid handle":

    • back\slash (mkdir 'back\slash')
    • thing﷐ (mkdir 'thing'$'\357\267\220')
    • new line (mkdir 'new'$'\n''line')
    • doublequotes");--injection (mkdir 'doublequotes");--injection')

    The following directories all fail without running the SQL query, with an error like unrecognized token: "");":

    • quotation"marks (mkdir 'quotation"marks')
    • comma",separated" (mkdir 'comma",separated"')

    This seems to be caused by building a SQL string using fmt.Sprintf and %q, which quotes/escapes strings in a way that Go understands rather than in a way that sqlite understands. Go will format " in the middle of a string as \", which Sqlite considers to be a literal backslash character followed by the end of a string, which is why most directories with the double quote character result in "unrecognized token". For other characters it seems like Go will escape them (e.g. \ becomes \\, newline becomes \n), and Sqlite happily passes the escaped versions to the modules' Create functions, which presumably try and fail to open a directory named e.g. back\\slash instead of back\slash

    This can be reproduced in the tests by changing the fixture repo from "repo" to e.g. "repo\\" or "repo\""

    opened by nhinds 5
  • Add support for `.mailmap` files

    Add support for `.mailmap` files

    See here for context. It would be useful to be able use mappings in a .mailmap of a repo to de-duplicate authors in queries.

    I'm not entirely sure how we add support for it - maybe as a helper function that takes the contents of a .mailmap and an email address, and returns the associated name.

    Something like SELECT mailmap(<mailmap-contents>, '[email protected]')

    enhancement 
    opened by patrickdevivo 4
  • Install error: cannot find package

    Install error: cannot find package "github.com/libgit2/git2go/v30"

    When trying to install by running this:

    go get -v -tags=sqlite_vtable github.com/augmentable-dev/askgit
    

    I get this:

    ➜ go get -v -tags=sqlite_vtable github.com/augmentable-dev/askgit
    
    github.com/libgit2/git2go (download)
    cannot find package "github.com/libgit2/git2go/v30" in any of:
    	/usr/local/go/src/github.com/libgit2/git2go/v30 (from $GOROOT)
    	/home/duncan/.go/src/github.com/libgit2/git2go/v30 (from $GOPATH)
    

    My environment is as follows:

     ➜ go version
    go version go1.15.5 linux/amd64
    
    ➜ go env
    GO111MODULE=""
    GOARCH="amd64"
    GOBIN=""
    GOCACHE="/home/duncan/.cache/go-build"
    GOENV="/home/duncan/.config/go/env"
    GOEXE=""
    GOFLAGS=""
    GOHOSTARCH="amd64"
    GOHOSTOS="linux"
    GOINSECURE=""
    GOMODCACHE="/home/duncan/.go/pkg/mod"
    GONOPROXY=""
    GONOSUMDB=""
    GOOS="linux"
    GOPATH="/home/duncan/.go"
    GOPRIVATE=""
    GOPROXY="https://proxy.golang.org,direct"
    GOROOT="/usr/local/go"
    GOSUMDB="sum.golang.org"
    GOTMPDIR=""
    GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
    GCCGO="gccgo"
    AR="ar"
    CC="gcc"
    CXX="g++"
    CGO_ENABLED="1"
    GOMOD=""
    CGO_CFLAGS="-g -O2"
    CGO_CPPFLAGS=""
    CGO_CXXFLAGS="-g -O2"
    CGO_FFLAGS="-g -O2"
    CGO_LDFLAGS="-g -O2"
    PKG_CONFIG="pkg-config"
    GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build228718343=/tmp/go-build -gno-record-gcc-switches"
    
     ➜ neofetch --backend off
    
    OS: Ubuntu 20.04.1 LTS x86_64 
    Kernel: 5.4.0-52-generic 
    Shell: bash 5.0.17 
    DE: Xfce 
    Memory: 38419MiB / 64206MiB 
    
    opened by dflock 4
  • Build fails looking for 'github.com/go-git/go-billy/v5/osfs'

    Build fails looking for 'github.com/go-git/go-billy/v5/osfs'

    Build log as follows:

    [email protected]:~/tmp$ go get -v -tags=sqlite_vtable github.com/augmentable-dev/askgit
    github.com/augmentable-dev/askgit (download)
    github.com/go-git/go-git (download)
    github.com/go-git/go-billy (download)
    Fetching https://golang.org/x/sys/unix?go-get=1
    Parsing meta tags from https://golang.org/x/sys/unix?go-get=1 (status code 200)
    get "golang.org/x/sys/unix": found meta tag get.metaImport{Prefix:"golang.org/x/sys", VCS:"git", RepoRoot:"https://go.googlesource.com/sys"} at https://golang.org/x/sys/unix?go-get=1
    get "golang.org/x/sys/unix": verifying non-authoritative meta tag
    Fetching https://golang.org/x/sys?go-get=1
    Parsing meta tags from https://golang.org/x/sys?go-get=1 (status code 200)
    golang.org/x/sys (download)
    github.com/go-git/gcfg (download)
    Fetching https://gopkg.in/warnings.v0?go-get=1
    Parsing meta tags from https://gopkg.in/warnings.v0?go-get=1 (status code 200)
    get "gopkg.in/warnings.v0": found meta tag get.metaImport{Prefix:"gopkg.in/warnings.v0", VCS:"git", RepoRoot:"https://gopkg.in/warnings.v0"} at https://gopkg.in/warnings.v0?go-get=1
    gopkg.in/warnings.v0 (download)
    github.com/mitchellh/go-homedir (download)
    github.com/jbenet/go-context (download)
    Fetching https://golang.org/x/net/context?go-get=1
    Parsing meta tags from https://golang.org/x/net/context?go-get=1 (status code 200)
    get "golang.org/x/net/context": found meta tag get.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at https://golang.org/x/net/context?go-get=1
    get "golang.org/x/net/context": verifying non-authoritative meta tag
    Fetching https://golang.org/x/net?go-get=1
    Parsing meta tags from https://golang.org/x/net?go-get=1 (status code 200)
    golang.org/x/net (download)
    github.com/emirpasic/gods (download)
    github.com/sergi/go-diff (download)
    Fetching https://golang.org/x/crypto/openpgp?go-get=1
    Parsing meta tags from https://golang.org/x/crypto/openpgp?go-get=1 (status code 200)
    get "golang.org/x/crypto/openpgp": found meta tag get.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/openpgp?go-get=1
    get "golang.org/x/crypto/openpgp": verifying non-authoritative meta tag
    Fetching https://golang.org/x/crypto?go-get=1
    Parsing meta tags from https://golang.org/x/crypto?go-get=1 (status code 200)
    golang.org/x/crypto (download)
    github.com/kevinburke/ssh_config (download)
    github.com/xanzy/ssh-agent (download)
    Fetching https://golang.org/x/crypto/ssh?go-get=1
    Parsing meta tags from https://golang.org/x/crypto/ssh?go-get=1 (status code 200)
    get "golang.org/x/crypto/ssh": found meta tag get.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/ssh?go-get=1
    get "golang.org/x/crypto/ssh": verifying non-authoritative meta tag
    Fetching https://golang.org/x/crypto/ssh/knownhosts?go-get=1
    Parsing meta tags from https://golang.org/x/crypto/ssh/knownhosts?go-get=1 (status code 200)
    get "golang.org/x/crypto/ssh/knownhosts": found meta tag get.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/ssh/knownhosts?go-get=1
    get "golang.org/x/crypto/ssh/knownhosts": verifying non-authoritative meta tag
    Fetching https://golang.org/x/net/proxy?go-get=1
    Parsing meta tags from https://golang.org/x/net/proxy?go-get=1 (status code 200)
    get "golang.org/x/net/proxy": found meta tag get.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at https://golang.org/x/net/proxy?go-get=1
    get "golang.org/x/net/proxy": verifying non-authoritative meta tag
    github.com/imdario/mergo (download)
    github.com/mattn/go-sqlite3 (download)
    github.com/gitsight/go-vcsurl (download)
    github.com/olekukonko/tablewriter (download)
    github.com/mattn/go-runewidth (download)
    github.com/spf13/cobra (download)
    github.com/spf13/pflag (download)
    ../go/src/github.com/go-git/go-git/remote.go:9:2: code in directory /home/simon/go/src/github.com/go-git/go-billy/osfs expects import "github.com/go-git/go-billy/v5/osfs"
    

    On investigation, https://github.com/go-git/go-billy/v5/osfs does not exist but https://github.com/go-git/go-billy/osfs does. Suggest this is bit-rot caused by the upstream package changing its directory structure?

    opened by simon-brooke 4
  • provide releases for non-Go developers

    provide releases for non-Go developers

    It would be nice to release this as a downloadable set of binaries. I have had some experience with GoReleaser and I have to say it's a pretty nice little tool, especially if you're working in Go. I'd prefer to be able to just brew install gitqlite, which GoReleaser has good support for: https://goreleaser.com/customization/homebrew/

    opened by klauern 4
  • fix panic when querying at the empty repository

    fix panic when querying at the empty repository

    This PR fixes panic when querying at the empty repository. This is my first Go language work :) I wish I could add a test, but after digging for a few hours, I gave up writing the test case :( Sorry for not adding a test case!

    master branch's behavior:

    $ mkdir empty-git
    $ cd empty-git
    $ git init
    Initialized empty Git repository in /home/youngminz/dist/empty-git-dir/.git/
    
    $ gitqlite "select * from commits"
    panic: invalid handle
            panic: invalid handle
    
    goroutine 1 [running]:
    github.com/mattn/go-sqlite3.lookupHandleVal(0x0, 0x0, 0x0, 0x0)
            /home/youngminz/go/pkg/mod/github.com/mattn/[email protected]+incompatible/callback.go:128 +0x13d
    github.com/mattn/go-sqlite3.lookupHandle(...)
            /home/youngminz/go/pkg/mod/github.com/mattn/[email protected]+incompatible/callback.go:135
    github.com/mattn/go-sqlite3.goVClose(0x0, 0x435a61)
            /home/youngminz/go/pkg/mod/github.com/mattn/[email protected]+incompatible/sqlite3_opt_vtable.go:448 +0x2f
    github.com/mattn/go-sqlite3._cgoexpwrap_7ec2bdc2f5b0_goVClose(0x0, 0x0)
            _cgo_gotypes.go:1506 +0x64
    github.com/mattn/go-sqlite3._Cfunc_sqlite3_finalize(0x29449d8, 0x0)
            _cgo_gotypes.go:962 +0x49
    github.com/mattn/go-sqlite3.(*SQLiteStmt).Close.func1(0xc0001227b0, 0x1)
            /home/youngminz/go/pkg/mod/github.com/mattn/[email protected]+incompatible/sqlite3.go:1767 +0x5f
    github.com/mattn/go-sqlite3.(*SQLiteStmt).Close(0xc0001227b0, 0x0, 0x0)
            /home/youngminz/go/pkg/mod/github.com/mattn/[email protected]+incompatible/sqlite3.go:1767 +0xac
    github.com/mattn/go-sqlite3.(*SQLiteRows).Close(0xc0000aaa20, 0x43520a, 0x7f44b710c6d0)
            /home/youngminz/go/pkg/mod/github.com/mattn/[email protected]+incompatible/sqlite3.go:1956 +0xa7
    database/sql.(*Rows).close.func1()
            /usr/lib/go-1.13/src/database/sql/sql.go:3076 +0x3c
    database/sql.withLock(0xc45a40, 0xc0000e8280, 0xc0000f3288)
            /usr/lib/go-1.13/src/database/sql/sql.go:3184 +0x6d
    database/sql.(*Rows).close(0xc0000e8300, 0x0, 0x0, 0x0, 0x0)
            /usr/lib/go-1.13/src/database/sql/sql.go:3075 +0x129
    database/sql.(*Rows).Close(0xc0000e8300, 0xc000124640, 0xc00009d8c0)
            /usr/lib/go-1.13/src/database/sql/sql.go:3059 +0x33
    panic(0xa7f040, 0xc31020)
            /usr/lib/go-1.13/src/runtime/panic.go:679 +0x1b2
    github.com/mattn/go-sqlite3.lookupHandleVal(0x0, 0x0, 0x0, 0x0)
            /home/youngminz/go/pkg/mod/github.com/mattn/[email protected]+incompatible/callback.go:128 +0x13d
    github.com/mattn/go-sqlite3.lookupHandle(...)
            /home/youngminz/go/pkg/mod/github.com/mattn/[email protected]+incompatible/callback.go:135
    github.com/mattn/go-sqlite3.goVFilter(0x0, 0x0, 0x2950630, 0x0, 0x29457e8, 0x2)
            /home/youngminz/go/pkg/mod/github.com/mattn/[email protected]+incompatible/sqlite3_opt_vtable.go:464 +0x40
    github.com/mattn/go-sqlite3._cgoexpwrap_7ec2bdc2f5b0_goVFilter(0x0, 0x0, 0x2950630, 0x0, 0x29457e8, 0x0)
            _cgo_gotypes.go:1535 +0x9e
    github.com/mattn/go-sqlite3._Cfunc__sqlite3_step_internal(0x29449d8, 0x0)
            _cgo_gotypes.go:414 +0x49
    github.com/mattn/go-sqlite3.(*SQLiteRows).nextSyncLocked.func1(0xc0000aaa20, 0xc00014a0c0)
            /home/youngminz/go/pkg/mod/github.com/mattn/[email protected]+incompatible/sqlite3.go:2030 +0x62
    github.com/mattn/go-sqlite3.(*SQLiteRows).nextSyncLocked(0xc0000aaa20, 0xc00013c1c0, 0xe, 0xe, 0xc00013c1c0, 0xe0)
            /home/youngminz/go/pkg/mod/github.com/mattn/[email protected]+incompatible/sqlite3.go:2030 +0x43
    github.com/mattn/go-sqlite3.(*SQLiteRows).Next(0xc0000aaa20, 0xc00013c1c0, 0xe, 0xe, 0x0, 0x0)
            /home/youngminz/go/pkg/mod/github.com/mattn/[email protected]+incompatible/sqlite3.go:2007 +0x2fb
    database/sql.(*Rows).nextLocked(0xc0000e8300, 0x430000)
            /usr/lib/go-1.13/src/database/sql/sql.go:2767 +0xd5
    database/sql.(*Rows).Next.func1()
            /usr/lib/go-1.13/src/database/sql/sql.go:2745 +0x3c
    database/sql.withLock(0xc47100, 0xc0000e8330, 0xc0000f3ad0)
            /usr/lib/go-1.13/src/database/sql/sql.go:3184 +0x6d
    database/sql.(*Rows).Next(0xc0000e8300, 0xc00013c000)
            /usr/lib/go-1.13/src/database/sql/sql.go:2744 +0x87
    github.com/augmentable-dev/gitqlite/cmd.tableDisplay(0xc0000e8300, 0x43520a, 0xc0000e8300)
            /home/youngminz/dist/gitqlite/cmd/root.go:233 +0x1ba
    github.com/augmentable-dev/gitqlite/cmd.displayDB(0xc0000e8300, 0x0, 0xc0000a4000)
            /home/youngminz/dist/gitqlite/cmd/root.go:136 +0x7d
    github.com/augmentable-dev/gitqlite/cmd.glob..func1(0x1031420, 0xc00010eaa0, 0x1, 0x1)
            /home/youngminz/dist/gitqlite/cmd/root.go:104 +0x2a6
    github.com/spf13/cobra.(*Command).execute(0x1031420, 0xc00009c030, 0x1, 0x1, 0x1031420, 0xc00009c030)
            /home/youngminz/go/pkg/mod/github.com/spf13/[email protected]/command.go:846 +0x2aa
    github.com/spf13/cobra.(*Command).ExecuteC(0x1031420, 0x0, 0x0, 0x0)
            /home/youngminz/go/pkg/mod/github.com/spf13/[email protected]/command.go:950 +0x349
    github.com/spf13/cobra.(*Command).Execute(...)
            /home/youngminz/go/pkg/mod/github.com/spf13/[email protected]/command.go:887
    github.com/augmentable-dev/gitqlite/cmd.Execute()
            /home/youngminz/dist/gitqlite/cmd/root.go:111 +0x2d
    main.main()
            /home/youngminz/dist/gitqlite/gitqlite.go:8 +0x20
    

    After my fix:

    $ gitqlite "select * from commits"
    repository is empty
    
    opened by youngminz 4
  • FEAT: improve git tables interface

    FEAT: improve git tables interface

    This pull-request improves the git virtual tables, building upon previous functionality and making following changes:

    • Make all git modules into table-valued functions [1] This switch allows us to support multi-repository queries.
    • Replace branches and tags with a unified refs table. See PRAGMA table_info(refs) for more info.
    • Drop the blame table
    • Add support for services.RepoLocator. Implementations of this interface could provide support for locating repositories at different locations while keeping the core agnostic of the fact where the repository lives.

    It also switches to go-git (away from libgit2) as the underlying git library provider. The switch is justified as go-git provides more Go-like access to git's data and is relatively more easier to extend. And being written in pure Go, it simplifies the build process.

    opened by riyaz-ali 3
  • HTTP 502 error querying private repo with GITHUB_TOKEN

    HTTP 502 error querying private repo with GITHUB_TOKEN

    I'm trying to get information about our pull requests from an internal repo and I'm getting a 502 error and I'm not sure how to debug what the issue is:

    mergestat "SELECT count(*) from github_repo_prs('private-org/private-repo');" -v
    Apr 19 12:57:06 INF starting GitHub repo_pull_requests iterator for private-org/private-repo name=private-repo owner=private-org per-page=100
    Apr 19 12:57:06 INF fetching page of repo_pull_requests for private-org/private-repo cursor=null name=private-repo owner=private-org per-page=100
    Apr 19 12:57:13 INF fetching page of repo_pull_requests for private-org/private-repo cursor=Y3Vyc29yOnYyOpHOGcCNUA== name=private-repo owner=private-org per-page=100
    +----------+
    | COUNT(*) |
    +----------+
    +----------+
    Apr 19 12:57:28 ERR failed to output resultset: non-200 OK status code: 502 Bad Gateway body: "{\n   \"data\": null,\n   \"errors\":[\n      {\n         \"message\":\"Something went wrong while executing your query. This may be the result of a timeout, or it could be a GitHub bug. Please include `51EA:050C:152958:3CEDAD:625EF7F3` when reporting this issue.\"\n      }\n   ]\n}\n"
    
    opened by klauern 5
  • Hardening binary & shared library

    Hardening binary & shared library

    I'm one of the package maintainers for Arch Linux and I also maintain a few packages on the AUR, which mergestat can be found on.

    Just wondering if there's any interest in RELRO/PIE being applied to the binary & shared library?

    I generally try and apply these to all the Go-related packages that I maintain due to our Go package guidelines. I've found that mergestat seems to be working fine with these applied, as per this commit.

    opened by grawlinson 2
  • Finding file paths affected by a commit

    Finding file paths affected by a commit

    I've been trying to work out how to filter file paths affected by a specific commit. Or in other words, show me the files modified by a particular commit, or maybe all the files affected when a particular merge happens.

    As far as I can tell, each commit hash in the files table has a record for every file in the tree at that time. I'm struggling to see where in the data model the file paths changed in each commit might be available.

    Thanks!

    opened by eddiesholl 1
  • handle null values in `summarize` queries better

    handle null values in `summarize` queries better

    https://github.com/mergestat/mergestat/blob/main/cmd/summarize/commits/commits.go#L63-L70

    It's possible for author_name / author_email to be null, in which case we have a Scan error - should use sql.NullString in these cases

    bug 
    opened by patrickdevivo 0
  • support for github enterprise

    support for github enterprise

    Hi

    Interesting project! Currently trying to run https://docs.mergestat.com/miscellaneous/cloning-all-org-repos against github enterprise, but failing on auth. Is there any way to set the custom github enterprise url so I can auth to that instead of github.com?

    enhancement 
    opened by andaag 1
Releases(v0.5.7)
SQL builder and query library for golang

__ _ ___ __ _ _ _ / _` |/ _ \ / _` | | | | | (_| | (_) | (_| | |_| | \__, |\___/ \__, |\__,_| |___/ |_| goqu is an expressive SQL bu

Doug Martin 1.6k Jun 24, 2022
SQL query builder for Go

GoSQL Query builder with some handy utility functions. Documentation For full documentation see the pkg.go.dev or GitBook. Examples // Open database a

Travis Harmon 23 Apr 25, 2022
Type safe SQL builder with code generation and automatic query result data mapping

Jet Jet is a complete solution for efficient and high performance database access, consisting of type-safe SQL builder with code generation and automa

null 723 Jun 24, 2022
A Go (golang) package that enhances the standard database/sql package by providing powerful data retrieval methods as well as DB-agnostic query building capabilities.

ozzo-dbx Summary Description Requirements Installation Supported Databases Getting Started Connecting to Database Executing Queries Binding Parameters

Ozzo Framework 551 Jun 3, 2022
Type safe SQL query builder and struct mapper for Go

sq (Structured Query) ?? ?? sq is a code-generated, type safe query builder and struct mapper for Go. ?? ?? Documentation β€’ Reference β€’ Examples This

null 162 Jun 24, 2022
Fast SQL query builder for Go

sqlf A fast SQL query builder for Go. sqlf statement builder provides a way to: Combine SQL statements from fragments of raw SQL and arguments that ma

Vlad Glushchuk 70 Jun 24, 2022
gosq is a parsing engine for a simplicity-focused, template-based SQL query builder for Go.

gosq is a parsing engine for a simplicity-focused, template-based SQL query builder for Go.

Sang-gon Lee 49 Apr 26, 2022
SQL query helper

SQL query helper

Aleksey Nikitin 1 Nov 7, 2021
Go fearless SQL. Sqlvet performs static analysis on raw SQL queries in your Go code base.

Sqlvet Sqlvet performs static analysis on raw SQL queries in your Go code base to surface potential runtime errors at build time. Feature highlights:

QP Hou 446 Jun 19, 2022
Generate type safe Go from SQL

sqlc: A SQL Compiler sqlc generates type-safe code from SQL. Here's how it works: You write queries in SQL.

Kyle Conroy 5.8k Jun 26, 2022
Go database query builder library for PostgreSQL

buildsqlx Go Database query builder library Installation Selects, Ordering, Limit & Offset GroupBy / Having Where, AndWhere, OrWhere clauses WhereIn /

Arthur Kushman 63 Jun 12, 2022
Query AWS Athena and download the result as CSV.

Overview This tool can download an Athena SQL query results in CSV format. Installation Using Homebrew: $ brew tap flowerinthenight/tap $ brew install

null 5 Nov 11, 2021
Bulk query SQLite database over the network

SQLiteQueryServer Bulk query SQLite database over the network. Way faster than SQLiteProxy!

Assaf Morami 48 May 20, 2022
Simple query builder for MongoDB

?? greenleaf - simple, type safe and easy to use query builder for MongoDB Installation To install use: go get github.com/slavabobik/greenleaf Quick

Slava 72 Jun 22, 2022
Easy JSON Query Processor with a Lispy syntax in Go

jql Hey there! You're probably here cause you're fed up with other json query processors being too complicated to use for anything surpassing simple s

Jacob Martin 863 Jun 21, 2022
Bluge, will this document match this query?

sour Will this bluge.Document match this bluge.Query? This library allows you to efficiently answer this question. s := sour.New(bluge.InMemoryOnlyCo

Bluge Labs 3 Jun 27, 2022
Tag based url Query parameters Constructor.

taqc ?? Tag based url Query parameters Constructor. (This is pronounced as same as "taxi") Synopsis type Query struct { Foo string `ta

moznion 2 Jan 11, 2022
qclean lets you to clean up search query in japanese.

qclean qclean lets you to clean up search query in japanese. This is mainly used to remove wasted space. Quick Start package main var cleaner *qclean

po3rin 0 Jan 4, 2022
Querydecoder - Optional query parameter decoder for Golang

Optional query parameter decoder for Golang Example import ( "github.com/ritwic

Ritwick Dey 8 Jun 10, 2022