Language Server Indexing Format (LSIF) generator for Go

Overview

Go LSIF indexer

Visit https://lsif.dev/ to learn about LSIF.

Installation

Binary downloads are available on the releases tab.

Installation: Linux

curl -L  https://github.com/sourcegraph/lsif-go/releases/download/v1.2.0/src_linux_amd64 -o /usr/local/bin/lsif-go
chmod +x /usr/local/bin/lsif-go

Installation: MacOS

curl -L  https://github.com/sourcegraph/lsif-go/releases/download/v1.2.0/src_darwin_amd64 -o /usr/local/bin/lsif-go
chmod +x /usr/local/bin/lsif-go

Installation: Docker

docker pull sourcegraph/lsif-go:v1.2.0

Indexing your repository

After installing lsif-go onto your PATH, run the command in the root where your go.mod file is located.

$ lsif-go -v
✔ Loading packages... Done (753.22ms)
✔ Emitting documents... Done (72.76µs)
✔ Adding import definitions... Done (86.24µs)
✔ Indexing definitions... Done (16.83ms)
✔ Indexing references... Done (93.36ms)
✔ Linking items to definitions... Done (8.46ms)
✔ Emitting contains relations... Done (294.13µs)

Stats:
	Wall time elapsed:   873.2ms
	Packages indexed:    14
	Files indexed:       53
	Definitions indexed: 1756
	Elements emitted:    35718
	Packages traversed:  40

Use lsif-go --help for more information.

Updating your index

To keep your index up-to-date, you can add a step to your CI to generate new data when your repository changes. See our documentation on adding LSIF to your workflows.

Issues
  • Add moniker support

    Add moniker support

    This PR

    • Adds moniker output to the lsif-go binary
    • Adds a new utility to add read go.mod and go.sum to and rewrite monikers for external use
    • Adds support for stdout output for lsif-go (this allows us to stream lsif-go output directly to lsif-gomod)
    • Refactors the binary arguments to act like lsif-tsc (now that we have two cmds, this was a good opportunity)
    opened by efritz 7
  •  indexer: prepare to index documentSymbols (do not index duplicate definitions)

    indexer: prepare to index documentSymbols (do not index duplicate definitions)

    After spending all day debugging, I think I finally understand what is going on here and answered my own issue :) PTAL @efritz

    Helps https://github.com/sourcegraph/sourcegraph/issues/19591

    The first commit

    This prepares the indexer to index textDocument/documentSymbols. The problem here is that indexing symbols involves emitting ranges for the symbol definitions, and so does our existing indexing for indexDefinition. LSIF ranges cannot overlap or be duplicative, so we must effectively make symbol and definition indexing work together. The idea is that symbol indexing will emit ranges for the definitions it encounters, and traditional definition indexing will merely reuse those or emit a new range if needed.

    At present, this causes all tests to fail due to the way we pre-emptively mark a range for generation during definition indexing:

    --- FAIL: TestIndexer (0.63s)
        --- FAIL: TestIndexer/check_Parallel_function_hover_text (0.00s)
            indexer_test.go:35: could not find target range
        --- FAIL: TestIndexer/check_external_package_hover_text (0.00s)
            indexer_test.go:62: could not find target range
        --- FAIL: TestIndexer/check_errs_definition (0.00s)
            indexer_test.go:94: incorrect definition count. want=1 have=0
        --- FAIL: TestIndexer/check_wg_references (0.00s)
            indexer_test.go:108: incorrect reference count. want=4 have=3
        --- FAIL: TestIndexer/check_NestedB_monikers (0.00s)
            indexer_test.go:122: could not find target range
        --- FAIL: TestIndexer/check_typeswitch (0.00s)
            indexer_test.go:143: could not find target range
        --- FAIL: TestIndexer/check_typealias (0.00s)
            indexer_test.go:209: could not find target range
        --- FAIL: TestIndexer/check_typealias_reference (0.00s)
            indexer_test.go:255: incorrection definition count. want=1 have=0
        --- FAIL: TestIndexer/check_typealias_anonymous_struct (0.00s)
            indexer_test.go:296: could not find target range
    FAIL
    FAIL	github.com/sourcegraph/lsif-go/internal/indexer	17.393s
    FAIL
    

    The second commit

    indexer: remove pre-emptive range marking for definitions

    Prior to this change, we would pre-emptively "mark" a range for a definition as being actively produced. This appears to have been because indexDefinition would always emit a new range, and this it was necessary to ensure that two parallel definition indexing processes would not emit the same range.

    Since my prior commit, however, indexDefinition only emits a new range if one does not exist. So pre-emptive range marking is not required anymore.

    This causes all tests to pass.

    api-docs 
    opened by slimsag 6
  • bug: no J2D or find-refs of switch-case blocks over an object's type

    bug: no J2D or find-refs of switch-case blocks over an object's type

    Code like the following:

    switch v := obj.(type) {
      case A: f1(v, v)
      case B: f2(v, v)
    }
    

    does not get indexed properly because when processing definitions, the definition for v has a nil associated def object, since there's a different def object for the v in each branch. Then when we encounter each use of v, there is an associated def object but since it's not indexed in our definitions we only output a hover result.

    Discussion:

    • If there's a way to know that this is why the def object is nil during the definitions pass, that would likely be the right way to solve.
    • What LSIF should we actually output here? IMO:
      • Each case branch should have a different hover result cause they have different types.
      • The def result is canonical.
      • I think the reference result should point to all uses in every branch. Kind of weird to have a reference return results with different types, but I think the experience would be weirder if you hover over one of these and only the symbols in that branch highlight. Also the definition would need to have everything as a reference result anyway.
      • Open question: what should go in the hover result when hovering the definition?

    Therefore the resulting LSIF structure should be:

    vdef -> resultSet1
    resultSet1 -> hoverResult1
    resultSet1 -> resultSetShared
    resultSetShared -> defResult1
    resultSetShared -> refResult1
    defResult1 -> vdef
    refResult1 -> vdef
    vref1 -> resultSet2
    resultSet2 -> hoverResult2
    resultSet2 -> resultSetShared
    refResult1 -> vref1
    vref2 -> resuiltSet2
    refResult1 -> vref2
    vref3 -> resultSet3
    resultSet3 -> hoverResult3
    resultSet3 -> resultSetShared
    refResult1 -> vref3
    vref4 -> resuiltSet3
    refResult1 -> vref4
    

    Thoughts?

    Also how does the worker process this graph?

    bug team/code-intelligence 
    opened by shrouxm 6
  • inVs is null

    inVs is null

    In the dump of gorilla/mux, there is an edge

    {
      id: '12458',
      type: 'edge',
      label: 'contains',
      outV: '96',
      inVs: null
    }
    

    this is invalid, inVs should be an empty array or this edge should not be included.

    bug feedback needed 
    opened by felixfbecker 6
  • What about a golang version of LSIF server?

    What about a golang version of LSIF server?

    I read some blogs for LSIF protocol recently, but I only found LSIF servers written in typescript. Is there any plan to write a golang version of server to serve the LSIF dumps?

    opened by blanet 5
  • Docker images not running go 1.16

    Docker images not running go 1.16

    Looking at the images on Docker Hub the latest one is f9c2d9cf3bdf which is running go 1.14.5 but before that image was even made the Dockerfile was updated in the repo to 1.16. Shouldn't the latest images on Docker Hub be running 1.16 as well?

    opened by jameshartig 4
  • Switch to alpine as base image for lsif-go

    Switch to alpine as base image for lsif-go

    This reduces the image size from 841MB to 346MB, should give a significant boost in startup time. Q: Do we rely on any tools other than git to be installed?

    opened by eseliger 4
  • internal/indexer: add method for determining whether a package should be indexed

    internal/indexer: add method for determining whether a package should be indexed

    This is pulled out of https://github.com/sourcegraph/lsif-go/pull/129#discussion_r580296179 - but with actual tests for it and more documentation explaining what it's doing and why.

    Helps https://github.com/sourcegraph/sourcegraph/issues/19591

    api-docs 
    opened by slimsag 4
  • Make monikers for struct fields more efficient

    Make monikers for struct fields more efficient

    Similar to the way we preload hover text (see https://github.com/sourcegraph/lsif-go/commit/c165bfff52e9d4f87891bba497e3b70fea144d89), we can also preload nested container names for moniker paths. This saves us from re-tracing the same AST nodes.

    We piggyback on the hover text preload step and keep track of the relevant parent nodes and stash them into a map for later fast retrieval. There's still some small opportunities here for cleaner code which I'll probably tackle later (for example, we keep the entire ast path to the current node but we only look at the last three, and only look at certain node types - we can store fewer things during this traversal).

    Reviewers: The logic changes are in the first two commits. The remaining commits are code style changes, a struct rename (HoverLoader to Preloader), and test updates.

    opened by efritz 4
  • Convert from string to int identifiers.

    Convert from string to int identifiers.

    The lsif-tsc utility uses either strings or uuid strings, not int-like strings. This change will make our implementation mirror that a bit more closely (with some small space savings during transmission as well).

    opened by efritz 4
  • Bump checkout action to v2.

    Bump checkout action to v2.

    Previously, we discovered a problem with the sg/sg repo, only when using v2 of the checkout action. However, this problem was not caught earlier because lsif-go was using v1. We should keep the action versions in sync across repos.

    opened by varungandhi-src 3
  • Support private git repos

    Support private git repos

    On a repo with a private dependency, the command go mod download fails when git prompts for a username + password.

    There are a few ways to authenticate:

    • Username + password
    • GitHub token
    • SSH key

    Perhaps the info could be passed via environment variables or mounting files in the lsif-go container.

    team/language-tools 
    opened by chrismwendt 0
  • Can't find associated interface method in some cases

    Can't find associated interface method in some cases

    This works: https://sourcegraph.com/github.com/sourcegraph/[email protected]/-/blob/internal/codeintel/dependencies/internal/store/store.go?L53:17#tab=implementations_go

    CleanShot 2022-07-13 at 16 24 09

    This doesn't: https://sourcegraph.com/github.com/sourcegraph/[email protected]/-/blob/internal/database/global_state.go?L45:28#tab=implementations_go

    CleanShot 2022-07-13 at 16 22 43

    Both have precise code intel.

    team/language-tools 
    opened by chrismwendt 0
  • Incorrect navigation for typealias RHS if typealias shadows it

    Incorrect navigation for typealias RHS if typealias shadows it

    It looks like when we have type X = p.X, the hover doc (and hence go to definition etc) end up referring to the X on the LHS instead of p.X. (try it here)

    incorrect hover doc

    Originally reported by Robert Lin in Slack.

    bug team/language-tools 
    opened by varungandhi-src 1
  • Missing code intel popover buttons for cross-repo references

    Missing code intel popover buttons for cross-repo references

    (Tentatively putting this on lsif-go, not sure if this is an extension issue. The buttons do show up for in-repo references, as well as for cross-repo lsif-typescript references.)

    Sourcegraph URL

    code intel popover with missing buttons

    This should be working, right?

    bug team/language-tools 
    opened by varungandhi-src 0
  • Incorrect Go to Definition for

    Incorrect Go to Definition for "context" import

    See screen recording below. Right now, it is taking me to https://sourcegraph.com/github.com/golang/[email protected]/-/blob/misc/cgo/testplugin/plugin_test.go?L9:3 despite having precise code intel (sorry the 'Precise' label isn't showing in the recording; I was doing it for accessibility).

    https://user-images.githubusercontent.com/93103176/167314970-ba50c302-71e7-4cb4-8367-4d695c0e9709.mov

    team/language-tools 
    opened by varungandhi-src 0
  • Incorrect Go to Definition for io.Writer from sourcegraph fork

    Incorrect Go to Definition for io.Writer from sourcegraph fork

    At https://sourcegraph.com/github.com/C-EO/[email protected]/-/blob/cmd/server/shared/copy.go?L73, the code intel popover for Writer in io.Writer says that the code intel is "Precise". However, if I click Go to Definition, it takes me to https://sourcegraph.com/github.com/golang/[email protected] 63b968f4f86f4c23ce92b7ac2feda4fc7ca17c8e/-/blob/src/io/multi_test.go?L84:3&subtree=true, which is not the definition of Go's io.Writer.

    Screenshot showing issue in action team/language-tools 
    opened by varungandhi-src 1
Releases(v1.9.0)
Owner
Sourcegraph
Code search and navigation for teams (self-hosted, OSS)
Sourcegraph
A collection of projects you can build for learning Go programming language.

Overview We've all been there before... You are just starting to pick up a new programming language (like Go!) and things are going great. The tutoria

Sujan Tamang 1 Mar 29, 2022
Phalanx is a cloud-native full-text search and indexing server written in Go built on top of Bluge that provides endpoints through gRPC and traditional RESTful API.

Phalanx Phalanx is a cloud-native full-text search and indexing server written in Go built on top of Bluge that provides endpoints through gRPC and tr

Minoru Osuka 247 Aug 10, 2022
BuntDB is an embeddable, in-memory key/value database for Go with custom indexing and geospatial support

BuntDB is a low-level, in-memory, key/value store in pure Go. It persists to disk, is ACID compliant, and uses locking for multiple readers and a sing

Josh Baker 3.8k Aug 7, 2022
A modern text indexing library for go

bleve modern text indexing in go - blevesearch.com Features Index any go data structure (including JSON) Intelligent defaults backed up by powerful co

bleve 8.5k Aug 9, 2022
indexing library for Go

Bluge modern text indexing in go - blugelabs.com Features Supported field types: Text, Numeric, Date, Geo Point Supported query types: Term, Phrase, M

Bluge Labs 1.3k Aug 10, 2022
Document Indexing and Searching Library in Go

Fehrist Fehrist is a pure Go library for indexing different types of documents. Currently it supports only CSV and JSON but flexible architecture give

Adnan Siddiqi 16 May 22, 2022
Package for indexing zip files and storing a compressed index

zipindex zipindex provides a size optimized representation of a zip file to allow decompressing the file without reading the zip file index. It will o

High Performance, Kubernetes Native Object Storage 27 May 10, 2022
DipDup plugin for selective metadata indexing

DipDup metadata indexer DipDup service for indexing contract and token metadata. Based on TzKT indexer. For start synchronization of DipDup state TzKT

DipDup 14 Jun 29, 2022
High-performance, columnar, in-memory store with bitmap indexing in Go

This package contains a high-performance, columnar, in-memory storage engine that supports fast querying, update and iteration with zero-allocations and bitmap indexing.

Roman Atachiants 963 Aug 13, 2022
A Go implementation of the core algorithm in paper

Boolean Expression Indexer Go library A Go implementation of the core algorithm in paper <Indexing Boolean Expression>, which already supports the fol

wangyi 49 Jul 19, 2022
google indexing api example (golang)

gindex google indexing api example (golang) 動機 search console のカバレージを見ると、除外が 170 ほどあった。 検出 - インデックス未登録が 84 件 クロール済み - インデックス未登録が 72 件 URL 検査で「インデックス登録

Seiji MORIYA 0 Nov 15, 2021
Go-explosion - Distributed indexing and searching in Go/Golang

go-explosion Distributed indexing and searching in Go/Golang. This library does

Mooij Technologies 1 Jan 1, 2022
This is a close to decentralized RSS3 Network implementation of RSS3 protocol v0.4.0 with full indexing function in Go

This is a close to decentralized RSS3 Network implementation of RSS3 protocol v0.4.0 with full indexing function in Go

Natural Selection Labs 48 Aug 4, 2022
Read metrics from a Message Queue in Json format and expose them in a Prometheus compatible format

mq2prom Read metrics from a Message Queue in Json format and expose them in a Prometheus compatible format. Currently only works for MQTT compatible M

International Business Machines 1 Jan 24, 2022
Using NFP (Number Format Parser) you can get an Abstract Syntax Tree (AST) from Excel number format expression

NFP (Number Format Parser) Using NFP (Number Format Parser) you can get an Abstract Syntax Tree (AST) from Excel number format expression. Installatio

fossabot 0 Feb 4, 2022
Musgo is a Go code generator for binary MUS format with validation support.

Musgo is a Go code generator for binary MUS format with validation support. Generated code converts data to and from MUS format.

null 41 Aug 7, 2022
Snowflake - Simple twitter's snowflake uniquely identifiable descriptors (IDs) format generator for Go

Snowflake Dead simple and fast Twitter's snowflake id generator in Go. Installation go get github.com/HotPotatoC/snowflake Usage Generating a snowflak

Juan Christian 3 Jan 8, 2022
A Go language binding for encodeing and decoding data in the bencode format that is used by the BitTorrent peer-to-peer file sharing protocol.

bencode-go A Go language binding for encoding and decoding data in the bencode format that is used by the BitTorrent peer-to-peer file sharing protoco

Jack Palevich 185 Jul 20, 2022
The Snappy compression format in the Go programming language.

The Snappy compression format in the Go programming language. To download and install from source: $ go get github.com/golang/snappy Unless otherwis

Go 1.3k Aug 11, 2022
indodate is a plugin for golang programming language for date convertion on indonesian format

indodate is a package for golang programming language for date conversion on indonesian format

Eko Kurniadi 1 Oct 23, 2021
DND-magic-item-Generator - D&D magic item generator like in Diablo

DND-magic-item-Generator D&D magic item generator like in Diablo Legendary items

Hex Kot 0 Mar 28, 2022
A Language Server Protocol (LSP) server for Jsonnet

Jsonnet Language Server Warning: This project is in active development and is likely very buggy. A Language Server Protocol (LSP) server for Jsonnet.

Jack Baldry 17 Apr 29, 2022
A Language Server Protocol (LSP) server for Jsonnet

Jsonnet Language Server A Language Server Protocol (LSP) server for Jsonnet. Features Jump to definition self-support.mp4 dollar-support.mp4 Error/War

Grafana Labs 61 Aug 3, 2022
Parser and generator of M3U8-playlists for Apple HLS. Library for Go language. :cinema:

M3U8 This is the most complete opensource library for parsing and generating of M3U8 playlists used in HTTP Live Streaming (Apple HLS) for internet vi

Alexander I.Grafov 949 Aug 12, 2022
Unit tests generator for Go programming language

GoUnit GoUnit is a commandline tool that generates tests stubs based on source function or method signature. There are plugins for Vim Emacs Atom Subl

Max Chechel 62 Jul 8, 2022
Port of the lemon parser generator to the Go programming language

From the golang-nuts mailing list (with few modifications): --== intro ==-- Hi. I just want to announce a simple port of the lemon parser generator

null 54 Feb 17, 2022
An LL(1) parser generator for the Go programming language.

What is it? I have implemented an LL(1) parser generator for the Go programming language. I did this to build parse trees for my HAML parser. You can

Curtis Schlak 14 Jan 18, 2022
accessor methods generator for Go programming language

accessory accessory is an accessor generator for Go programming language. What is accessory? Accessory is a tool that generates accessor methods from

masaushi 5 Jul 19, 2022
Rest Api Generator for Golang Programming Language

Rest Api Generator for Golang Programming Language

Eko Kurniadi 3 Nov 29, 2021