Sometimes we want to search Go code that isn't in a package. Perhaps follow the lead of the Go tool and treat an argument ending in ".go" as a Go file not a package, and search that directly.

Since $x can match any node, it can be limiting when matching expressions. For example, we might want to only match strings.

A few syntaxes come to mind, like $x!string or $x: string.

Here's a thought - what about untyped constants? For example should, 0 match both int and uint, or neither?

It would be useful to search for $x of type that have a type that satisfies some predicate.

For example, I would like to grep for slices with elements of type $T. More precisely, I wanted to find all slicing expressions of form s[:] where s is a string or slice.

$ gogrep '$(s type(string))[:]' std
strings/replace.go:450:22: s[:]

This works, but I need to manually substitute string with all other types I want to check.

With type predicates, it would be possible to describe function that accepts node that is being matched with some kind of context object (that includes types info) and returns bool.

I have no idea about how to enable user-defined predicates. (Go plugins doesn't seem first class citizen.)

I was using gogrep to easily spot ways to simplify the cmd/compile/internal/gc code, which was once C and was transpiled. As such, it has tons of verbosity that is unnecessary.

I was focusing on the following two:

i := 0
for _, v := range slice {
    use(i, v)
    i++
}

for _, v := range slice {
    elem := v
    use(elem)
}

So, for example, for the first I used for _, $_ := range $_ { $*_; $_++ }. It had no false negatives that I can see, but it had one obvious false positive:

for _, v := range slice {
    use(v)
    someMap[someKey]++
}

You can imagine plenty of others. The root of the problem here is that $_ in our $_++ is not limited to just idents, as it can be any expression.

One could say that I should have grepped for $i := 0; for _, $_ := range $_ { $*_; $i++ }. But I think that is a worse solution. What if $i was actually declared as a parameter, or was of the form $i = 0 as it was being reused? Or what if it had statements between the decl and the for?

I think this could be orthogonal to #3. That issue is about constraining to go/types types, i.e. what the code ends up representing. What I am suggesting here is constraining to go/ast, i.e. what type of node it is. The syntax for the two should be different, as x could be both an *ast.Ident in the AST and be of *ast.Ident type in the resulting program.

This could also have other uses. For example, one could easily say "find all for statements whose body starts with an if statement" by constraining the AST type of the first body statement to IfStmt. Could still be possible with agressive matching, but this is more direct.

/cc @rogpeppe

**Reproducer: **

1. cd $GOROOT/src
2. gogrep -x 'func $x()' ./...

Result:

gogrep -x 'func $x()' ./...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x65a6f1]

goroutine 1 [running]:
main.(*matcher).node(0xc00008add0, 0x750fc0, 0x0, 0x750fc0, 0xc00025e4e0, 0x1)
	$GOPATH/src/mvdan.cc/gogrep/match.go:513 +0x2101
main.(*matcher).node(0xc00008add0, 0x751440, 0xc000084e40, 0x751440, 0xc00025e510, 0x0)
	$GOPATH/src/mvdan.cc/gogrep/match.go:440 +0x32d6
main.(*matcher).topNode(0xc00008add0, 0x751440, 0xc000084e40, 0x751440, 0xc00025e510, 0x0, 0x0)
	$GOPATH/src/mvdan.cc/gogrep/match.go:264 +0xad
main.(*matcher).cmdRange.func1(0x751440, 0xc000084e40, 0x751440, 0xc00025e510)
	$GOPATH/src/mvdan.cc/gogrep/match.go:106 +0xce
main.(*matcher).walkWithLists.func1(0x751440, 0xc00025e510, 0x751601)
	$GOPATH/src/mvdan.cc/gogrep/match.go:242 +0x7c
go/ast.inspector.Visit(0xc01991a120, 0x751440, 0xc00025e510, 0x7506e0, 0xc01991a120)
	$GOROOT/src/go/ast/walk.go:373 +0x3a
go/ast.Walk(0x7506e0, 0xc01991a120, 0x751440, 0xc00025e510)
	$GOROOT/src/go/ast/walk.go:52 +0x66
go/ast.walkDeclList(0x7506e0, 0xc01991a120, 0xc00025cc80, 0x4, 0x4)
	$GOROOT/src/go/ast/walk.go:38 +0x9e
go/ast.Walk(0x7506e0, 0xc01991a120, 0x7513c0, 0xc000283a00)
	$GOROOT/src/go/ast/walk.go:353 +0x2656
go/ast.Inspect(0x7513c0, 0xc000283a00, 0xc01991a120)
	$GOROOT/src/go/ast/walk.go:385 +0x4b
main.inspect(0x7513c0, 0xc000283a00, 0xc01991a120)
	$GOPATH/src/mvdan.cc/gogrep/main.go:279 +0x162
main.(*matcher).walkWithLists(0xc00008add0, 0x751440, 0xc000084e40, 0x7513c0, 0xc000283a00, 0xc0193a56b0)
	$GOPATH/src/mvdan.cc/gogrep/match.go:254 +0x93
main.(*matcher).cmdRange(0xc00008add0, 0x6ffc5e, 0x1, 0x7ffe8b993097, 0x9, 0x6b6760, 0xc000084e40, 0xc014b23900, 0xd, 0xd, ...)
	$GOPATH/src/mvdan.cc/gogrep/match.go:121 +0x194
main.(*matcher).cmdRange-fm(0x6ffc5e, 0x1, 0x7ffe8b993097, 0x9, 0x6b6760, 0xc000084e40, 0xc014b23900, 0xd, 0xd, 0x6e8be0, ...)
	$GOPATH/src/mvdan.cc/gogrep/match.go:70 +0x79
main.(*matcher).submatches(0xc00008add0, 0xc000084cf0, 0x1, 0x1, 0xc014b23900, 0xd, 0xd, 0xc01867bdc8, 0xc0193a53b0, 0x1c)
	$GOPATH/src/mvdan.cc/gogrep/match.go:89 +0x136
main.(*matcher).matches(0xc00008add0, 0xc000084cf0, 0x1, 0x1, 0xc0000f6400, 0xd, 0x10, 0xc00000e000, 0xc00bbce000, 0x11e)
	$GOPATH/src/mvdan.cc/gogrep/match.go:24 +0x1df
main.(*matcher).fromArgs(0xc00008add0, 0xc00000e0d0, 0x3, 0x3, 0xc000055f88, 0x405539)
	$GOPATH/src/mvdan.cc/gogrep/main.go:164 +0x3ca
main.main()
	$GOPATH/src/mvdan.cc/gogrep/main.go:63 +0xe1

If gogrep doesn't support such patterns, there should be something like pattern-compile error. Descriptive error message would help.

Case in point:

$ gogrep 'for $_, $_ := range $_ { $*_; $_++ }'
bexport.go:255:2: for _, n := range exportlist { sym := n.Sym; if sym.Exported() { continue; }; sym.SetExported(true); if strings.Contains(sym.Name, ".") { Fatalf("exporter: unexpected symbol: %v", sym); }; if sym.Def == nil { Fatalf("exporter: unknown export symbol: %v", sym); }; if p.trace { p.tracef("\n"); }; p.obj(sym); objcount++; }
esc.go:859:3: for _, lrn := range Curfn.Func.Dcl { if i >= retList.Len() { break; }; if lrn.Op != ONAME || lrn.Class() != PPARAMOUT { continue; }; e.escassignWhyWhere(lrn, retList.Index(i), "return", n); i++; }
fmt.go:1541:3: for _, n1 := range n.List.Slice() { if i != 0 { fmt.Fprint(s, " + "); }; n1.exprfmt(s, nprec, mode); i++; }
plive.go:981:2: for i, live := range lv.livevars { h := hashbitmap(H0, live) % uint32(tablesize); for { j := table[h]; if j < 0 { break; }; jlive := lv.livevars[j]; if live.Eq(jlive) { remap[i] = j; continue Outer; }; h++; if h == uint32(tablesize) { h = 0; }; }; table[h] = uniq; remap[i] = uniq; lv.livevars[uniq] = live; uniq++; }
reflect.go:1337:3: for _, t1 := range t.Fields().Slice() { dtypesym(t1.Type); n++; }
sinit.go:1244:3: for _, a := range n.List.Slice() { if a.Op == OKEY { k = nonnegintconst(a.Left); a = a.Right; }; addvalue(p, k*n.Type.Elem().Width, a); k++; }
subr.go:264:2: for _, s := range opkg.Syms { if s.Def == nil { continue; }; if !exportname(s.Name) || strings.ContainsRune(s.Name, 0xb7) { continue; }; s1 = lookup(s.Name); if s1.Def != nil { pkgerror = fmt.Sprintf("during import %q", opkg.Path); redeclare(s1, pkgerror); continue; }; s1.Def = s.Def; s1.Block = s.Block; if asNode(s1.Def).Name == nil { Dump("s1def", asNode(s1.Def)); Fatalf("missing Name"); }; asNode(s1.Def).Name.Pack = pack; s1.Origpkg = opkg; n++; }
typecheck.go:2650:2: for _, tl := range tstruct.Fields().Slice() { t = tl.Type; if tl.Isddd() { if isddd { if i >= nl.Len() { goto notenough; }; if nl.Len()-i > 1 { goto toomany; }; n = nl.Index(i); setlineno(n); if n.Type != nil { nl.SetIndex(i, assignconvfn(n, t, desc)); }; goto out; }; for ; i < nl.Len(); i++ { n = nl.Index(i); setlineno(n); if n.Type != nil { nl.SetIndex(i, assignconvfn(n, t.Elem(), desc)); }; }; goto out; }; if i >= nl.Len() { goto notenough; }; n = nl.Index(i); setlineno(n); if n.Type != nil { nl.SetIndex(i, assignconvfn(n, t, desc)); }; i++; }
typecheck.go:3487:3: for _, r := range s { l = append(l, nod(OKEY, nodintconst(int64(i)), nodintconst(int64(r)))); i++; }

I get reeeeally long lines and it's hard to see what I was after. Would be neat if, for example, double underscores meant any node and discard, like:

$ gogrep 'for $_, $_ := range $_ { $*__; $_++ }'
bexport.go:255:2: for _, n := range exportlist { [...]; objcount++; }

The problem I think is important, but the solution I just came up with rather quickly without much thought. Perhaps there is a cleaner or better way in the long run to achieve this.

/cc @rogpeppe

Reasons to:

• Useful. I just needed it right now.
• Using $*_; b; c; $*_ is not as useful, as it also prints the stuff on the sides.

Reasons not to:

• If we really want a statement list with just b; c, there wouldn't be another way to express that. { b; c } does not cover it, as it wouldn't match statement lists outside block statements such as switch case bodies.

Previously following match tests will get inconsistent result:

• {[]string{"-x", "struct{a int}{a: $_}"}, "struct{a int}{a: 1}", 1}: OK
• {[]string{"-x", "struct{a int}{a: $*_}"}, "struct{a int}{a: 1}", 1}: NOK

This is a quick fix while not sure if this makes sense. The cause why original code doesn't work for the 2nd test case is because when comparing expr (*ast.Ident{Name: "gogrep_0"}) and node (*ast.BasicLit{Value: "1"}) in matcher.node(), it will return false simply because the wildcard is any. I'm not sure the design behind this, so...

I was trying to find places where I had rolled my own rotate function instead of using math/bits.

I tried this query:

gogrep -x  '(a << k) | (a >> (32 - k))' .

But it failed to match this code:

func rotl32(k uint32, rot uint32) uint32 {
        return (k << rot) | (k >> (32 - rot))
}

Note that the equivalent gofmt rewrite query does work:

gofmt -d -r '(a << k) | (a >> (32 - k)) -> bits.RotateLeft32(a, k)' .

The gogrep query does work if I use the exact same variable names though:

gogrep -x '(k << rot) | (k >> (32 - rot))' .

Continuation from slack thread, here is the panic:

panic: resolveType TODO: *ast.BinaryExpr

goroutine 1 [running]:
main.(*matcher).resolveType(0xc0000cab60, 0xc0059c0d70, 0x139f940, 0xc000075bc0, 0xc004f99ef0, 0xc0000cab01)
        /Users/dmatrenichev/go/src/mvdan.cc/gogrep/match.go:634 +0x555
main.(*matcher).attrApplies(0xc0000cab60, 0x139e680, 0xc006230580, 0x1302c60, 0xc00000acc0, 0x100813b)
        /Users/dmatrenichev/go/src/mvdan.cc/gogrep/match.go:194 +0x6e5
main.(*matcher).cmdAttr(0xc0000cab60, 0x1344535, 0x1, 0x7ffeefbff978, 0x3d, 0x1302c60, 0xc00000acc0, 0xc001c8ff40, 0x1, 0x1, ...)
        /Users/dmatrenichev/go/src/mvdan.cc/gogrep/match.go:155 +0xff
main.(*matcher).cmdAttr-fm(0x1344535, 0x1, 0x7ffeefbff978, 0x3d, 0x1302c60, 0xc00000acc0, 0xc001c8ff40, 0x1, 0x1, 0xc006223080, ...)
        /Users/dmatrenichev/go/src/mvdan.cc/gogrep/match.go:78 +0x79
main.(*matcher).submatches(0xc0000cab60, 0xc0000a2120, 0x1, 0x2, 0xc001c8ff40, 0x1, 0x1, 0x1, 0x1, 0xc001c8ff40)
        /Users/dmatrenichev/go/src/mvdan.cc/gogrep/match.go:89 +0x136
main.(*matcher).submatches(0xc0000cab60, 0xc0000a20f0, 0x2, 0x3, 0xc005badee0, 0x1, 0x1, 0x1f, 0x1f, 0xc005badee0)
        /Users/dmatrenichev/go/src/mvdan.cc/gogrep/match.go:89 +0x1a3
main.(*matcher).submatches(0xc0000cab60, 0xc0000a20c0, 0x3, 0x4, 0xc003bbc000, 0x1f, 0x1f, 0x1636000, 0x0, 0x0)
        /Users/dmatrenichev/go/src/mvdan.cc/gogrep/match.go:89 +0x1a3
main.(*matcher).matches(0xc0000cab60, 0xc0000a20c0, 0x3, 0x4, 0xc0019e1600, 0x1f, 0x1f, 0x0, 0x0, 0x0)
        /Users/dmatrenichev/go/src/mvdan.cc/gogrep/match.go:24 +0x1d9
main.(*matcher).fromArgs(0xc0000cab60, 0x1344515, 0x1, 0xc00000e0f0, 0x1, 0x1, 0x12e8a60, 0xc000037dd8)
        /Users/dmatrenichev/go/src/mvdan.cc/gogrep/main.go:155 +0x20f
main.main()
        /Users/dmatrenichev/go/src/mvdan.cc/gogrep/main.go:63 +0xf9

This could especially be useful when loading with types, as it can easily take seconds to load even a small program if we need type information.

We could have something like readline:

$ gogrep -repl
> -x expr
[...]
> -x expr -g expr_with_types
[...]
> ^D
$

This project was fun and interesting, but I haven't put a significant amount of work into it for over a year.

The project aimed to solve two main use cases, which I'll cover below. It's also worth noting that a significant part of the initial design came from @rogpeppe.

Querying Go code

One good example is writing custom linting checks. One can always use go/ast and go/types, but having to do that for every little query or check is overkill.

We have https://pkg.go.dev/golang.org/x/tools/go/analysis today, so the barrier to entry has been significantly lowered.

There have also been other projects started in this space since, such as https://semgrep.dev/ (which supports many languages) and https://github.com/quasilyte/go-ruleguard (which actually forked our pattern-matching syntax and code, I think).

Refactoring Go code

This was done by gofmt -r or eg, but they were both too simple to accomplish anything non-trivial.

I firmly believe that refactoring Go code is better served by https://github.com/rsc/rf in the future. That tool is still alpha, but its design is clearly better in multiple ways. It's already been used to refactor Go's own compiler in many ways that gogrep simply does not support, for example.

Current design problems

The API is a list of commands, which is too limiting if one wants to write complex queries. See https://github.com/mvdan/gogrep/issues/32. I tried to do a refactor for this in early 2020 (https://github.com/mvdan/gogrep/commit/e933f34b68d0b10e7f5b1fb1fa14491b688ab9ff), but I didn't particularly like the result either.

The pattern matching is sometimes too stiff; see https://github.com/mvdan/gogrep/issues/4 and https://github.com/mvdan/gogrep/issues/52.

The matching could be more clever, such as comparing evaluated constant values: https://github.com/mvdan/gogrep/issues/43

Moving forward

I don't plan to invest the significant amount of time it would take to make gogrep a competitive tool again, given that it's been outpaced in both use cases as outlined above. Frankly, I'm pretty okay with go/analysis and rf as Go-specific ways to solve both problems in a satisfactory way, and they are both well maintained and better designed.

So I think that gogrep, as a tool in its current form, should be deprecated. There's little reason to recommend its use to others. It would also help reflect the current state of the project.

I still think that there might be value in continuing to develop the internal matching syntax and logic. See https://github.com/mvdan/gogrep/issues/55, for example. I would have absolutely no problem with that, but I don't plan to do that myself.

If someone is willing to step up and take ownership of this project, including the end-user tool, I'm happy to give that person write access - and eventually transfer the repository, once it's shown a bit of progress.

Assuming noone steps up, I think the best course of action is to deprecate the tool and freeze this GitHub repository. Current users of the tool could continue to use it, though the deprecation should discourage them. This would also impact ruleguard, but I think they could simply continue to maintain the pattern matching code in their own repository.

--

Either way the project goes, it's going to be a relief for me - I have other cool stuff that I want to work on, and it's unfair for me to keep this project in its current state if I don't plan to continue it :)

I'll leave this issue open for two weeks before making a decision.

CC @rogpeppe @quasilyte, given your past contributions.

This brings README.md and the usage printed by the command into sync, and also adds information about available attributes.

I've reverse-engineered the descriptions from my idea of what the code might be doing, so they're quite likely wrong :)

How about having a -d (or -v) argument to print the AST type information of the matched node and the $x patterns? Currently, when a pattern match occurs, gogrep prints the value of the AST node: https://github.com/mvdan/gogrep/blob/master/main.go#L326

For example, given the following code:

func main() {
  v := mypkg.MyStruct{}

The command gogrep -x '$x.$y{$*_}' will print something like this:

main.go:24:10: mypkg.MyStruct{}

This is great, but when exploring various patterns, I find it would be very useful to know the exact ast nodes of the entire pattern and each of the $X sub matches.

Suppose you're about to do some refactoring inside some specific file. You want to do a -x pattern with -s replacement to get the job done. The problem is that we can't reliably run a gogrep over a single file since it looks like it does a typechecking even if no type-related filters are involved (I may be wrong here).

We can get a cheap fix of this problem by adding a way of telling "I'm giving you a package as a target, but I'm only interested in one specific file". It could either be a bool param that makes gogrep infer the file package from the specified filename, load the package and then print results only for that file. A simpler approach is to make that param a string that tells which filename to include into the output, all other matches should be discarded.

If we generalize, it looks like I need an additional file filter, which may not necessarily be limited to 1 file, but I can't come up with a good use case for that at the moment.

In writing linters (in my case, go/analysis passes to be run as golangci-lint plugins) I find myself wanting some sort of AST-matching engine. This tool seems like a really great one! But I want to use it inside my own tool, rather than as a simple global search, like this package or go-ruleguard is designed to do. To do that, I need to use the gogrep matcher, but then refer back to the AST or types in an arbitrary way.

A sample API that I think would be sufficient, and which looks to my quick glance similar to what you're already using internally:

type Matcher

// Compile compiles a gogrep pattern into a Matcher object.
func Compile(pattern string) (*Matcher, error)

// FindAll returns all matches to the given pattern within the given AST node.
func (m *Matcher) FindAll(node ast.Node) []Match

// A single match of a pattern.
type Match struct {
  // The top-level node that matched the pattern.
  Node ast.Node
  // The nodes that matched each variable; for example if $x + $_ matched 2 + 3,
  // captures would be {"$x": <node for 2>}
  Captures map[string]ast.Node
}

// optionally other regexp-style APIs like MustCompile, Matcher.Find, Matcher.Match.

Related to #32, but it seems like that won't actually do what we would want, for the same reason go-ruleguard isn't quite enough for us: we would still have no direct access to the underlying AST/types.

Prior to commit 58f4747 (port to go/packages), it was possible to run gogrep on a file that doesn't pass type checking. But that no longer works. It would be nice to have that functionality back.