Decorated Syntax Tree - manipulate Go source with perfect fidelity.

Related tags

Compiler dst
Overview

Build Status Documentation codecov stability-stable Sourcegraph

Decorated Syntax Tree

The dst package enables manipulation of a Go syntax tree with high fidelity. Decorations (e.g. comments and line spacing) remain attached to the correct nodes as the tree is modified.

Where does go/ast break?

The go/ast package wasn't created with source manipulation as an intended use-case. Comments are stored by their byte offset instead of attached to nodes, so re-arranging nodes breaks the output. See this Go issue for more information.

Consider this example where we want to reverse the order of the two statements. As you can see the comments don't remain attached to the correct nodes:

code := `package a

func main(){
	var a int    // foo
	var b string // bar
}
`
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "", code, parser.ParseComments)
if err != nil {
	panic(err)
}

list := f.Decls[0].(*ast.FuncDecl).Body.List
list[0], list[1] = list[1], list[0]

if err := format.Node(os.Stdout, fset, f); err != nil {
	panic(err)
}

//Output:
//package a
//
//func main() {
//	// foo
//	var b string
//	var a int
//	// bar
//}

Here's the same example using dst:

code := `package a

func main(){
	var a int    // foo
	var b string // bar
}
`
f, err := decorator.Parse(code)
if err != nil {
	panic(err)
}

list := f.Decls[0].(*dst.FuncDecl).Body.List
list[0], list[1] = list[1], list[0]

if err := decorator.Print(f); err != nil {
	panic(err)
}

//Output:
//package a
//
//func main() {
//	var b string // bar
//	var a int    // foo
//}

Usage

Parsing a source file to dst and printing the results after modification can be accomplished with several Parse and Print convenience functions in the decorator package.

For more fine-grained control you can use Decorator to convert from ast to dst, and Restorer to convert back again.

Comments

Comments are added at decoration attachment points. See here for a full list of these points, along with demonstration code of where they are rendered in the output.

The decoration attachment points have convenience functions Append, Prepend, Replace, Clear and All to accomplish common tasks. Use the full text of your comment including the // or /**/ markers. When adding a line comment, a newline is automatically rendered.

code := `package main

func main() {
	println("Hello World!")
}`
f, err := decorator.Parse(code)
if err != nil {
	panic(err)
}

call := f.Decls[0].(*dst.FuncDecl).Body.List[0].(*dst.ExprStmt).X.(*dst.CallExpr)

call.Decs.Start.Append("// you can add comments at the start...")
call.Decs.Fun.Append("/* ...in the middle... */")
call.Decs.End.Append("// or at the end.")

if err := decorator.Print(f); err != nil {
	panic(err)
}

//Output:
//package main
//
//func main() {
//	// you can add comments at the start...
//	println /* ...in the middle... */ ("Hello World!") // or at the end.
//}

Spacing

The Before property marks the node as having a line space (new line or empty line) before the node. These spaces are rendered before any decorations attached to the Start decoration point. The After property is similar but rendered after the node (and after any End decorations).

code := `package main

func main() {
	println(a, b, c)
}`
f, err := decorator.Parse(code)
if err != nil {
	panic(err)
}

call := f.Decls[0].(*dst.FuncDecl).Body.List[0].(*dst.ExprStmt).X.(*dst.CallExpr)

call.Decs.Before = dst.EmptyLine
call.Decs.After = dst.EmptyLine

for _, v := range call.Args {
	v := v.(*dst.Ident)
	v.Decs.Before = dst.NewLine
	v.Decs.After = dst.NewLine
}

if err := decorator.Print(f); err != nil {
	panic(err)
}

//Output:
//package main
//
//func main() {
//
//	println(
//		a,
//		b,
//		c,
//	)
//
//}

Decorations

The common decoration properties (Start, End, Before and After) occur on all nodes, and can be accessed with the Decorations() method on the Node interface:

code := `package main

func main() {
	var i int
	i++
	println(i)
}`
f, err := decorator.Parse(code)
if err != nil {
	panic(err)
}

list := f.Decls[0].(*dst.FuncDecl).Body.List

list[0].Decorations().Before = dst.EmptyLine
list[0].Decorations().End.Append("// the Decorations method allows access to the common")
list[1].Decorations().End.Append("// decoration properties (Before, Start, End and After)")
list[2].Decorations().End.Append("// for all nodes.")
list[2].Decorations().After = dst.EmptyLine

if err := decorator.Print(f); err != nil {
	panic(err)
}

//Output:
//package main
//
//func main() {
//
//	var i int  // the Decorations method allows access to the common
//	i++        // decoration properties (Before, Start, End and After)
//	println(i) // for all nodes.
//
//}

dstutil.Decorations

While debugging, it is often useful to have a list of all decorations attached to a node. The dstutil package provides a helper function Decorations which returns a list of the attachment points and all decorations for any node:

code := `package main

// main comment
// is multi line
func main() {

	if true {

		// foo
		println( /* foo inline */ "foo")
	} else if false {
		println /* bar inline */ ("bar")

		// bar after

	} else {
		// empty block
	}
}`

f, err := decorator.Parse(code)
if err != nil {
	panic(err)
}

dst.Inspect(f, func(node dst.Node) bool {
	if node == nil {
		return false
	}
	before, after, points := dstutil.Decorations(node)
	var info string
	if before != dst.None {
		info += fmt.Sprintf("- Before: %s\n", before)
	}
	for _, point := range points {
		if len(point.Decs) == 0 {
			continue
		}
		info += fmt.Sprintf("- %s: [", point.Name)
		for i, dec := range point.Decs {
			if i > 0 {
				info += ", "
			}
			info += fmt.Sprintf("%q", dec)
		}
		info += "]\n"
	}
	if after != dst.None {
		info += fmt.Sprintf("- After: %s\n", after)
	}
	if info != "" {
		fmt.Printf("%T\n%s\n", node, info)
	}
	return true
})

//Output:
//*dst.FuncDecl
//- Before: NewLine
//- Start: ["// main comment", "// is multi line"]
//
//*dst.IfStmt
//- Before: NewLine
//- After: NewLine
//
//*dst.ExprStmt
//- Before: NewLine
//- Start: ["// foo"]
//- After: NewLine
//
//*dst.CallExpr
//- Lparen: ["/* foo inline */"]
//
//*dst.ExprStmt
//- Before: NewLine
//- End: ["\n", "\n", "// bar after"]
//- After: NewLine
//
//*dst.CallExpr
//- Fun: ["/* bar inline */"]
//
//*dst.BlockStmt
//- Lbrace: ["\n", "// empty block"]

Newlines

The Before and After properties cover the majority of cases, but occasionally a newline needs to be rendered inside a node. Simply add a \n decoration to accomplish this.

Clone

Re-using an existing node elsewhere in the tree will panic when the tree is restored to ast. Instead, use the Clone function to make a deep copy of the node before re-use:

code := `package main

var i /* a */ int`

f, err := decorator.Parse(code)
if err != nil {
	panic(err)
}

cloned := dst.Clone(f.Decls[0]).(*dst.GenDecl)

cloned.Decs.Before = dst.NewLine
cloned.Specs[0].(*dst.ValueSpec).Names[0].Name = "j"
cloned.Specs[0].(*dst.ValueSpec).Names[0].Decs.End.Replace("/* b */")

f.Decls = append(f.Decls, cloned)

if err := decorator.Print(f); err != nil {
	panic(err)
}

//Output:
//package main
//
//var i /* a */ int
//var j /* b */ int

Apply

The dstutil package is a fork of golang.org/x/tools/go/ast/astutil, and provides the Apply function with similar semantics.

Imports

The decorator can automatically manage the import block, which is a non-trivial task.

Use NewDecoratorWithImports and NewRestorerWithImports to create an import aware decorator / restorer.

During decoration, remote identifiers are normalised - *ast.SelectorExpr nodes that represent qualified identifiers are replaced with *dst.Ident nodes with the Path field set to the path of the imported package.

When adding a qualified identifier node, there is no need to use *dst.SelectorExpr - just add a *dst.Ident and set Path to the imported package path. The restorer will wrap it in a *ast.SelectorExpr where appropriate when converting back to ast, and also update the import block.

To enable import management, the decorator must be able to resolve the imported package for selector expressions and identifiers, and the restorer must be able to resolve the name of a package given it's path. Several implementations for these resolvers are provided, and the best method will depend on the environment. See below for more details.

Load

The Load convenience function uses go/packages to load packages and decorate all loaded ast files, with import management enabled:

// Create a simple module in a temporary directory
dir, err := tempDir(map[string]string{
	"go.mod":	"module root",
	"main.go":	"package main \n\n func main() {}",
})
defer os.RemoveAll(dir)
if err != nil {
	panic(err)
}

// Use the Load convenience function that calls go/packages to load the package. All loaded
// ast files are decorated to dst.
pkgs, err := decorator.Load(&packages.Config{Dir: dir, Mode: packages.LoadSyntax}, "root")
if err != nil {
	panic(err)
}
p := pkgs[0]
f := p.Syntax[0]

// Add a call expression. Note we don't have to use a SelectorExpr - just adding an Ident with
// the imported package path will do. The restorer will add SelectorExpr where appropriate when
// converting back to ast. Note the new Path field on *dst.Ident. Set this to the package path
// of the imported package, and the restorer will automatically add the import to the import
// block.
b := f.Decls[0].(*dst.FuncDecl).Body
b.List = append(b.List, &dst.ExprStmt{
	X: &dst.CallExpr{
		Fun:	&dst.Ident{Path: "fmt", Name: "Println"},
		Args: []dst.Expr{
			&dst.BasicLit{Kind: token.STRING, Value: strconv.Quote("Hello, World!")},
		},
	},
})

// Create a restorer with the import manager enabled, and print the result. As you can see, the
// import block is automatically managed, and the Println ident is converted to a SelectorExpr:
r := decorator.NewRestorerWithImports("root", gopackages.New(dir))
if err := r.Print(p.Syntax[0]); err != nil {
	panic(err)
}

//Output:
//package main
//
//import "fmt"
//
//func main() { fmt.Println("Hello, World!") }

Mappings

The decorator exposes Dst.Nodes and Ast.Nodes which map between ast.Node and dst.Node. This enables systems that refer to ast nodes (such as go/types) to be used:

code := `package main

func main() {
	var i int
	i++
	println(i)
}`

// Parse the code to AST
fset := token.NewFileSet()
astFile, err := parser.ParseFile(fset, "", code, parser.ParseComments)
if err != nil {
	panic(err)
}

// Invoke the type checker using AST as input
typesInfo := types.Info{
	Defs:	make(map[*ast.Ident]types.Object),
	Uses:	make(map[*ast.Ident]types.Object),
}
conf := &types.Config{}
if _, err := conf.Check("", fset, []*ast.File{astFile}, &typesInfo); err != nil {
	panic(err)
}

// Create a new decorator, which will track the mapping between ast and dst nodes
dec := decorator.NewDecorator(fset)

// Decorate the *ast.File to give us a *dst.File
f, err := dec.DecorateFile(astFile)
if err != nil {
	panic(err)
}

// Find the *dst.Ident for the definition of "i"
dstDef := f.Decls[0].(*dst.FuncDecl).Body.List[0].(*dst.DeclStmt).Decl.(*dst.GenDecl).Specs[0].(*dst.ValueSpec).Names[0]

// Find the *ast.Ident using the Ast.Nodes mapping
astDef := dec.Ast.Nodes[dstDef].(*ast.Ident)

// Find the types.Object corresponding to "i"
obj := typesInfo.Defs[astDef]

// Find all the uses of that object
var astUses []*ast.Ident
for id, ob := range typesInfo.Uses {
	if ob != obj {
		continue
	}
	astUses = append(astUses, id)
}

// Find each *dst.Ident in the Dst.Nodes mapping
var dstUses []*dst.Ident
for _, id := range astUses {
	dstUses = append(dstUses, dec.Dst.Nodes[id].(*dst.Ident))
}

// Change the name of the original definition and all uses
dstDef.Name = "foo"
for _, id := range dstUses {
	id.Name = "foo"
}

// Print the DST
if err := decorator.Print(f); err != nil {
	panic(err)
}

//Output:
//package main
//
//func main() {
//	var foo int
//	foo++
//	println(foo)
//}

Resolvers

There are two separate interfaces defined by the resolver package which allow the decorator and restorer to automatically manage the imports block.

The decorator uses a DecoratorResolver which resolves the package path of any *ast.Ident. This is complicated by dot-import syntax (see below).

The restorer uses a RestorerResolver which resolves the name of any package given the path. This is complicated by vendoring and Go modules.

When Resolver is set on Decorator or Restorer, the Path property must be set to the local package path.

Several implementations of both interfaces that are suitable for different environments are provided:

DecoratorResolver

gotypes

The gotypes package provides a DecoratorResolver with full dot-import compatibility. However it requires full export data for all imported packages, so the Uses map from go/types.Info is required. There are several methods of generating go/types.Info. Using golang.org/x/tools/go/packages.Load is recommended for full Go modules compatibility. See the decorator.Load convenience function to automate this.

goast

The goast package provides a simplified DecoratorResolver that only needs to scan a single ast file. This is unable to resolve identifiers from dot-imported packages, so will panic if a dot-import is encountered in the import block. It uses the provided RestorerResolver to resolve the names of all imported packages. If no RestorerResolver is provided, the guess implementation is used.

RestorerResolver

gopackages

The gopackages package provides a RestorerResolver with full compatibility with Go modules. It uses golang.org/x/tools/go/packages to load the package data. This may be very slow, and uses the go command line tool to query package data, so may not be compatible with some environments.

gobuild

The gobuild package provides an alternative RestorerResolver that uses the legacy go/build system to load the imported package data. This may be needed in some circumstances and provides better performance than go/packages. However, this is not Go modules aware.

guess and simple

The guess and simple packages provide simple RestorerResolver implementations that may be useful in certain circumstances, or where performance is critical. simple resolves paths only if they occur in a provided map. guess guesses the package name based on the last part of the path.

Example

Here's an example of supplying resolvers for the decorator and restorer:

code := `package main

	import "fmt"

	func main() {
		fmt.Println("a")
	}`

dec := decorator.NewDecoratorWithImports(token.NewFileSet(), "main", goast.New())

f, err := dec.Parse(code)
if err != nil {
	panic(err)
}

f.Decls[1].(*dst.FuncDecl).Body.List[0].(*dst.ExprStmt).X.(*dst.CallExpr).Args = []dst.Expr{
	&dst.CallExpr{
		Fun: &dst.Ident{Name: "A", Path: "foo.bar/baz"},
	},
}

res := decorator.NewRestorerWithImports("main", guess.New())
if err := res.Print(f); err != nil {
	panic(err)
}

//Output:
//package main
//
//import (
//	"fmt"
//
//	"foo.bar/baz"
//)
//
//func main() {
//	fmt.Println(baz.A())
//}

Alias

To control the alias of imports, use a FileRestorer:

code := `package main

	import "fmt"

	func main() {
		fmt.Println("a")
	}`

dec := decorator.NewDecoratorWithImports(token.NewFileSet(), "main", goast.New())

f, err := dec.Parse(code)
if err != nil {
	panic(err)
}

res := decorator.NewRestorerWithImports("main", guess.New())

fr := res.FileRestorer()
fr.Alias["fmt"] = "fmt1"

if err := fr.Print(f); err != nil {
	panic(err)
}

//Output:
//package main
//
//import fmt1 "fmt"
//
//func main() {
//	fmt1.Println("a")
//}

Details

For more information on exactly how the imports block is managed, read through the test cases.

Dot-imports

Consider this file...

package main

import (
	. "a"
)

func main() {
	B()
	C()
}

B and C could be local identifiers from a different file in this package, or from the imported package a. If only one is from a and it is removed, we should remove the import when we restore to ast. Thus the resolver needs to be able to resolve the package using the full info from go/types.

Status

This package is well tested and used in many projects. The API should be considered stable going forward.

Chat?

Feel free to create an issue or chat in the #dst Gophers Slack channel.

Comments
  • Add support for Generics (requires Go 1.18+)

    Add support for Generics (requires Go 1.18+)

    This patch represents a first attempt at adding support for generics.

    Thank you @dave for the amazing work on this tool! The code is really fun to work with! I am sure that there are many other things that will need to be done to get this going, but I made pretty good progress today and wanted to let you take a look!

    Thanks for letting me help! Will

    Towards #63

    opened by hawkinsw 29
  • Comments in empty function/loop/if-statement bodies not found

    Comments in empty function/loop/if-statement bodies not found

    Comments that are placed inside empty function, loop or if statement bodies are not found in the respective node's decorations. I would expect them to be placed somewhere in the e.g. dst.FuncDeclDecorations of the function or similarly the dst.ForStmtDecorations of the loop declaration. This snipped demonstrates the issue:

    package main
    
    func main() {
    }
    
    func empty() {
    	// inside empty function, not found
    }
    
    func foo() {
    	i := 1
    	for i < 2 {
    		// inside empty for statement, not found
    	}
    	// after empty for statement, found and associated with for statement
    
    	if i == 3 {
    		// inside empty if statement, not found
    	}
    	i = 4
    }
    

    And here you can find code to reproduce the issue: https://play.golang.org/p/Yg9OOkoP6IF [Disclaimer: sometimes the playground needs multiple tries to import all packages, so if you encounter a timeout running go build error, just execute the code again.]

    I tried using dst in order to fix the issue https://github.com/golang/go/issues/39753 I had with go/ast. As I am a fairly new user of this module, I might also have overlooked something - Please let me know if this is the case and how I could get the correct comment associations :) Thanks!

    $ go version
    go version go1.14.2 darwin/amd64
    
    go env Output
    $ go env
    GO111MODULE=""
    GOARCH="amd64"
    GOBIN=""
    GOCACHE="/Users/.../Library/Caches/go-build"
    GOENV="/Users/.../Library/Application Support/go/env"
    GOEXE=""
    GOFLAGS=""
    GOHOSTARCH="amd64"
    GOHOSTOS="darwin"
    GOINSECURE=""
    GONOPROXY=""
    GONOSUMDB=""
    GOOS="darwin"
    GOPATH="/Users/.../go"
    GOPRIVATE=""
    GOPROXY="https://proxy.golang.org,direct"
    GOROOT="/usr/local/go"
    GOSUMDB="sum.golang.org"
    GOTMPDIR=""
    GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
    GCCGO="gccgo"
    AR="ar"
    CC="clang"
    CXX="clang++"
    CGO_ENABLED="1"
    GOMOD="/Users/.../go.mod"
    CGO_CFLAGS="-g -O2"
    CGO_CPPFLAGS=""
    CGO_CXXFLAGS="-g -O2"
    CGO_FFLAGS="-g -O2"
    CGO_LDFLAGS="-g -O2"
    PKG_CONFIG="pkg-config"
    GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/yb/hqncynqs0b5_3hyxcjpdsyr00000gr/T/go-build447564702=/tmp/go-build -gno-record-gcc-switches -fno-common"
    
    opened by ec-m 12
  • Dst doesn't support generics

    Dst doesn't support generics

    This is not a small change, and I will be hiking in Patagonia from December 2021 until April or May 2022 without my laptop, so I won't be able to get this done until I get back.

    opened by dave 8
  • Ident Paths always empty string of loaded packages

    Ident Paths always empty string of loaded packages

    I'm trying to load a package and walk the resulting tree. I'm noticing that all of the dst.Ident Path values are set to an empty string if the type is in the package being loaded but the Path values are set correctly if the type refers to another package. I pasted a simple program below that loads the net/http package and finds the CookieJar interface for which it prints an identity of just CookieJar (I expected net/http.CookieJar). It also prints the identity of the type of the first parameter to the SetCookies method as net/url.URL.

    Is there some way to have the path info filled in?

    package main
    
    import (
    	"github.com/dave/dst"
    	"github.com/dave/dst/decorator"
    	"golang.org/x/tools/go/packages"
    )
    
    func main() {
    	pkgs, err := decorator.Load(&packages.Config{
    		Mode: packages.NeedName |
    			packages.NeedFiles |
    			packages.NeedCompiledGoFiles |
    			packages.NeedImports |
    			packages.NeedTypes |
    			packages.NeedTypesSizes |
    			packages.NeedSyntax |
    			packages.NeedTypesInfo,
    	}, "net/http")
    	if err != nil {
    		panic(err)
    	}
    
    	for _, pkg := range pkgs {
    		for _, file := range pkg.Syntax {
    			for _, decl := range file.Decls {
    				if genDecl, ok := decl.(*dst.GenDecl); ok {
    					for _, spec := range genDecl.Specs {
    						if typeSpec, ok := spec.(*dst.TypeSpec); ok {
    							if typeSpec.Name.Name == "CookieJar" {
    								println("CookieJar ident:", typeSpec.Name.String())
    								for _, method := range typeSpec.Type.(*dst.InterfaceType).Methods.List {
    									if method.Names[0].Name == "SetCookies" {
    										param1Type := method.Type.(*dst.FuncType).Params.List[0].Type.(*dst.StarExpr)
    										println("  Param1 StarExpr X ident:", param1Type.X.(*dst.Ident).String())
    									}
    								}
    							}
    						}
    					}
    				}
    			}
    		}
    	}
    }
    

    Output:

    CookieJar ident: CookieJar
      Param1 StarExpr X ident: net/url.URL
    
    opened by myshkin5 8
  • question: support/implement positions for dst nodes and decorations?

    question: support/implement positions for dst nodes and decorations?

    could you please help me to figure out one thing? in the docs here https://github.com/dave/dst/blob/e3c208037a64131003377785f5ccddc79a887a29/dst.go#L24 it is said that

    // All nodes contain position information marking the beginning of
    // the corresponding source text segment; it is accessible via the
    // Pos accessor method. Nodes may contain additional position info
    // for language constructs where comments may be found between parts
    // of the construct (typically any larger, parenthesized subpart).
    // That position information is needed to properly position comments
    // when printing the construct.
    

    but at the same time the interface and it's implementations do not contain this method at all. I have a doubt if dst nodes contain positions at all

    If there're no positions provided, what approach would you suggest in implementing them?

    opened by lwsanty 8
  • Duplicate node restoring new function

    Duplicate node restoring new function

    I'm trying to generate code similar to this:

    type something struct {
    	here int
    }
    
    func newSomething() *something {
    	return &something{
    		here: 42,
    	}
    }
    

    I've got the struct in place and a start on the func but the body of the func is giving me an error:

    panic: duplicate node: &dst.Ident{Name:"something", Obj:(*dst.Object)(nil), Path:"", Decs:dst.IdentDecorations{NodeDecs:dst.NodeDecs{Before:0, Start:dst.Decorations(nil), End:dst.Decorations(nil), After:0}, X:dst.Decorations(nil)}}
    
    goroutine 1 [running]:
    github.com/dave/dst/decorator.(*FileRestorer).restoreNode(0xc000626e10, 0x1457e60, 0xc000625a80, 0x13dd1ff, 0xc, 0x13db130, 0x4, 0x13db0c0, 0x4, 0x0, ...)
    	/Users/dschultz/workspace/go/pkg/mod/github.com/dave/[email protected]/decorator/restorer-generated.go:15 +0x14356
    github.com/dave/dst/decorator.(*FileRestorer).restoreNode(0xc000626e10, 0x1457c80, 0xc000559c20, 0x13dc48c, 0x9, 0x13dacc7, 0x1, 0x13db0c0, 0x4, 0x0, ...)
    	/Users/dschultz/workspace/go/pkg/mod/github.com/dave/[email protected]/decorator/restorer-generated.go:459 +0x7fe
    github.com/dave/dst/decorator.(*FileRestorer).restoreNode(0xc000626e10, 0x1458140, 0xc000619730, 0x13dc8fe, 0xa, 0x13dbaef, 0x7, 0x13db0c0, 0x4, 0x0, ...)
    	/Users/dschultz/workspace/go/pkg/mod/github.com/dave/[email protected]/decorator/restorer-generated.go:1839 +0x1b0d
    

    I dumped the DST of a go file similar to what I want to create:

      1280  .  .  .  Body: *dst.BlockStmt {
      1281  .  .  .  .  List: []dst.Stmt (len = 1) {
      1282  .  .  .  .  .  0: *dst.ReturnStmt {
      1283  .  .  .  .  .  .  Results: []dst.Expr (len = 1) {
      1284  .  .  .  .  .  .  .  0: *dst.UnaryExpr {
      1285  .  .  .  .  .  .  .  .  Op: &
      1286  .  .  .  .  .  .  .  .  X: *dst.CompositeLit {
      1287  .  .  .  .  .  .  .  .  .  Type: *dst.Ident {
      1288  .  .  .  .  .  .  .  .  .  .  Name: "something"
      1289  .  .  .  .  .  .  .  .  .  .  Obj: *(obj @ 50)
      1290  .  .  .  .  .  .  .  .  .  .  Path: ""
    

    My code is creating an identical structure but it looks like the restorer code thinks that something is duplicate when defined on line 1288 above. Also not sure what to do about the Op field. I'm setting it to token.AND but not sure if that's right.

    Any help is greatly appreciated!

    opened by myshkin5 7
  • Printing a single node

    Printing a single node

    This is problematic because only *ast.File stores comments. If we convert a dst.Expr to an ast.Expr, there's nowhere to put the comments so they will be lost. Workarounds? For Decl perhaps we could create an empty File with one Decl, but what for Stmt and Expr?

    opened by dave 7
  • Lost package qualifier on function call

    Lost package qualifier on function call

    Thanks for starting this package, the problem it solves is one I encountered a very long time ago!

    I just started off with a very simple test:

    package b
    
    func B() {
    	println("B")
    }
    
    package a
    
    import "github.com/pwaller/example/testdata/a/b"
    
    func A() {
    	b.B()
    }
    

    With the following program I get unexpected output:

    package main
    
    import (
    	"golang.org/x/tools/go/packages"
    
    	"github.com/dave/dst/decorator"
    )
    
    func main() {
    	pkgs, err := decorator.Load(&packages.Config{Mode: packages.LoadAllSyntax}, ".")
    	if err != nil {
    		panic(err)
    	}
    
    	for _, p := range pkgs {
    		for _, s := range p.Syntax {
    			decorator.Print(s)
    		}
    	}
    }
    

    Here is the output:

    package a
    
    import "github.com/pwaller/example/testdata/a/b"
    
    func A() {
    	B()
    }
    

    I expected that I should see a call to b.B(), not B(), so it seems the package qualifier is lost.

    This is using github.com/dave/dst v0.23.0.

    I'm a bit surprised by the bug, I would have thought this case should work - have I done something wrong? Thanks!

    opened by pwaller 6
  • Hanging indent comment

    Hanging indent comment

    This is very similar to https://github.com/dave/dst/issues/9 but a more general case.

    A statement that flows onto multiple lines will be indented. This means that comments that follow will be able to have two indent levels:

    package a
    
    const a = 1 +
    	1
    	// a1
    
    	// a2
    const b = 1
    
    const c = 1 +
    	1
    
    // d1
    
    // d2
    const d = 1
    
    opened by dave 5
  • Will the dst package be updated when generics land?

    Will the dst package be updated when generics land?

    Thanks for the very useful dst package, I have used it a few times for rewriting code with great success! :)

    Now that the Go language proposal https://github.com/golang/go/issues/43651 has been accepted, I was wondering what your thoughts are on updating the dst package when generics land?

    Do you plan to make the required changes so that dst can read, modify and write code that uses generics? (I’m assuming changes will be required.)

    Thanks,

    opened by stapelberg 3
  • Can't generate an *empty* struct

    Can't generate an *empty* struct

    I can't seem to figure out how to make an empty struct without a new line. I've tried the struct decorator and the field list decorator. It feels like I should be altering the field decorator but there are no fields. I'm fighting with a new linter that is removing the empty line from empty structs and I don't want generated code to be affected by the linter. Here's what I get:

    type MyEmptyStruct struct {
    }
    

    Here's what I want:

    type MyEmptyStruct struct {}
    

    Thanks in advance!

    opened by myshkin5 3
  • dst removes line break between /* … */ comments and package statement

    dst removes line break between /* … */ comments and package statement

    When taking the example program from the README, and adding a /* single line comment */ like so:

    package main
    
    import "github.com/dave/dst/decorator"
    
    func main() {
    	code := `/* single line comment */
    package a
    
    func main(){
    	var a int    // foo
    	var b string // bar
    }
    `
    	f, err := decorator.Parse(code)
    	if err != nil {
    		panic(err)
    	}
    	if err := decorator.Print(f); err != nil {
    		panic(err)
    	}
    }
    

    …the output is:

    /* single line comment */package a
    
    func main() {
    	var a int    // foo
    	var b string // bar
    }
    

    Note how package a is now at the end of the previous line.

    The same issue is reproducible with multi-line /* … */ comments.

    I tested this with github.com/dave/dst v0.27.0, which is current at the time of writing.

    opened by stapelberg 1
  • Empty lines inside structs/functions are sometimes removed

    Empty lines inside structs/functions are sometimes removed

    I have a source file that uses cgo a lot, and running it through dst causes (some) empty lines in structs/functions to be removed.

    I tried some testing, and I am not entirely sure what causes the empty lines to be removed, because e.g. this struct is left alone

    type StructTest struct {
    	aaa int32
    
    	b int32
    }
    

    but a similar struct

    type AType struct {
    	c      *C.StructNameCensored
    	f SomeComplexGoType
    
    	using int32
    }
    

    has the empty line removed, so it looks like this:

    type AType struct {
    	c     *C.StructNameCensored
    	f     SomeComplexGoType
    	using int32
    }
    

    I also have a similar issue inside (some) functions -- I guess this is related to the C APIs? However, the newline between the package line and the copyright notice is also removed:

    // Copyright 2020 Pexeso Inc. All Rights reserved.
    
    package matcher
    
    opened by horenmar 2
  • Multi-line string literal indentation

    Multi-line string literal indentation

    I'm trying to generate some code using dst package. Part of the code involves multi-line string literal. I'd like to keep the code prettier by propagating the indentation level into the string literal

    ... // node :=
    &dst.CallExpr{
    	Fun: &dst.SelectorExpr{
    		X:   &dst.Ident{Name: "mypackage"},
    		Sel: &dst.Ident{Name: "MyFunc"},
    	},
    	Args: []dst.Expr{
    		&dst.BasicLit{
    			Kind:  token.STRING,
    			Value: fmt.Sprintf("`\n%v\n`", "multi\nline\nliteral"),
    		},
    	},
    },
    

    The generated code will look like

    func GeneratedFunc() {
    	node := mypackage.MyFunc(`
    multi
    line
    literal
    `) // <-- This non-indent looks rather bothering
    }
    

    Is it possible that I can make the string literal indent aligned with the caller, or somehow retrieve the 'indentation level' via the dst package, so that I can manually Tweak the literal? e.g.

    func GeneratedFunc() {
    	node := mypackage.MyFunc(`
    		multi
    		line
    		literal
    		`)
    }
    
    opened by shichuzhu 2
  • FileRestorer doesn't preserve import grouping

    FileRestorer doesn't preserve import grouping

    This is minor, but curious whether you'd take a patch to grant users of the library control over import grouping.

    Some organizations (including my own) tend to group imports beyond stdlib/non-stdlib. I've seen two permutations at different orgs:

    1. stdlib, public, private
    2. stdlib, public, private (shared), private (local)

    goimports supports the former via its -local flag, such that:

    import (
    	"net/http"
    	"github.com/public/pkg"
    	"github.com/private/pkg"
    )
    

    when processed via goimports -local github.com/private will result in:

    import (
    	"net/http"
    
    	"github.com/public/pkg"
    
    	"github.com/private/pkg"
    )
    

    Further, goimports does not remove pre-existing groups.

    I'd like dst's import management to provide the ability to retain the original grouping of imports - though this does introduce the problem of needing to find an appropriate place to insert any new imports that aren't stdlib.

    Alternately, perhaps the best bet is to avoid using an import-aware FileRestorer and implement similar logic by hand.

    opened by Deiz 4
  • panic when incorrectly formatted code

    panic when incorrectly formatted code

    Hello,

    This might be a real edge case, but just wanting to share. When for example, missing a package main you will get a panic: panic: runtime error: invalid memory address or nil pointer dereference

    func TestDecorator(t *testing.T) {
    	code := `import "github.com/MyPackage/Name/package1"
    
    
    func main() {
    	fmt.Println("Hello!")
    }`
    	decorator.Parse(code)
    }
    

    I think earlier versions did not fail this test, and feel like it should gracefully return an error.

    opened by reinierkors 1
Owner
Dave Brophy
Dave Brophy
Grumpy is a Python to Go source code transcompiler and runtime.

Grumpy: Go running Python Overview Grumpy is a Python to Go source code transcompiler and runtime that is intended to be a near drop-in replacement fo

Google 10.5k Dec 1, 2022
HEP Fidelity Proxy

HFP HEP Fidelity Proxy Reliable way of relaying all your HEP to any HEP remote server that is behind unreliable networks. It is buffered TCP proxy wit

Ivica Lovric 10 Sep 26, 2022
Tool that can parse Go files into an abstract syntax tree and translate it to several programming languages.

GoDMT GoDMT, the one and only Go Data Model Translator. The goal of this project is to provide a tool that can parse Go files that include var, const,

Josep Jesus Bigorra Algaba 44 Nov 28, 2022
Search for HCL(v2) using syntax tree

hclgrep Search for HCL(v2) using syntax tree. The idea is heavily inspired by ht

magodo 77 Oct 21, 2022
Using NFP (Number Format Parser) you can get an Abstract Syntax Tree (AST) from Excel number format expression

NFP (Number Format Parser) Using NFP (Number Format Parser) you can get an Abstract Syntax Tree (AST) from Excel number format expression. Installatio

fossabot 0 Feb 4, 2022
Dependency-free replacement for GNU parallel, perfect fit for usage in an initramfs.

coshell v0.2.5 A no-frills dependency-free replacement for GNU parallel, perfect for initramfs usage. Licensed under GNU/GPL v2. How it works An sh -c

gdm85 41 May 11, 2022
Example of how to achieve pixel perfect shadows in a 2D view

Lights and Shadows Example of how to achieve pixel perfect shadows in a 2D view. The sample is based on the following sources: 2D Pixel Perfect Shadow

Defold 13 Oct 26, 2022
kubetnl tunnels TCP connections from within a Kubernetes cluster to a cluster-external endpoint, e.g. to your local machine. (the perfect complement to kubectl port-forward)

kubetnl kubetnl (kube tunnel) is a command line utility to tunnel TCP connections from within a Kubernetes to a cluster-external endpoint, e.g. to you

null 4 Sep 18, 2022
Memorize and perfect your Go movements 🥋

Practice yourself, for heaven's sake, in little things; and thence proceed to greater. -- Epictetus Go katas Katas (形) are practiced in martial arts a

Jozef Reisinger 39 Nov 10, 2022
Exp-tree: go library for parsing expression tree

Vinshop expression tree Exp-tree is go library for parsing expression tree Installation go get -u github.com/vinshop/exp-tree Quick start Format Expre

VinShop 6 May 11, 2022
app-services-go-linter plugin analyze source tree of Go files and validates the availability of i18n strings in *.toml files

app-services-go-linter app-services-go-linter plugin analyze source tree of Go files and validates the availability of i18n strings in *.toml files. A

Red Hat Developer 2 Nov 29, 2021
Golang package to manipulate time intervals.

timespan timespan is a Go library for interacting with intervals of time, defined as a start time and a duration. Documentation API Installation Insta

null 82 Sep 26, 2022
A Golang library to manipulate strings according to the word parsing rules of the UNIX Bourne shell.

shellwords A Golang library to manipulate strings according to the word parsing rules of the UNIX Bourne shell. Installation go get github.com/Wing924

Wei He 18 Sep 27, 2022
Manipulate subtitles in GO (.srt, .ssa/.ass, .stl, .ttml, .vtt (webvtt), teletext, etc.)

This is a Golang library to manipulate subtitles. It allows you to manipulate srt, stl, ttml, ssa/ass, webvtt and teletext files for now. Available op

Quentin Renard 422 Nov 22, 2022
Explore Docker registries and manipulate Docker images!

L/S tags Utility and API to manipulate (analyze, synchronize and aggregate) images across different Docker registries. Example invocation $ lstags alp

Ivan Ilves 302 Nov 25, 2022
manipulate and inspect VCS repositories in Go

go-vcs - manipulate and inspect VCS repositories go-vcs is a library for manipulating and inspecting VCS repositories in Go. It currently supports Git

Sourcegraph 73 Nov 27, 2022
Handy tools to manipulate korean character.

About hangul hangul is a set of handy tools for manipulate korean character in Go language. Example package main import ( "fmt" hangu

Homin Lee 43 Oct 27, 2022
QueryCSV enables you to load CSV files and manipulate them using SQL queries then after you finish you can export the new values to a CSV file

QueryCSV enable you to load CSV files and manipulate them using SQL queries then after you finish you can export the new values to CSV file

Mohamed Shapan 100 Dec 22, 2021
datatable is a Go package to manipulate tabular data, like an excel spreadsheet.

datatable is a Go package to manipulate tabular data, like an excel spreadsheet. datatable is inspired by the pandas python package and the data.frame R structure. Although it's production ready, be aware that we're still working on API improvements

Datasweet 221 Nov 23, 2022
Tiny lib to manipulate the .line format (.rm in the reMarkable2) in Go

linestogo Tiny lib to manipulate the .line format

Olivier Wulveryck 3 Apr 30, 2021