Ruby -> Go at the snap of your fingers

Related tags

DevOps Tools thanos


Thanos aims to be a source-source compiler from Ruby to human-readable Go. It's still a few stones short of universe-altering power. Run thanos help for a list of commands.


The Big Caveats

  • Type hints/annotations -- the current type inference model relies on tracking method calls back to literal values. For library code that only ever is called in test or in client applications, this is insufficient.
  • Exception handling -- the impedance mismatch between trapping runtime exceptions in Ruby and the comma-error pattern in Go is one of the largest differences between the two languages. It is large enough that I am avoiding implementing it entirely for now. It may be possible but poses a number of difficulties that I believe can only be fully addressed by refactoring the source Ruby or refactoring the target Go.
  • Metaprogramming -- I have no interest in building a Ruby runtime, which makes the extent of the metaprogramming that is realistic to support fairly small.
  • Dependencies -- since thanos isn't a runtime, and doesn't support metaprogramming, pulling in existing Ruby libraries is more or less out of the question.
  • Heterogenous arrays and hashes aren't on the menu. I do hope to support detection and generation of common interface types but in a fairly limited way.
  • Hashes are translated directly into Go maps without any sort of shim type, which means that ordering guarantees provided by Ruby are not respected; thus a number of Enumerable methods that depend on those guarantees are not supported.


  1. Short-term goals

    a. Complete planned grammar support. Explicitly excluded from the plan are:

    • BEGIN and END blocks
    • =begin/=end comments
    • Interpolation of instance/class variables using #@foo instead of #{@foo} (did you even know you could do this??)
    • Global variables, including all the goofy automatically populated ones

    b. Flesh out support for core classes. Many methods are missing from built-in primitive types, support for Range and Proc are very limited, and several important classes (namely Struct, Date, and Time) have no support at all. The methods missing from classes with any support at all are documented using thanos report.

  2. Long-term goals

    a. Allow comments to be used as simple, example-based type annotation with literal values (as opposed to actually having to annotate source using the Ruby 3 syntax)

    b. Provide some sort of support for exception handling, even if it just means eliding the begin/rescue/end

    c. Support automatic generation of a ruby-ffi wrapper gem

How it works

The flow from bytes representing Ruby to bytes representing Go is as follows:

  • The parser, generated with goyacc (see parser/ruby.y), consumes tokens using the lexer in parser/lexer.go and generates a parse tree using types implementing the Node interface in parser/ast.go.
  • The resulting AST, stored on *parser.Root, then undergoes type inference by calling Analyze() on *parser.Root. This relies on:
    • the parser.GetType function and the Type, SetType, and TargetType methods on Node
    • the types package, which contains:
      • a Type interface
      • predefined implementations of the Type interface for Ruby primitive types and a small (but growing!) set of other classes from the Ruby standard library
      • facilities for generating new Types for classes at parse time and for generating adapters from Ruby classes to Go functions when necessary
  • The type-annotated AST is handed off to compiler.Compile, which translates parser.Nodes into the appropriate analogs in the go/ast package. For method calls, this involves retrieving a types.TransformAST function, the return value of which supplies statements to prepend and an expression to substitute (more about this later). The resulting Go AST is then formatted and printed.

Translation guide

Primitive Ruby objects, despite their duck-typed squishiness, have typed translation targets in Go that are hopefully easy to guess. Other constructs might have less predictable behavior.


Classes are translated to a struct. When class method and variable support is added, this will probably be a set of two structs. Some specific class features:

  • Instance variables are translated to struct fields. If the instance variable has attr_accessor called, it will be an exported struct field, and if instance variable x has attr_reader or attr_writer called, the struct will have an exported X or SetX method respectively.
  • A class named Foo will also have a NewFoo constructor function generated."name", false) will translate to NewFoo("name", false). If the class defines an initialize method, it will be called inside this function.
  • super calls inline the parent method body inside an function literal invocation.


Constants are translated to const declarations where the type allows, and package level var declarations for any other type. Constants declared inside another constant (class or module) have a Go name in the form (ModuleName)*(ClassName)?ConstantName.


Block arguments to the collections methods implemented thus far are unfolded into for ... range loops in the target, with an additional slice or map declaration where appropriate.

For user-defined methods that take a block, a function type will be generated based on the inferred types of the arguments and return values of blocks passed in method calls. That type will be used in creating the signature for the function or method in Go. Block arguments will be translated to func literals conforming to this type signature.

Regular expressions

Regular expression literals are translated to *regexp.Regexp values. If there is no interpolation in the regex,it is created with a top-level variable declaration using regexp.MustCompile; otherwise it is created in the local scope. In either case it is assigned to a local variable named pattX, where X is len(patt idents added to local scope) + 1.

A compatibility layer for MatchData instances is provided in the stdlib package, since some of that functionality is not directly analogous to convenience functions provided by Go's regexp package.

The =~ operator returns a boolean rather than an integer-or-nil since it is translated directly to regexp.MatchString. regexp.MatchString is also used when a Ruby regex literal is provide as the argument to when in a case expression.

Adding functionality

Adding a method to a built-in type

Let's imagine that Ruby has a method Array#snap that eliminates every other element from its receiver. We can imagine implementing this in Ruby with something like{|elem, i| i % 2 == 0 }.map(&:first), but in MRI, it's probably implemented in C. In Go, we probably wouldn't have this method at all, but instead would just range over the array like so:

unsnapped := []*Hero{}
for i, hero := range heroes {
  if i % 2 == 0 {
    unsnapped = append(unsnapped, hero)

Like most methods on Array, Array#snap returns the resulting array, allowing chaining. So while in Ruby we have a single expression, in Go we have a statement initializing a variable and a for...range statement. However, when implementing this method in thanos, that's not enough. We also must somehow return the result of the method call as an expression in case we are compiling an expression like heroes.snap.first, or if heroes.snap is the last expression in a method and needs to be returned. In this case that expression is unsnapped.

We start by opening up types/array.go and looking at the massive init() function at the bottom of the file. We'll add our snap implemention to the other methods already there.

	arrayProto.Def("select", MethodSpec{
		ReturnType: func(r Type, b Type, args []Type) (Type, error) {
			return r, nil
		TransformAST: func(rcvr TypeExpr, args []TypeExpr, blk *Block, it bst.IdentTracker) Transform {
      // TODO implement me!
      return Transform{}

And with this step, we've satisfied the thanos type inference engine, so parsing Array#snap will now work -- the ReturnType field of a types.MethodSpec will be called with the type (a types.Type value) for the receiver, the return type of a block, if the method takes one, and the types of any arguments given. In this case, we expect the return type to be exactly the type of the receiver, so we just return the first argument without an error.

Now it's time to figure out how to compile our snap call into a for-loop, which is what goes in the body of the anonymous function given as the TransformAST field. We start by declaring and initializing our unsnapped variable:

unsnapped := it.New("unsnapped")
initSlice := emptySlice(unsnapped, rcvr.Type.(Array).Element.GoType())

What's happening here:

  • The bst.IdentTracker provided to the function is keeping track of all the identifiers in the current block. When we call it.New, it is checking to see if we've already initialized an unsnapped variable in the current scope, and if we have, it'll call it unsnapped1 instead.
  • emptySlice is a utility function that generates the Go AST fragment for <variable name> := []<Go type>{}. We pass in a *go/ast.Ident and a string representing the type in Go.
  • A types.TypeExpr is a struct with two fields: the inferred type from the Ruby source (types.Type) and the precompiled Go AST node (go/ast.Expr). types.Array implements types.Type; it also has a struct field Element that is also a types.Type. the types.Type interface requires a GoType() string method to satisfy it.

Next, we add our loop. This is where things get a little dirty:

i, x := it.Get("i"), it.Get("x")
loop := &ast.RangeStmt{
  Key:   i,
  Value: x,
  Tok:   token.DEFINE,
  X:     rcvr.Expr,
  Body: &ast.BlockStmt{
    List: []ast.Stmt{
        Cond: bst.Binary(bst.Binary(i, token.REM, bst.Int(2)), token.EQL, bst.Int(0)),
        Body: &ast.BlockStmt{
          List: []ast.Stmt{
            bst.Assign(unsnapped, bst.Call(nil, "append", unsnapped, x)),

The first line gives us locals for the identifiers in our loop. it.Get, unlike it.New, will recycle existing identifiers and assume there are no collisions.

Then we get to the definition of the loop itself. The bst package provides some utilities for AST generation (naming is hard). As you can see, it is far from complete, and we have to specify several levels of the AST by hand. The behavior of the functions used here from bst are hopefully self-evident:

  • bst.Assign returns the appropriate Go AST nodes for <variable_name> = <rhs>
  • bst.Call produces the right fragment for a method or function call, depending on whether the first argument is nil
  • bst.Binary takes LHS, operator token, RHS and returns the appropriate expression node
  • bst.Int returns an *ast.BasicLit with Kind set to token.INT

We now have everything we need to transform an Array#snap call into a simple loop in Go. All that's left to do is to send that info back to the compiler.

return Transform{
  Stmts: []ast.Stmt{initSlice, loop},
  Expr: unsnapped,

The four code snippets above are enough for thanos. It will happily compile these method calls now.

Leveraging dependencies

Thanos strives to generate Go code with few dependencies on itself. However, purity of principles must not stand in the way of the mission, and sacrifices will have to be made. The stdlib provides a place to house such dependencies. In the case of Array#snap, we could implement a method in stdlib/snap.go using Go's new generics:

func Snap[Elem any](beings []Elem) []Elem {
  unsnapped := []Elem{}
  for i, x := range beings {
    if i%2 == 0 {
      unsnapped = append(unsnapped, x)
  return unsnapped

I would say this is use case is overkill, and the handrolled AST is the right approach -- especially since this function probably has little to no reuse. Nonetheless, we could now simplify our TransformAST function body to the following, specifying the required dependency:

unsnapped := it.New("unsnapped")
assignment := bst.Assign(unsnapped, bst.Call("stdlib", "Snap", rcvr.Expr))
return Transform {
  Stmts: []ast.Stmt{assignment},
  Expr: unsnapped,
  Imports: []string{""},

Writing tests

While most of the thanos test suite is rather conventional, the compiler package works a bit differently. There are two separate sets of tests:

Style tests

Given the project's focus on producing human-readable output, it is important to validate that the Go resulting from the compilation step looks like something that would at minimum be a reasonable departure point for a refactor. The compiler package thus operates on parallel Ruby input and expected Go output files in the compiler/testdata directory. go test ./compiler -filename <name of test file without extension> can be used to run the test for a single file, or just go test ./compiler to run them them all.

It is important to note that these tests do not compile the Go output. There are two reasons for this:

  • It takes more than a string of valid Go expressions to make a Go program, but for testing purposes we often don't care that, for example, a variable is declared but never used. Feeding the output to the compiler would get in the way of efficiently testing the compilation of specific methods and expressions.
  • I imagine cases where the thanos output doesn't compile because of a bug or missing features, but a human being can look at it and quickly identify the fix and move on. I don't want the tests to assume that this use case doesn't exist.

Gauntlet tests

Gauntlet tests are end-to-end verifications that the stdout from given Ruby matches the stdout from the Go program thanos produces when compiling that Ruby. You run them with the thanos test command -- run thanos test --help to see options. You write them using the gauntlet pseudo-method, which is implemented using some rather nasty hackery housed primarily in the thanos lexer. They look like this:

gauntlet("drop") do
  [1,2,3,4,5].drop(3).each do |x|
    puts x

When run, this will execute the body of the block argument using whatever version of ruby corresponds to the ruby on your path, run that same code through thanos, and feed the output to go run. Because the gauntlet method isn't real Ruby, it's okay for the block to contain Ruby that wouldn't normally be valid, like declaring constants.

You might also like...
k8s-image-swapper Mirror images into your own registry and swap image references automatically.
k8s-image-swapper Mirror images into your own registry and swap image references automatically.

k8s-image-swapper Mirror images into your own registry and swap image references automatically. k8s-image-swapper is a mutating webhook for Kubernetes

Easily run your Compose application to the cloud with compose-cli

This CLI tool makes it easy to run Docker containers and Docker Compose applications in the cloud using either Amazon Elastic Container Service (ECS) or Microsoft Azure Container Instances (ACI) using the Docker commands you already know.

🐶 Kubernetes CLI To Manage Your Clusters In Style!
🐶 Kubernetes CLI To Manage Your Clusters In Style!

K9s - Kubernetes CLI To Manage Your Clusters In Style! K9s provides a terminal UI to interact with your Kubernetes clusters. The aim of this project i

Open URL in your local web browser from the SSH-connected remote environment.

opener Open URL in your local web browser from the SSH-connected remote environment. How does opener work? opener is a daemon process that runs locall

Validation of best practices in your Kubernetes clusters
Validation of best practices in your Kubernetes clusters

Best Practices for Kubernetes Workload Configuration Fairwinds' Polaris keeps your clusters sailing smoothly. It runs a variety of checks to ensure th

🛅 Backup your Kubernetes Stateful Applications

Stash Stash by AppsCode is a cloud-native data backup and recovery solution for Kubernetes workloads. If you are running production workloads in Kuber

A cli that exposes your local resources to kubernetes
A cli that exposes your local resources to kubernetes

ktunnel Expose your local resources to kubernetes 📝 Table of Contents About Getting Started Usage Documentation Contributing Authors Acknowledgments

Bubbly is an open-source platform that gives you confidence in your continuous release process.
Bubbly is an open-source platform that gives you confidence in your continuous release process.

Bubbly Bubbly - Release Readiness in a Bubble Bubbly emerged from a need that many lean software teams practicing Continuous Integration and Delivery

Putting serverless on your server
Putting serverless on your server

Matterless: putting serverless on your server Serverless computing enables you to build applications that automatically scale with demand, and your wa

  • Support `*args` and `**kwargs`

    Support `*args` and `**kwargs`

    In method signatures, *args support will translate into a Go variadic parameter, and if the Ruby method's *args has heterogenous types, it's gonna break.

    On the LHS of an assignment, *foo will only be supported if the corresponding RHS members are of the same type. If the LHS has multiple values and the RHS is singular and an array, we can translate the RHS into a series of index and slice expressions.

    **kwargs in a method signature should be compiled away to positional arguments for any key in the Hash that is ever accessed. A first pass should not support anything beyond that.

    opened by redneckbeard 0
  • Defaults for keyword parameters not handled correctly if double-splat identifier argument given

    Defaults for keyword parameters not handled correctly if double-splat identifier argument given

    def foo(bar: 10, **kwargs); end; foo(**some_hash) requires that the target explicitly check the splatted hash for presence of a bar key before blowing away the default. Current implementation does not do this.

    opened by redneckbeard 0
  • Support `Date` and `Time`

    Support `Date` and `Time`

    Should be relatively straightforward with the exception of formatting. Because of Go's rather unconventional parse string format, will require a compatibility layer in the stdlib package for parsing format strings.

    opened by redneckbeard 0
Jonathan Paul Lukens
Jonathan Paul Lukens
Monitor your Website and APIs from your Computer. Get Notified through Slack, E-mail when your server is down or response time is more than expected.

StatusOK Monitor your Website and APIs from your computer.Get notified through Slack or E-mail when your server is down or response time is more than

Sanath Kumar 1.6k Dec 27, 2022
Supporting your devops by shortening your strings using common abbreviations and clever guesswork

abbreviate Shorten your strings using common abbreviations. Supported by Tidelift Motivation This tool comes out of a frustration of the name of resou

Pascal Dennerly 193 Dec 14, 2022
💓 小米手环实时心率数据采集 - Your Soul, Your Beats!

?? mebeats 小米手环实时心率数据采集 - Your Soul, Your Beats! cmd/mebeats-client: the mebeats client. It collects the heart rate data from Mi Band and reports to s

E99p1ant 193 Dec 31, 2022
Kusk makes your OpenAPI definition the source of truth for API resources in your cluster

Kusk - use OpenAPI to configure Kubernetes What is Kusk? Developers deploying their REST APIs in Kubernetes shouldn't have to worry about managing res

kubeshop 7 Dec 16, 2022
A API for your youtube channel to protect your api key

Youtube-Channel-Badge A API for your youtube channel to protect your

Mallikarjun H 0 Dec 23, 2021
A simple download file manager that sorts your files into your desired folders, This was meant to be a small project and nothing big.

GoDFM Simply go to the tags and download the .exe file (or compile it yourself by running go build). Add it to your environment paths by going to sett

null 0 Aug 9, 2022
Enable your Go applications to self update

go-selfupdate Enable your Golang applications to self update. Inspired by Chrome based on Heroku's hk. Features Tested on Mac, Linux, Arm, and Windows

Mark 1.1k Jan 3, 2023
Jenkins CLI allows you manage your Jenkins as an easy way

Quick start 简体中文 Jenkins CLI Jenkins CLI allows you manage your Jenkins in an easy way. No matter if you're a plugin developer, administrator or just

Jenkins Chinese Community 346 Jan 4, 2023
Manage your ssh alias configs easily.

manssh manssh is a command line tool for managing your ssh alias config easily, inspired by storm project, powered by Go. Note: This project is actual

Wendell Sun 276 Nov 9, 2022
grafana-sync Keep your grafana dashboards in sync.

grafana-sync Keep your grafana dashboards in sync. Table of Contents grafana-sync Table of Contents Installing Getting Started Pull Save all dashboard

Maksym Postument 169 Dec 14, 2022