Grumpy is a Python to Go source code transcompiler and runtime.

Related tags

Compiler grumpy
Overview

Grumpy: Go running Python

Build Status Join the chat at https://gitter.im/grumpy-devel/Lobby

Overview

Grumpy is a Python to Go source code transcompiler and runtime that is intended to be a near drop-in replacement for CPython 2.7. The key difference is that it compiles Python source code to Go source code which is then compiled to native code, rather than to bytecode. This means that Grumpy has no VM. The compiled Go source code is a series of calls to the Grumpy runtime, a Go library serving a similar purpose to the Python C API (although the API is incompatible with CPython's).

Limitations

Things that will probably never be supported by Grumpy

  1. exec, eval and compile: These dynamic features of CPython are not supported by Grumpy because Grumpy modules consist of statically-compiled Go code. Supporting dynamic execution would require bundling Grumpy programs with the compilation toolchain, which would be unwieldy and impractically slow.

  2. C extension modules: Grumpy has a different API and object layout than CPython and so supporting C extensions would be difficult. In principle it's possible to support them via an API bridge layer like the one that JyNI provides for Jython, but it would be hard to maintain and would add significant overhead when calling into and out of extension modules.

Things that Grumpy will support but doesn't yet

There are three basic categories of incomplete functionality:

  1. Language features: Most language features are implemented with the notable exception of old-style classes. There are also a handful of operators that aren't yet supported.

  2. Builtin functions and types: There are a number of missing functions and types in __builtins__ that have not yet been implemented. There are also a lot of methods on builtin types that are missing.

  3. Standard library: The Python standard library is very large and much of it is pure Python, so as the language features and builtins get filled out, many modules will just work. But there are also a number of libraries in CPython that are C extension modules which will need to be rewritten.

  4. C locale support: Go doesn't support locales in the same way that C does. As such, some functionality that is locale-dependent may not currently work the same as in CPython.

Running Grumpy Programs

Method 1: make run:

The simplest way to execute a Grumpy program is to use make run, which wraps a shell script called grumprun that takes Python code on stdin and builds and runs the code under Grumpy. All of the commands below are assumed to be run from the root directory of the Grumpy source code distribution:

echo "print 'hello, world'" | make run

Method 2: grumpc and grumprun:

For more complicated programs, you'll want to compile your Python source code to Go using grumpc (the Grumpy compiler) and then build the Go code using go build. Since Grumpy programs are statically linked, all the modules in a program must be findable by the Grumpy toolchain on the GOPATH. Grumpy looks for Go packages corresponding to Python modules in the __python__ subdirectory of the GOPATH. By convention, this subdirectory is also used for staging Python source code, making it similar to the PYTHONPATH.

The first step is to set up the shell so that the Grumpy toolchain and libraries can be found. From the root directory of the Grumpy source distribution run:

make
export PATH=$PWD/build/bin:$PATH
export GOPATH=$PWD/build
export PYTHONPATH=$PWD/build/lib/python2.7/site-packages

You will know things are working if you see the expected output from this command:

echo 'import sys; print sys.version' | grumprun

Next, we will write our simple Python module into the __python__ directory:

echo 'def hello(): print "hello, world"' > $GOPATH/src/__python__/hello.py

To build a Go package from our Python script, run the following:

mkdir -p $GOPATH/src/__python__/hello
grumpc -modname=hello $GOPATH/src/__python__/hello.py > \
    $GOPATH/src/__python__/hello/module.go

You should now be able to build a Go program that imports the package "__python__/hello". We can also import this module into Python programs that are built using grumprun:

echo 'from hello import hello; hello()' | grumprun

grumprun is doing a few things under the hood here:

  1. Compiles the given Python code to a dummy Go package, the same way we produced __python__/hello/module.go above
  2. Produces a main Go package that imports the Go package from step 1. and executes it as our __main__ Python package
  3. Executes go run on the main package generated in step 2.

Developing Grumpy

There are three main components and depending on what kind of feature you're writing, you may need to change one or more of these.

grumpc

Grumpy converts Python programs into Go programs and grumpc is the tool responsible for parsing Python code and generating Go code from it. grumpc is written in Python and uses the pythonparser module to accomplish parsing.

The grumpc script itself lives at tools/grumpc. It is supported by a number of Python modules in the compiler subdir.

Grumpy Runtime

The Go code generated by grumpc performs operations on data structures that represent Python objects in running Grumpy programs. These data structures and operations are defined in the grumpy Go library (source is in the runtime subdir of the source distribution). This runtime is analogous to the Python C API and many of the structures and operations defined by grumpy have counterparts in CPython.

Grumpy Standard Library

Much of the Python standard library is written in Python and thus "just works" in Grumpy. These parts of the standard library are copied from CPython 2.7 (possibly with light modifications). For licensing reasons, these files are kept in the third_party subdir.

The parts of the standard library that cannot be written in pure Python, e.g. file and directory operations, are kept in the lib subdir. In CPython these kinds of modules are written as C extensions. In Grumpy they are written in Python but they use native Go extensions to access facilities not otherwise available in Python.

Source Code Overview

  • compiler: Python package implementating Python -> Go transcompilation logic.
  • lib: Grumpy-specific Python standard library implementation.
  • runtime: Go source code for the Grumpy runtime library.
  • third_party/ouroboros: Pure Python standard libraries copied from the Ouroboros project.
  • third_party/pypy: Pure Python standard libraries copied from PyPy.
  • third_party/stdlib: Pure Python standard libraries copied from CPython.
  • tools: Transcompilation and utility binaries.

Contact

Questions? Comments? Drop us a line at [email protected] or join our Gitter channel

Comments
  • Build gets into strange state when first invocation uses wrong version of Python

    Build gets into strange state when first invocation uses wrong version of Python

    [@localhost grumpy]$ make build/src/grumpy/lib/itertools/module.go:5: can't find import: "grumpy/lib/sys" make: *** [build/pkg/linux_amd64/grumpy/lib/itertools.a] Error 1

    duplicate 
    opened by yyh-graphsql 43
  • Tag `attr_mode:

    Tag `attr_mode:"rw"` to read-write native attributes

    Use tag attr_mode:"rw" to allow native attributes to be changed by Python code. The attribute should be public on Golang (start with a Capital).

    • makeStructFieldDescriptor needs a last boolean argument. Please provide false for private fields.
    • When the new argument is true and the struct field is public, it can be changed (set) via Python
    • Will not be locked on changes. If locking is needed, do it in Python or via a custom Property setter
    opened by alanjds 32
  • ImportError: no such module: socket

    ImportError: no such module: socket

    How do I get past this error:

    Traceback (most recent call last):
      File "./build/bin/grumpc", line 118, in <module>
        sys.exit(main(parser.parse_args()))
      File "./build/bin/grumpc", line 76, in main
        visitor.visit(mod)
      File "/home/localhost/Documents/Dev/play/grumpy/build/lib/python2.7/site-packages/grumpy/pythonparser/algorithm.py", line 41, in visit
        return self._visit_one(obj)
      File "/home/localhost/Documents/Dev/play/grumpy/build/lib/python2.7/site-packages/grumpy/pythonparser/algorithm.py", line 32, in _visit_one
        return getattr(self, visit_attr)(node)
      File "/home/localhost/Documents/Dev/play/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/stmt.py", line 320, in visit_Module
        self._visit_each(node.body)
      File "/home/localhost/Documents/Dev/play/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/stmt.py", line 734, in _visit_each
        self.visit(node)
      File "/home/localhost/Documents/Dev/play/grumpy/build/lib/python2.7/site-packages/grumpy/pythonparser/algorithm.py", line 41, in visit
        return self._visit_one(obj)
      File "/home/localhost/Documents/Dev/play/grumpy/build/lib/python2.7/site-packages/grumpy/pythonparser/algorithm.py", line 32, in _visit_one
        return getattr(self, visit_attr)(node)
      File "/home/localhost/Documents/Dev/play/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/stmt.py", line 287, in visit_Import
        for imp in self.block.root.importer.visit(node):
      File "/home/localhost/Documents/Dev/play/grumpy/build/lib/python2.7/site-packages/grumpy/pythonparser/algorithm.py", line 41, in visit
        return self._visit_one(obj)
      File "/home/localhost/Documents/Dev/play/grumpy/build/lib/python2.7/site-packages/grumpy/pythonparser/algorithm.py", line 32, in _visit_one
        return getattr(self, visit_attr)(node)
      File "/home/localhost/Documents/Dev/play/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/imputil.py", line 88, in visit_Import
        imp = self._resolve_import(node, alias.name)
      File "/home/localhost/Documents/Dev/play/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/imputil.py", line 153, in _resolve_import
        raise util.ImportError(node, 'no such module: {}'.format(modname))
    grumpy.compiler.util.ImportError: line 9: no such module: socket
    
    
    

    I"m trying to transpile python to Go code on Fedora 25

    opened by ch3ck 17
  • Replace use of builtin ast module with pythonparser.

    Replace use of builtin ast module with pythonparser.

    pythonparser is a high quality Python parser written in Python that produces an AST very similar to the one produced by the standard ast module. There are three very appealing qualities to switching Grumpy to pythonparser:

    1. It's pure Python so there's a reasonable chance of getting it to build under Grumpy in the near future, whereas the ast module, being written in C, is unlikely to be work anytime soon.

    2. The AST produced by pythonparser contains detailed span location data. The standard ast module provides line numbers only and not for all nodes. pythonparser provides column data, sub-node locations, locations for comments and more.

    3. Experimenting with custom syntax (e.g. to properly support importing Go packages with arbitrary names, like from __go__ import "github.com/labstack/echo") will become much simpler.

    I suspect pythonparser is quite a bit slower than the ast module but it doesn't seem to matter since Go compilation dominates. Timings for make test were 9m35.968s under the ast module and 10m7.250s under pythonparser.

    There are a few minor outstanding issues with the library I've filed. These seem relatively straightforward to fix and the maintainer is open to PRs. Overall I'm very satisfied with the quality and completeness of the library.

    opened by trotterdylan 12
  • Implementation of raw_input()

    Implementation of raw_input()

    I implemented a simplified version of raw_input().

    Please review on focusing these. First, It might be not 100% compatible with CPython and pypy's raw_input(). and also need more implementation to handle error cases.

    Second, I need some guide for writing unit test codes to check it is well working for raw_input(). I don't know how to test user's input from keyboard.

    Third, prompt printing is slower when running with go run *.go so if you have solutions then please give me a feedback.

    Related PR= #247

    opened by corona10 11
  • STD lib roadmap

    STD lib roadmap

    Very nice project.

    I am a golang programmer and it would be nice to have many of the machine learning python code available to golang programmers.

    Is there a CI build system to build and test the pythons libs yet ?

    opened by joeblew99 11
  • Add support for the `callable` builtin

    Add support for the `callable` builtin

    Implement the callable as defined here:

    • https://docs.python.org/2.7/library/functions.html#callable

    make precommit passed.

    Signed-off-by: Cholerae Hu [email protected]

    opened by choleraehyq 11
  • Added math module

    Added math module

    This request contains a shim package that wraps Go functions from the math package in functions that are identical to those in the Python math module. There are three comments that begin with "NOTE", that I think require input from others and probably some corrections.

    opened by lalaithion 10
  • Create Standard Library checklist.md

    Create Standard Library checklist.md

    #!/usr/bin/env python3
    import bs4, requests
    url = 'https://docs.python.org/2.7/py-modindex.html'
    soup = bs4.BeautifulSoup(requests.get(url).content, 'html.parser')
    for module in soup.find_all('code', class_='xref'):
        if '.' not in module.string:
            print('- [ ] ' + module.string.replace('__', '\_\_', 1))
    
    opened by cclauss 9
  • Add _struct, _md5 to third_party/pypy

    Add _struct, _md5 to third_party/pypy

    Added _struct, _md5, md5(wrapper to _md5) to third_party/pypy.

    Hard-coded byteorder = 'little' on sys module for now, which is needed by _struct Let me know if I better hardcode inside _struct module.

    Note that I cannot add wrapper struct.py because struct is a keyword in go as you know.

    opened by S-YOU 9
  • Homebrew does not have a formula for Grumpy

    Homebrew does not have a formula for Grumpy

    https://github.com/Homebrew/homebrew-core/issues/9213

    Does anyone know how the homebrew installer for Mac OSX works? I spent an few hours trying to build a formula to install Grumpy based on these examples but I failed.

    opened by cclauss 8
  • make: *** No rule to make target 'run'.  Stop.

    make: *** No rule to make target 'run'. Stop.

    I installed grumpy using snap (sudo snap install grumpy). I would like to know which exactly is the root directory of the Grumpy source code distribution.

    I'm asking this because I tried to do echo "print 'hello, world'" | make run into /snap/grumpy/ and get error:

    $ echo "print 'hello, world'" | make run
    make: *** No rule to make target 'run'.  Stop.
    

    here my content from directory

    $ ls /snap/grumpy/
    15 current
    
    opened by DiegoBneiNoah 1
  • Rebuild tool for automated refactoring

    Rebuild tool for automated refactoring

    opened by XVilka 0
  • Failed to import some python std libs(e.g. json) in python code and then called in go code

    Failed to import some python std libs(e.g. json) in python code and then called in go code

    My python code is as below

    def feature1(inputJsonString):
        print inputJsonString
        import json
        import sys
        print json.dumps(1)
        return sys.version
    

    and the go main code is as below

    package main
    
    import ... // I have imported all the package under the "$GOPATH/src/__python__" and thus "json" related libs should be imported
    
    func main() {
        f := grumpy.NewRootFrame()
        mods, _ := grumpy.ImportModule(f, "feature1")
        fmt.Printf("mods: %v\n", mods)
        feature1, _ := grumpy.GetAttr(f, mods[0], grumpy.NewStr("feature1"), nil)
        args := f.MakeArgs(1)
        args[0] = grumpy.NewStr(`{"a": 1, "b": "I am string"}`).ToObject()
        result, err := feature1.Call(f, args, nil)
        fmt.Printf("result type: %v\n", reflect.TypeOf(result))
        fmt.Printf("result: %v\n", result.String())
        fmt.Printf("err: %v\n", err)
    }
    

    When I tried to run the go main code, and I got result like this

    result: nil
    err: TypeError('bad operand type for unary -: \'tuple\'',)
    

    I found this error is raised from grumpy.ImportModule function in generated go code feature1.go but I am not clear how this error is raised. Also when I try to import some other python std libs like "sys", and it works fine.

    My Golang version is go1.10.3 darwin/amd64

    opened by always-less 0
Owner
Google
Google ❤️ Open Source
Google
Go -> Haxe -> JS Java C# C++ C Python Lua

go2hx Compile: Go -> Haxe -> Js, Lua, C#, C++, Java, C, Python warning: heavily experimental still a ways to go before an alpha. Come give feedback on

go2hx 78 Nov 30, 2022
Transpiling C code to Go code

A tool for transpiling C code to Go code. Milestones of the project: Transpiling project GNU GSL. Transpiling project GTK+. Notes: Transpiler works on

Konstantin 329 Dec 2, 2022
Transpiling fortran code to golang code

f4go Example of use > # Install golang > # Compile f4go > go get -u github.com/Konstantin8105/f4go > cd $GOPATH/src/github.com/Konstantin8105/f4go > g

Konstantin 34 Sep 26, 2022
Decorated Syntax Tree - manipulate Go source with perfect fidelity.

Decorated Syntax Tree The dst package enables manipulation of a Go syntax tree with high fidelity. Decorations (e.g. comments and line spacing) remain

Dave Brophy 992 Nov 28, 2022
A compiler from Go to JavaScript for running Go code in a browser

GopherJS - A compiler from Go to JavaScript GopherJS compiles Go code (golang.org) to pure JavaScript code. Its main purpose is to give you the opport

GopherJS 11.6k Dec 5, 2022
Transform Go code into it's AST

Welcome to go2ast ?? Transform Go code into it's AST Usage echo "a := 1" | go run main.go Example output []ast.Stmt { &ast.AssignStmt {

Eli Yukelzon 106 Nov 11, 2022
Compile Go regular expressions to Go code

regexp2go regexp2go is an alternate backend for the regexp package that allows to perform ahead-of-time compilation of regular expressions to Go code.

Carlo Alberto Ferraris 85 Jul 11, 2022
Syntax-aware Go code search, based on the mvdan/gogrep

gogrep WIP: this is an attempt to move modified gogrep from the go-ruleguard project, so it can be used outside of the ruleguard as a library. Acknowl

Iskander (Alex) Sharipov 30 Nov 9, 2022
Promise to the Go compiler that your Reads and Writes are well-behaved

noescape go get lukechampine.com/noescape noescape provides Read and Write functions that do not heap-allocate their argument. Normally, when you pas

Luke Champine 33 Jun 30, 2022
Go compiler for small places. Microcontrollers, WebAssembly, and command-line tools. Based on LLVM.

TinyGo - Go compiler for small places TinyGo is a Go compiler intended for use in small places such as microcontrollers, WebAssembly (Wasm), and comma

TinyGo 11.9k Dec 2, 2022
GopherLua: VM and compiler for Lua in Go

GopherLua: VM and compiler for Lua in Go. GopherLua is a Lua5.1 VM and compiler written in Go. GopherLua has a same goal with Lua: Be a scripting lang

Yusuke Inuzuka 5.1k Dec 6, 2022
A Lua 5.3 VM and compiler written in Go.

DCLua - Go Lua Compiler and VM: This is a Lua 5.3 VM and compiler written in Go. This is intended to allow easy embedding into Go programs, with minim

Milo Christiansen 901 Nov 28, 2022
High-performance PHP application server, load-balancer and process manager written in Golang

RoadRunner is an open-source (MIT licensed) high-performance PHP application server, load balancer, and process manager. It supports running as a serv

Spiral Scout 6.9k Nov 27, 2022
Mathematical expression parsing and calculation engine library. 数学表达式解析计算引擎库

Math-Engine 使用 Go 实现的数学表达式解析计算引擎库,它小巧,无任何依赖,具有扩展性(比如可以注册自己的函数到引擎中),比较完整的完成了数学表达式解析执行,包括词法分析、语法分析、构建AST、运行。 go get -u github.com/dengsgo/math-engine 能够

Deng.Liu 251 Nov 28, 2022
Runcmd - just golang binary that runs commands from url or local file and logs output

runcmd just golang binary that runs commands from url or local file and logs out

boredhackerblog 0 Feb 2, 2022
Golang library to act on structure fields at runtime. Similar to Python getattr(), setattr(), hasattr() APIs.

go-attr Golang library to act on structure fields at runtime. Similar to Python getattr(), setattr(), hasattr() APIs. This package provides user frien

Shyamsunder Rathi 27 Jul 5, 2022
Identify containers at runtime and observe them. No container runtime required. Read only access to the kernel.

Linux Telemetry The Double Slit Experiment Taken from an interesting physics anomaly where the behavior of a physical system mutates simply by being o

Kris Nóva 14 Sep 18, 2022
golang-runtime-di is a framework for runtime dependency injection in go

golang-runtime-di description golang-runtime-di is a framework for runtime dependency injection in go. usage quickstart add it to your go.mod: go get

DB Systel GmbH 2 Aug 1, 2022
Open Source runtime tool which help to detect malware code execution and run time mis-configuration change on a kubernetes cluster

Kube-Knark Project Trace your kubernetes runtime !! Kube-Knark is an open source tracer uses pcap & ebpf technology to perform runtime tracing on a de

Chen Keinan 32 Sep 19, 2022
:triangular_ruler:gofmtmd formats go source code block in Markdown. detects fenced code & formats code using gofmt.

gofmtmd gofmtmd formats go source code block in Markdown. detects fenced code & formats code using gofmt. Installation $ go get github.com/po3rin/gofm

po3rin 91 Oct 31, 2022