naive go bindings to the CPython C-API

Overview

go-python

Build Status Build status

Naive go bindings towards the C-API of CPython-2.

this package provides a go package named "python" under which most of the PyXYZ functions and macros of the public C-API of CPython have been exposed.

theoretically, you should be able to just look at:

http://docs.python.org/c-api/index.html

and know what to type in your go program.

this package also provides an executable "go-python" which just loads "python" and then call python.Py_Main(os.Args). the rational being that under such an executable, go based extensions for C-Python would be easier to implement (as this usually means calling into go from C through some rather convoluted functions hops)

Install

With Go 1 and the go tool, cgo packages can't pass anymore additional CGO_CFLAGS from external programs (except pkg-config) to the "fake" #cgo preprocessor directive.

go-python now uses pkg-config to get the correct location of headers and libraries. Unfortunately, the naming convention for the pkg-config package is not standardised across distributions and OSes, so you may have to edit the cgoflags.go file accordingly.

 $ go get github.com/sbinet/go-python

If go get + pkg-config failed:

 $ cd go-python
 $ edit cgoflags.go
 $ make VERBOSE=1

Note: you'll need the proper header and python development environment. On Debian, you'll need to install the python-all-dev package

Documentation

Is available on godoc:

http://godoc.org/github.com/sbinet/go-python

Example:

package main

import "fmt"
import "github.com/sbinet/go-python"

func init() {
   err := python.Initialize()
   if err != nil {
          panic(err.Error())
   } 
}

func main() {
 	 gostr := "foo" 
	 pystr := python.PyString_FromString(gostr)
	 str := python.PyString_AsString(pystr)
	 fmt.Println("hello [", str, "]")
}
$ go run ./main.go
hello [ foo ]

TODO:

  • fix handling of integers (I did a poor job at making sure everything was ok)

  • add CPython unit-tests

  • do not expose C.FILE pointer and replace it with os.File in "go-python" API

  • provide an easy way to extend go-python with go based extensions

  • think about the need (or not) to translate CPython exceptions into go panic/recover mechanism

  • use SWIG to automatically wrap the whole CPython api ?

Comments
  • fatal error: unexpected signal during runtime execution

    fatal error: unexpected signal during runtime execution

    hi sbinet: While using go-python , occurred the error "fatal error: unexpected signal during runtime execution". The error looks like c programming error. I am using python 2.7 Can you have any suggestion?

    Thank you.

    bug question 
    opened by tangleiao 12
  • Pandas and go Python are incompatible?

    Pandas and go Python are incompatible?

    I introduced the pandas package into my Python function, and then used read_ csv function can be executed normally and r eturns 25. If you call this function through go, it will return - 1. Why?

    opened by dil1gent 9
  • object: always return the same value for Py_None, Py_True, and Py_False

    object: always return the same value for Py_None, Py_True, and Py_False

    https://docs.python.org/2/c-api/none.html

    You're supposed to be able to compare a python.PyObject to python.Py_None but this doesn't work.

    I think we should consider adding a PyNone_Check function to compare the underlying pointer.

    // Return true if self is None
    func PyNone_Check(self *PyObject) bool {
    	return Py_None.ptr == self.ptr
    }
    
    opened by icholy 8
  • asking about pkg-config problem

    asking about pkg-config problem

    I go get in window. It's come up with

    $ go get github.com/sbinet/go-python
    # pkg-config --cflags python-2.7
    Package python-2.7 was not found in the pkg-config search path.
    Perhaps you should add the directory containing `python-2.7.pc'
    to the PKG_CONFIG_PATH environment variable
    No package 'python-2.7' found
    pkg-config: exit status 1
    

    This not happen for me in OSX Now I cant even find .pc files in any directories. So I don't know what to add in PKG_CONFIG_PATH

    Sorry for asking stupid question I'm pretty new in window, python, pkg-config

    bug enhancement 
    opened by benzsuankularb 8
  • go-python does not execute a python module anymore if that module once raises an exception

    go-python does not execute a python module anymore if that module once raises an exception

    Having an issue where, we invoke a python code using go-python. If the python code raises some exception once, then onwards any call to that python module returns immediately from CallObject() with blank return value.

    Sample code to invoke python looks as below

    var pyinit = false
    func Init() (err error) {
        if !pyinit {
            if err = python.Initialize(); err == nil {
                pyinit = true
            }
        }
        return
    }
    
    type PyFunction struct {
        *python.PyObject
    }
    
    var mutex sync.Mutex
    func Import(module string, functions ...string) (funcs map[string]*PyFunction, err error) {
        if err = Init(); err != nil {
            return
        }
    
        if pymod := python.PyImport_ImportModuleNoBlock(module); pymod == nil {
            err = errors.New(fmt.Sprintf("gopy:%s: module import failed", module))
        } else {
            funcs = make(map[string]*PyFunction)
            for _, name := range functions {
                if pyfunc := pymod.GetAttrString(name); pyfunc == nil {
                    err = errors.New(fmt.Sprintf("gopy:%s:%s: function not found", module, name))
                    return
                } else {
                    funcs[name] = &PyFunction{pyfunc}
                }
            }
        }
        return
    }
    
    func (f *PyFunction) Call(args ...interface{}) {
        var pyargs *python.PyObject
    
        if pyargs, err = ToPyObject(reflect.ValueOf(args)); err != nil {
            return
        }
        mutex.Lock()
        defer mutex.Unlock()
        name := python.PyString_AsString(f.GetAttrString("__name__"))
        if r = f.CallObject(pyargs); r == nil {
            err = errors.New(fmt.Sprintf("%s(): function failed at python side", name))
        }
     }
    
    var funcNames = [...]string{
        "MyFunc",
    }
    
    var pyFuncs map[string]*PyFunction
    func main() {
            pyFuncs, err := gopy.Import("mypackage", funcNames[:]...)
            if err != nil {
            panic(err)
        }
           for {
               if pyFuncs["MyFunc"].Call(); err !=nil {
                   // handle the error
               }
          }
    }
    

    In above scenario say the python function mypackage.MyFunc raises an exception in some scenario. Then onwards, it never enters the python code at all and keeps showing the error "function failed at python side".

    Not sure if we should re-load the python interpreter from go code once exception or something else. Kindly suggest.

    Regards, Shubhendu

    opened by shtripat 7
  • Question: how to set the embedded Python to virtualenv Python ?

    Question: how to set the embedded Python to virtualenv Python ?

    Hi, I'm trying to write a Go executable that is able to pick up whichever Python version (hence libraries) it's running on. For example, in macOS the default system Python is 2.7.14, but could be executed within Python 3.6.5 virtualenv. As you know, working with virtualenv it's quite nice too, because the libraries installed within the env won't have to be globally installed.

    I have tried to prefix the PYTHON_HOME env variable with the VIRTUAL_ENV (Python bin) before python.Initialize(). However this seems to result in still picking the system Python 2.7.14. I can see the PATH env variable started with the correct VIRTUAL_ENV path, but the embedded Python once again is still picking up the system Python.

    One of the suggested workarounds on StackOverflow: 7492855 with (Python C API) was to set the program name via Py_SetProgramName. Although I don't seem to be able to find this function on go-python.

    Anything suggestions ? or, perhaps I'm missing something quite obvious.

    Thanks and regards,
    Wan.

    opened by sindbach 6
  • Linux RSS infintely increasing though go heap is empty

    Linux RSS infintely increasing though go heap is empty

    Hi,

    I'm not sure what exactly is happening here but the resident memory occupied by the golang process never seems to get free though there is nothing in the heap. Here is a sample code:

    package main
    
    import python "github.com/sbinet/go-python"
    
    func main() {
        initPython()
    
        for {
            //Infinite loop, make some data and create python dict
            message := createMessage()
            messageDict := getPyDict(message)
            messageDict.Clear()
        }
    }
    
    func createMessage() map[string]interface{} {
        payload := make(map[string]interface{})
        payload["key_1"] = "value_1"
        payload["key_2"] = "value_2"
        payload["key_3"] = "value_3"
        return payload
    }
    
    func getPyDict(message map[string]interface{}) *python.PyObject {
        messageDict := python.PyDict_New()
        for key, value := range message {
            pyKey := python.PyString_FromString(key)
            pyValue := python.PyString_FromString(value.(string))
            python.PyDict_SetItem(messageDict, pyKey, pyValue)
        }
    
        return messageDict
    } 
    
    func initPython() {
        //Init Python
        if err := python.Initialize(); err != nil {
              panic(err)
        }
    }
    

    The above script takes 1GB of RSS within a minute and the memory usage keeps increasing and doesn't seem to go down (golang heap is 0MB). Here are my local box details:

    uname -a

    Linux AE-LP-059 4.2.0-25-generic #30-Ubuntu SMP Mon Jan 18 12:31:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
    

    go version

    go version go1.6 linux/amd64
    

    go env

    GOARCH="amd64"
    GOBIN=""
    GOEXE=""
    GOHOSTARCH="amd64"
    GOHOSTOS="linux"
    GOOS="linux"
    GOPATH="/home/mpoornima/Work/go"
    GORACE=""
    GOROOT="/usr/local/go"
    GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
    GO15VENDOREXPERIMENT="1"
    CC="gcc"
    GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
    CXX="g++"
    CGO_ENABLED="1"
    
    opened by mpoornima 6
  • Make returns error

    Make returns error

    running make returns the following error:

    CGO_LDFLAGS="-L/usr/lib/python2.7 -lpython2.7" CGO_CFLAGS="-I/usr/include/python2.7 -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security" go install -compiler="gc" .
    go install: no install location for _/home/smith/go/pkg/github.com/sbinet/go-python
    make: *** [install] Error 1
    

    My GOPATH is /home/smith/go.

    opened by glittershark 6
  • go run error?

    go run error?

    fatal error: unexpected signal during runtime execution fatal error: unexpected signal during runtime execution [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x7f17cdca447b]

    runtime stack: runtime.throw(0x74c3c2, 0x2a) /usr/local/go/src/runtime/panic.go:616 +0x81 runtime.sigpanic() /usr/local/go/src/runtime/signal_unix.go:372 +0x28e

    goroutine 506 [syscall]: runtime.cgocall(0x68bad0, 0xc420384e68, 0x4) /usr/local/go/src/runtime/cgocall.go:128 +0x64 fp=0xc420384e38 sp=0xc420384e00 pc=0x40eb44 github.com/sbinet/go-python._Cfunc_PyImport_ImportModule(0x7f17bc0008c0, 0x0) _cgo_gotypes.go:3023 +0x4a fp=0xc420384e68 sp=0xc420384e38 pc=0x4f9dba github.com/sbinet/go-python.PyImport_ImportModule(0x73fc40, 0x4, 0x0)

    opened by casiphia 5
  • passing in NULL to go-python function?

    passing in NULL to go-python function?

    I'm trying to use PyList_SetSlice and according to the documentation, we can set itemlist to NULL. How would I do this? I tried C.NULL, nil, and even 0 but nothing seems to work.

    opened by reiinakano 5
  • panic error by import turbodbc lib

    panic error by import turbodbc lib

    System info: MacOs Sierra 10.12.6

    Error could not inport turbodbc

    package main
    
    import (
    	"fmt"
    
    	python "github.com/sbinet/go-python"
    )
    
    func init() {
    	err := python.Initialize()
    	if err != nil {
    		panic(err.Error())
    	}
    }
    
    func main() {
    	turboExasol := python.PyImport_ImportModule("turbodbc")
    	if turboExasol != nil {
    		panic("could not inport turbodbc")
    	}
    	fmt.Printf("%v", turboExasol)
    	connection := turboExasol.GetAttrString(`dsn=exasolution-io352fw_64`)
    	if connection != nil {
    		panic("could not get turbodbc(dsn=exasolution-io352fw_64)")
    	}
    	fmt.Printf(" %v", connection)
    }
    
    opened by cooljl31 5
  • Cannot pass Integer type to CallFunction

    Cannot pass Integer type to CallFunction

    the case is that when i use CallFunction to execute python module functions, and pass int type variable, it occur error like this:

    [[email protected] rootwd]# go build gptest/main.go
    [[email protected] rootwd]# ./main 
    panic: runtime error: cgo argument has Go pointer to Go pointer
    
    goroutine 1 [running]:
    github.com/sbinet/go-python.(*PyObject).CallFunction.func4(0xd0d510?, {0xc00004cf60?, 0x1, 0x0?}, 0xc00004ce48?, 0xc00004cef8)
            /usr/local/gopath/pkg/mod/github.com/sbinet/[email protected]/object.go:377 +0x89
    github.com/sbinet/go-python.(*PyObject).CallFunction(0x4ae0f3?, {0xc00004cf60?, 0x1, 0x1})
            /usr/local/gopath/pkg/mod/github.com/sbinet/[email protected]/object.go:377 +0x205
    main.main()
            /root/rootwd/gptest/main.go:21 +0x66
    

    here is my code:

    package main
    
    import (
    	"github.com/sbinet/go-python"
    )
    
    func init() {
    	err := python.Initialize()
    	if err != nil {
    		panic(err.Error())
    	}
    }
    
    func main() {
    	mod := python.PyImport_ImportModule("tpy.test")
    	fobj := mod.GetAttrString("test")
    	fobj.CallFunction("aaa") // ok
    	fobj.CallFunction(1)     // panic: runtime error: cgo argument has Go pointer to Go pointer
    }
    

    python module file:

    # tpy.test
    def test(data):
        print "xxxx", data
    

    I'm not sure if i use it in a wrong way.

    opened by hedongho 0
  • Exception SystemError: '/builddir/build/BUILD/Python-2.7.5/Objects/classobject.c:521: bad argument to internal function'

    Exception SystemError: '/builddir/build/BUILD/Python-2.7.5/Objects/classobject.c:521: bad argument to internal function'

    Hi, I have a problem and I need help !!!

    When used python.PyInstance_New to instance my python script Class, if Class inherit other Class, then have bad error

    Exception SystemError: '/builddir/build/BUILD/Python-2.7.5/Objects/classobject.c:521: bad argument to internal function'
    

    my golang programmer

    package main
    
    import (
    	"fmt"
    	"github.com/sbinet/go-python"
    	"os"
    )
    
    var PyStr = python.PyString_FromString
    
    func init() {
    	os.Setenv("PYTHONPATH", ".")
    	fmt.Println("PYTHONPATH:", os.Getenv("PYTHONPATH"))
    	err := python.Initialize()
    	if err != nil {
    		panic(err.Error())
    	}
    }
    
    func import_and_use_obj() {
    	// Import class from module
    	objsModule := python.PyImport_ImportModule("objs")
    	if objsModule == nil {
    		panic("Error importing module: objs")
    	}
    	fmt.Printf("[MODULE REF] repr(objsModule) = %s\n", python.PyString_AS_STRING(objsModule.Repr()))
    
    	obj := objsModule.GetAttrString("obj")
    	if obj == nil {
    		panic("[CLASS REF] Error importing object: obj")
    	}
    	fmt.Printf("[CLASS REF] repr(obj) = %s\n", python.PyString_AS_STRING(obj.Repr()))
    
    	// Instantiate obj
    	newObj := python.PyInstance_New(obj, python.PyTuple_New(0), nil)
    	if newObj == nil {
    		panic("[INSTANCE REF] Error instantiating object: obj")
    	}
    	fmt.Printf("[INSTANCE REF] repr(newObj) = %s\n", python.PyString_AS_STRING(newObj.Repr()))
    }
    
    func main() {
    	err := python.Initialize()
    	if err != nil {
    		panic(err.Error())
    	}
    	defer python.Finalize()
    	import_and_use_obj()
    	python.Finalize()
    }
    

    my python script

    #!/usr/bin/env python2
    import pickle
    import json
    
    class MyObj(object):
        def __init__(self):
            self.name = "ABC"
    
    
    class obj(MyObj):
        def __init__(self): pass
    
        def obj_method(self):
            print("Python object method called!")
            return 1
    
        def obj_method_arg(self, arg1):
            print("Python object method called, with argument: %s" % arg1)
            return 1
    
        def obj_method_args(self, arg1, arg2):
            print("Python object method called with arguments: %s" % arg1, arg2)
            return 1
    
        def obj_method_kwargs(self, **kwargs):
            print("Python object method called with arguments:" )
            print(kwargs)
            return 1
    

    Error

    [[email protected] demo]# go run application/demo1/main.go 
    PYTHONPATH: .
    [MODULE REF] repr(objsModule) = <module 'objs' from '/root/demo/objs.py'>
    [CLASS REF] repr(obj) = <class 'objs.obj'>
    Exception SystemError: '/builddir/build/BUILD/Python-2.7.5/Objects/classobject.c:521: bad argument to internal function' in <module 'threading' from '/usr/lib64/python2.7/threading.pyc'> ignored
    panic: [INSTANCE REF] Error instantiating object: obj
    
    goroutine 1 [running]:
    main.import_and_use_obj()
            /root/demo/application/demo1/main.go:37 +0x195
    main.main()
            /root/demo/application/demo1/main.go:48 +0x49
    exit status 2
    
    opened by anye88 0
  • Hi,

    Hi,

    Hi,

    Apologies for the belated answer.

    IIRC, what I meant with that to-do note was that I wasn't completely sure what I did for integers would work consistently between 32b and 64b platforms. I mapped both 'PyInt' and 'PyLong' to Go's 'int'. On a 64b machine, that should work. Probably not so much on a 32b one.

    In hindsight, I should have probably matched them to int32 and int64... (Probably something to do when/if I migrate this project to the go-python organization)

    Originally posted by @sbinet in https://github.com/sbinet/go-python/issues/70#issuecomment-410344914

    opened by popeye8777 0
  • any plan to support python3?

    any plan to support python3?

    I'm using an api interface(python) which requires at least python 3.6. I need this api for my project(I cannot use older version because i need some updated functions in current version). I changed this line: //#cgo pkg-config: python-2.7
    to //#cgo pkg-config: python-3.7 but go build reports a lot of errors. is there any plan to upgrade it to python3? this will keep this program popular/alive.

    opened by NickQian 1
  • How to import 3rd packages in the main python file

    How to import 3rd packages in the main python file

    I just found there is no any 3rd packages were imported in your test examples. I tried to import 3rd package, import numpy in a py file, then go run main.go, will report a failed import. Please tell me how to import other 3rd packages installed in anaconda2/lib/python2.7/site-packages/

    thanks

    opened by camelhehe 2
Owner
Sebastien Binet
Sebastien Binet
PHP bindings for the Go programming language (Golang)

PHP bindings for Go This package implements support for executing PHP scripts, exporting Go variables for use in PHP contexts, attaching Go method rec

Alex Palaistras 878 Nov 22, 2022
Go bindings to QuickJS: a fast, small, and embeddable ES2020 JavaScript interpreter.

quickjs Go bindings to QuickJS: a fast, small, and embeddable ES2020 JavaScript interpreter. These bindings are a WIP and do not match full parity wit

Kenta Iwasaki 135 Nov 30, 2022
A Go API for the V8 javascript engine.

V8 Bindings for Go The v8 bindings allow a user to execute javascript from within a go executable. The bindings are tested to work with several recent

Augusto Roman 381 Nov 13, 2022
naive go bindings to the CPython C-API

go-python Naive go bindings towards the C-API of CPython-2. this package provides a go package named "python" under which most of the PyXYZ functions

Sebastien Binet 1.4k Dec 2, 2022
naive go bindings to GnuPlot

go-gnuplot Simple-minded functions to work with gnuplot. go-gnuplot runs gnuplot as a subprocess and pushes commands via the STDIN of that subprocess.

Sebastien Binet 26 Nov 8, 2021
Naive Bayesian Classification for Golang.

Naive Bayesian Classification Perform naive Bayesian classification into an arbitrary number of classes on sets of strings. bayesian also supports ter

Jake Brukhman 743 Nov 22, 2022
Naive Bayesian Classification for Golang.

Naive Bayesian Classification Perform naive Bayesian classification into an arbitrary number of classes on sets of strings. bayesian also supports ter

Jake Brukhman 743 Nov 22, 2022
A Naive Bayes SMS spam classifier written in Go.

Ham (SMS spam classifier) Summary The purpose of this project is to demonstrate a simple probabilistic SMS spam classifier in Go. This supervised lear

Dan Wolf 13 Sep 9, 2022
Naive Bayes spam-filtering in Go

Naive Bayes Spam-Filtering Spam is a simple implementation of naive Bayes spam-filtering algorithm. Resources youtube - live coding(farsi). License Th

Mobocrat 4 Nov 20, 2021
Naive LEGO helper for SberCloud DNS to be used with the EXEC plugin

Naive LEGO helper for SberCloud DNS Very basic, no any checks performed To be used with the exec plugin as described here Environment variables SBC_AC

null 0 Nov 3, 2021
A naive implementation of Raft consensus algorithm.

This implementation is used to learn/understand the Raft consensus algorithm. The code implements the behaviors shown in Figure 2 of the Raft paper wi

Martin 0 Dec 3, 2021
Paxoskv: a Naive and Basic paxos kv storage

paxoskv: a Naive and Basic paxos kv storage 这个repo 目前仅是用于学习的实例代码. 这是一个基于paxos, 只有200行代码的kv存储系统的简单实现, 以最简洁的形式展示paxos如何运行, 作为 可靠分布式系统-paxos的直观解释 这篇教程中的代

null 0 Nov 29, 2021
A naive and simple implementation of blockchains.

naivechain A naive and simple implementation of blockchains. Build And Run Download and compile go get -v github.com/kofj/naivechain Start First Node

疯魔慕薇 324 Dec 5, 2022
Gocfg - A naive and simple cfg parser that uses maps internally done in Go

gocfg A simple ini-like parser based on maps. Key iteration can be done using th

Lucas Eduardo 2 Sep 13, 2022
Go bindings for OpenCV / 2.x API in gocv / 1.x API in opencv

Go OpenCV binding A Golang binding for OpenCV. OpenCV 1.x C API bindings through CGO, and OpenCV 2+ C++ API (GoCV) through SWIG. Disclaimer This is a

go-opencv 1.3k Dec 1, 2022
Golang bindings for the Telegram Bot API

Golang bindings for the Telegram Bot API All methods are fairly self explanatory, and reading the godoc page should explain everything. If something i

null 4.2k Dec 3, 2022
Go bindings for Lua C API - in progress

Go Bindings for the lua C API Simplest way to install: # go get github.com/aarzilli/golua/lua You can then try to run the examples: $ cd golua/_examp

Alessandro Arzilli 601 Nov 21, 2022
Cgo bindings to PulseAudio's Simple API, for easily playing or capturing raw audio.

pulse-simple Cgo bindings to PulseAudio's Simple API, for easily playing or capturing raw audio. The full Simple API is supported, including channel m

Tommy 23 Nov 8, 2022
Vulkan API bindings for Go programming language

Golang Bindings for Vulkan API Package vulkan provides Go bindings for Vulkan — a low-overhead, cross-platform 3D graphics and compute API. Updated Oc

null 656 Dec 1, 2022
Go bindings for Lua C API - in progress

Go Bindings for the lua C API Simplest way to install: # go get github.com/aarzilli/golua/lua You can then try to run the examples: $ cd golua/_examp

Alessandro Arzilli 601 Nov 21, 2022