Tight bidirectional integration between Go and Python

Related tags

Network python go cgo
Overview

py4go

License Go Report Card

Call Python 3 functions and methods from within your Go program while exposing Go functions and methods to Python.

This is not an implementation of Python in Go. Rather, py4go works by embedding the CPython runtime into your Go program using cgo functionality.

The expected use cases are not low-latency integration, but rather tight bidirectional integration. You can combine the full Go ecosystem with the full Python ecosystem.

Though you can achieve some integration by using Go's exec package to run python, with py4go you get fine-grained access to individual functions, objects, methods, and variables.

To get started try running scripts/example. Note that you need the Python development libraries installed. E.g. in Fedora:

sudo dnf install python3-devel

Example Usage

package main

import (
    "fmt"
    "github.com/tliron/py4go"
)

func main() {
    // Initialize Python
    python.Initialize()
    defer python.Finalize()

    // Import Python code (foo.py)
    foo, _ := python.Import("foo")
    defer foo.Release()

    // Get access to a Python function
    hello, _ := foo.GetAttr("hello")
    defer hello.Release()

    // Call the function with arguments
    r, _ := hello.Call("myargument")
    defer r.Release()
    fmt.Printf("Returned: %s\n", r.String())

    // Expose a Go function to Python via a C wrapper
    // (Just use "import api" from Python)
    api, _ := python.CreateModule("api")
    defer api.Release()
    api.AddModuleCFunctionNoArgs("my_function", C.api_my_function)
    api.EnableModule()
}

Calling Python code from Go is relatively straightforward because Python is a dynamic language and CPython is an interpreted runtime. Exposing Go code to Python is more involved as it requires writing wrapper functions in C, which we omitted in the example above. See the examples directory for more detail.

In the future we are hoping to make this easier, perhaps via a C code generator based on static analysis of function signatures and types.

Caveats

There are several issues to be aware of:

  • It's more difficult to distribute your Go program because you must have the CPython library available on the target operating system with a specific name. Because different operating systems have their own conventions for naming this library, to create a truly portable distribution it may be best to distribute your program as a packaged container, e.g. using Flatpak, Docker, or Snap.
  • It is similarly more difficult to build your Go program. We are using pkg-config: python3-embed to locate the CPython SDK, which works on Fedora-based operating systems. But, because where you build will determine the requirements for where you will run, it may be best to build on Fedora, either directly or in a virtual machine or container. Unfortunately cgo does not let us parameterize that pkg-config directive, thus you will have to modify our source files in order to build on/for other operating systems.
  • Calling functions and passing data between these two high-level language's runtime environments obviously incurs some overhead. Notably strings are sometimes copied multiple times internally, and may be encoded and decoded (Go normally uses UTF-8, Python defaults to UCS4). If you are frequently calling back and forth be aware of possible performance degradation. As always, if you experience a problem measure first and identify the bottleneck before prematurely optimizing!
  • Similarly, be aware that you are simultaneously running two memory management runtimes, each with its own heap allocation and garbage collection threads, and that Go is unaware of Python's. Your Go code will thus need to explicitly call Release on all Python references to ensure that they are garbage collected. Luckily, the defer keyword makes this easy enough in many circumstances.
  • Concurrency is a bit tricky in Python due to its infamous Global Interpreter Lock (GIL). If you are calling Python code from a Goroutine make sure to call python.SaveThreadState and python.EnsureGilState as appropriate. See the examples for more detail.

References

  • go-python is a similar and more mature project for Python 2.
  • goPy is a much older project for Python 2.
  • gopy generates Python wrappers for Go functions.
  • setuptools-golang allows you to include Go libraries in Python packages.
Owner
Tal Liron
Tal Liron
Partial port of Python difflib package to Go

go-difflib THIS PACKAGE IS NO LONGER MAINTAINED. At this point, I have no longer the time nor the interest to work on go-difflib. I apologize for the

Patrick M├ęzard 290 Jan 14, 2022
Python's webbrowser in Go. Simple API for opening web pages on your default browser.

webbrowser webbrowser provides a simple API for opening web pages on your default browser. It's inspired on Python's webbrowser package but lacks some

Carlos Cobo 170 Jan 7, 2022
Fastest python library for making asynchronous group requests.

FGrequests: Fastest Asynchronous Group Requests Installation Install using pip: pip install fgrequests Documentation Pretty easy to use. import fgrequ

Farid Chowdhury 12 Dec 9, 2021
A python comtrade load library accelerated by go

Comtrade-GRPC Code for python used is mainly from dparrini/python-comtrade. Just

Bo 1 Dec 27, 2021
Simple Relay between a Unix socket and a TCP socket, and vice versa.

Simple TCP <-> Unix Relay simpletcpunixrelay is a program which exposes a TCP endpoint as a Unix socket and vice versa. Usecase Let's say you are runn

Antoine Catton 0 Dec 4, 2021
Our aim is to expand the capabilities of blockchain and make a secure way for transferring NFT between RMRK and MOVR blockchain.

remov Inspiration Our aim is to expand the capabilities of blockchain and make a secure way for transferring NFT between RMRK and MOVR blockchain. The

RMRK Team 4 Jan 10, 2022
Make TCP connection storm between server and client for benchmarking network stuff

Make TCP connection storm between server and client for benchmarking network stuff

Masahiro Nagano 2 Nov 14, 2021
Bridge between mattermost, IRC, gitter, xmpp, slack, discord, telegram, rocketchat, twitch, ssh-chat, zulip, whatsapp, keybase, matrix, microsoft teams, nextcloud, mumble, vk and more with REST API

bridge between mattermost, IRC, gitter, xmpp, slack, discord, telegram, rocketchat, twitch, ssh-chat, zulip, whatsapp, keybase, matrix, microsoft teams, nextcloud, mumble, vk and more with REST API (mattermost not required!)

Wim 4.5k Jan 17, 2022
Powered by Matterbridge, MatterAMXX is a plugin for AMXX that allows simple bridging between your game servers, Mattermost, IRC, XMPP, Gitter, Slack, Discord, Telegram, and more.

Powered by Matterbridge, MatterAMXX is a plugin for AMXX that allows simple bridging between your game servers, Mattermost, IRC, XMPP, Gitter, Slack, Discord, Telegram, and more.

Gabriel Iggy N. 7 Nov 6, 2021
A REST API for the DN42 registry, written in Go, to provide a bridge between interactive applications and the registry.

dn42regsrv A REST API for the DN42 registry, written in Go, to provide a bridge between interactive applications and registry data. A public instance

Simon Marsh 0 Dec 2, 2021
Plugs module to see different types of plug types needed in different countries, and a comparison tool between two countries plug socket types.

plugs Importing the module: go get github.com/matthewboyd/plugs "github.com/matthewboyd/plugs" How to use the module: There are two functions wi

Matthew Boyd 2 Dec 28, 2021
RuuviBridge - Utility to bridge RuuviTag data between various sources and consumers

RuuviBridge RuuviBridge is designed to act as a "data bridge" between various so

Scrin 4 Jan 9, 2022
Transfer files between machines in the same network

GoTrans Transfer files from computer A to computer B. The program have a sender and a receiver, if you choose a sender you'll need to provide the ip a

Danilo Marques 0 Nov 7, 2021
Code to be shared between EduVPN clients

EduVPN shared library This repository contains a Go library with functions that all EduVPN clients can use. The goal is to let EduVPN clients link aga

null 0 Dec 17, 2021
A document scanner that creates a graph of the analogy between documents.

Social Analogizer The idea is that this program scans documents for keyword that correspond to other documents. Starting point is that a user has a pr

Tibart 0 Dec 20, 2021
Socketlogger allows multiple processes to log to a shared log file between processes.

socketlogger socketlogger is a language agnostic logging utility designed to allow mulitple applications, running on one or multiple computers, to wri

Ryan Johnson 1 Dec 7, 2021
An anonymous, encrypted Point-to-Point (Layer 3) tunnel between two peers.

NKN-Link An anonymous, encrypted Point-to-Point (Layer 3) tunnel between two peers. NKN-Link Table of Contents Preface Description Install Setup Run P

HAH! Sun 4 Dec 29, 2021
The ultimate way to move between folders in the CLI

goto-command The ultimate way to move between folders in the command line Goto is a command that can be used like cd, and also allows you to add speci

null 0 Jan 5, 2022
Go rate limiter used to ensure a minimum duration between executions.

Ratelimiter Rate limiter used to ensure a minimum duration between executions. Additionally supports the optional limit of max queue size. This can be

Branden 0 Jan 15, 2022