Reinforcement Learning in Go

Overview

logo
cartpole
GoDoc Go Report Card

Overview

Gold is a reinforcement learning library for Go. It provides a set of agents that can be used to solve challenges in various environments. The library further contains a composable tooling for creating agents and visualizing their performance.

Quick Start

go run ./pkg/v1/agent/deepq/experiments/cartpole/main.go

Requirements

  • Go >= v13.0
  • Docker
  • A browser that isn't IE

Agents

All of the agent implementations can be found in pkg/v1/agent each agent has an experiments folder providing demos across various environments.

agent algorithm
pkg/v1/agent/q Q Learning
pkg/v1/agent/deepq Deep Q learning with Double Q
pkg/v1/agent/reinforce REINFORCE aka Monte Carlo Policy Gradients
pkg/v1/agent/nes Natural Evolution Strategies
pkg/v1/agent/her Hindsight Experience Replay
pkg/v1/agent/ppo Proximal Policy Optimization ⚠️ in progress

Docs

Each package contains a README explaining the usage, also see GoDoc.

Contributing

Please open an MR for any issues or feature requests.

The project is fairly simple to hack on, environments are provided through Sphere, models are provided through Goro.

Feel free to ping @pbarker on Gopher slack.

Roadmap

  • More agents, more environments; see Future Thoughts
  • Accelerated compute support
  • Tuning libraries
  • Deployments

Inspiration

Comments
  • Taxi q experiment panics

    Taxi q experiment panics

    Using go1.14.1 with https://github.com/aunum/gold/commit/8dedf68d0993b8fb4355edfc560c1567ea9fb80c I get a panic in taxi experiment:

    go run -trimpath ./pkg/v1/agent/q/experiments/taxi/main.go
    ℹ  tracking data in /var/folders/zy/lzyp__pd7vx6762s5jcgkkl80000gn/T/stats.104998610.json
    ℹ  connecting to existing container "stupefied_taussig"
    ✔  connected to server "gym"
    ✔  created env: f07159db-890d-45ca-a295-aa9e383507de
    ℹ  serving agent api/ui on http://localhost:39421
    ℹ  running for 5000 episodes
    ✖  rpc error: code = Unknown desc = Exception calling application: 'int' object has no attribute 'ravel'
    goroutine 1 [running]:
    runtime/debug.Stack(0x75, 0x0, 0x0)
            runtime/debug/stack.go:24 +0x9d
    runtime/debug.PrintStack()
            runtime/debug/stack.go:16 +0x22
    github.com/aunum/log.fatalf(0xc0007e0100, 0x1c76e46, 0x3, 0xc0007e1f28, 0x1, 0x1)
            github.com/aunum/[email protected]/log.go:77 +0x11d
    github.com/aunum/log.Fatalf(...)
            github.com/aunum/[email protected]/log.go:64
    github.com/aunum/log.Fatal(0xc0007e1f28, 0x1, 0x1)
            github.com/aunum/[email protected]/log.go:109 +0xa3
    github.com/aunum/gold/pkg/v1/common/require.NoError(...)
            github.com/aunum/gold/pkg/v1/common/require/require.go:18
    main.main()
            command-line-arguments/main.go:36 +0x303
    exit status 1
    
    opened by pmalek 15
  • Cannot convert FauxZero

    Cannot convert FauxZero

    Cloned repository and using the quick start instructions.

    $ go run ./pkg/v1/agent/deepq/experiments/cartpole/main.go # github.com/aunum/gold/pkg/v1/dense pkg/v1/dense/value.go:57:19: cannot convert FauxZero (type float64) to type complex64 pkg/v1/dense/value.go:59:20: cannot convert FauxZero (type float64) to type complex128

    Using go v1.14 on linux/amd64

    bug 
    opened by JetSetIlly 9
  • Exception calling application: module 'gym.envs.box2d' has no attribute 'LunarLander'

    Exception calling application: module 'gym.envs.box2d' has no attribute 'LunarLander'

    It seems that the only working environment is CartPole-v1. If I try to use LunarLander-v2 or something else from Box2D, I get an error, for example:

    	server, err := env.FindOrCreate(env.GymServerConfig)
    	if err != nil {
    		log.Fatal(err)
    	}
    	defer server.Close()
    
    	environment, err = server.Make("LunarLander-v2", env.WithRecorder())
    	if err != nil {
    		log.Fatal(err)
    	}
    

    Error:

    2020/04/10 18:15:13 rpc error: code = Unknown desc = Exception calling application: module 'gym.envs.box2d' has no attribute 'LunarLander'
    

    I wonder whether it's expected behavior or not? Should I perform some extra steps to enable additional environments?

    opened by dtitov 3
  • constant conversion fix

    constant conversion fix

    @JetSetIlly reported https://github.com/aunum/gold/issues/2 and discovered it was caused by https://go-review.googlesource.com/c/go/+/226197/ and recommended the following fix

    opened by pbarker 0
  • 1.14 hot fix

    1.14 hot fix

    A breaking change occurred in Go between 13.6 and 14.1 with float64 to complex64 casting, still need to investigate more closes https://github.com/aunum/gold/issues/2

    opened by pbarker 0
  • Environment Step: float or int?

    Environment Step: float or int?

    Hi,

    I'm trying to create my own agent, following the example here: https://github.com/aunum/gold/tree/master/pkg/v1/agent/reinforce It's an example for CartPole-v0, while I'm playing around with LunarLander-v2.

    I've come now to implementing the "step" function. And I realized that Env.Step() function takes int as an argument. Maybe that's what is needed for the CartPole-v0, but LunarLander-v2's step is float: https://github.com/openai/gym/blob/master/gym/envs/box2d/lunar_lander.py#L111

    # Action is two floats [main engine, left-right engines].
    # Main engine: -1..0 off, 0..+1 throttle from 50% to 100% power. Engine can't work with less than 50% power.
    # Left-right:  -1.0..-0.5 fire left engine, +0.5..+1.0 fire right engine, -0.5..0.5 off
    

    Pretty much everywhere in the gold library where I meet anything action-related - it's int.

    How should it be mapped to the continuous environments? Or is it not possible at the moment? If so, would it be difficult to implement it?

    opened by dtitov 6
  • Error with build

    Error with build

    C:\Users\<...>\go\pkg\mod\github.com\aunum\[email protected]\pkg\v1\agent\q\experiments\taxi>go run main.go
    # github.com/ory/dockertest/docker/pkg/system
    ..\..\..\..\..\..\..\..\ory\[email protected]+incompatible\docker\pkg\system\filesys_windows.go:113:24: cannot use uintptr(unsafe.Pointer(&sd[0])) (type uintptr) as type *"golang.org/x/sys/windows".SECURITY_DESCRIPTOR in assignment
    

    go version go1.16.6 windows/amd64

    opened by ogau 1
  • Not all videos are being saved...

    Not all videos are being saved...

    Hi,

    I'm running a lot of episodes of LunarLander in a loop (maybe hundreds). When I call environment.End() in the end, I get all the episodes information printed, like this:

    ...
    
      "15":
        episode_id: 15
        episode_length: 80
        reward: 20.416142
        timestamp:
          seconds: 1588666897
    
    ...
    

    But, for some reason, it records only a few videos:

    Videos:
      "0":
        content_type: video/mp4
      "1":
        content_type: video/mp4
        episode_id: 1
      "8":
        content_type: video/mp4
        episode_id: 8
      "27":
        content_type: video/mp4
        episode_id: 27
    

    I'm having a hard time trying to find the parameter that can turn on saving video for each and every episode. Is it even possible?

    opened by dtitov 2
  • visualize toy text problems

    visualize toy text problems

    Currently toy text problems don't have visualization because the only way they are rendered is with ansi color codes. Should provide a means of rendering that locally in the terminal.

    enhancement 
    opened by pbarker 0
  • build pipelines

    build pipelines

    https://github.com/aunum/gold/issues/5 was caused by a local build of the sphere container that hadn't been pushed and highlighted the need for build pipelines

    enhancement 
    opened by pbarker 0
Owner
AUNUM
AUNUM
On-line Machine Learning in Go (and so much more)

goml Golang Machine Learning, On The Wire goml is a machine learning library written entirely in Golang which lets the average developer include machi

Conner DiPaolo 1.4k Nov 23, 2022
Gorgonia is a library that helps facilitate machine learning in Go.

Gorgonia is a library that helps facilitate machine learning in Go. Write and evaluate mathematical equations involving multidimensional arrays easily

Gorgonia 4.7k Nov 19, 2022
Machine Learning libraries for Go Lang - Linear regression, Logistic regression, etc.

package ml - Machine Learning Libraries ###import "github.com/alonsovidales/go_ml" Package ml provides some implementations of usefull machine learnin

Alonso Vidales 196 Nov 10, 2022
Gorgonia is a library that helps facilitate machine learning in Go.

Gorgonia is a library that helps facilitate machine learning in Go. Write and evaluate mathematical equations involving multidimensional arrays easily

Gorgonia 4.7k Nov 23, 2022
Prophecis is a one-stop machine learning platform developed by WeBank

Prophecis is a one-stop machine learning platform developed by WeBank. It integrates multiple open-source machine learning frameworks, has the multi tenant management capability of machine learning compute cluster, and provides full stack container deployment and management services for production environment.

WeBankFinTech 386 Nov 19, 2022
Go Machine Learning Benchmarks

Benchmarks of machine learning inference for Go

Nikolay Dubina 24 Sep 27, 2022
Deploy, manage, and scale machine learning models in production

Deploy, manage, and scale machine learning models in production. Cortex is a cloud native model serving platform for machine learning engineering teams.

Cortex Labs 7.8k Nov 23, 2022
A High-level Machine Learning Library for Go

Overview Goro is a high-level machine learning library for Go built on Gorgonia. It aims to have the same feel as Keras. Usage import ( . "github.

AUNUM 351 Nov 20, 2022
Standard machine learning models

Cog: Standard machine learning models Define your models in a standard format, store them in a central place, run them anywhere. Standard interface fo

Replicate 3.4k Nov 28, 2022
Spice.ai is an open source, portable runtime for training and using deep learning on time series data.

Spice.ai Spice.ai is an open source, portable runtime for training and using deep learning on time series data. ⚠️ DEVELOPER PREVIEW ONLY Spice.ai is

Spice.ai 772 Nov 25, 2022
Katib is a Kubernetes-native project for automated machine learning (AutoML).

Katib is a Kubernetes-native project for automated machine learning (AutoML). Katib supports Hyperparameter Tuning, Early Stopping and Neural Architec

Kubeflow 1.3k Nov 22, 2022
FlyML perfomant real time mashine learning libraryes in Go

FlyML perfomant real time mashine learning libraryes in Go simple & perfomant logistic regression (~100 LoC) Status: WIP! Validated on mushrooms datas

Vadim Kulibaba 1 May 30, 2022
PaddleDTX is a solution that focused on distributed machine learning technology based on decentralized storage.

中文 | English PaddleDTX PaddleDTX is a solution that focused on distributed machine learning technology based on decentralized storage. It solves the d

null 82 Nov 18, 2022
Self-contained Machine Learning and Natural Language Processing library in Go

Self-contained Machine Learning and Natural Language Processing library in Go

NLP Odyssey 1.3k Nov 17, 2022
Go (Golang) encrypted deep learning library; Fully homomorphic encryption over neural network graphs

DC DarkLantern A lantern is a portable case that protects light, A dark lantern is one who's light can be hidden at will. DC DarkLantern is a golang i

Raven 2 Oct 31, 2022
This is the repository for the LinkedIn Learning course Learning Go.

Learning Go This is the repository for the LinkedIn Learning course Learning Go. The full course is available from LinkedIn Learning. What is Go? Go i

Zhenguan Tang 0 Nov 2, 2021
This repository is where I'm learning to write a CLI using Go, while learning Go, and experimenting with Docker containers and APIs.

CLI Project This repository contains a CLI project that I've been working on for a while. It's a simple project that I've been utilizing to learn Go,

Tamir Arnesty 0 Dec 12, 2021
Selected Machine Learning algorithms for natural language processing and semantic analysis in Golang

Natural Language Processing Implementations of selected machine learning algorithms for natural language processing in golang. The primary focus for t

James Bowman 381 Nov 20, 2022
Machine Learning for Go

GoLearn GoLearn is a 'batteries included' machine learning library for Go. Simplicity, paired with customisability, is the goal. We are in active deve

Stephen Whitworth 8.6k Nov 27, 2022
On-line Machine Learning in Go (and so much more)

goml Golang Machine Learning, On The Wire goml is a machine learning library written entirely in Golang which lets the average developer include machi

Conner DiPaolo 1.4k Nov 23, 2022