Go language interface to the Libcircle distributed-queue API

Related tags

circle
Overview

Circle

Description

The Circle package provides a Go interface to the Libcircle distributed-queue API. Despite the name, Circle has nothing to do with graphics. Instead, Circle provides a mechanism for enqueueing "work" (currently, text strings) on a distributed queue then letting numerous processes distributed across a local-area network dequeue and process that work.

Use Circle when you have a huge number of independent tasks to perform and want an easy way to distribute these across a large cluster or supercomputer.

Features

The underlying Libcircle library has the following features:

  • proximity-aware, work-stealing scheduler

  • used daily on production supercomputers at Los Alamos National Laboratory to perform various maintenance activities across a multi-petabyte parallel filesystem

  • fast—communication is implemented with user-level messaging (specifically, MPI), not kernel-level sockets.

Circle provides a Go interface to Libcircle:

  • a low-level API that maps directly to the Libcircle API but supports all of the Go niceties such as using Go strings for work items and Go functions for Libcircle callbacks

  • a higher-level API that forgoes Libcircle's callback mechanism in favor Go channels: one for enqueueing work and one for dequeueing work

Installation

You'll need to download and install Libcircle, which is available from https://github.com/hpc/libcircle. After that,

go get github.com/lanl/circle

ought to work.

Documentation

Pre-built documentation for the Circle API is available online at http://godoc.org/github.com/lanl/circle, courtesy of GoDoc.

Once you install Circle, you can view the API locally with godoc, for example by running

godoc github.com/lanl/circle

to display the Goop documentation on screen or by running

cd $GOPATH
godoc -http=:6060

to start a local Web server then viewing the HTML-formatted documentation at http://localhost:6060/pkg/github.com/lanl/circle/ in your favorite browser.

License

Circle is provided under a BSD-ish license with a "modifications must be indicated" clause. See the LICENSE file for the full text.

Circle is part of the LANL Go Suite, LA-CC-11-056.

Author

Scott Pakin, [email protected]

Owner
Los Alamos National Laboratory
Los Alamos National Laboratory
Golang client for NATS, the cloud native messaging system.

NATS - Go Client A Go client for the NATS messaging system. Installation # Go client go get github.com/nats-io/nats.go/ # Server go get github.com/na

NATS - The Cloud Native Messaging System 3.4k Jun 13, 2021
A pluggable backend API that enforces the Event Sourcing Pattern for persisting & broadcasting application state changes

A pluggable "Application State Gateway" that enforces the Event Sourcing Pattern for securely persisting & broadcasting application state changes

null 24 Apr 15, 2021
HTTP rate limiting module for Caddy 2

This module implements both internal and distributed HTTP rate limiting. Requests can be rejected after a specified rate limit is hit.

Matt Holt 42 Jun 10, 2021
A Lightweight VPN Built on top of Libp2p for Truly Distributed Networks.

Hyprspace A Lightweight VPN Built on top of Libp2p for Truly Distributed Networks. demo.mp4 Table of Contents A Bit of Backstory Use Cases A Digital N

Hyprspace 84 Jun 13, 2021
A persistent queue implemented in Go.

goq goq is a persistent queue implemented in Go. Features Communication over HTTP RESTful with JSON responses (trivial to write client libraries) Mini

Kunal Anand 31 May 23, 2021
A Crypto-Secure, Production-Grade Reliable-UDP Library for golang with FEC

Introduction kcp-go is a Production-Grade Reliable-UDP library for golang. This library intents to provide a smooth, resilient, ordered, error-checked

xtaci 3k Jun 13, 2021
NFF-Go -Network Function Framework for GO (former YANFF)

Network Function Framework for Go (former YANFF) Wonderful news : we are now supporting AF_XDP and supporting(almost) getting packets directly from Li

Intel Go Team 1.1k Jun 13, 2021
secure scuttlebutt network simulator

Network Simulator a brief exposition on the network simulator to be built as part of SSB's NGI Pointer grant For the remainder of May—and likely parts

null 12 Jun 4, 2021
webrpc is a schema-driven approach to writing backend services for modern Web apps and networks

webrpc is a schema-driven approach to writing backend servers for the Web. Write your server's api interface in a schema format of RIDL or JSON, and t

null 368 May 29, 2021
A Stable & Secure Tunnel based on KCP with N:M multiplexing and FEC. Available for ARM, MIPS, 386 and AMD64。KCPプロトコルに基づく安全なトンネル。KCP 프로토콜을 기반으로 하는 보안 터널입니다。

Disclaimer: kcptun maintains a single website — github.com/xtaci/kcptun. Any websites other than github.com/xtaci/kcptun are not endorsed by xtaci. Re

xtaci 12.5k Jun 14, 2021
Super fault-tolerant gateway for HTTP clusters, written in Go. White paper for reference - https://github.com/gptankit/serviceq-paper

ServiceQ ServiceQ is a fault-tolerant gateway for HTTP clusters. It employs probabilistic routing to distribute load during partial cluster shutdown (

Ankit Gupta 49 May 20, 2021
Distributed RTC System by pure Go and Flutter

ION is a distributed real-time communication system, the goal is to chat anydevice, anytime, anywhere! Distributed Real-time Communication System ION

Pion 3k Jun 11, 2021
IPFS implementation in Go

go-ipfs What is IPFS? IPFS is a global, versioned, peer-to-peer filesystem. It combines good ideas from previous systems such as Git, BitTorrent, Kade

IPFS 11.5k Jun 12, 2021