Scalable, fault-tolerant application-layer sharding for Go applications

Overview

ringpop-go Build Status Coverage Status

(This project is no longer under active development.)

Ringpop is a library that brings cooperation and coordination to distributed applications. It maintains a consistent hash ring on top of a membership protocol and provides request forwarding as a routing convenience. It can be used to shard your application in a way that's scalable and fault tolerant.

Getting started

To install ringpop-go:

go get github.com/uber/ringpop-go

Developing

First make certain that thrift (OSX: brew install thrift) and glide are in your path (above). Then,

make setup

to install remaining golang dependencies and install the pre-commit hook.

Finally, run the tests by doing:

make test

Documentation

Interested in where to go from here? Read the docs at ringpop.readthedocs.org

Comments
  • Feature: Labels

    Feature: Labels

    This PR will add a possibility to add labels to a node in ringpop. Labels set on a node will be gossiped around the network in the same fashion as updates to the status of a node do and therefore will converge so all members of the membership will have the same labels for all nodes during normal operation where the membership is converged.

    Labels can be used to annotate nodes with extra information. In the examples an application is provided where a role for a node is set on its labels. Other members of the membership can then query ringpop for all nodes that fulfill a specific role.

    • [x] Add test to make sure setting private labels from public interface returns error (from: #172)
    • [x] Revisit locking to prevent races
    opened by thanodnl 31
  • Move ringpop-go to glide

    Move ringpop-go to glide

    • Works on a local OSX machine with a fresh $GOPATH.
    • Works on travis-ci.
    • Does not work on a fresh Docker golang:1.6 instance due to possibly different thrift version (not verified). However, the dev branch doesn't work too, so it's a separate problem.
    opened by motiejus 24
  • [Labels] Implement internal labels

    [Labels] Implement internal labels

    To make sure that applications can't interfere with ringpop internals on labels we reserve all labels prefixed with __ (double lodash) for ringpop. Applications will receive an error when they try to set a label that starts with this prefix.

    opened by thanodnl 20
  • Add proxy endpoints for ringpop and basic proxy support.

    Add proxy endpoints for ringpop and basic proxy support.

    This patch just adds very basic proxy functionality (I am actually in the process of writing some integration test which Jeff pointed so that I can actually validate the handling of the proxy request, etc.) I am just putting it out there for others to see.

    TODO:

    • Add more tests to properly test this functionality
    • Fixup the proxy request handling after getting an actual proxy going
    • Increase code coverage (go test --cover gives only 74.2% now)
    opened by aravindvs 19
  • [Labels] Implement sane limits on Labels.

    [Labels] Implement sane limits on Labels.

    Since labels are gossiped around for all members it can cause gossip bloat when used irresponsibly. This will add enforceable limits on labels to prevent the such bloat from happening. Labels will be ignored when the application tries to set a label or multiple labels that would violate the allowed budget for the labels.

    opened by thanodnl 15
  • Status refactor 1: Refactor membership to use localMember as the source of truth

    Status refactor 1: Refactor membership to use localMember as the source of truth

    With this refactor we keep an explicit copy of what the local state of a member is and reincarnate from there if we receive a gossip that would overwrite the state of our node in the network.

    It is analogous to https://github.com/uber/ringpop-node/pull/291

    opened by thanodnl 13
  • Test examples/ping-*/README.md

    Test examples/ping-*/README.md

    Using cram to execute and verify instructions in the following files:

    • examples/ping-json/README.md
    • examples/ping-thrift-gen/README.md
    • examples/ping-thrift/README.md

    Currently, all of the tests are failing due to incomplete instructions in the README, but opening the pull request anyway to set the path forward to fixing them.

    opened by motiejus 13
  • Emit lookup stat correctly

    Emit lookup stat correctly

    Previously the lookup stat was emitted from the event-handler inside of ringpop.go when a LookupEvent was emitted. However, the lookup-event is emitted on ringpop itself so it's internal event-handler isn't triggered.

    opened by mennopruijssers 11
  • Update testpop to emit stats to network or file

    Update testpop to emit stats to network or file

    In preparation for automated perf. tests we want to be able to emit stats to a file/network from testpop.

    This PR adds a new flag to testpop to control where the stats should go; if the flag is omitted, no stats are emitted.

    opened by severb 11
  • Set a better default for maxP and fix maxP value after bootstrap.

    Set a better default for maxP and fix maxP value after bootstrap.

    Two things going on:

    • maxP should never be lower than 15, considering the math used to adjust it.
    • after bootstrap, the initial value for maxP doesn't reflect the ring size.
    opened by severb 11
  • Ws ping reqs bug fix

    Ws ping reqs bug fix

    Restructure ping-reqs and put them into a indirectPing -> (reached bool, errs []errors) function The test is said to be inconclusive if all nodes that should ping the target are unreachable.

    Add tests for all seven possible ping-req scenarios.

    Add a ListenerFunc type to the swim/events.go that turns a function into a object that implements EventListener.

    opened by CorgiMan 11
  • Ringpop with HTTP server

    Ringpop with HTTP server

    Hi, I am trying to use ringpop with http.Server. The server's handler receives requests and uses ringpop to handle or forward the requests to other servers on the ring. But I am unable to find any docs or guides or examples. Can someone please share a link to relevant examples? Thanks.

    opened by lamnguyen21 0
  • Fix function comments based on best practices from Effective Go

    Fix function comments based on best practices from Effective Go

    Every exported function in a program should have a doc comment. The first sentence should be a summary that starts with the name being declared. From effective go.

    I generated this with CodeLingo and I'm keen to get some feedback, but this is automated so feel free to close it and just say "opt out" to opt out of future CodeLingo outreach PRs.

    opened by Daanikus 2
  • (This project is no longer under active development.)

    (This project is no longer under active development.)

    Could we please get some more info about what this means? Should people avoid looking at ringpop as it's a discontinued project that uber is moving on from, or is it because you consider it feature complete and in maintenance mode?

    opened by tjstebbing 3
  • Generated rinpop adapter does not compile when there is service extension

    Generated rinpop adapter does not compile when there is service extension

    service Foo {
       string ping()
    }
    
    service Bar extends Foo {
    }
    

    The generated code fails with this error:

    cannot use adapter (type *RingpopBarAdapter) as type TChanBar in return argument:
    	*RingpopBarAdapter does not implement TChanBar (missing Ping method)
    

    Examples of how tchannel-go thrift gen template handles it https://github.com/uber/tchannel-go/blob/dev/thrift/thrift-gen/tchannel-template.go#L55 https://github.com/uber/tchannel-go/blob/dev/thrift/thrift-gen/tchannel-template.go#L45

    opened by chucknthem 1
  • Fix leaking timer goroutines

    Fix leaking timer goroutines

    This fixes two leaking goroutines from the ringpop timers. There is still one more leaking goroutine that I'm aware of in rcrowley/go-metrics:

    https://github.com/rcrowley/go-metrics/blob/master/meter.go#L218-L225

    That's a ticker that would also require a stop channel.

    opened by andrewdeandrade 1
Releases(v0.10.0)
Owner
Uber Open Source
Open Source Software at Uber
Uber Open Source
Dkron - Distributed, fault tolerant job scheduling system https://dkron.io

Dkron - Distributed, fault tolerant job scheduling system for cloud native environments Website: http://dkron.io/ Dkron is a distributed cron service,

Distributed Works 3.4k Dec 5, 2022
A distributed fault-tolerant order book matching engine

Go - Between A distributed fault-tolerant order book matching engine. Features Limit orders Market orders Order book depth Calculate market price for

Daniel Gatis 1 Dec 24, 2021
Easy to use Raft library to make your app distributed, highly available and fault-tolerant

An easy to use customizable library to make your Go application Distributed, Highly available, Fault Tolerant etc... using Hashicorp's Raft library wh

Richard Bertok 60 Nov 16, 2022
An implementation of a distributed KV store backed by Raft tolerant of node failures and network partitions 🚣

barge A simple implementation of a consistent, distributed Key:Value store which uses the Raft Concensus Algorithm. This project launches a cluster of

Shehjad Khan 0 Nov 24, 2021
Flowgraph package for scalable asynchronous system development

flowgraph Getting Started go get -u github.com/vectaport/flowgraph go test Links Wiki Slides from Minneapolis Golang Meetup, May 22nd 2019 Overview F

Scott Johnston 53 Sep 26, 2022
Fast, efficient, and scalable distributed map/reduce system, DAG execution, in memory or on disk, written in pure Go, runs standalone or distributedly.

Gleam Gleam is a high performance and efficient distributed execution system, and also simple, generic, flexible and easy to customize. Gleam is built

Chris Lu 3.1k Dec 7, 2022
Simple, fast and scalable golang rpc library for high load

gorpc Simple, fast and scalable golang RPC library for high load and microservices. Gorpc provides the following features useful for highly loaded pro

Aliaksandr Valialkin 665 Nov 29, 2022
Sandglass is a distributed, horizontally scalable, persistent, time sorted message queue.

Sandglass is a distributed, horizontally scalable, persistent, time ordered message queue. It was developed to support asynchronous tasks and message

Sandglass 1.5k Nov 18, 2022
Dapr is a portable, event-driven, runtime for building distributed applications across cloud and edge.

Dapr is a portable, serverless, event-driven runtime that makes it easy for developers to build resilient, stateless and stateful microservices that run on the cloud and edge and embraces the diversity of languages and developer frameworks.

Dapr 20k Dec 3, 2022
Build share and run your distributed applications.

sealer[ˈsiːlər] provides the way for distributed application package and delivery based on kubernetes.

Alibaba 1.7k Nov 23, 2022
Gorm-sharding - High performance table sharding plugin for Gorm

Gorm Sharding This project has moved to Gorm offical organization: https://githu

Longbridge 72 Nov 18, 2022
Lightweight, fault-tolerant message streams.

Liftbridge provides lightweight, fault-tolerant message streams by implementing a durable stream augmentation for the NATS messaging system. It extend

Liftbridge 2.4k Dec 2, 2022
Distributed, fault-tolerant key-value storage written in go.

A simple, distributed, fault-tolerant key-value storage inspired by Redis. It uses Raft protocotol as consensus algorithm. It supports the following data structures: String, Bitmap, Map, List.

Igor German 360 Aug 4, 2022
A distributed, fault-tolerant pipeline for observability data

Table of Contents What Is Veneur? Use Case See Also Status Features Vendor And Backend Agnostic Modern Metrics Format (Or Others!) Global Aggregation

Stripe 1.6k Nov 30, 2022
Super fault-tolerant gateway for HTTP clusters, written in Go. White paper for reference - https://github.com/gptankit/serviceq-paper

ServiceQ ServiceQ is a fault-tolerant gateway for HTTP clusters. It employs probabilistic routing to distribute load during partial cluster shutdown (

Ankit Gupta 66 Jul 16, 2022
Fault tolerant, sharded key value storage written in GoLang

Ravel is a sharded, fault-tolerant key-value store built using BadgerDB and hashicorp/raft. You can shard your data across multiple clusters with mult

Aditya Meharia 77 Nov 1, 2022
Dkron - Distributed, fault tolerant job scheduling system https://dkron.io

Dkron - Distributed, fault tolerant job scheduling system for cloud native environments Website: http://dkron.io/ Dkron is a distributed cron service,

Distributed Works 3.4k Dec 5, 2022
A distributed fault-tolerant order book matching engine

Go - Between A distributed fault-tolerant order book matching engine. Features Limit orders Market orders Order book depth Calculate market price for

Daniel Gatis 1 Dec 24, 2021
Tendermint Core is a Byzantine Fault Tolerant (BFT) middleware that takes a state transition machine

Tendermint Core is a Byzantine Fault Tolerant (BFT) middleware that takes a state transition machine - written in any programming language - and securely replicates it on many machines.

null 0 Sep 8, 2022
Easy to use Raft library to make your app distributed, highly available and fault-tolerant

An easy to use customizable library to make your Go application Distributed, Highly available, Fault Tolerant etc... using Hashicorp's Raft library wh

Richard Bertok 60 Nov 16, 2022