High-performance framework for building redis-protocol compatible TCP servers/services

Related tags

go redis service rpc low-latency
Overview

Redeo

GoDoc Build Status Go Report Card License

The high-performance Swiss Army Knife for building redis-protocol compatible servers/services.

Parts

This repository is organised into multiple components:

  • root package contains the framework for building redis-protocol compatible, high-performance servers.
  • resp implements low-level primitives for dealing with RESP (REdis Serialization Protocol), client and server-side. It contains basic wrappers for readers and writers to read/write requests and responses.
  • client contains a minimalist pooled client.

For full documentation and examples, please see the individual packages and the official API documentation: https://godoc.org/github.com/bsm/redeo.

Examples

A simple server example with two commands:

package main

import (
  "net"

  "github.com/bsm/redeo"
)

func main() {
	srv := redeo.NewServer(nil)

	// Define handlers
	srv.HandleFunc("ping", func(w resp.ResponseWriter, _ *resp.Command) {
		w.AppendInlineString("PONG")
	})
	srv.HandleFunc("info", func(w resp.ResponseWriter, _ *resp.Command) {
		w.AppendBulkString(srv.Info().String())
	})

	// More handlers; demo usage of redeo.WrapperFunc
	srv.Handle("echo", redeo.WrapperFunc(func(c *resp.Command) interface{} {
		if c.ArgN() != 1 {
			return redeo.ErrWrongNumberOfArgs(c.Name)
		}
		return c.Arg(0)
	}))

	// Open a new listener
	lis, err := net.Listen("tcp", ":9736")
	if err != nil {
		panic(err)
	}
	defer lis.Close()

	// Start serving (blocking)
	srv.Serve(lis)
}

More complex handlers:

func main() {
	mu := sync.RWMutex{}
	data := make(map[string]string)
	srv := redeo.NewServer(nil)

	srv.HandleFunc("set", func(w resp.ResponseWriter, c *resp.Command) {
		if c.ArgN() != 2 {
			w.AppendError(redeo.WrongNumberOfArgs(c.Name))
			return
		}

		key := c.Arg(0).String()
		val := c.Arg(1).String()

		mu.Lock()
		data[key] = val
		mu.Unlock()

		w.AppendInt(1)
	})

	srv.HandleFunc("get", func(w resp.ResponseWriter, c *resp.Command) {
		if c.ArgN() != 1 {
			w.AppendError(redeo.WrongNumberOfArgs(c.Name))
			return
		}

		key := c.Arg(0).String()
		mu.RLock()
		val, ok := data[key]
		mu.RUnlock()

		if ok {
			w.AppendBulkString(val)
			return
		}
		w.AppendNil()
	})
}

Redeo also supports command wrappers:

func main() {
	mu := sync.RWMutex{}
	data := make(map[string]string)
	srv := redeo.NewServer(nil)

	srv.Handle("set", redeo.WrapperFunc(func(c *resp.Command) interface{} {
		if c.ArgN() != 2 {
			return redeo.ErrWrongNumberOfArgs(c.Name)
		}

		key := c.Arg(0).String()
		val := c.Arg(1).String()

		mu.Lock()
		data[key] = val
		mu.Unlock()

		return 1
	}))

	srv.Handle("get", redeo.WrapperFunc(func(c *resp.Command) interface{} {
		if c.ArgN() != 1 {
			return redeo.ErrWrongNumberOfArgs(c.Name)
		}

		key := c.Arg(0).String()
		mu.RLock()
		val, ok := data[key]
		mu.RUnlock()

		if ok {
			return val
		}
		return nil
	}))
}
Issues
  • Double-quoted string shoud be treated as one argument

    Double-quoted string shoud be treated as one argument

    Very sweet library, thanks!

    I found a litte problem when I want use it in my project: If I type echo "hello world", c.ArgN() returns 2 rather then 1, redis do the right thing. Maybe it's better to parse double-quoted string as one argument rather then just split them by blank?

    opened by zsimple 5
  • add Close() method to server

    add Close() method to server

    New simple implementation, please review it

    opened by ngaut 5
  • Allow the client to be set manually

    Allow the client to be set manually

    I'm implementing Redis transactions (https://github.com/tsileo/silokv/blob/master/main.go), so I keep a buffer of request I manually apply on EXEC, but I need a way to set the client (the context) manually.

    opened by tsileo 3
  • add close event handler

    add close event handler

    this is a my hack resolution. does there any similar resolution?

    opened by kitech 3
  • Performance much slower than Redis

    Performance much slower than Redis

    Even with empty handlers that basically NOOP the performance of this library is still lacking compared to Redis itself which is doing more work. I wonder if the bottleneck could be removed? I haven't looked into it yet but I thought I would report some benchmarks.

    Redis

    ./redis-benchmark -t set -n 100000 -q -P 100
    SET: 806451.62 requests per second
    

    Redeo

    ./redis-benchmark -t set -n 100000 -q -P 100
    SET: 216919.73 requests per second
    

    Server code is as follows.

        srv := redeo.NewServer(&redeo.Config{Addr: "localhost:6379"})
        srv.HandleFunc("ping", func(out *redeo.Responder, _ *redeo.Request) error {
            out.WriteInlineString("PONG")
            return nil
        })
    
        srv.HandleFunc("set", func(out *redeo.Responder, _ *redeo.Request) error {
            out.WriteInlineString("OK")
            return nil
        })
    
        log.Fatal(srv.ListenAndServe())
    }
    

    Profiling shows the following.

    (pprof) top10
    12.40s of 13.13s total (94.44%)
    Dropped 158 nodes (cum <= 0.07s)
    Showing top 10 nodes out of 113 (cum >= 0.22s)
          flat  flat%   sum%        cum   cum%
         8.13s 61.92% 61.92%      8.15s 62.07%  syscall.Syscall
         1.54s 11.73% 73.65%      1.54s 11.73%  runtime.mach_semaphore_signal
         0.76s  5.79% 79.44%      0.76s  5.79%  runtime.kevent
         0.71s  5.41% 84.84%      0.71s  5.41%  runtime.usleep
         0.64s  4.87% 89.72%      0.64s  4.87%  runtime.mach_semaphore_wait
         0.29s  2.21% 91.93%      0.29s  2.21%  runtime.(*mcentral).grow
         0.11s  0.84% 92.76%      0.11s  0.84%  runtime.mach_semaphore_timedwait
         0.09s  0.69% 93.45%      0.09s  0.69%  nanotime
         0.07s  0.53% 93.98%      0.12s  0.91%  runtime.scanobject
         0.06s  0.46% 94.44%      0.22s  1.68%  runtime.scanstack
    
    opened by simongui 3
  • Don't close on unknown command.

    Don't close on unknown command.

    No need to close the connection on a unknown command.

    opened by alicebob 2
  • Provides a way to access Responder raw bytes.Buffer

    Provides a way to access Responder raw bytes.Buffer

    It's currently not possible to perform a res.WriteTo(anotherRes), this change allows to write: res.WriteTo(anotherRes.Buffer()).

    opened by tsileo 2
  • Missing LICENSE File

    Missing LICENSE File

    You have a copy of a license in your README file, but you do not include an actual LICENSE file. A license should typically be it's own file and not included as part of the documentation. Would you be willing to fix this so that I can continue getting this source (ultimately Gitea) into Debian?

    opened by MTecknology 2
Owner
Black Square Media
Black Square Media
Type-safe Redis client for Golang

Redis client for Golang ❤️ Uptrace.dev - distributed traces, logs, and errors in one place Join Discord to ask questions. Documentation Reference Exam

null 12k Jul 20, 2021
Redis Sorted Sets Benchmark

redis-zbench-go Redis Sorted Sets Benchmark Overview This repo contains code to trigger load ( ZADD ) or query (ZRANGEBYLEX key min max) benchmarks, w

filipe oliveira 3 May 18, 2021
Distributed WebSocket broker

dSock dSock is a distributed WebSocket broker (in Go, using Redis). Clients can authenticate & connect, and you can send text/binary message as an API

Charles Crete 200 Jul 22, 2021
Simple key-value store abstraction and implementations for Go (Redis, Consul, etcd, bbolt, BadgerDB, LevelDB, Memcached, DynamoDB, S3, PostgreSQL, MongoDB, CockroachDB and many more)

gokv Simple key-value store abstraction and implementations for Go Contents Features Simple interface Implementations Value types Marshal formats Road

Philipp Gillé 350 Jul 27, 2021
Google Go Client and Connectors for Redis

Go-Redis Go Clients and Connectors for Redis. The initial release provides the interface and implementation supporting the (~) full set of current Red

Joubin Houshyar 436 Jul 22, 2021
GoBigdis is a persistent database that implements the Redis server protocol. Any Redis client can interface with it and start to use it right away.

GoBigdis GoBigdis is a persistent database that implements the Redis server protocol. Any Redis client can interface with it and start to use it right

Riccardo 4 Jul 10, 2021
redis client implement by golang, inspired by jedis.

godis redis client implement by golang, refers to jedis. this library implements most of redis command, include normal redis command, cluster command,

piaohao 93 Jul 19, 2021
Golang client for redislabs' ReJSON module with support for multilple redis clients (redigo, go-redis)

Go-ReJSON - a golang client for ReJSON (a JSON data type for Redis) Go-ReJSON is a Go client for ReJSON Redis Module. ReJSON is a Redis module that im

Nitish Malhotra 172 Jul 24, 2021
🐺 Deploy Databases and Services Easily for Development and Testing Pipelines.

Peanut provides an API and a command line tool to deploy and configure the commonly used services like databases, message brokers, graphing tools ... etc. It perfectly suited for development, manual testing, automated testing pipelines where mocking is not possible and test drives.

Ahmed 35 Jul 24, 2021
Redis client library for Go

go-redis go-redis is a Redis client library for the Go programming language. It's built on the skeleton of gomemcache. It is safe to use by multiple g

Alexandre Fiori 45 Jul 14, 2020
Go Redis Client

xredis Built on top of github.com/garyburd/redigo with the idea to simplify creating a Redis client, provide type safe calls and encapsulate the low l

Raed Shomali 15 Jun 2, 2021
Go client for Redis

Redigo Redigo is a Go client for the Redis database. Features A Print-like API with support for all Redis commands. Pipelining, including pipelined tr

null 8.5k Jul 21, 2021
High-performance framework for building redis-protocol compatible TCP servers/services

Redeo The high-performance Swiss Army Knife for building redis-protocol compatible servers/services. Parts This repository is organised into multiple

Black Square Media 397 Jun 21, 2021
Redis client Mock Provide mock test for redis query

Redis client Mock Provide mock test for redis query, Compatible with github.com/go-redis/redis/v8 Install Confirm that you are using redis.Client the

null 50 Jul 21, 2021