High-performance framework for building redis-protocol compatible TCP servers/services

Overview

Redeo

GoDoc Build Status Go Report Card License

The high-performance Swiss Army Knife for building redis-protocol compatible servers/services.

Parts

This repository is organised into multiple components:

  • root package contains the framework for building redis-protocol compatible, high-performance servers.
  • resp implements low-level primitives for dealing with RESP (REdis Serialization Protocol), client and server-side. It contains basic wrappers for readers and writers to read/write requests and responses.
  • client contains a minimalist pooled client.

For full documentation and examples, please see the individual packages and the official API documentation: https://godoc.org/github.com/bsm/redeo.

Examples

A simple server example with two commands:

package main

import (
  "net"

  "github.com/bsm/redeo"
)

func main() {
	srv := redeo.NewServer(nil)

	// Define handlers
	srv.HandleFunc("ping", func(w resp.ResponseWriter, _ *resp.Command) {
		w.AppendInlineString("PONG")
	})
	srv.HandleFunc("info", func(w resp.ResponseWriter, _ *resp.Command) {
		w.AppendBulkString(srv.Info().String())
	})

	// More handlers; demo usage of redeo.WrapperFunc
	srv.Handle("echo", redeo.WrapperFunc(func(c *resp.Command) interface{} {
		if c.ArgN() != 1 {
			return redeo.ErrWrongNumberOfArgs(c.Name)
		}
		return c.Arg(0)
	}))

	// Open a new listener
	lis, err := net.Listen("tcp", ":9736")
	if err != nil {
		panic(err)
	}
	defer lis.Close()

	// Start serving (blocking)
	srv.Serve(lis)
}

More complex handlers:

func main() {
	mu := sync.RWMutex{}
	data := make(map[string]string)
	srv := redeo.NewServer(nil)

	srv.HandleFunc("set", func(w resp.ResponseWriter, c *resp.Command) {
		if c.ArgN() != 2 {
			w.AppendError(redeo.WrongNumberOfArgs(c.Name))
			return
		}

		key := c.Arg(0).String()
		val := c.Arg(1).String()

		mu.Lock()
		data[key] = val
		mu.Unlock()

		w.AppendInt(1)
	})

	srv.HandleFunc("get", func(w resp.ResponseWriter, c *resp.Command) {
		if c.ArgN() != 1 {
			w.AppendError(redeo.WrongNumberOfArgs(c.Name))
			return
		}

		key := c.Arg(0).String()
		mu.RLock()
		val, ok := data[key]
		mu.RUnlock()

		if ok {
			w.AppendBulkString(val)
			return
		}
		w.AppendNil()
	})
}

Redeo also supports command wrappers:

func main() {
	mu := sync.RWMutex{}
	data := make(map[string]string)
	srv := redeo.NewServer(nil)

	srv.Handle("set", redeo.WrapperFunc(func(c *resp.Command) interface{} {
		if c.ArgN() != 2 {
			return redeo.ErrWrongNumberOfArgs(c.Name)
		}

		key := c.Arg(0).String()
		val := c.Arg(1).String()

		mu.Lock()
		data[key] = val
		mu.Unlock()

		return 1
	}))

	srv.Handle("get", redeo.WrapperFunc(func(c *resp.Command) interface{} {
		if c.ArgN() != 1 {
			return redeo.ErrWrongNumberOfArgs(c.Name)
		}

		key := c.Arg(0).String()
		mu.RLock()
		val, ok := data[key]
		mu.RUnlock()

		if ok {
			return val
		}
		return nil
	}))
}
Comments
  • Double-quoted string shoud be treated as one argument

    Double-quoted string shoud be treated as one argument

    Very sweet library, thanks!

    I found a litte problem when I want use it in my project: If I type echo "hello world", c.ArgN() returns 2 rather then 1, redis do the right thing. Maybe it's better to parse double-quoted string as one argument rather then just split them by blank?

    opened by zsimple 5
  • Performance much slower than Redis

    Performance much slower than Redis

    Even with empty handlers that basically NOOP the performance of this library is still lacking compared to Redis itself which is doing more work. I wonder if the bottleneck could be removed? I haven't looked into it yet but I thought I would report some benchmarks.

    Redis

    ./redis-benchmark -t set -n 100000 -q -P 100
    SET: 806451.62 requests per second
    

    Redeo

    ./redis-benchmark -t set -n 100000 -q -P 100
    SET: 216919.73 requests per second
    

    Server code is as follows.

        srv := redeo.NewServer(&redeo.Config{Addr: "localhost:6379"})
        srv.HandleFunc("ping", func(out *redeo.Responder, _ *redeo.Request) error {
            out.WriteInlineString("PONG")
            return nil
        })
    
        srv.HandleFunc("set", func(out *redeo.Responder, _ *redeo.Request) error {
            out.WriteInlineString("OK")
            return nil
        })
    
        log.Fatal(srv.ListenAndServe())
    }
    

    Profiling shows the following.

    (pprof) top10
    12.40s of 13.13s total (94.44%)
    Dropped 158 nodes (cum <= 0.07s)
    Showing top 10 nodes out of 113 (cum >= 0.22s)
          flat  flat%   sum%        cum   cum%
         8.13s 61.92% 61.92%      8.15s 62.07%  syscall.Syscall
         1.54s 11.73% 73.65%      1.54s 11.73%  runtime.mach_semaphore_signal
         0.76s  5.79% 79.44%      0.76s  5.79%  runtime.kevent
         0.71s  5.41% 84.84%      0.71s  5.41%  runtime.usleep
         0.64s  4.87% 89.72%      0.64s  4.87%  runtime.mach_semaphore_wait
         0.29s  2.21% 91.93%      0.29s  2.21%  runtime.(*mcentral).grow
         0.11s  0.84% 92.76%      0.11s  0.84%  runtime.mach_semaphore_timedwait
         0.09s  0.69% 93.45%      0.09s  0.69%  nanotime
         0.07s  0.53% 93.98%      0.12s  0.91%  runtime.scanobject
         0.06s  0.46% 94.44%      0.22s  1.68%  runtime.scanstack
    
    opened by simongui 3
  • Allow the client to be set manually

    Allow the client to be set manually

    I'm implementing Redis transactions (https://github.com/tsileo/silokv/blob/master/main.go), so I keep a buffer of request I manually apply on EXEC, but I need a way to set the client (the context) manually.

    opened by tsileo 3
  • Missing LICENSE File

    Missing LICENSE File

    You have a copy of a license in your README file, but you do not include an actual LICENSE file. A license should typically be it's own file and not included as part of the documentation. Would you be willing to fix this so that I can continue getting this source (ultimately Gitea) into Debian?

    opened by MTecknology 2
  • Fix slice out of bound exception in rare cases.

    Fix slice out of bound exception in rare cases.

    In rare cases, '\n' may not be buffered while '\r' is. This will cause consecutive "slice out of bound" failures of compact() calls due to b.r > b.w by 1.

    opened by zhangjyr 1
  • Give example for multi exec commands

    Give example for multi exec commands

    When using multi each command returns the string "QUEUED" exec returns the values of the executed commands how do i fetch that value in the client ?

    for example for this code:

    multi := tx.TxPipeline()
    cmd := multi.RPushX(key, ".")
    cmds, err := multi.Exec()
    

    i try to use those mocks:

    
    srv.HandleFunc("multi", func(w resp.ResponseWriter, c *resp.Command) {
    		w.AppendOK()
    	})
    
    srv.Handle("rpushx", redeo.WrapperFunc(func(c *resp.Command) interface{} {
    		return "QUEUED"
    	}))
    
    srv.Handle("exec", redeo.WrapperFunc(func(c *resp.Command) interface{} {
    		return []int{1}
    	}))
    
    

    how can i get the value 1 ? i tried:

    c := cmd.Val() // returns 0

    or:

    count = cmds[0].(*redis.IntCmd).Val() //returns 0 too

    Thanks Dan

    opened by buzzdan 1
  • Added Client.Close() method

    Added Client.Close() method

    In order to implement a 'QUIT' method the client now has a Close() method. It's more of a graceful close, where the socket is only closed once the buffer has been send.

    opened by alicebob 1
  • Return of TotalConnections and TotalCommands

    Return of TotalConnections and TotalCommands

    The number of processed commands and the count of connections made are not available as plain integer values anymore, only as strings in a more general info string. This PR re-introduces them.

    I used them in https://github.com/alicebob/miniredis/blob/master/miniredis.go#L186 and https://github.com/alicebob/miniredis/blob/master/miniredis.go#L202

    opened by alicebob 0
Owner
Black Square Media
Black Square Media
Stream new blocks to various services (redis/elasticsearch/...)

AlgoNode algostreamer utility About algostreamer Small utility to stream past and/or current Algorand node JSON blocks to Redis or stdout. About AlgoN

AlgoNode 12 Apr 25, 2022
Golang client for redislabs' ReJSON module with support for multilple redis clients (redigo, go-redis)

Go-ReJSON - a golang client for ReJSON (a JSON data type for Redis) Go-ReJSON is a Go client for ReJSON Redis Module. ReJSON is a Redis module that im

Nitish Malhotra 291 Nov 17, 2022
Redis client Mock Provide mock test for redis query

Redis client Mock Provide mock test for redis query, Compatible with github.com/go-redis/redis/v8 Install Confirm that you are using redis.Client the

null 149 Nov 11, 2022
Bxd redis benchmark - Redis benchmark tool for golang

使用 redis benchmark 工具, 测试 10 20 50 100 200 1k 5k 字节 value 大小,redis get set 性能。 r

bingxindan 0 Jan 22, 2022
redis protocol server for go.

redigosrv what redis server侧的golang版本实现。 why 作为后端RD,你开发的,维护的最多的应该就是你的server,作为服务的提供方来说,想要和客户端进行有效的交互,首先要要能理解信息的含义,因此一套通信协议是必须的。 为什么选择redis协议,应用层有很多成熟的

Saner 7 Nov 30, 2021
Schema based, typed Redis caching/memoize framework for Go

cacheme - Redis Caching Framework For Go English | 中文 Statically Typed - 100% statically typed using code generation. Scale Efficiently - thundering h

null 22 Sep 27, 2022
A MongoDB compatible embeddable database and toolkit for Go.

lungo A MongoDB compatible embeddable database and toolkit for Go. Installation Example Motivation Architecture Features License Installation To get s

Joël Gähwiler 397 Nov 19, 2022
🐺 Deploy Databases and Services Easily for Development and Testing Pipelines.

Peanut provides an API and a command line tool to deploy and configure the commonly used services like databases, message brokers, graphing tools ... etc. It perfectly suited for development, manual testing, automated testing pipelines where mocking is not possible and test drives.

Ahmed 682 Nov 14, 2022
redis client implement by golang, inspired by jedis.

godis redis client implement by golang, refers to jedis. this library implements most of redis command, include normal redis command, cluster command,

piaohao 107 Nov 18, 2022
Go client for Redis

Redigo Redigo is a Go client for the Redis database. Features A Print-like API with support for all Redis commands. Pipelining, including pipelined tr

null 9.3k Nov 27, 2022
Type-safe Redis client for Golang

Redis client for Golang ❤️ Uptrace.dev - distributed traces, logs, and errors in one place Join Discord to ask questions. Documentation Reference Exam

null 15.9k Nov 20, 2022
Go Redis Client

xredis Built on top of github.com/garyburd/redigo with the idea to simplify creating a Redis client, provide type safe calls and encapsulate the low l

Raed Shomali 19 Sep 26, 2022
Simple key-value store abstraction and implementations for Go (Redis, Consul, etcd, bbolt, BadgerDB, LevelDB, Memcached, DynamoDB, S3, PostgreSQL, MongoDB, CockroachDB and many more)

gokv Simple key-value store abstraction and implementations for Go Contents Features Simple interface Implementations Value types Marshal formats Road

Philipp Gillé 490 Nov 21, 2022
godis - an old Redis client for Go

godis Implements a few database clients for Redis. There is a stable client and an experimental client, redis and exp, respectively. To use any of the

Simon Zimmermann 86 Apr 16, 2022
Google Go Client and Connectors for Redis

Go-Redis Go Clients and Connectors for Redis. The initial release provides the interface and implementation supporting the (~) full set of current Red

Joubin Houshyar 442 Oct 25, 2022
Redis client library for Go

go-redis go-redis is a Redis client library for the Go programming language. It's built on the skeleton of gomemcache. It is safe to use by multiple g

Alexandre Fiori 45 Nov 8, 2022
Type-safe Redis client for Golang

Redis client for Golang ❤️ Uptrace.dev - distributed traces, logs, and errors in one place Join Discord to ask questions. Documentation Reference Exam

null 15.9k Nov 23, 2022
Redis Sorted Sets Benchmark

redis-zbench-go Redis Sorted Sets Benchmark Overview This repo contains code to trigger load ( ZADD ) or query (ZRANGEBYLEX key min max) benchmarks, w

filipe oliveira 3 May 18, 2021
Use Redis' MONITOR to draw things in a terminal

Redis Top Redistop uses MONITOR to watch Redis commands and shows per command and per host statistics. Because MONITOR streams back all commands, its

Factory 31 Aug 30, 2022