Go Memcached client library #golang

Overview

About

This is a memcache client library for the Go programming language (http://golang.org/).

Installing

Using go get

$ go get github.com/bradfitz/gomemcache/memcache

After this command gomemcache is ready to use. Its source will be in:

$GOPATH/src/github.com/bradfitz/gomemcache/memcache

Example

import (
        "github.com/bradfitz/gomemcache/memcache"
)

func main() {
     mc := memcache.New("10.0.0.1:11211", "10.0.0.2:11211", "10.0.0.3:11212")
     mc.Set(&memcache.Item{Key: "foo", Value: []byte("my value")})

     it, err := mc.Get("foo")
     ...
}

Full docs, see:

See https://godoc.org/github.com/bradfitz/gomemcache/memcache

Or run:

$ godoc github.com/bradfitz/gomemcache/memcache
Issues
  • Use ReadFull instead of ioutil.ReadAll to read objects

    Use ReadFull instead of ioutil.ReadAll to read objects

    ioutil.ReadAll uses a bytes.Buffer, which allocates memory by doubling. Since we know exactly how muh data we expect to get, we can allocate it in advance. This reduces the total amount of allocation, and ensures that the slice stored in the item won't have excess capacity (which can affect memory usage if the item is held for a long time, for instance in a secondary cache).

    cla: yes 
    opened by arodland 5
  • App engine standard

    App engine standard

    Brad,

    Not sure if you would be interested, but I made your client work on App Engine Standard Environment -- using a memcached instance on Compute Engine. Just had to add the appengine context.

    I'm a newb at this GitHub stuff, so hopefully I did this right...

    Brian

    cla: yes 
    opened by cavebeavis 5
  • fix problems with the dns resolving cache

    fix problems with the dns resolving cache

    There are problems with connecting to the memcash server by DNS name when changing the IP address of the server.

    Due to the fact that the IP address is cached and there is no mechanism for resolving a new address (updating the cache), we get constant errors connecting to the server.

    I have added some solution.

    cla: yes 
    opened by HejVkt 4
  • Add facebook/mcrouter support

    Add facebook/mcrouter support

    Problem

    We're using mcrouter (v0.40) as a scalable memcache solution, however it doesn't seem to support gets command which this library uses by default even for fetching single key values.

    Basically we get this kind of error:

    memcache: unexpected line in get response: "SERVER_ERROR unexpected result mc_res_unknown (0) for get\r\n"
    

    Probably related issue - https://github.com/bradfitz/gomemcache/issues/37#issuecomment-133867638

    Changes

    This PR

    • adds private getOneFromAddr() which uses get command under the hood & is used for fetching single key values now from Get()
    • adds private getMultiFromAddr() which uses gets from GetMulti()
    • adds public setting GetMultiSupported to the client (set to true by default)
    • changes GetMulti- if GetMultiSupported is set to false, then for mcrouter compatibility, it reads keys in sequence using getOneFromAddr
    • changes integration tests to have more visible hierarchy with t.Run inside (would be nice to make them independent though, looks like they are coupled)
    • changes integration tests to be mandatory and not be skipped if memcached server is not found
    • adds integration test for GetMulti for mcrouter case
    • fixes Set malformed keys unit test, because in go 1.15+ string(0x7f) fails to convert a string conversion from untyped int to string yields a string of one rune, not a string of digits
    cla: yes 
    opened by tot-ra 4
  • Use buffer pool to read from parseGetResponse bufio.Reader instead of using ReadAll everytime

    Use buffer pool to read from parseGetResponse bufio.Reader instead of using ReadAll everytime

    My main concern is using ioutil.ReadAll everytime the bufio.Reader is read in parseGetResponse since it will then trigger bytes.makeSlice and allocate new space each and every time the function is called.

    Instead I think using a buffer pool like fmt does would be a fairly better solution.

    I've added a benchmark in memcache_test.go that Set 4 []byte of different sizes in memcache, and then randomly Get them. Here's the result before using the buffer pool:

    $ go test ./memcache -v -run=^$ -bench=BenchmarkGet$ -benchtime=10s -parallel=4 -memprofile=prof.mem
    PASS
    BenchmarkGet-4    200000         53816 ns/op       74349 B/op         36 allocs/op
    $ go tool pprof --alloc_space memcache.test prof.mem
    (pprof) list parseGetResponse
    Total: 15.85GB
    ROUTINE ======================== github.com/asticode/gomemcache/memcache.parseGetResponse in /home/asticode/projects/go/src/github.com/asticode/gomemcache/memcache/memcache.go
       51.50MB    15.41GB (flat, cum) 97.23% of Total
             .          .    453:           return err
             .          .    454:       }
             .          .    455:       if bytes.Equal(line, resultEnd) {
             .          .    456:           return nil
             .          .    457:       }
          46MB       46MB    458:       it := new(Item)
             .       66MB    459:       size, err := scanGetResponseLine(line, it)
             .          .    460:       if err != nil {
             .          .    461:           return err
             .          .    462:       }
        5.50MB    15.30GB    463:       it.Value, err = ioutil.ReadAll(io.LimitReader(r, int64(size)+2))
             .          .    464:       if err != nil {
             .          .    465:           return err
             .          .    466:       }
             .          .    467:       if !bytes.HasSuffix(it.Value, crlf) {
             .          .    468:           return fmt.Errorf("memcache: corrupt get result read")
    

    And here's the result after using the buffer pool:

    $ go test ./memcache -v -run=^$ -bench=BenchmarkGet$ -benchtime=10s -parallel=4 -memprofile=prof.mem
    PASS
    BenchmarkGet-4    500000         27828 ns/op        1082 B/op         31 allocs/op
    $ go tool pprof --alloc_space memcache.test prof.mem
    (pprof) list parseGetResponse
    Total: 549.90MB
    ROUTINE ======================== github.com/asticode/gomemcache/memcache.parseGetResponse in /home/asticode/projects/go/src/github.com/asticode/gomemcache/memcache/memcache.go
          49MB   341.24MB (flat, cum) 62.06% of Total
             .          .    466:           return err
             .          .    467:       }
             .          .    468:       if bytes.Equal(line, resultEnd) {
             .          .    469:           return nil
             .          .    470:       }
       33.50MB    33.50MB    471:       it := new(Item)
             .   115.50MB    472:       size, err := scanGetResponseLine(line, it)
             .          .    473:       if err != nil {
             .          .    474:           return err
             .          .    475:       }
             .          .    476:       buf := bufferPool.Get().(*bytes.Buffer)
       15.50MB   192.23MB    477:       _, err = buf.ReadFrom(io.LimitReader(r, int64(size)+2))
             .          .    478:       if err != nil {
             .          .    479:           freeBuffer(buf)
             .          .    480:           return err
             .          .    481:       }
             .          .    482:       it.Value = buf.Bytes()
    
    
    cla: yes 
    opened by asticode 4
  • How to use Codecs?

    How to use Codecs?

    Hi,

    I'm trying to store a struct, but I can find no examples of doing so with this library.

    Googling, I see examples for using memcache.Gob.set, but these advice seem to refer to Appengine/memcache.

    At https://github.com/bradfitz/gomemcache/blob/master/memcache/memcache.go#L150 the comments refer to using a codec to set an Item, but I can't find any reference to codecs, Gob, etc in the source.

    Is this a mistake?

    opened by jfeldstein 4
  • Make the maxIdleConnsPerAddr configurable

    Make the maxIdleConnsPerAddr configurable

    Write a new ‘NewWithConnThreshold’ func to configure maxIdleConnsPerAddr. Decide not to change the original 'New' func in consideration of backward compatibility.

    cla: yes 
    opened by varstr 4
  • Exported Metrics Views and Tags

    Exported Metrics Views and Tags

    Moved the views to their own exported variables so that they can be accessed and registered individually. Changed the tags to be exported as well to be able to add or adjust the tags assigned to the views.

    cla: no 
    opened by KolbyMcGarrah 3
  • encounter fault,

    encounter fault, "Get failed memcache: unexpected line in get response: "ERROR\r\n""

    memcached version: 1.5.1 code: // Set foo := &memcache.Item{Key: "home", Value: []byte("fooval")} err := mc.Add(foo) // work 0k here if err != nil { fmt.Println("Set failed once", err.Error()) } err = mc.Add(foo) // work 0k here if err != nil { fmt.Println("Set failed secondly", err.Error()) }

    // Get it, err := mc.Get("home") if err != nil { fmt.Println("Get failed", err.Error()) // failed here } else { if string(it.Key) == "foo" { fmt.Println("Add value is ", string(it.Value)) } else { fmt.Println("Get failed") } }

    opened by clee1602 3
  • Add Ping method.

    Add Ping method.

    Checks if instances are alive.

    errorCollector := make(chan error)
    m := memcache.New("/var/run/memcached.sock")
    
    go func() {
    	tick := time.NewTicker(time.Second)
    	for range tick.C {
    		if err := m.Ping(); err != nil {
    			errorCollector <- err
    		}
    	}
    }()
    
    go func() {
    	for err := range errorCollector {
    		fmt.Println(err)
    	}
    }()
    
    cla: yes 
    opened by andreiavrammsd 3
  • Make bufferSize&maxIdleConnsPerAddr configurable

    Make bufferSize&maxIdleConnsPerAddr configurable

    • Add bufferSize field to client, add SetBufferSize optional function to constructor
    • Add maxIdleConnsPerAddr field to client, add SetMaxIdleConnsPerAddr optional function to constructo
    • Fix some lint warning for method comment

    This change is Reviewable

    cla: yes 
    opened by lysu 3
  • Adding Support For Meta & Stats Commands

    Adding Support For Meta & Stats Commands

    Adds new functionality for Meta commands, which provides more flexibility with the use of in-line flags, and the stats commands which returns information for each connection. There are also tests for each of these commands.

    opened by moh-stripe 0
  • Connections are attempted on idle connections that have already been closed

    Connections are attempted on idle connections that have already been closed

    I'm attempting to query memcache on localhost every 10 minutes. The first Get("key") works fine, but the one 10 minutes later results in an error. The cycle repeats, so every other query succeeds. tcpdump shows that the memcache server attempted to close the TCP connection after about 30 seconds.

    The errors I get are either of the following.

    panic: write tcp 127.0.0.1:13590->127.0.0.1:11211: write: broken pipe
    EOF
    

    I've found I can work around this problem by calling Ping() and ignoring the returned error immediately before any Get() call.

    I've boiled down a minimal repro of the problem I'm facing.

    package main
    
    import (
            "github.com/bradfitz/gomemcache/memcache"
            "fmt"
            "time"
    )
    
    func query(mc *memcache.Client) {
            // This Ping() call mitigates the problem
            // err := mc.Ping()
            // fmt.Printf("Ping(): %v\n", err)
    
            r, err := mc.Get("key")
            if err != nil {
                    panic(err)
            }
            fmt.Printf("%v\n", string(r.Value))
    }
    
    func main() {
            mc := memcache.New("127.0.0.1:11211")
    
            query(mc)
    
            fmt.Printf("Sleeping...\n")
            time.Sleep(35 * time.Second)
    
            // This second query call results in an error from mc.Get("key")
            query(mc)
    }
    
    opened by joel0 2
  • Support for contexts

    Support for contexts

    This MR, which pretty much adds contexts to all exported methods, allows us to use that information not only in the main library component but also in custom server list, which then can be used for other stuff such as instrumentation for new relic, which was the original objective for which I forked the library.

    opened by paulozenida 0
  • Transfer gomemcache to memcached org

    Transfer gomemcache to memcached org

    Hey @bradfitz / @dormando

    What do you think of moving this project to the memcached GitHub org?

    There are quite a few open issues and PRs that could use some attention. It would be nice to increase the number of maintainers able to work on it.

    opened by SuperQ 0
Owner
Brad Fitzpatrick
Xoogler. Ex @golang team (2010-2020). Currently making WireGuard easier and more magical @Tailscale.
Brad Fitzpatrick
Redisx: a library of Go utilities built on the redigo redis client library

redisx redisx is a library of Go utilities built on the redigo redis client libr

null 1 Dec 24, 2021
Go client library for Pilosa

Go Client for Pilosa Go client for Pilosa high performance distributed index. What's New? See: CHANGELOG Requirements Go 1.12 and higher. Install Down

Pilosa 53 Jun 9, 2022
Redis client library for Go

go-redis go-redis is a Redis client library for the Go programming language. It's built on the skeleton of gomemcache. It is safe to use by multiple g

Alexandre Fiori 45 Jul 15, 2020
Golang client for redislabs' ReJSON module with support for multilple redis clients (redigo, go-redis)

Go-ReJSON - a golang client for ReJSON (a JSON data type for Redis) Go-ReJSON is a Go client for ReJSON Redis Module. ReJSON is a Redis module that im

Nitish Malhotra 274 Aug 5, 2022
redis client implement by golang, inspired by jedis.

godis redis client implement by golang, refers to jedis. this library implements most of redis command, include normal redis command, cluster command,

piaohao 105 Jul 20, 2022
Neo4j REST Client in golang

DEPRECATED! Consider these instead: https://github.com/johnnadratowski/golang-neo4j-bolt-driver https://github.com/go-cq/cq Install: If you don't ha

null 76 Mar 14, 2022
Neo4j client for Golang

neoism - Neo4j client for Go Package neoism is a Go client library providing access to the Neo4j graph database via its REST API. Status System Status

Jason McVetta 388 Aug 2, 2022
Type-safe Redis client for Golang

Redis client for Golang ❤️ Uptrace.dev - distributed traces, logs, and errors in one place Join Discord to ask questions. Documentation Reference Exam

null 15k Aug 2, 2022
Type-safe Redis client for Golang

Redis client for Golang ❤️ Uptrace.dev - distributed traces, logs, and errors in one place Join Discord to ask questions. Documentation Reference Exam

null 15.1k Aug 11, 2022
HDFS for Go - This is a native golang client for hdfs.

HDFS for Go This is a native golang client for hdfs. It connects directly to the namenode using the protocol buffers API. It tries to be idiomatic by

Colin Marc 1.2k Aug 4, 2022
A CouchDB client in Go(Golang)

pillow pillow is a CouchDB client in Go(Golang). Resources Installation Usage Example Installation Install pillow as you normally would for any Go pac

Alex Munene 3 Feb 3, 2022
Redis client for Golang

Redis client for Golang To ask questions, join Discord or use Discussions. Newsl

null 0 Dec 23, 2021
Redis client for Golang

Redis client for Golang Discussions. Newsletter to get latest updates. Documentation Reference Examples RealWorld example app Other projects you may l

null 0 Dec 30, 2021
Aerospike Client Go

Aerospike Go Client An Aerospike library for Go. This library is compatible with Go 1.9+ and supports the following operating systems: Linux, Mac OS X

Aerospike 394 Aug 1, 2022
Couchbase client in Go

A smart client for couchbase in go This is a unoffical version of a Couchbase Golang client. If you are looking for the Offical Couchbase Golang clien

null 316 Jun 26, 2022
Neo4j Rest API Client for Go lang

neo4j.go Implementation of client package for communication with Neo4j Rest API. For more information and documentation please read Godoc Neo4j Page s

Cihangir 27 Jan 26, 2022
Go client for Redis

Redigo Redigo is a Go client for the Redis database. Features A Print-like API with support for all Redis commands. Pipelining, including pipelined tr

null 9.2k Aug 9, 2022
Go Redis Client

xredis Built on top of github.com/garyburd/redigo with the idea to simplify creating a Redis client, provide type safe calls and encapsulate the low l

Raed Shomali 18 Jan 23, 2022
godis - an old Redis client for Go

godis Implements a few database clients for Redis. There is a stable client and an experimental client, redis and exp, respectively. To use any of the

Simon Zimmermann 86 Apr 16, 2022