Go Memcached client library #golang

Overview

About

This is a memcache client library for the Go programming language (http://golang.org/).

Installing

Using go get

$ go get github.com/bradfitz/gomemcache/memcache

After this command gomemcache is ready to use. Its source will be in:

$GOPATH/src/github.com/bradfitz/gomemcache/memcache

Example

import (
        "github.com/bradfitz/gomemcache/memcache"
)

func main() {
     mc := memcache.New("10.0.0.1:11211", "10.0.0.2:11211", "10.0.0.3:11212")
     mc.Set(&memcache.Item{Key: "foo", Value: []byte("my value")})

     it, err := mc.Get("foo")
     ...
}

Full docs, see:

See https://godoc.org/github.com/bradfitz/gomemcache/memcache

Or run:

$ godoc github.com/bradfitz/gomemcache/memcache
Issues
  • Use ReadFull instead of ioutil.ReadAll to read objects

    Use ReadFull instead of ioutil.ReadAll to read objects

    ioutil.ReadAll uses a bytes.Buffer, which allocates memory by doubling. Since we know exactly how muh data we expect to get, we can allocate it in advance. This reduces the total amount of allocation, and ensures that the slice stored in the item won't have excess capacity (which can affect memory usage if the item is held for a long time, for instance in a secondary cache).

    cla: yes 
    opened by arodland 5
  • App engine standard

    App engine standard

    Brad,

    Not sure if you would be interested, but I made your client work on App Engine Standard Environment -- using a memcached instance on Compute Engine. Just had to add the appengine context.

    I'm a newb at this GitHub stuff, so hopefully I did this right...

    Brian

    cla: yes 
    opened by cavebeavis 5
  • fix problems with the dns resolving cache

    fix problems with the dns resolving cache

    There are problems with connecting to the memcash server by DNS name when changing the IP address of the server.

    Due to the fact that the IP address is cached and there is no mechanism for resolving a new address (updating the cache), we get constant errors connecting to the server.

    I have added some solution.

    cla: yes 
    opened by HejVkt 4
  • Add facebook/mcrouter support

    Add facebook/mcrouter support

    Problem

    We're using mcrouter (v0.40) as a scalable memcache solution, however it doesn't seem to support gets command which this library uses by default even for fetching single key values.

    Basically we get this kind of error:

    memcache: unexpected line in get response: "SERVER_ERROR unexpected result mc_res_unknown (0) for get\r\n"
    

    Probably related issue - https://github.com/bradfitz/gomemcache/issues/37#issuecomment-133867638

    Changes

    This PR

    • adds private getOneFromAddr() which uses get command under the hood & is used for fetching single key values now from Get()
    • adds private getMultiFromAddr() which uses gets from GetMulti()
    • adds public setting GetMultiSupported to the client (set to true by default)
    • changes GetMulti- if GetMultiSupported is set to false, then for mcrouter compatibility, it reads keys in sequence using getOneFromAddr
    • changes integration tests to have more visible hierarchy with t.Run inside (would be nice to make them independent though, looks like they are coupled)
    • changes integration tests to be mandatory and not be skipped if memcached server is not found
    • adds integration test for GetMulti for mcrouter case
    • fixes Set malformed keys unit test, because in go 1.15+ string(0x7f) fails to convert a string conversion from untyped int to string yields a string of one rune, not a string of digits
    cla: yes 
    opened by tot-ra 4
  • Use buffer pool to read from parseGetResponse bufio.Reader instead of using ReadAll everytime

    Use buffer pool to read from parseGetResponse bufio.Reader instead of using ReadAll everytime

    My main concern is using ioutil.ReadAll everytime the bufio.Reader is read in parseGetResponse since it will then trigger bytes.makeSlice and allocate new space each and every time the function is called.

    Instead I think using a buffer pool like fmt does would be a fairly better solution.

    I've added a benchmark in memcache_test.go that Set 4 []byte of different sizes in memcache, and then randomly Get them. Here's the result before using the buffer pool:

    $ go test ./memcache -v -run=^$ -bench=BenchmarkGet$ -benchtime=10s -parallel=4 -memprofile=prof.mem
    PASS
    BenchmarkGet-4    200000         53816 ns/op       74349 B/op         36 allocs/op
    $ go tool pprof --alloc_space memcache.test prof.mem
    (pprof) list parseGetResponse
    Total: 15.85GB
    ROUTINE ======================== github.com/asticode/gomemcache/memcache.parseGetResponse in /home/asticode/projects/go/src/github.com/asticode/gomemcache/memcache/memcache.go
       51.50MB    15.41GB (flat, cum) 97.23% of Total
             .          .    453:           return err
             .          .    454:       }
             .          .    455:       if bytes.Equal(line, resultEnd) {
             .          .    456:           return nil
             .          .    457:       }
          46MB       46MB    458:       it := new(Item)
             .       66MB    459:       size, err := scanGetResponseLine(line, it)
             .          .    460:       if err != nil {
             .          .    461:           return err
             .          .    462:       }
        5.50MB    15.30GB    463:       it.Value, err = ioutil.ReadAll(io.LimitReader(r, int64(size)+2))
             .          .    464:       if err != nil {
             .          .    465:           return err
             .          .    466:       }
             .          .    467:       if !bytes.HasSuffix(it.Value, crlf) {
             .          .    468:           return fmt.Errorf("memcache: corrupt get result read")
    

    And here's the result after using the buffer pool:

    $ go test ./memcache -v -run=^$ -bench=BenchmarkGet$ -benchtime=10s -parallel=4 -memprofile=prof.mem
    PASS
    BenchmarkGet-4    500000         27828 ns/op        1082 B/op         31 allocs/op
    $ go tool pprof --alloc_space memcache.test prof.mem
    (pprof) list parseGetResponse
    Total: 549.90MB
    ROUTINE ======================== github.com/asticode/gomemcache/memcache.parseGetResponse in /home/asticode/projects/go/src/github.com/asticode/gomemcache/memcache/memcache.go
          49MB   341.24MB (flat, cum) 62.06% of Total
             .          .    466:           return err
             .          .    467:       }
             .          .    468:       if bytes.Equal(line, resultEnd) {
             .          .    469:           return nil
             .          .    470:       }
       33.50MB    33.50MB    471:       it := new(Item)
             .   115.50MB    472:       size, err := scanGetResponseLine(line, it)
             .          .    473:       if err != nil {
             .          .    474:           return err
             .          .    475:       }
             .          .    476:       buf := bufferPool.Get().(*bytes.Buffer)
       15.50MB   192.23MB    477:       _, err = buf.ReadFrom(io.LimitReader(r, int64(size)+2))
             .          .    478:       if err != nil {
             .          .    479:           freeBuffer(buf)
             .          .    480:           return err
             .          .    481:       }
             .          .    482:       it.Value = buf.Bytes()
    
    
    cla: yes 
    opened by asticode 4
  • How to use Codecs?

    How to use Codecs?

    Hi,

    I'm trying to store a struct, but I can find no examples of doing so with this library.

    Googling, I see examples for using memcache.Gob.set, but these advice seem to refer to Appengine/memcache.

    At https://github.com/bradfitz/gomemcache/blob/master/memcache/memcache.go#L150 the comments refer to using a codec to set an Item, but I can't find any reference to codecs, Gob, etc in the source.

    Is this a mistake?

    opened by jfeldstein 4
  • Make the maxIdleConnsPerAddr configurable

    Make the maxIdleConnsPerAddr configurable

    Write a new ‘NewWithConnThreshold’ func to configure maxIdleConnsPerAddr. Decide not to change the original 'New' func in consideration of backward compatibility.

    cla: yes 
    opened by varstr 4
  • Exported Metrics Views and Tags

    Exported Metrics Views and Tags

    Moved the views to their own exported variables so that they can be accessed and registered individually. Changed the tags to be exported as well to be able to add or adjust the tags assigned to the views.

    cla: no 
    opened by KolbyMcGarrah 3
  • encounter fault,

    encounter fault, "Get failed memcache: unexpected line in get response: "ERROR\r\n""

    memcached version: 1.5.1 code: // Set foo := &memcache.Item{Key: "home", Value: []byte("fooval")} err := mc.Add(foo) // work 0k here if err != nil { fmt.Println("Set failed once", err.Error()) } err = mc.Add(foo) // work 0k here if err != nil { fmt.Println("Set failed secondly", err.Error()) }

    // Get it, err := mc.Get("home") if err != nil { fmt.Println("Get failed", err.Error()) // failed here } else { if string(it.Key) == "foo" { fmt.Println("Add value is ", string(it.Value)) } else { fmt.Println("Get failed") } }

    opened by clee1602 3
  • Add Ping method.

    Add Ping method.

    Checks if instances are alive.

    errorCollector := make(chan error)
    m := memcache.New("/var/run/memcached.sock")
    
    go func() {
    	tick := time.NewTicker(time.Second)
    	for range tick.C {
    		if err := m.Ping(); err != nil {
    			errorCollector <- err
    		}
    	}
    }()
    
    go func() {
    	for err := range errorCollector {
    		fmt.Println(err)
    	}
    }()
    
    cla: yes 
    opened by andreiavrammsd 3
  • Make bufferSize&maxIdleConnsPerAddr configurable

    Make bufferSize&maxIdleConnsPerAddr configurable

    • Add bufferSize field to client, add SetBufferSize optional function to constructor
    • Add maxIdleConnsPerAddr field to client, add SetMaxIdleConnsPerAddr optional function to constructo
    • Fix some lint warning for method comment

    This change is Reviewable

    cla: yes 
    opened by lysu 3
  • Adding Support For Meta & Stats Commands

    Adding Support For Meta & Stats Commands

    Adds new functionality for Meta commands, which provides more flexibility with the use of in-line flags, and the stats commands which returns information for each connection. There are also tests for each of these commands.

    opened by moh-stripe 0
  • Connections are attempted on idle connections that have already been closed

    Connections are attempted on idle connections that have already been closed

    I'm attempting to query memcache on localhost every 10 minutes. The first Get("key") works fine, but the one 10 minutes later results in an error. The cycle repeats, so every other query succeeds. tcpdump shows that the memcache server attempted to close the TCP connection after about 30 seconds.

    The errors I get are either of the following.

    panic: write tcp 127.0.0.1:13590->127.0.0.1:11211: write: broken pipe
    EOF
    

    I've found I can work around this problem by calling Ping() and ignoring the returned error immediately before any Get() call.

    I've boiled down a minimal repro of the problem I'm facing.

    package main
    
    import (
            "github.com/bradfitz/gomemcache/memcache"
            "fmt"
            "time"
    )
    
    func query(mc *memcache.Client) {
            // This Ping() call mitigates the problem
            // err := mc.Ping()
            // fmt.Printf("Ping(): %v\n", err)
    
            r, err := mc.Get("key")
            if err != nil {
                    panic(err)
            }
            fmt.Printf("%v\n", string(r.Value))
    }
    
    func main() {
            mc := memcache.New("127.0.0.1:11211")
    
            query(mc)
    
            fmt.Printf("Sleeping...\n")
            time.Sleep(35 * time.Second)
    
            // This second query call results in an error from mc.Get("key")
            query(mc)
    }
    
    opened by joel0 2
  • Support for contexts

    Support for contexts

    This MR, which pretty much adds contexts to all exported methods, allows us to use that information not only in the main library component but also in custom server list, which then can be used for other stuff such as instrumentation for new relic, which was the original objective for which I forked the library.

    opened by paulozenida 0
  • Transfer gomemcache to memcached org

    Transfer gomemcache to memcached org

    Hey @bradfitz / @dormando

    What do you think of moving this project to the memcached GitHub org?

    There are quite a few open issues and PRs that could use some attention. It would be nice to increase the number of maintainers able to work on it.

    opened by SuperQ 0
Owner
Brad Fitzpatrick
Xoogler. Ex @golang team (2010-2020). Currently making WireGuard easier and more magical @Tailscale.
Brad Fitzpatrick
An in-memory key:value store/cache (similar to Memcached) library for Go, suitable for single-machine applications.

go-cache go-cache is an in-memory key:value store/cache similar to memcached that is suitable for applications running on a single machine. Its major

Patrick Mylund Nielsen 6.4k Aug 9, 2022
A memcached binary protocol toolkit for go.

gomemcached This is a memcached binary protocol toolkit in go. It provides client and server functionality as well as a little sample server showing h

Dustin Sallings 318 Jun 14, 2022
A memcached proxy that manages data chunking and L1 / L2 caches

Rend: Memcached-Compatible Server and Proxy Rend is a proxy whose primary use case is to sit on the same server as both a memcached process and an SSD

Netflix, Inc. 1.1k Aug 3, 2022
memcached operator

memcached-operator Operator SDK 中的 Go 编程语言支持可以利用 Operator SDK 中的 Go 编程语言支持,为 Memcached 构 建基于 Go 的 Operator 示例、分布式键值存储并管理其生命周期。 前置条件 安装 Docker Desktop,

jxlwqq 3 Apr 6, 2022
Go memcache client package

Description This is a memcache client package for the Go programming language. The following commands are implemented: get (single key) set, add, repl

Krzysztof Kliś 66 Jan 8, 2022
Cache library for golang. It supports expirable Cache, LFU, LRU and ARC.

GCache Cache library for golang. It supports expirable Cache, LFU, LRU and ARC. Features Supports expirable Cache, LFU, LRU and ARC. Goroutine safe. S

Jun Kimura 2k Aug 4, 2022
An in-memory cache library for golang. It supports multiple eviction policies: LRU, LFU, ARC

GCache Cache library for golang. It supports expirable Cache, LFU, LRU and ARC. Features Supports expirable Cache, LFU, LRU and ARC. Goroutine safe. S

Jun Kimura 318 May 31, 2021
gdcache is a pure non-intrusive distributed cache library implemented by golang

gdcache is a pure non-intrusive distributed cache library implemented by golang, you can use it to implement your own distributed cache

Jovan 9 Mar 24, 2022
Gocodecache - An in-memory cache library for code value master in Golang

gocodecache An in-memory cache library for code master in Golang. Installation g

Takuo Oki 3 Jun 23, 2022
Concurrency-safe Go caching library with expiration capabilities and access counters

cache2go Concurrency-safe golang caching library with expiration capabilities. Installation Make sure you have a working Go environment (Go 1.2 or hig

Christian Muehlhaeuser 1.8k Aug 5, 2022
☔️ A complete Go cache library that brings you multiple ways of managing your caches

Gocache Guess what is Gocache? a Go cache library. This is an extendable cache library that brings you a lot of features for caching data. Overview He

Vincent Composieux 1.4k Aug 4, 2022
A REST-API service that works as an in memory key-value store with go-minimal-cache library.

A REST-API service that works as an in memory key-value store with go-minimal-cache library.

HarunBuyuktepe 1 Oct 25, 2021
A multi-level cache library with stampede prevention for Go

HybridCache A multi-level cache library with cache stampede prevention for Go import "github.com/cshum/hybridcache" // Redis cache adapter based on R

Adrian Shum 116 May 17, 2022
An in-memory key:value store/cache library written in Go 1.18 generics

go-generics-cache go-generics-cache is an in-memory key:value store/cache that is suitable for applications running on a single machine. This in-memor

Kei Kamikawa 202 Aug 6, 2022
Design and Implement an in-memory caching library for general use

Cache Implementation in GoLang Problem Statement Design and Implement an in-memory caching library for general use. Must Have Support for multiple Sta

null 1 Dec 28, 2021
Ristretto - A fast, concurrent cache library built with a focus on performance and correctness

Ristretto Ristretto is a fast, concurrent cache library built with a focus on pe

Koichi Shiraishi 1 Mar 23, 2022
A zero-dependency cache library for storing data in memory with generics.

Memory Cache A zero-dependency cache library for storing data in memory with generics. Requirements Golang 1.18+ Installation go get -u github.com/rod

Rodrigo Brito 11 May 26, 2022
Redis caching layer for Cloudflare KV in Golang

Redis caching layer for Cloudflare KV in Golang

Dinesh Sonachalam 30 May 21, 2022