a generic object pool for golang

Overview

Go Commons Pool

Build Status CodeCov Go Report Card GoDoc

The Go Commons Pool is a generic object pool for Golang, direct rewrite from Apache Commons Pool.

Features

  1. Support custom PooledObjectFactory.
  2. Rich pool configuration option, can precise control pooled object lifecycle. See ObjectPoolConfig.
    • Pool LIFO (last in, first out) or FIFO (first in, first out)
    • Pool cap config
    • Pool object validate config
    • Pool object borrow block and max waiting time config
    • Pool object eviction config
    • Pool object abandon config

Pool Configuration Option

Configuration option table, more detail description see ObjectPoolConfig

Option Default Description
LIFO true If pool is LIFO (last in, first out)
MaxTotal 8 The cap of pool
MaxIdle 8 Max "idle" instances in the pool
MinIdle 0 Min "idle" instances in the pool
TestOnCreate false Validate when object is created
TestOnBorrow false Validate when object is borrowed
TestOnReturn false Validate when object is returned
TestWhileIdle false Validate when object is idle, see TimeBetweenEvictionRuns
BlockWhenExhausted true Whether to block when the pool is exhausted
MinEvictableIdleTime 30m Eviction configuration,see DefaultEvictionPolicy
SoftMinEvictableIdleTime math.MaxInt64 Eviction configuration,see DefaultEvictionPolicy
NumTestsPerEvictionRun 3 The maximum number of objects to examine during each run evictor goroutine
TimeBetweenEvictionRuns 0 The number of milliseconds to sleep between runs of the evictor goroutine, less than 1 mean not run

Usage

Use Simple Factory

import (
	"context"
	"fmt"
	"strconv"
	"sync/atomic"

	"github.com/jolestar/go-commons-pool/v2"
)

func Example_simple() {
	type myPoolObject struct {
		s string
	}

	v := uint64(0)
	factory := pool.NewPooledObjectFactorySimple(
		func(context.Context) (interface{}, error) {
			return &myPoolObject{
					s: strconv.FormatUint(atomic.AddUint64(&v, 1), 10),
				},
				nil
		})

	ctx := context.Background()
	p := pool.NewObjectPoolWithDefaultConfig(ctx, factory)

	obj, err := p.BorrowObject(ctx)
	if err != nil {
		panic(err)
	}

	o := obj.(*myPoolObject)
	fmt.Println(o.s)

	err = p.ReturnObject(ctx, obj)
	if err != nil {
		panic(err)
	}

	// Output: 1
}

Use Custom Factory

import (
	"context"
	"fmt"
	"strconv"
	"sync/atomic"

	"github.com/jolestar/go-commons-pool/v2"
)

type MyPoolObject struct {
	s string
}

type MyCustomFactory struct {
	v uint64
}

func (f *MyCustomFactory) MakeObject(ctx context.Context) (*pool.PooledObject, error) {
	return pool.NewPooledObject(
			&MyPoolObject{
				s: strconv.FormatUint(atomic.AddUint64(&f.v, 1), 10),
			}),
		nil
}

func (f *MyCustomFactory) DestroyObject(ctx context.Context, object *pool.PooledObject) error {
	// do destroy
	return nil
}

func (f *MyCustomFactory) ValidateObject(ctx context.Context, object *pool.PooledObject) bool {
	// do validate
	return true
}

func (f *MyCustomFactory) ActivateObject(ctx context.Context, object *pool.PooledObject) error {
	// do activate
	return nil
}

func (f *MyCustomFactory) PassivateObject(ctx context.Context, object *pool.PooledObject) error {
	// do passivate
	return nil
}

func Example_customFactory() {
	ctx := context.Background()
	p := pool.NewObjectPoolWithDefaultConfig(ctx, &MyCustomFactory{})
	p.Config.MaxTotal = 100
    
	obj1, err := p.BorrowObject(ctx)
	if err != nil {
		panic(err)
	}

	o := obj1.(*MyPoolObject)
	fmt.Println(o.s)

	err = p.ReturnObject(ctx, obj1)
	if err != nil {
		panic(err)
	}

	// Output: 1
}

For more examples please see pool_test.go and example_simple_test.go, example_customFactory_test.go.

Note

PooledObjectFactory.MakeObject must return a pointer, not value. The following code will complain error.

p := pool.NewObjectPoolWithDefaultConfig(ctx, pool.NewPooledObjectFactorySimple(
    func(context.Context) (interface{}, error) {
        return "hello", nil
    }))
obj, _ := p.BorrowObject()
p.ReturnObject(obj)

The right way is:

p := pool.NewObjectPoolWithDefaultConfig(ctx, pool.NewPooledObjectFactorySimple(
    func(context.Context) (interface{}, error) {
        s := "hello"
        return &s, nil
    }))

For more examples please see example_simple_test.go.

Dependency

PerformanceTest

The results of running the pool_perf_test is almost equal to the java version PerformanceTest

go test --perf=true

For Apache commons pool user

  • Direct use pool.Config.xxx to change pool config
  • Default config value is same as java version
  • If TimeBetweenEvictionRuns changed after ObjectPool created, should call ObjectPool.StartEvictor to take effect. Java version do this on set method.
  • No KeyedObjectPool (TODO)
  • No ProxiedObjectPool
  • No pool stats (TODO)

FAQ

FAQ

How to contribute

  • Choose one open issue you want to solve, if not create one and describe what you want to change.
  • Fork the repository on GitHub.
  • Write code to solve the issue.
  • Create PR and link to the issue.
  • Make sure test and coverage pass.
  • Wait maintainers to merge.

中文文档

License

Go Commons Pool is available under the Apache License, Version 2.0.

Issues
  • Cannot update to v2.1 when using gomod

    Cannot update to v2.1 when using gomod

    Using go get -u github.com/jolestar/[email protected] to attempt to update to 2.1 results in this error:

    go: finding github.com/jolestar/go-commons-pool v2.1.0
    go: finding github.com/jolestar/go-commons-pool v2.1.0
    go get github.com/jolestar/[email protected]: github.com/jolestar/[email protected]: invalid version: module contains a go.mod file, so major version must be compatible: should be v0 or v1, not v2
    

    To resolve this, the module needs to refer to itself as github.com/jolestar/go-commons-pool/v2, as per the documentation, and likely the commit should be tagged as a new release (v2.1.1, at least).

    opened by dcormier 6
  • dead lock when  goroutines exceeds MaxTotal

    dead lock when goroutines exceeds MaxTotal

    Seeing

    dead lock when goroutines exceeds MaxTotal:

    Environment

    go 1.11 go-commons-pool newest ubuntu 18.04

    Code

    package main
    
    import (
    	"bytes"
    	"context"
    	"crypto/md5"
    	"encoding/binary"
    	"fmt"
    	"github.com/gomodule/redigo/redis"
    	"github.com/jolestar/go-commons-pool"
    	"log"
    	"math/rand"
    	"net/http"
    	_ "net/http/pprof"
    	"strconv"
    	"sync"
    	"time"
    )
    
    func main() {
    	// dump stack info by http://localhost:6060/debug/pprof/goroutine?debug=2
    	go func() {
    		log.Println(http.ListenAndServe("localhost:6060", nil))
    	}()
    	// create commons pool
    	factory := pool.NewPooledObjectFactorySimple(
    		func(context.Context) (interface{}, error) {
    			return redis.Dial("tcp", ":6400")
    		})
    	ctx := context.TODO()
    	p := pool.NewObjectPoolWithDefaultConfig(ctx, factory)
    	p.Config.MaxTotal = 100
    	p.Config.MaxIdle = 100
    	defer p.Close(ctx)
    	// write to redis parallel
    	total := 100000
    	idCount := 10
    	// when <= 100 (MaxTotal) is OK, or else dead lock
    	parallel := 200
    	today := time.Now().Format("20060102")
    	rand.Seed(time.Now().Unix())
    	log.Println("begin write to redis")
    	for i := 0; i < total; {
    		batchWg := &sync.WaitGroup{}
    		for j := 0; j < parallel/idCount && i < total; j++ {
    			uuid := createUuid()
    			for k := 0; k < idCount; k++ {
    				idInt := rand.Intn(1000000)
    				id := strconv.Itoa(idInt)
    				field := ":" + today + ":" + id
    				batchWg.Add(1)
    				go incrCounter(batchWg, p, uuid+field)
    			}
    			i++
    		}
    		batchWg.Wait()
    	}
    	log.Println("end write to redis")
    }
    
    func incrCounter(wg *sync.WaitGroup, pool *pool.ObjectPool, key string) {
    	defer wg.Done()
    	ctx := context.TODO()
    	conn, _ := pool.BorrowObject(ctx)
    	if conn != nil {
    		defer pool.ReturnObject(ctx, conn)
    	} else {
    		return
    	}
    	redisConn := conn.(redis.Conn)
    	_, err := redisConn.Do("incr", key)
    	if err != nil {
    		log.Printf("incr error %v", err)
    	}
    }
    
    func createUuid() string {
    	randInt := rand.Int63()
    	intBytes := make([]byte, 8)
    	binary.BigEndian.PutUint64(intBytes, uint64(randInt))
    	md5Bytes := md5.Sum(intBytes)
    	buf := new(bytes.Buffer)
    	fmt.Fprintf(buf, "%x", md5Bytes)
    	return buf.String()
    }
    

    Stack Trace

    goroutine 228 [running]:
    runtime/pprof.writeGoroutineStacks(0x857620, 0xc00048e000, 0x40d62f, 0xc000392360)
    	/home/zhyhang/workdoc/dev/go/src/runtime/pprof/pprof.go:678 +0xa7
    runtime/pprof.writeGoroutine(0x857620, 0xc00048e000, 0x2, 0xae41e0, 0x0)
    	/home/zhyhang/workdoc/dev/go/src/runtime/pprof/pprof.go:667 +0x44
    runtime/pprof.(*Profile).WriteTo(0xad5d60, 0x857620, 0xc00048e000, 0x2, 0xc00048e000, 0x7f5c13)
    	/home/zhyhang/workdoc/dev/go/src/runtime/pprof/pprof.go:328 +0x3e4
    net/http/pprof.handler.ServeHTTP(0xc00037cb51, 0x9, 0x85a5c0, 0xc00048e000, 0xc0002ce000)
    	/home/zhyhang/workdoc/dev/go/src/net/http/pprof/pprof.go:245 +0x210
    net/http/pprof.Index(0x85a5c0, 0xc00048e000, 0xc0002ce000)
    	/home/zhyhang/workdoc/dev/go/src/net/http/pprof/pprof.go:268 +0x723
    net/http.HandlerFunc.ServeHTTP(0x80d5b0, 0x85a5c0, 0xc00048e000, 0xc0002ce000)
    	/home/zhyhang/workdoc/dev/go/src/net/http/server.go:1964 +0x44
    net/http.(*ServeMux).ServeHTTP(0xae3400, 0x85a5c0, 0xc00048e000, 0xc0002ce000)
    	/home/zhyhang/workdoc/dev/go/src/net/http/server.go:2361 +0x127
    net/http.serverHandler.ServeHTTP(0xc0002b60d0, 0x85a5c0, 0xc00048e000, 0xc0002ce000)
    	/home/zhyhang/workdoc/dev/go/src/net/http/server.go:2741 +0xab
    net/http.(*conn).serve(0xc0002ae8c0, 0x85a940, 0xc000371a80)
    	/home/zhyhang/workdoc/dev/go/src/net/http/server.go:1847 +0x646
    created by net/http.(*Server).Serve
    	/home/zhyhang/workdoc/dev/go/src/net/http/server.go:2851 +0x2f5
    
    goroutine 1 [semacquire]:
    sync.runtime_Semacquire(0xc00043c558)
    	/home/zhyhang/workdoc/dev/go/src/runtime/sema.go:56 +0x39
    sync.(*WaitGroup).Wait(0xc00043c550)
    	/home/zhyhang/workdoc/dev/go/src/sync/waitgroup.go:130 +0x64
    main.main()
    	/home/zhyhang/code-go/src/github.com/zhyhang/gofirst/tools/deadpool.go:56 +0x36c
    
    goroutine 20 [IO wait]:
    internal/poll.runtime_pollWait(0x7faf71b262b0, 0x72, 0x0)
    	/home/zhyhang/workdoc/dev/go/src/runtime/netpoll.go:173 +0x66
    internal/poll.(*pollDesc).wait(0xc0001d1718, 0x72, 0xc0001aa100, 0x0, 0x0)
    	/home/zhyhang/workdoc/dev/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
    internal/poll.(*pollDesc).waitRead(0xc0001d1718, 0xffffffffffffff00, 0x0, 0x0)
    	/home/zhyhang/workdoc/dev/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
    internal/poll.(*FD).Accept(0xc0001d1700, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    	/home/zhyhang/workdoc/dev/go/src/internal/poll/fd_unix.go:384 +0x1a0
    net.(*netFD).accept(0xc0001d1700, 0x40d62f, 0xc0002ae960, 0xa0)
    	/home/zhyhang/workdoc/dev/go/src/net/fd_unix.go:238 +0x42
    net.(*TCPListener).accept(0xc0002a4078, 0xc000042df0, 0x8dcf19d1, 0x6b34adfcd281e637)
    	/home/zhyhang/workdoc/dev/go/src/net/tcpsock_posix.go:139 +0x2e
    net.(*TCPListener).AcceptTCP(0xc0002a4078, 0xc000042e18, 0x491246, 0x5bbc77ec)
    	/home/zhyhang/workdoc/dev/go/src/net/tcpsock.go:247 +0x47
    net/http.tcpKeepAliveListener.Accept(0xc0002a4078, 0xc000042e68, 0x18, 0xc000062900, 0x6a6825)
    	/home/zhyhang/workdoc/dev/go/src/net/http/server.go:3232 +0x2f
    net/http.(*Server).Serve(0xc0002b60d0, 0x85a7c0, 0xc0002a4078, 0x0, 0x0)
    	/home/zhyhang/workdoc/dev/go/src/net/http/server.go:2826 +0x22f
    net/http.(*Server).ListenAndServe(0xc0002b60d0, 0xc0002b60d0, 0x0)
    	/home/zhyhang/workdoc/dev/go/src/net/http/server.go:2764 +0xb6
    net/http.ListenAndServe(0x7f66e8, 0xe, 0x0, 0x0, 0x0, 0x0)
    	/home/zhyhang/workdoc/dev/go/src/net/http/server.go:3004 +0x74
    main.main.func1()
    	/home/zhyhang/code-go/src/github.com/zhyhang/gofirst/tools/deadpool.go:23 +0x3e
    created by main.main
    	/home/zhyhang/code-go/src/github.com/zhyhang/gofirst/tools/deadpool.go:22 +0x47
    
    goroutine 229 [IO wait]:
    internal/poll.runtime_pollWait(0x7faf71aea958, 0x72, 0xc000167858)
    	/home/zhyhang/workdoc/dev/go/src/runtime/netpoll.go:173 +0x66
    internal/poll.(*pollDesc).wait(0xc0001d1f18, 0x72, 0xffffffffffffff00, 0x8581c0, 0xaa6618)
    	/home/zhyhang/workdoc/dev/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
    internal/poll.(*pollDesc).waitRead(0xc0001d1f18, 0xc000406000, 0x1000, 0x1000)
    	/home/zhyhang/workdoc/dev/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
    internal/poll.(*FD).Read(0xc0001d1f00, 0xc000406000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    	/home/zhyhang/workdoc/dev/go/src/internal/poll/fd_unix.go:169 +0x179
    net.(*netFD).Read(0xc0001d1f00, 0xc000406000, 0x1000, 0x1000, 0xc0001aba00, 0xc000167968, 0x69cff0)
    	/home/zhyhang/workdoc/dev/go/src/net/fd_unix.go:202 +0x4f
    net.(*conn).Read(0xc0002a4098, 0xc000406000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    	/home/zhyhang/workdoc/dev/go/src/net/net.go:177 +0x68
    net/http.(*connReader).Read(0xc0001cd080, 0xc000406000, 0x1000, 0x1000, 0x435c7c, 0xc0001b9680, 0x298)
    	/home/zhyhang/workdoc/dev/go/src/net/http/server.go:786 +0xfc
    bufio.(*Reader).fill(0xc0001c39e0)
    	/home/zhyhang/workdoc/dev/go/src/bufio/bufio.go:100 +0x106
    bufio.(*Reader).ReadSlice(0xc0001c39e0, 0xc0002ca00a, 0xc000167ac8, 0x40d1a3, 0x7faf71b6db30, 0x0, 0x203000)
    	/home/zhyhang/workdoc/dev/go/src/bufio/bufio.go:341 +0x36
    bufio.(*Reader).ReadLine(0xc0001c39e0, 0xc0002ca000, 0x100, 0xf8, 0x7e4760, 0x1, 0x1000000ae41e0)
    	/home/zhyhang/workdoc/dev/go/src/bufio/bufio.go:370 +0x34
    net/textproto.(*Reader).readLineSlice(0xc0001cd0b0, 0xc000404701, 0xc000167b98, 0x40deb8, 0x100, 0x7e4760)
    	/home/zhyhang/workdoc/dev/go/src/net/textproto/reader.go:55 +0x6f
    net/textproto.(*Reader).ReadLine(0xc0001cd0b0, 0xc0002ca000, 0x0, 0x0, 0x0)
    	/home/zhyhang/workdoc/dev/go/src/net/textproto/reader.go:36 +0x2b
    net/http.readRequest(0xc0001c39e0, 0x0, 0xc0002ca000, 0x0, 0x0)
    	/home/zhyhang/workdoc/dev/go/src/net/http/request.go:958 +0x8c
    net/http.(*conn).readRequest(0xc0002ae960, 0x85a940, 0xc0001ab980, 0x0, 0x0, 0x0)
    	/home/zhyhang/workdoc/dev/go/src/net/http/server.go:966 +0x162
    net/http.(*conn).serve(0xc0002ae960, 0x85a940, 0xc0001ab980)
    	/home/zhyhang/workdoc/dev/go/src/net/http/server.go:1788 +0x49e
    created by net/http.(*Server).Serve
    	/home/zhyhang/workdoc/dev/go/src/net/http/server.go:2851 +0x2f5
    
    goroutine 2642 [runnable]:
    syscall.Syscall(0x0, 0x69, 0xc000209241, 0x1, 0xffffffffffffffff, 0x0, 0xb)
    	/home/zhyhang/workdoc/dev/go/src/syscall/asm_linux_amd64.s:18 +0x5
    syscall.read(0x69, 0xc000209241, 0x1, 0x1, 0xc00005a000, 0x0, 0x0)
    	/home/zhyhang/workdoc/dev/go/src/syscall/zsyscall_linux_amd64.go:732 +0x5a
    syscall.Read(0x69, 0xc000209241, 0x1, 0x1, 0x0, 0xc000060000, 0xc00044ef00)
    	/home/zhyhang/workdoc/dev/go/src/syscall/syscall_unix.go:172 +0x49
    internal/poll.(*FD).Read(0xc0001d1e80, 0xc000209241, 0x1, 0x1, 0x0, 0x0, 0x0)
    	/home/zhyhang/workdoc/dev/go/src/internal/poll/fd_unix.go:165 +0x119
    net.(*netFD).Read(0xc0001d1e80, 0xc000209241, 0x1, 0x1, 0x1, 0xc00039adc0, 0xc00001e7b8)
    	/home/zhyhang/workdoc/dev/go/src/net/fd_unix.go:202 +0x4f
    net.(*conn).Read(0xc0002a4090, 0xc000209241, 0x1, 0x1, 0x0, 0x0, 0x0)
    	/home/zhyhang/workdoc/dev/go/src/net/net.go:177 +0x68
    net/http.(*connReader).backgroundRead(0xc000209230)
    	/home/zhyhang/workdoc/dev/go/src/net/http/server.go:676 +0x5a
    created by net/http.(*connReader).startBackgroundRead
    	/home/zhyhang/workdoc/dev/go/src/net/http/server.go:672 +0xd2
    
    goroutine 2624 [select]:
    github.com/jolestar/go-commons-pool/concurrent.(*TimeoutCond).Wait(0xc00008a420, 0x85a980, 0xc000094018, 0x0)
    	/home/zhyhang/code-go/src/github.com/jolestar/go-commons-pool/concurrent/cond.go:55 +0x14f
    github.com/jolestar/go-commons-pool/collections.(*LinkedBlockingDeque).PollFirstWithContext(0xc00009c340, 0x85a980, 0xc000094018, 0x0, 0x0, 0x0, 0x0)
    	/home/zhyhang/code-go/src/github.com/jolestar/go-commons-pool/collections/queue.go:289 +0xba
    github.com/jolestar/go-commons-pool.(*ObjectPool).borrowObject(0xc000122000, 0x85a980, 0xc000094018, 0x8, 0xc0001aab40, 0x1, 0xc00019a718)
    	/home/zhyhang/code-go/src/github.com/jolestar/go-commons-pool/pool.go:276 +0x45d
    github.com/jolestar/go-commons-pool.(*ObjectPool).BorrowObject(0xc000122000, 0x85a980, 0xc000094018, 0xc0004558c0, 0x1, 0x1, 0xc000218320)
    	/home/zhyhang/code-go/src/github.com/jolestar/go-commons-pool/pool.go:141 +0x3f
    main.incrCounter(0xc00043c550, 0xc000122000, 0xc000442f00, 0x30)
    	/home/zhyhang/code-go/src/github.com/zhyhang/gofirst/tools/deadpool.go:64 +0x8a
    created by main.main
    	/home/zhyhang/code-go/src/github.com/zhyhang/gofirst/tools/deadpool.go:52 +0x2f1
    
    bug 
    opened by zhyhang 4
  • unit test failed on respberry, 32bit ARM, linux system

    unit test failed on respberry, 32bit ARM, linux system

    --- FAIL: TestBaseInvalidateObject (0.00s) panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x12208]

    goroutine 27 [running]: testing.tRunner.func1(0x24cce60) /usr/local/go/src/testing/testing.go:792 +0x30c panic(0x2f5e68, 0x59bfb0) /usr/local/go/src/runtime/panic.go:513 +0x194 runtime/internal/atomic.goLoad64(0x250b7ac, 0x243ed44, 0x243ed4c) /usr/local/go/src/runtime/internal/atomic/atomic_arm.go:124 +0x1c github.com/jolestar/go-commons-pool/concurrent.(*TimeoutCond).HasWaiters(0x250b7a0, 0x344f10) /home/pi/go/src/github.com/jolestar/go-commons-pool/concurrent/cond.go:42 +0x24 github.com/jolestar/go-commons-pool/collections.(*LinkedBlockingDeque).HasTakeWaiters(0x250b7e0, 0x2479100) /home/pi/go/src/github.com/jolestar/go-commons-pool/collections/queue.go:441 +0x60 github.com/jolestar/go-commons-pool.(*ObjectPool).ensureIdle(0x249c720, 0x38dea0, 0x24160a8, 0x1, 0x2479000) /home/pi/go/src/github.com/jolestar/go-commons-pool/pool.go:339 +0x124 github.com/jolestar/go-commons-pool.(*ObjectPool).InvalidateObject(0x249c720, 0x38dea0, 0x24160a8, 0x2ceac0, 0x251c8c8, 0x2ceac0, 0x251c8cc) /home/pi/go/src/github.com/jolestar/go-commons-pool/pool.go:477 +0x118 github.com/jolestar/go-commons-pool.(*PoolTestSuite).TestBaseInvalidateObject(0x24ea040) /home/pi/go/src/github.com/jolestar/go-commons-pool/pool_test.go:426 +0x234 reflect.Value.call(0x2510540, 0x24ee450, 0x13, 0x32da11, 0x4, 0x24fafac, 0x1, 0x1, 0x24faf78, 0xc, ...) /usr/local/go/src/reflect/value.go:447 +0x368 reflect.Value.Call(0x2510540, 0x24ee450, 0x13, 0x24fafac, 0x1, 0x1, 0x2d1b6e3, 0x0, 0xb9930664) /usr/local/go/src/reflect/value.go:308 +0x74 github.com/jolestar/go-commons-pool/vendor/github.com/stretchr/testify/suite.Run.func2(0x24cce60) /home/pi/go/src/github.com/jolestar/go-commons-pool/vendor/github.com/stretchr/testify/suite/suite.go:95 +0x190 testing.tRunner(0x24cce60, 0x2511e80) /usr/local/go/src/testing/testing.go:827 +0xac created by testing.(*T).Run /usr/local/go/src/testing/testing.go:878 +0x2b0 exit status 2 FAIL github.com/jolestar/go-commons-pool 0.073s

    go test github.com/jolestar/go-commons-pool/concurrent panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x120a8]

    goroutine 23 [running]: runtime/internal/atomic.goXadd64(0x145310c, 0x1, 0x0, 0x0, 0x0) /usr/local/go/src/runtime/internal/atomic/atomic_arm.go:96 +0x1c github.com/jolestar/go-commons-pool/concurrent.(*TimeoutCond).addWaiter(0x1453100) /home/pi/go/src/github.com/jolestar/go-commons-pool/concurrent/cond.go:25 +0x34 github.com/jolestar/go-commons-pool/concurrent.(*TimeoutCond).Wait(0x1453100, 0x2d2690, 0x1454b00, 0x71a00) /home/pi/go/src/github.com/jolestar/go-commons-pool/concurrent/cond.go:47 +0x24 github.com/jolestar/go-commons-pool/concurrent.(*LockTestObject).lockAndWaitWithTimeout(0x140c240, 0x498d0000, 0x71afd, 0x0) /home/pi/go/src/github.com/jolestar/go-commons-pool/concurrent/cond_test.go:31 +0xc4 github.com/jolestar/go-commons-pool/concurrent.TestTimeoutCondWaitTimeoutNotify.func1(0x140c240, 0x77359400, 0x0, 0x1418e40, 0x14166f0) /home/pi/go/src/github.com/jolestar/go-commons-pool/concurrent/cond_test.go:96 +0x5c created by github.com/jolestar/go-commons-pool/concurrent.TestTimeoutCondWaitTimeoutNotify /home/pi/go/src/github.com/jolestar/go-commons-pool/concurrent/cond_test.go:94 +0x1a0 FAIL github.com/jolestar/go-commons-pool/concurrent 0.034s

    bug 
    opened by ludanfeng 3
  • Added support for contexts (including for canceling and timing out)

    Added support for contexts (including for canceling and timing out)

    In order to support more modern ways of timing out or otherwise canceling calls to borrow an object from the pool, this PR adds support for passing a context to the new method, BorrowObjectWithContext.

    opened by dcormier 3
  • deadlock in BorrowObject

    deadlock in BorrowObject

    Problem

    When creating object fails and using n goroutines (n > MaxTotal) to call BorrowObject, it will deadlock. But when I lock BorrowObject, the deadlock will not happen. Is this a bug?

    Env

    go version 1.13 windows10 go-commons-pool version v2.11

    Code

    package pool
    
    import (
    	"context"
    	"errors"
    	"fmt"
    	"github.com/jolestar/go-commons-pool/v2"
    	"sync"
    	"testing"
    	"time"
    )
    
    func TestPool(t *testing.T) {
    	factory := pool.NewPooledObjectFactorySimple(
    		func(context.Context) (interface{}, error) {
    			time.Sleep(time.Second)
    			return nil, errors.New("create object fail") // simulate the failure of creating objects
    		})
    
    	ctx := context.Background()
    	conf := pool.NewDefaultPoolConfig()
    	conf.MaxTotal = 1
    	p := pool.NewObjectPool(ctx, factory, conf)
    
    	n := 2
    	wg := sync.WaitGroup{}
    	wg.Add(n)
    	//lock := sync.Mutex{}
    	for i := 0; i < n; i++ {
    		id := i
    		go func() { // when using goroutines without locking BorrowObject, it will deadlock
    			defer wg.Done()
    			fmt.Println(id, "before borrow")
    			//lock.Lock()
    			obj, err := p.BorrowObject(ctx)
    			//lock.Unlock()
    			fmt.Println(id, "after borrow")
    			if err != nil {
    				fmt.Println(id, "borrow err:", err)
    			}
    
    			err = p.ReturnObject(ctx, obj)
    			if err != nil {
    				fmt.Println(id, "return err:", err)
    			}
    		}()
    	}
    	wg.Wait()
    }
    
    
    question 
    opened by XH-JMC 2
  • Context Support

    Context Support

    It would be nice if context.Context could be passed to methods on the pool, and also if the context was passed to the factory methods.

    I already have a PR that adds a BorrowWithContext method that will cancel based on the supplied context (#26), but this is only a first step.

    If the author is willing to break backwards compatibility (which means releasing this as v2.0.0), I can change the package to pass a context down all the way to the factory methods.

    feature 
    opened by dcormier 2
  • Added support for contexts (including for canceling and timing out)

    Added support for contexts (including for canceling and timing out)

    In order to support more modern ways of timing out or otherwise canceling function calls, this PR adds support for passing a context to the new method, BorrowObjectWithContext.

    opened by dcormier 2
  • Bump github.com/stretchr/testify from 1.7.4 to 1.7.5

    Bump github.com/stretchr/testify from 1.7.4 to 1.7.5

    Bumps github.com/stretchr/testify from 1.7.4 to 1.7.5.

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 1
  • Bump github.com/stretchr/testify from 1.7.2 to 1.7.4

    Bump github.com/stretchr/testify from 1.7.2 to 1.7.4

    Bumps github.com/stretchr/testify from 1.7.2 to 1.7.4.

    Commits
    • 48391ba Fix panic in AssertExpectations for mocks without expectations (#1207)
    • 840cb80 arrays value types in a zero-initialized state are considered empty (#1126)
    • 07dc7ee Bump actions/setup-go from 3.1.0 to 3.2.0 (#1191)
    • c33fc8d Bump actions/checkout from 2 to 3 (#1163)
    • 3c33e07 Added Go 1.18.1 as a build/supported version (#1182)
    • e2b56b3 Bump github.com/stretchr/objx from 0.1.0 to 0.4.0
    • See full diff in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 1
  • Bump github.com/stretchr/testify from 1.7.1 to 1.7.2

    Bump github.com/stretchr/testify from 1.7.1 to 1.7.2

    Bumps github.com/stretchr/testify from 1.7.1 to 1.7.2.

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 1
  • deadlock in Close

    deadlock in Close

    Problem

    Whenever eviction and close run at the same time. There is a possible deadlock because of the closeLock mutex usage on poo.go:487. We can unlock it whenever we don't need it rather than using defer.

    Repreduce UT

    func (suit *PoolTestSuite) TestConcurrentCloseAndEvict() {
    	ctx := context.Background()
    	suit.pool.Config.MinIdle = 1
    	suit.pool.Config.SoftMinEvictableIdleTime = time.Millisecond * 100
    	suit.pool.Config.TimeBetweenEvictionRuns = time.Millisecond * 500
    	suit.factory.destroyLatency = time.Millisecond * 1000 // Destroy takes 1000 ms
    	suit.pool.PreparePool(ctx)
    	suit.pool.StartEvictor()
    	suit.Equal(1, suit.pool.GetNumIdle())
    	ticker := time.NewTicker(time.Millisecond * 1000)
    	testTimeoutTicker := time.NewTicker(time.Millisecond * 5000) // if time exceeds test fails
    	go func() {
    		select {
    		case <-testTimeoutTicker.C:
    			// Time-out
    			suit.FailNow("Time is exceeds on pool close")
    		}
    	}()
    	select {
    	case <-ticker.C:
    		suit.pool.Close(ctx)
    	}
    }
    
    bug 
    opened by ykalayy 1
  • Optimization: prepare config in init-time rather than in run-time

    Optimization: prepare config in init-time rather than in run-time

    Introduction: There are some instructions in the code, that can be executed once in init-time rather than every time in running:

    1. evictionConfig is been recreated every time in eviction-phase.
    2. EvictionPolicy is being looked in map every time in eviction-phase. Map operations aren't so fast as we might think: they calculate hash, look object in buckets by the hash.
    3. Something else?

    Problem: Probably, it's not reasonable to do these operations every time, because the config is been changed hardly ever after pool is created.

    Possible solution:

    1. Create such objects once in startEvictor(). So, when you change config - just restart evictor;
    2. Make a new method to apply config changes. So, when you change config - just call the method;
    3. Another solution?
    enhancement 
    opened by PROger4ever 0
Releases(v2.1.2)
  • v2.1.2(Apr 21, 2022)

    What's Changed

    • Bump github.com/stretchr/testify from 1.4.0 to 1.6.1 by @dependabot-preview in https://github.com/jolestar/go-commons-pool/pull/51
    • Fix typo EvitionContext -> EvictionContext by @jolestar in https://github.com/jolestar/go-commons-pool/pull/53
    • Bump github.com/stretchr/testify from 1.6.1 to 1.7.0 by @dependabot-preview in https://github.com/jolestar/go-commons-pool/pull/54
    • Upgrade to GitHub-native Dependabot by @dependabot-preview in https://github.com/jolestar/go-commons-pool/pull/55
    • Bump github.com/stretchr/testify from 1.7.0 to 1.7.1 by @dependabot in https://github.com/jolestar/go-commons-pool/pull/56
    • fix deadlock #57 by @ykalayy in https://github.com/jolestar/go-commons-pool/pull/58

    New Contributors

    • @dependabot-preview made their first contribution in https://github.com/jolestar/go-commons-pool/pull/51
    • @dependabot made their first contribution in https://github.com/jolestar/go-commons-pool/pull/56
    • @ykalayy made their first contribution in https://github.com/jolestar/go-commons-pool/pull/58

    Full Changelog: https://github.com/jolestar/go-commons-pool/compare/v2.1.1...v2.1.2

    Source code(tar.gz)
    Source code(zip)
  • v2.1.1(Nov 15, 2019)

  • v2.1.0(Nov 13, 2019)

  • v2.0.0(Jun 21, 2018)

  • v1.1.2(Aug 9, 2017)

  • v1.1.1(Oct 31, 2016)

  • v1.1(Jan 23, 2016)

  • v1.0(Jan 18, 2016)

    Go commons pool rewrite the most test case of Apache commons pool, test coverage reached 90%, performance results are similar as Java version, can be used in a production environment, then released version 1.0.

    Source code(tar.gz)
    Source code(zip)
Owner
jolestar
jolestar
Go-generic-unboxing - A quick ready to ship demo for go generic using the official example

Go generic This repo contain basic demo for installing and running go1.18beta1 v

Shenouda Fawzy 1 Feb 1, 2022
[TOOL, CLI] - Filter and examine Go type structures, interfaces and their transitive dependencies and relationships. Export structural types as TypeScript value object or bare type representations.

typex Examine Go types and their transitive dependencies. Export results as TypeScript value objects (or types) declaration. Installation go get -u gi

Daniel T. Gorski 150 Jun 21, 2022
State observer - StateObserver used to synchronize the local(cached) state of the remote object with the real state

state observer StateObserver used to synchronize the local(cached) state of the

Ilya 2 Jan 19, 2022
Go worker pool

description This is a generic worker pool for the Go language. It's useful when you want to limit the number of goroutines running in parallel. instal

Ștefan Talpalaru 62 Jul 4, 2021
Optimistic rollup tech, minimal and generic.

Opti Optimistic rollup tech, minimal and generic. VERY experimental, just exploratory code, question is: 1:1 EVM rollup with interactive fraud proof p

Diederik Loerakker 13 Oct 29, 2021
Generic mapStringInterface tool for extracting of data for CSV output

Generic mapStringInterface tool for extracting of data for CSV output

tbal999 1 Nov 2, 2021
Generic slices for Go 1.8+

Slice A simple package that makes working with slices a little bit easier with the help of generics. Install go get github.com/twharmon/slice Example

Travis Harmon 5 Jun 10, 2022
Generic tools for go 1.18+

Gtools Generic tools for go 1.18+ FT (func tools) Provide func tools over iterators Iterators for functions like Filter, Map, Reduce, etc solve 3 main

null 1 Jan 12, 2022
Ecsgo - Cache friendly, Multi threading Entity Component System in Go (with Generic)

ECSGo ECSGo is an Entity Component System(ECS) in Go. This is made with Generic

Vong Kong 11 Jun 17, 2022
Go-generic - A collection of experiments using Go Generics coming out in Go 1.18

Go Generic - experiments with Go 1.18 beta Data structures: iter.Iter[T any] - l

Matthew Hall 2 Feb 15, 2022
Nune - High-performance numerical engine based on generic tensors

Nune (v0.1) Numerical engine is a library for performing numerical computation i

Lord Larker 61 May 31, 2022
Generic-list-go - Go container/list but with generics

generic-list-go Go container/list but with generics. The code is based on contai

Arne Bahlo 5 May 16, 2022
Nune-go - High-performance numerical engine based on generic tensors

Nune (v0.1) Numerical engine is a library for performing numerical computation i

Lord Larker 61 May 31, 2022
Golang CS:GO external base. Development currently halted due to compiler/runtime Golang bugs.

gogo Golang CS:GO External cheat/base. Also, my first Golang project. Wait! Development momentarily halted due to compiler/runtime bugs. Disclaimer Th

cristei 2 Jun 25, 2022
Belajar Golang Install Golang

Golang belajar Golang Install Golang = download di https://golang.org/dl/ = pilih yg Zip = extract file zipnya = buka foldernya - copy folder go = pas

Arif Fadilah 1 Nov 15, 2021
Golang-module-references - A reference for how to setup a Golang project with modules - Task Management + Math Examples

Golang Module Project The purpose of this project is to act as a reference for setting up future Golang projects using modules. This project has a mat

Bob Bass 0 Jan 2, 2022
Golang-echo-sample - Make an out-of-the-box backend based on golang-echo

Golang-echo-sample - Make an out-of-the-box backend based on golang-echo

Haitwang 0 Dec 31, 2021
Minimalistic, pluggable Golang evloop/timer handler with dependency-injection

Anagent Minimalistic, pluggable Golang evloop/timer handler with dependency-injection - based on codegangsta/inject - go-macaron/inject and chuckpresl

Ettore Di Giacinto 14 Jan 26, 2022
GoLang Library for Browser Capabilities Project

Browser Capabilities GoLang Project PHP has get_browser() function which tells what the user's browser is capable of. You can check original documenta

Maksim N. 40 Jun 13, 2022