Cache Slow Database Queries

Overview

dataframe-go

Cache Slow Database Queries

This package is used to cache the results of slow database queries in memory or Redis. It can be used to cache any form of data. A Redis and in-memory storage driver is provided.

See Article for further details including a tutorial.

The package is production ready and the API is stable. A variant of this package has been used in production for over 3 years.

the project to show your appreciation.

Installation

go get -u github.com/rocketlaunchr/remember-go

Create a Key

Let’s assume the query’s argument is an arbitrary search term and a page number for pagination.

CreateKeyStruct

CreateKeyStruct can generate a JSON based key by providing a struct.

type Key struct {
    Search string
    Page   int `json:"page"`
}

var key string = remember.CreateKeyStruct(Key{"golang", 2})

CreateKey

CreateKey provides more flexibility to generate keys:

// Key will be "search-golang-2"
key :=  remember.CreateKey(false, "-", "search-x-y", "search", "golang", 2)

Initialize the Storage Driver

In-Memory

import "github.com/rocketlaunchr/remember-go/memory"

var ms = memory.NewMemoryStore(10 * time.Minute)

Redis

The Redis storage driver relies on Gary Burd’s excellent Redis client library.

import red "github.com/rocketlaunchr/remember-go/redis"
import "github.com/gomodule/redigo/redis"

var rs = red.NewRedisStore(&redis.Pool{
    Dial: func() (redis.Conn, error) {
        return redis.Dial("tcp", "localhost:6379")
    },
})

Memcached

An experimental (and untested) memcached driver is provided. It relies on Brad Fitzpatrick's memcache driver.

Ristretto

DGraph's Ristretto is a fast, fixed size, in-memory cache with a dual focus on throughput and hit ratio performance.

Create a SlowRetrieve Function

The package initially checks if data exists in the cache. If it doesn’t, then it elegantly fetches the data directly from the database by calling the SlowRetrieve function. It then saves the data into the cache so that next time it doesn’t have to refetch it from the database.

type Result struct {
    Title string
}

slowQuery := func(ctx context.Context) (interface{}, error) {
    results := []Result{}

    stmt := `
        SELECT title
        FROM books WHERE title LIKE ?
        ORDER BY title LIMIT ?, 20
    `

    rows, _ := db.QueryContext(ctx, stmt, search, (page-1)*20)

    for rows.Next() {
        var title string
        rows.Scan(&title)
        results = append(results, Result{title})
    }

    return results, nil
}

Usage

key := remember.CreateKeyStruct(Key{"golang", 2})
exp := 10*time.Minute

results, found, err := remember.Cache(ctx, ms, key, exp, slowQuery, remember.Options{GobRegister: false})

return results.([]Result) // Type assert in order to use

Gob Register Errors

The Redis storage driver stores the data in a gob encoded form. You have to register with the gob package the data type returned by the SlowRetrieve function. It can be done inside a func init(). Alternatively, you can set the GobRegister option to true. This will slightly impact concurrency performance however.

Other useful packages

  • dataframe-go - For statistics, machine-learning, and data manipulation/exploration
  • dbq - Zero boilerplate database operations for Go
  • electron-alert - SweetAlert2 for Electron Applications
  • igo - A Go transpiler with cool new syntax such as fordefer (defer for for-loops)
  • mysql-go - Properly cancel slow MySQL queries
  • react - Build front end applications using Go

Legal Information

The license is a modified MIT license. Refer to LICENSE file for more details.

© 2019-20 PJ Engineering and Business Solutions Pty. Ltd.

Final Notes

Feel free to enhance features by issuing pull-requests.

You might also like...
Slow and unreliable JSON parser generator (in progress)

VivaceJSON Fast and reliable JSON parser generator Todo List parse fields parse types generate struct generate (keypath+key) to struct Value Mapping F

bf.go - A dead simple brainfuck interpreter Slow and simple

bf.go - A dead simple brainfuck interpreter Slow and simple. Can execute pretty much all tested Brainfuck scripts. Installation If you have Go install

Simple program that uploads large files to Amazon S3 over slow connections.

shrimp is a small program that can reliably upload large files to Amazon S3. My personal use case is to upload large files to S3 over a slow residenti

parse mysql slow log file

MSLP Tool For Parse MySQL Slow Log File And Send Via DingTalk Usage ./mslp --help Usage of ./mslp: -exclude-db string db1,db2,... exclude th

Fsyncperf - A very simple program to tell how fast/slow is fsync on your disk

fsyncperf This is a very simple program to tell you who fast/slow is fsync on yo

"Go SQL DB" is a relational database that supports SQL queries for research purposes

A pure golang SQL database for database theory research

A simple Golang-based application that queries a PostgreSQL database

Qwik-E-Mart Demo App A simple Golang-based application that queries a PostgreSQL database named qwikemart to read and return customer data stored in t

Structscanner is a simple library to make going from database queries to structs easier

structscanner is a simple library to make going from database queries to structs easier, while retaining the flexibility of joins and mapping using struct tags.

Cache library for golang. It supports expirable Cache, LFU, LRU and ARC.
Cache library for golang. It supports expirable Cache, LFU, LRU and ARC.

GCache Cache library for golang. It supports expirable Cache, LFU, LRU and ARC. Features Supports expirable Cache, LFU, LRU and ARC. Goroutine safe. S

A high performance gin middleware to cache http response. Compared to gin-contrib/cache, It has a huge performance improvement. 高性能gin缓存中间件,相比于官方版本,有明显性能提升。
Package cache is a middleware that provides the cache management for Flamego.

cache Package cache is a middleware that provides the cache management for Flamego. Installation The minimum requirement of Go is 1.16. go get github.

A mem cache base on other populator cache, add following feacture

memcache a mem cache base on other populator cache, add following feacture add lazy load(using expired data, and load it asynchronous) add singlefligh

A server for TurboRepo Remote Cache to store cache artefacts in Google Cloud Storage or Amazon S3

Tapico Turborepo Remote Cache This is an implementation of Vercel's Turborepo Remote Cache API endpoints used by the turborepo CLI command. This solut

Cache - A simple cache implementation

Cache A simple cache implementation LRU Cache An in memory cache implementation

Gin-cache - Gin cache middleware with golang

Gin-cache - Gin cache middleware with golang

🧩 Redify is the optimized key-value proxy for quick access and cache of any other database throught Redis and/or HTTP protocol.

Redify (Any database as redis) License Apache 2.0 Redify is the optimized key-value proxy for quick access and cache of any other database throught Re

Go native library for fast point tracking and K-Nearest queries

Geo Index Geo Index library Overview Splits the earth surface in a grid. At each cell we can store data, such as list of points, count of points, etc.

A package for Go that can be used for range queries on large number of intervals

go-stree go-stree is a package for Go that can be used to process a large number of intervals. The main purpose of this module is to solve the followi

Write your SQL queries in raw files with all benefits of modern IDEs, use them in an easy way inside your application with all the profit of compile time constants

About qry is a general purpose library for storing your raw database queries in .sql files with all benefits of modern IDEs, instead of strings and co

Releases(v1.0.1)
Owner
null
Go native library for fast point tracking and K-Nearest queries

Geo Index Geo Index library Overview Splits the earth surface in a grid. At each cell we can store data, such as list of points, count of points, etc.

Hailo Network IP Ltd 344 Dec 3, 2022
A package for Go that can be used for range queries on large number of intervals

go-stree go-stree is a package for Go that can be used to process a large number of intervals. The main purpose of this module is to solve the followi

Thomas Oberndörfer 39 May 14, 2022
Fast in-memory key:value store/cache with TTL

MCache library go-mcache - this is a fast key:value storage. Its major advantage is that, being essentially a thread-safe . map[string]interface{} wit

O.J 86 Nov 11, 2022
Golang LRU cache

golang-lru This provides the lru package which implements a fixed-size thread safe LRU cache. It is based on the cache in Groupcache. Documentation Fu

HashiCorp 3.3k Dec 29, 2022
A fast little LRU cache for Go

tinylru A fast little LRU cache. Getting Started Installing To start using tinylru, install Go and run go get: $ go get -u github.com/tidwall/tinylru

Josh Baker 128 Dec 24, 2022
☔️ A complete Go cache library that brings you multiple ways of managing your caches

Gocache Guess what is Gocache? a Go cache library. This is an extendable cache library that brings you a lot of features for caching data. Overview He

Vincent Composieux 1.6k Jan 1, 2023
A skip list of arbitrary elements that can be filtered using roaring bitmaps stored in an LRU cache

Skipfilter This package provides a data structure that combines a skiplist with a roaring bitmap cache. This library was created to efficiently filter

Kevin Burns 22 Aug 4, 2022
A threadsafe single-value cache for Go with a simple but flexible API

SVCache SVCache is a threadsafe, single-value cache with a simple but flexible API. When there is no fresh value in the cache, an attempt to retrieve

softwaretechnik.berlin 1 Jan 23, 2022
This Service collects slow queries and returns them in response.

pgsql-api-go This Service collects slow queries and returns them in response. Status This service is the very first version of the project. App is up

ngh 0 Dec 30, 2021
A database connection wrapper to cache prepared statements by transforming queries to use with array arguments.

sqlpp sqlpp is a sql(MySQL and PostgreSQL) database connection wrapper to cache prepared statements by transforming queries ("...in (?)...", []) to us

Nazım Parlar 9 Feb 9, 2022