MySQL to Redis caching made easy

Overview

redisql

GoDoc

MySQL to Redis caching made easy

Example Usage

CLI

Installation:

go install github.com/DGKSK8LIFE/redisql/redisql

Configuration:

Create a YAML file with the following structure:

sqluser: 
sqlpassword: 
sqldatabase:
sqltable:
redisaddr:
redispass:
log:

Usage:

redisql copy -config=pathtofile.yml 

Library

Installation:

go get github.com/DGKSK8LIFE/redisql

Usage:

package main

import (
    "github.com/DGKSK8LIFE/redisql"
)

func main() {
	config := redisql.Config{
		SQLUser:     "josh",
		SQLPassword: "joshmark52",
		SQLDatabase: "celebrities",
		SQLTable:    "celebrity",
		RedisAddr:   "localhost:6379",
		RedisPass:   "joshmark52",
		Log:         true,
	}
	err := config.Copy()
	if err != nil {
		panic(err)
	}
}

Current Functionality and Limitations

  • Simple copying of entire MySQL tables to Redis via CLI and Go Module
  • Improved logs (optional CLI output, improved formatting)
  • Support for multiple Redis data types (lists, sets, etc)
  • Support for other SQL servers
Issues
  • [FEATURE REQUEST] Add Docker support for redisql so that things like GitHub Actions can be used for CI/CD; and IDE setup is seamless.

    [FEATURE REQUEST] Add Docker support for redisql so that things like GitHub Actions can be used for CI/CD; and IDE setup is seamless.

    Describe the solution you'd like

    Provide a docker compose solution so a maintainer or user can run this box in isolation of needing PostgreSQL/MySQL and Redis to test this.

    Describe alternatives you've considered

    A docker image could be appropriate for testing which combines Go, MySQL, PostgreSQL and/or Redis in one box, however it would require maintenance of upstream Docker images and there's no performance penalty using docker compose, a docker-compose template allows maintainers to just completely retrash/rebuild their development environment by services.

    enhancement help wanted good first issue Hacktoberfest 
    opened by stemount 4
  • [BUG] Forgot to remove

    [BUG] Forgot to remove "Improved Logs" in README roadmap.

    Expected Behavior

    image

    • The highlighted list item should be removed as I've since removed logging entirely from the library.

    Current Behavior

    • The highlighted list item is not removed
    done 
    opened by DGKSK8LIFE 3
  • use redis pipeline to write CopyToString

    use redis pipeline to write CopyToString

    To solve issue #45

    before data pipelining

    > go test -v -db mysql -rows 10000
    Preparing Test...
    === RUN   TestCopyToString
    --- PASS: TestCopyToString (11.00s)
    === RUN   TestCopyToList
    --- PASS: TestCopyToList (1.25s)
    === RUN   TestCopyToHash
    --- PASS: TestCopyToHash (1.25s)
    PASS
    ok      github.com/DGKSK8LIFE/redisql   18.056s
    

    after data pipelining

    > go test -v -db mysql -rows 10000
    Preparing Test...
    === RUN   TestCopyToString
    --- PASS: TestCopyToString (1.31s)
    === RUN   TestCopyToList
    --- PASS: TestCopyToList (1.05s)
    === RUN   TestCopyToHash
    --- PASS: TestCopyToHash (1.15s)
    PASS
    ok      github.com/DGKSK8LIFE/redisql   7.865s
    
    enhancement Hacktoberfest performance hacktoberfest-accepted 
    opened by Fajrinmk 3
  • [BUG] CopyToString is over 100% slower than other methods

    [BUG] CopyToString is over 100% slower than other methods

    Expected Behavior

    Performance in CopyToString should be comparable to that of CopyToList and CopyToHash.

    Current Behavior

    CopyToString is over 100% slower than the aforementioned two.

    Possible Solution

    Because of Redis' data structures, CopyToString (strings) has to write one key per field, compared to CopyToList and CopyToHash that both write one key per row. This is not necessarily something that can be "quick fixed", but with other performance optimizations, especially transactions and pipelining, performance should improve drastically.

    Steps to Reproduce

    1. Run test cases with ~10k rows or so and your preferred DB.
    2. View output:
    ➜  redisql git:(master) go test -v -db mysql -rows 10000 
    Preparing Test...
    === RUN   TestCopyToString
    --- PASS: TestCopyToString (8.43s)
    === RUN   TestCopyToList
    --- PASS: TestCopyToList (0.75s)
    === RUN   TestCopyToHash
    --- PASS: TestCopyToHash (0.92s)
    PASS
    ok      github.com/DGKSK8LIFE/redisql   17.995s
    
    done 
    opened by DGKSK8LIFE 2
  • Added logging with levels.

    Added logging with levels.

    Issue #50: reimplement logging.

    • Initialize it by setting the logging level you want displayed.
    • Create Log outputs with a level for when it should be displayed.
    • Set file output.

    I didn't write any logging in the copying part since that will be affected by the Redis Pipeline change.

    I set it to 3 output levels because I thought that seemed reasonable. Another option could be to have different categories, like Reads, Writes, Actions, etc.

    enhancement Hacktoberfest performance 
    opened by cappe987 4
  • [FEATURE REQUEST] Add Docker support for redisql so that things like GitHub Actions can be used for CI/CD; and IDE setup is seamless.

    [FEATURE REQUEST] Add Docker support for redisql so that things like GitHub Actions can be used for CI/CD; and IDE setup is seamless.

    Describe the solution you'd like

    Provide a docker compose solution so a maintainer or user can run this box in isolation of needing PostgreSQL/MySQL and Redis to test this.

    Describe alternatives you've considered

    A docker image could be appropriate for testing which combines Go, MySQL, PostgreSQL and/or Redis in one box, however it would require maintenance of upstream Docker images and there's no performance penalty using docker compose, a docker-compose template allows maintainers to just completely retrash/rebuild their development environment by services.

    enhancement help wanted good first issue Hacktoberfest 
    opened by stemount 4
  • [BUG] Forgot to remove

    [BUG] Forgot to remove "Improved Logs" in README roadmap.

    Expected Behavior

    image

    • The highlighted list item should be removed as I've since removed logging entirely from the library.

    Current Behavior

    • The highlighted list item is not removed
    done 
    opened by DGKSK8LIFE 3
  • use redis pipeline to write CopyToString

    use redis pipeline to write CopyToString

    To solve issue #45

    before data pipelining

    > go test -v -db mysql -rows 10000
    Preparing Test...
    === RUN   TestCopyToString
    --- PASS: TestCopyToString (11.00s)
    === RUN   TestCopyToList
    --- PASS: TestCopyToList (1.25s)
    === RUN   TestCopyToHash
    --- PASS: TestCopyToHash (1.25s)
    PASS
    ok      github.com/DGKSK8LIFE/redisql   18.056s
    

    after data pipelining

    > go test -v -db mysql -rows 10000
    Preparing Test...
    === RUN   TestCopyToString
    --- PASS: TestCopyToString (1.31s)
    === RUN   TestCopyToList
    --- PASS: TestCopyToList (1.05s)
    === RUN   TestCopyToHash
    --- PASS: TestCopyToHash (1.15s)
    PASS
    ok      github.com/DGKSK8LIFE/redisql   7.865s
    
    enhancement Hacktoberfest performance hacktoberfest-accepted 
    opened by Fajrinmk 3
  • [BUG] CopyToString is over 100% slower than other methods

    [BUG] CopyToString is over 100% slower than other methods

    Expected Behavior

    Performance in CopyToString should be comparable to that of CopyToList and CopyToHash.

    Current Behavior

    CopyToString is over 100% slower than the aforementioned two.

    Possible Solution

    Because of Redis' data structures, CopyToString (strings) has to write one key per field, compared to CopyToList and CopyToHash that both write one key per row. This is not necessarily something that can be "quick fixed", but with other performance optimizations, especially transactions and pipelining, performance should improve drastically.

    Steps to Reproduce

    1. Run test cases with ~10k rows or so and your preferred DB.
    2. View output:
    ➜  redisql git:(master) go test -v -db mysql -rows 10000 
    Preparing Test...
    === RUN   TestCopyToString
    --- PASS: TestCopyToString (8.43s)
    === RUN   TestCopyToList
    --- PASS: TestCopyToList (0.75s)
    === RUN   TestCopyToHash
    --- PASS: TestCopyToHash (0.92s)
    PASS
    ok      github.com/DGKSK8LIFE/redisql   17.995s
    
    done 
    opened by DGKSK8LIFE 2
Releases(v1.0.3)
  • v1.0.3(Nov 28, 2021)

    New Features

    1. Reimplemented logging with levels, optimized logging system as a whole.
    2. Huge performance optimizations via custom pipelining and chunks.

    Features Yet to Be Completed

    1. Autosync
    2. Dockerization
    Source code(tar.gz)
    Source code(zip)
  • v2.0.0(Sep 29, 2021)

    Major Release v2.0.0

    New Features/Patches

    • Created issue + PR templates (#42)
    • POSTGRES SUPPORT LES GOOO!
    • Replicated test cases for Postgres (#41)
    • Moved to feature branches over the monolithic unstable branch for new feature contributions (#40)
    • Added CONTRIBUTING.md (#39)
    • Made project icon/image (refactored multiple times; #38)
    • Added Discord link in README (#35)
    • Added Test Harness (unit testing) thanks to @stemount (#33)

    Features Removed

    • Logging in its entirety; was a large bottleneck and didn't have much value to the end-user so rethinking it atm

    TODO

    • Performance optimizations
    • Exploring Redis Streams/PubSub
    • Autosync (#23)
    Source code(tar.gz)
    Source code(zip)
  • v1.0.2(Aug 23, 2021)

    Release 1.0.2

    Features Added

    • Improved logs with unique characters (fixes #26)

    Features To Be Added

    1. Support for other SQL servers
    2. Autosync (if I can do it in a performant way)
    Source code(tar.gz)
    Source code(zip)
  • v1.0.1(Jul 26, 2021)

  • v1.0.0(Jul 20, 2021)

    Release 1.0

    New Features

    1. Improved logs with optional command line output
    2. Proper CLI tool with YAML configuration
    3. Support for Redis strings, lists, and hashes

    Features to be implemented

    1. Support for other SQL servers
    Source code(tar.gz)
    Source code(zip)
  • v0.2(Jul 2, 2021)

Owner
Tarek Ali
Open source developer, student, freelancer.
Tarek Ali
POC de caching en Go en utilisant go-redis/cache

Test-web POC de caching en Go en utilisant go-redis/cache, cette lib permet d'avoir un cache local et un cache redis (appel cache local puis cache red

HUANG Manutea 0 Nov 19, 2021
LFU Redis implements LFU Cache algorithm using Redis as data storage

LFU Redis cache library for Golang LFU Redis implements LFU Cache algorithm using Redis as data storage LFU Redis Package gives you control over Cache

Mohamed Shapan 6 Apr 23, 2021
Concurrency-safe Go caching library with expiration capabilities and access counters

cache2go Concurrency-safe golang caching library with expiration capabilities. Installation Make sure you have a working Go environment (Go 1.2 or hig

Christian Muehlhaeuser 1.8k Jun 23, 2022
groupcache is a caching and cache-filling library, intended as a replacement for memcached in many cases.

groupcache Summary groupcache is a distributed caching and cache-filling library, intended as a replacement for a pool of memcached nodes in many case

Go 11.5k Jun 26, 2022
Multi-level caching service in Go

IgoVIUM Multi-level caching service in Go. Specifically: Distributed in-memory cache (L1) DB-based cache (L2) Long term historization on persistent vo

Andrea 7 Dec 23, 2021
API Cache is a simple caching server, using grpc to accept messages.

API Cache is a simple caching server, using grpc to accept messages. It allows to store key-value pairs, where key is string and value is []byte.

null 1 Nov 16, 2021
Design and Implement an in-memory caching library for general use

Cache Implementation in GoLang Problem Statement Design and Implement an in-memory caching library for general use. Must Have Support for multiple Sta

null 1 Dec 28, 2021
A simple generic in-memory caching layer

sc sc is a simple in-memory caching layer for golang. Usage Wrap your function with sc - it will automatically cache the values for specified amount o

motoki317 7 Jun 27, 2022
🧩 Redify is the optimized key-value proxy for quick access and cache of any other database throught Redis and/or HTTP protocol.

Redify (Any database as redis) License Apache 2.0 Redify is the optimized key-value proxy for quick access and cache of any other database throught Re

Dmitry Ponomarev 2 Mar 21, 2022
Minicache - Distributed cache implemented in Go. Like Redis but simpler.

Distributed cache with client-side consistent hashing, distributed leader-elections, and dynamic node discovery. Supports both HTTP/gRPC interfaces secured with mTLS.

null 261 Jun 25, 2022
Go-mysql-orm - Golang mysql orm,dedicated to easy use of mysql

golang mysql orm 个人学习项目, 一个易于使用的mysql-orm mapping struct to mysql table golang结构

magacy 88 Jun 27, 2022
Redis-shake is a tool for synchronizing data between two redis databases. Redis-shake是一个用于在两个redis之间同步数据的工具,满足用户非常灵活的同步、迁移需求。

RedisShake is mainly used to synchronize data from one redis to another. Thanks to the Douyu's WSD team for the support. 中文文档 English tutorial 中文使用文档

Alibaba 2.4k Jun 24, 2022
Golang mysql orm, a personal learning project, dedicated to easy use of mysql

golang mysql orm 个人学习项目, 一个易于使用的mysql-orm mapping struct to mysql table golang结构

magacy 1 Dec 30, 2021
Redis caching layer for Cloudflare KV in Golang

Redis caching layer for Cloudflare KV in Golang

Dinesh Sonachalam 30 May 21, 2022
Schema based, typed Redis caching/memoize framework for Go

cacheme - Redis Caching Framework For Go English | 中文 Statically Typed - 100% statically typed using code generation. Scale Efficiently - thundering h

null 20 Jun 14, 2022
POC de caching en Go en utilisant go-redis/cache

Test-web POC de caching en Go en utilisant go-redis/cache, cette lib permet d'avoir un cache local et un cache redis (appel cache local puis cache red

HUANG Manutea 0 Nov 19, 2021
☔🎀 Translation made with simplicity, yet robust. ~ Backend portion of Arisu, made in Go.

☔ Tsubaki Website • Discord • Discussions Translation made with simplicity, yet robust. Made with ?? using TypeScript, React with Next.js. Tsubaki is

Arisu 0 Jan 13, 2022
Go MySQL Driver is a MySQL driver for Go's (golang) database/sql package

Go-MySQL-Driver A MySQL-Driver for Go's database/sql package Features Requirements Installation Usage DSN (Data Source Name) Password Protocol Address

Go SQL Drivers 12.4k Jun 30, 2022
mysql to mysql 轻量级多线程的库表数据同步

goMysqlSync golang mysql to mysql 轻量级多线程库表级数据同步 测试运行 设置当前binlog位置并且开始运行 go run main.go -position mysql-bin.000001 1 1619431429 查询当前binlog位置,参数n为秒数,查询结

null 13 Jun 14, 2022
Quick and dirty test to compare MySQL perf between ARM64 & Rosetta MySQL on M1Pro mac

Quick and dirty test to compare MySQL perf between ARM64 & Rosetta MySQL on M1Pro mac

Chris Atkins 0 Nov 5, 2021
Beerus-DB: a database operation framework, currently only supports Mysql, Use [go-sql-driver/mysql] to do database connection and basic operations

Beerus-DB · Beerus-DB is a database operation framework, currently only supports Mysql, Use [go-sql-driver/mysql] to do database connection and basic

Beerus 6 Mar 5, 2022
a url shortener made using go and redis

__ | |--.--.--.----.-----. .-----.-----. | <| | | _|-- __|__| _ | _ | |__|__|_____|__| |_____|__|___

André Kelpe 76 Dec 10, 2020
🔎Sniffing and parsing mysql,redis,http,mongodb etc protocol. 抓包截取项目中的数据库请求并解析成相应的语句。

go-sniffer Capture mysql,redis,http,mongodb etc protocol... 抓包截取项目中的数据库请求并解析成相应的语句,如mysql协议会解析为sql语句,便于调试。 不要修改代码,直接嗅探项目中的数据请求。 中文使用说明 Support List: m

Four 1.6k Jun 29, 2022
Docker LNMP (Nginx, PHP7/PHP5, MySQL, Redis)

DNMP(Docker + Nginx + MySQL + PHP7/5 + Redis)是一款全功能的LNMP一键安装程序。 使用前最好提前阅读一遍目录,以便快速上手,遇到问题也能及时排除。

Gary Meng 3.1k Jun 26, 2022
基于Golang的框架Gin开发,项目结构和理念参考Laravel。现支持:MySQL、Redis、MVC、拦截器、助手函数、fresh热更、swagger-UI、tpl模版输出、安全的Api。

GinLaravel的构建和运行周期与Beego、Vue、React、Laravel、ThinkPHP、Django等都会有类似的引导思路、参数设置、插件扩展、服务部署、代码统一性、生态护城河等。

方圆 37 Jun 16, 2022
WAL-G is an archival restoration tool for PostgreSQL, MySQL/MariaDB, and MS SQL Server (beta for MongoDB and Redis).

WAL-G is an archival restoration tool for PostgreSQL, MySQL/MariaDB, and MS SQL Server (beta for MongoDB and Redis).

null 2.2k Jun 28, 2022
null 949 Jun 28, 2022
Examples and code to assign a name to your MongoDB, MySQL, PostgreSQL, RabbitMQ, and redis connection.

your connection deserves a name ?? When your app interacts with an external system, assign a name to the connection. An external system in this contex

Andy Grunwald 25 Feb 15, 2022
Devcloud-go provides a sql-driver for mysql which named devspore driver and a redis client which named devspore client,

Devcloud-go Devcloud-go provides a sql-driver for mysql which named devspore driver and a redis client which named devspore client, you can use them w

HUAWEI CLOUD 11 Jun 9, 2022