A river for elasticsearch to automatically index mysql content using the replication feed.

Overview

Mysql River Plugin for ElasticSearch

The Mysql River plugin allows to hook into Mysql replication feed using the excellent python-mysql-replication and automatically index it into elasticsearch.

This plugin is based on the elasticsearch-river-couchdb plugin.

Installation

This river consists of two parts :

  • one is the elasticsearch plugin
  • the other one is a little webserver to serve an http stream of the mysql replication stream.

The river

To install this plugin, first build it using mvn clean package. Then, use the plugin binary distributed with elasticsearch /path/to/elasticsearch/bin/plugin -url file:./target/releases/elasticsearch-river-mysql-*-SNAPSHOT.zip -install mysql-river

Mysql replication stream

You first need to install dependencies :

  • easy_install pymysql
  • easy_install cherrypy
  • git clone git://github.com/noplay/python-mysql-replication.git ; cd python-mysql-replication ; python setup.py install

Running the river

  • python http_stream/http_stream.py
  • /path/to/elasticsearch/bin/elasticsearch

Plugin activation and configuration

curl -XPUT 'localhost:9200/_river/mydb/_meta' -d '{ "type" : "mysql", "streamer" : { "host" : "localhost", "port" : 8080, }, "mysql" : { # Future usage to configure mysql access from the river. _ } }'

Editing the plugin

To edit the plugin using eclipse, run mvn eclipse:eclipse, then you can import the current directory as an existing project in eclipse.

Bulking

Bulking is automatically done in order to speed up the indexing process. If within the specified bulk_timeout more changes are detected, changes will be bulked up to bulk_size before they are indexed.

License

This software is licensed under the Apache 2 license, quoted below.

Copyright 2009-2012 Shay Banon and ElasticSearch <http://www.elasticsearch.org>

Licensed under the Apache License, Version 2.0 (the "License"); you may not
use this file except in compliance with the License. You may obtain a copy of
the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations under
the License.
You might also like...
Go-fiber - Implement CRUD Data Go and Mysql using Authentication & Authorization

Implement CRUD Data Go and Mysql using Authentication & Authorization

A demo project that automatically restarts with a trio of docker, redis and go and transmits page visits.
A demo project that automatically restarts with a trio of docker, redis and go and transmits page visits.

A demo project that automatically restarts with a trio of docker, redis and go and transmits page visits.

a powerful mysql toolset with Go
a powerful mysql toolset with Go

go-mysql A pure go library to handle MySQL network protocol and replication. Call for Committer/Maintainer Sorry that I have no enough time to maintai

A high-performance MySQL proxy

kingshard 中文主页 Overview kingshard is a high-performance proxy for MySQL powered by Go. Just like other mysql proxies, you can use it to split the read

Vitess is a database clustering system for horizontal scaling of MySQL.

Vitess Vitess is a database clustering system for horizontal scaling of MySQL through generalized sharding. By encapsulating shard-routing logic, Vite

db-recovery is a tool for recovering MySQL data.

db-recovery is a tool for recovering MySQL data. It is used in scenarios where the database has no backup or binlog. It can parse data files and redo/undo logs to recover data.

一个使 mysql,pgsql 数据库表自动生成 go struct 的工具

db2go 一个使 mysql、pgsql 数据库表自动生成 go struct 的工具 快速使用 将项目放入到GOPATH/src目录下

🐳 A most popular sql audit platform for mysql
🐳 A most popular sql audit platform for mysql

🐳 A most popular sql audit platform for mysql

Dumpling is a fast, easy-to-use tool written by Go for dumping data from the database(MySQL, TiDB...) to local/cloud(S3, GCP...) in multifarious formats(SQL, CSV...).

🥟 Dumpling Dumpling is a tool and a Go library for creating SQL dump from a MySQL-compatible database. It is intended to replace mysqldump and mydump

Comments
  • python http_stream/http_stream.py error

    python http_stream/http_stream.py error

    Traceback (most recent call last): File "http_stream/http_stream.py", line 43, in cherrypy.quickstart(Streamer()) File "http_stream/http_stream.py", line 16, in init self.stream = BinLogStreamReader(connection_settings = mysql_settings, only_events = [DeleteRowsEvent, WriteRowsEvent, UpdateRowsEvent], blocking = True, resume_stream = True) TypeError: init() takes at least 3 non-keyword arguments (2 given)

    Python 2.6.6 PyMySQL-0.7.0-py2.6.egg CherryPy-4.0.0-py2.6.egg

    opened by kala55 1
Owner
null
MySQL replication topology management and HA

orchestrator [Documentation] orchestrator is a MySQL high availability and replication management tool, runs as a service and provides command line ac

null 4.7k Sep 22, 2022
MySQL replication topology manager - agent (daemon)

orchestrator-agent MySQL topology agent (daemon) orchestrator-agent is a sub-project of orchestrator. It is a service that runs on MySQL hosts and com

GitHub 51 Mar 8, 2022
Sync MySQL data into elasticsearch

go-mysql-elasticsearch is a service syncing your MySQL data into Elasticsearch automatically. It uses mysqldump to fetch the origin data at first, the

siddontang 3.8k Sep 21, 2022
Enhanced PostgreSQL logical replication

pgcat - Enhanced postgresql logical replication Why pgcat? Architecture Build from source Install Run Conflict handling Table mapping Replication iden

jinhua luo 357 Sep 10, 2022
Streaming replication for SQLite.

Litestream Litestream is a standalone streaming replication tool for SQLite. It runs as a background process and safely replicates changes incremental

Ben Johnson 7.4k Sep 24, 2022
mysql to mysql 轻量级多线程的库表数据同步

goMysqlSync golang mysql to mysql 轻量级多线程库表级数据同步 测试运行 设置当前binlog位置并且开始运行 go run main.go -position mysql-bin.000001 1 1619431429 查询当前binlog位置,参数n为秒数,查询结

null 14 Aug 17, 2022
convert sql to elasticsearch DSL in golang(go)

_____ _ _ ____ _____ ___ ____ ____ ___ _ | ____| | / \ / ___|_ _|_ _|/ ___|/ ___| / _ \ | | | _| | | / _ \ \___ \ |

Xargin 962 Sep 27, 2022
Zinc Search engine. A lightweight alternative to elasticsearch that requires minimal resources, written in Go.

Zinc Zinc is a search engine that does full text indexing. It is a lightweight alternative to elasticsearch and runs in less than 100 MB of RAM. It us

Prabhat Sharma 12.3k Sep 28, 2022
This package can parse date match expression, which used by ElasticSearch

datemath-parser this package is pure go package, this package can parse date match expression, which used by ElasticSearch. Date Math Definition you c

zhuliquan 0 Jan 8, 2022
CRUD API example is written in Go using net/http package and MySQL database.

GoCrudBook CRUD API example is written in Go using net/http package and MySQL database. Requirements Go MySQL Code Editor Project Structure GoCrudBook

Serhat Karabulut 4 Aug 1, 2022