d18n is a data desensitization tool for RDBMS.

Related tags

Database tools d18n
Overview

Introduction

Language

logo

d18n is a numeronym short for "data-desensitization", sounds like "d-eighteen-n".

As its name says, d18n can mask data to make it desensitized. In addition, d18n can do many other things.

  • d18n is a portable RDBMS cmd client. e.g., MySQL, PostgreSQL, Oracle, SQL Server ...
  • save query result into a file, e.g., xlsx, csv, txt, sql, html, json ...
  • detect sensitive info (like PII) from a file or a SQL query.
  • import data from files into different types of databases.
  • lint data file, to check if its format is compatible before import it into some database.

It can be used as a portable cmd client or imported as a package by other tools.

For more details and latest updates, see doc and release notes.

Build

d18n develop with Golang 1.16+, please install first.

git clone github.com/LianjiaTech/d18n
cd d18n

# Mac or Linux
make build

# Windows
go build -o d18n cmd\d18n\d18n.go

Cross-platform compile

Golang support many

~ $ make release
...

~ $ ls release
d18n.darwin-amd64  d18n.darwin-arm64  d18n.linux-amd64  d18n.windows-amd64

Quick Start

Simple Example

~ $ d18n --defaults-extra-file test/my.cnf --query "show databases"
+--------------------+
|      DATABASE      |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sakila             |
| sys                |
| test               |
| world_x            |
+--------------------+

License

d18n is under the Apache 2.0 license. See the LICENSE file for details.

Issues
  • 项目引用了github.com/dgrijalva/jwt-go等457个开源组件,存在14个漏洞,建议升级

    项目引用了github.com/dgrijalva/jwt-go等457个开源组件,存在14个漏洞,建议升级

    大佬,你好,我是@abbykimi,我IDE运行您这个项目的时候,提示有几个漏洞,项目调用了github.com/dgrijalva/jwt-go等457个开源组件,存在14个安全漏洞,建议你升级下。

    漏洞标题:jwt-go 安全漏洞
    漏洞编号:CVE-2020-26160
    漏洞描述:
    jwt-go是个人开发者的一个Go语言的JWT实现。
    jwt-go 4.0.0-preview1之前版本存在安全漏洞。攻击者可利用该漏洞在使用[]string{} for m[\"aud\"](规范允许)的情况下绕过预期的访问限制。
    影响范围:(∞, 4.0.0-preview1)
    最小修复版本:4.0.0-preview1
    引入路径:
    github.com/LianjiaTech/[email protected]>github.com/pingcap/[email protected]>go.etcd.io/[email protected]>github.com/dgrijalva/[email protected]+incompatible
    

    另外13个漏洞,如需查看详细报告、复测或持续监测您的项目,戳这里https://www.mfsec.cn/jr?p=m5a8ab

    如果你对这个issues有任何疑问可以回复我哈( @abbykimi ),我会及时回复你的。

    opened by ghost 1
  • Can you support random dates?

    Can you support random dates?

    能支持日期随机生成吗?比如生日,给个100年以内的date,Birthday生成的‘NNNN-NN-NN’对于日期类型无法识别

    Google translate Can you support random dates? For example, birthdays, given a date within 100 years, the "NNNN-NN-NN" generated by the birth cannot be recognized for a certain date type

    feature request 
    opened by guanshanyue 0
Owner
Ke Technologies
Ke Technologies
Dumpling is a fast, easy-to-use tool written by Go for dumping data from the database(MySQL, TiDB...) to local/cloud(S3, GCP...) in multifarious formats(SQL, CSV...).

?? Dumpling Dumpling is a tool and a Go library for creating SQL dump from a MySQL-compatible database. It is intended to replace mysqldump and mydump

PingCAP 261 Jun 24, 2022
OctoSQL is a query tool that allows you to join, analyse and transform data from multiple databases and file formats using SQL.

OctoSQL OctoSQL is a query tool that allows you to join, analyse and transform data from multiple databases, streaming sources and file formats using

Jacob Martin 3.1k Jun 27, 2022
Redis-shake is a tool for synchronizing data between two redis databases. Redis-shake是一个用于在两个redis之间同步数据的工具,满足用户非常灵活的同步、迁移需求。

RedisShake is mainly used to synchronize data from one redis to another. Thanks to the Douyu's WSD team for the support. 中文文档 English tutorial 中文使用文档

Alibaba 2.4k Jun 24, 2022
Small tool that analyzes the data of my crappy file format for catalogging things

Things Catalog Analyzer I recently started catalogging all the things I own. I simply wanted to have an overview over my things, think about what I ca

Marcel Schramm 0 Nov 7, 2021
A tool for finding corrupted data rows in TiDB

tidb-bad-rows A tool for finding corrupted data rows in TiDB. It scans the target table and using a divide and conquer paradigm to locate all corrupte

Wenxuan 1 Nov 17, 2021
A database connection tool for sensitive data

go-sql 用于快速统计数据库行数、敏感字段匹配、数据库连接情况。 usage ./go-sql_darwin_amd64 -h ./go-sql_darwin_amd64 -f db.yaml -k name,user ./go-sql_darwin_amd64 -f db.yaml --min

null 5 Apr 4, 2022
Mergestat - a command-line tool for running SQL queries on git repositories and related data sources

Query git repositories with SQL. Generate reports, perform status checks, analyze codebases. ?? ??

MergeStat 3.1k Jul 3, 2022
A fast data generator that's multi-table aware and supports multi-row DML.

If you need to generate a lot of random data for your database tables but don't want to spend hours configuring a custom tool for the job, then datage

Coding Concepts 48 Apr 29, 2022
Sync MySQL data into elasticsearch

go-mysql-elasticsearch is a service syncing your MySQL data into Elasticsearch automatically. It uses mysqldump to fetch the origin data at first, the

siddontang 3.8k Jun 25, 2022
Convert data exports from various services to a single SQLite database

Bionic Bionic is a tool to convert data exports from web apps to a single SQLite database. Bionic currently supports data exports from Google, Apple H

Bionic 138 Jun 29, 2022
Library for scanning data from a database into Go structs and more

scany Overview Go favors simplicity, and it's pretty common to work with a database via driver directly without any ORM. It provides great control and

Georgy Savva 590 Jun 21, 2022
Workshop: Building data-intensive apps with SingleStore, Redpanda, and Golang

This repo provides a starting point for building applications using SingleStore, Redpanda (by Vectorized), and the Go language. SingleStore is a scale-out relational database built for data-intensive workloads. Redpanda is a Kafka API compatible streaming platform for mission-critical workloads created by the team at Vectorized.

SingleStore Labs 13 Jun 22, 2022
An observability database aims to ingest, analyze and store Metrics, Tracing and Logging data.

BanyanDB BanyanDB, as an observability database, aims to ingest, analyze and store Metrics, Tracing and Logging data. It's designed to handle observab

The Apache Software Foundation 105 Jun 28, 2022
Create key value sqlite3 database from tabular data, fast.

Turn tabular data into a lookup table using sqlite3. This is a working PROTOTYPE with limitations, e.g. no customizations, the table definition is fixed, etc.

Martin Czygan 5 Apr 2, 2022
Use SQL to data from CSV files. Open source CLI. No DB required.

CSV Plugin for Steampipe Use SQL to query data from CSV files. Get started → Documentation: Table definitions & examples Community: Slack Channel Get

Turbot 6 Mar 16, 2022
MatrixOne is a planet scale, cloud-edge native big data engine crafted for heterogeneous workloads.

What is MatrixOne? MatrixOne is a planet scale, cloud-edge native big data engine crafted for heterogeneous workloads. It provides an end-to-end data

Matrix Origin 996 Jun 29, 2022
Go library that stores data in Redis with SQL-like schema

Go library that stores data in Redis with SQL-like schema. The goal of this library is we can store data in Redis with table form.

kaharman 2 Mar 14, 2022
Make a sqlite3 database from tabular data, fast.

MAKTA make a database from tabular data Turn tabular data into a lookup table using sqlite3. This is a working PROTOTYPE with limitations, e.g. no cus

Martin Czygan 5 Apr 2, 2022
Get data from .csv files use SQL-like queries.

csvql Get data from .csv files use SQL-like queries. Задание Необходимо написать консольную программу, которая по заданному клиентом запросу осуществл

Mikhail Belov 0 Dec 7, 2021