tbls is a CI-Friendly tool for document a database, written in Go.

Overview


tbls

Build Status GitHub release codecov Go Report Card Docker Build

tbls is a CI-Friendly tool for document a database, written in Go.

Key features of tbls are:

Table of Contents


Quick Start

Document a database with one command.

$ tbls doc postgres://dbuser:[email protected]:5432/dbname

Using docker image.

$ docker run --rm -v $PWD:/work k1low/tbls doc postgres://dbuser:[email protected]:5432/dbname

Install

deb:

Use dpkg-i-from-url

$ export TBLS_VERSION=X.X.X
$ curl -L https://git.io/dpkg-i-from-url | bash -s -- https://github.com/k1LoW/tbls/releases/download/v$TBLS_VERSION/tbls_$TBLS_VERSION-1_amd64.deb

RPM:

$ export TBLS_VERSION=X.X.X
$ yum install https://github.com/k1LoW/tbls/releases/download/v$TBLS_VERSION/tbls_$TBLS_VERSION-1_amd64.rpm

homebrew tap:

$ brew install k1LoW/tap/tbls

manually:

Download binary from releases page

go get:

$ go get github.com/k1LoW/tbls

docker:

$ docker pull ghcr.io/k1low/tbls:latest

temporary:

$ source <(curl https://git.io/use-tbls)
$ curl -sL https://git.io/use-tbls > /tmp/use-tbls.tmp && . /tmp/use-tbls.tmp

Getting Started

Document a database

Add .tbls.yml ( or tbls.yml ) file to your repository.

# .tbls.yml

# DSN (Database Source Name) to connect database
dsn: postgres://dbuser:[email protected]:5432/dbname

# Path to generate document
# Default is `dbdoc`
docPath: doc/schema

Notice: If you are using a symbol such as # < in database password, URL-encode the password

Run tbls doc to analyzes the database and generate document in GitHub Friendly Markdown format.

$ tbls doc

Commit .tbls.yml and the document.

$ git add .tbls.yml doc/schema
$ git commit -m 'Add database document'
$ git push origin master

View the document on GitHub.

Sample document

sample

Diff database and ( document or database )

Update database schema.

$ psql -U dbuser -d dbname -h hostname -p 5432 -c 'ALTER TABLE users ADD COLUMN phone_number varchar(15);'
Password for user dbuser:
ALTER TABLE

tbls diff shows the difference between database schema and generated document.

$ tbls diff
diff postgres://dbuser:*****@hostname:5432/dbname doc/schema/README.md
--- postgres://dbuser:*****@hostname:5432/dbname
+++ doc/schema/README.md
@@ -4,7 +4,7 @@

 | Name | Columns | Comment | Type |
 | ---- | ------- | ------- | ---- |
-| [users](users.md) | 7 | Users table | BASE TABLE |
+| [users](users.md) | 6 | Users table | BASE TABLE |
 | [user_options](user_options.md) | 4 | User options table | BASE TABLE |
 | [posts](posts.md) | 8 | Posts table | BASE TABLE |
 | [comments](comments.md) | 6 | Comments<br>Multi-line<br>table<br>comment | BASE TABLE |
diff postgres://dbuser:*****@hostname:5432/dbname doc/schema/users.md
--- postgres://dbuser:*****@hostname:5432/dbname
+++ doc/schema/users.md
@@ -14,7 +14,6 @@
 | email | varchar(355) |  | false |  |  | ex. [email protected] |
 | created | timestamp without time zone |  | false |  |  |  |
 | updated | timestamp without time zone |  | true |  |  |  |
-| phone_number | varchar(15) |  | true |  |  |  |

 ## Constraints

And, tbls diff support for diff checking between database and other database

$ tbls diff postgres://dbuser:*****@local:5432/dbname postgres://dbuser:*****@production:5432/dbname

Notice: tbls diff shows the difference Markdown documents only.

Re-generating database documentation

Existing documentation can re-generated using either --force or --rm-dist flag.

--force forces overwrite of the existing documents. It does not, however, remove files of removed tables.

$ tbls doc --force

--rm-dist removes files in docPath before generating the documents.

$ tbls doc --rm-dist

Lint a database

Add linting rule to .tbls.yml following

# .tbls.yml
lint:
  requireColumnComment:
    enabled: true
    exclude:
      - id
      - created
      - updated
  columnCount:
    enabled: true
    max: 10

Run tbls lint to check the database according to lint: rules

$ tbls lint
users.username: column comment required.
users.password: column comment required.
users.phone_number: column comment required.
posts.user_id: column comment required.
posts.title: column comment required.
posts.labels: column comment required.
comments.post_id: column comment required.
comment_stars.user_id: column comment required.
post_comments.comment: column comment required.
posts: too many columns. [12/10]
comments: too many columns. [11/10]

11 detected

Measure document coverage

tbls coverage measure and show document coverage ( description, comments ).

$ tbls coverage
Table                       Coverage
All tables                  16.1%
 public.users               20%
 public.user_options        37.5%
 public.posts               35.3%
 public.comments            14.3%
 public.comment_stars       0%
 public.logs                12.5%
 public.post_comments       87.5%
 public.post_comment_stars  0%
 public.CamelizeTable       0%
 public.hyphen-table        0%
 administrator.blogs        0%
 backup.blogs               0%
 backup.blog_options        0%
 time.bar                   0%
 time.hyphenated-table      0%
 time.referencing           0%

Continuous Integration

Continuous integration using tbls.

  1. Commit the document using tbls doc.
  2. Update the database schema in the development cycle.
  3. Check for document updates by running tbls diff or tbls lint in CI.
  4. Return to 1.

Example: Travis CI

# .travis.yml
language: go

install:
  - source <(curl -sL https://git.io/use-tbls)
script:
  - tbls diff
  - tbls lint

Tips: If your CI based on Debian/Ubuntu (/bin/sh -> dash), you can use following install command curl -sL https://git.io/use-tbls > use-tbls.tmp && . ./use-tbls.tmp && rm ./use-tbls.tmp

Tips: If the order of the columns does not match, you can use the --sort option.

Configuration

Name

name: is used to specify the database name of the document.

# .tbls.yml
name: mydatabase

Description

desc: is used to specify the database description.

# .tbls.yml
desc: This is My Database

Labels

labels: is used to label the database or tables.

label database:

# .tbls.yml
labels:
  - cmdb
  - analytics

label tables:

# .tbls.yml
comments:
  -
    table: users
    labels:
      - user
      - privacy data

DSN

dsn: (Data Source Name) is used to connect to database.

# .tbls.yml
dsn: my://dbuser:[email protected]:3306/dbname

dsn: can expand environment variables.

# .tbls.yml
dsn: my://${MYSQL_USER}:${MYSQL_PASSWORD}@hostname:3306/${MYSQL_DATABASE}

Support Datasource

tbls support following databases/datasources.

PostgreSQL:

# .tbls.yml
dsn: postgres://dbuser:[email protected]:5432/dbname
# .tbls.yml
dsn: pg://dbuser:[email protected]:5432/dbname

MySQL:

# .tbls.yml
dsn: mysql://dbuser:[email protected]:3306/dbname
# .tbls.yml
dsn: my://dbuser:[email protected]:3306/dbname

MariaDB:

# .tbls.yml
dsn: mariadb://dbuser:[email protected]:3306/dbname
# .tbls.yml
dsn: maria://dbuser:[email protected]:3306/dbname

SQLite:

# .tbls.yml
dsn: sqlite:///path/to/dbname.db
# .tbls.yml
dsn: sq:///path/to/dbname.db

BigQuery:

# .tbls.yml
dsn: bigquery://project-id/dataset-id?creds=/path/to/google_application_credentials.json
# .tbls.yml
dsn: bq://project-id/dataset-id?creds=/path/to/google_application_credentials.json

To set GOOGLE_APPLICATION_CREDENTIALS environment variable, you can use

  1. export GOOGLE_APPLICATION_CREDENTIALS or env GOOGLE_APPLICATION_CREDENTIALS
  2. Add query to DSN
    • ?google_application_credentials=/path/to/client_secrets.json
    • ?credentials=/path/to/client_secrets.json
    • ?creds=/path/to/client_secrets.json

Required permissions: bigquery.datasets.get bigquery.tables.get bigquery.tables.list

Cloud Spanner:

# .tbls.yml
dsn: spanner://project-id/instance-id/dbname?creds=/path/to/google_application_credentials.json

To set GOOGLE_APPLICATION_CREDENTIALS environment variable, you can use

  1. export GOOGLE_APPLICATION_CREDENTIALS or env GOOGLE_APPLICATION_CREDENTIALS
  2. Add query to DSN
    • ?google_application_credentials=/path/to/client_secrets.json
    • ?credentials=/path/to/client_secrets.json
    • ?creds=/path/to/client_secrets.json

Amazon Redshift:

# .tbls.yml
dsn: redshift://dbuser:[email protected]:5432/dbname
# .tbls.yml
dsn: rs://dbuser:[email protected]:5432/dbname

Microsoft SQL Server:

# .tbls.yml
dsn: mssql://DbUser:[email protected]:1433/testdb
# .tbls.yml
dsn: sqlserver://DbUser:[email protected]:1433/testdb
# .tbls.yml
dsn: ms://DbUser:[email protected]:1433/testdb

Amazon DynamoDB:

# .tbls.yml
dsn: dynamodb://us-west-2
# .tbls.yml
dsn: dynamo://ap-northeast-1?aws_access_key_id=XXXXXxxxxxxxXXXXXXX&aws_secret_access_key=XXXXXxxxxxxxXXXXXXX

To set AWS credentials, you can use

  1. Use default credential provider chain of AWS SDK for Go
  2. Add query to DSN
    • ?aws_access_key_id=XXXXXxxxxxxxXXXXXXX&aws_secret_access_key=XXXXXxxxxxxxXXXXXXX

Snowflake (Experimental):

---
# .tbls.yml
dsn: snowflake://user:[email protected]/mydb/myschema

See also: https://pkg.go.dev/github.com/snowflakedb/gosnowflake

JSON:

The JSON file output by the tbls out -t json command can be read as a datasource.

---
# .tbls.yml
dsn: json://path/to/testdb.json

HTTP:

---
# .tbls.yml
dsn: https://hostname/path/to/testdb.json
---
# .tbls.yml
dsn:
  url: https://hostname/path/to/testdb.json
  headers:
    Authorization: token GITHUB_OAUTH_TOKEN

Document path

tbls doc generates document in the directory specified by docPath:.

# .tbls.yml
# Default is `dbdoc`
docPath: doc/schema

docPath: can expand environment variables.

# .tbls.yml
docPath: ${DOC_PATH}

Table format

format: is used to change the document format.

# .tbls.yml
format:
  # Adjust the column width of Markdown format table
  # Default is false
  adjust: true
  # Sort the order of table list and  columns
  # Default is false
  sort: false

ER diagram

tbls doc generate ER diagram images at the same time.

# .tbls.yml
er:
  # Skip generation of ER diagram
  # Default is false
  skip: false
  # ER diagram image format
  # Default is `svg`
  format: svg
  # Add table/column comment to ER diagram
  # Default is false
  comment: true
  # Distance between tables that display relations in the ER
  # Default is 1
  distance: 2
  # ER diagram (png/jpg) font (font name, font file, font path or keyword)
  # Default is "" ( system default )
  font: M+

It is also possible to personalize the output by providing your own templates. See the Personalized Templates section below.

Lint

tbls lint work as linter for database.

# .tbls.yml
lint:
  # require table comment
  requireTableComment:
    enabled: true
    # all commented, or all uncommented.
    allOrNothing: false
  # require column comment
  requireColumnComment:
    enabled: true
    # all commented, or all uncommented.
    allOrNothing: true
    # exclude columns from warnings
    exclude:
      - id
      - created_at
      - updated_at
    # exclude tables from warnings
    excludeTables:
      - logs
      - comment_stars
  # require index comment
  requireIndexComment:
    enabled: true
    # all commented, or all uncommented.
    allOrNothing: false
    # exclude indexes from warnings
    exclude:
      - user_id_idx
    # exclude tables from warnings
    excludeTables:
      - logs
      - comment_stars
  # require constraint comment
  requireConstraintComment:
    enabled: true
    # all commented, or all uncommented.
    allOrNothing: false
    # exclude constrains from warnings
    exclude:
      - unique_user_name
    # exclude tables from warnings
    excludeTables:
      - logs
      - comment_stars
  # require trigger comment
  requireTriggerComment:
    enabled: true
    # all commented, or all uncommented.
    allOrNothing: false
    # exclude triggers from warnings
    exclude:
      - update_count
    # exclude tables from warnings
    excludeTables:
      - logs
      - comment_stars
  # find a table that has no relation
  unrelatedTable:
    enabled: true
    # all related, or all unrelated.
    allOrNothing: true
    # exclude tables from warnings
    exclude:
      - logs
  # check max column count
  columnCount:
    enabled: true
    max: 10
    # exclude tables from warnings
    exclude:
      - user_options
  # require columns
  requireColumns:
    enabled: true
    columns:
      -
        name: created
      -
        name: updated
        exclude:
          - logs
          - CamelizeTable
  # check duplicate relations
  duplicateRelations:
    enabled: true
  # check if the foreign key columns have an index
  requireForeignKeyIndex:
    enabled: true
    exclude:
      - comments.user_id
  # checks if labels are in BigQuery style ( https://cloud.google.com/resource-manager/docs/creating-managing-labels#requirements )
  labelStyleBigQuery:
    enabled: true
    exclude:
      - schema_migrations

Filter tables

filter tables

include: and exclude: are used to filter target tables from tbls *.

# .tbls.yml
include:
  - some_prefix_*
exclude:
  - some_prefix_logs
  - CamelizeTable

lintExclude: is used to exclude tables from tbls lint.

# .tbls.yml
lintExclude:
  - CamelizeTable

Comments

comments: is used to add table/column comment to database document without ALTER TABLE.

For example, you can add comment about VIEW TABLE or SQLite tables/columns.

# .tbls.yml
comments:
  -
    table: users
    # table comment
    tableComment: Users table
    # column comments
    columnComments:
      email: Email address as login id. ex. [email protected]
    # labels for tables
    labels:
      - privary data
      - backup:true
  -
    table: post_comments
    tableComment: post and comments View table
    columnComments:
      id: comments.id
      title: posts.title
      post_user: posts.users.username
      comment_user: comments.users.username
      created: comments.created
      updated: comments.updated
  -
    table: posts
    # index comments
    indexComments:
      posts_user_id_idx: user.id index
    # constraints comments
    constraintComments:
      posts_id_pk: PRIMARY KEY
    # triggers comments
    triggerComments:
      update_posts_updated: Update updated when posts update

Relations

relations: is used to add table relation to database document without FOREIGN KEY.

You can create ER diagrams with relations without having foreign key constraints.

relations:
  -
    table: logs
    columns:
      - user_id
    parentTable: users
    parentColumns:
      - id
    # Relation definition
    # Default is `Additional Relation`
    def: logs->users
  -
    table: logs
    columns:
      - post_id
    parentTable: posts
    parentColumns:
      - id
  -
    table: logs
    columns:
      - comment_id
    parentTable: comments
    parentColumns:
      - id
  -
    table: logs
    columns:
      - comment_star_id
    parentTable: comment_stars
    parentColumns:
      - id

img

Automatically detect relations

detectVirtualRelations: if enabled, automatically detect relations from table and column names.

detectVirtualRelations:
  enabled: true

Dictionary

dict: is used to replace title/table header of database document

# .tbls.yml
---
dict:
  Tables: テーブル一覧
  Description: 概要
  Columns: カラム一覧
  Indexes: INDEX一覧
  Constraints: 制約一覧
  Triggers: トリガー
  Relations: ER図
  Name: 名前
  Comment: コメント
  Type: タイプ
  Default: デフォルト値
  Children: 子テーブル
  Parents: 親テーブル
  Definition: 定義
  Table Definition: テーブル定義

Personalized Templates

It is possible to provide your own templates to personalize the documentation generated by tbls by adding a templates section to your configuration. For example:

templates:
  dot:
    schema: 'templates/schema.dot.tmpl'
    table: 'templates/table.dot.tmpl'
  puml:
    schema: 'templates/schema.puml.tmpl'
    table: 'templates/table.puml.tmpl'
  md:
    index: 'templates/index.md.tmpl'
    table: 'templates/table.md.tmpl'

A good starting point to design your own template is to modify a copy the default ones for Dot, PlantUML and markdown.

Output formats

tbls out output in various formats.

Markdown:

$ tbls out -t md -o schema.md

DOT:

$ tbls out -t dot -o schema.dot

PlantUML:

$ tbls out -t plantuml -o schema.puml

Image (svg, png, jpg):

$ tbls out -t svg --table users --distance 2 -o users.svg

JSON:

$ tbls out -t json -o schema.json

Tips: tbls doc can load schema.json as DSN.

$ tbls doc json:///path/to/schema.json

YAML:

$ tbls out -t yaml -o schema.yaml

Excel:

$ tbls out -t xlsx -o schema.xlsx

.tbls.yml:

$ tbls out -t config -o .tbls.new.yml

Command arguments

tbls subcommands ( doc,diff, etc) accepts arguments and options

$ tbls doc my://root:[email protected]:3306/testdb doc/schema

You can check available arguments and options using tbls help [COMMAND].

$ tbls help doc
'tbls doc' analyzes a database and generate document in GitHub Friendly Markdown format.

Usage:
  tbls doc [DSN] [DOC_PATH] [flags]

Flags:
  -j, --adjust-table       adjust column width of table
  -c, --config string      config file path
  -t, --er-format string   ER diagrams output format (png, svg, jpg, ...). default: svg
  -f, --force              force
  -h, --help               help for doc
      --sort               sort
      --when string        command execute condition
      --without-er         no generate ER diagrams

Environment variables

tbls accepts environment variables TBLS_DSN and TBLS_DOC_PATH

$ env TBLS_DSN=my://root:[email protected]:3306/testdb TBLS_DOC_PATH=doc/schema tbls doc
Comments
  • sql: Scan error on column index 1: unsupported Scan, storing driver.Value type <nil> into type *string

    sql: Scan error on column index 1: unsupported Scan, storing driver.Value type into type *string

    mysqlのドキュメントを生成しようとするとエラーが出てしまいます。 対処方法などありましたら、教えていただけないでしょうか。

    # cat /etc/redhat-release 
    CentOS release 6.6 (Final)
    # mysql --version
    mysql  Ver 14.14 Distrib 5.6.23, for Linux (x86_64) using  EditLine wrapper
    # go version
    go version go1.9.6 linux/amd64
    # tbls version
    0.9.3
    # tbls doc mysql://db_user:[email protected]:3306/cs_production ./dbdoc
    sql: Scan error on column index 1: unsupported Scan, storing driver.Value type <nil> into type *string
    
    bug good first issue 
    opened by ymaeyama 16
  • Tbls crashes

    Tbls crashes

    What happened

    tbls crashed

    What you expected to happed

    Generate doc

    What stacktrace or error message from tbls did you see?

    panic: runtime error: index out of range [0] with length 0

    goroutine 1 [running]: github.com/k1LoW/tbls/drivers/mysql.(*Mysql).Analyze(0xc00089ce50, 0xc0000297a0, 0x0, 0x0) /Users/k1low/src/github.com/k1LoW/tbls/drivers/mysql/mysql.go:421 +0x4876 github.com/k1LoW/tbls/datasource.Analyze(0xc0008b5c00, 0x33, 0x0, 0x0, 0x0, 0x0) /Users/k1low/src/github.com/k1LoW/tbls/datasource/datasource.go:114 +0xba8 github.com/k1LoW/tbls/cmd.glob..func5(0x2132420, 0x225b488, 0x0, 0x0) /Users/k1low/src/github.com/k1LoW/tbls/cmd/doc.go:73 +0x1a5 github.com/spf13/cobra.(*Command).execute(0x2132420, 0x225b488, 0x0, 0x0, 0x2132420, 0x225b488) /Users/k1low/go/pkg/mod/github.com/spf13/[email protected]/command.go:854 +0x2c2 github.com/spf13/cobra.(*Command).ExecuteC(0x2132c00, 0x4, 0x225b488, 0x0) /Users/k1low/go/pkg/mod/github.com/spf13/[email protected]/command.go:958 +0x375 github.com/spf13/cobra.(*Command).Execute(...) /Users/k1low/go/pkg/mod/github.com/spf13/[email protected]/command.go:895 github.com/k1LoW/tbls/cmd.Execute() /Users/k1low/src/github.com/k1LoW/tbls/cmd/root.go:187 +0x86 main.main() /Users/k1low/src/github.com/k1LoW/tbls/main.go:34 +0x25

    Anything else we need to know?

    No

    Environment

    tbls version: 1.49.2 Database version: Mariadb 10.4.17 OS ( Ubuntu bionic, macOS Calalina, WSL2, etc ): WSL2 ubuntu, i tried also docker, same issue Shell ( bash, zsh, poworshell, etc ): bash panic: runtime error: index out of range [0] with length 0

    goroutine 1 [running]: github.com/k1LoW/tbls/drivers/mysql.(*Mysql).Analyze(0xc00089ce50, 0xc0000297a0, 0x0, 0x0) /Users/k1low/src/github.com/k1LoW/tbls/drivers/mysql/mysql.go:421 +0x4876 github.com/k1LoW/tbls/datasource.Analyze(0xc0008b5c00, 0x33, 0x0, 0x0, 0x0, 0x0) /Users/k1low/src/github.com/k1LoW/tbls/datasource/datasource.go:114 +0xba8 github.com/k1LoW/tbls/cmd.glob..func5(0x2132420, 0x225b488, 0x0, 0x0) /Users/k1low/src/github.com/k1LoW/tbls/cmd/doc.go:73 +0x1a5 github.com/spf13/cobra.(*Command).execute(0x2132420, 0x225b488, 0x0, 0x0, 0x2132420, 0x225b488) /Users/k1low/go/pkg/mod/github.com/spf13/[email protected]/command.go:854 +0x2c2 github.com/spf13/cobra.(*Command).ExecuteC(0x2132c00, 0x4, 0x225b488, 0x0) /Users/k1low/go/pkg/mod/github.com/spf13/[email protected]/command.go:958 +0x375 github.com/spf13/cobra.(*Command).Execute(...) /Users/k1low/go/pkg/mod/github.com/spf13/[email protected]/command.go:895 github.com/k1LoW/tbls/cmd.Execute() /Users/k1low/src/github.com/k1LoW/tbls/cmd/root.go:187 +0x86 main.main() /Users/k1low/src/github.com/k1LoW/tbls/main.go:34 +0x25 Stacktrace or error message from tbls ( If you can, set DEBUG=1) Config ( .tbls.yml ): dsn: mariadb://root:[email protected]:3306/warehouse docPath: doc/schema

    bug 
    opened by svservices 15
  • Hide `parent`, `children`, `comment` columns without data

    Hide `parent`, `children`, `comment` columns without data

    Fix for https://github.com/k1LoW/tbls/issues/321

    thank you for the great tool!

    it feels like it is possible to make reports more readable by removing columns with no data.

    opened by YauhenPylAurea 12
  • Segfault when trying to document Redshift database

    Segfault when trying to document Redshift database

    panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x41f7cf6]
    
    goroutine 1 [running]:
    database/sql.(*Rows).close(0x0, 0x0, 0x0, 0x0, 0x0)
            /Users/k1low/.anyenv/envs/goenv/versions/1.12.5/src/database/sql/sql.go:2976 +0x66
    database/sql.(*Rows).Close(0x0, 0x48c2a0c, 0x172)
            /Users/k1low/.anyenv/envs/goenv/versions/1.12.5/src/database/sql/sql.go:2972 +0x33
    github.com/k1LoW/tbls/drivers/postgres.(*Postgres).Analyze(0xc0003397c0, 0xc0000dd4f0, 0x498d880, 0xc0003b6f40)
            /Users/k1low/src/github.com/k1LoW/tbls/drivers/postgres/postgres.go:131 +0x2f8a
    github.com/k1LoW/tbls/datasource.Analyze(0x7ffeefbfe421, 0xa0, 0x0, 0x0, 0x0)
            /Users/k1low/src/github.com/k1LoW/tbls/datasource/datasource.go:74 +0x683
    github.com/k1LoW/tbls/cmd.glob..func2(0x4e5dbc0, 0xc0001f7460, 0x1, 0x1)
            /Users/k1low/src/github.com/k1LoW/tbls/cmd/doc.go:72 +0x111
    github.com/spf13/cobra.(*Command).execute(0x4e5dbc0, 0xc0001f7430, 0x1, 0x1, 0x4e5dbc0, 0xc0001f7430)
            /Users/k1low/go/pkg/mod/github.com/spf13/[email protected]/command.go:766 +0x2ae
    github.com/spf13/cobra.(*Command).ExecuteC(0x4e5d4a0, 0x0, 0x0, 0x45eabea)
            /Users/k1low/go/pkg/mod/github.com/spf13/[email protected]/command.go:850 +0x2fc
    github.com/spf13/cobra.(*Command).Execute(...)
            /Users/k1low/go/pkg/mod/github.com/spf13/[email protected]/command.go:800
    github.com/k1LoW/tbls/cmd.Execute()
            /Users/k1low/src/github.com/k1LoW/tbls/cmd/root.go:59 +0x2e
    main.main()
            /Users/k1low/src/github.com/k1LoW/tbls/main.go:32 +0x20
    
    $ tbls version
    1.15.1
    
    $ go version
    go version go1.12.5 darwin/amd64
    

    Please let me know if there's any other information I can provide to help debugging!

    bug 
    opened by craigrosie 12
  • Invalid links can be generated when not selecting all the tables

    Invalid links can be generated when not selecting all the tables

    What happened

    Suppose you have tables A, B, and C. both A and B point to table C. If we specify to generate docs only for tables A and B, the link in theese pointing to non existing table C documentation will be generated.

    What you expected to happed

    Should not create a link to avoid user errors when browsing generated output. A switch not to generate links at all can do the job as well.

    Environment

    • tbls version : 1.37.0
    • postgreSQL database
    • Windows 10
    • config:
    comments:
    - table: A
      tableComment: table A
    - table: B
      tableComment: table B
    desc: 'My description'
    docPath: doc/schema
    dsn: postgres://user:[email protected]:5432/dbname?sslmode=disable
    er:
      skip: true
    include:
    - A
    - B
    name: Db overview
    
    bug 
    opened by palucki 11
  • [sqlite] tbls doc error - near

    [sqlite] tbls doc error - near ".": syntax error

    What happened

    tbls doc sq://./data.db # => near ".": syntax error
    

    Tried different paths - absolute, relative with dot and without.

    What you expected to happed

    No errors

    What stacktrace or error message from tbls did you see?

    near ".": syntax error
    github.com/k1LoW/tbls/drivers/sqlite.(*Sqlite).Analyze
            /Users/k1low/src/github.com/k1LoW/tbls/drivers/sqlite/sqlite.go:251
    github.com/k1LoW/tbls/datasource.Analyze
            /Users/k1low/src/github.com/k1LoW/tbls/datasource/datasource.go:85
    github.com/k1LoW/tbls/cmd.glob..func4
            /Users/k1low/src/github.com/k1LoW/tbls/cmd/doc.go:69
    github.com/spf13/cobra.(*Command).execute
            /Users/k1low/go/pkg/mod/github.com/spf13/[email protected]/command.go:844
    github.com/spf13/cobra.(*Command).ExecuteC
            /Users/k1low/go/pkg/mod/github.com/spf13/[email protected]/command.go:945
    github.com/spf13/cobra.(*Command).Execute
            /Users/k1low/go/pkg/mod/github.com/spf13/[email protected]/command.go:885
    github.com/k1LoW/tbls/cmd.Execute
            /Users/k1low/src/github.com/k1LoW/tbls/cmd/root.go:59
    main.main
            /Users/k1low/src/github.com/k1LoW/tbls/main.go:33
    runtime.main
            /Users/k1low/.anyenv/envs/goenv/versions/1.13.6/src/runtime/proc.go:203
    runtime.goexit
            /Users/k1low/.anyenv/envs/goenv/versions/1.13.6/src/runtime/asm_amd64.s:1357
    

    Environment

    • tbls version v1.29.2
    • Database version - sqlite 3
    • Windows 10
    • Config ( .tbls.yml ) - empty
    bug help wanted 
    opened by unlight 11
  • Support Snowflake

    Support Snowflake

    This is an initial implementation for Snowflake. (If snowflake support is not needed for tbls, please close this PR 🙏 )

    Note

    I'm not sure what to do with the testing environment. So I don't add test cases to snowflake driver yet.

    Demo

    The following is DDLs and configurations.

    create schema tbls_sample;
    use schema tbls_sample;
    create or replace table tbls_sample (
      id int primary key
      , ext_id int
      , raw variant
      , created_at timestamp_ltz
      , updated_at timestamp_ltz
      , unique (ext_id)
    );
    
    dsn: snowflake:/<user>:<password>@<account_id>.ap-northeast-1.aws/SANDBOX/TBLS_SAMPLE
    docPath: doc/schema
    
    TBLS_SAMPLE.md # TBLS_SAMPLE

    Description

    Table Definition
    create or replace TABLE TBLS_SAMPLE (
    	ID NUMBER(38,0) NOT NULL,
    	EXT_ID NUMBER(38,0),
    	RAW VARIANT,
    	CREATED_AT TIMESTAMP_LTZ(9),
    	UPDATED_AT TIMESTAMP_LTZ(9),
    	unique (EXT_ID),
    	primary key (ID)
    );
    

    Columns

    | Name | Type | Default | Nullable | Children | Parents | Comment | | ---- | ---- | ------- | -------- | -------- | ------- | ------- | | ID | NUMBER | | false | | | | | EXT_ID | NUMBER | | true | | | | | RAW | VARIANT | | true | | | | | CREATED_AT | TIMESTAMP_LTZ | | true | | | | | UPDATED_AT | TIMESTAMP_LTZ | | true | | | |

    Relations

    er


    Generated by tbls

    </details>
    
    enhancement 
    opened by kanata2 9
  • panic: runtime error: index out of range

    panic: runtime error: index out of range

    link https://github.com/k1LoW/tbls/issues/36#issuecomment-407610853

    # cat /etc/redhat-release 
    CentOS release 6.9 (Final)
    # mysql --version
    mysql  Ver 14.14 Distrib 5.7.22, for Linux (x86_64) using  EditLine wrapper
    # go version
    go version go1.9.6 linux/amd64
    # go get -u github.com/k1LoW/tbls
    # DEBUG=1 tbls doc mysql://user:[email protected]:3306/tbls_mysql57 ./dbdoc
    panic: runtime error: index out of range
    
    goroutine 1 [running]:
    github.com/k1LoW/tbls/db.Analyze(0x7ffdb65f48b3, 0x46, 0x0, 0x0, 0x0)
            /root/go/src/github.com/k1LoW/tbls/db/db.go:28 +0x453
    github.com/k1LoW/tbls/cmd.glob..func4(0xaa7620, 0xc420118e40, 0x2, 0x2)
            /root/go/src/github.com/k1LoW/tbls/cmd/doc.go:56 +0x65
    github.com/k1LoW/tbls/vendor/github.com/spf13/cobra.(*Command).execute(0xaa7620, 0xc420118e00, 0x2, 0x2, 0xaa7620, 0xc420118e00)
            /root/go/src/github.com/k1LoW/tbls/vendor/github.com/spf13/cobra/command.go:766 +0x2c1
    github.com/k1LoW/tbls/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xaa7ae0, 0xc42001c0b8, 0x0, 0xc42004ff48)
            /root/go/src/github.com/k1LoW/tbls/vendor/github.com/spf13/cobra/command.go:852 +0x334
    github.com/k1LoW/tbls/vendor/github.com/spf13/cobra.(*Command).Execute(0xaa7ae0, 0x0, 0x0)
            /root/go/src/github.com/k1LoW/tbls/vendor/github.com/spf13/cobra/command.go:800 +0x2b
    github.com/k1LoW/tbls/cmd.Execute()
            /root/go/src/github.com/k1LoW/tbls/cmd/root.go:53 +0x2d
    main.main()
            /root/go/src/github.com/k1LoW/tbls/main.go:34 +0x20
    
    bug 
    opened by k1LoW 9
  • PostgreSQL+PostGIS: not found table 'public.topology' (non-default schema issue)

    PostgreSQL+PostGIS: not found table 'public.topology' (non-default schema issue)

    What happened

    $ tbls doc
    not found table 'public.topology'
    

    Seems related to #28 and #153

    You can reproduce with docker image mdillon/postgis:9.5

    What you expected to happed

    I expected a generated documentation.

    What stacktrace or error message from tbls did you see?

    not found table 'public.topology'

    Anything else we need to know?

    PostGIS is enabled and manages the topology schema. This is the output of \dt:

    foobar> \dt
    +----------+-------------------+--------+--------+
    | Schema   | Name              | Type   | Owner  |
    |----------+-------------------+--------+--------|
    | public   | migration         | table  | dbuser |
    | public   | spatial_ref_sys   | table  | dbuser |
    | public   | subscriber        | table  | dbuser |
    | public   | user              | table  | dbuser |
    | public   | user_role         | table  | dbuser |
    | public   | users_homes       | table  | dbuser |
    | public   | users_roles       | table  | dbuser |
    | public   | users_tokens      | table  | dbuser |
    | topology | layer             | table  | dbuser |
    | topology | topology          | table  | dbuser |
    +----------+-------------------+--------+--------+
    

    Environment

    • tbls version: 1.28.2
    • PostgreSQL v9.5.16
    • PostGIS v2.5 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
    • .tbls.yml:
    ---
    
    name: foobar
    dsn: postgres://dbuser:[email protected]:5432/foobar?sslmode=disable
    docPath: docs/schema
    

    Stacktrace

    $ DEBUG=1 tbls doc
    not found table 'public.topology'
    github.com/k1LoW/tbls/schema.(*Schema).FindTableByName
            /Users/k1low/src/github.com/k1LoW/tbls/schema/schema.go:110
    github.com/k1LoW/tbls/drivers/postgres.(*Postgres).Analyze
            /Users/k1low/src/github.com/k1LoW/tbls/drivers/postgres/postgres.go:337
    github.com/k1LoW/tbls/datasource.Analyze
            /Users/k1low/src/github.com/k1LoW/tbls/datasource/datasource.go:85
    github.com/k1LoW/tbls/cmd.glob..func4
            /Users/k1low/src/github.com/k1LoW/tbls/cmd/doc.go:69
    github.com/spf13/cobra.(*Command).execute
            /Users/k1low/go/pkg/mod/github.com/spf13/[email protected]/command.go:844
    github.com/spf13/cobra.(*Command).ExecuteC
            /Users/k1low/go/pkg/mod/github.com/spf13/[email protected]/command.go:945
    github.com/spf13/cobra.(*Command).Execute
            /Users/k1low/go/pkg/mod/github.com/spf13/[email protected]/command.go:885
    github.com/k1LoW/tbls/cmd.Execute
            /Users/k1low/src/github.com/k1LoW/tbls/cmd/root.go:59
    main.main
            /Users/k1low/src/github.com/k1LoW/tbls/main.go:33
    runtime.main
            /Users/k1low/.anyenv/envs/goenv/versions/1.13.6/src/runtime/proc.go:203
    runtime.goexit
            /Users/k1low/.anyenv/envs/goenv/versions/1.13.6/src/runtime/asm_amd64.s:1357
    
    bug 
    opened by rafi 7
  • [FEATURE REQUEST] Support service account impersonation with Cloud Spanner

    [FEATURE REQUEST] Support service account impersonation with Cloud Spanner

    Overview

    Hello, I am a tbls user, mainly with postgres and Cloud spanner.

    We want to use service account impersonation for Cloud Spanner to keep granting minimum permissions to each service accounts.

    It is not difficult to implement this feature, so I made draft implementation (this PR). Please review design of this feature.

    Design

    Using URL values to pass params about impersonate service account.

    New values are below.

    • impersonate_service_account: An email of service account that is impersonated by caller.
    • impersonate_service_account_duration: The duration parameter for impersonation request.

    Example

    • ServiceAccount A: the account who issues a request to impersonate ServiceAccountB. ServiceAccountA does not have permission to connect spanner directly.
    • ServiceAccount B: the account who can connect to spanner databases, and used from ServiceAccountA.

    In this case, the command will be below.

    tbls doc \
      “spanner://{project}/{instance}/{database}?creds={credential path of ServiceAccount A}&impersonate_service_account={Email of ServiceAccount B}&impersonate_service_account_duration={duration of impersonation}”
    

    Next

    • If ok, I will implement below
      • unit tests
      • documents
      • support bigquery
    enhancement 
    opened by Attsun1031 6
  • Mongo basic support

    Mongo basic support

    Hello, thanks for the great tool!

    There is a basic mongodb support. It was tested on docker-mongo-sample-datasets.

    README example:


    Tables

    | Name | Attributes | Comment | Type | | ---- | ------- | ------- | ---- | | admin.system.version | 2 | Estimated count of documents is 1 | collection | | config.system.sessions | 0 | Estimated count of documents is 0 | collection | | local.startup_log | 7 | Estimated count of documents is 4 | collection | | test.restaurants | 7 | Estimated count of documents is 25359 | collection |

    Relations

    image


    Generated by tbls


    Table example:


    test.restaurants

    Description

    Estimated count of documents is 25359

    Attributes

    | Name | Type | Default | Nullable | Children | Parents | Comment | | ---- | ---- | ------- | -------- | -------- | ------- | ------- | | name | string | | false | | | | | restaurant_id | string | | false | | | | | _id | objectId | | false | | | | | address | document | | false | | | | | borough | string | | false | | | | | cuisine | string | | false | | | | | grades | array | | false | | | |

    Indexes

    | Name | Definition | Comment | | ---- | ---------- | ------- | | id | {"_id": {"$numberInt":"1"}} | Non-unique, Version 2 |

    Relations

    image


    Generated by tbls


    opened by YauhenPylAurea 6
  • Unable to filter schemas using include/exclude for PostgreSQL DB

    Unable to filter schemas using include/exclude for PostgreSQL DB

    What happened

    tbls is always generating all of the tables in all schemas instead of the ones for the specified in the include/exclude filters provided at the end of the ticket

    What you expected to happened

    I expect tbls to generate only the lint and er for the public schema, excluding all of the tables from the products schema

    Environment

    • tbls version 1.56.8
    • Database version 13
    • OS WSL2
    • Shell zsh
    • Config:

    Tried with:

    include:
      - public
    
    include:
      - public.
    
    include:
      - public.*
    
    include:
      - public.*
    exclude:
      - products.*
    
    include:
      - public.*
    exclude:
      - products.
    
    include:
      - public.
    exclude:
      - products.*
    
    include:
      - public
    exclude:
      - products
    
    include:
      - public.*
    exclude:
      - products
    
    include:
      - public
    exclude:
      - products.*
    
    bug 
    opened by Bare7aOVO 0
  • Percentage columns are always output in Excel format

    Percentage columns are always output in Excel format

    What happened

    Percentage columns are always output in Excel format issue

    What you expected to happened

    Percentage columns are not output by default in excel format fixed

    What stack trace or error message from tbls did you see?

    nothing

    Anything else we need to know?

    Perhaps, the Percent below is incorrect. (I think, this is simple typo mistake)

    Error: Percent Correct: Percents https://github.com/k1LoW/tbls/blob/d90f7ab181f97abf46371f0048aa1e2e07cf9434/output/xlsx/xlsx.go#L191

    Environment

    • tbls version
      • 1.56.8
    • Database version
      • postgresql:15
    • OS ( Ubuntu bionic, macOS Calalina, WSL2, etc )
      • macOS Ventura
    • Shell ( bash, zsh, poworshell, etc )
      • zsh
    • Stacktrace or error message from tbls ( If you can, set DEBUG=1)
      • nothing
    • Config ( .tbls.yml )
      • nothing
    bug pull request wanted 
    opened by zonbitamago 1
  • Output schema data file by default

    Output schema data file by default

    1. I'm thinking of making a change so that schema data (JSON) is also output by default when the tbls doc command is executed.
      • It is also possible to output it with tbls out -t json even now.
      • Also add a setting to disable output of schema data
    2. I plan to use the schema data in 1 to create commands to handle the schema without accessing the database (e.g., tbls ls-tables).

    I also want the schema data files to be committed to the repository. I believe this will make tools such as https://github.com/harakeishi/trv more powerful.

    I am currently struggling with two points.

    1. File name of schema data
      • schema.json
      • .schema.json
      • tbls.schema.json
      • .tbls.schema.json
      • tbls.schema
      • .tbls.schema
      • etc...
    2. Output path of schema data file
      • Same directory as .tbls.yml
      • Same directory as the generated schema document
      • Always working directory
      • etc...

    If you have any advice or input, I'd love to hear it.

    enhancement help wanted 
    opened by k1LoW 5
  • Exclude all tables except specific table

    Exclude all tables except specific table

    I have a database where I want to exclude all tables from the auth schema, except auth.users. This database also has tables in other schemas that I want to be included.

    I tried

    # .tbls.yml
    include:
      - * # also `"*"`, `'*'`, `"**/*"` as well as removing this line
      - auth.users
    exclude:
      - auth.*
    

    However, this causes all output to be excluded, producing an empty output.

    1. How can I exclude all tables in the auth schema, except auth.users?
    2. How can I include all other tables when using an include section? (* does not seem to work when used alone)

    Environment

    • tbls 1.56.2
    • OS Debian bullseye
    • Shell bash
    bug pull request wanted 
    opened by FelixZY 3
  • Support Mermaid output

    Support Mermaid output

    GitHub now supports Mermaid https://github.blog/2022-02-14-include-diagrams-markdown-files-mermaid/. Therefore, it is helpful if tbls outputs ER diagrams as Mermaid figures.

    enhancement pull request wanted 
    opened by arahatashun 0
Releases(v1.56.8)
Owner
Ken’ichiro Oyama
Ken’ichiro Oyama
A Go recursive coverage testing tool

roveralls A Go recursive coverage testing tool. roveralls runs coverage tests on a package and all its sub-packages. The coverage profile is output as

Lawrence Woodman 19 Sep 26, 2022
This is a simple graph database in SQLite, inspired by "SQLite as a document database".

About This is a simple graph database in SQLite, inspired by "SQLite as a document database". Structure The schema consists of just two structures: No

Denis Papathanasiou 1.1k Nov 30, 2022
A lightweight document-oriented NoSQL database written in pure Golang.

Lightweight document-oriented NoSQL Database ???? English | ???? 简体中文 | ???? Spanish CloverDB is a lightweight NoSQL database designed for being simpl

Stefano Scafiti 302 Nov 27, 2022
A rudimentary implementation of a basic document (NoSQL) database in Go

tiedot Documentation Keywords: Golang, go, document database, NoSQL, JSON tiedot - Your NoSQL database powered by Golang tiedot is a document database

Howard 2.7k Nov 14, 2022
Powerful and versatile MIME sniffing package using pre-compiled glob patterns, magic number signatures, XML document namespaces, and tree magic for mounted volumes, generated from the XDG shared-mime-info database.

mimemagic Powerful and versatile MIME sniffing package using pre-compiled glob patterns, magic number signatures, xml document namespaces, and tree ma

Ronen Ulanovsky 83 Nov 3, 2022
Document-oriented, embedded SQL database

Genji Document-oriented, embedded, SQL database Table of contents Table of contents Introduction Features Installation Usage Using Genji's API Using d

Genji 1.1k Nov 29, 2022
Graphik is a Backend as a Service implemented as an identity-aware document & graph database with support for gRPC and graphQL

Graphik is a Backend as a Service implemented as an identity-aware, permissioned, persistant document/graph database & pubsub server written in Go.

null 308 Nov 10, 2022
Schema-free, document-oriented streaming database that optimized for monitoring network traffic in real-time

Basenine Schema-free, document-oriented streaming database that optimized for monitoring network traffic in real-time. Featured Aspects Has the fastes

UP9 38 Nov 2, 2022
sq is a command line tool that provides jq-style access to structured data sources such as SQL databases, or document formats like CSV or Excel.

sq: swiss-army knife for data sq is a command line tool that provides jq-style access to structured data sources such as SQL databases, or document fo

Neil O'Toole 398 Nov 25, 2022
CLI tool to convert many common document types to plane text.

Textify. CLI tool to convert many common document types to plane text. Goals. SO many different document types exist today. PDFs, EPUB books, Microsof

Quin 1 Nov 19, 2021
Online server tool to made markdown document.

An authenticated(basic auth with digest) docsify server for private markdown documents. Embedded docsify-mermaid as UML tool.

null 0 Oct 16, 2021
A user-friendly CMS written in Go (golang)

Fragmenta CMS Fragmenta CMS is a user-friendly Content Management System built with Go. For more information and a demo of the CMS in action, see the

Fragmenta 587 Nov 15, 2022
A user friendly RabbitMQ library written in Golang.

TurboCookedRabbit A user friendly RabbitMQ library written in Golang to help use streadway/amqp. Based on my work found at CookedRabbit. Work Recently

Tristan (HouseCat) Hyams 102 Jul 18, 2022
A lightweight Vault client module written in Go, with no dependencies, that is intuitive and user-friendly

libvault A lightweight Hashicorp Vault client written in Go, with no dependencies. It aims to provide an intuitive, simple API that is easy to use. Ju

null 74 Sep 18, 2022
A clean, safe, user-friendly implementation of GraphQL's Dataloader, written with generics in go

go-dataloader A clean, safe, user-friendly implementation of GraphQL's Dataloader, written with generics in go (go1.18beta1). Features written in gene

yckao 17 Sep 7, 2022
Command line tool for improving typing skills (programmers friendly)

Command line tool for improving typing speed and accuracy. The main goal is to help programmers practise programming languages. Demo Installation Pyth

Jan 375 Dec 5, 2022
🚀 goprobe is a promising command line tool for inspecting URLs with modern and user-friendly way.

goprobe Build go build -o ./bin/goprobe Example > goprobe https://github.com/gaitr/goprobe > cat links.txt | goprobe > echo "https://github.com/gaitr/

null 3 Oct 24, 2021
Dbmate is a database migration tool, to keep your database schema in sync across multiple developers and your production servers.

Dbmate is a database migration tool, to keep your database schema in sync across multiple developers and your production servers. It is a stand

Adrian Macneil 2.9k Nov 30, 2022
A PDF document generator with high level support for text, drawing and images

GoFPDF document generator Package gofpdf implements a PDF document generator with high level support for text, drawing and images. Features UTF-8 supp

Kurt Jung 4.1k Nov 28, 2022