Tracking down a Memory Leak in Go/SQLite

Overview

Tracking down a Memory Leak in Go/SQLite

run make test - WARNING: long running - several minutes on my workstation


OSs supported:

  • Windows_NT => memory leak not observable - at least not to the same extent
    • uses tasklist to measure process memory footprint
  • Linux => memory leak observed
    • uses ps to measure process memory footprint

First, the test starts a OS child process that

  • contains a "large" in-mem sqlite db filled with dummy data
  • offers a web endpoint http://localhost:8890/dumpdb ... once the db is initialized

Then, the test loops over

  • calling that web endpoint of its testee, who
    • makes an in-mem copy - I call it a "snapshot" - of the db using SQLite's backup mechanism copying memory pages

    • dumps that snapshot as INSERT stmts into a file using github.com/schollz/sqlite3dump

      note: this is a fairly long operation, that blocks the db, which is why we are using that "snapshot" mechanism

    • and [supposedly] disposes that snapshot

  • records memory usage (rss) of the testee using OS tools

At the end, the recorded memory usage is printed for each loop.


example results (process memory footprint - in KB) on my work station:

Iteration Win Lin Lin with WORKAROUND
(file based snapshot db)
0 625’928 592132 593480
1 637’648 1005364 595088
2 637’188 1429180 594708
3 637’396 1429928 594408
4 638’196 1585408 594280
5 636’824 1869536 594568
6 638’256 2347204 594480
7 637’596 2757300 594864
8 638’728 2822888 594816
9 637’732 2822320 597340
10 648’708 3280812 597196
11 671’804 3609528 597048
12 672’932 4133240 598112
13 718’468 4264400 597220
14 834’328 4788828 597144
15 832’080 4854132 597308
16 845’052 4854172 598084
17 832’348 4854140 597512
18 836’280 4914684 597648
19 838’816 5328012 597424
20 844’752 5392820 598224

Some observations

  • Not in every iteration of every run a memory growth can be observed, but running our productive online service 24/7, we can monitor continuous growth over time always leading to a OutOfMemory exception.

  • It seems to be a linux related problem.

    At least this test shows not the same memory growth when run in Windows.

  • We assume the in-mem "snapshot" of an in-mem db to be a part of the problem.

    Why? Our current workaround (search for code comment with WORKAROUND) is to change the "snapshot" db to be a file based db by patching mode in the connection string of the snapshot db: memory -> rwc, which makes the memory growth disappear.

    Why "part of"? Also, the fact of having snapshot db activity (in our case: db dump - search for code comment with `snapshot db activity``) seems to affect the memory behavior. Without snapshot db activity - just change the corresponding code line - the growth seems to be capped after ~6 iterations.

  • the schema complexity of the db seems to influence the problem.

    When [further] simplifying the db schema, the problem seems to disappear even when adjusting the db size of the remaining tables by adding more rows. That is why we left the structure as shown in this repo.

  • As the go managed memory is not increasing, but only the OS view process footprint, we suspect sqlite to be the culprit.

=> our assumption: the snapshot db is not properly "disposed". What's wrong with our code? Or is it a problem somewhere in golang or even deeper in SQLite code?

Owner
Stefan Thiel
Stefan Thiel
Convert data exports from various services to a single SQLite database

Bionic Bionic is a tool to convert data exports from web apps to a single SQLite database. Bionic currently supports data exports from Google, Apple H

Bionic 138 Jun 29, 2022
Pure Go SQLite file reader

Package SQLittle provides pure Go, read-only, access to SQLite (version 3) database files. What SQLittle reads SQLite3 tables and indexes. It iterates

Harmen 181 Jun 10, 2022
Low-level Go interface to SQLite 3

zombiezen.com/go/sqlite This package provides a low-level Go interface to SQLite 3. It is a fork of crawshaw.io/sqlite that uses modernc.org/sqlite, a

Ross Light 262 Jun 19, 2022
Streaming replication for SQLite.

Litestream Litestream is a standalone streaming replication tool for SQLite. It runs as a background process and safely replicates changes incremental

Ben Johnson 6.9k Jun 30, 2022
Go sqlite3 http vfs: query sqlite databases over http with range headers

sqlite3vfshttp: a Go sqlite VFS for querying databases over http(s) sqlite3vfshttp is a sqlite3 VFS for querying remote databases over http(s). This a

Peter Sanford 42 Jun 22, 2022
BQB is a lightweight and easy to use query builder that works with sqlite, mysql, mariadb, postgres, and others.

Basic Query Builder Why Simple, lightweight, and fast Supports any and all syntax by the nature of how it works Doesn't require learning special synta

Aaron M 37 Jun 11, 2022
Experimental implementation of a SQLite backend for go-mysql-server

go-mysql-sqlite-server This is an experimental implementation of a SQLite backend for go-mysql-server from DoltHub. The go-mysql-server is a "frontend

MergeStat 5 May 8, 2022
SQLite extension for accessing other SQL databases

dblite SQLite extension for accessing other SQL databases, in SQLite. Similar to how Postgres Foreign Data Wrappers enable access to other databases i

MergeStat 9 May 20, 2022
Golang database driver for SQLite

go-sqlite Golang database driver for SQLite. Does not use cgo. This driver is based on pure-Go SQLite implementation (https://gitlab.com/cznic/sqlite)

glebarez 43 Jun 23, 2022
RecordLite: a library (and executable) that declaratively maintains SQLite tables and views of semi-structured data

RecordLite RecordLite is a library (and executable) that declaratively maintains

François Saint-Jacques 22 May 29, 2022
Dbench - An unscientific benchmark of SQLite vs the file system (btrfs)

DBENCH Basic benchmarks for SQLite vs file system (btrfs on a 2020 Dell XPS SSD)

Chris Davies 38 May 20, 2022
Sqlair - SQLite Query Layer With Golang

sqlair SQLite Query Layer Creates an abstract over the go sql package to provide

Simon Richardson 1 Feb 18, 2022
Simple key-value store on top of SQLite or MySQL

KV Work in progress, not ready for prime time. A simple key/value store on top of SQLite or MySQL (Go port of GitHub's KV). Aims to be 100% compatible

Sergio Rubio 1 Mar 15, 2022
Minimal memory usage, cloud native logstash alternative

Mr-Plow Tiny and minimal tool to export data from relational db (postgres or mysql) to elasticsearch. The tool does not implement all the logstash fea

RingLoop 28 Apr 27, 2022
This is a simple graph database in SQLite, inspired by "SQLite as a document database".

About This is a simple graph database in SQLite, inspired by "SQLite as a document database". Structure The schema consists of just two structures: No

Denis Papathanasiou 1.1k Jun 21, 2022
Package sqlite is a CGo-free port of SQLite.

sqlite Package sqlite is a CGo-free port of SQLite. SQLite is an in-process implementation of a self-contained, serverless, zero-configuration, transa

Joe 3 Nov 30, 2021
Friends don't let friends leak secrets on their terminal window

senv - safer env Friends don't let friends leak secrets in terminal windows. ?? Print your environment to the terminal without worry.

null 99 Jun 16, 2022
Goroutine leak detector

goleak Goroutine leak detector to help avoid Goroutine leaks. Installation You can use go get to get the latest version: go get -u go.uber.org/goleak

Uber Go 2.7k Jun 26, 2022
Goroutine Leak Detector

Leaktest Refactored, tested variant of the goroutine leak detector found in both net/http tests and the cockroachdb source tree. Takes a snapshot of r

Ian 957 Jun 28, 2022
Continuous profiling for analysis of CPU, memory usage over time, and down to the line number. Saving infrastructure cost, improving performance, and increasing reliability.

Continuous profiling for analysis of CPU, memory usage over time, and down to the line number. Saving infrastructure cost, improving performance, and increasing reliability.

Parca 2.3k Jun 24, 2022
High-performance minimalist queue implemented using a stripped-down lock-free ringbuffer, written in Go (golang.org)

This project is no longer maintained - feel free to fork the project! gringo A high-performance minimalist queue implemented using a stripped-down loc

Darren Elwood 126 Jun 8, 2022
Monitor your Website and APIs from your Computer. Get Notified through Slack, E-mail when your server is down or response time is more than expected.

StatusOK Monitor your Website and APIs from your computer.Get notified through Slack or E-mail when your server is down or response time is more than

Sanath Kumar 1.5k Jun 26, 2022
Control external Fan to cool down your raspi cluster

Fan control for Raspberry Pi This is a small project that I build in order to cool down my raspi home cluster The case I use have some external fans t

Carlos Tadeu Panato Junior 13 Dec 11, 2021
Application to shut down a machine using HTTP requests.

shutdownd Service to shut down a system using HTTP requests. Usage Here's a quick example of how you can use this software on Linux. Download or build

Lerk 1 Nov 15, 2021
A build tool from space, down on earth.

Bob Inspired by Make and Bazel · Made for humans Bob is a build system, a task runner as well as tooling for Git Multi-repos, all bundled into a singl

benchkram 115 Jun 27, 2022
Automatically power off system when network interface is down

passer A tiny tool can automatically power off system when network interface is

Zongzi 0 Apr 23, 2022
Medieval-themed, top-down zombie survival game.

rage-of-ishtar Medieval-themed, top-down zombie survival game. If you do not open the gate for me to come in, I shall smash the door and shatter the b

Luc Capaldi 1 Dec 31, 2021
Automated-gke-cilium-networkpolicy-demo - Quickly provision and tear down a GKE cluster with Cilium enabled for working with Network Policy.

Automated GKE Network Policy Demo Before running the automation, make sure you have the correct variables in env-automation/group_vars/all.yaml. There

Casey Wylie 0 Jan 1, 2022
RaspChecker - A Telegram bot that allows you to monitor your Raspberry Pi's temperature and shut it down. Written in Golang.

RaspChecker Remotely monitor your Raspberry Pi's temperature and shut it down if you want. All through Telegram. How to set up In order to run the bot

Drull 0 Jan 2, 2022