Stochastic flame graph profiler for Go programs

Overview

go-torch Build Status Coverage Status GoDoc

go-torch is deprecated, use pprof instead

As of Go 1.11, flamegraph visualizations are available in go tool pprof directly!

# This will listen on :8081 and open a browser.
# Change :8081 to a port of your choice.
$ go tool pprof -http=":8081" [binary] [profile]

If you cannot use Go 1.11, you can get the latest pprof tool and use it instead:

# Get the pprof tool directly
$ go get -u github.com/google/pprof

$ pprof -http=":8081" [binary] [profile]

Synopsis

Tool for stochastically profiling Go programs. Collects stack traces and synthesizes them into a flame graph. Uses Go's built in pprof library.

Example Flame Graph

Inception

Basic Usage

$ go-torch -h
Usage:
  go-torch [options] [binary] <profile source>

pprof Options:
  -u, --url=         Base URL of your Go program (default: http://localhost:8080)
  -s, --suffix=      URL path of pprof profile (default: /debug/pprof/profile)
  -b, --binaryinput= File path of previously saved binary profile. (binary profile is anything accepted by https://golang.org/cmd/pprof)
      --binaryname=  File path of the binary that the binaryinput is for, used for pprof inputs
  -t, --seconds=     Number of seconds to profile for (default: 30)
      --pprofArgs=   Extra arguments for pprof

Output Options:
  -f, --file=        Output file name (must be .svg) (default: torch.svg)
  -p, --print        Print the generated svg to stdout instead of writing to file
  -r, --raw          Print the raw call graph output to stdout instead of creating a flame graph; use with Brendan Gregg's flame graph perl script (see https://github.com/brendangregg/FlameGraph)
      --title=       Graph title to display in the output file (default: Flame Graph)
      --width=       Generated graph width (default: 1200)
      --hash         Colors are keyed by function name hash
      --colors=      Set color palette. Valid choices are: hot (default), mem, io, wakeup, chain, java,
                     js, perl, red, green, blue, aqua, yellow, purple, orange
      --hash         Graph colors are keyed by function name hash
      --cp           Graph use consistent palette (palette.map)
      --inverted     Icicle graph
Help Options:
  -h, --help         Show this help message

Write flamegraph using /debug/pprof endpoint

The default options will hit http://localhost:8080/debug/pprof/profile for a 30 second CPU profile, and write it out to torch.svg

$ go-torch
INFO[19:10:58] Run pprof command: go tool pprof -raw -seconds 30 http://localhost:8080/debug/pprof/profile
INFO[19:11:03] Writing svg to torch.svg

You can customize the base URL by using -u

$ go-torch -u http://my-service:8080/
INFO[19:10:58] Run pprof command: go tool pprof -raw -seconds 30 http://my-service:8080/debug/pprof/profile
INFO[19:11:03] Writing svg to torch.svg

Or change the number of seconds to profile using --seconds:

$ go-torch --seconds 5
INFO[19:10:58] Run pprof command: go tool pprof -raw -seconds 5 http://localhost:8080/debug/pprof/profile
INFO[19:11:03] Writing svg to torch.svg

Using pprof arguments

go-torch will pass through arguments to go tool pprof, which lets you take existing pprof commands and easily make them work with go-torch.

For example, after creating a CPU profile from a benchmark:

$ go test -bench . -cpuprofile=cpu.prof

# This creates a cpu.prof file, and the $PKG.test binary.

The same arguments that can be used with go tool pprof will also work with go-torch:

$ go tool pprof main.test cpu.prof

# Same arguments work with go-torch
$ go-torch main.test cpu.prof
INFO[19:00:29] Run pprof command: go tool pprof -raw -seconds 30 main.test cpu.prof
INFO[19:00:29] Writing svg to torch.svg

Flags that are not handled by go-torch are passed through as well:

$ go-torch --alloc_objects main.test mem.prof
INFO[19:00:29] Run pprof command: go tool pprof -raw -seconds 30 --alloc_objects main.test mem.prof
INFO[19:00:29] Writing svg to torch.svg

Integrating With Your Application

To add profiling endpoints in your application, follow the official Go docs here. If your application is already running a server on the DefaultServeMux, just add this import to your application.

import _ "net/http/pprof"

If your application is not using the DefaultServeMux, you can still easily expose pprof endpoints by manually registering the net/http/pprof handlers or by using a library like this one.

Installation

$ go get github.com/uber/go-torch

You can also use go-torch using docker:

$ docker run uber/go-torch -u http://[address-of-host] -p > torch.svg

Using -p will print the SVG to standard out, which can then be redirected to a file. This avoids mounting volumes to a container.

Get the flame graph script:

When using the go-torch binary locally, you will need the Flamegraph scripts in your PATH:

$ cd $GOPATH/src/github.com/uber/go-torch
$ git clone https://github.com/brendangregg/FlameGraph.git

Development and Testing

Install the Go dependencies:

$ go get github.com/Masterminds/glide
$ cd $GOPATH/src/github.com/uber/go-torch
$ glide install

Run the Tests

$ go test ./...
ok    github.com/uber/go-torch   0.012s
ok    github.com/uber/go-torch/graph   0.017s
ok    github.com/uber/go-torch/visualization 0.052s
Issues
  • Simplify go-torch: use raw pprof output, and use go-flags

    Simplify go-torch: use raw pprof output, and use go-flags

    Simplify go-torch:

    • use go-flags instead of cli since we only need flags, we don't have subcommands etc. This simplifies the code and lets us use structs to access the command line flags.
    • Use raw output from pprof rather, which contains every single sample that we can convert to call stacks for flame graph scripts
    • reduce use of interfaces, test the real concrete implementations by having tests invoke "go tool pprof" and creating fake flamegraph scripts in our PATH for tests etc
    • removed logrus to reduce dependencies since we don't really need a smart logger.

    A lot of logic has been moved around into 2 packages: pprof: for invoking pprof and parsing the raw output from it renderer: for invoking flamegraph scripts

    r: @sectioneight

    opened by prashantv 11
  • Add dockerfile and Glide files

    Add dockerfile and Glide files

    Hi!

    I added a couple of changes to the project:

    1. A brand new Dockerfile, so people can use go-torch from within a docker container using docker run uber/go-torch go-torch ...
    2. I removed the old Godeps file and added Glide files as requested.
    opened by calavera 7
  • pprof: fix incorrect parsing for locations

    pprof: fix incorrect parsing for locations

    Fixes #61.

    Extrapolating from https://github.com/google/pprof/blob/master/profile/profile.go#L360 and https://github.com/uber/go-torch/issues/61 we can have locations of the following format:

    <id>: 0x<address> M=<mapping id>
    

    Note: This doesn't address all malformed inputs, malformed inputs that may have gotten through before will still do so. This patch only addresses #61.

    Added the raw pprof out as an additional test for one of our binaries that failed prior.

    opened by irfansharif 6
  • Drop all but first line during location parsing

    Drop all but first line during location parsing

    This is a fix but not really a good solution for #63. The problem arises because the raw format (at least in go1.9) can contain lines of the form

    [...]
       729: 0x4d6a395 github.com/cockroachdb/cockroach/pkg/sql.(*tableState).release /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:791 s=0
       730: 0x4021625 runtime.heapBits.next /usr/local/Cellar/go/1.9beta2/libexec/src/runtime/mbitmap.go:464 s=0
                 runtime.scanobject /usr/local/Cellar/go/1.9beta2/libexec/src/runtime/mgcmark.go:1162 s=0
    [...]
    

    and the last line is not parsed correctly. The way parsing is set up is line-by-line, so it's not straightforward to recover. Instead of a bigger refactor to try to address this, this change just drops these extra lines, and we should really implement #64.

    Note also that I changed some lines in the parsing test to match what is actually produced by go1.9 -- I'm not sure if the previous snippet was simply not true-to-original or whether perhaps the raw format has changed. Either way, using the raw format seems to invite regular breakage.

    Closes #63.

    opened by tbg 4
  • Improve help + flags, update README

    Improve help + flags, update README

    Flags now pass through unknown flags (like --alloc_objects) to go tool pprof, and are broken into multiple sections.

    Restructure the README into:

    • Usage
    • Integration
    • Installation
    • Development

    Can see the rendered output here: https://github.com/uber/go-torch/blob/readme/README.md

    Focus examples on more common usage, and remove less frequently used examples like outputting an SVG to stdout or outputting the raw flamegraph input.

    opened by prashantv 4
  • Fix sample selection to not always use first index

    Fix sample selection to not always use first index

    Currently, go-torch always uses the first sample it finds in the profile. With 1.8, memory profiles always have inuse/alloc so we need to select the right sample, otherwise we're always showing alloc.

    This allows allows selection of a custom sample using sample_index for non-memory profiles.

    Fixes #52.

    opened by prashantv 3
  • Add perl to Docker image, and improve README instructions

    Add perl to Docker image, and improve README instructions

    Fixes "no such file or directory" error from #41

    @sectioneight: once we merge this, can you start a new build. We should look into automating the builds as well.

    cc @calavera

    opened by prashantv 3
  • Allow to load the pprof raw data from a file.

    Allow to load the pprof raw data from a file.

    This change allows to analyze historic data without having to run pprof in real time. It's also useful when you cannot connect go-torch to a production server but the server has pprof installed.

    It adds the flag load to the CLI. This flag takes the file path to raw data. Go-torch tries to run the profiler as usual if this flag is empty.

    PS: Thanks for this project, it has already saved me lots of time.

    Signed-off-by: David Calavera [email protected]

    opened by calavera 3
  • pprof requires binary as well as pprof output when using files.

    pprof requires binary as well as pprof output when using files.

    from pprof usage: usage: pprof [options] [binary] ...

    require a binaryname arg when using binaryfile. verified this with local -cpuprofile output from tests.

    r @sandlerben

    opened by prashantv 3
  • Fix pprof parsing when there's more than 2 columns

    Fix pprof parsing when there's more than 2 columns

    Previously, the pprof parser assumed there were 2 columns, and stored them as "samples" and "total". Instead parse all values and store columns which we can use in future.

    Fixes #51 but I noticed a new issue with how we select the sample to display: #52

    opened by prashantv 2
  • Support pprof arg format in go-torch

    Support pprof arg format in go-torch

    pprof is usually run with either:

    go tool pprof [binary-name] [binary-file]
    

    or

    go tool pprof [url]
    

    This change makes it so go tool pprof can be replaced with go-torch and it just works. We support --seconds as well as --time (for backwards compatibility).

    opened by prashantv 2
  • Improve instructions on fetching flamegraph scripts

    Improve instructions on fetching flamegraph scripts

    The first time I set this up on a new machine it pointed me to the repo, but didn't say which files I needed. So it took some trial and error before I got it working. We should be explicit in what we use.

    opened by ascandella 6
Owner
Uber Archive
Uber's open source projects archive. Support and/or new releases are not available.
Uber Archive
Nats-subject-profiler - NATS Subject Profiler With Golang

NATS Subject Profiler Example Connect it to the demo NATS server. nats-subject-p

Byron Ruth 2 Feb 7, 2022
A highly-performant command runtime server written in Go for Flame 🔥

Flame Command Runtime A highly-performant command runtime server written in Go for Flame ?? Flame Command Runtime(FCR) is powerful, highly customizabl

Flame 3 Aug 29, 2021
FlameScope is a visualization tool for exploring different time ranges as Flame Graphs.

FlameScope FlameScope is a visualization tool for exploring different time ranges as Flame Graphs, allowing quick analysis of performance issues such

Netflix, Inc. 2.7k Jun 23, 2022
JSONL graph tools - Graph is represented as JSONL of nodes and edges.

JSONL graph tools - Graph is represented as JSONL of nodes and edges.

Nikolay Dubina 58 Apr 22, 2022
The DGL Operator makes it easy to run Deep Graph Library (DGL) graph neural network training on Kubernetes

DGL Operator The DGL Operator makes it easy to run Deep Graph Library (DGL) graph neural network distributed or non-distributed training on Kubernetes

Qihoo 360 40 Apr 28, 2022
🚀 fgprof is a sampling Go profiler that allows you to analyze On-CPU as well as Off-CPU (e.g. I/O) time together.

?? fgprof - The Full Go Profiler fgprof is a sampling Go profiler that allows you to analyze On-CPU as well as Off-CPU (e.g. I/O) time together. Go's

Felix Geisendörfer 2.2k Jun 27, 2022
🚀Statsview is a real-time Golang runtime stats visualization profiler

Statsview is a real-time Golang runtime stats visualization profiler. It is built top on another open-source project, go-echarts, which helps statsview to show its graphs on the browser.

go-echarts 486 Jun 26, 2022
System resource usage profiler tool which regularly takes snapshots of the memory and CPU load of one or more running processes so as to dynamically build up a profile of their usage of system resources.

Vegeta is a system resource usage tracking tool built to regularly take snapshots of the memory and CPU load of one or more running processes, so as to dynamically build up a profile of their usage of system resources.

Kartik 8 Jan 16, 2022
Progress OpenEdge Profiler data parsing to OpenTracing format

openedge-profiler-parser Progress OpenEdge Profiler data parsing to OpenTracing format. Prerequisites In order to RUN you will be enough with Docker:

Baltic Amadeus 4 Nov 9, 2021
A simple application to show how to use dd-trace-go's tracer and profiler.

dd-trace-go-demo A simple application to show how to use dd-trace-go's tracer and profiler. Usage To run this demo application, simply clone this repo

Felix Geisendörfer 3 Mar 2, 2022
Metrics collector and ebpf-based profiler for C, C++, Golang, and Rust

Apache SkyWalking Rover SkyWalking Rover: Metrics collector and ebpf-based profiler for C, C++, Golang, and Rust. Documentation Official documentation

The Apache Software Foundation 42 Jun 29, 2022
BTree provides a simple, ordered, in-memory data structure for Go programs.

BTree implementation for Go This package provides an in-memory B-Tree implementation for Go, useful as an ordered, mutable data structure. The API is

Google 2.8k Jun 28, 2022
webbased debugging for Go programs

Hopwatch, a debugging tool for Go Hopwatch is a simple tool in HTML5 that can help debug Go programs. It works by communicating to a WebSockets based

Ernest Micklei 256 Jun 20, 2022
An embeddable implementation of the Ngaro Virtual Machine for Go programs

Ngaro Go Overview This is an embeddable Go implementation of the Ngaro Virtual Machine. This repository contains the embeddable virtual machine, a rud

Denis Bernard 22 Jan 23, 2022
Read and write Netpbm images from Go programs

netpbm Introduction netpbm is a package for the Go programming language that implements image decoders and encoders for the Netpbm image formats. The

Scott Pakin 15 Aug 31, 2021
Simple logger for Go programs. Allows custom formats for messages.

go-logger A simple go logger for easy logging in your programs. Allows setting custom format for messages. Preview Install go get github.com/apsdehal/

Amanpreet Singh 276 Jun 14, 2022
Example programs for the Gio project.

Gio Examples Example programs for the Gio project. Issues File bugs and TODOs through the issue tracker or send an email to ~eliasnaur/[email protected]

Gio 67 Jun 8, 2022
Simple project to demonstrate the loading of eBPF programs via florianl/go-tc.

tc-skeleton Simple project to demonstrate the loading of eBPF programs via florianl/go-tc.

Florian Lehner 28 May 22, 2022
Library to work with eBPF programs from Go

Go eBPF A nice and convenient way to work with eBPF programs / perf events from Go. Requirements Go 1.10+ Linux Kernel 4.15+ Supported eBPF features e

Dropbox 933 Jun 28, 2022
7 days golang programs from scratch (web framework Gee, distributed cache GeeCache, object relational mapping ORM framework GeeORM, rpc framework GeeRPC etc) 7天用Go动手写/从零实现系列

7 days golang programs from scratch README 中文版本 7天用Go从零实现系列 7天能写什么呢?类似 gin 的 web 框架?类似 groupcache 的分布式缓存?或者一个简单的 Python 解释器?希望这个仓库能给你答案

Dai Jie 10.7k Jun 26, 2022
Build self-updating Go programs

selfupdate: Build self-updating Go programs NOTE: Original work at github.com/inconshreveable/go-update, modified for the needs within MinIO project P

High Performance, Kubernetes Native Object Storage 202 Jun 23, 2022
Very simple charts with some debug data for Go programs

debugcharts Go memory debug charts. This package uses Plotly chart library. It is open source and free for use. Installation go get -v -u github.com/m

Marko Kevac 476 Jun 19, 2022
A GitLab API client enabling Go programs to interact with GitLab in a simple and uniform way

go-gitlab A GitLab API client enabling Go programs to interact with GitLab in a simple and uniform way NOTE Release v0.6.0 (released on 25-08-2017) no

Xanzy 1.7k Jun 27, 2022
A GitLab API client enabling Go programs to interact with GitLab in a simple and uniform way

A GitLab API client enabling Go programs to interact with GitLab in a simple and uniform way

Xanzy 1.7k Jun 28, 2022
go-i18n is a Go package and a command that helps you translate Go programs into multiple languages.

go-i18n is a Go package and a command that helps you translate Go programs into multiple languages.

Nick Snyder 2k Jun 26, 2022
Generate Go clients from anchor IDLs for Solana blockchain programs

usage anchor-go --src=/path/to/idl.json Generated Code will be generated and saved to ./generated/. TODO instructions accounts types events errors han

Slavomir 69 Jun 30, 2022
This manager helps handle the life cycle of your eBPF programs

eBPF Manager This repository implements a manager on top of Cilium's eBPF library. This declarative manager simplifies attaching and detaching eBPF pr

Datadog, Inc. 22 Jun 20, 2022
Go clients for the Metaplex Solana programs

metaplex-go A suite of Go clients for the 5 metaplex contracts. This is an alpha version. For usage examples, you can get inspired by their Rust/Types

Slavomir 23 Jun 22, 2022
A GitLab API client enabling Go programs to interact with GitLab in a simple and uniform way

TEMPORARY REPOSITORY This repo will exist only temporarily waiting for the PR has been merged.

null 0 Nov 5, 2021