Simple profiling for Go

Overview

profile

Simple profiling support package for Go

Build Status GoDoc

installation

go get github.com/pkg/profile

usage

Enabling profiling in your application is as simple as one line at the top of your main function

import "github.com/pkg/profile"

func main() {
    defer profile.Start().Stop()
    ...
}

options

What to profile is controlled by config value passed to profile.Start. By default CPU profiling is enabled.

import "github.com/pkg/profile"

func main() {
    // p.Stop() must be called before the program exits to
    // ensure profiling information is written to disk.
    p := profile.Start(profile.MemProfile, profile.ProfilePath("."), profile.NoShutdownHook)
    ...
    // You can enable different kinds of memory profiling, either Heap or Allocs where Heap
    // profiling is the default with profile.MemProfile.
    p := profile.Start(profile.MemProfileAllocs, profile.ProfilePath("."), profile.NoShutdownHook)
}

Several convenience package level values are provided for cpu, memory, and block (contention) profiling.

For more complex options, consult the documentation.

contributing

We welcome pull requests, bug fixes and issue reports.

Before proposing a change, please discuss it first by raising an issue.

Comments
  • Proposal

    Proposal

    Because you liked my first suggestion, I took the liberty of suggesting some more changes to simplify readability. Hope they are welcome and I'd love to hear what you think of them. I also hope I didn't break anything :100:

    opened by gbbr 14
  • Multi-modal profiling

    Multi-modal profiling

    Problem Statement

    From my viewpoint, the biggest thing stopping me from using github.com/pkg/profile is its hard stance on "no more than one kind of profiling at a time". Especially since my usual starting assumption is "show me both a CPU profile and a memory (allocs) profile".

    Taking a CPU + memory profile, or a CPU + memory + goroutines + mutexes profile is actually quite a sane thing to do in my viewpoint. Further I do sometimes want to get both a trace and a cpu profile from the same long-running program, rather than go to all the trouble of running it twice.

    So with that introduction, let me lay out my understanding of why this is an okay thing to do (conversely, why the restriction on "one at a time" is actually necessary and overly restrictive).

    Kinds of Profiling

    For the sake of my argument I'm going to focus primarily on the distinction between active and passive profiles.

    There are two kinds of active profiles: CPU profiling and execution tracing. Here the cost of instrumentation is so high that they only run for a set amount of time (say take a 10-second CPU profile from a server, or re-running a tool with trace profiling turned on for the duration).

    On the other hand there are passive profiles including allocs, block, goroutine, heap, mutex, and threadcreate. Here the instrumentation cost is low enough that these profiles are latently always on; their counts are collected all the time, and can be collected at any point in time.

    NOTE this model is of course complicated by the fact that there is a tuning knob for the passive/always-on memory profiler, which pkg/profile is fairly unique in making such a user-facing API-out of (to say nothing of the basically unusable CPU profiler hz tuning knob...)

    Common Combinations

    Of all those lovely modes of profiling CPU, memory, and tracing are the most generally known, used, and useful; as evidenced by the prevalent flag idioms -cpuprofile FILE, -memprofile FILE, and -trace FILE.

    There's really no problem in combining CPU profiling and memory profiling (at normal rates); to the contrary: since the memory profiler is always on any how, you may as well dump it at the end (in the context of an offline tool, which seems to be the use-case that pkg/profile is most suited for). The same goes for any/all of the other passive profilers: they already have their counts just sitting there, you're only doing yourself a harm by not dumping the data.

    The most concern comes when combining CPU profiling and tracing. But even there, at least in my experience, any cross-chatter is fairly easy to pick out or compensate for:

    • in the event trace, you can at least see that a CPU profile was going on, since there will be at least one dedicated goroutine writing it out over some time span (you should even be able to see the os-interrupt timing... but I've not actually tried to do that, and now I'm purely speculating in parenthetical...)
    • there is real concern when it comes to skewing the CPU profile however, since every trace instrumentation branch is now hot, further inflating (the probably already dominant) impact of runtime functions
    • in practice, for online servers, I always sequence tracing and CPU profiling for this reason, rather than do them in parallel; however for an offline tool where your profiling its entire lifecycle, there are times when you want to see both concurrently (even if you also re-run it to also get a "pure" trace and cpu profile)

    Why I Care

    In my view pkg/profile is very close to fully solving the use case of "lifetime profiling for an offline tool / batch task". I'd like to wrap some flag.Value implementation around it and use it as a dependency (maybe even send a pull request for adding the flag convenience).

    However in its current form, not being able to take several at once is a bit of a blocker for that use case.

    opened by jcorbin 11
  • parsing profile: unrecognized profile format under MacOs Sierra 10.12.5

    parsing profile: unrecognized profile format under MacOs Sierra 10.12.5

    Hi, I'm trying to profile my application using your package following the documentation. I have a cpu.pprof non empty file generated but pprof is unable to read it

    $ go tool pprof —pdf ./app /var/folders/x7/wnp2zjn563j2dr45dsfpvdz80000gn/T/profile464041113/cpu.pprof > out.pdf              [email protected]
    parsing profile: unrecognized profile format
    

    I've tried multiple configuration from the default one defer profile.Start().Stop() to a more elaborated defer profile.Start(profile.BlockProfile, profile.MemProfile, profile.CPUProfile).Stop() but pprof is still unable to parse it. Any ideas?

    FYI

    $ go version                                                                                                                  
    go version go1.8.3 darwin/amd64
    

    Thanks

    opened by nsitbon 11
  • Empty CPU Profile

    Empty CPU Profile

    I'm trying to profile a simple application:

    package main
    
    import (
        "github.com/pkg/profile"
        "github.com/spf13/cobra"
        "strings"
    )
    
    /*
    Tokenize TODO
    */
    func Tokenize(text string) []string {
        return strings.Split(text, "\n")
    }
    
    func main() {
        profileCommand := &cobra.Command{
            Use: "profile",
            Run: func(cmd *cobra.Command, args []string) {
                defer profile.Start(profile.CPUProfile, profile.ProfilePath(".")).Stop()
    
                Tokenize("Hello\nWorld")
            },
        }
    
        rootCommand := &cobra.Command{Use: "bern"}
        rootCommand.AddCommand(profileCommand)
        rootCommand.Execute()
    }
    

    But when I look at the cpu.pprof file it simply contains ' and nothing else. What am I doing incorrectly?

    opened by ianwalter 8
  • 10 byte profile file can't be opened

    10 byte profile file can't be opened

    After profiling I get a 10 byte file (cpu.pprof.zip) that when read produces:

    • cpu.pprof: decompressing profile: unexpected EOF ( With go tool pprof cpu.pprof)
    • Unexpected end of ZLIB input stream (With Goland)

    Tried both:

    defer profile.Start(profile.ProfilePath(os.Getenv("HOME"))).Stop()
    

    as well as:

    p := profile.Start(profile.ProfilePath(os.Getenv("HOME")), profile.NoShutdownHook)
    // ...
    p.Stop()
    

    As of 3704c8d23324a3fc0771ad2c1cef0aa4e9459f6f on macOS Big Sur 11.1.

    With the shutdown hook I saw the library was intercepting shutdown to stop profiling. Without it I see my code called to .Stop().

    Troubleshooting ideas?

    Thanks for your work.

    opened by thnk2wn 7
  • profile.Start() should only be called once

    profile.Start() should only be called once

    Now that we've made it impossible to activate more than one profile at a time, users may attempt to work around this with something like

     defer profile.Start(profile.CPUProfile).Stop
     defer profile.Start(profile.MemProfile).Stop
    

    We should decide to support this, and if so, make sure that we move anything which is shared across profiles into the *Profile, see #5. Or, we should prohibit it with some package global lock.

    opened by davecheney 7
  • Switched to switch

    Switched to switch

    I was looking over this with the hope to learn something new and I found it was a great example of using the functional options you spoke about during your talk!

    I thought I'd make this change because I personally find it more readable to use a switch when there are multiple if's, it kind of prepares the reader for a lot of conditions. The fact that Go also allows us to have a "no parameter" switch gives us the opportunity to do such things.

    This is obviously just a personal opinion and you don't have to merge it if you don't agree.

    opened by gbbr 6
  • No support to run multiple profile

    No support to run multiple profile

    profile.Start(profile.MemProfile, profile.ProfilePath("."), profile.NoShutdownHook)
    profile.Start(profile.CPUProfile, profile.ProfilePath("."), profile.NoShutdownHook)
    

    Starting multiple profiles throws an error : profile: Start() already called

    opened by prakashettigar 5
  • TestProfile fails

    TestProfile fails

    I'm trying to create a Debian package for profile. The test TestProfile fails during the build of the packages:

    go test -v -p 4 github.com/pkg/profile
    === RUN   TestProfile
    --- FAIL: TestProfile (1.75s)
    profile_test.go:174: default profile (cpu)
    profile_test.go:174: memory profile
    profile_test.go:174: memory profile (rate 2048)
    profile_test.go:174: double start
    profile_test.go:174: block profile
    profile_test.go:174: profile path
    profile_test.go:174: profile path error
    profile_test.go:194: stderr: wanted '[could not create initial output]', got '2017/05/09 06:50:47 profile: cpu profiling enabled, README.md/cpu.pprof
    	2017/05/09 06:50:47 profile: cpu profiling disabled, README.md/cpu.pprof
    	'
    profile_test.go:209: expected error
    profile_test.go:174: multiple profile sessions
    profile_test.go:174: profile quiet
    
    opened by legrostdg 5
  • 0 Byte files when attempting to generate CPU and Memory profiles

    0 Byte files when attempting to generate CPU and Memory profiles

    I have been trying to collect CPU and Memory profiles for a program I wrote. I have imported "github.com/pkg/profile" and added the appropriate line for either CPU or Memory profile.

    import (
        ...
        "github.com/pkg/profile"
        ...
    )
    
    ...
    
    func main() {
        // memory profile
        defer profile.Start(profile.MemProfile, profile.ProfilePath("./profiles/")).Stop()
        // OR cpu profile
        defer profile.Start(profile.CPUProfile, profile.ProfilePath("./profiles/")).Stop()
    }
    

    When I run the program, I see messages like the following on the command line:

    2019/10/11 12:33:24 profile: memory profiling enabled (rate 4096), profiles/mem.pprof
    

    However, the cpu.pprof or mem.pprof files are always 0 bytes. This used to work before for the same program, I do not understand why this no longer works.

    ls -alhrt profiles
    total 8.0K
    -rw-rw-r-- 1 manas manas    0 Oct 11 12:33 mem.pprof
    drwxrwxr-x 9 manas manas 4.0K Oct 11 12:33 ..
    drwxrwxr-x 2 manas manas 4.0K Oct 11 12:33 .
    
    opened by manasbhatnagar 4
  • How to profile loading of libraries ?

    How to profile loading of libraries ?

    Fantastic tool.

    One thing, say I have a binary that is loading slowly from a massive amount of imports that may possibly be having side effects upon code being loaded.

    I don't see how to use profile here because I put defer profile.Start().Stop() at the beginning of main but that doesn't capture the loading/code execution of library code.

    thank you

    opened by fxfactorial 4
  • Move the CI to Github Actions

    Move the CI to Github Actions

    Travis-ci.org is ceased. I just did the migration to GH actions for two of my repos so when I saw this I thought if you also need to move this can help and took me just few minutes.

    Ignore if you have other plans.

    opened by kavehmz 0
  • Doesn't releases the file even after calling profile.Stop()

    Doesn't releases the file even after calling profile.Stop()

    Failed to remove the files created by the profile pkg "error":"remove mem.pprof: The process cannot access the file because it is being used by another process."

    opened by prakashettigar 1
  • Named interface for Stop()

    Named interface for Stop()

    Problem

    Currently the signature of the Start function is

    func Start(options ...func(*Profile)) interface {
    	Stop()
    } 
    

    Use case

    The lifecycle in our case is managed by another library (fluent-bit-go)

    • The Start() has to happen in lifecycle function FLBPluginInit
    • The Stop() has to happen in lifecycle function FLBPluginExit

    Proposal

    have a named interface for Stop

    type Stopper interface {
    	Stop()
    } 
    

    So the Start will become

    func Start(options ...func(*Profile)) Stopper
    
    opened by kishaningithub 10
  • Timing profiling?

    Timing profiling?

    Hi. I hope everyone is having a great new year so far.

    I was told by a colleague I could use this package for determining how much time is being spent in each function during an application's run. Is this correct and, if so, how? I saw nothing in the documentation which makes this clear if so.

    Thanks in advance.

    opened by FrankDMartinez 7
  • added power support arch ppc64le on yml file.

    added power support arch ppc64le on yml file.

    Hi Team, Added power support for the travis.yml file with ppc64le. This is part of the Ubuntu distribution for ppc64le. This helps us simplify testing later when distributions are re-building and re-releasing. For more info tag @gerrith3. continuous integration build has passed after adding power support arch ppc64le without any failures. please check and close this component.

    opened by srinivas32 0
  • Add ProfileFilename

    Add ProfileFilename

    Every time I use this package, I think that I can do something like:

    var flagCPUProfile = flag.String("cpuprofile", "", "write a cpuprofile to `cpuprofile`")
    
    func main() {
    	if *flagCPUProfile != "" {
    		defer profile.Start(profile.ProfilePath(*flagCPUProfile)).Stop()
    	}
    

    And then I run the program with -cpuprofile=somename.pprof. And there is no profile to be found.

    And then I dig through the docs and discover ProfilePath is supposed to be a directory. And then I wish there was a way to provide a filename. This is because I sometimes do multiple runs, and I want to write the results to different profiles so that I can combine them, and I don't want to have to deal with creating and cleaning up a directory per profile.

    May I send a PR to add ProfileFilename, or something like it?

    opened by josharian 2
Releases(v1.7.0)
  • v1.7.0(Oct 20, 2022)

  • v1.5.0(May 20, 2020)

  • v1.4.0(Nov 21, 2019)

  • v1.2.1(May 10, 2017)

    New features

    • Version 1.2.1 adds support for the Go mutex profile. Thanks @valyala

    Bug fixes

    • Version 1.2.1 also fixes a bug executing tests in some environments where README.md had been removed. Fixes #40
    • Missing Stop() call in ExampleProfilePath was fixed. Thanks @mark-rushakoff
    Source code(tar.gz)
    Source code(zip)
  • v1.2.0(Aug 1, 2016)

    Version 1.2.0 adds support for the Go execution tracer. This feature was added to Go 1.5 but hasn't really been usable til Go 1.7. Support for profile.Start(profile.TraceProfile) is only available in Go 1.7 and later.

    Source code(tar.gz)
    Source code(zip)
  • v1.1.0(Jun 1, 2016)

    Version 1.1.0 adds explicit support for stopping and starting a profile multiple times. This was previously support, but untested, and contained a race condition. The race has been fixed and profiling multiple times during the execution of a program is now supported. Thanks to @valyala for adding this feature.

    Fixes #18, #19

    Source code(tar.gz)
    Source code(zip)
Owner
Artisanal, hand crafted, barrel aged, Go packages
null
pprof is a tool for visualization and analysis of profiling data

Introduction pprof is a tool for visualization and analysis of profiling data. pprof reads a collection of profiling samples in profile.proto format a

Google 6k Dec 3, 2022
Continuous profiling of golang program based on pprof

基于 pprof 的 Golang 程序连续分析 Demo 点击 point Quick Start 需要被收集分析的golang程序,需要提供net/http/pprof端点,并配置在collector.yaml配置文件中 #run server :8080 go run ser

xyctruth 111 Dec 8, 2022
Simple and blazing fast lockfree logging library for golang

glg is simple golang logging library Requirement Go 1.11 Installation go get github.com/kpango/glg Example package main import ( "net/http" "time"

Yusuke Kato 175 Nov 28, 2022
Simple and configurable Logging in Go, with level, formatters and writers

go-log Logging package similar to log4j for the Golang. Support dynamic log level Support customized formatter TextFormatter JSONFormatter Support mul

Guoqiang Chen 13 Sep 26, 2022
Simple logger for Go programs. Allows custom formats for messages.

go-logger A simple go logger for easy logging in your programs. Allows setting custom format for messages. Preview Install go get github.com/apsdehal/

Amanpreet Singh 282 Nov 18, 2022
Simple, configurable and scalable Structured Logging for Go.

log Log is a simple, highly configurable, Structured Logging library Why another logging library? There's allot of great stuff out there, but also tho

Go Playgound 283 Sep 26, 2022
A simple logging module for go, with a rotating file feature and console logging.

A simple logging module for go, with a rotating file feature and console logging. Installation go get github.com/jbrodriguez/mlog Usage Sample usage W

Juan B. Rodriguez 28 Sep 28, 2022
Dead simple, super fast, zero allocation and modular logger for Golang

Onelog Onelog is a dead simple but very efficient JSON logger. It is one of the fastest JSON logger out there. Also, it is one of the logger with the

Francois Parquet 402 Sep 26, 2022
Simple and extensible monitoring agent / library for Kubernetes: https://gravitational.com/blog/monitoring_kubernetes_satellite/

Satellite Satellite is an agent written in Go for collecting health information in a kubernetes cluster. It is both a library and an application. As a

Teleport 197 Nov 10, 2022
A simple daemon which will watch files on your filesystem, mirror them to MFS, automatically update related pins, and update related IPNS keys.

ipfs-sync is a simple daemon which will watch files on your filesystem, mirror them to MFS, automatically update related pins, and update related IPNS keys, so you can always access your directories from the same address. You can use it to sync your documents, photos, videos, or even a website!

null 78 Nov 25, 2022
A simple web service for storing text log files

logpaste A minimalist web service for uploading and sharing log files. Run locally go run main.go Run in local Docker container The Docker container a

Michael Lynch 258 Dec 1, 2022
A simple logging interface for Go

A more minimal logging API for Go Before you consider this package, please read this blog post by the inimitable Dave Cheney. I really appreciate what

null 879 Dec 7, 2022
Very simple charts with some debug data for Go programs

debugcharts Go memory debug charts. This package uses Plotly chart library. It is open source and free for use. Installation go get -v -u github.com/m

Marko Kevac 481 Nov 23, 2022
Distributed simple and robust release management and monitoring system.

Agente Distributed simple and robust release management and monitoring system. **This project on going work. Road map Core system First worker agent M

StreetByters Community 30 Nov 17, 2022
Simple, customizable, leveled and efficient logging in Go

log Simple, customizable, leveled and efficient logging in Go Installation go get -u github.com/ermanimer/log Features log is a simple logging package

Erman İmer 21 Dec 20, 2021
A simple http service that generates *.PDF reports from Grafana dashboards.

Grafana reporter A simple http service that generates *.PDF reports from Grafana dashboards. Requirements Runtime requirements pdflatex installed and

Izak Marais 806 Dec 5, 2022
gtl - Gemini Tiny Logs - A simple TUI for the tinylog format on gemini

GTL: Gemini Tiny Logs Goal: A TUI for the tinylogs format on the gemini space. See screenshots Installation gtl requires go ≥ 1.16 From Source git clo

bacardi55 18 Dec 1, 2022
Simple Yet Powerful Logger

sypl sypl provides a Simple Yet Powerful Logger built on top of the Golang sypl. A sypl logger can have many Outputs, and each Output is responsible f

Sauce Labs 8 Sep 23, 2022
simple concurrent logger

XMUS-LOGGER pure golang logger compatible with golang io standards. USAGE : logOptions := logger.LoggerOptions{ LogLevel: 6, // read more about lo

amupxm [amir hossein mokarrami far] 6 Aug 1, 2022