gProfiler combines multiple sampling profilers to produce unified visualization of what your CPU

Related tags

Utilities gprofiler
Overview

gProfiler

gProfiler combines multiple sampling profilers to produce unified visualization of what your CPU is spending time on, displaying stack traces of your processes across native programs1 (includes Golang), Java and Python runtimes, and kernel routines.

gProfiler can upload its results to the Granulate Performance Studio, which aggregates the results from different instances over different periods of time and can give you a holistic view of what is happening on your entire cluster. To upload results, you will have to register and generate a token on the website.

gProfiler runs on Linux.

Granulate Performance Studio example view

Running

This section describes the possible options to control gProfiler's output, and the various execution modes (as a container, as an executable, etc...)

Output options

gProfiler can produce output in two ways:

  • Create an aggregated, collapsed stack samples file (profile_.col) and a flamegraph file (profile_.html). Two symbolic links (last_profile.col and last_flamegraph.html) always point to the last output files.

    Use the --output-dir/-o option to specify the output directory.

    If --rotating-output is given, only the last results are kept (available via last_profle.col and last_flamegraph.html). This can be used to avoid increasing gProfiler's disk usage over time. Useful in conjunction with --upload-results (explained ahead) - historical results are available in the Granulate Performance Studio, and the very latest results are available locally.

    --no-flamegraph can be given to avoid generation of the profile_.html file - only the collapsed stack samples file will be created.

  • Send the results to the Granulate Performance Studio for viewing online with filtering, insights, and more.

    Use the --upload-results/-u flag. Pass the --token option to specify the token provided by Granulate Performance Studio, and the --service-name option to specify an identifier for the collected profiles, as will be viewed in the Granulate Performance Studio. Profiles sent from numerous gProfilers using the same service name will be aggregated together.

Note: both flags can be used simultaneously, in which case gProfiler will create the local files and upload the results.

Profiling options

  • --profiling-frequency: The sampling frequency of the profiling, in hertz.
  • --profiling-duration: The duration of the each profiling session, in seconds.
  • --profiling-interval: The interval between each profiling session, in seconds.

The default profiling frequency is 11 hertz. Using higher frequency will lead to more accurate results, but will create greater overhead on the profiled system & programs.

The default duration is 60 seconds, and the default interval matches it. So gProfiler runs the profiling sessions back-to-back - the next session starts as soon as the previous session is done.

  • --no-java, --no-python: Disable the runtime-specific profilers of Java and/or Python, accordingly.

Continuous mode

gProfiler can be run in a continuous mode, profiling periodically, using the --continuous/-c flag. Note that when using --continuous with --output-dir, a new file will be created during each sampling interval. Aggregations are only available when uploading to the Granulate Performance Studio.

Running as a Docker container

Run the following to have gProfiler running continuously, uploading to Granulate Performance Studio:

docker pull granulate/gprofiler:latest
docker run --name gprofiler -d --restart=always \
    --pid=host --userns=host --privileged \
    -v /lib/modules:/lib/modules:ro -v /usr/src:/usr/src:ro \
    -v /var/run/docker.sock:/var/run/docker.sock \
	granulate/gprofiler:latest -cu --token <token> --service-name <service> [options]

For profiling with eBPF, kernel headers must be accessible from within the container at /lib/modules/$(uname -r)/build. On Ubuntu, this directory is a symlink pointing to /usr/src. The command above mounts both of these directories.

Running as an executable

Run the following to have gprofiler running continuously, uploading to Granulate Performance Studio:

wget https://github.com/Granulate/gprofiler/releases/latest/download/gprofiler
sudo chmod +x gprofiler
sudo ./gprofiler -cu --token <token> --service-name <service> [options]

gProfiler unpacks executables to /tmp by default; if your /tmp is marked with noexec, you can add TMPDIR=/proc/self/cwd to have everything unpacked in your current working directory.

sudo TMPDIR=/proc/self/cwd ./gprofiler -cu --token <token> --service-name <service> [options]

Executable known issues

The following platforms are currently not supported with the gProfiler executable:

  • Ubuntu 14.04
  • Alpine

Remark: container-based execution works and can be used in those cases.

Running as a Kubernetes DaemonSet

See gprofiler.yaml for a basic template of a DaemonSet running gProfiler. Make sure to insert the GPROFILER_TOKEN and GPROFILER_SERVICE variables in the appropriate location!

Running from source

gProfiler requires Python 3.6+ to run.

pip3 install -r requirements.txt
./scripts/build.sh

Then, run the following as root:

python3 -m gprofiler [options]

Theory of operation

Each profiling interval, gProfiler invokes perf in system wide mode, collecting profiling data for all running processes. Alongside perf, gProfiler invokes runtime-specific profilers for processes based on these environments:

  • Java runtimes (version 7+) based on the HotSpot JVM, including the Oracle JDK and other builds of OpenJDK like AdoptOpenJDK and Azul Zulu.
    • Uses async-profiler.
  • The CPython interpreter, versions 2.7 and 3.5-3.9.
    • eBPF profiling (based on PyPerf) requires Linux 4.14 or higher. Profiling using eBPF incurs lower overhead. This requires kernel headers to be installed.
    • If eBPF is not available for whatever reason, py-spy is used.
  • PHP (Zend Engine), versions 7.0-8.0.

The runtime-specific profilers produce stack traces that include runtime information (i.e, stacks of Java/Python functions), unlike perf which produces native stacks of the JVM / CPython interpreter. The runtime stacks are then merged into the data collected by perf, substituting the native stacks perf has collected for those processes.

Contribute

We welcome all feedback and suggestion through Github Issues:

Releasing a new version

  1. Update __version__ in __init__.py.
  2. Create a tag with the same version (after merging the __version__ update) and push it.

We recommend going through our contribution guide for more details.

Credits

Footnotes

1: Currently requires profiled native programs to be compiled with frame pointer.

Issues
  • perf segfaults

    perf segfaults

    On Fedora 35 running Kernel 5.16.19-200.fc35.x86_64, gProfiler's perf segfaults on exeuction.

    # setsid ./gprofiler -cu --token '<my_token>' --service-name 'test' &
    

    Yields

    [16:34:56] Running gprofiler (version 1.2.19), commandline: '-cu --token lN4td5dwQNA5PyF7D6wYQGstP8Vxh-RYPJIv6ZB0eZA --service-name pmemdev1'
    [16:34:56] gProfiler Python version: 3.6.8 (default, Nov 16 2020, 16:55:22)
    [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]
    [16:34:56] gProfiler deployment mode: standalone_executable
    [16:34:56] Kernel uname release: 5.16.19-200.fc35.x86_64
    [16:34:56] Kernel uname version: #1 SMP PREEMPT Fri Apr 8 15:34:44 UTC 2022
    [16:34:56] Total CPUs: 96
    [16:34:56] Total RAM: 250.58 GB
    [16:34:56] Linux distribution: Fedora Linux | 35 |
    [16:34:56] libc version: glibc-2.34
    [16:34:56] Hostname: hostname
    [16:34:57] The connection to the server was successfully established (service 'pmemdev1')
    [16:35:03] Initialized JavaProfiler (frequency: 11hz, duration: 60s)
    [16:35:03] Couldn't create the Java profiler, continuing without it
    Traceback (most recent call last):
      File "gprofiler/profilers/factory.py", line 41, in get_profilers
      File "gprofiler/profilers/java.py", line 608, in __init__
      File "gprofiler/profilers/java.py", line 624, in _init_ap_mode
      File "gprofiler/utils/perf.py", line 19, in can_i_use_perf_events
      File "gprofiler/utils/__init__.py", line 265, in run_process
    gprofiler.exceptions.CalledProcessError: Command '['/tmp/_MEID3BPp6/gprofiler/resources/perf', 'record', '-o', '/dev/null', '--', '/bin/true']' died with <Signals.SIGSEGV: 11>.
    stdout: b''
    stderr: b''
    [16:35:03] Initialized SystemProfiler (frequency: 11hz, duration: 60s)
    [16:35:03] Initialized PythonEbpfProfiler (frequency: 11hz, duration: 60s)
    [16:35:05] Initialized RbSpyProfiler (frequency: 11hz, duration: 60s)
    [16:35:05] Could not find a Docker daemon or CRI-compatible daemon, profiling data will not include the container names. If you do have a containers runtime and it's not supported, please open a new issue here: https://github.com/Granulate/gprofiler/issues/new
    [16:35:05] gProfiler initialized and ready to start profiling
    [16:35:05] Starting profiling of Python processes with PyPerf
    [16:35:06] Starting perf (fp mode)
    [16:35:11] perf failed to start. stdout b'' stderr b''
    [16:35:11] Unexpected error occurred
    Traceback (most recent call last):
      File "gprofiler/main.py", line 771, in main
      File "gprofiler/main.py", line 357, in run_continuous
      File "gprofiler/main.py", line 148, in __enter__
      File "gprofiler/main.py", line 233, in start
      File "gprofiler/profilers/perf.py", line 192, in start
      File "gprofiler/profilers/perf.py", line 72, in start
      File "gprofiler/utils/__init__.py", line 155, in wait_event
    TimeoutError
    

    I see the following in dmesg

    # dmesg -T | tail
    [Fri Apr 15 16:35:02 2022] perf[13307]: segfault at 10 ip 00007fabc1498b34 sp 00007ffc7b92c5b0 error 4 in libssh.so.4.8.7[7fabc1486000+44000]
    [Fri Apr 15 16:35:02 2022] Code: 00 00 00 31 c0 5b c3 b8 ff ff ff ff c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 48 83 ec 08 48 8d 3d 49 c4 04 00 e8 2c fc fe ff <8b> 80 10 00 00 00 48 83 c4 08 c3 90 f3 0f 1e fa 48 85 ff 74 1b 53
    

    Running the command from gProfiler manually also yields a core dump

    # /tmp/_MEID3BPp6/gprofiler/resources/perf record -o /dev/null -- /bin/true
    Segmentation fault (core dumped)
    

    Running the included perf does not

    # /usr/bin/perf record -o /dev/null -- /bin/true
    [ perf record: Woken up 5 times to write data ]
    [ perf record: Captured and wrote 0.000 MB /dev/null ]
    
    opened by sscargal 16
  • Feature/stacks container name

    Feature/stacks container name

    Description

    The V2 profile upload API now sends the container name for stacks from processes that are inside a Docker container An example profile file:

    #{"containers": ["amazing_lamport", "datadog_1"], "hostname": "ip-10-0-0-9", "container_names_disabled": false}
    ;swapper;secondary_startup_64_[k];start_secondary_[k];cpu_startup_entry_[k];do_idle_[k];default_idle_call_[k];arch_cpu_idle_[k];native_safe_halt_[k] 51
    ;swapper;secondary_startup_64_[k];x86_64_start_kernel_[k];x86_64_start_reservations_[k];start_kernel_[k];arch_call_rest_init_[k];rest_init_[k];cpu_startup_entry_[k];do_idle_[k];default_idle_call_[k];arch_cpu_idle_[k];native_safe_halt_[k] 46
    datadog_1;process-agent;[/opt/datadog-agent/embedded/bin/process-agent];[/opt/datadog-agent/embedded/bin/process-agent];_start;_start;_start 1
    ;py-spy;process_vm_readv 1
    amazing_lamport;python;<module> (<string>) 100
    

    Checklist:

    • [x] My code follows the code style of this project.
    • [x] I have updated the relevant documentation.
    • [x] I have read the CONTRIBUTING document.
    • [ ] I have added tests to cover my changes.
    • [ ] All new and existing tests passed.
    opened by alonbentamar 14
  • k8s Daemonset pods stuck in

    k8s Daemonset pods stuck in "CrashLoopBackOff"

    just installed gprofiler daemonset on eks k8s cluster. Pods keep restarting:

    [[email protected] ~]$ kubectl get pods | grep granulate granulate-gprofiler-t7dwh 0/1 CrashLoopBackOff 19 (75s ago) 74m granulate-gprofiler-t855m 0/1 CrashLoopBackOff 19 (96s ago) 74m granulate-gprofiler-w4x96 0/1 CrashLoopBackOff 19 (82s ago) 74m granulate-maestro-6df6db74c9-t26bj 2/2 Running 0 73m

    [[email protected] ~]$ kubectl describe pods granulate-gprofiler-t7dwh Name: granulate-gprofiler-t7dwh Namespace: default Priority: 0 Node: ip-192-168-17-133.us-east-2.compute.internal/192.168.17.133 Start Time: Wed, 20 Jul 2022 19:02:24 +0000 Labels: app=granulate-gprofiler controller-revision-hash=57dbd966bd pod-template-generation=1 Annotations: kubernetes.io/psp: eks.privileged Status: Running IP: 192.168.34.230 IPs: IP: 192.168.34.230 Controlled By: DaemonSet/granulate-gprofiler Containers: granulate-gprofiler: Container ID: docker://90d9dcab2ac6423596e6553fb12b9c651f72e65e5c7bf31bc2b8ef39dbc30d1d Image: index.docker.io/granulate/gprofiler:latest Image ID: docker-pullable://granulate/[email protected]:ab80eda157f96962e0dcbfd4e6f878a185802f5346b5c93c447f085de59203b8 Port: Host Port: Args: -cu --token $(GPROFILER_TOKEN) --service-name $(GPROFILER_SERVICE) State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 2 Started: Wed, 20 Jul 2022 20:45:46 +0000 Finished: Wed, 20 Jul 2022 20:45:47 +0000 Ready: False Restart Count: 25 Limits: cpu: 500m memory: 1Gi Requests: cpu: 100m memory: 256Mi Environment: GPROFILER_TOKEN: -2VzyckD4HHifIF8fuvspXEgHgJ5WRhfdlqVNzzZWDw GPROFILER_SERVICE: EKS_Demo GPROFILER_IN_K8S: 1 Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fpz88 (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: kube-api-access-fpz88: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/disk-pressure:NoSchedule op=Exists node.kubernetes.io/memory-pressure:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists node.kubernetes.io/pid-pressure:NoSchedule op=Exists node.kubernetes.io/unreachable:NoExecute op=Exists node.kubernetes.io/unschedulable:NoSchedule op=Exists Events: Type Reason Age From Message


    Warning BackOff 113s (x492 over 106m) kubelet Back-off restarting failed container [[email protected] ~]$

    opened by brijean 13
  • Support merging the result of fp and dwarf perfs for accuracy

    Support merging the result of fp and dwarf perfs for accuracy

    Description

    In order to increase the accuracy and reliability of perf results, 2 global perfs will run in parallel - one with FP (frame pointers) and one with Dwarf. The result with the highest average of frames per stack will be chosen for the final output, per process.

    How Has This Been Tested?

    I used the new --perf-mode parameter to test all of the different modes - FP, dwarf and "smart" and compared the results.

    Screenshots:

    FP: fp flamegraph

    Dwarf: dwarf flamegraph

    Smart: smart flamegraph

    Types of changes

    • [ ] Bug fix (non-breaking change which fixes an issue)
    • [x] New feature (non-breaking change which adds functionality)
    • [ ] Breaking change (fix or feature that would cause existing functionality to change)

    Checklist:

    • [x] My code follows the code style of this project.
    • [x] I have updated the relevant documentation.
    • [x] I have read the CONTRIBUTING document.
    • [ ] I have added tests to cover my changes.
    • [ ] All new and existing tests passed.
    opened by alonbentamar 12
  • Application metadata

    Application metadata

    Description

    MVP of application metadata: retrieve per-pid information like runtime version, file buildids, ...

    Tasks:

    • [x] Update protocol version to v3
    • [x] Make sure that if v1 is given, no app metadata & no container names are sent (and strip the two ;)
    • [x] Remove code dups
    • [x] ~~Rethink of design with objects instead of classes?~~
    enhancement 
    opened by Jongy 10
  • Profile spawning processes

    Profile spawning processes

    Based on:

    • https://github.com/Granulate/gprofiler/pull/280
    • https://github.com/Granulate/gprofiler/pull/279

    Description

    This PR aims to solve the "short processes" issue: gProfiler runs profiling sessions of 60 seconds (that's the default). Upon each session, the different profilers (JavaProfiler, PySpyProfiler etc) scan through all currently running processes and invoke their respective profilers for each relevant process.

    This works nicely for processes running for long durations (and still, it has some disadvantages which I'll describe later). However, it fails miserably in environments where processes are spawning and exiting quickly. For example, for an app with average lifetime of 10 seconds, we'll only see profiling data from the respective profiler for less than 1/6 of the time! (because on average, we'll catch only one out of 6 processes, and that process might already be halfway through its 10 seconds...). The situation was even worse for Java before https://github.com/Granulate/gprofiler/pull/265, because we didn't read the output file if the process exited before the profiling session has ended. So by design - processes shorter than 60 seconds were not profiled... (py-spy & rb-spy do not suffer from this problem; PyPerf & phpspy, by design, do not suffer from it).

    This PR is the first step towards the "final" design of gProfiler, as I see it. I'll explain what I did here, then explain the next steps.

    The SpawningProcessProfilerBase class was added (name T.B.D). It extends ProcessProfilerBase which is the base class for process-based profilers - profilers invoked per process (AP, py-spy, rbspy). This class extends the existing behavior of ProcessProfilerBase using proc_events. Here's how it works:

    1. SpawningProcessProfilerBase.snapshot() is called
    2. ProcessProfilerBase.snapshot() is called. Upon selecting the currently running processes to profile, it notifies its base classes with the selected processes. At this point, SpawningProcessProfilerBase starts listening for new execs via proc_events (had it been listening earlier, we could "double catch" processes). This section is racy; fixing it is not too hard, but it's irrelevant in the final solution, and the race is not very likely anyway. So I don't intend to do it now.
    3. Upon new execs, we place the psutil.Process object of the new process into a queue. It will now be checked for "should we profile it" multiple times over 1.5 seconds (with some basic exponential backoff logic). The reason we need to check multiple times is that our "should profile" checks are based on parsing /proc/pid/maps. Once the exec completes, libraries are not yet loaded (it is ld.so's job). Furthermore, in the case of Java, libjvm.so is not even an (in)direct dependency of java - it's loaded later by whatever. This may take a short while, also depending on the load on the machine... In the case of Python, we sometimes select processes to profile based on site-packages in their maps - that definitely doesn't show up in maps right after the exec.
    4. Once a spawned process was deemed relevant for profiling, we submit it to _profile_process in a TPE, with the duration decreased by how much time has passed since this session started.
    5. Upon ProcessProfilerBase.snapshot() return, we also stop listening for new execs, and terminate the TPE.
    6. Results from ProcessProfilerBase.snapshot() are merged with the results from our TPE.

    Issues with the current impl:

    • If used for rbspy, py-spy and AP: we're cloning the scheduler thread 3 times, Not sure it's that bad.
    • SpawningProcessProfilerBase contains lots of logic that should be split.

    Roadmap & end goal

    The final idea of gProfiler is this:

    • all profilers run continuously (like perf, PyPerf & phpspy are now).
    • Process-based profilers (AP, py-spy, rbspy) can run indefinitely, and dump output upon signal / request / process exit. So upon each session, we just "ask" profilers to give their current state.
    • gProfiler runs a single thread using proc_events, and decides for each PID if any of the profilers should get it (as an optimization - phpsy & PyPerf should be modified to accept PIDs this way, instead of scanning themselves - scanning has a cost on the system, and thus if we have multiple unrelated "scanners", they run at lower frequency compared to the effectiveness of proc_events.
      • for low-priv executions, where we can't use proc_events, gProfiler will continue scanning for processes, but at a much higher interval (e.g each 0.5 seconds)

    Roadmap:

    1. Implement an alternative to proc_events for low-priv.
    2. Get rid of _select_processes_to_profile - that is, don't "select processes" upon each session start, just select processes based on proc_events, and start profiling until the current session end.
    3. Add support for continuous profiling in AP, rbspy, py-spy. I think that AP might have it (just gotta use the dump action instead of stop), not sure about the others.
    4. Run all profilers independently of the profiling sessions, and just call a "dump data from all currently running py-spys" method.

    How Has This Been Tested?

    T.B.A

    Checklist:

    • [ ] Decide on the final design & implement it
    • [ ] Make sure it is enabled for rbspy & py-spy as well.
    • [ ] Add at least one automatic test, so that it runs in CI.
    • [ ] Document theory of operation (I plan to add that in README)
    • [ ] Document my future plans (as noted shortly in this PR)
    enhancement 
    opened by Jongy 9
  • Java: Check for Hotspot error files after process termination

    Java: Check for Hotspot error files after process termination

    Description

    If java process terminates due to an error, detect it and report a log message. In addition, extract siginfo, native frames, vm info, and container info from the hotspot log and report them in our log.

    Motivation and Context

    Debugging async-profiler misbehaviors.

    How Has This Been Tested?

    1. Changed async-profiler to force a crash.
    2. Started OpenJDK example process.
    3. Ran gprofiler built from this branch.
    4. Ran crashing async-profiler.

    Log output

    [15:40:25] Found Hotspot error log at /proc/1/root/home/ubuntu/jbb/21-11-16_153905/hs_err_pid32629.log
    [15:40:25] Pid 32629 Hotspot VM info: OpenJDK 64-Bit Server VM (25.292-b10) for linux-amd64 JRE (1.8.0_292-8u292-b10-0ubuntu1~18.04-b10), built on Apr 21 2021 11:46:09 by "buildd" with gcc 7.5.0
    [15:40:25] Pid 32629 Hotspot siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000004
    [15:40:25] Pid 32629 Hotspot native frames:
    C  [libc.so.6+0x18e4e1]
    C  [libasyncProfiler.so+0x1bb4e]  Profiler::dump(std::ostream&, Arguments&)+0xce
    C  [libasyncProfiler.so+0x1bcae]  Profiler::runInternal(Arguments&, std::ostream&)+0x9e
    C  [libasyncProfiler.so+0x1c242]  Profiler::run(Arguments&)+0x212
    C  [libasyncProfiler.so+0x48d81]  Agent_OnAttach+0x1e1
    V  [libjvm.so+0x7ea65b]
    V  [libjvm.so+0x2f5e62]
    V  [libjvm.so+0xb08d2f]
    V  [libjvm.so+0xb0a0fa]
    V  [libjvm.so+0x990552]
    C  [libpthread.so.0+0x76db]  start_thread+0xdb
    [15:40:25] Pid 32629 Hotspot container info:
    container_type: cgroupv1
    cpu_cpuset_cpus: 0-3
    cpu_memory_nodes: 0
    active_processor_count: 4
    cpu_quota: -1
    cpu_period: 100000
    cpu_shares: -1
    memory_limit_in_bytes: -1
    memory_and_swap_limit_in_bytes: -2
    memory_soft_limit_in_bytes: -1
    memory_usage_in_bytes: 5046411264
    memory_max_usage_in_bytes: 7147831296
    

    Checklist:

    • [ ] I have read the CONTRIBUTING document.
    • [ ] I have updated the relevant documentation.
    • [ ] I have added tests for new logic.
    Runtime/Java 
    opened by d3dave 9
  • Initial Aarch64 support

    Initial Aarch64 support

    Description

    This PR adds support for building gProfiler's container as Aarch64.

    Only part of the features are supported & enabled:

    • perf is built
    • async-profiler is built
    • py-spy is built
    • gProfiler is built as a container

    This is all done via scripts/build_aarch64_container.sh.

    The following are not supported:

    • ~~py-spy~~, PyPerf, rbspy, phpspy and burn - they are not built & their respective profilers are disabled (in the case of burn, local flamegraphs are not generated - only collapsed files are written).
    • pyi.Dockerfile is not supported - gProfiler can not be built as an executable.

    async-profiler was chosen to be supported first; perf is obvious. Machines based on Aarch64 are modern and usually (if not always) have containers support, hence containers are supported first. We can add exe support later.

    I have tried hard to use the same Dockerfile & to retain it a simple Dockerfile - i.e, avoided falling back to generating it via some script. So, x86_64 can still be built in a standard fashion - docker build . The Aarch64 related consts are embedded in scripts/build_aarch64_container.sh which contains the relevant docker buildx build command.

    What might be improved here is the set of hacks I used to maintain a single Dockerfile. CR fellows, tell me if you think this one is too complex and I'll try to simplify things. Otherwise I suppose it's not too bad and I actually improved a bunch of things along the way ;)

    ~~This doesn't run as part of the CI at the moment (docker buildx build is very slow over qemu). I'll check if we can get some Aarch64 machine for the CI so it could run this build & tests regularly.~~ I checked, it seems like GH actions doesn't support Aarch64, you need a self-hosted runner / use an emulator. Not going to do that now.

    Motivation and Context

    Allows profiling on Linux Aarch64 :)

    How Has This Been Tested?

    Tested perf and async-profiler with some demo apps on this machine:

    [email protected]:~$ uname -a
    Linux xxx 4.18.0-305.10.2.el8_4.aarch64 #1 SMP Tue Jul 20 17:47:41 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
    
    • [x] Run gProfiler's test suite on Aarch64 - see https://github.com/Granulate/gprofiler/pull/170#issuecomment-905311765

    Checklist:

    • [x] I have read the CONTRIBUTING document.
    • [x] I have updated the relevant documentation.
    • [ ] I have added tests for new logic.
    enhancement 
    opened by Jongy 9
  • Use proc_events on Java processes

    Use proc_events on Java processes

    Description

    Use proc_events from granulate-utils to track async-profiled processes and log when they exit with non-zero code.

    Related Issue

    Motivation and Context

    How Has This Been Tested?

    Screenshots

    Checklist:

    • [ ] I have read the CONTRIBUTING document.
    • [ ] I have updated the relevant documentation.
    • [ ] I have added tests for new logic.
    opened by YishaiZinkin 8
  • perf: Add build logic, and upgrade it

    perf: Add build logic, and upgrade it

    ~~Based on https://github.com/Granulate/gprofiler/pull/68~~

    Description

    Dockerfile stuff are based on https://github.com/Granulate/linux/blob/granulate-perf/tools/perf/Dockerfile.

    @DanielShaulov note - bringing back the process comm & not thread.

    Motivation and Context

    For the build logic - see #68. For the upgrade - it's a revert of the revert https://github.com/Granulate/gprofiler/pull/14.

    How Has This Been Tested?

    • [x] On <3.16 kernel - where there's no PERF_RECORD_MISC_COMM_EXEC, what happens
    • [x] Post 3.16 - do we really get the process comm instead of thread?

    Types of changes

    • [ ] Bug fix (non-breaking change which fixes an issue)
    • [x] New feature (non-breaking change which adds functionality)
    • [ ] Breaking change (fix or feature that would cause existing functionality to change)

    Checklist:

    • [x] My code follows the code style of this project.
    • [ ] I have updated the relevant documentation.
    • [x] I have read the CONTRIBUTING document.
    • [ ] I have added tests to cover my changes.
    • [ ] All new and existing tests passed.
    enhancement 
    opened by Jongy 8
  • bugfix: gProfiler excessive restarts when run in a container and failing to grab the mutex

    bugfix: gProfiler excessive restarts when run in a container and failing to grab the mutex

    Change exit code to 0 when failing to grab the mutex. This help gprofiler container avoid excessive restarts when running using K8s or docker

    Description

    Exiting with exit code 0 when failing to grab the mutex.

    Checklist:

    • [ ] I have read the CONTRIBUTING document.
    • [x] I have updated the relevant documentation.
    • [ ] I have added tests for new logic.
    bug 
    opened by tomersa 7
  • when ran gprofiler in minikube using kubernates

    when ran gprofiler in minikube using kubernates

    what connection it's refused, how can i fix this error `

    [15:13:47] Initialized JavaProfiler (frequency: 11hz, duration: 60s)
    [15:13:49] Initialized SystemProfiler (frequency: 11hz, duration: 60s)
    [15:13:49] Initialized PythonEbpfProfiler (frequency: 11hz, duration: 60s)
    [15:13:49] Python eBPF profiler initialization failed
    [15:13:49] Initialized PySpyProfiler (frequency: 11hz, duration: 60s)
    [15:13:49] Initialized RbSpyProfiler (frequency: 11hz, duration: 60s)
    [15:13:49] Discovered container runtimes: ['docker', 'containerd']
    [15:13:49] gProfiler initialized and ready to start profiling
    [15:13:49] Failed to enable proc_events listener for exited Java processes (this does not prevent Java profiling)
    Traceback (most recent call last):
    File "/usr/local/lib/python3.8/dist-packages/granulate_utils/linux/ns.py", line 231, in _switch_and_run
    ret = callback()
    File "/usr/local/lib/python3.8/dist-packages/granulate_utils/linux/proc_events.py", line 249, in _start_listener
    listener.start()
    File "/usr/local/lib/python3.8/dist-packages/granulate_utils/linux/proc_events.py", line 212, in start
    self._register_for_connector_events(self._socket)
    File "/usr/local/lib/python3.8/dist-packages/granulate_utils/linux/proc_events.py", line 125, in _register_for_connector_events
    socket.send(nl_msg)
    ConnectionRefusedError: [Errno 111] Connection refused

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last): File "/app/gprofiler/profilers/java.py", line 1050, in start proc_events.register_exit_callback(self._proc_exit_callback) File "/usr/local/lib/python3.8/dist-packages/granulate_utils/linux/proc_events.py", line 261, in wrapper _proc_events_listener = run_in_ns(["net"], _start_listener) File "/usr/local/lib/python3.8/dist-packages/granulate_utils/linux/ns.py", line 246, in run_in_ns raise Exception("run_in_ns execution failed") from exc Exception: run_in_ns execution failed [15:13:49] Starting perf (fp mode) [15:13:49] Started perf (fp mode)

    `

    opened by sridharchary 0
  • got issue when ran gprofiler in local and kubernets server cluster

    got issue when ran gprofiler in local and kubernets server cluster

    can anyone help me to how to fix this issue

    from granulate_utils.linux.elf import get_mapped_dso_elf_id ImportError: cannot import name 'get_mapped_dso_elf_id' from 'granulate_utils.linux.elf' (/home/ubuntu/Desktop/gprofiler/granulate_utils/linux/elf.py)

    from granulate_utils.linux.elf import read_process_execfn ImportError: cannot import name 'read_process_execfn' from 'granulate_utils.linux.elf' (/home/ubuntu/.local/lib/python3.8/site-packages/granulate_utils/linux/elf.py)

    opened by sridharchary 0
  • perf: Parse new event family names

    perf: Parse new event family names

    This field used to be "cpu-clock:", with the new perf (and on a recent kernel) I get "cpu_core/cycles/", so add _ and / to the pattern.

    It's probably for the better to remove this complex regex and remain only with parsing the stack itself. I didn't want to make a larger change at this point, given that this already works, and that we don't have good tests coverage here.

    bug 
    opened by Jongy 1
  • Pass --token, --service-name with = everywhere

    Pass --token, --service-name with = everywhere

    PR #325 did it partially. Finish the job here.

    Fixes the issue from https://github.com/Granulate/gprofiler/issues/417.

    I manually tested the new k8s DaemonSet and it works fine.

    The pattern --token[^=] doesn't have any other matches :)

    enhancement 
    opened by Jongy 0
  • windows support for gprofiler with py-spy

    windows support for gprofiler with py-spy

    PR provides batch scripts to build py-spy and gprofiler using py-spy on Windows

    Description

    Provided batch scripts to build py-spy and gprofiler from source on Windows. This PR contains a build.bat file, a build-pyspy.bat file, py-spy (v0.3.10g1) source, gprofiler source along with some Windows specific dependencies. The additions can all be found in the Windows subfolder.

    Motivation and Context

    This PR addresses Granulates initial foray into providing Windows support. To be specific, this PR provides python profiling on Windows using py-spy.

    opened by teoluwaf 0
Releases(1.4.0)
Owner
Granulate
Granulate
CPU feature identification for Go

cpuid Package cpuid provides information about the CPU running the current program. CPU features are detected on startup, and kept for fast access thr

Klaus Post 715 Aug 4, 2022
Automatically set GOMAXPROCS to match Linux container CPU quota.

automaxprocs Automatically set GOMAXPROCS to match Linux container CPU quota. Installation go get -u go.uber.org/automaxprocs Quick Start import _ "go

Uber Go 2.1k Aug 2, 2022
Continuous profiling for analysis of CPU, memory usage over time, and down to the line number. Saving infrastructure cost, improving performance, and increasing reliability.

Continuous profiling for analysis of CPU, memory usage over time, and down to the line number. Saving infrastructure cost, improving performance, and increasing reliability.

Parca 2.5k Aug 7, 2022
A tool to find redirection chains in multiple URLs

UnChain A tool to find redirection chains in multiple URLs Introduction UnChain automates process of finding and following `30X` redirects by extracti

RedCode Labs 79 Aug 2, 2022
go-i18n is a Go package and a command that helps you translate Go programs into multiple languages.

go-i18n is a Go package and a command that helps you translate Go programs into multiple languages.

Nick Snyder 2.1k Jul 27, 2022
LogAnalyzer - Analyze logs with custom regex patterns.Can search for particular patterns on multiple files in a directory.

LogAnalyzer Analyze logs with custom regex patterns.Can search for particular patterns on multiple files in a directory

Johnson Simon 6 May 31, 2022
Split multiple Kubernetes files into smaller files with ease. Split multi-YAML files into individual files.

Split multiple Kubernetes files into smaller files with ease. Split multi-YAML files into individual files.

Patrick D'appollonio 170 Jul 28, 2022
Split multiple Kubernetes files into smaller files with ease. Split multi-YAML files into individual files.

kubectl-slice: split Kubernetes YAMLs into files kubectl-slice is a neat tool that allows you to split a single multi-YAML Kubernetes manifest into mu

Patrick D'appollonio 170 Jul 28, 2022
Tiny Go tool for running multiple functions concurrently and collecting their results into an error slice.

Overview Short for "ConCurrent". Tiny Go tool for running multiple functions concurrently and collecting their results into an error slice. Dependency

Nelo Mitranim 0 Nov 22, 2021
Small tool for splitting files found in a path into multiple groups

Small tool for splitting files found in a path into multiple groups. Usefull for parallelisation of whatever can be paralleled with multiple files.

Antonio Martinović 0 Jan 30, 2022
Hotswap provides a solution for reloading your go code without restarting your server, interrupting or blocking any ongoing procedure.

Hotswap provides a solution for reloading your go code without restarting your server, interrupting or blocking any ongoing procedure. Hotswap is built upon the plugin mechanism.

Edwin 91 Jul 28, 2022
Get cloud instances with your favourite software pre-loaded

This Golang package can be used to provision cloud hosts using a simple CRUD-style API along with a cloud-init user-data script. It could be used to automate anything from k3s clusters, to blogs, or CI runners. We use it to create the cheapest possible hosts in the cloud with a public IP address.

inlets 32 Jul 22, 2022
流媒体NetFlix解锁检测脚本 / A script used to determine whether your network can watch native Netflix movies or not

netflix-verify 流媒体NetFlix解锁检测脚本,使用Go语言编写 在VPS网络正常的情况下,哪怕是双栈网络也可在几秒内快速完成IPv4/IPv6的解锁判断 鸣谢 感谢 @CoiaPrant 指出对于地域检测更简便的方法 感谢 @XmJwit 解决了IPV6 Only VPS无法下载脚

null 1.7k Jul 29, 2022
Visualize your Go data structures using graphviz

memviz How would you rather debug a data structure? "Pretty" printed Visual graph (*test.fib)(0xc04204a5a0)({ index: (int) 5, prev: (*test.fib)(0xc0

Bradley Kemp 1.3k Jul 17, 2022
Dynamically generated Last.fm stats for your profile readme

GitHub Readme Last.fm Stats Dynamically generated last.fm stats in your profile readme Contents Usage Options Demo Development & Deployment Issues, Re

Rafael Wiska-Ilnicki 6 Mar 17, 2022
this is an api that execute your deno code and send you the output

this a simple api that execute your deno code and send you the output, has not limit per request example request: in deno: const rawResponse = await f

pai 13 May 28, 2022
🚀 Use Lanyard API easily in your Go app!

?? Go Lanyard Use Lanyard API easily in your Go app! ?? Installation Initialize your project (go mod init example.com/example) Add package (go get git

Barış DEMİRCİ 8 Mar 11, 2022
A simple API for computing diffs of your documents over the time built on a scalable technology stack.

Diffme API WIP - this is an API to compute diffs between documents. It serves as a way to easily create audit logs for documents in your system, think

diffme 11 Sep 8, 2021
A comprehensive list of alternatives to your favorite software

alternativeto A comprehensive list of alternatives to your favorite software. Please do not edit this file directly. Instead, follow the steps outline

Brent Adamson 33 Jun 16, 2022