Trace Go program execution with uprobes and eBPF

Overview

Weaver

PLEASE READ! - I am currently refactoring Weaver to use libbpf instead of bcc which would include various other major improvements. If you're currently using weaver please be aware that features/bug fixes are being held off until the major refactor occurs. This will be tracked in the branch "refactor"

gopher

Weaver is a CLI tool that allows you to trace Go programs in order to inspect what values are passed to specified functions. It leverages eBPF attached to uprobes.

Go Report Card

Quick Start

There are two modes of operation, one that uses a 'functions file', and one that extracts a symbol table from a passed binary and filters by Go packages. More information on functionality in docs.

Functions file

Take the following example program:

test_prog.go

package main

//go:noinline
func test_function(int, [2]int) {}

//go:noinline
func other_test_function(rune, int64) {}

func main() {
	test_function(3, [2]int{1, 2})
	other_test_function('a', 33)
}

Let's say we want to know what values are passed to test_function and other_test_function whenever the program is run. Once the program is compiled (make) we just have to create a file which specifies each function to trace:

functions_to_trace.txt

main.test_function(int, [2]int)
main.other_test_function(rune, int64)

Notice that we have to specify the parameter data types. (You can use weaver --types to see what data types are supported.)

Now we can call weaver like so:

sudo weaver -f /path/to/functions_to_trace.txt /path/to/test-prog-binary

Weaver will then sit idle without any output until test-prog is run and the test_function and other_test_function functions are called. This will also work on an already running Go Program.

{"functionName":"main.other_test_function","args":[{"type":"RUNE","value":"a"},{"type":"INT64","value":"33"}],"procInfo":{"pid":43300,"ppid":42754,"comm":"test-prog-binar"}}
{"functionName":"main.test_function","args":[{"type":"INT","value":"3"},{"type":"INT_ARRAY","value":"1, 2"}],"procInfo":{"pid":43300,"ppid":42754,"comm":"test-prog-binar"}}

Package mode

For the same example Go program as above, you can choose to not specify a functions file. The command would like like this:

sudo weaver /path/to/test-prog-binary

This will default to only tracing functions in the main package, however you can use the --packages flag to specify a comma seperated list of packages (typially of the form github.com/x/y)

Output does include argument vlaues in this mode.

{"functionName":"main.main","procInfo":{"pid":44411,"ppid":42754,"comm":"test-prog-binar"}}
{"functionName":"main.test_function","procInfo":{"pid":44411,"ppid":42754,"comm":"test-prog-binar"}}

Note on supported types

Currently weaver supports basic data types but getting support for user defined types is a high priority. Getting following types defined are a work in progress:

  • user/stdlib defined structs
  • user/stdlib defined interfaces

System Dependencies

  • bcc / bcc-devel
  • linux kernel version > 4.14 (please make bug reports if your kernel version doesn't work)

Build

make will compile the weaver binary to bin/weaver (It also creates the smoke test binary and print-stack utility)

Can't build? Please make an issue!

Roadmap

Check issues for tasks currently being tracked. Please open bug reports, i'm sure there are plenty :-)

Short term goals include:

  • Testing
  • Output options
  • Inspecting binaries for parameter data types instead of specifying them with a functions file
  • CI/CD infrastructre

image modified version of art by Ashley McNamara (license) based on art by Renee French.

Comments
  • Support loading probes on struct methods and minor bugfixes

    Support loading probes on struct methods and minor bugfixes

    Hi, I added support for struct methods

    • Parsing struct method and functions containing /
    • Adding 8 byte of offset (struct pointer) when loading probe on struct method
    • Added one test (we might want to add more...)
    • Kill weaver process even if diff command fail in run_smoke_test.sh
    • Do not attach probe if context is empty
    opened by rrreeezzz 4
  • Add 'package' mode, remove table writer

    Add 'package' mode, remove table writer

    This adds a 'package' mode, which is default if a functions file is not specified. This will attach uprobes to each function symbol in specified packages (default is just main). Function arguments are extracted for now. This commit also removes the table writer (output is now only JSON).

    Signed-off-by: grantseltzer [email protected]

    opened by grantseltzer 3
  •  #14 added PID PPID and COMM info

    #14 added PID PPID and COMM info

    Related to #14 Added PID, PPID and COMM.

    JSON: {"FunctionName":"main.test_uint64_array","Args":[{"Type":"UINT64_ARRAY","Value":"412412456, 1234134"}],"procInfo":{"pid":121841,"ppid":39630,"comm":"tester"}}

    TAB:

    +-----------------------+--------------+------+-------+-----------+--------+-------+
    |     FUNCTION NAME     | ARG POSITION | TYPE | VALUE | PROC NAME |  PID   | PPID  |
    +-----------------------+--------------+------+-------+-----------+--------+-------+
    | main.test_single_int8 | 0            | INT8 | 1     | tester    | 121892 | 39630 |
    +-----------------------+--------------+------+-------+-----------+--------+-------+
    
    opened by rrreeezzz 3
  • Tracing by PID #12

    Tracing by PID #12

    Added the ability to filter by PID with -pid option. It reads the symbols from /proc/PID/exe.

    Added a Filters struture field to functionTraceContext, could be use in the future to add more filters (ie. by user, by comm process name, etc...)

    opened by rrreeezzz 1
  • Fix flaky unit testcase in TestReadFunctionsFile

    Fix flaky unit testcase in TestReadFunctionsFile

    Some of tests occasionally fail because the ordering of the slice is off:

        --- FAIL: TestReadFunctionsFile/empty_line_at_begining_of_file (0.00s)
            functions_file_test.go:122: 
                	Error Trace:	functions_file_test.go:122
                	Error:      	Not equal: 
                	            	expected: []main.functionTraceContext{main.functionTraceContext{binaryName:"", FunctionName:"main.foobar", HasArguments:true, Arguments:[]main.argument{main.argument{CType:"long", goType:1, StartingOffset:8, VariableName:"argument1", PrintfFormat:"%ld", TypeSize:8, ArrayLength:0}, main.argument{CType:"long", goType:5, StartingOffset:16, VariableName:"argument2", PrintfFormat:"%ld", TypeSize:8, ArrayLength:0}}}, main.functionTraceContext{binaryName:"", FunctionName:"main.buzbaz", HasArguments:true, Arguments:[]main.argument{main.argument{CType:"char", goType:13, StartingOffset:8, VariableName:"argument1", PrintfFormat:"%t", TypeSize:1, ArrayLength:0}, main.argument{CType:"int", goType:16, StartingOffset:12, VariableName:"argument2", PrintfFormat:"%c", TypeSize:4, ArrayLength:0}}}}
                	            	actual  : []main.functionTraceContext{main.functionTraceContext{binaryName:"", FunctionName:"main.buzbaz", HasArguments:true, Arguments:[]main.argument{main.argument{CType:"char", goType:13, StartingOffset:8, VariableName:"argument1", PrintfFormat:"%t", TypeSize:1, ArrayLength:0}, main.argument{CType:"int", goType:16, StartingOffset:12, VariableName:"argument2", PrintfFormat:"%c", TypeSize:4, ArrayLength:0}}}, main.functionTraceContext{binaryName:"", FunctionName:"main.foobar", HasArguments:true, Arguments:[]main.argument{main.argument{CType:"long", goType:1, StartingOffset:8, VariableName:"argument1", PrintfFormat:"%ld", TypeSize:8, ArrayLength:0}, main.argument{CType:"long", goType:5, StartingOffset:16, VariableName:"argument2", PrintfFormat:"%ld", TypeSize:8, ArrayLength:0}}}}
                	            	
                	            	Diff:
                	            	--- Expected
                	            	+++ Actual
                	            	@@ -1,2 +1,27 @@
                	            	 ([]main.functionTraceContext) (len=2) {
                	            	+ (main.functionTraceContext) {
                	            	+  binaryName: (string) "",
                	            	+  FunctionName: (string) (len=11) "main.buzbaz",
                	            	+  HasArguments: (bool) true,
                	            	+  Arguments: ([]main.argument) (len=2) {
                	            	+   (main.argument) {
                	            	+    CType: (string) (len=4) "char",
                	            	+    goType: (main.goType) 13,
                	            	+    StartingOffset: (int) 8,
                	            	+    VariableName: (string) (len=9) "argument1",
                	            	+    PrintfFormat: (string) (len=2) "%t",
                	            	+    TypeSize: (int) 1,
                	            	+    ArrayLength: (int) 0
                	            	+   },
                	            	+   (main.argument) {
                	            	+    CType: (string) (len=3) "int",
                	            	+    goType: (main.goType) 16,
                	            	+    StartingOffset: (int) 12,
                	            	+    VariableName: (string) (len=9) "argument2",
                	            	+    PrintfFormat: (string) (len=2) "%c",
                	            	+    TypeSize: (int) 4,
                	            	+    ArrayLength: (int) 0
                	            	+   }
                	            	+  }
                	            	+ },
                	            	  (main.functionTraceContext) {
                	            	@@ -25,27 +50,2 @@
                	            	   }
                	            	- },
                	            	- (main.functionTraceContext) {
                	            	-  binaryName: (string) "",
                	            	-  FunctionName: (string) (len=11) "main.buzbaz",
                	            	-  HasArguments: (bool) true,
                	            	-  Arguments: ([]main.argument) (len=2) {
                	            	-   (main.argument) {
                	            	-    CType: (string) (len=4) "char",
                	            	-    goType: (main.goType) 13,
                	            	-    StartingOffset: (int) 8,
                	            	-    VariableName: (string) (len=9) "argument1",
                	            	-    PrintfFormat: (string) (len=2) "%t",
                	            	-    TypeSize: (int) 1,
                	            	-    ArrayLength: (int) 0
                	            	-   },
                	            	-   (main.argument) {
                	            	-    CType: (string) (len=3) "int",
                	            	-    goType: (main.goType) 16,
                	            	-    StartingOffset: (int) 12,
                	            	-    VariableName: (string) (len=9) "argument2",
                	            	-    PrintfFormat: (string) (len=2) "%c",
                	            	-    TypeSize: (int) 4,
                	            	-    ArrayLength: (int) 0
                	            	-   }
                	            	-  }
                	            	  }
                	Test:       	TestReadFunctionsFile/empty_line_at_begining_of_file
            functions_file_test.go:124: Test failed.
    
    bug 
    opened by grantseltzer 1
  • Create mode of operation where instead of functions file, just trace the function calls without param types

    Create mode of operation where instead of functions file, just trace the function calls without param types

    We can just inspect the symbol table and attach uprobes to every function in a list of a specified packages (default: just main), and have it print out just when the function is called. If #8 is confirmed then this would just be the default mode of operation.

    Functions files might still be useful for people who want fine grained control of what's traced

    feature request 
    opened by grantseltzer 1
  • Create indicator that all uprobes/eBPF programs are loaded.

    Create indicator that all uprobes/eBPF programs are loaded.

    There's some delay before all eBPF programs are loaded. It would be nice to have a visual indicator that all specified functions now have uProbes attached.

    good first issue feature request 
    opened by grantseltzer 1
  • `weaver --types` output should be in logical order

    `weaver --types` output should be in logical order

    This output order doesn't make much sense:

    [*] ./bin/oster -t    
    UINT
    RUNE
    INT
    UINT64
    FLOAT64
    STRING
    UINT16
    INT32
    INT64
    UINT32
    BYTE
    STRUCT
    POINTER
    INT16
    UINT8
    FLOAT32
    BOOL
    INT8
    
    bug good first issue 
    opened by grantseltzer 1
  • Strip null's from end of strings

    Strip null's from end of strings

    Currently when tracing strings a lot of padded null's are appended to the end of strings:

    [*] sudo ./bin/oster --function-to-trace='main.test_function(int, string)' ./bin/test-prog
    {"Type":"INT","Value":"3"}
    {"Type":"STRING","Value":"hello\u0000\u0000\u0000\u0000\u0000\u0000\u0000"}
    

    Let's remove those after treating the user space byte array as little endian.

    bug 
    opened by grantseltzer 1
  • [feature request] Create a helper/tester tool or library for inspectig the full contents of stack

    [feature request] Create a helper/tester tool or library for inspectig the full contents of stack

    When trying to determine how the Go stack is organized when entering into a function I had an eBPF program that would print out the stack one byte at a time for a Go program with various arrangements of argument data types.

    Creating a general solution would be great for testing different versions of Go/system compatibility

    	        int i;
    		char y;
    		for (i = 0; i < 50; i++) {
    			char *x = (char *)decSP+i; 
    			bpf_probe_read(&y, sizeof(y), x);
    			bpf_trace_printk("%d\n", y);
    		}
    
    opened by grantseltzer 1
  • Add the ability to read in the value, or address of strings

    Add the ability to read in the value, or address of strings

    Currently strings aren't a supported data type (just haven't gotten to it yet). Go stores a pointer to the string on the stack, which needs to be read in as a long (8 bytes) and then de-referenced. That may be able to happen in Go instead of in eBPF, but probably not.

    bug feature request 
    opened by grantseltzer 1
  • Minimum types to index when parsing DWARF

    Minimum types to index when parsing DWARF

    dwarf.TagCompileUnit dwarf.TagSubprogram dwarf.TagVariable dwarf.TagTag(0) dwarf.TagFormalParameter dwarf.TagLexDwarfBlock dwarf.TagInlinedSubroutine dwarf.TagConstant dwarf.TagUnspecifiedType dwarf.TagPointerType dwarf.TagBaseType dwarf.TagStructType dwarf.TagMember dwarf.TagTypedef dwarf.TagSubroutineType dwarf.TagArrayType dwarf.TagSubrangeType

    question 
    opened by grantseltzer 0
  • Go 1.17 will support register based arguments instead of the stack (feature flagged)

    Go 1.17 will support register based arguments instead of the stack (feature flagged)

    Will certainly still be retrievable but will need to be feature flagged. I believe the go version string is passed into the binary debug information too so could auto-detect.

    https://github.com/golang/go/issues/40724

    bug 
    opened by grantseltzer 1
  • Sequence numbering/ordering for function calls

    Sequence numbering/ordering for function calls

    Currently there's no order for function calls, just whenever they come out of the kernel. It'd be useful to associate invocations in a map and order them.

    opened by grantseltzer 1
  • Add CLI syntax like dtrace

    Add CLI syntax like dtrace

    In this blog post Brendan Gregg has some examples of using a utility for installing probes on userspace probes: http://www.brendangregg.com/blog/2015-06-28/linux-ftrace-uprobe.html

    Being able to put a one line probe in would be nice, perhaps daemonize it.

    opened by grantseltzer 0
  • More robust integration testing

    More robust integration testing

    The smoke-test breaks pretty easily. We should have go tests that actually execute weaver and do typed assertions for checking output.

    This'll make it really easy to port to new systems too.

    P0 
    opened by grantseltzer 0
Releases(v0.1)
Owner
grantseltzer
www.grant.pizza
grantseltzer
Edb - An eBPF program debugger

EDB (eBPF debugger) edb is a debugger(like gdb and dlv) for eBPF programs. Norma

null 140 Sep 4, 2022
PinGo is a standalone and feature-rich tool for common IP-based reachability checking tasks. Ping or Trace and Observe in real-time the statistics.

pingo As a network champion from designing and implementing to troubleshooting large scale networks - I know that is usually not easy for administrato

Jerome Amon 2 Jul 19, 2022
A tool based on eBPF, prometheus and grafana to monitor network connectivity.

Connectivity Monitor Tracks the connectivity of a kubernetes cluster to its api server and exposes meaningful connectivity metrics. Uses ebpf to obser

Gardener 25 Aug 30, 2022
SailFirewall - Linux firewall powered by eBPF and XDP

SailFirewall Linux firewall powered by eBPF and XDP Requirements Go 1.16+ Linux

Hevienz 0 May 4, 2022
eBPF based TCP observability.

TCPDog is a total solution from exporting TCP statistics from Linux kernel by eBPF very efficiently to store them at your Elasticsearch or InfluxDB da

Mehrdad Arshad Rad 201 Sep 12, 2022
Library to work with eBPF programs from Go

Go eBPF A nice and convenient way to work with eBPF programs / perf events from Go. Requirements Go 1.10+ Linux Kernel 4.15+ Supported eBPF features e

Dropbox 955 Sep 23, 2022
eBPF library for Go based on Linux libbpf

libbpfgo libbpfgo is a Go library for working with Linux's eBPF. It was created for Tracee, our open source Runtime Security and eBPF tracing tools wr

Aqua Security 349 Sep 16, 2022
eBPF Library for Go

eBPF eBPF is a pure Go library that provides utilities for loading, compiling, and debugging eBPF programs. It has minimal external dependencies and i

Cilium 3.4k Sep 24, 2022
A distributed Layer 2 Direct Server Return (L2DSR) load balancer for Linux using XDP/eBPF

VC5 A distributed Layer 2 Direct Server Return (L2DSR) load balancer for Linux using XDP/eBPF This is very much a proof of concept at this stage - mos

David Coles 33 Aug 29, 2022
eBPF-based EDR for Linux

ebpf-edr A proof-of-concept eBPF-based EDR for Linux Seems to be working fine with the 20 basic rules implemented. Logs the alerts to stdout at the mo

null 15 Aug 3, 2022
An ebpf's tool to watch traffic

watch-dog watch-dog利用ebpf的能力,监听指定网卡的流量来达到旁路检测流量的目的,并使用图数据库neo4j保存节点之间的流量关系。 Get go get github.com/TomatoMr/watch-dog Install make build Usage sudo ./w

null 0 Feb 5, 2022
Command-line tool and library for Windows remote command execution in Go

WinRM for Go Note: if you're looking for the winrm command-line tool, this has been splitted from this project and is available at winrm-cli This is a

Brice Figureau 378 Sep 21, 2022
Bell is the simplest event system written in Go (Golang) which is based on the execution of handlers independent of the main channel.

Bell Bell is the simplest event system written in Go (Golang) which is based on the execution of handlers independent of the main channel. Written in

NUT.Tech 27 Sep 8, 2022
Go library providing an abstraction to Ethereum execution nodes

go-execution-client Go library providing an abstraction to Ethereum execution nodes. Its external API follows the official Ethereum consensus APIs spe

Attestant 2 Aug 12, 2022
Moviefetch: a simple program to search and download for movies from websites like 1337x and then stream them

MovieFetch Disclaimer I am NOT responisble for any legal issues or other you enc

Hashm 2 May 12, 2022
Aidos Kuneen (v2 network) daemon program that is controlled through the command line and remotely via RPC calls

adk-daemon: aidosd.v2 aidosd (v2) is a deamon which acts as bitcoind for adk. This version has been built specifically for network mesh version 2+ For

Aidos Kuneen 0 Dec 1, 2021
A Golang program that receives DNSTAP traffic and relays it to multiple other listeners.

socket-proxy socket-proxy is a Golang program that is used to proxy dnstap messages from one socket to multiple other sockets. Overview Name Servers t

Andrew Fried 0 Jan 10, 2022
Program to simultaneously listen and respond on multiple TCP/UDP ports

listen Program to simultaneously listen on multiple TCP/UDP ports and reply back to anything sent along with IP addresses and lengths of data received

Purplecarrot 1 Feb 20, 2022
rconn is a multiplatform program for creating generic reverse connections. Lets you consume services that are behind firewall or NAT without opening ports or port-forwarding.

rconn (r[everse] conn[ection]) is a multiplatform program for creating reverse connections. It lets you consume services that are behind NAT and/or fi

Hikmat Jafarli 233 Sep 4, 2022