I am scanning files against a large collection of compiled yara rules. I am using yara-4.0.0

I am encountering this error when I try to run the cross-compiled version of the same go code on windows OS.

When I scan the same files with the same yara ruleset on linux, everything goes smoothly. I already searched a bit for this issue and I was hoping getting rid of the hash import in yara would solve the problem. Unfortunately it did not.

Any idea what might be causing this?

Is this compatible with YARA 4.1.0? When trying to build this with that version in Debian unstable, I get test failures due to segfaults on amd64:

github.com/hillu/go-yara
   dh_auto_test -O--buildsystem=golang
	cd obj-x86_64-linux-gnu && go test -vet=off -v -p 6 github.com/hillu/go-yara
=== RUN   TestBasic
    cbpool_test.go:25: Get: Got expected panic: Attempt to get nonexistent value from pool
    cbpool_test.go:43: Delete: Got expected panic: Attempt to delete nonexistent value from pool
    cbpool_test.go:57: full pool: Got expected panic: cbPool storage exhausted
--- PASS: TestBasic (0.00s)
=== RUN   TestCompiler
    compiler_test.go:21: expected error: syntax error
--- PASS: TestCompiler (0.00s)
=== RUN   TestPanic
    compiler_test.go:31: Everything ok, MustCompile panicked: syntax error
--- PASS: TestPanic (0.00s)
=== RUN   TestWarnings
    compiler_test.go:43: Recorded Errors=[]yara.CompilerMessage{yara.CompilerMessage{Filename:"", Line:1, Text:"syntax error"}}, Warnings=[]yara.CompilerMessage(nil)
--- PASS: TestWarnings (0.00s)
=== RUN   TestCompilerIncludeCallback
    compiler_test.go:52: Processing include "existing" (from ns="default", file="")
    compiler_test.go:52: Processing include "non-existing" (from ns="default", file="")
    compiler_test.go:73: Compiler returned error on attempt to include non-existing rule: callback failed to provide include resource: non-existing
--- PASS: TestCompilerIncludeCallback (0.00s)
=== RUN   TestIterator
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x0]

runtime stack:
runtime.throw(0x5985c8, 0x2a)
	/usr/lib/go-1.15/src/runtime/panic.go:1116 +0x72
runtime.sigpanic()
	/usr/lib/go-1.15/src/runtime/signal_unix.go:726 +0x4ac

goroutine 11 [syscall]:
runtime.cgocall(0x544e60, 0xc000058dd0, 0x1)
	/usr/lib/go-1.15/src/runtime/cgocall.go:133 +0x5b fp=0xc000058da0 sp=0xc000058d68 pc=0x407ffb
github.com/hillu/go-yara._Cfunc_yr_rules_scan_mem_blocks(0x79c630, 0xc0001003f0, 0x8, 0x5443b0, 0x76b838, 0xc000000000, 0xc000000000)
	_cgo_gotypes.go:1567 +0x4d fp=0xc000058dd0 sp=0xc000058da0 pc=0x5353ad
github.com/hillu/go-yara.(*Rules).ScanMemBlocks.func1(0xc000010098, 0xc0001003f0, 0x0, 0x5c2080, 0xc00000e200, 0x76b838, 0x0, 0x7f5a029ce108)
	/build/golang-github-hillu-go-yara-4.0.5/obj-x86_64-linux-gnu/src/github.com/hillu/go-yara/rules.go:182 +0x145 fp=0xc000058e30 sp=0xc000058dd0 pc=0x540625
github.com/hillu/go-yara.(*Rules).ScanMemBlocks(0xc000010098, 0x5c36c0, 0xc00000e220, 0x0, 0x0, 0x5c2080, 0xc00000e200, 0x0, 0x0)
	/build/golang-github-hillu-go-yara-4.0.5/obj-x86_64-linux-gnu/src/github.com/hillu/go-yara/rules.go:182 +0x285 fp=0xc000058ef0 sp=0xc000058e30 pc=0x53a545
github.com/hillu/go-yara.TestIterator(0xc000130480)
	/build/golang-github-hillu-go-yara-4.0.5/obj-x86_64-linux-gnu/src/github.com/hillu/go-yara/mem_blocks_test.go:54 +0xda fp=0xc000058f80 sp=0xc000058ef0 pc=0x52a4da
testing.tRunner(0xc000130480, 0x59c310)
	/usr/lib/go-1.15/src/testing/testing.go:1123 +0xef fp=0xc000058fd0 sp=0xc000058f80 pc=0x4e206f
runtime.goexit()
	/usr/lib/go-1.15/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc000058fd8 sp=0xc000058fd0 pc=0x471541
created by testing.(*T).Run
	/usr/lib/go-1.15/src/testing/testing.go:1168 +0x2b3

goroutine 1 [chan receive]:
testing.(*T).Run(0xc000130480, 0x58fcf9, 0xc, 0x59c310, 0x489901)
	/usr/lib/go-1.15/src/testing/testing.go:1169 +0x2da
testing.runTests.func1(0xc000001980)
	/usr/lib/go-1.15/src/testing/testing.go:1439 +0x78
testing.tRunner(0xc000001980, 0xc000057cf0)
	/usr/lib/go-1.15/src/testing/testing.go:1123 +0xef
testing.runTests(0xc00000e0a0, 0x685300, 0x27, 0x27, 0xc01331720406f728, 0x8bb31abbc6, 0x69a980, 0xc000057db8)
	/usr/lib/go-1.15/src/testing/testing.go:1437 +0x2fe
testing.(*M).Run(0xc000122000, 0x0)
	/usr/lib/go-1.15/src/testing/testing.go:1345 +0x1eb
github.com/hillu/go-yara.TestMain(0xc000122000)
	/build/golang-github-hillu-go-yara-4.0.5/obj-x86_64-linux-gnu/src/github.com/hillu/go-yara/main_test.go:35 +0x1a5
main.main()
	_testmain.go:125 +0x165
FAIL	github.com/hillu/go-yara	0.024s
FAIL

Hello, Thank you for the great work. I was able compile go-yara statically under Linux and Windows (using msys2) and it works.

I am new to yara and my question is about scanning process(es) using yara executable or go-yara. Both Scanner.ScanProc or yr_scanner_scan_proc show same behavior on my laptop, Ubuntu 18.04 amd64. The scanned (target) process's RSS memory increases almost up to VSS memory. This becomes impossible when I want to scan all the available processes due to memory, I ran out of memory. Besides, I use the simplest yara rule just to search for a string existence.

What I see from yara Windows code, it does not scan uncommited memory pages of a process but under Linux story is different.

I really need a guidance to scan all processes without crashing the Linux OSes. One can try to scan chrome/firefox/Xorg processes and watch with top/htop to see how their memory usage changes.

Hi,

I'm trying to use your library on Windows. I'm not familiar with C/C++ and linker/compilation problems but i've searched a lot before posting here. Maybe you could help me. First, i've installed mingw64 and also the latest version of visual studio. I've cloned the official yara repository and successfuly compiled libyara with visual studio. I saw that i've got the following folders:

• C:\YARA\windows\libyara (with a lot of *.obj files)
• C:\YARA\libyara (with a lot of *.c files)
• C:\YARA\libyara\include (with *.h files)
• C:\YARA\windows\vs2017\libyara\Release (with a 22MB libyara64.lib)

I didn't understood what i need to put in CGO_LDFLAGS and CGO_CFLAGS env vars or if i need additional compilations to successfully go get github.com/hillu/go-yara.

Actually it just return the following error:

# runtime/cgo
C:/mingw64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find -lyara

Any help would be appreciated :)

I have been struggling to get this compiled for windows using MinGW.

It would be good to have more instructions on how to proper compile the Yara project and generate libyara for windows 32 and 64bit. Even thought Yara has the Visual Studio solutions and I can easily generate libyara32.lib and libyara64.lib.... Not sure if these are compatible with MinGW and CGO

And how to have this package use these libs with CGO.

When scanning a byte slice with ScanMem go panics with:

panic: runtime error: cgo argument has Go pointer to Go pointer

goroutine 7 [running]:
panic(0x5fbd00, 0xc82000e8b0)
    /usr/lib/go/src/runtime/panic.go:481 +0x3e6
github.com/hillu/go-yara._cgoCheckPointer0(0x5a9be0, 0xc820016fe4, 0x0, 0x0, 0x0, 0x54417e)
    ??:0 +0x4d
github.com/hillu/go-yara.addString(0xc82000e888, 0x11b3dd4, 0x0, 0xc820016fe4, 0xc800000030)
    /home/tom/src/github.com/hillu/go-yara/rules.go:117 +0xce
github.com/hillu/go-yara._cgoexpwrap_f3bc5924bb1c_addString(0xc82000e888, 0x11b3dd4, 0x0, 0xc820016fe4, 0xc800000030)
    ??:0 +0x47
github.com/hillu/go-yara._Cfunc_yr_rules_scan_mem(0x12360e0, 0xc820016fe4, 0x3b, 0x1, 0x58a900, 0xc82000e888, 0x1, 0x0)
    ??:0 +0x44
github.com/hillu/go-yara.(*Rules).ScanMem(0xc82002c068, 0xc820016fe4, 0x3b, 0x40, 0x1, 0x3b9aca00, 0x0, 0x0, 0x0, 0x0, ...)
    /home/tom/src/github.com/hillu/go-yara/rules.go:147 +0x23d

Using a bytes.Buffer in rules_test.go/TestReader seems to trigger the above error

diff --git a/rules_test.go b/rules_test.go
index ca59966..83b912d 100644
--- a/rules_test.go
+++ b/rules_test.go
@@ -109,7 +109,10 @@ func TestReader(t *testing.T) {
        if err != nil {
                t.Fatalf("ReadRules: %+v", err)
        }
-       m, err := r.ScanMem([]byte(" abc "), 0, 0)
+
+       buf := &bytes.Buffer{}
+       buf.Write([]byte(" abc "))
+       m, err := r.ScanMem(buf.Bytes(), 0, 0)
        if err != nil {
                t.Errorf("ScanMem: %s", err)
        }

Problem overview

I have been attempting to deploy a web service in Go within a docker container (ubuntu 19.04) that accept a file and returns the names for each rule that matches. When testing this on my local machine (macOS) with yara 3.11 and yarac 3.11 I am able to scan the wannaCry sample [ed01ebfbc9eb5bbea545af4d01bf5f1071661840480439c6e5babe8e080e41aa](https://www.virustotal.com/gui/file/ed01ebfbc9eb5bbea545af4d01bf5f1071661840480439c6e5babe8e080e41aa/detection). Although when I go deploy this service within the docker container I get the following errors.

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x2f pc=0xcd3e97]

runtime stack:
runtime.throw(0xf1dee4, 0x2a)
	/usr/local/go/src/runtime/panic.go:774 +0x72
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:378 +0x47c

goroutine 61 [syscall, locked to thread]:
runtime.cgocall(0xcd3e60, 0xc000344d20, 0x1)
	/usr/local/go/src/runtime/cgocall.go:128 +0x5b fp=0xc000344cf0 sp=0xc000344cb8 pc=0x40b96b
github.com/hillu/go-yara._Cfunc_string_matches(0x7fb6c544ec05, 0x0, 0xc0004c9bfc)
	_cgo_gotypes.go:1121 +0x45 fp=0xc000344d20 sp=0xc000344cf0 pc=0xb07415
github.com/hillu/go-yara.(*String).Matches(0xc000344e48, 0x0, 0x0, 0x0)
	/go/src/github.com/hillu/go-yara/rule.go:225 +0x6b fp=0xc000344db8 sp=0xc000344d20 pc=0xb0c2ab
github.com/hillu/go-yara.(*Rule).getMatchStrings(0xc0000a8ce0, 0x0, 0x0, 0x0)
	/go/src/github.com/hillu/go-yara/rule.go:244 +0x2c2 fp=0xc000344ec0 sp=0xc000344db8 pc=0xb0c722
github.com/hillu/go-yara.(*MatchRules).RuleMatching(0xc000324220, 0xc0000a8ce0, 0xc000324220, 0x7fb6ec1219f0, 0xc000324220)
	/go/src/github.com/hillu/go-yara/rules_callback.go:151 +0x224 fp=0xc000345020 sp=0xc000344ec0 pc=0xb0e3e4
github.com/hillu/go-yara.scanCallbackFunc(0x1, 0x7fb6c3d6916d, 0x21bb7e0, 0xf395a8)
	/go/src/github.com/hillu/go-yara/rules_callback.go:90 +0x220 fp=0xc000345100 sp=0xc000345020 pc=0xb0ddf0
github.com/hillu/go-yara._cgoexpwrap_08a63f7e5a95_scanCallbackFunc(0x7fb600000001, 0x7fb6c3d6916d, 0x21bb7e0, 0x0)
	_cgo_gotypes.go:1621 +0x3d fp=0xc000345130 sp=0xc000345100 pc=0xb08aad
runtime.call32(0x0, 0x7fb6cdffa3f0, 0x7fb6cdffa480, 0x20)
	/usr/local/go/src/runtime/asm_amd64.s:539 +0x3b fp=0xc000345160 sp=0xc000345130 pc=0x461f3b
runtime.cgocallbackg1(0x0)
	/usr/local/go/src/runtime/cgocall.go:314 +0x1b7 fp=0xc000345248 sp=0xc000345160 pc=0x40bd17
runtime.cgocallbackg(0x0)
	/usr/local/go/src/runtime/cgocall.go:191 +0xc1 fp=0xc0003452b0 sp=0xc000345248 pc=0x40bac1
runtime.cgocallback_gofunc(0x40b98f, 0xcd40b0, 0xc000345340, 0xc000345330)
	/usr/local/go/src/runtime/asm_amd64.s:793 +0x9b fp=0xc0003452d0 sp=0xc0003452b0 pc=0x46350b
runtime.asmcgocall(0xcd40b0, 0xc000345340)
	/usr/local/go/src/runtime/asm_amd64.s:640 +0x42 fp=0xc0003452d8 sp=0xc0003452d0 pc=0x4633a2
runtime.cgocall(0xcd40b0, 0xc000345340, 0x10279e1)
	/usr/local/go/src/runtime/cgocall.go:131 +0x7f fp=0xc000345310 sp=0xc0003452d8 pc=0x40b98f
github.com/hillu/go-yara._Cfunc_yr_rules_scan_mem(0x7fb6d4000eb0, 0xc001c5d19a, 0x35a000, 0x1, 0xcd3790, 0x21bb7e0, 0x0, 0x0)
	_cgo_gotypes.go:1524 +0x4d fp=0xc000345340 sp=0xc000345310 pc=0xb0855d
github.com/hillu/go-yara.(*Rules).ScanMemWithCallback.func1(0xc0000a8960, 0xc001c5d19a, 0xc001c5d19a, 0x35a000, 0xd4a656, 0x1, 0x21bb7e0, 0x0, 0x7523cf)
	/go/src/github.com/hillu/go-yara/rules.go:91 +0xfc fp=0xc0003453a0 sp=0xc000345340 pc=0xb1099c
github.com/hillu/go-yara.(*Rules).ScanMemWithCallback(0xc0000a8960, 0xc001c5d19a, 0x35a000, 0xd4a656, 0x1, 0x0, 0xdc17a0, 0xc000324220, 0x0, 0x0)
	/go/src/github.com/hillu/go-yara/rules.go:91 +0x1b7 fp=0xc000345488 sp=0xc0003453a0 pc=0xb0cb27
github.com/hillu/go-yara.(*Rules).ScanMem(...)
	/go/src/github.com/hillu/go-yara/rules.go:68
github.secureserver.net/threat/util/yara.(*Rules).ScanBuffer(0xc0001559e0, 0x1054b20, 0xc000022080, 0xc001c5d19a, 0x35a000, 0xd4a656, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.secureserver.net/threat/util/yara/yara.go:84 +0x184 fp=0xc000345660 sp=0xc000345488 pc=0xb142d4
project/classification.(*Plugin).classifySamples(0xc0002b6c80, 0x1054b20, 0xc000022080, 0xc0001f5000, 0xc00014c660, 0x0, 0x17, 0xc000575960)
	/go/src/project/classification/classification.go:176 +0x1dd fp=0xc000345800 sp=0xc000345660 pc=0xb16b0d

This stack trace basically shows that we're taking in a sample, attempting to classify it, using ScanBuffer as the mechanism for scanning and returning the matching rules. The top function within the stack trace points to github.com/hillu/go-yara._Cfunc_string_matches so I've been trying to figure out how this function could cause issues within a docker a container but nowhere else.

As for the ruleset, I have been using a relatively large ruleset for testing and when using rules that just check for the PE header or the Cannot Run in DOS mode string, there are no errors so this could also be the source of error here.

With regards to how I am installing Yara in the docker container, following is a snippet for how I am installing yara on the host that contains the go built application as well

# steps required to install Yara and remove the source files
RUN git clone https://github.com/VirusTotal/yara.git /tmp/yara
WORKDIR ${YARA_SRC_PATH}
RUN \
    apt update && \
    apt install automake libtool make pkg-config libyara-dev libc-dev -y && \
    ./bootstrap.sh && \
    ./configure && \
    make && \
    make install && \
    rm -rf ${YARA_SRC_PATH}

Things I have tried so far

• compiling the rules with yarac and compiling them within the web service
  • had no effect, still caused issues with the wannacry sample
• Check if its an issue within the docker container or something that would be an issue on my host machine as well
  • Its not an issue on the host machine, so that means something is missing within the docker container or how yara interacts with the ubuntu base image
• Checks yara versions
  • Yara and Yarac in the deploy container are both running version 3.11
• Adding libc-dev and file-dev as those pacakges are added in the official yara docker image
  • Does not seem to fix the issue

Go-Yara is used in my program. Now I want to reload a batch of rules according to the reload rule command I received without restarting the program. I use the command 'top' to monitor the proportion of memory, but after reloading, the memory has been increasing without being released.

As per the title, would it be possible to provide a scanning interface that accepts an io.Reader. The implementation could be based on a streaming interface by implementing the YR_MEMORY_BLOCK_ITERATOR* parameter for yr_scanner_scan_mem_block()?

With an io.Reader it'd be much easier to integrate go-yara in a clean and Go-like manner. Unless I missed something of course - happy to hear about alternatives (currently I believe there's either ScanMem(), which takes a singular []byte, ScanFile, which takes a filename, and ScanFileDescriptor, which takes a fd) :-)

Trying to cross compile from OS X to Linux using Yara 3.8.1. Following appears on build:

[REDACTED]/go/src/github.com/hillu/go-yara/util.go:9:20: undefined: makecbPool

Compiles fine for OS X.

Still seeing "cbPool storage exhausted" in 4.1.0 similar to #88. Haven't dug into cbPool but is there any particular limitation or resource constraint that can be reconfigured as a workaround?

% go list -m all | grep yara github.com/hillu/go-yara/v4 v4.1.0

% yara --version 4.1.3

cbPool storage exhausted

goroutine 963 [running]: github.com/hillu/go-yara/v4.(*cbPool).Put(0xc0001b60a0, {0x4183a40, 0xc000013350}) /Users/user/go/pkg/mod/github.com/hillu/go-yara/[email protected]v4.1.0/cbpool.go:56 +0x158 github.com/hillu/go-yara/v4.(*Scanner).putCallbackData(0xc000013200) /Users/user/go/pkg/mod/github.com/hillu/go-yara/[email protected]/scanner.go:138 +0x12d github.com/hillu/go-yara/v4.(*Scanner).ScanMem(0xc000013200, {0xc001358500, 0x4c9, 0x4ca}) /Users/user/go/pkg/mod/github.com/hillu/go-yara/[email protected]/scanner.go:153 +0x5d main.Scan({0xc00082c500, 0x9c}, 0x0, 0xc000832018)

This is probably "just" a matter of documentation, though here's the idea:

For an analysis tool running in a web browser based on Go and WASM, libyara would need to support that compile target.

There is already libyara-wasm and wasmer-go. Now the question is what exactly needs to be done such that I can write a Go function to be called from JavaScript through WASM.

So far, I have been using webpack-golang-wasm-async-loader in Fiedka (branch to replace main somewhat soon).

My overall goal is to get https://github.com/Mimoja/MFT-AnalyserV2 in as a package, so I'm dealing with multiple modules and transitive dependencies, for which I do have forks and branches, especially https://github.com/orangecms/MFT-AnalyserV2/tree/fiedka. Anyhow, every little hint would be highly appreciated. :bow:

Hi!

Can't seem to load files that has include statements like this:

include "subdir/SomeGreatYaraRules.yar"

I load these files with err := compiler.AddFile(f, "ns")

I am trying to cross compile go-yara for x86 and x64 and I am getting some errors. The tool set for cross compiling is xgo.

I can cross compile correctly for darwin or windows, but I am unable to do it for linux.

x64

/usr/local/lib/libyara.a(math.o): In function `string_entropy':
math.c:(.text+0xecc): undefined reference to `log2'
/usr/local/lib/libyara.a(math.o): In function `data_entropy':
math.c:(.text+0x1150): undefined reference to `log2'
collect2: error: ld returned 1 exit status

x86

/usr/bin/ld: skipping incompatible /usr/local/lib/libyara.a when searching for -lyara
/usr/bin/ld: skipping incompatible //usr/local/lib/libyara.a when searching for -lyara
/usr/bin/ld: cannot find -lyara
collect2: error: ld returned 1 exit status

I am not sure that the errors came either from go-yara or yara itself.

If you need more datils, please ask them.

It turns out that the repo doesn't contain much in the way of documentation, so I think a prominent link to the docs would be awesome. Adding some examples wouldn't be bad either (I'll add a PR for that later if I get some time).