A golang CLI tool to download malware from a variety of sources.

Related tags

Command Line mlget
Overview

mlget

          _____                    _____            _____                    _____                _____          
         /\    \                  /\    \          /\    \                  /\    \              /\    \         
        /::\____\                /::\____\        /::\    \                /::\    \            /::\    \        
       /::::|   |               /:::/    /       /::::\    \              /::::\    \           \:::\    \       
      /:::::|   |              /:::/    /       /::::::\    \            /::::::\    \           \:::\    \      
     /::::::|   |             /:::/    /       /:::/\:::\    \          /:::/\:::\    \           \:::\    \     
    /:::/|::|   |            /:::/    /       /:::/  \:::\    \        /:::/__\:::\    \           \:::\    \    
   /:::/ |::|   |           /:::/    /       /:::/    \:::\    \      /::::\   \:::\    \          /::::\    \   
  /:::/  |::|___|______    /:::/    /       /:::/    / \:::\    \    /::::::\   \:::\    \        /::::::\    \  
 /:::/   |::::::::\    \  /:::/    /       /:::/    /   \:::\ ___\  /:::/\:::\   \:::\    \      /:::/\:::\    \ 
/:::/    |:::::::::\____\/:::/____/       /:::/____/  ___\:::|    |/:::/__\:::\   \:::\____\    /:::/  \:::\____\
\::/    / ~~~~~/:::/    /\:::\    \       \:::\    \ /\  /:::|____|\:::\   \:::\   \::/    /   /:::/    \::/    /
 \/____/      /:::/    /  \:::\    \       \:::\    /::\ \::/    /  \:::\   \:::\   \/____/   /:::/    / \/____/ 
             /:::/    /    \:::\    \       \:::\   \:::\ \/____/    \:::\   \:::\    \      /:::/    /          
            /:::/    /      \:::\    \       \:::\   \:::\____\       \:::\   \:::\____\    /:::/    /           
           /:::/    /        \:::\    \       \:::\  /:::/    /        \:::\   \::/    /    \::/    /            
          /:::/    /          \:::\    \       \:::\/:::/    /          \:::\   \/____/      \/____/             
         /:::/    /            \:::\    \       \::::::/    /            \:::\    \                              
        /:::/    /              \:::\____\       \::::/    /              \:::\____\                             
        \::/    /                \::/    /        \::/____/                \::/    /                             
         \/____/                  \/____/                                   \/____/                              

Build

What is it

Use mlget to query multiple sources for a given malware hash and download it. The thought is to save time querying each source individually.

Currently queries:

  • tg (triage)
  • mb (malwarebazaar)
  • ms (malshare)
  • ha (HybirdAnlysis)
  • vt (VirusTotal)
  • cp (Cape Sandbox)
  • mw (Malware Database)
  • ps (PolySwarm)
  • iq (InquestLabs)

Only Malware Bazaar does not require a key, the rest require a key. The config file needs to be placed in the user's home directory (essentially where os.UserHomeDir() resolves to).

Build

Assuming golang is installed and this repo is checked out; simply run:

go get -u
go mod tidy
go build

from the cloned directory to compile the application and ./mlget to run it.

Setup Config

Run mlget on the commandline without any flags to have it generate the yaml config template.

> ./mlget
Config does not exist.  Create config? [Y|n]

Hit enter and it will create the config, .mlget.yml, in the user's home dir.

Created /home/dir/.mlget.yml.  Make sure to fill out the API keys for the services you want to use.
No hashes found
mlget - A command line tool to download malware from a variety of sources

Usage: ./mlget [OPTIONS] hash_arguments...
      --comment strings     Add comment to the sample when uploading to your own instance of MWDB.
      --config              Parse and print the config file
      --from string         The service to download the malware from.
                              Must be one of:
                              - tg (Triage)
                              - mb (Malware Bazaar)
                              - ms (Malshare)
                              - ha (HybirdAnlysis)
                              - vt (VirusTotal)
                              - cp (Cape Sandbox)
                              - mw (Malware Database)
                              - ps (PolySwarm)
                              - iq (InquestLabs)
                            If omitted, all services will be tried.
      --help                Print the help message
      --noextraction        Do not extract malware from archive file.
                            Currently this only effects MalwareBazaar and HybridAnalysis
      --output              Write to a file the hashes not found (for later use with the --read flag)
      --read string         Read in a file of hashes (one per line)
      --readupdate string   Read hashes from file to download.  Replace entries in the file with just the hashes that were not found (for next time).
      --tag strings         Tag the sample when uploading to your own instance of MWDB.
      --upload              Upload downloaded files to the MWDB instance specified in the mlget.yml file.
      --uploaddelete        Upload downloaded files to the MWDB instance specified in the mlget.yml file.
                            Delete the files after successful upload

Example Usage: mlget <sha256>
Example Usage: mlget --from mb <sha256>
Example Usage: mlget --tag tag_one --tag tag_two --uploaddelete <sha256> <sha1> <md5>

Fill out the API keys where needed. If the service does not have an API key then it will be skipped.

malshare:
  api: ""
  host: https://malshare.com
malwarebazar:
  host: https://mb-api.abuse.ch/api/v1
mwdb:
  api: ""
  host: https://mwdb.cert.pl/api
virustotal:
  api: ""
  host: https://www.virustotal.com/api/v3
triage:
  api: ""
  host: https://api.tria.ge/v0
hybridanalysis:
  api: ""
  host: https://www.hybrid-analysis.com/api/v2
polyswarm:
  api: ""
  host: https://api.polyswarm.network/v2
capesandbox:
  api: ""
  host: https://www.capesandbox.com/apiv2
inquestlabs:
  api: ""
  host: https://labs.inquest.net/api
uploadtomwdb:
  api: ""
  host: ""

Fill out the last last one uploadtomwdb if you want to take advantage of a private MWDB instance to upload the files to. All of the others are for download only.

Suggested Setup

When uploadtomwdb is Configured

Add an alias that maps mlget to mlget --uploaddelete --readupdate /path/to/malware_download_tracker.txt

When uploadtomwdb is Not Configured

Add an alias that maps mlget to mlget --readupdate /path/to/malware_download_tracker.txt

Example Usage

Download from All sources

mlget e6ba5de3a9b0287291def0317789b871fa1984a11021d55d3a0371c6d65a872b 1c7897a902b35570a9620c64a2926cd5d594d4ff5a033e28a400981d14516600

Multiple hash can be passed.

Download from Specific Source

mlget --from tg e6ba5de3a9b0287291def0317789b871fa1984a11021d55d3a0371c6d65a872b 1c7897a902b35570a9620c64a2926cd5d594d4ff5a033e28a400981d14516600

Hashes from a file

Hashes can be read from both the command line and a file (one hash per line) at the same time

mlget --read hashes.txt e6ba5de3a9b0287291def0317789b871fa1984a11021d55d3a0371c6d65a872b 1c7897a902b35570a9620c64a2926cd5d594d4ff5a033e28a400981d14516600

Recording the hashes not found for next time

This will output the unfound hashes to a new file:

mlget --read hashes.txt --output e6ba5de3a9b0287291def0317789b871fa1984a11021d55d3a0371c6d65a872b 1c7897a902b35570a9620c64a2926cd5d594d4ff5a033e28a400981d14516600

This will update current file so it only contains the hashes not found

mlget --readupdate download.txt e6ba5de3a9b0287291def0317789b871fa1984a11021d55d3a0371c6d65a872b 1c7897a902b35570a9620c64a2926cd5d594d4ff5a033e28a400981d14516600

Other Commands

mlget --help

mlget - A command line tool to download malware from a variety of sources

Usage: ./mlget [OPTIONS] hash_arguments...
      --comment strings     Add comment to the sample when uploading to your own instance of MWDB.
      --config              Parse and print the config file
      --from string         The service to download the malware from.
                              Must be one of:
                              - tg (Triage)
                              - mb (Malware Bazaar)
                              - ms (Malshare)
                              - ha (HybirdAnlysis)
                              - vt (VirusTotal)
                              - cp (Cape Sandbox)
                              - mw (Malware Database)
                              - ps (PolySwarm)
                              - iq (InquestLabs)
                            If omitted, all services will be tried.
      --help                Print the help message
      --noextraction        Do not extract malware from archive file.
                            Currently this only effects MalwareBazaar and HybridAnalysis
      --output              Write to a file the hashes not found (for later use with the --read flag)
      --read string         Read in a file of hashes (one per line)
      --readupdate string   Read hashes from file to download.  Replace entries in the file with just the hashes that were not found (for next time).
      --tag strings         Tag the sample when uploading to your own instance of MWDB.
      --upload              Upload downloaded files to the MWDB instance specified in the mlget.yml file.
      --uploaddelete        Upload downloaded files to the MWDB instance specified in the mlget.yml file.
                            Delete the files after successful upload

Example Usage: mlget <sha256>
Example Usage: mlget --from mb <sha256>
Example Usage: mlget --tag tag_one --tag tag_two --uploaddelete <sha256> <sha1> <md5>

License

MIT License

Copyright (c) 2021 @xorhex

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

You might also like...
Symfony-cli - The Symfony CLI tool For Golang

Symfony CLI Install To install Symfony CLI, please download the appropriate vers

tmux-wormhole - download files and directories with tmux!
tmux-wormhole - download files and directories with tmux!

tmux-wormhole Use tmux and magic wormhole to get things from your remote computer to your tmux. If tmux has DISPLAY set, open the file locally! Demo U

๐Ÿ“ท Command-line utility to download all photos from Instagram
๐Ÿ“ท Command-line utility to download all photos from Instagram

Instagram Downloader This is a simple command-line tool, written in Go, to download all images from an Instagram account. Getting Started Install inst

A small utility command line application that can recursively download Notion pages

notionbackup A small utility command line application that can recursively download Notion pages. I needed something scriptable that could periodicall

A terminal-based download manager written in Go!

goload-manager About A terminal-based download manager written in Go! This project uses grab for downloading and tcell for the terminal interface. Hea

Download and install binaries from GitHub Releases, interactively.

bget Download and install binary files from GitHub Releases. Preview Install For Mac/Linux users, you can use Homebrew to install it: brew install ego

Mass download all github repositories(public & private) of an organization, ideally in a few seconds.

Git Mass Mass download all github repositories(public & private) of an organization, ideally in a few seconds. Writing this as a simple bash script wo

Command-line program to download videos from YouTube.com and other video sites

youtube-dl - download videos from youtube.com or other video platforms INSTALLATION DESCRIPTION OPTIONS CONFIGURATION OUTPUT TEMPLATE FORMAT SELECTION

You-Get is a tiny command-line utility to download media contents (videos, audios, images) from the Web,
You-Get is a tiny command-line utility to download media contents (videos, audios, images) from the Web,

You-Get NOTICE: Read this if you are looking for the conventional "Issues" tab. You-Get is a tiny command-line utility to download media contents (vid

Comments
  • Malpedia 404 is not considered a failure

    Malpedia 404 is not considered a failure

    Summary

    Title. Malpedia's API can return a 404 message in the form of {"detail":"Not found."}. MLGet currently considers this a success.

    opened by Still34 2
  • Read from URL

    Read from URL

    Added a feature to allow for urls to be submitted to the --read option. If the url points to a list of hashes, then this allow for things like:

    mlget --read https://raw.githubusercontent.com/avast/ioc/master/OperationDragonCastling/samples.sha256
    
    mlget --read https://raw.githubusercontent.com/eset/malware-ioc/master/mustang_panda/samples.sha256
    
    opened by xorhex 0
  • v2.5 Release

    v2.5 Release

    Added two more sources: FileScanIO and VxShare Fixed a bug with the Inquest downloader. Fixed a bug with the UnpacMe downloader. Fixed a bug with the Malpedia downloader. Added some sanity checks when parsing an input file.

    opened by xorhex 0
  • Look into why the Post line is returned when the sample is not found on MalwareBazaar.

    Look into why the Post line is returned when the sample is not found on MalwareBazaar.

    Look up bbe855f9259345af18de5f2cfd759eb78782b664bb22c43f19177dab51d782da (sha256)

    Looking on MalwareBazaar [*] MalwareBazaar: https://mb-api.abuse.ch/api/v1 Post "https://mb-api.abuse.ch/api/v1": EOF [!] Not Found

    opened by xorhex 0
Releases(v2.5.2_take_4)
Owner
null
Got: Simple golang package and CLI tool to download large files faster ๐Ÿƒ than cURL and Wget!

Got. Simple and fast concurrent downloader. Installation โ˜ CLI Usage โ˜ Module Usage โ˜ License Comparison Comparison in cloud server: [[email protected]

Mohamed El Bahja 616 Nov 25, 2022
CLI tool to upload object to s3-compatible storage backend and set download policy for it.

typora-s3 CLI tool to upload object to s3-compatible storage backend and set download policy for it. Build $ git clone https://github.com/fengxsong/ty

fengxsong 0 Dec 29, 2021
zipspy - a CLI tool to extract files from zip archives in S3 without needing to download the entire archive

Zipspy allows you interact with ZIP archives stored in remote locations without requiring a local copy. For example, you can list the filenames in an S3 ZIP archive, download a subset of files, search and retrieve files with regular expressions, and more!

Alec Rabold 0 Feb 19, 2022
Gowget - A simple wget tool built using golang to download files from the internet whether it is zip

gowget wget built using golang About: A simple wget tool built using golang to d

Krisna Pranav 3 May 21, 2022
๐Ÿ“ฅ Command-line tool to download videos from hanime.tv

hanime Command-line tool to download videos from hanime.tv Requirements Installation Install via go get Install from source Install from release Usage

็งใฏใƒฌใ‚ชใƒณใงใ™ 32 Nov 24, 2022
This is the tool to download files from qiniu cruster manually.

This is the tool to download files from qiniu cruster manually. toCheck = []string{ sealPath, filepath.Join(cachePath, "p_aux"), filepath.Join(cachePa

lyswifter 1 Nov 25, 2021
Nebula Diagnosis CLI Tool is an information diagnosis cli tool for the nebula service and the node to which the service belongs.

Nebula Diagnosis CLI Tool is an information diagnosis cli tool for the nebula service and the node to which the service belongs.

Katz 1 Jan 12, 2022
Downloader written in golang to download the public data files from RUC Paraguay.

rucpy-downloader Downloader written in golang to download the public data files(RUC Paraguay) from set.gov.py. The downloader will download the public

bitebait 1 Dec 6, 2021
Little golang app that allows you to download a youtube video as mp3, and optionally embed ID3 tags -Cover Art, Artist ...-

yt2mp3 Little golang app that allows you to download a youtube video as mp3, and optionally embed ID3 tags -Cover Art, Artist ...- Instructions At the

null 0 Dec 25, 2021