This is a small tool designed to scrape one or more URLs given as command arguments.

Overview

HTTP-FETCH

This is a small tool designed to scrape one or more URLs given as command arguments.

Usage

http-fetch [--metadata] ...URLs

The output files will be found under a folder called "output" under the root of this project's directory structure.

Example

http-fetch --metadata https://www.google.com

Docker Usage

docker build -t http-fetch .
docker run -v /Users/daniel/http-fetch-output:/app/output http-fetch --metadata https://www.google.com

In the above run case, you may replace the /Users/daniel/http-fetch-output portion with any path on your local filesystem.

TODO

  • Add tests based around mocked web server. Test would ensure that file contents match between on-disk sample and scraped results
  • Add support for nested resources loaded via JS and CSS
Owner
Daniel Sullivan
Daniel Sullivan
DataHen Till is a standalone tool that instantly makes your existing web scraper scalable, maintainable, and more unblockable, with minimal code changes on your scraper.

DataHen Till is a standalone tool that instantly makes your existing web scraper scalable, maintainable, and more unblockable, with minimal code changes on your scraper.

DataHenHQ 769 Nov 26, 2021
Cirno-go A tool for downloading books from hbooker in Go.

Cirno-go A tool for downloading books from hbooker in Go. Features Login your own account Search books by book name Download books as txt and epub fil

沚水 27 Oct 13, 2021
DorkScout - Golang tool to automate google dork scan against the entiere internet or specific targets

dorkscout dokrscout is a tool to automate the finding of vulnerable applications or secret files around the internet throught google searches, dorksco

R4yan 138 Nov 24, 2021
🦙 acao(阿草), the tool man for data scraping of https://asoul.video/.

?? acao acao(阿草), the tool man for data scraping of https://asoul.video/. Deploy to Aliyun serverless function with Raika update_member Update A-SOUL

A-SOUL Video 4 Nov 30, 2021
A simple go program which checks if your websites are running and runs forever (stop it with ctrl+c). It takes two optional arguments, comma separated string with urls and an interval.

uptime A simple go program which checks if your websites are running and runs forever (stop it with ctrl+c). It takes two optional arguments: -interva

Markus Tenghamn 5 May 26, 2021
HTTP Echo is a go web server that echos back the arguments given to it.

HTTP Echo is a go web server that echos back the arguments given to it. This is especially useful for demos or a more extensive "hello world" application in Docker or Kubernetes.

jxlwqq 3 Nov 28, 2021
Link converter service converts URLs to deeplinks or deeplinks to URLs.

Link converter Link converter service converts URLs to deeplinks or deeplinks to URLs. The service responds to the incoming request and first checks w

İlker Rişvan 0 Nov 27, 2021
A proxy that authorizes and enforces a given label in a given PromQL query

prom-authzed-proxy prom-authzed-proxy is a proxy for Prometheus that authorizes the request's Bearer Token with Authzed and enforces a label in a Prom

authzed 25 Nov 26, 2021
You had one job, or more then one, which can be done in steps

Leprechaun Leprechaun is tool where you can schedule your recurring tasks to be performed over and over. In Leprechaun tasks are recipes, lets observe

Strahinja 84 Nov 30, 2021
A tool to enumerate all the command-line arguments used to start a Linux process written in Go.

ranwith A tool to enumerate all the command-line arguments used to start a Linux process written in Go. ranwith uses the Linux /proc directory to obta

Alexis Rodriguez 1 Oct 16, 2021
Parse any web page for URLs and return the HTTP response code of each one.

ParseWebPage - Fully Functional WebPage Parser Parse any web page for URLs and return the HTTP response code of each one. Creators ?? Steven Williams

null 0 Oct 25, 2021
Scrape the Twitter Frontend API without authentication with Golang.

Twitter Scraper Twitter's API is annoying to work with, and has lots of limitations — luckily their frontend (JavaScript) has it's own API, which I re

Nomadic 154 Dec 7, 2021
Scrape the web in the eink era. Convert websites into books.

Scrape the web in the eink era. Convert websites into books.

null 10 Nov 18, 2021
🤖 Automatically scrape PortableApps.com (or official release page) and convert into Edgeless plugin package

Edgeless 自动插件机器人 2 简介 该项目是为了使用 Golang 重新实现 Edgeless 自动插件机器人 特性 (WIP) 完全兼容 Edgeless 自动插件机器人,包括 Tasks,以实现无缝迁移 更快的构建速度 更好的代码结构 更高的拓展性 工作进度 截止至 2021/11/28

Hydrogen 1 Nov 28, 2021
argv - Go library to split command line string as arguments array using the bash syntax.

Argv Argv is a library for Go to split command line string into arguments array. Documentation Documentation can be found at Godoc Example func TestAr

null 31 Nov 15, 2021
A command-line arguments parser that will make you smile.

docopt-go An implementation of docopt in the Go programming language. docopt helps you create beautiful command-line interfaces easily: package main

null 1.3k Nov 30, 2021
🚀 goprobe is a promising command line tool for inspecting URLs with modern and user-friendly way.

goprobe Build go build -o ./bin/goprobe Example > goprobe https://github.com/gaitr/goprobe > cat links.txt | goprobe > echo "https://github.com/gaitr/

null 3 Oct 24, 2021
Small gh extension that suggests issues to work on in a given GitHub repository

gh contribute being a gh extension for finding issues to help with in a GitHub repository. This extension suggests an issue in a given repository to w

Nate Smith 16 Nov 30, 2021
An open-source, distributed, cloud-native CD (Continuous Delivery) product designed for developersAn open-source, distributed, cloud-native CD (Continuous Delivery) product designed for developers

Developer-oriented Continuous Delivery Product ⁣ English | 简体中文 Table of Contents Zadig Table of Contents What is Zadig Quick start How to use? How to

null 0 Oct 19, 2021
System resource usage profiler tool which regularly takes snapshots of the memory and CPU load of one or more running processes so as to dynamically build up a profile of their usage of system resources.

Vegeta is a system resource usage tracking tool built to regularly take snapshots of the memory and CPU load of one or more running processes, so as to dynamically build up a profile of their usage of system resources.

Kartik 8 Oct 28, 2021
Brigodier is a command parser & dispatcher, designed and developed for command lines such as for Discord bots or Minecraft chat commands. It is a complete port from Mojang's "brigadier" into Go.

brigodier Brigodier is a command parser & dispatcher, designed and developed to provide a simple and flexible command framework. It can be used in man

Minekube 11 Aug 29, 2021
To Make frp with no arguments and ini file ,which Conveniently in red teaming

frpBuilder To Make frp with no arguments ,which Conveniently in red teaming I will give a simple modified source code of frp and Builder(MFC C++) in t

evilash 12 Nov 11, 2021
Go linter that warns about number of arguments in functions.

argslen linter Argslen is a linter that checks for long list of argument in functions. The default limit is 5 (maxArguments) and skip the test files (

Federico Guerinoni 5 Sep 28, 2021
A database connection wrapper to cache prepared statements by transforming queries to use with array arguments.

sqlpp sqlpp is a sql(MySQL and PostgreSQL) database connection wrapper to cache prepared statements by transforming queries ("...in (?)...", []) to us

Nazım Parlar 4 Sep 23, 2021
Clockwise is a meeting cost calculator designed to encourage more efficient meetings.

↻lockwise Clockwise is a meeting cost calculator designed to encourage more efficient meetings. The meetings will continue until morale improves.

null 470 Nov 17, 2021
Time-Based One-Time Password (TOTP) and HMAC-Based One-Time Password (HOTP) library for Go.

otpgo HMAC-Based and Time-Based One-Time Password (HOTP and TOTP) library for Go. Implements RFC 4226 and RFC 6238. Contents Supported Operations Read

Jose Torres 26 Nov 12, 2021
A pair of local reverse proxies (one in Windows, one in Linux) for Tailscale on WSL2

tailscale-wsl2 TL;DR Running two reverse proxies (one in Windows, one in the WSL2 Linux VM), the Windows Tailscale daemon can be accessed via WSL2: $

Danny Hermes 0 Dec 6, 2021