File Processor in Concurrency Pattern using Golang goroutine.

Overview

File Processor in Concurrency Pattern

Implement a file processor solution in concurrency pattern using Golang goroutine.

App architecture

Get Started

  1. Run docker-compose up to start the localstack and redis services.
  2. Create a Dynamodb table if the storage is Dynamodb.
    aws dynamodb create-table --table-name Users --attribute-definitions AttributeName=UUID,AttributeType=S --key-schema AttributeName=UUID,KeyType=HASH --billing-mode PAY_PER_REQUEST --endpoint-url=http://localhost:4566 --profile localstack
    
  3. There are 3 types of storage services defined, Dynamodb, Redis, and Fake storages. The storage service is injected via this code.
    writer := &fileprocessor.StorageWriter{
    	// Storage: storage.NewDynamodbService(),
    	// Storage: storage.NewFakeStorage(),
    	Storage: storage.NewRedisService(),
    }
    
  4. Run go run main.go --path /path/to/file

Tips

Localstack service health checks

http://localhost:4566/health?reload

Start a Redis cli to connect to the Redis server in docker container.

docker run -it --rm \
> --network go-file-processor_app-tier \
> bitnami/redis:6.2.6 redis-cli -h go-file-processor_redis_1
Pluggable, extensible virtual file system for Go

vfs Package vfs provides a pluggable, extensible, and opinionated set of file system functionality for Go across a number of file system types such as

C2FO 133 Oct 20, 2021
A Go filesystem package for working with files and directories

Stowage A Go filesystem package for working with files and directories, it features a simple API with support for the common files and directories ope

null 19 May 28, 2021
Run a command when files change

Reflex Reflex is a small tool to watch a directory and rerun a command when certain files change. It's great for automatically running compile/lint/te

Caleb Spare 2.4k Oct 21, 2021
pprofutils pprofutils is a swiss army knife for pprof files.

pprofutils pprofutils is a swiss army knife for pprof files. You can use it as a command line utility or as a free web service. Install Utilities: ano

Felix Geisendörfer 74 Oct 1, 2021
A FileSystem Abstraction System for Go

A FileSystem Abstraction System for Go Overview Afero is a filesystem framework providing a simple, uniform and universal API interacting with any fil

Steve Francia 4k Oct 25, 2021
A package to allow one to concurrently go through a filesystem with ease

skywalker Skywalker is a package to allow one to concurrently go through a filesystem with ease. Features Concurrency BlackList filtering WhiteList fi

Will Dixon 65 Aug 31, 2021
High Performance, Kubernetes Native Object Storage

MinIO Quickstart Guide MinIO is a High Performance Object Storage released under Apache License v2.0. It is API compatible with Amazon S3 cloud storag

High Performance, Kubernetes Native Object Storage 29.7k Oct 25, 2021
Golang PDF library for creating and processing PDF files (pure go)

UniPDF - PDF for Go UniDoc UniPDF is a PDF library for Go (golang) with capabilities for creating and reading, processing PDF files. The library is wr

UniDoc 1.4k Oct 21, 2021
FSManager - Tree view Simple util to displays the directory structure of a path or of the disk in a drive graphically.

FSManager - Tree view Simple util to displays the directory structure of a path or of the disk in a drive graphically. If you don't specify a drive or

Kostiantyn Denysov 1 Oct 9, 2021
Dragonfly is an intelligent P2P based image and file distribution system.

Dragonfly Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in o

dragonflyoss 5.7k Oct 14, 2021
Plik is a scalable & friendly temporary file upload system ( wetransfer like ) in golang.

Want to chat with us ? Telegram channel : https://t.me/plik_root_gg Plik Plik is a scalable & friendly temporary file upload system ( wetransfer like

root.gg 743 Oct 24, 2021
Cross-platform file system notifications for Go.

File system notifications for Go fsnotify utilizes golang.org/x/sys rather than syscall from the standard library. Ensure you have the latest version

fsnotify 6.1k Oct 16, 2021
Easily create & extract archives, and compress & decompress files of various formats

archiver Introducing Archiver 3.1 - a cross-platform, multi-format archive utility and Go library. A powerful and flexible library meets an elegant CL

Matt Holt 3.3k Oct 23, 2021
Encrypted overlay filesystem written in Go

An encrypted overlay filesystem written in Go. Official website: https://nuetzlich.net/gocryptfs (markdown source). gocryptfs is built on top the exce

null 2k Oct 15, 2021
Fast, dependency-free, small Go package to infer the binary file type based on the magic numbers signature

filetype Small and dependency free Go package to infer file and MIME type checking the magic numbers signature. For SVG file type checking, see go-is-

Tom 1.4k Oct 17, 2021
The best HTTP Static File Server, write with golang+vue

gohttpserver Goal: Make the best HTTP File Server. Features: Human-friendly UI, file uploading support, direct QR-code generation for Apple & Android

Sound Sun 1.3k Oct 21, 2021
File system for GitHub

HUBFS · File System for GitHub HUBFS is a read-only file system for GitHub and Git. Git repositories and their contents are represented as regular dir

Bill Zissimopoulos 144 Oct 21, 2021
gsheet is a CLI tool (and Golang package) for piping csv data to and from Google Sheets

gsheet Table of Contents Introduction Why? Installation Authentication and Authorization What about OAuth authentication? CLI Usage Sheet commands Dri

chris 13 Aug 5, 2021