An easy-to-use Map Reduce Go parallel-computing framework inspired by 2021 6.824 lab1. It supports multiple workers on a single machine right now.

Overview

MapReduce

github build status Go Reference

This is an easy-to-use Map Reduce Go framework inspired by 2021 6.824 lab1.

Feature

  • Multiple workers on single machine right now.
  • Easy to parallel your code with just Map and Reduce function.

Usage

Here's a simply example for word count program.

main.go

package main

import (
	mp "github.com/BWbwchen/MapReduce"
)

func main() {
	mp.StartSingleMachineJob(mp.ParseArg())
}

And write down your own Map and Reduce function.

wc.go

package main
import (
	"strconv"
	"strings"
	"unicode"

	"github.com/BWbwchen/MapReduce/worker"
)
func Map(filename string, contents string, ctx worker.MrContext) {
	// function to detect word separators.
	ff := func(r rune) bool { return !unicode.IsLetter(r) }

	// split contents into an array of words.
	words := strings.FieldsFunc(contents, ff)

	for _, w := range words {
		ctx.EmitIntermediate(w, "1")
	}
}
func Reduce(key string, values []string, ctx worker.MrContext) {
	// return the number of occurrences of this word.
	ctx.Emit(key, strconv.Itoa(len(values)))
}

Run with :

go build -race -buildmode=plugin -o wc.so wc.go
go run -race main.go -i 'txt/*' -p 'wc.so' -r 1 -w 8

Output file name is mr-out-0.txt

More example can be found in the mrapps/ folder, and we will add more example in the future.

Made by Bo-Wei Chen. All code is licensed under the MIT License.
Releases(v1.0.0)
Advent of Code 2021 https://adventofcode.com/2021

AOC 2021 How to use Not sure yet. Maybe cd into your day folder and go run, or maybe better to try go test Dev environment Open in VSCode, enable (Rem

bozdoz 3 Apr 22, 2022
mit 6.824 lab

lab-6.824 0. How to run? go版本: 1.13+ 在高于1.11的版本中报unexpected directory layout, 但是用较低版本goLand无法调试... unexpected dir layout 原因是不支持相对路径包引入, 遇到时在import 删掉.

JinyuChata 1 Dec 9, 2021
learn mit 6.824 lab

MIT6.824 lab1 MapReduce timeout: command not found -> brew install coreutils panic data race -> 加锁 内层变量会屏蔽外部同名变量 test1 word-count 测试基本功能 test2 indexer

GoodbyeYesterDay 0 Jan 5, 2022
My code of the course mit6.824

MIT6.824 Lab1 Rules 最后文件需要输出nReduce个,文件名格式为mr-out-X 输出到文件的格式在mrsequential.go中 只用写worker.go/coordinator.go/rpc.go这三个文件 worker将中间文件输出到当前文件夹下,之后worker执行r

TheR1sing3un 8 Jun 17, 2022
Implement based on course of mit_6.824 and raft paper

Mit6.824_raft implement based on course of mit_6.824 and raft paper ##Mit 6.824 ###2A leader election Implement Raft leader election and heartbeats (A

Razer 0 Feb 13, 2022
labs & notes on 6.824

Lab 1: MapReduce MapReduce论文 概念 MapReduce is a programming model and an associated implementation for processing and generating large data sets. Users

Vurtne 1 Mar 6, 2022
MIT 6.824: Distributed Systems (Spring 2020)

MIT6.824 MIT 6.824: Distributed Systems (Spring 2020) Lab 1 Lab 2 Lab 2A Lab 2B Lab 2C Lab 2D Lab 3 Lab 3A Lab 3B Lab 4 Lab 4A Lab 4B Lab 4 Challenge

Ray Eldath 50 Jun 21, 2022
Clone of the old gasnow.org, now available at gasnow.dev

Gasnow 2 Clone of gasnow.org Quickstart The application connects to infura to get data from its ethereum full node. In order to do so the following en

Pablo Lopez 4 Mar 30, 2022
An ease to use finit state machine golang implementation.Turn any struct to a fsm with graphviz visualization supported.

go-fsm An ease to use finit state machine golang implementation.Turn any struct to a fsm with graphviz visualization supported. usage import github.co

FingerLiu 5 Dec 26, 2021
A library for parallel programming in Go

pargo A library for parallel programming in Go Package pargo provides functions and data structures for expressing parallel algorithms. While Go is pr

null 176 Jun 10, 2022
DSV Parallel Processor takes input files and query specification via a spec file

DSV Parallel Processor Spec file DSV Parallel Processor takes input files and query specification via a spec file (conventionally named "spec.toml").

Wattanit Hotrakool 0 Oct 9, 2021
A simple package for executing work in parallel up to a limit.

concurrencylimiter A simple package for executing work concurrently - up to a limit. The intended usecase looks something like: func concurrentlyDo(ta

Edward Stell 0 Dec 19, 2021
elPrep: a high-performance tool for analyzing sequence alignment/map files in sequencing pipelines.

Overview elPrep is a high-performance tool for analyzing .sam/.bam files (up to and including variant calling) in sequencing pipelines. The key advant

null 254 May 30, 2022
Onmap - Go package onmap puts pins on a world map image

onmap Go package onmap puts pins on a world map image. The images (mercator.jpg,

Dmitry Chestnykh 1 Feb 3, 2022
:sunglasses:Package captcha provides an easy to use, unopinionated API for captcha generation

Package captcha provides an easy to use, unopinionated API for captcha generation. Why another captcha generator? I want a simple and framework-indepe

Weilin Shi 106 Jun 26, 2022
An easy to use, extensible health check library for Go applications.

Try browsing the code on Sourcegraph! Go Health Check An easy to use, extensible health check library for Go applications. Table of Contents Example M

Claudemiro 431 May 31, 2022
Hrple is an easy to use tool to help you create habits

Hrple is an easy to use tool to help you create habits. This is loosely inspired by the book Atomic Habits by James Clear and techniques or frameworks like Kanban and the Pomodoro Technique.

Brett Mostert 1 Jun 2, 2022
Eye - An easy-use lib for event-driven pattern

?? Eye Eye 是一个简单易用的事件驱动模式库。 Read me in English ?? 功能特性 敬请期待。。。 历史版本的特性请查看 HISTOR

水不要鱼 1 Jan 17, 2022
A fast and easy-to-use gutenberg book downloader

Gutenberg Downloader A brief description of what this project does and who it's for Usage download books Download all english books as epubs with imag

Lukas f. Paluch 2 Jan 11, 2022