Labs, solutions and related materials from the MIT 6.824 Distributed Systems course.

Overview

MIT 6.824 Distributed Systems

Labs, solutions and related materials from the MIT 6.824 Distributed Systems course.

distributed-systems

ci-tests status

Overview

From the official website:

What is 6.824 about?

6.824 is a core 12-unit graduate subject with lectures, readings, programming labs, an optional project, a mid-term exam, and a final exam. It will present abstractions and implementation techniques for engineering distributed systems. Major topics include fault tolerance, replication, and consistency. Much of the class consists of studying and discussing case studies of distributed systems.

This repository was created to serve as an environment for working on the labs as well as hosting my solutions for them. It's primary purpose is to serve as learning resource and a pedagogical tool. Each lab will be accompanied with a corresponding specification and design document to outline the approach used to solve the tasks. Effort will be made to make the solutions production grade, but at the same time, approachable for any Gopher.

The tests from the original source tree at git://g.csail.mit.edu/6.824-golabs-2021 have been preserved to a great extent to ensure the authenticity of the labs. The only changes to the tests are updated paths for different artefacts due to the changes in the directory structure.

Organization

The repository is organized as follows:

.
├── assets
│   └── logo.png
├── cmd
│   ├── diskvd
│   ├── lockservice
│   ├── mr
│   ├── pbservice
│   └── viewd
├── datasets
│   └── project-gutenberg
├── LICENSE
├── pkg
│   ├── kvraft
│   ├── labgob
│   ├── labrpc
│   ├── models
│   ├── mr
│   ├── porcupine
│   ├── raft
│   ├── shardctrler
│   └── shardkv
└── README.md

The core of each module is represented as a package under the pkg/ subdirectories. Most of these packages are accompanied by tests in the same directory. The user facing utlitiy command/program binaries are kept under cmd/.

Labs Progress

Each completed checklist item is also a link to its spec and design document.

  • Lab 1: MapReduce
  • Lab 2: Raft Consensus Algorithm
    • Lab 2A: Raft Leader Election
    • Lab 2B: Raft Log Entries Append
    • Lab 2C: Raft state persistence
  • Lab 3: Fault-tolerant Key/Value Service
    • Lab 3A: Key/value Service Without Log Compaction
    • Lab 3B: Key/value Service With Log Compaction
  • Lab 4: Sharded Key/Value Service

Self study

If you wish to work on the labs on your own, checkout from the tag v0.1.0 into a new branch.

git checkout v0.1.0
git checkout -b ${branch}

In order to use github actions to check your labs, add your branch to the test workflow:

--- a/.github/workflows/ci-tests.yml
+++ b/.github/workflows/ci-tests.yml
@@ -4,12 +4,14 @@ on:
   push:
     branches:
       - main
+      - ${branch}
     paths:
       - cmd/**
       - pkg/**
   pull_request:
     branches:
       - main
+      - ${branch}
     paths:
       - cmd/**
       - pkg/**

Testing

All of the labs from the original source tree has been migrated to go modules. Except for mapreduce, the tests for all labs may be run as follows:

git clone https://github.com/arindas/mit-6.824-distributed-systems.git
cd mit-6.824-distributed-systems
cd pkg/${PACKAGE_TO_TEST}
go test

For the mapreduce lab, run the tests as follows:

cd cmd/mr/scripts
bash test-mr.sh  # single trial
bash test-mr-many.sh $N_TRIALS  # multiple trials

Building

Required binaries can be built by running go build in the respective cmd/ subdirectories.

Attribution

Materials presented in this repository are based on the materials from MIT 6.824 Distributed Systems course, which are distributed under the Creative Commons License, as mentioned on their website. This repository falls under fair use of the Creative Commons License.

In case of any disputes, please contact: Arindam Das [email protected]

License

The solutions presented here are licensed under the MIT License. See LICENSE for full license text.

Releases(v1.0.0)
Owner
Arindam Das
Specializes in distributed systems, deep learning inference and AI SaaS at scale.
Arindam Das
MIT6.824 Distributed Systems

MIT6.824-Distributed-Systems My Solutions for MIT6.824

null 0 Jan 28, 2022
Distributed-Services - Distributed Systems with Golang to consequently build a fully-fletched distributed service

Distributed-Services This project is essentially a result of my attempt to under

Hamza Yusuff 6 Jun 1, 2022
Distributed lock manager. Warning: very hard to use it properly. Not because it's broken, but because distributed systems are hard. If in doubt, do not use this.

What Dlock is a distributed lock manager [1]. It is designed after flock utility but for multiple machines. When client disconnects, all his locks are

Sergey Shepelev 25 Dec 24, 2019
null 1 Feb 3, 2022
A distributed systems library for Kubernetes deployments built on top of spindle and Cloud Spanner.

hedge A library built on top of spindle and Cloud Spanner that provides rudimentary distributed computing facilities to Kubernetes deployments. Featur

null 21 Jan 4, 2022
Go Micro is a framework for distributed systems development

Go Micro Go Micro is a framework for distributed systems development. Overview Go Micro provides the core requirements for distributed systems develop

Asim Aslam 18.6k Jul 6, 2022
Go Micro is a standalone framework for distributed systems development

Go Micro Go Micro is a framework for distributed systems development. Overview Go Micro provides the core requirements for distributed systems develop

Asim Aslam 18.6k Jul 1, 2022
Tarmac is a unique framework designed for the next generation of distributed systems

Framework for building distributed services with Web Assembly

Benjamin Cane 192 Jun 26, 2022
Distributed Systems 2021 - Miniproject 3

Distributed Systems 2021 -- Miniproject 3 Hand-in Date: 1 December 2021 (at 23:59) What to submit on learnit: a single zip-compressed file containing:

null 0 Dec 11, 2021
dht is used by anacrolix/torrent, and is intended for use as a library in other projects both torrent related and otherwise

dht Installation Install the library package with go get github.com/anacrolix/dht, or the provided cmds with go get github.com/anacrolix/dht/cmd/....

Matt Joiner 239 Jun 29, 2022
The TinyKV course builds a key-value storage system with the Raft consensus algorithm.

The TinyKV Course The TinyKV course builds a key-value storage system with the Raft consensus algorithm. It is inspired by MIT 6.824 and TiKV Project.

jaegerwang 1 Nov 19, 2021
Distributed reliable key-value store for the most critical data of a distributed system

etcd Note: The main branch may be in an unstable or even broken state during development. For stable versions, see releases. etcd is a distributed rel

etcd-io 40.4k Jun 30, 2022
High performance, distributed and low latency publish-subscribe platform.

Emitter: Distributed Publish-Subscribe Platform Emitter is a distributed, scalable and fault-tolerant publish-subscribe platform built with MQTT proto

emitter 3.3k Jul 4, 2022
Fast, efficient, and scalable distributed map/reduce system, DAG execution, in memory or on disk, written in pure Go, runs standalone or distributedly.

Gleam Gleam is a high performance and efficient distributed execution system, and also simple, generic, flexible and easy to customize. Gleam is built

Chris Lu 3.1k Jun 28, 2022
Go Open Source, Distributed, Simple and efficient Search Engine

Go Open Source, Distributed, Simple and efficient full text search engine.

ego 6.1k Jul 4, 2022
Dapr is a portable, event-driven, runtime for building distributed applications across cloud and edge.

Dapr is a portable, serverless, event-driven runtime that makes it easy for developers to build resilient, stateless and stateful microservices that run on the cloud and edge and embraces the diversity of languages and developer frameworks.

Dapr 18.4k Jun 28, 2022
Asynq: simple, reliable, and efficient distributed task queue in Go

Asynq: simple, reliable, and efficient distributed task queue in Go

Ken Hibino 3.6k Jul 5, 2022