UCSD CSE 124 Fall 2021 Project 3 Starter Code

Related tags

Miscellaneous proj3
Overview

Surfstore

This is the starter code for Module 3: Surfstore.

Before you get started, make sure you understand the following 2 things about Go. (These will also be covered in class and in discussions)

  1. Interfaces: They are named collections of method signatures. Here are some good resources to understand interfaces in Go: a. https://gobyexample.com/interfaces b. https://jordanorelli.com/post/32665860244/how-to-use-interfaces-in-go

  2. RPC: You should know how to write RPC servers and clients in Go. The online documentation of the rpc package is a good resource.

Data Types

Recall from the module write-up the following things:

  1. The SurfStore service is composed of two services: BlockStore and MetadataStore
  2. A file in SurfStore is broken into an ordered sequence of one or more blocks which are stored in the BlockStore.
  3. The MetadataStore maintains the mapping of filenames to hashes of these blocks (and versions) in a map.

The starter code defines the following types for your usage in SurfstoreInterfaces.go:

type Block struct {
	BlockData []byte
	BlockSize int
}

type FileMetaData struct {
	Filename      string
	Version       int
	BlockHashList []string
}

Surfstore Interface

SurfstoreInterfaces.go also contains interfaces for the BlockStore and the MetadataStore:

type MetaStoreInterface interface {
	// Retrieves the server's FileInfoMap
	GetFileInfoMap(_ignore *bool, serverFileInfoMap *map[string]FileMetaData) error
	
	// Update a file's fileinfo entry
	UpdateFile(fileMetaData *FileMetaData, latestVersion *int) (err error)

	// Retrieve the mapping of BlockStore addresses to block hashes
	GetBlockStoreMap(blockHashesIn []string, blockStoreMap *map[string][]string) error
}

type BlockStoreInterface interface {

	// Get a block based on its hash
	GetBlock(blockHash string, block *Block) error

	// Put a block
	PutBlock(block Block, succ *bool) error

	// Check if certain blocks are alredy present on the server
	HasBlocks(blockHashesIn []string, blockHashesOut *[]string) error
}

Server

BlockStore.go provides a skeleton implementation of the BlockStoreInterface and MetaStore.go provides a skeleton implementation of the MetaStoreInterface You must implement the methods in these 2 files which have panic("todo") as their body.

init/SurfstoreServerExec/main.go also has a method startServer which you must implement. Depending on the service type specified, it should register a MetaStore, BlockStore, or Both and start listening for connections from clients.

Client

SurfstoreRPCClient.go provides the rpc client stub for the surfstore rpc server. You must implement the methods in this file which have panic("todo") as their body. (Hint: one of them has been implemented for you)

SurfstoreClientUtils.go also has the following method which you need to implement for the sync logic of clients:

/*
Implement the logic for a client syncing with the server here.
*/
func ClientSync(client RPCClient) {
	panic("todo")
}

Setup

You will need to setup your runtime environment variables so that you can build your code and also use the executables that will be generated.

  1. If you are using a Mac, open ~/.bash_profile or if you are using a unix/linux machine, open ~/.bashrc. Then add the following:
export GOPATH=
   
    
export PATH=$PATH:$GOPATH/bin
export GO111MODULE=off

   
  1. Run source ~/.bash_profile or source ~/.bashrc

Usage

  1. Only after you have implemented all the methods and completed the Setup steps, run the build.sh script provided with the starter code. This should create 2 executables in the bin folder inside your starter code directory.
> ./build.sh
> ls bin
SurfstoreClientExec SurfstoreServerExec
  1. Run your server using the script provided in the starter code.
./run-server.sh -s <service> -p <port> -l -d (BlockStoreAddr*)

Here, service should be one of three values: meta, block, or both. This is used to specify the service provided by the server. port defines the port number that the server listens to (default=8080). -l configures the server to only listen on localhost. -d configures the server to output log statements. Lastly, (BlockStoreAddr*) is zero or more initial BlockStore addresses that the server is configured with. For module 3, the MetaStore should always start with 1 BlockStore address and if service=both then the BlockStoreAddr should be the ip:port of this server.

Examples:

./run-server.sh -s both -p 8070 -l localhost:8070

This starts a server that listens only to localhost on port 8070 and services both the BlockStore and MetaStore interface.

Run the commands below on separate terminals (or nodes)
> ./run-server.sh -s block -p 8081 -l
> ./run-server.sh -s meta -l localhost:8081

The first line starts a server that services only the BlockStore interface and listens only to localhost on port 8081. The second line starts a server that services only the MetaStore interface, listens only to localhost on port 8080, and references the BlockStore we created as the underlying BlockStore. (Note: if these are on separate nodes, then you should use the public ip address and remove -l)

  1. From a new terminal (or a new node), run the client using the script provided in the starter code (if using a new node, build using step 1 first). Use a base directory with some files in it.
> mkdir dataA
> cp ~/pic.jpg dataA/ 
> ./run-client.sh server_addr:port dataA 4096

This would sync pic.jpg to the server hosted on server_addr:port, using dataA as the base directory, with a block size of 4096 bytes.

  1. From another terminal (or a new node), run the client to sync with the server. (if using a new node, build using step 1 first)
> ls dataB/
> ./run-client.sh server_addr:port dataB 4096
> ls dataB/
pic.jpg index.txt

We observe that pic.jpg has been synced to this client.

Testing

On gradescope, only a subset of test cases will be visible, so we highly encourage you to come up with different scenarios like the one described above. You can then match the outcome of your implementation to the expected output based on the theory provided in the writeup.

To avoid autograder configuration issues, we also encourage you to test your solution on AWS which is likely configured with the same go version (1.14.6). This also means that your solution should avoid using methods and packages introduced after go 1.14.6.

Owner
UCSD CSE 124 Fall 2021
UCSD CSE 124 Fall 2021
A boilerplate/starter project for quickly building APIs using Golang with Clean Architechture

A boilerplate/starter project for quickly building APIs using Golang with Clean Architechture

Nyoman Frastyawan 1 Mar 2, 2022
Example hello-world service uses go-fx-grpc-starter boilerplate code

About Example hello-world service uses https://github.com/srlk/go-fx-grpc-starter boilerplate code. Implementation A hello world grpc service is creat

null 0 Nov 14, 2021
🎄 My code for the Advent of Code of year 2021 in Go.

Advent of Code 2021 This repository contains all code that I wrote for the Advent of Code 2021. This year I chose to try and learn Go. Enjoy! Built wi

Nick van Ravenzwaaij 0 Dec 9, 2021
bytecamp 2021 project

Calldiff 背景 复杂项目的日常迭代中,研发同学想了解某个改动的影响点,往往需要深入代码细节才能获得尽可能全的 checklist 。然而,由于项目的复杂性,我们获得的 checklist 又难免会有一些遗漏。 为了减轻研发同学的心智负担,辅助同学们更有效地保证服务稳定性,本项目应运而生。 本

null 3 Oct 4, 2021
Code for the keycloak demo for the talk I gave at blablaconf 2021

This demo illustrates the usage of keycloak to secure several service accross different clients using roles. How to Run: From a terminal you need to r

Zakaria Amine 1 Oct 30, 2021
Advent of Code 2021 Solutions

Advent of Code 2021 Solutions for the 2021 Advent of Code Building This project makes use of Go 1.17. go mod download go test ./... Running the Soluti

Nathan Lowe 7 Jan 7, 2022
Finished snippets I did for Advent of Code 2021.

Advent of Code 2021 This repository includes my Go code for the Advent of Code 2021. If you want to solve the puzzles yourself first, do so, then look

Mario Liebisch 0 Dec 25, 2021
Advent of Code 2021 - Time to learn Go

aoc2021 Advent of Code 2021 - Time to learn Go Will contain my solutions for aoc2021, so avoid reading the files in .src/aoc2021/ unless you want spoi

null 1 Dec 22, 2021
Advent of Code 2021, experiment in learning Go

aoc_2021 Advent of Code 2021, experiment in learning Go To run, open a terminal in the root folder of the project and run with go run . <day to run> <

Jonathan den Herder 1 Mar 1, 2022
Advent of Code 2021, this time in Go

Go Lang Notes Advent of Code Day 3 The distinction between chars and bytes is a bit annoying. I got tripped up by doing: int(str[pos]) which gives yo

Dan Vanderkam 15 Apr 14, 2022
My solutions for 2021's Advent of Code

Advent of Code 2021 These are my solutions to this year's Advent of Code. I used it as an excuse to practice with a language which I'm not yet very fa

Merlijn Visser 1 Dec 27, 2021
Advent of Code 2021 (Go)

Advent of Code 2021 (Go) This project includes an implementation of the Advent of Code 2021 problems, implemented in Go. These are ports from my C# so

Brad Wilson 1 Dec 18, 2021
Solutions to Advent-of-Code 2021, in Go.

?? advent-of-code-2021 ?? Solutions to 2021 Advent of code. Summary Advent of Code is an annual advent-calendar of programming puzzles. Here are my 20

James Cockbain 2 Jan 3, 2022
Advent of code 2021 solutions by: me :)

Advent of code 2021 These are my solutions for the advent of code 2021 event. My idea is to solve all the problems in go, but I may use another langua

Maitgon 4 Dec 27, 2021
A little repository for my Advent of Code 2021 solutions in Go

Advent of Code 2021 What's this repo all about? Well, this is a collection of code written in Golang to solve the daily problems presented in Advent o

Lars Joseph Norlander 1 Dec 12, 2021
Advent of Code 2021 solutions using Go 1.18 Generics

advent-of-code-2021 Here are my solutions for Advent of Code 2021. This year, I chose to write my solutions using Go 1.18 with generics (by building t

Glenn Lewis 3 Feb 16, 2022
My solutions for Advent of Code 2021.

My solutions for Advent of Code 2021 in go. The solution and input data (i.e. my custom input data) for day XX is contained in the subdirectory dayXX.

Florian Pigorsch 2 Dec 22, 2021
Auto Judger for BUAA-SE-OOP Course (2021 Spring)

patpat Auto Judger for BUAA-SE-OOP Course (2021 Spring) 1. 评测机使用方法 1.1. 一些准备工作 1.1.1. 下载评测机 见 GitHub 中的 Releases,下载对应版本即可。目前提供 Windows, Linux, MacOS(I

null 28 Feb 27, 2022
An easy-to-use Map Reduce Go parallel-computing framework inspired by 2021 6.824 lab1. It supports multiple workers on a single machine right now.

MapReduce This is an easy-to-use Map Reduce Go framework inspired by 2021 6.824 lab1. Feature Multiple workers on single machine right now. Easy to pa

Bo-Wei Chen 193 Jun 9, 2022