Distributed-Services - Distributed Systems with Golang to consequently build a fully-fletched distributed service

Overview

Distributed-Services

This project is essentially a result of my attempt to understand the concepts of Distributed Systems, and implement them with Golang to consequently build a fully-fletched distributed service. Since, the learning curve itself is madly steep, I have separated the project developement in the stages listed below. However, the main aim is to build a distributed service with it's very own storage handling, networking over a client and server, and a way to distribute the server instances. At the end, if possible, I plan to deploy the service with Kubernetes to the cloud.

At this point in time ( 8th Janauary 2022), the first step has been successfully completed and tested. The second step is expected to be completed by the end of January.

The stages were decided in this order to reflect the content structure of the book Distributed Services with Go, written by Travis Jeffery. As the book proceeds with the concepts, I have tried to independently and simultaneously learn and build the different components of the service. Finally, the stages are as follows :

Building the project's storage layer, a web server to faciliate JSON over HTTP, and a custom made log libray

  • Develop the JSON over HTTP commit log service
  • Setup protobufs, and ways to aumatically generate the data structures based on the protobuf message structures
  • Building a commit log library that will essentially be the log for the entire service, to store and lookup data
    • The commit log library has the following structure:
      • A component that allows appending and reading records from the log by provisioning independent structures and methods to faciliate the following
        • Store file handling for record entries
        • Index file handling for index entries of the corresponding records
      • A component that combines the store and file handling components to provision a Segment file handling module to coordinate operations across store and index files.
      • Lasty, the final component ties all the components above, specially the Segment module, to create the final Log handling package for the entire libaray.
    • All the files for the log library can be found under the internal/log directory.

Creating the service over a network

  • Setting up gRPC, define the client server APIs in protobuf together with builing the client and server
  • Securing the service with authentication of the server with SSL/TLS, to encrypy/decrypt the data exchanged by authenticating requests with accress tokens.
  • Making service observable by addings logs, metrics and tracing

Distribute - making the service distributed

  • Building discovery into service to make server instances aware of each other
  • Adding Raft consensus to coordinate the efforts of our servers, and turn them into a cluster
  • Putting discovery into out gRPC clients, so that discover and connect to servers with client side load balancing.

Development

The entire development of the project is dependent on my learning curve, and ability to grasp the concepts of distirbued services. Since, the project is entirely for educational purposes, it is hard to predict a possible timeline. However, by the end of this month, the entire project can be expected to be completed.

Disclaimer

As stated above, the project is being built by closely following the content and concepts outlined in the book Distributed Services with Go by Travis Jeffery. Hence, it's being developed purely for my own exploration and learning about distribued services. Howvever, anyone willing to contribue is more than welcomed. Thanks!.

Author

Hamza Yusuff - Email: [email protected]

You might also like...
A distributed, proof of stake blockchain designed for the financial services industry.

Provenance Blockchain Provenance is a distributed, proof of stake blockchain designed for the financial services industry.

This library contains utilities that are useful for building distributed services.

Grafana Dskit This library contains utilities that are useful for building distributed services. Current state This library is still in development. D

Build share and run your distributed applications.
Build share and run your distributed applications.

sealer[ˈsiːlər] provides the way for distributed application package and delivery based on kubernetes.

A distributed lock service in Go using etcd

locker A distributed lock service client for etcd. What? Why? A distributed lock service is somewhat self-explanatory. Locking (mutexes) as a service

A Go library for master-less peer-to-peer autodiscovery and RPC between HTTP services

sleuth sleuth is a Go library that provides master-less peer-to-peer autodiscovery and RPC between HTTP services that reside on the same network. It w

An experimental library for building clustered services in Go

Donut is a library for building clustered applications in Go. Example package main import ( "context" "log" "os" // Wait for etcd client v3.4, t

Leader-follower-pattern - Build leader-follower system pattern with etcd election

主备系统模式 原理 使用分布式锁实现主备节点系统。通过对分布式锁进行续期,保持长期锁, 从而使当前服务节点处于主服务节点 无法获取分布式锁的服务节点,则作为备选

Distributed reliable key-value store for the most critical data of a distributed system

etcd Note: The main branch may be in an unstable or even broken state during development. For stable versions, see releases. etcd is a distributed rel

Library for enabling asynchronous health checks in your service

go-health A library that enables async dependency health checking for services running on an orchestrated container platform such as kubernetes or mes

Owner
Hamza Yusuff
Hamza Yusuff
Distributed lock manager. Warning: very hard to use it properly. Not because it's broken, but because distributed systems are hard. If in doubt, do not use this.

What Dlock is a distributed lock manager [1]. It is designed after flock utility but for multiple machines. When client disconnects, all his locks are

Sergey Shepelev 25 Dec 24, 2019
Go Micro is a framework for distributed systems development

Go Micro Go Micro is a framework for distributed systems development. Overview Go Micro provides the core requirements for distributed systems develop

Asim Aslam 19.1k Sep 21, 2022
Go Micro is a standalone framework for distributed systems development

Go Micro Go Micro is a framework for distributed systems development. Overview Go Micro provides the core requirements for distributed systems develop

Asim Aslam 19.1k Sep 26, 2022
Tarmac is a unique framework designed for the next generation of distributed systems

Framework for building distributed services with Web Assembly

Benjamin Cane 212 Sep 14, 2022
A distributed systems library for Kubernetes deployments built on top of spindle and Cloud Spanner.

hedge A library built on top of spindle and Cloud Spanner that provides rudimentary distributed computing facilities to Kubernetes deployments. Featur

null 21 Jan 4, 2022
MIT 6.824: Distributed Systems

MIT 6.824 is a core 12-unit graduate subject with lectures, readings, programming labs, an optional project, a mid-term exam, and a final exam.

CC Chang 2 Jul 6, 2022
Distributed Systems 2021 - Miniproject 3

Distributed Systems 2021 -- Miniproject 3 Hand-in Date: 1 December 2021 (at 23:59) What to submit on learnit: a single zip-compressed file containing:

null 0 Dec 11, 2021
Labs, solutions and related materials from the MIT 6.824 Distributed Systems course.

MIT 6.824 Distributed Systems Labs, solutions and related materials from the MIT 6.824 Distributed Systems course. Overview From the official website:

Arindam Das 44 Sep 16, 2022
MIT6.824 Distributed Systems

MIT6.824-Distributed-Systems My Solutions for MIT6.824

null 0 Jan 28, 2022
Skynet is a framework for distributed services in Go.

##Introduction Skynet is a communication protocol for building massively distributed apps in Go. It is not constrained to Go, so it will lend itself n

null 2k Sep 29, 2022