ftgogo - event-driven architecture demonstration application

Overview

ftgogo - event-driven architecture demonstration application

Introduction

ftgogo (food-to-gogo) is a Golang implementation of the FTGO application described in the book "Microservice Patterns" by Chris Richardson. A library edat was developed to provide for Golang many of the solutions that Eventuate, the framework used by FTGO, provides for Java.

Purpose

This repository exists to demonstrate the patterns and processes involved when constructing a distributed application using event-driven architecture.

Prerequisites

Docker - Everything is built and run from a docker compose environment.

Execution

Open a command prompt and then execute the following docker command

NOTE: The first time you bring everything up the init script for Postgres will run automatically. The services will crash-loop for a bit because of that. Eventually things will stabilize.

Mac and Linux Users

COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose up

Windows Users

set "COMPOSE_DOCKER_CLI_BUILD=1" & set "DOCKER_BUILDKIT=1" & docker-compose up

Use Ctrl-C to stop all services.

Running individual services

Not recommended but each service can be run using go run .. You'll need to use an .env file or set some environment variables to properly run the service.

Use go run . --help to see what flags and the list of environment variables that can be set.

Application

Services

Several services also have sibling CDC services.

Design

  • Services exist within a "domain" folder. Within that folder you'll find the following layout.

    /"domain"        - Domain that is a subdomain in the application domain
    |-/cmd           - Parent for servers, cli, and tools that are built using the code in this domain
    | |-/cdc         - CDC (Change Data Capture) server. If the service publishes messages it will also have this
    | |-/service     - Primary domain service
    |-/internal      - Use Golangs special treatment of "internal" to sequester our code from the other services
      |-/adapters    - The "Adapters" from "Ports & Adapters". These are the implementations of domain interfaces
      |-/application - CQRS parent. Processes under this will implement business rules and logic
      | |-/commands  - Application commands. Processes that apply some change to the subdomain
      | |-/queries   - Application queries. Processes that request information from the subdomain
      |-/domain      - The definitions, interfaces, and the domain rules and logic
      |-/sagas       - The definitions for complex multi-service interactions
    
  • The api for each service has been defined in an /"domain"/cmd/service/openapi.yaml file.

    An issue in the openapi library being used prevents the serving the openapi document via HTTP request. A swagger UI may be added sometime later if this is resolved, or a different solution is used.

  • Each "domain" uses go modules for dependency management and has a go.mod file just for it.

  • The services use the following components from edat

    • edat/es - implements event sourcing
    • edat/msg - implements transactional messaging
    • edat/sagas - implements orchestrated sagas
  • Application domain apis, events, commands, and entities are available to all subdomains from the serviceapis folder and its child packages.

DDD / Hexagonal Architecture / Ports & Adapters

The organizational layout of the services is to bring separation to the layers. The /"domain"/cmd/service/application.go file found in all the services is where ports (web handlers, message receivers) are combined with the application commands and queries.

  • The application is connected to the /internal/domain using interfaces backed by real implementations found in the /internal/adapters folder.
  • Commands and queries have no dependencies on any primary ports or on any adapters directly. In fact, they may only receive adapters that implement secondary ports defined by interfaces in the /internal/domain folder.
  • No dependencies exist on any ports or adapters from the code in /internal/domain and this is crucial to having a clean architecture.

Regarding the layout. What I landed on felt alright to work with, felt like it provided the right separations, and felt like it wasn't an over complication of DDD applied to a folder structure. You be the judge.

CQRS

Each service divides the requests it receives into commands and queries. Using a simple design described here by Three Dots Labs all of our handlers can be setup to use a command or query.

While most application commands in this repository will adhere to the "commands return nothing" rule, some commands return simple scalar values, mostly entity ids, in addition to the error and I am OK with that.

Event Sourcing

Several services use event sourcing and keep track of the changes to aggregates using commands and recorded events. Check out the Order aggregate for an example.

Sagas

The same three sagas found in FTGO have been implemented here in the order-service.

  • CreateOrderSaga - saga responsible for the creation of a new order
  • CancelOrderSaga - saga responsible for the cancelling and releasing of order resources like tickets and accounting reserves
  • ReviseOrderSaga - saga responsible for the processing the changes made to an open order

Event-driven architecture

All inter-service communication is handled asynchronously using messages.

Dual-Writes / Outbox / CDC

An implementation of the outbox pattern is included. It provides the solution to the dual write problem. Any service that publishes messages is actually publishing the message into the database. A CDC sibling service then processes the messages from the database and publishes the message into NATS Streaming. This process provides at-least-once delivery.

Services can be made to publish messages directly without having to use an outbox and CDC. The pattern works best, only?, with backends that provide transactional support.

Other

Metrics/Instrumentation

Prometheus metrics for each service are available at http://localhost:[port]/metrics. The order-service has a few additional counters. See the order-service application for more information.

Mono-repository

This demonstration application is a mono-repository for the Golang services. I chose to use as few additional frameworks as possible so you'll find there is also quite a bit of shared code in packages under /shared-go

/shared-go is named the way it is because I intended to build one of the services in another language. I didn't but left the name the way it was.

Type Registration

Commands, Events, Snapshots, and other serializable entities and value objects are registered in groups in each /"domain"/internal/domain/register_types.go and in the child packages of serviceapis. This type registration is a feature of edat/core and is not unique to this application.

The purpose of doing this type registration is to avoid boilerplate marshalling and unmarshalling code for every struct.

Changes from FTGO

I intend for this demonstration to exist as a faithful Golang recreation of the original. If a difference exists either because of opinion or is necessary due of the particulars of Go, I will try my best to include them all here.

Changed

  • I've kept most API requests and responses the same "shape" but routes are prefixed with /api and use snake_case instead of camelCase for property names.
  • In FTGO many apis and messages that operated on Tickets used the OrderID as the TicketID. I could have done the same but chose to let the Ticket aggregates use their own IDs. The TicketID was then included in responses and messages where it was needed.
  • Order-History is not using DynamoDB. The purpose of Order-History is to provide a "view" or "query" service and it should demonstrate using infrastructure best suited for that purpose. For now, I'm using Postgres but intend to use Elasticsearch soon.
  • The OrderService->createOrder method I felt was doing too much. The command implementation creates the order like before, but the published entity event that results from that command is now the catalyst for starting the CreateOrderSaga.

Missing

  • Tests. Examples of testing these services. Both Unit and Integration
  • Api-Gateway. I haven't gotten around to creating the gateway.

Out Of Scope

Just like the original the following are outside the scope of the demonstration.

  • Logins & Authentication
  • Accounts & Authorization
  • AWS/Azure/GCP or any other cloud deployment instructions or scripts
  • Tuning guidance
  • CI/CD guidance
  • Chaos testing - although feel free to terminate a service or cdc process for a bit and see if it breaks (it shouldn't)

Quick Demo

Postman can be used to load the api collection for an easy demo.

With the application running using one of the commands above you can make the following calls to see the processes, events, and entities involved.

  • Consumer Service: Register Consumer
  • Restaurant Service: Create Restaurant
  • Order: Create Order

Loading FTGOGO.postman_collection.json into Postman will provide pre-built calls with semi-random data that can be used to test the above.

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please make sure to update tests as appropriate.

No warranties

From time to time I expect to make improvements that may be breaking. I provide no expectation that local copies of this demonstration application won't be broken after fetching any new commit(s). If it does fail to run; simply remove the related docker volumes and re-run the demonstration.

Issues
  • BDD tests using Ubiquitous Language

    BDD tests using Ubiquitous Language

    A lovely :heart: project you have created here! I knew about Wild Workouts, but this gives even more detail, fills more blanks. Thank you!

    I saw that there are no tests, and this is an example app after all. But one cool concept that is on-topic with the whole DDD theme, is to provide BDD tests and - if possible - formulate them according to the Ubiquitous Language (and understandable for non-technical stakeholders).

    Example:

      Feature: Select community member role
    
        As an organizer who owns and facilitates a community,
        I want to be able to change member roles,
        So that I can select other organizers to facilitate with me.
    
      Background:
        Given the community is active,
        And the organizer is the community owner.
    
        # In the MVP community ownership cannot be changed. The
        # creator is the owner.
    
        Scenario: Change membership role settings
          Given the list of community members,
          And changing members' role settings,
          When I save the role changes,
          Then campaigners can be promoted to organizers,
          And organizers can be demoted to campaigners.
    

    An example project that uses BDD in Go (though for a bit more technical features) is https://github.com/trustbloc/orb

    (OT: Another thing very cool to test-drive in Go is functional domains. See Domain Modeling made Functional by Scott Wlaschin)

    documentation 
    opened by aschrijver 3
  • Feedback and ideas

    Feedback and ideas

    Hi @stackus

    Love what you are doing with this demo app. Great to see all the BDD feature tests you added. I wanted to post some bits of feedback and some additional ideas that may inspire.

    Feedback

    Note: This is merely an observation. I don't have deep insight in all the ins and outs, design decision and possible alternative.

    Each of the sub-domains / bounded contexts is conveniently packaged in its own Go module. Except that, should I add a command / event etc. there's need to update serviceapis, and the domain is not self-contained but depends on this. In an application I intend to develop there are no microservices, but a plugin architecture based on HashiCorp go-plugin, which would allow dynamic addition of plugins at runtime (e.g. downloaded from a Marketplace of 3rd-party components). The serviceapis setup would not work in my case then, and I feel that in a microservices environment with, say, distributed remote teams working on them, this would be inconvenient too.

    Ideas

    There is a very interesting addition to DDD/CQRS/ES that may significantly reduce overall complexity, and that is: adding an Actor Model to the design. One of the original devs behind Akka has created a Golang-based actor framework named Proto Actor.

    With actors part of the design you can delegate all kinds of complex things, like location transparency, guaranteed message delivery, while increasing the overall resilience of the application. There are basically 2 ways to add to an existing DDD design:

    1. DDD Aggregates are actors (Vaughn Vernon's approach)
    2. Actors in application layer load, cache and invoke aggregates (Alexey Zimarev's approach)

    The second approach looks most appealing to me. The domain stays fully self-contained, and once you've retrieved / hydrated an aggregate (slow) it is cached within the actor / memory for the next use (fast).

    Note that Alexey is involved with proto.actor. In this video he explains his approach.


    Finally there's an interesting 2-part video that demonstrates Event Storming for an OAuth2 implementation in Go. The event storming is imho a great, maybe obligatory tool at the start and during DDD projects. The DDD process (strategic design) is as important as the resulting code (elaborated via tactical design). It is here where you keep the (non-technical) customer most likely on the same page as the dev team and vice versa.

    Maybe there's something interesting on the documentation side you can borrow from these video's, and maybe even the OAuth2 specific example can play a role in ftgogo, idk. Here are the code and videos:

    opened by aschrijver 0
Releases(v0.0.1)
Owner
Michael Stack
Michael Stack
Source code related to an on-site demonstration (for Ardan Labs) of packaging Go applications with Nix

Ardan Labs Nix Demo High-Level Overview We bumbled the scheduling of an earlier presentation about git worktree that a few co-workers attended. In eff

John Rinehart 4 Apr 30, 2022
Fast, intuitive, and powerful configuration-driven engine for faster and easier REST development

aicra is a lightweight and idiomatic configuration-driven engine for building REST services. It's especially good at helping you write large APIs that remain maintainable as your project grows.

xdrm-brackets 7 Jul 2, 2022
This is an example to demonstrate implementation golang microservices using domain driven design principles and sugestions from go-kit

go-kit DDD Domain Driven Design is prevelent and rising standard for organizing your microservice code. This design architecture emphasis on Code orga

Sourabh Mandal 1 Feb 9, 2022
Pact: a consumer-driven contract testing framework

CDC Pact Pact is a consumer-driven contract testing framework. Born out of a microservices boom, Pact was created to solve the problem of integration

A.Samet İleri 48 Jun 2, 2022
This example showcases an event-sourced CQRS system based on github.com/romshark/eventlog

Eventlog Example This example is showcasing an eventually consistent, fault-tolerant, event sourced system following the CQRS (Command-Query-Responsib

Roman Sharkov 3 Mar 13, 2022
CRUD API server of Clean Architecture with Go(Echo), Gorm, MySQL, Docker and Swagger

CRUD API Server of Clean Architecture Go(echo) gorm mysql docker swagger build docker-compose up -d --build API Postman and Fiddler is recommended to

null 303 May 30, 2021
Demo Fully Isolated System Architecture

Fully Isolated System Architecture (Microservices) Arsitektur Request | | | Api Gateway --- Auth Provider |\________________________

Muhamad Surya Iksanudin 28 Jul 9, 2022
Hexagonal Architecture implemented in GO

MSGO - Microservices in Go Hexagonal Architecture (Ports & Adapter) Wiki: Hexagonal Architecture Architecture benefits: loosely coupled interchangeabl

Vincent Villaluna 38 Jul 26, 2022
🚀 link shortener which is using hexagonal architecture in Go

?? go-link-shortener-hexagonal Link Shortener which is using Hexagonal Architecture that compliant with clean code guidelines in Go. ├── README.md ├──

Ali Baran Eser 7 Feb 26, 2022
This is a template service for development first with go monolithic architecture

This is a template service for development first with go monolithic architecture

Muhammad Rivaldy 1 Oct 28, 2021
This is demo / sample / example project using microservices architecture for Online Food Delivery App.

Microservices This is demo / sample / example project using microservices architecture for Online Food Delivery App. Architecture Services menu-servic

Nurali Virani 0 Nov 10, 2021
A pastebin clone implemented as microservice architecture.

pastebean Implementing a pastebin clone as a microservice architecture. Written using go-gin and gorm alongwith many other awesome open source Go liba

Rudraksh Pareek 3 Jan 24, 2022
Hexagonal architecture bank account example

Hexagonal architecture bank account example Esse repositório contém um pouco dos meus estudos sobre arquitetura hexagonal. Go Lang é a linguagem utili

João Paulo Vanzuita 8 Jan 8, 2022
The microservices have a loosely coupled architecture design

ETI_Assignment1 5.1.3.1 Design consideration The microservices have a loosely coupled architecture design. The microservices created include Passenger

null 0 Dec 12, 2021
Hands-on Labs on Microservices Architecture

Giới thiệu Khóa học Building Distributed Applications with Microservices sẽ giúp bạn tìm hiểu nhanh chóng về công nghệ Microservices để ứng dụng xây c

null 6 Jul 17, 2022
Go-rifa-microservice - Clean Architecture template for Golang services

Test CI Go Clean template Clean Architecture template for Golang services Overvi

Evandro Martinelli 1 Jul 14, 2022
Assignment2 - A shared project making use of microservice architecture

This project is a shared project making use of microservice architecture, API's and a simple frontend to implement a start-up new concept called EduFi. The concept combines education and financial systems to create profit from studying.

Gabriel Goh 1 Jan 26, 2022
Microservice - A sample architecture of a microservice in go

#microservice Folder structure required. service certs config config.yaml loggin

null 0 Feb 3, 2022
Go-hexagonal - Go Hexagonal architecture implemented with interfaces, tests and mockgen

Go Hexagonal Go Hexagonal architecture implemented with interfaces, tests and mo

Matheus Araujo 1 Jan 22, 2022