goTempM is a full stack Golang microservices sample application built on top of the Micro platform.

Overview

goTempM

Lint

goTempM is a full stack Golang microservices sample application built on top of the Micro platform. Note that the original goTemp, which is built directly on the go-Micro framework, is still available and can be found in the goTemp repo. It is worth noting that Micro itself uses go-Micro as its underlying framework.

In it current incarnation (this is wip), this mono-repo uses the following stack as backend:

  • Golang as its main implementation technology
  • Micro as the microservices platform running our services
  • gRPC for inter-service real time communication
  • NATS for Pub/Sub event driven communication
  • multicast DNS for service registration and discovery
  • PostgreSql for transactional data storage
  • TimescaleDB time series DB used for historical audit data storage
  • ArangoDBis a multi-model database used for master data storage
  • Redis is used to cache data and reduce number of data requests to other services
  • Docker for creating application images
  • Docker-compose to run the application
  • Minikube to run the application in Kubernetes

In terms of the web front end, the stack is as follows:

  • Javascript as its main implementation technology
  • Svelte is used as the compilation engine (via rollup)
  • Sapper is the javascript framework
  • Sveltestrap provides the css framework and is based on bootstrap
  • Font Awesome to display icons in the application

Below is a diagram that displays the overall setup of the application:

Diagram showing goTempM components

In a nutshell. the application functionality is as follows in the backend:

  • The frontend connects to the different services through the Micro API gateway
  • For each service the frontend provides:
    • Search page
    • Detail page
  • Additionally, the frontend provides:
    • Landing page
    • Login page
    • Register page
  • Each service performs the basic CRUD operations to their underlying databases
  • All services authenticate via the user service
  • All completed CUD operations are forwarded to the NATS broker which in turn forwards the message to the auditing service. This service saves the data into TimescaleDB.
  • Each service has a client which can be used to test all basic CRUD functionality

Starting the application

Note: At this time, starting multiple services in a row (which the startup scripts do) sometimes causes an error to be thrown. The details of the issue are described in issue 8. As such, portions of the startup scripts may have to be re-run manually to bring the application up fully.

goTempM landing page

Before running the application the first time:

  • Clone the repository
  • cd into gotempM/web/sapper
  • Run the following command to generate the javascript dependencies.
    npm install

Running with the Micro Docker image

Prerequisites
Software
  • Docker must be installed and running

Running the app

To start the application:

  • Ensure that Docker is installed and running. Then, execute the following command from a terminal in the goTempM root folder:
   make microup

Depending on whether you have run the application before, docker may have to download all the dependent images (Micro, PostgreSql, TimescaleDB, Nodejs, etc). This may take a while depending on your internet connection speed. Once everything has been downloaded and started, you should see a message in the terminal indicating that the application is listening at localhost:3000. At that point, yo can open your browser and navigate to:

    http://localhost:3000

To stop the application:

    make microdown

Running on Kubernetes (Minikube) using the Micro Helm chart

Prerequisites
Software
  • Ensure Micro is installed on the host
  • Docker must be installed and running
  • Ensure that Minikube is installed and running
  • Similarly, ensure that Helm is installed on the host.
  • Install the Micro Helm Chart
Ingress

The application front end connects with the API gateway using via a K8s ingress resource. As such, the ingress addon must be enabled in Minikube. To enabled it, run:

    minikube addons enable ingress

Check the ingress is working using the command below. The command's results should include an entry for the ingress.

    kubectl get pods -n kube-system
Building and pushing images (optional)

Out of the box, the Kubernetes manifest will pull existing Bolbeck goTemp images for the front end and some of the DBs from Docker Hub. You are welcome to change the Kubernetes manifests in the ./cicd/K8s folder to pull your own images. To build your own image of the front end and push it to docker hub run the command below for each of the services:

    make hubpushcontext SERVICE=web FOLDER=web

where serviceName is the name of the service for which the image should be built folderName is the folder that contains the docker file used to build the service image

Note that Micro will automagically create and deploy the containers for the different services

Running

Enable Port forwarding:

    kubectl port-forward svc/proxy -n micro 8081:443

Note that the application itself does not need this port forwarding, this is only needed so that we can send commands to Micro and start the app.

Start application by running:

    make microk8sup

Once the application is deployed, check the address and host assigned to the ingress:

    kubectl get ingress -n micro

Note that it takes a couple of minutes for K8s to assign the IP to the ingress. As such wait for that happens before moving ahead.

If this is the first time running the app in Minikube: Grab the address & the host from the result of the command above, and add it to your /etc/hosts file.

Finally, access app:

     minikube service web -n micro

Stop the application:

    make microK8sdown

Note: you will need stop the port forwarding as well

Running with Micro locally

Prerequisites

Software
  • Ensure Micro is installed
  • Docker must be installed and running

Running

Start the micro server:

    micro server

Wait for the server to be up and ensure that you can access localhost:8080 before continuing to next step. Start the application:

    make microlocalup

Access the application in your browser at:

    http://localhost:3000

To stop the application, run:

    make microlocaldown

Repo organization

The project is organized in a way that each folder represents either a service, a database or a shared library package. Currently, we have the following:

  • arangodb: Volumes mounted to the ArangoDB container as well as data initialization scripts
  • audit: Audit service to collect and store historical audit information
  • cicd : Holds files related to CI/CD and orchestration
  • customer: Customer master data service
  • diagramforDocs: Diagrams used in the readme documents
  • globalCache : Enables Micro to use Redis as a cache store which can then be used in our services
  • globalErrors: Generic errors shared package
  • globalProtos: Generic protobuf message definitions shared across packages
  • globalUtils: Generic utilities shared package
  • micro : Hosts the custom Dockerfile that is used to run Micro.
  • nats: NATS dockerfile and configuration
  • postgres: Volumes mounted to the PostgreSQL DB container as well as data initialization scripts
  • product: Product master data service
  • promotion: Promotion service to track product discounts (this was the first service built)
  • redis: Volumes mounted on the redis container as well as config files (if any)
  • timescaleDB: Volumes mounted to the Timescale DB container as well as data initialization scripts
  • user: User and authentication service
  • web: application web frontend

Additionally, we have the following files in the root directory as well:

  • .dockerignore: Files to be ignored when building service images
  • .gitignore: Files to be ignored by git
  • docker-compose: File controls the building of the different services and their dependencies
  • go.mod and go.sum: Go modules control
  • main.go: Not used for services yet
  • Makefile: shortcuts to common actions
  • Readme.md: Well... this file...

Services

We use Micro as the main GO microservices platform. Using Micro simplifies many of the tasks associated with building microservices including (but not limited to):

  • Service discovery
  • gRPC for inter service communication
  • Built in async messaging (in our case used to set up pub/sub messages to NATS )
  • Built-in data storage interface (in our case used to interact with Redis)
  • API gateway:
    • Request routing
    • Load balancing
    • Automatic conversion of frontend JSON payloads to backend gRPC messages
Organization

Each one of the services has a similar structure:

  • client: Contains a client service that calls the server service to perform multiple operations
  • proto: Proto buffer messages and services definitions. Empty if service does not handle real time inter-service communication.
  • server: Service that performs a number of actions like interacting with the DB
  • Dockerfile: Build the image for the server service
  • DockerfileCLI: Build the image of the client service
  • docker-compose.env: Environment variable required to run the service when running the service with docker-compose
  • docker-compose-cli.env: Environment variable required to run the client when running the client with docker-compose
Running individual services
Starting the service

The services must be started using Micro run. Since we are using the Dockerized version of Micro, we can start a service as follows:

If Micro is not already running:

make microserveup

This will start Micro as well as all the DBs used by goTempM. Get into the Micro container, if not in it already:

docker exec -it microservercont bash

Start the service:

make micro

where can be usersrv, auditsrv, customersrv, productsrv or promotionsrv.

Testing the service

Note: all the commands below must be run within the Micro container

To run some data through a service once it is started, we can run the service client:

micro run --name /client

where can be usercli, auditcli, customercli, productcli or promotioncli. For example, we could start the user client as follows:

micro run --name usercli user/client

This will bring up run the client service which will attempt to create,update and delete a user. The results will be printed to the log. To see the logs, run:

micro logs -f

The server user service will update the DB as necessary and send the updated information to the broker (NATS) so that the audit service may eventually store it in the time series DB. The audit service can be started using:

make microauditsrv

Databases Initialization

The project initializes each of the DBs and seeds them with tables and data. Data changes made at run time are automatically persisted using mounted volumes when running via docker-compose. See the folders for each DB for details as well as the docker-compose file.

Web front end

Our web front end is built with Svelte and Sapper which have some interesting benefits:

  • Server-side initial rendering of our pages
  • File based routing
  • Smaller code base than other Javascript frameworks. Does more with less.
  • Svelte translates the code to vanilla javascript. Thus, smaller application footprint than most frameworks
  • Emphasis on component re-usability
Organization

The web application lives in the ./web folder. Since Sapper and Svelte generate multiple files and folders, we will just discuss the relevant folders below:

  • sapper: The main folder containing the web app
    • src: This is where the bulk of the application resides
      • components: Contains re-usable Svelte components
      • globalUtils: Shared javascript utilities
      • routes: application routes to the different pages
      • client.js: This is a required file. It is used to start Sapper.
      • server.js: Used to configure the app with items like middleware and compression
      • template.html: Main page that contains our application. We added Bootstrap and Font Awesome CDN references in this page.
    • static: Holds static items
  • Dockerfile: Used to build the docker image for the web app

Routes

All of our main routes are pretty standard in terms of organization. We will use the customer route (./web/sapper/src/routes/customer) as an example:

  • index.svelte: Main customers search page that serves at localhost:3000/customer
  • _searchGridSlot: Component holds the template for the search grid to be display in hte search page (index.svelte)
  • new.svelte: Page to be displayed when user want to create a new customer. Displayed at localhost:3000/customer/new .
  • [slug].svelte: Page to view and modify existing customers. Displayed at localhost:3000/customer/[customerid]
  • _detail.svelte: Holds the gui and bulk of the logic for adding or editing customers. It is called by new.svelte and [slug].svelte .

There are three routes that do not share the structure above as they have very little functionality and thus are server by a single index.svelte component: root, register and login.

Kubernetes

The application configuration in K8s can be seen in the diagram below. Note that the diagram shows just one of the different microservices and its associated database. The configuration for all other microservices, beyond the shared ingress and API Gateway, is similar to the one depicted in the diagram. Note that Micro builds and spins out the service pods.

Diagram showing goTempM components

Organization

The K8s files live in the ./cicd/K8s folder, and is organized as follows:

  • dbsAndBroker: Contains the manifests for all the databases and for the broker.
  • Ingress: Manifest to create the ingress resource that allows the front end and the back end to communicate
  • web: Manifest for the web front end.

Note that within each of the folders, most related manifests are organized using a prefix. For example, all the front end related services start with the 'web' prefix.

Additional information:

Additional information can be found in the individual folders either in a readme.md or a doc.go file. Additionally, the Makefile contains many command samples that can be used for development.

Issues
  • Rpc error: code = Canceled desc = grpc: the client connection is closing

    Rpc error: code = Canceled desc = grpc: the client connection is closing

    Issue description:

    When starting multiple services one after the other using micro run, many times the following error is thrown: Rpc error: code = Canceled desc = grpc: the client connection is closing One can just re-run the command and the service comes up. This is specially visible when running in docker or K8s, but also happens locally ‌

    Steps to reproduce:

    Try to start all services with the command below, which will start all the services one after the other make microstartsrvs or if running inside the container :

    bug 
    opened by camba1 0
  • Improve startup script

    Improve startup script

    The startup script currently has sleep commands between services startups to try to avoid Micro throwing a gRPC timeout error. Need to find a better way to do this. Currently if Micro does throw the error, one can just rerun the startup script. Not the best situation though.

    enhancement 
    opened by camba1 0
  • user registration needs its own endpoint

    user registration needs its own endpoint

    Currently the user registration shares the same end point with the regular user save when saving the data. However that means that the user save does not check for a token. That in turn causes the user service to not send new user record to the audit service since the audit service gets uses the token to identify which user performed the action. While there are multiple ways to fix this, the cleanest is to have a separate user service end point that handles anonymous registration separately

    enhancement 
    opened by camba1 0
  • User is missing the modified by field

    User is missing the modified by field

    The user in the userSrv does not have the standard ‘modifiedBy’ field. The createDate field is incorrectly set to createdate and updateDate is incorrectly set to updatedate

    enhancement 
    opened by camba1 0
  • Standardize the way validity dates are sent out from the services

    Standardize the way validity dates are sent out from the services

    Currently, depending on wether the DB is SQL or No SQL, the validity dates and the modification fields are been sent out to the client either inline or as part of an object within the dataset. This is due to how the fields are mapped in the porto definition. When creating the services we switched midway the way we handled those fields in the porto definitions to use imported global definitions but never went back to change it in the promotion and customer services. Bottom line, all services should handle those fields the same way It is done in the product and customer services, independent of how they are stored in the DB

    enhancement 
    opened by camba1 0
Releases(v0.6.0)
  • v0.6.0(Apr 10, 2021)

    Release notes

    • Integration with Prometheus to scrape metrics from the microservices, databases and broker
    • Integration with Grafana to display metrics scraped by Prometheus
    • Addition of default dashboards in Grafana
    • Change target names in Docker and Docker Compose files to be all lower case (docker cli issue #1414 )
    Source code(tar.gz)
    Source code(zip)
  • v0.5.1(Mar 10, 2021)

    Release notes

    • Update to micro v3.1.1 which includes among other things micro update instead of having to do micro kill and micro start every time.
    • Update to Go 1.16
    • Removed old dependency on jackc/pgx from audit service and consolidated on jackc/pgx/v4 for all services
    • App is now tested on Go 1.16, minikube v 1.18 and Kubernetes v1.20.2
    • Reduced cpu request for Vault injected init containers when running app integrated with Vault in K8s
    Source code(tar.gz)
    Source code(zip)
  • v0.5.0(Mar 4, 2021)

    Release notes

    I have not published a new release for a while, so figured it is a good time for an update. Since release 0.4.0 (several months ago) and in descending order of implementation from newest to oldest:

    • Optional integration with Hashicorp Vault when running in Kubernetes for static credentials management (Feb 2021). See ./vault for details
    • Migration from go-Micro to Micro
    • Setup application to run in Kubernetes
    • Creation of Svelte based front end
    Source code(tar.gz)
    Source code(zip)
Owner
Independent solution architect, project manager, developer and everything in between. Changing the world, one semi-colon at a time
null
Elastic Stack Docker + Sample Go AppElastic Stack Docker + Sample Go App

?? Elastic Stack Docker + Sample Go App Test Elastic Stack which includes Elasticsearch, Kibana, Filebeat and Metricbeat. It comes with a very simple

Ruben Delgado 0 Jan 14, 2022
micro-draft-manager is a microservice that helps you to manage unstructured data in your application with sorting and full-text search

micro-draft-manager is a microservice that helps you to manage unstructured data in your application with sorting and full-text search. For example, y

Hamed Abdollahpour 1 Nov 24, 2021
Sample cloud-native application with 10 microservices showcasing Kubernetes, Istio, gRPC and OpenCensus.

Online Boutique is a cloud-native microservices demo application. Online Boutique consists of a 10-tier microservices application. The application is

Google Cloud Platform 12.1k May 12, 2022
Create production ready microservices mono repo pattern wired with Neo4j. Microservices for other languages and front end repos to be added as well in future.

Create Microservices MonoRepo in GO/Python Create a new production-ready project with backend (Golang), (Python) by running one CLI command. Focus on

GoChronicles 13 Dec 31, 2021
Go-kit-microservices - Example microservices implemented with Go Kit

Go Kit Microservices Example microservices implemented with go kit, a programmin

Hao-Ming, Hsu 1 Jan 18, 2022
This is demo / sample / example project using microservices architecture for Online Food Delivery App.

Microservices This is demo / sample / example project using microservices architecture for Online Food Delivery App. Architecture Services menu-servic

Nurali Virani 0 Nov 10, 2021
Micro is a platform for cloud native development

Micro Overview Micro addresses the key requirements for building services in the cloud. It leverages the microservices architecture pattern and provid

Micro 11.1k May 15, 2022
A Micro-UTP, plug-able sanity checker for any on-prem JFrog platform instance

hello-frog About this plugin This plugin is a template and a functioning example for a basic JFrog CLI plugin. This README shows the expected structur

rdar 0 Dec 7, 2021
Study Project for the application of micro services and requisition controls

Starting Endpoint GO with Retry Request Install GoLang for Linux Tutorial: LINK

Antenor Pires 3 Dec 29, 2021
Istio - An open platform to connect, manage, and secure microservices

Istio An open platform to connect, manage, and secure microservices. For in-dept

null 0 Jan 5, 2022
An open platform to connect, manage, and secure microservices.

Istio An open platform to connect, manage, and secure microservices. For in-depth information about how to use Istio, visit istio.io To ask questions

Baalaji 0 Feb 6, 2022
Online Boutique: a cloud-native microservices demo application

This is an application forked from the GCP Microservice demo. Online Boutique is

null 0 Jan 5, 2022
GoLang utility packages to assist with the development of web micro-services.

GoTil Golang utility packages to assist with the development of web micro-services. Installation As a library. go get github.com/ccthomas/gotil Usage

Christopher Thomas 0 Nov 26, 2021
Micro-serviço em Golang para processar pagamentos de um Gateway.

Go Payment Processor Projeto Este repositório contém um micro-serviço em Golang que faz parte de um conjunto de serviços necessário para o projeto do

Daniel Hessel 0 Dec 13, 2021
Go Micro is a framework for distributed systems development

Go Micro Go Micro is a framework for distributed systems development. Overview Go Micro provides the core requirements for distributed systems develop

Asim Aslam 18.2k May 15, 2022
Micro-service framework in Go

Kite Micro-Service Framework Kite is a framework for developing micro-services in Go. Kite is both the name of the framework and the micro-service tha

Koding, Inc. 3.1k May 15, 2022
NewSQL distributed storage database based on micro service framework

QLite 是基于微服务的 NewSQL 型数据库系统,与传统的一体化数据库不同,该系统将本该内置的多种数据结构(STL)拆分成多个服务模块,每个模块都是独立的一个节点,每个节点都与其主网关进行连接,从而形成分布式存储结构。

null 30 Mar 25, 2022
A code generator that turns plain old Go services into RPC-enabled (micro)services with robust HTTP APIs.

Frodo is a code generator and runtime library that helps you write RPC-enabled (micro) services and APIs.

Monadic 17 Apr 21, 2022
Automatic Service Mesh and RPC generation for Go micro services, it's a humble alternative to gRPC with Istio.

Mesh RPC MeshRPC provides automatic Service Mesh and RPC generation for Go micro services, it's a humble alternative to gRPC with Istio. In a nutshell

AstraNet Toolkit 68 Apr 19, 2022