A Realtime API Gateway used with NATS to build REST, real time, and RPC APIs, where all your clients are synchronized seamlessly.

Overview

Resgate logo

Realtime API Gateway
Synchronize Your Clients

License Report Card Build Status Coverage

Visit Resgate.io for guides, live demos, and resources.


Resgate is a Go project implementing a realtime API gateway for the RES protocol with NATS server as messaging system.

It is a simple server that lets you create REST, real time, and RPC APIs, where all your clients are synchronized seamlessly.

Used for building new REST APIs with real-time functionality, or when creating single page applications using reactive frameworks such as React, Vue.js, or Modapp.

Book Collection Animation
Screen capture from the Book Collection Example. Try out the Live demo version yourself.

How it works

Resgate handles all API requests from your clients, instead of directly exposing your micro-services (represented by Node.js and Java below). Clients will connect to Resgate, using either HTTP or WebSocket, to make requests. These requests are sent to the micro-services over NATS server, and Resgate will keep track on which resource each client has requested.

Whenever there is a change to the data, the responsible micro-service sends an event. Resgate will use this event to both update its own cache, and make sure each subscribing client is kept up-to-date.

RES network diagram

Quickstart

If you install Docker, it is easy to run both NATS server and Resgate as containers:

docker network create res
docker run -d --name nats -p 4222:4222 --net res nats
docker run --name resgate -p 8080:8080 --net res resgateio/resgate --nats nats://nats:4222

Both images are small, about 10 MB each.

See Resgate.io - Installation for other ways of installation.

Examples

While Resgate may be used with any language, the examples in this repository are written in Javascript for Node.js, without using any additional library.

Example Description
Hello World Simple service serving a static message.
Edit Text Text field that can be edited by multiple clients concurrently.
Book Collection List of book titles & authors that can be edited by many.
JWT Authentication Showing how JWT tokens can be used for authentication.
Password Authentication Showing authentication with user and password credentials.
Client Session Creating client sessions that survive reloads and reconnects.

Note

All examples are complete with both service and client.

Protocol Specification

For more in depth information on the protocol:

Usage

resgate [options]

Server options

Option Description Default value
-n, --nats <url> NATS Server URL nats://127.0.0.1:4222
-i, --addr <host> Bind to HOST address 0.0.0.0
-p, --port <port> HTTP port for client connections 8080
-w, --wspath <path> WebSocket path for clients /
-a, --apipath <path> Web resource path for clients /api/
-r, --reqtimeout <seconds> Timeout duration for NATS requests 3000
-u, --headauth <method> Resource method for header authentication
--tls Enable TLS for HTTP false
--tlscert <file> HTTP server certificate file
--tlskey <file> Private key for HTTP server certificate
--apiencoding <type> Encoding for web resources: json, jsonflat json
--creds <file> NATS User Credentials file
--alloworigin <origin> Allowed origin(s): *, or <scheme>://<hostname>[:<port>] *
--putmethod <methodName> Call method name mapped to HTTP PUT requests
--deletemethod <methodName> Call method name mapped to HTTP DELETE requests
--patchmethod <methodName> Call method name mapped to HTTP PATCH requests
-c, --config <file> Configuration file in JSON format

Logging options

Option Description
-D, --debug Enable debugging output
-V, --trace Enable trace logging
-DV Debug and trace

Common options

Option Description
-h, --help Show usage message
-v, --version Show version

Configuration

Configuration is a JSON encoded file. If no config file is found at the given path, a new file will be created with default values as follows.

Properties

{
    // URL to the NATS server.
    "natsUrl": "nats://127.0.0.1:4222",
    // NATS User Credentials file path.
    // Eg. "ngs.creds"
    "natsCreds": null,
    // Timeout in milliseconds for NATS requests
    "requestTimeout": 3000,
    // Bind to HOST IPv4 or IPv6 address.
    // Empty string ("") means all IPv4 and IPv6 addresses.
    // Invalid or missing IP address defaults to 0.0.0.0.
    "addr": "0.0.0.0",
    // Port for the http server to listen on.
    // If the port value is missing or 0, standard http(s) port is used.
    "port": 8080,
    // Path for accessing the RES API WebSocket.
    "wsPath": "/",
    // Path prefix for accessing web resources.
    "apiPath": "/api",
    // Encoding for web resources.
    // Available encodings are:
    // * json - JSON encoding with resource reference meta data.
    // * jsonflat - JSON encoding without resource reference meta data.
    "apiEncoding": "json",
    // Flag enabling WebSocket per message compression (RFC 7692).
    "wsCompression": false,
    // Call method name to map HTTP PUT method requests to.
    // Eg. "put"
    "putMethod": null,
    // Call method name to map HTTP DELETE method requests to.
    // Eg. "delete"
    "deleteMethod": null,
    // Call method name to map HTTP PATCH method requests to.
    // Eg. "patch"
    "patchMethod": null,
    // Header authentication resource method for web resources.
    // Prior to accessing the resource, this resource method will be
    // called, allowing an auth service to set a token using
    // information such as the request headers.
    // Missing value or null will disable header authentication.
    // Eg. "authService.headerLogin"
    "headerAuth": null,
    // Flag enabling tls encryption.
    "tls": false,
    // Certificate file path for tls encryption.
    "tlsCert": "",
    // Key file path for tls encryption.
    "tlsKey": "",
    // Allowed origin for CORS requests, or * to allow all origins.
    // Multiple origins are separated by semicolon.
    // Eg. "https://example.com;https://api.example.com"
    "allowOrigin": "*",
    // Flag enabling debug logging.
    "debug": false,
    // Flag enabling trace logging.
    "trace": false
}

Running Resgate

By design, Resgate will exit if it fails to connect to the NATS server, or if it loses the connection. This is to allow clients to try to reconnect to another Resgate instance and resume from there, and to give Resgate a fresh new start if something went wrong.

A simple bash script can keep it running:

#!/bin/bash
until ./resgate; do
    echo "Resgate exited with code $?.  Restarting.." >&2
    sleep 2
done

Documentation

Visit Resgate.io for documentation and resources.

It has guides on installation, configuration, writing services, scaling, queries, and other useful things. It also contains guides for ResClient when working with frameworks such as React, Vue.js, and Modapp.

Support Resgate

Resgate is an MIT-licensed open source project where development is made possible through community support.

If you'd like help out, please consider:

Contribution

Any feedback on the protocol and its implementation is highly appreciated!

If you find any issues with the protocol or the gateway, feel free to report them.

If you have created a service library, a client library, or some other tool or utility, please contact me to have it added to the list of resources.

Issues
  • Discussion: Microservices interacting with RES over http and websockets

    Discussion: Microservices interacting with RES over http and websockets

    Proble: The problem with NATS is that is does not support Multi-Tenancy. Its been talked about being fixed and its unlikely to be. It also does not support on the fly config.

    SO i was thinking that because there is a Client API for Res Clients to use, why cant there also be one for Microservices to use over HTTP and websockets ?

    Why ?

    1. Enterprise patterns such as Security, metrics, logging, tracing etc can all be done at the Res Gateway level.
    2. You can secure Microservices so that they can only access parts of the NATS namespace ( loosely using this but i guess you know what i mean ).
    opened by ghost 6
  • CORS - allowing content-type on headers

    CORS - allowing content-type on headers

    Greetings!

    I've been dealing with an application that has different servers for the frontend and its API. Resgate is the entrypoint for the API, and I was trying to do some REST requests from the frontend, but having trouble when setting the header "Content-Type: application/json", because resgate's answer didn't have the header "Access-Control-Allow-Headers: content-type". I added it to the code, and things started to work perfectly for me.

    If you think this change might bring value to the project, here's the PR: https://github.com/resgateio/resgate/pull/173

    opened by g-belmonte 6
  • How Microservice interface with RES

    How Microservice interface with RES

    Does the RES gateway support a typical Microservice interfacing with it over http ? As opposed to using the golang NATS client.

    If yes, does the Microservice need to support websockets ? I presume a definite yes.

    Additionally can a Microservice also use the golang NATS drivers instead. There are certain use cases where that prefer that.

    Also if I have a Microservice that needs to provide web GUI to the browser can the RES gateway be used for it ? I presume yes. It would mean I don't need a sperate reverse proxy & the security is all applied unifoly at the RES gateway level. It also means that I only have to run a single docker of my web service that also provides the web rendering.

    Lastly .. there is no lastly. That's alot of questions ..

    opened by ghost 6
  • Incompatible with nats:latest (nats ^2.1.0)

    Incompatible with nats:latest (nats ^2.1.0)

    Expected Behaviour

    As a user of resgate I am able to replicate the functionality of the jwt-authentication example

    Actual Behaviour

    The browser is not authenticated and gives the following error

     {"error":{"code":"system.accessDenied","message":"Access denied"},"id":3}
    

    Steps to reproduce

    Instrumentation:

    1. $ nats sub >
    • watches all nats messages being sent to the broker
    1. append --trace to Resgate docker run command

    Steps to reproduce:

    1. Bring up auth service (../jwt-authentication/authService.js) (yarn start)
    2. Comment out the two publish lines below
    3. Bring up this service (yarn run dev)
    4. Navigate to http://localhost:8084
    5. Login
    6. Go back
    7. Refresh the page
    8. Observe the terminal output of both services

    So the other service is the one handling these requests, it works as expected the browser displays hello world.

    Presumably we are doing the exact same thing in both services, as show by the terminal output...right?

    Let's make sure:

    1. Comment out the publish lines in the other service
    2. Uncomment the lines in this service
    3. Refresh the browser (do it twice, it doesn't always work first try)
    4. Observe how the request fails yet we have the "same" thing being sent as the other service.

    ¯_(ツ)_/¯

    ie:

    Original Service:

     [Auth] Received new message _INBOX.7Ap64yZ6OjkInVcCNbwmv2
     [Auth] Response 1:  {"token":{"foo":"bar","iat":1629071958}}
     [Auth] Response 2:  {"result":null}
     [Auth] Processed message
     [Example] Token is:  { foo: 'bar', iat: 1629071958 }
    

    This Service:

     [Auth 2.0] Received new message _INBOX.7Ap64yZ6OjkInVcCNbwmv2
     [Auth 2.0] Repsponse 1:  {"token":{"foo":"bar","iat":1629071958}}
     [Auth 2.0] Response 2:  {"result":null}
     [Auth 2.0] Processed message
     ... [Example] Token is:  null
    

    Yields:

     {"error":{"code":"system.accessDenied","message":"Access denied"},"id":3}
    

    Speculation

    I'm either doing something wildly wrong, because I swear the implementation is the same, or there's a version incompatibility with the latest version of nats.

    Perhaps it has something to do with the new string encoding that the nats client uses?

    I'm willing to make a PR, I just need some guidance.

    I submitted a PR and would like to contribute an example that works with the newest client. I think this project is awesome and it would solve a lot of pain points on our team if I could resolve this issue.

    opened by LucienBrule 6
  • Security

    Security

    Could you point me to a few links in resgate, client and server JS code to show me the security architecture.

    I want to try to I integrate Vault (https://www.vaultproject.io/) to make it really enterprise extensible. At least to see what sensible opportunity there is to make it easy and powerful.

    Vault has a Web GUI too written in golang which makes it easy to get the hang of: https://github.com/Caiyeon/goldfish I think that there is also a Web GUI built into Vault too, but i have not tried it.

    discussion 
    opened by ghost 6
  • Clarification needed on collection events

    Clarification needed on collection events

    It's a bit unclear to me how to use the collection events.

    If a new value is added, an add event is sent. If a value is deleted, remove event is sent. What event should be send if a value is replaced?

    I'm guessing that for a collection ["a", "b"], an add event with payload {"value": "x", "idx": 1} implies that "x" was inserted at position 1 to get the result ["a", "x", "b"]. Or does it imply that a replace with the result ["a", "x"] took place? If my guess is correct, what event do I send to denote that the value at index 1 was replaced?

    Please add some text that clarifies this.

    Side note: I would have expected an add to mean "new value was inserted at index (and all values from index was shifted right)" and a set event meaning "value at index was replaced".

    opened by thallgren 5
  • go client

    go client

    Hey @jirenius Congrats on getting onto the NATS blog :) The article is well written.

    I am wondering what happened to the go-client ? Cant find the repo for it.

    question 
    opened by ghost 4
  • Server not stopping with correct signals

    Server not stopping with correct signals

    I setup resgate to run in tandem with some other processes as part of my setup.

    Using go-cmd, and raised an issue here:

    https://github.com/go-cmd/cmd/issues/19#issuecomment-406827169

    The test code to reproduce I don't have accessible but the 20 line example code in go-cmd is the same and so very easy to try.

    It is important that spawned servers inside NATS terminate correctly in order for them to be well managed by the process manager. I have not tried resgate with systemd or launchd or windows service yet. I intend to try that next

    bug 
    opened by ghost 4
  • different project but the same vision

    different project but the same vision

    the same protocol maybe interested for you https://github.com/dunglas/mercure

    discussion 
    opened by hbakhtiyor 3
  • why not use Redis as the storage?

    why not use Redis as the storage?

    I personally use Redis as the backend storage.

    How to let the client listen to certain event based on the resource id without a cached resource?

    discussion 
    opened by akeyboardlife 3
  • Client connection tracking

    Client connection tracking

    Issue

    Services want to be able to track which clients are currently connected to any of the Resgates.

    Scope

    This feature should consider the following:

    1. Resgate events on client connects/disconnects
    2. Handling of Resgate restart or Resgate disconnect from NATS server
    3. Handling of service restart or service disconnect from NATS server

    Notes

    The tracking of client connections should only be done for WebSocket connections, and not HTTP requests. The short-lived connection created for HTTP requests would only produce unnecessary traffic of limited use.

    enhancement 
    opened by jirenius 4
  • Resource not sent on successive get request

    Resource not sent on successive get request

    Issue

    In case a client get request is immediately followed by another client get or subscribe request, while the first request is still collecting resources, the response for the second request may not contain the resources that was included in the response to the first request.

    Since any resource sent in a response to a client get request should not be considered subscribed, any successive get or subscribe request should resend those resources if needed.

    How to recreate

    Assume we have a resource, test.a, which has a resource reference to another resource, test.b.

    1. Client sends request: get.test.a
    2. Resgate sends NATS request: get.test.a
    3. Client sends request: get.test.b
    4. Resgate gets result from request 3)
    5. Resgate sends NATS request: get.test.b
    6. Resgate gets result from request 4)
    7. Resgate responds to request 1), including both test.a and test.b
    8. Resgate responds to request 3) with no resource data, wrongly assuming client already has test.a

    Impact

    The bug has no security impact, and low impact on usage as it only affects a get request corner case. In addition, get requests are currently not used by any known client. ResClient only uses subscribe requests to fetch data, and is therefor unaffected.

    bug 
    opened by jirenius 0
  • Support for NGS

    Support for NGS

    Issue

    Add support to connect Resgate to NGS.

    enhancement 
    opened by jirenius 4
  • Add logging

    Add logging

    Issue

    Similar logging capabilities that exists in NATS Server should be available in Resgate, to allow a unified way of logging activities in both servers.

    Related discussion

    https://github.com/jirenius/resgate/issues/34

    enhancement 
    opened by jirenius 0
  • Logging

    Logging

    So far, all the logging stuff has been ignored.

    The problem is that all the different golang log libraries don't share a common standard interface, so the solution would be to design a standard interface and then make an adapter for each one, similar to the way the SQL package has all the different database drivers, at the cost of not being able to use specific features to each logger package. That itself could be a big task. After a quick Google there seems to be an example which uses adapters approach here: https://goa.design/implement/logging/

    discussion 
    opened by ghost 3
Releases(v1.7.5)
Owner
Resgate.io - Synchronize Your Clients
Resgate is a project for creating microservice based REST and real time APIs where all clients are synchronized seamlessly.
Resgate.io - Synchronize Your Clients
The Durudex gateway combines all durudex services so that it can be used through a single gateway.

The Durudex gateway combines all durudex services so that it can be used through a single gateway.

null 7 Jan 6, 2022
A LoRaWAN nodes' and network simulator that works with a real LoRaWAN environment (such as Chirpstack) and equipped with a web interface for real-time interaction.

LWN Simulator A LoRaWAN nodes' simulator to simulate a LoRaWAN Network. Table of Contents General Info Requirements Installation General Info LWN Simu

ARSLab 19 Jan 14, 2022
Json to rpc example with envoy, go, grpc, nats

grpc-nats-envoy json to rpc example with envoy, go, grpc, redis This repo is a mirror of https://github.com/charlesonunze/grpc-redis-envoy-example It

Charles Onunze 1 Dec 7, 2021
List running processes that are acting as DCE/RPC servers or clients

rpcls This project was made to assist in a larger research project. It pulls from a running process' PEB to enumerate the loaded DLLs. If a process im

Alex Flores 3 Aug 22, 2021
Grpc-gateway-map-null - gRPC Gateway test using nullable values in map

Demonstrate gRPC gateway behavior with nullable values in maps Using grpc-gatewa

null 1 Jan 6, 2022
No-frills all-in-one HTTP API gateway

SX: a minimal, declarative API gateway WARNING: not production ready. Use at your own risk! Need something better? Check out nginx, Caddy or Envoy SX

Giorgio Pellero 1 Dec 18, 2021
Antenna RPC is an RPC protocol for distributed computing, it's based on QUIC and Colfer. its currently an WIP.

aRPC - Antenna Remote Procedure Call Antenna remote procedure call (aRPC) is an RPC protocol focused on distributed processing and HPC. aRPC is implem

Raphael de Carvalho Almeida 3 Jun 16, 2021
rpc/v2 support for JSON-RPC 2.0 Specification.

rpc rpc/v2 support for JSON-RPC 2.0 Specification. gorilla/rpc is a foundation for RPC over HTTP services, providing access to the exported methods of

High Performance, Kubernetes Native Object Storage 3 Jul 4, 2021
Go Substrate RPC Client (GSRPC)Go Substrate RPC Client (GSRPC)

Go Substrate RPC Client (GSRPC) Substrate RPC client in Go. It provides APIs and types around Polkadot and any Substrate-based chain RPC calls. This c

Chino Chang 1 Nov 11, 2021
All-in-one Network Gateway for Malware analysis

aio-gw [EXPERIMENTAL]: All-in-one Network Gateway for Malware analysis. currently at Alpha stage. HELP NEEDED: if you're keen to contribute to aio-gw,

Ali Mosajjal 1 Oct 15, 2021
Ephemeral One Time/Build-Time gRPC TLS PKI system.

PkiSauce Ephemeral Build Time TLS PKI saucing for your intra services GRPC (or not) communications. Description A simple attempt to avoid deploying co

Ermites.IO 3 Dec 14, 2021
Transparent TLS and HTTP proxy serve and operate on all 65535 ports, with domain regex whitelist and rest api control

goshkan Transparent TLS and HTTP proxy serve & operating on all 65535 ports, with domain regex whitelist and rest api control tls and http on same por

Sina Ghaderi 7 Dec 26, 2021
PinGo is a standalone and feature-rich tool for common IP-based reachability checking tasks. Ping or Trace and Observe in real-time the statistics.

pingo As a network champion from designing and implementing to troubleshooting large scale networks - I know that is usually not easy for administrato

Jerome Amon 1 Dec 18, 2021
Fast and idiomatic client-driven REST APIs.

Vulcain is a brand new protocol using HTTP/2 Server Push to create fast and idiomatic client-driven REST APIs. An open source gateway server which you

Kévin Dunglas 3.2k Jan 14, 2022
Generate HTTP load and plot the results in real-time

ali A load testing tool capable of performing real-time analysis, inspired by vegeta and jplot. ali comes with an embedded terminal-based UI where you

Ryo Nakao 3.1k Jan 20, 2022
the pluto is a gateway new time, high performance, high stable, high availability, easy to use

pluto the pluto is a gateway new time, high performance, high stable, high availability, easy to use Acknowledgments thanks nbio for providing low lev

mobus 2 Sep 19, 2021
A Wireguard VPN Server Manager and API to add and remove clients

Wireguard Manager And API A manager and API to add, remove clients as well as other features such as an auto reapplier which deletes and adds back a c

null 46 Dec 31, 2021