Fast and idiomatic client-driven REST APIs.

Overview

Vulcain: Use HTTP/2 Server Push to create fast and idiomatic client-driven REST APIs

Vulcain is a brand new protocol using HTTP/2 Server Push to create fast and idiomatic client-driven REST APIs.

An open source gateway server which you can put on top of any existing web API to instantly turn it into a Vulcain-compatible one is also provided!

It supports hypermedia APIs but also any "legacy" API by documenting its relations using OpenAPI.

Plant Tree PkgGoDev Build Status codecov Go Report Card

Vulcain Schema

Grab What You Need... Burn The REST!

The protocol has been published as an Internet Draft that is maintained in this repository.

A reference, production-grade, implementation gateway server is also available in this repository. It's free software (AGPL) written in Go. A Docker image is provided.

Introduction

Over the years, several formats have been created to fix performance bottlenecks impacting web APIs: over fetching, under fetching, the n+1 problem...

Current solutions for these problems (GraphQL, JSON:API's embedded resources and sparse fieldsets, ...) are smart network hacks for HTTP/1. But these hacks come with (too) many drawbacks when it comes to HTTP cache, logs and even security.

Fortunately, thanks to the new features introduced in HTTP/2, it's now possible to create true REST APIs fixing these problems with ease and class! Here comes Vulcain!

See also the comparison between Vulcain and GraphQL and other API formats.

Pushing Relations

Preload Schema

Considering the following resources:

/books

{
    "member": [
        "/books/1",
        "/books/2"
    ]
}

/books/1

{
    "title": "1984",
    "author": "/authors/1"
}

/books/2

{
    "title": "Homage to Catalonia",
    "author": "/authors/1"
}

/authors/1

{
    "givenName": "George",
    "familyName": "Orwell"
}

The Preload HTTP header introduced by Vulcain can be used to ask the server to immediately push resources related to the requested one using HTTP/2 Server Push:

GET /books/ HTTP/2
Preload: "/member/*/author"

In addition to /books, a Vulcain server will use HTTP/2 Server Push to push the /books/1, /books/2 and /authors/1 resources!

Example in JavaScript:

const bookResp = await fetch("/books/1", { headers: { Preload: `"/author"` } });
const bookJSON = await bookResp.json();

// Returns immediately, the resource has been pushed and is already in the push cache
const authorResp = await fetch(bookJSON.author);
// ...

Full example, including collections, see also use GraphQL as query language for Vulcain.

Thanks to HTTP/2 multiplexing, pushed responses will be sent in parallel.

When the client will follow the links and issue a new HTTP request (for instance using fetch()), the corresponding response will already be in cache, and will be used instantly!

For non-hypermedia APIs (when the identifier of the related resource is a simple string or int), use an OpenAPI specification to configure links between resources. Tip: the easiest way to create a hypermedia API is to use the API Platform framework (by the same author as Vulcain).

More than 90% of users have devices supporting HTTP/2. However, for the remaining 10%, and for cases where using HTTP/2 Server Push isn't allowed such as when resources are served by different authorities, Vulcain allows to gracefully fallback to preload links, which can be used together with the 103 status code.

Query Parameter

Alternatively to HTTP headers, the preload query parameter can be used:

Preload Query Schema

Filtering Resources

Fields Schema

The Fields HTTP header allows the client to ask the server to return only the specified fields of the requested resource, and of the preloaded related resources.

Multiple Fields HTTP headers can be passed. All fields matching at least one of these headers will be returned. Other fields of the resource will be omitted.

Considering the following resources:

/books/1

{
    "title": "1984",
    "genre": "novel",
    "author": "/authors/1"
}

/authors/1

{
    "givenName": "George",
    "familyName": "Orwell"
}

And the following HTTP request:

GET /books/1 HTTP/2
Preload: "/author"
Fields: "/author/familyName", "/genre"

A Vulcain server will return a response containing the following JSON document:

{
    "genre": "novel",
    "author": "/authors/1"
}

It will also push the following filtered /authors/1 resource:

{
    "familyName": "Orwell"
}

Query Parameter

Alternatively to HTTP headers, the fields query parameter can be used to filter resources:

Preload Query Schema

See Also

License and Copyright

tl;dr:

  • proprietary software can implement the Vulcain specification
  • proprietary software can be used behind the Vulcain Gateway Server without having to share their sources
  • modifications made to the Vulcain Gateway Server must be shared
  • alternatively, a commercial license is available for the Vulcain Gateway Server

The specification is available under the IETF copyright policy. The Vulcain specification can be implemented by any software, including proprietary software.

The Vulcain Gateway Server is licensed under AGPL-3.0. This license implies that if you modify the Vulcain Gateway Server, you must share those modifications. However, the AGPL-3.0 license applies only to the gateway server itself, not to software used behind the gateway.

For companies not wanting, or not able to use AGPL-3.0 licensed software, commercial licenses are also available. Contact us for more information.

Treeware

This package is Treeware. If you use it in production, then we ask that you buy the world a tree to thank us for our work. By contributing to the Treeware forest you’ll be creating employment for local families and restoring wildlife habitats.

Credits

Created by Kévin Dunglas. Sponsored by Les-Tilleuls.coop.

Some ideas and code used in Vulcain's reference implementation have been taken from Hades by Gabe Sullice, an HTTP/2 reverse proxy for JSON:API backend.

See also the prior arts.

Issues
  • [Question] Relations preloaded within first 'main' request, making the gateway server requests very slow

    [Question] Relations preloaded within first 'main' request, making the gateway server requests very slow

    Hi,

    First of all, very interesting project!

    I'm implementing the Vulcain Gateway server in my application and it's performing slow. To be sure it's not the fault of my application I downloaded and ran the demo and this is also slow at preloading the relations. It looks like Vulcain is preloading every relation in the main request and slowing down the request when it's calling the 'conferences' endpoint:
    Schermafbeelding 2020-03-10 om 15 27 05 This is in contrast to the slide in the Symfonycon slides, where the preloading 'conferences' request takes about the same time (bit more than 4sec) it needs to request the 'conferences' endpoint without preloading: image

    I'm not sure if this is how Vulcain is supposed to work, but if it is I don't really see how Vulcain can be faster than GraphQL for example?

    opened by matthijsoberon 6
  • [Question] Can/should i use Nginx and/or Traefik with Vulcain? Where should be Varnish?

    [Question] Can/should i use Nginx and/or Traefik with Vulcain? Where should be Varnish?

    Hello, everybody! Cool idea, cool project!

    But i'm not understand how to modify my current scheme to use Vulcain.

    Now:

    (https, http/2)-> Traefik -(http)-> Nginx -(static or fastcgi)-> php-fpm (ApiPlatform)

    • Traefik is my ingress controller with auto routing to some microservices via docker labels. +http/2 endpoint.
    • Nginx handles static files and routes dynamic to php-fpm.
    1. How it will looks with Vulcain?
    2. Where should be Varnish?
    opened by dvc 6
  • Passing query params to sub-resources

    Passing query params to sub-resources

    There is a mention about using GraphQL as a query language for Vulcain here : https://github.com/dunglas/vulcain/blob/master/docs/graphql.md#using-graphql-as-query-language-for-vulcain

    I've been playing on a toyproject about Vulcain and I'm wondering if all GraphQL queries can be converted to a HTTP Request using Vulcain, and especially, deep arguments like the following :

    Given this query

    {
      persons(orderBy: name_ASC) {
        id
        name,
        starships(type: destroyer) {
          name,
          weapons(category: nuclear) {
            name
          }
        }
      }
    }
    

    I believe it may be translated to the following requests :

    GET /persons?orderBy=name
    Preload: /*/starships/*/weapons
    
    [
      {
        "id": "/persons/1",
        "name": "Person 1",
        "starships": "/starships?person=/persons/1&type=destroyer"
      } 
    ]
    
    GET /starships?person=/persons/1&type=destroyer
    
    [
      {
        "id": "/starships/1",
        "name": "Starship 1",
        "weapons": "/weapons?starship=/startships/1&category=nuclear"
      }
    ]
    
    GET /weapons?starship=/startships/1&category=nuclear
    
    [
      {
        "id": "/weapons/1",
        "name": "Weapon 1",
      }
    ]
    

    Now It's easy to change weapons(category: nuclear) to weapons(category: explosives) in GraphQL, but I'm not sure how to do it with Vulcain. Do you have any insight about this problem?

    opened by polc 4
  • ci: bump go to 1.16, resolve golang-ci lint flakyness (#1)

    ci: bump go to 1.16, resolve golang-ci lint flakyness (#1)

    • Bumped Go to v1.16, updated modules with go mod tidy

    • Update ci.yml: golang-ci Actions job acting flaky when staging my PR. Linter is now pinned to the same go version the project is using at 1.16. Also configured it to only run against newly committed code when triggered by a pull_request. A merge into main from a PR will still trigger a full run of the golang-ci linter suite against the codebase.

    opened by kevholmes 3
  • spec: allow to select rel and media types

    spec: allow to select rel and media types

    • Allow to select specific media types as requested in #46 but also specific relation types and languages
    • Allow to select all links by passing an empty string (best used with the new rel parameter
    • Switch to the format defined in the structured headers I-D
    • Add an "Implementation Status" section (RFC6982
    • Add an "Acknowledgment" section

    @andrerom could you review this change? It should address the eZ Publish's use case (closes #46). @evert as discussed during API Days Interface, this change to the spec should allow to cover the use cases also covered by your Prefer-Push I-D. Would you mind to take a look?

    Thanks!

    opened by dunglas 3
  • Why is it AGPL licensed?

    Why is it AGPL licensed?

    While AGPL is a valid license, it might be blocking the usage of Vulcain in enterprise environments (legal considerations). Is there any reason to not publish it under an even more less restrictive license as MIT?

    opened by gabel 3
  • Link to IETF

    Link to IETF

    Hi,

    I noticed that the link to IETF page (https://datatracker.ietf.org/doc/draft-dunglas-vulcain/) from README page is not working. I tried to find correct link on IETF website but could not find it.

    Is it because the request for this new standard is not registered yet by IETF?

    Laurent

    opened by laurent35240 3
  • Is it ok to use the name

    Is it ok to use the name "Vulcain" in an alternative implementation of the protocol?

    I am working on a (WIP) implementation of parts of the Vulcain protocol to be used in a server less (OpenWhisk) environment here: https://github.com/trieloff/helix-vulcain-filters

    Is it ok to use the term "Vulcain" in the project name?

    opened by trieloff 3
  • Curren state

    Curren state

    Vulcain's approach is quite intriguing. However major browsers per my understanding have deprecated the h2 server push feature.

    Does this mean that vulcain in current state is unusable?

    I understand that it might make use of Early Hints feature in the future.

    opened by AntonioL 2
  •  feat!: make the library usable without the builtin reverse proxy

    feat!: make the library usable without the builtin reverse proxy

    • Make the library usable with any Go HTTP handler or reverse proxy, including Caddy server
    • Improve the documentation
    • Add an example
    • Switch from Logrus to Uber Zap
    opened by dunglas 2
  • Will 'object' remain a permanent key for targeting nested content ?

    Will 'object' remain a permanent key for targeting nested content ?

    Hi Dunglas,

    I see examples and I have to say that the object attribute is a little weird for me..

    This key name is coming for your conception or from another architecture?

    IMHO, I'll prefer to use a key name named like content or value or more specific to Vulca(i)n outbreak or rash.

    Thanks for reading !

    opened by edimitchel 2
  • Fields issue on preload entity

    Fields issue on preload entity

    Hi! I'm playing with Vulcain but i'm facing to an issue... I've 2 entities in an Api Platform project (greeting and author, author have many greetings, for relationship)

    If i fetch greetings endpoint:

    window.fetch('https://localhost/greetings', {
            headers: {
                preload: '"/hydra:member/*/author"',
                fields: '"/hydra:member/*/@id", "/hydra:member/*/name", "/hydra:member/*/author/@id"'
            }
        }).then(() => window.fetch('https://localhost/authors/1'))
    

    Of course authors/1 exists and it linked to all my greetings for more simplicity.

    So... In this case, with fields contains "/hydra:member/*/authors/@id" (it the same with firstname or anyother property) push doesn't work image

    But if instead "/hydra:member/*/authors/@id", i've "/hydra:member/*/authors" (without /@id) push works (but i get full object after, of course) image

    What am i missing....? Documentation seems clear about, i think this should work...

    opened by BenWaNH 3
  • Link Varnish X-Key broken

    Link Varnish X-Key broken

    In the Cache-Documentaion there is a broken link to X-Key. https://github.com/dunglas/vulcain/blob/main/docs/cache.md

    Broken in Master-Branch https://github.com/varnish/varnish-modules/blob/master/docs/vmod_xkey.rst

    Branch 6.5 is available https://github.com/varnish/varnish-modules/blob/6.5/docs/vmod_xkey.rst

    opened by art-cg 1
  • Preload link should include the

    Preload link should include the "crossorigin" attribute, if initial request uses CORS

    I've been trying to get Vulcain working in a situation where CORS is necessary. At the moment it seems to me that browsers are blocking preload requests, because the generated link headers do not include the crossorigin attribute (docs).

    2021-04-25_17-03

    I'd propose adding crossorigin=anonymous, if the Access-Control-Allow-Origin header is present. And crossorigin=use-credentials, if the Access-Control-Allow-Credentials header is true (docs).

    However, I'm not entirely sure what should happen, if the related resources are owned by different API-s (e.g. bookapi.com/book/1 is referencing authorapi.com/author/2). In that case the abovementioned could not be implied from the response headers.

    Since the Link headers are added by Vulcain, there's no way for the upstream to handle this or is there?

    opened by kgilden 0
  • ACME_HOSTS on prod env

    ACME_HOSTS on prod env "x509: certificate signed by unknown authority"

    Hi,

    i'm trying to use Vulcain in my prod stack (no issue in dev with self signed certificate) So, i'm Docker user and this is my service config in my docker-compose.yml file :

    vulcain:
            image: dunglas/vulcain
            environment:
                - ACME_HOSTS=api.example.com
                - UPSTREAM=http://api
            depends_on:
                - api
            deploy:
                labels:
                    - traefik.enable=true
                    - traefik.tcp.routers.isc-api.rule=HostSNI(`api.example.com`)
                    - traefik.tcp.routers.isc-api.entrypoints=websecure
                    - traefik.tcp.routers.isc-api.tls=true
                    - traefik.tcp.routers.isc-api.tls.passthrough=true
                    - traefik.tcp.routers.isc-api.service=service-api-isc
                    - traefik.tcp.services.service-api-isc.loadbalancer.server.port=443
            <<: *network
    

    And below, content docker log for Vulcain service:

    {"level":"info","ts":1606489658.6425629,"caller":"vulcain/server.go:147","msg":"vulcain started","protocol":"https","addr":""}
    2020/11/27 15:09:10 http: TLS handshake error from 10.0.7.11:39992: Get "https://acme-v02.api.letsencrypt.org/directory": x509: certificate signed by unknown authority
    2020/11/27 15:09:10 http: TLS handshake error from 10.0.7.11:39990: acme/autocert: missing certificate
    2020/11/27 15:09:10 http: TLS handshake error from 10.0.7.11:39996: acme/autocert: missing certificate
    2020/11/27 15:09:10 http: TLS handshake error from 10.0.7.11:39998: acme/autocert: missing certificate
    

    For additional informations, i'm using Traefik and Docker Swarm but i don't think that is an impact to this issue

    What's wrong with acme config?

    Thanks BenWa

    opened by BenWaNH 0
  • Clear up inconsistency in documentation of production-readiness of the Gateway Server

    Clear up inconsistency in documentation of production-readiness of the Gateway Server

    While reading the documentation, I stumbled on this inconsistency in the documentation.

    On the readme.md it says:

    A reference, production-grade, implementation gateway server is also available in this repository.

    While on the linked page (Install The Gateway Server), it says about the Gateway server:

    The Gateway Server is still experimental

    So what is it? Either it is "experimental" or it is "production-grade". So one of these wordings should be changed.

    Or it is somehow both, but some clarification on "why" would be in order in that case.

    opened by steefw2w 1
Releases(v0.2.1)
Owner
Kévin Dunglas
Founder of @coopTilleuls / Creator of @api-platform, Mercure.rocks and Vulcain.rocks / @symfony Core Team
Kévin Dunglas
A Realtime API Gateway used with NATS to build REST, real time, and RPC APIs, where all your clients are synchronized seamlessly.

Realtime API Gateway Synchronize Your Clients Visit Resgate.io for guides, live demos, and resources. Resgate is a Go project implementing a realtime

Resgate.io - Synchronize Your Clients 567 Apr 13, 2022
Minimal and idiomatic WebSocket library for Go

websocket websocket is a minimal and idiomatic WebSocket library for Go. Install go get nhooyr.io/websocket Highlights Minimal and idiomatic API First

Anmol Sethi 2.2k May 5, 2022
A simple Hasura client for query and metadata APIs

Hasura Client A Hasura client for schema APIs https://hasura.io/docs/latest/graphql/core/api-reference/index.html Usage Create client client := hasura

Toan Nguyen 0 Oct 31, 2021
The Go client to access APIs on the Micro Platform

This is the Go client to access APIs on the Micro Platform

Micro 4 Nov 9, 2021
Fake server, Consumer Driven Contracts and help with testing performance from one configuration file with zero system dependencies and no coding whatsoever

mockingjay server Mockingjay lets you define the contract between a consumer and producer and with just a configuration file you get: A fast to launch

Chris James 517 Apr 22, 2022
webrpc is a schema-driven approach to writing backend services for modern Web apps and networks

webrpc is a schema-driven approach to writing backend servers for the Web. Write your server's api interface in a schema format of RIDL or JSON, and t

null 447 May 5, 2022
🚀 gnet is a high-performance, lightweight, non-blocking, event-driven networking framework written in pure Go./ gnet 是一个高性能、轻量级、非阻塞的事件驱动 Go 网络框架。

English | ???? 中文 ?? Introduction gnet is an event-driven networking framework that is fast and lightweight. It makes direct epoll and kqueue syscalls

Andy Pan 6.4k May 9, 2022
Go driven rpc code generation tool for right now.

Go driven rpc code generation tool for right now. 100% Go Describe services with Go interfaces (with structs, methods, comments, etc.) Generate server

pace.dev 586 May 13, 2022
High-performance, non-blocking, event-driven, easy-to-use networking framework written in Go, support tls/http1.x/websocket.

High-performance, non-blocking, event-driven, easy-to-use networking framework written in Go, support tls/http1.x/websocket.

lesismal 664 May 10, 2022
A sample web API in GO (with GIn) under a domain driven architecture.

Golang Sample API Domain Driven Design Pattern 1. About This sample project presents a custom made domain driven API architecture in Golang using the

George Batagiannis ⚡ 1 Jan 10, 2022
Event driven modular status-bar for dwm; written in Go & uses Unix sockets for signaling.

dwmstat A simple event-driven modular status-bar for dwm. It is written in Go & uses Unix sockets for signaling. The status bar is conceptualized as a

Navaz Alani 1 Dec 25, 2021
A yaml data-driven testing format together with golang testing library

Specified Yaml data-driven testing Specified is a yaml data format for data-driven testing. This enforces separation between feature being tested the

Design it, Run it 0 Jan 9, 2022
An event driven remote access trojan for experimental purposes.

erat An event driven remote access trojan for experimental purposes. This example is very simple and leverages ssh failed login events to trigger erat

siovador 0 Jan 16, 2022
Package event-driven makes it easy for you to drive events between services

Event-Driven Event-driven architecture is a software architecture and model for application design. With an event-driven system, the capture, communic

Ramooz 3 Apr 20, 2022
kcp is a prototype of a Kubernetes API server that is not a Kubernetes cluster - a place to create, update, and maintain Kube-like APis with controllers above or without clusters.

kcp is a minimal Kubernetes API server How minimal exactly? kcp doesn't know about Pods or Nodes, let alone Deployments, Services, LoadBalancers, etc.

Prototype of Future Kubernetes Ideas 1.4k May 15, 2022
llb - It's a very simple but quick backend for proxy servers. Can be useful for fast redirection to predefined domain with zero memory allocation and fast response.

llb What the f--k it is? It's a very simple but quick backend for proxy servers. You can setup redirect to your main domain or just show HTTP/1.1 404

Kirill Danshin 12 Jan 23, 2022
Spin up a local instance of an Avalanche network to interact with the standard APIs or to test a custom VM

ava-sim ava-sim makes it easy for anyone to spin up a local instance of an Avalanche network to interact with the standard APIs or to test a custom VM

Ava Labs 26 Apr 15, 2022
Manabie todo apis

Manabie TODO apis Introduction This is a simple backend for a todo service, right now this service can handle login/list/create simple tasks, to make

null 0 Nov 13, 2021
iceportal-api is a Golang client implementation to interact with the REST API of iceportal.de when connected to the WiFi-Network offered in German ICE Trains.

iceportal-api is a Golang client implementation to interact with the REST API of iceportal.de when connected to the WiFi-Network offered in German ICE Trains.

Fabian Siegel 39 Dec 18, 2021