Multi-cluster api gateway based on apiserver-aggregation.

Overview

Cluster Gateway

"Cluster-Gateway" is a gateway apiserver for routing kubernetes api traffic to multiple kubernetes clusters. Additionally, the gateway is completely pluggable for a running kubernetes cluster natively because it is developed based on the native api extensibility named apiserver-aggregation. A new extended resource "cluster.core.oam.dev/ClusterGateway" will be registered into the hosting cluster after properly applying corresponding APIService objects, and the new subresource named "proxy" will be available for every existing "ClusterGateway" resource which is inspired by the original kubernetes "service/proxy", "pod/proxy" subresource.

Overall our "Cluster-Gateway" also has the following merits as a multi-cluster api-gateway solution:

  • Zero-Dependency: Normally an aggregated apiserver must be deployed along with a dedicated etcd cluster which is bringing extra costs for the admins. While our "Cluster-Gateway" can be running completely without etcd instances, because the extended "ClusterGateway" resource are physically stored as secret resources in the hosting kubernetes cluster.

  • Scalability: We can scale out the gateway instances in arbitrary replicas freely. Also it's proven stably working in production for years.

Image

$ docker pull oamdev/cluster-gateway:v1.1.4 # Or other newer tags

Documentation

Run Local: https://github.com/oam-dev/cluster-gateway/blob/master/docs/non-etcd-apiserver/local-run.md

Resource Example

spec: provider: "" access: endpoint: "https://127.0.0.1:9443" caBundle: "..." credential: type: X509Certificate x509: certificate: "..." privateKey: "..." status: { } ">
apiVersion: "cluster.core.oam.dev/v1alpha1"
kind: "ClusterGateway"
metadata:
  name: <..>
spec:
  provider: ""
  access:
    endpoint: "https://127.0.0.1:9443"
    caBundle: "..."
    credential:
      type: X509Certificate
      x509:
        certificate: "..."
        privateKey: "..."
status: { }      
Comments
  • cluster-gateway addon status

    cluster-gateway addon status

    How to check the cluster-gateway addon deployed successfully? I have install the cluster-gateway addon and got some info.

    Message :"The status of cluster-gateway add-on is unknow" Reason: "ManagedClusterAddOnLeaseNotFound" Status:"Unknow" Type:"Available"

    And I got nothing after exec "kubectl get clustergateway".Is there any way to troubleshoot?

    opened by wangzd1 9
  • multicluster wrap add enhance roundtripper, support informer cache

    multicluster wrap add enhance roundtripper, support informer cache

    Signed-off-by: champly [email protected]

    multicluster.NewClusterGatewayRoundTripper not support informer cache. Because informerFactory.Start use ctx listen for stop signals. So add enhance rounttripper include clusterName.

    opened by champly 3
  • Add ManagedCluster informer to improve performance

    Add ManagedCluster informer to improve performance

    We got a higher latency when using cluster-gateway with ocm-integration=true to access managed cluster k8s resources. And through e2e benchmark test, I found that cluster-gateway will get ManagedCluster per request, and sometimes it will cost X*100ms(sometimes X senconds in the production environment). ManagedCluster is not got from cache,so I add ManagedCluster informer to improve performance, and according to the result of e2e benchmark, the latency is greatly reduced.

    add arg --v=4 to gateway-deployment, and then run multicluster e2e benchmark test case test case, I copy the logs of some high latency requests below:

    I1206 10:22:12.321915       1 request.go:597] Waited for 392.608513ms due to client-side throttling, not priority and fairness, request: GET:https://10.96.0.1:443/apis/cluster.open-cluster-management.io/v1/managedclusters/cluster-81
    I1206 10:22:12.326916       1 httplog.go:129] "HTTP" verb="GET" URI="/apis/cluster.core.oam.dev/v1alpha1/clustergateways/cluster-81/proxy/api/v1/namespaces/kube-system" latency="398.586785ms" userAgent="Go-http-client/2.0" audit-ID="94579a16-45ba-4333-8c83-7bb95dd703ea" srcIP="172.23.211.87:58464" resp=200
    I1206 10:22:12.522535       1 request.go:597] Waited for 387.186395ms due to client-side throttling, not priority and fairness, request: GET:https://10.96.0.1:443/apis/cluster.open-cluster-management.io/v1/managedclusters/cluster-82
    I1206 10:22:12.526706       1 httplog.go:129] "HTTP" verb="GET" URI="/apis/cluster.core.oam.dev/v1alpha1/clustergateways/cluster-82/proxy/healthz" latency="391.656812ms" userAgent="Go-http-client/2.0" audit-ID="1e02f8f4-8bcd-4fdb-a96d-978074d2f0c5" srcIP="172.23.211.88:36280" resp=200
    I1206 10:22:12.722840       1 request.go:597] Waited for 382.323062ms due to client-side throttling, not priority and fairness, request: GET:https://10.96.0.1:443/apis/cluster.open-cluster-management.io/v1/managedclusters/cluster-77
    I1206 10:22:12.727957       1 httplog.go:129] "HTTP" verb="GET" URI="/apis/cluster.core.oam.dev/v1alpha1/clustergateways/cluster-77/proxy/api/v1/namespaces/kube-system" latency="387.744393ms" userAgent="Go-http-client/2.0" audit-ID="5022a7df-d6e8-48a7-a53a-2901cc15f187" srcIP="172.23.211.87:58464" resp=200
    I1206 10:22:12.922070       1 request.go:597] Waited for 387.643281ms due to client-side throttling, not priority and fairness, request: GET:https://10.96.0.1:443/apis/cluster.open-cluster-management.io/v1/managedclusters/cluster-80
    I1206 10:22:12.926795       1 httplog.go:129] "HTTP" verb="GET" URI="/apis/cluster.core.oam.dev/v1alpha1/clustergateways/cluster-80/proxy/healthz" latency="393.948768ms" userAgent="Go-http-client/2.0" audit-ID="ac30211c-7c27-45f2-97b8-c6885826022b" srcIP="172.23.211.88:36280" resp=200
    I1206 10:22:13.122193       1 request.go:597] Waited for 392.955174ms due to client-side throttling, not priority and fairness, request: GET:https://10.96.0.1:443/apis/cluster.open-cluster-management.io/v1/managedclusters/cluster-78
    I1206 10:22:13.126164       1 httplog.go:129] "HTTP" verb="GET" URI="/apis/cluster.core.oam.dev/v1alpha1/clustergateways/cluster-78/proxy/api/v1/namespaces/kube-system" latency="397.184533ms" userAgent="Go-http-client/2.0" audit-ID="5a27ffae-58d2-4b63-badf-4c6d832bc8e9" srcIP="172.23.211.87:58464" resp=200
    

    the result of e2e benchmark before adding ManagedCluster informer:

    • [MEASUREMENT]
    Basic RoundTrip Test
    /root/cluster-gateway/e2e/benchmark/configmap.go:32
      list namespace from managed cluster
      /root/cluster-gateway/e2e/benchmark/configmap.go:143
    
      Ran 10 samples:
      runtime:
        Fastest Time: 0.384s
        Slowest Time: 1.212s
        Average Time: 0.921s ± 0.311s
    ------------------------------
    
    Ran 1 of 1 Specs in 9.210 seconds
    SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped
    
    You're using deprecated Ginkgo functionality:
    =============================================
    Ginkgo 2.0 is under active development and will introduce several new features, improvements, and a small handful of breaking changes.
    A release candidate for 2.0 is now available and 2.0 should GA in Fall 2021.  Please give the RC a try and send us feedback!
      - To learn more, view the migration guide at https://github.com/onsi/ginkgo/blob/ver2/docs/MIGRATING_TO_V2.md
      - For instructions on using the Release Candidate visit https://github.com/onsi/ginkgo/blob/ver2/docs/MIGRATING_TO_V2.md#using-the-beta
      - To comment, chime in at https://github.com/onsi/ginkgo/issues/711
    
      Measure is deprecated and will be removed in Ginkgo V2.  Please migrate to gomega/gmeasure.
      Learn more at: https://github.com/onsi/ginkgo/blob/ver2/docs/MIGRATING_TO_V2.md#removed-measure
        /root/cluster-gateway/e2e/benchmark/configmap.go:143
    
    To silence deprecations that can be silenced set the following environment variable:
      ACK_GINKGO_DEPRECATIONS=1.16.5
    
    --- PASS: TestE2E (9.23s)
    PASS
    ok  	github.com/oam-dev/cluster-gateway/e2e/benchmark	9.250s
    

    the result of e2e benchmark after adding ManagedCluster informer:

    • [MEASUREMENT]
    Basic RoundTrip Test
    /root/cluster-gateway/e2e/benchmark/configmap.go:32
      list namespace from managed cluster
      /root/cluster-gateway/e2e/benchmark/configmap.go:143
    
      Ran 10 samples:
      runtime:
        Fastest Time: 0.020s
        Slowest Time: 0.058s
        Average Time: 0.025s ± 0.011s
    ------------------------------
    
    Ran 1 of 1 Specs in 0.250 seconds
    SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped
    
    You're using deprecated Ginkgo functionality:
    =============================================
    Ginkgo 2.0 is under active development and will introduce several new features, improvements, and a small handful of breaking changes.
    A release candidate for 2.0 is now available and 2.0 should GA in Fall 2021.  Please give the RC a try and send us feedback!
      - To learn more, view the migration guide at https://github.com/onsi/ginkgo/blob/ver2/docs/MIGRATING_TO_V2.md
      - For instructions on using the Release Candidate visit https://github.com/onsi/ginkgo/blob/ver2/docs/MIGRATING_TO_V2.md#using-the-beta
      - To comment, chime in at https://github.com/onsi/ginkgo/issues/711
    
      Measure is deprecated and will be removed in Ginkgo V2.  Please migrate to gomega/gmeasure.
      Learn more at: https://github.com/onsi/ginkgo/blob/ver2/docs/MIGRATING_TO_V2.md#removed-measure
        /root/cluster-gateway/e2e/benchmark/configmap.go:143
    
    To silence deprecations that can be silenced set the following environment variable:
      ACK_GINKGO_DEPRECATIONS=1.16.5
    
    --- PASS: TestE2E (0.26s)
    PASS
    ok  	github.com/oam-dev/cluster-gateway/e2e/benchmark	0.286s
    
    opened by ivan-cai 1
  • do authorization on local cluster before forward request

    do authorization on local cluster before forward request

    Today gateway using cluster credential to forward the request. In multiple tenants, it means all tenants will share same credential and permission to access the target cluster, it is a security risk.

    In multiple clusters management, we can assume hub cluster and managed cluster have same structure, for example, create namespace1 for user1 with dev permission on hub cluster, multiple clusters management will create namespace1 on all managed cluster.

    With this assumption, to fix the issue above, gateway could do authorization on hub cluster before forward the request, for example, if user1 wants to access namespace1 on managed cluster, then forward, but if user2 wants to access namespace1 on managed cluster, then reject.

    feature flag - --proxy-local-authorization=true to enable the change, default is false

    opened by zychina 1
  • Feat: support proxy dryrun request

    Feat: support proxy dryrun request

    Signed-off-by: Yin Da [email protected]

    In Kubernetes apiserver lib, the dryRun request for rest.Connector will be filtered which means ClusterGateway cannot make proxy request for dryRun request previously. This PR solves this issue by

    1. Escape the dryRun query parameter to __dryRun at the beginning of handling proxy requests.
    2. Unescape it while making proxy requests.

    https://github.com/kubernetes/kubernetes/blob/b0246c64309ce1ce504ae0a2ca3c66be6ff814e5/staging/src/k8s.io/apiserver/pkg/endpoints/handlers/rest.go#L181

    opened by Somefive 0
  • Feat: support client identity exchange

    Feat: support client identity exchange

    Signed-off-by: Yin Da [email protected]

    Implement #120, except:

    1. priority is not implemented. The implicit priority for the rules is the lower index rule will have higher priority. Cluster configuration will have higher priority than global configuration.
    2. ExternalIdentityExchanger is not implemented, but interfaces are left for future development.
    opened by Somefive 0
  • Feat: add log args

    Feat: add log args

    Signed-off-by: Yin Da [email protected]

    Add logging arguments for cluster-gateway.

    Fixes https://github.com/oam-dev/cluster-gateway/issues/104.

    opened by Somefive 0
  • Fix: allow plain secret use when ocm integration enabled

    Fix: allow plain secret use when ocm integration enabled

    Signed-off-by: Yin Da [email protected]

    When --ocm-integration enabled, clusters managed by direct secrets are unavailable to provide services. This PR fixes it.

    opened by Somefive 0
  • [Feat] Client Identity Exchanger

    [Feat] Client Identity Exchanger

    Proposal: Client Identity Exchanger

    Background

    In v1.4.0, we have ClientIdentityPenetration feature flag which allows cluster-gateway to pass the hub cluster identity to the managed cluster requests. This ensures that the requests through cluster-gateway to keep consistent identity across all clusters and therefore we can leverage the native Kubernetes RBAC mechanism to deal with the authorization.

    Purpose

    But there are times that the hub cluster identity is not aligned with the managed cluster identity.

    1. For example, in Kubernetes provided by cloud providers, each cluster may have unique ID for its issued identity. Therefore the identity across clusters are not natively consistent. When the problem goes to hybrid cloud, it will be more difficult to manage that.
    2. And there are times the user of cluster-gateway may want to use privileges provided by the joined cluster itself, instead of using its own hub cluster identity. This can be seen as a super-user behaviour.

    To address the above issues, the ClientIdentityExchanger is proposed here. The ClientIdentityExchanger should be able to exchange identities between the hub cluster and managed clusters depending on the given configuration.

    Solution

    The ClientIdentityExchanger should work when both ClientIdentityExchanger and ClientIdentityPenetration feature flags are switched on. The ClientIdentityExchanger can be classified into two scopes, one is the global configuration and the other one is the cluster configuration.

    1. The global configuration works for all clusters. It can be stored in static files and used by the cluster-gateway. In Kubernetes, we can use ConfigMap/Secret to store it and mount it into the cluster-gateway pod.
    2. The cluster configuration works for specific cluster. This data is stored along with the cluster object, such as the cluster.core.oam.dev/client-identity-exchanger-config field in the annotation of cluster secret or OCM managed cluster.

    The configuration should be able to describe the rules for exchanging identity, as follows

    apiVersion: cluster.core.oam.dev/v1alpha1
    kind: ClusterGatewayConfiguration
    spec:
      clientIdentityExchanger:
        rules:
        - name: super-user
          source:
            group: sudoer
          type: PrivilegedIdentityExchanger
          priority: 10
        - name: mapping
          source:
            user: user-12345
            cluster: cluster-34567
          target:
            user: user-34567
          type: StaticMappingIdentityExchanger
          priority: 100
        - name: external
          source:
            userPattern: ext-*
          url: http://external.exchanger/
          type: ExternalIdentityExchanger
          priority: 20
    

    We can support three types of exchanger.

    • PrivilegedIdentityExchanger will directly use the original identity registered by the cluster.
    • StaticMappingIdentityExchanger will make static mappings for the given identity.
    • ExternalIdentityExchanger will call external service to provide the identity mapping resources.
    opened by Somefive 2
  • [Feat] Support the joined clusters can be set OFFLINE

    [Feat] Support the joined clusters can be set OFFLINE

    OFFLINE means that all the requests come to the offline clusters will be bypass and return normally, and

    • vela-core can recognize the defined return code, and set the status for applicaiton cr depending on the defined return code
    • vela-apiserver can recognize the defined return code, and return the Rest-API or CLI call depending on the defined return code
    opened by oeular 0
  • Get pod log timeout

    Get pod log timeout

    当我通过cluster-gateway调用 pod GetLogs接口是会出现连接断开的问题,大概是1分钟左右。 version:v1.3.2 cluster-gateway日志如下: 2022/04/28 13:17:37 httputil: ReverseProxy read error during body copy: context deadline exceeded E0428 13:17:37.307985 1 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/cluster.core.oam.dev/v1alpha1/clustergateways/cluster1/proxy/api/v1/namespaces/kube-system/pods/kube-apiserver-node1/log?follow=true&tailLines=150" audit-ID="25c2a7a7-7a2f-4bf6-b805-105771b6ebfd" E0428 13:17:37.308822 1 timeout.go:137] post-timeout activity - time-elapsed: 803.726µs, GET "/apis/cluster.core.oam.dev/v1alpha1/clustergateways/cluster1/proxy/api/v1/namespaces/kube-system/pods/kube-apiserver-node1/log" result: net/http: abort Handler 2022/04/28 13:21:46 httputil: ReverseProxy read error during body copy: context deadline exceeded E0428 13:21:46.937195 1 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/cluster.core.oam.dev/v1alpha1/clustergateways/cluster1/proxy/api/v1/namespaces/kube-system/pods/kube-apiserver-node1/log?follow=true&tailLines=150" audit-ID="7cd14081-a0d9-4bf7-a85d-a8db310e7b40" E0428 13:21:46.943292 1 timeout.go:137] post-timeout activity - time-elapsed: 6.018041ms, GET "/apis/cluster.core.oam.dev/v1alpha1/clustergateways/cluster1/proxy/api/v1/namespaces/kube-system/pods/kube-apiserver-node1/log" result: net/http: abort Handler

    bug 
    opened by ghostloda 0
Releases(v1.7.0-alpha.3)
Owner
Open Application Model
Make shipping applications more enjoyable.
Open Application Model
:mailbox_closed: Your own local SMS gateway in Go

gosms Your own local SMS gateway What's the use ? Can be used to send SMS, where you don't have access to internet or cannot use Web SMS gateways or w

null 1.4k Jan 2, 2023
grpc + grpc gateway demo

grpc + grpc gateway demo

null 0 Nov 5, 2021
Nodebook - Multi-Lang Web REPL + CLI Code runner

nodebook Nodebook - Multi-Language REPL with Web UI + CLI code runner Useful to practice algorithms and datastructures for coding interviews. What is

Jérôme Schneider 1.6k Dec 29, 2022
did:ar | multi-chain DIDs backed by Arweave

did:ar | multi-chain DIDs backed by Arweave - THIS IS A WIP ABSOLUTELY SHOULD NOT BE USED IN PRODUCTION ___ ___

Glass 16 Sep 20, 2022
Packer Plugin Vagrant - The Vagrant multi-component plugin can be used with HashiCorp Packer to create custom images

Packer Plugin Vagrant - The Vagrant multi-component plugin can be used with HashiCorp Packer to create custom images

null 1 Jul 13, 2022
Ecsgo - Cache friendly, Multi threading Entity Component System in Go (with Generic)

ECSGo ECSGo is an Entity Component System(ECS) in Go. This is made with Generic

Vong Kong 17 Oct 19, 2022
Executor - Wrapper for exec.Command for simple using and multi commands executing

executor Examples package main import ( "fmt" "github.com/solar-jsoc/execut

null 5 Feb 12, 2022
cluster registration and lifecycle

Cluster Registration Contains controllers that support the registration of managed clusters to a hub to place them under management. Community, discus

Open Cluster Management 39 Nov 6, 2022
Control external Fan to cool down your raspi cluster

Fan control for Raspberry Pi This is a small project that I build in order to cool down my raspi home cluster The case I use have some external fans t

Carlos Tadeu Panato Junior 13 Dec 11, 2021
A restart tracker that gives context to what is restarting in your cluster

A restart tracker that gives context to what is restarting in your cluster

Soraro Labs 52 Dec 20, 2022
Yubigo is a Yubikey client API library that provides an easy way to integrate the Yubico Yubikey into your existing Go-based user authentication infrastructure.

yubigo Yubigo is a Yubikey client API library that provides an easy way to integrate the Yubikey into any Go application. Installation Installation is

Geert-Johan Riemer 123 Oct 27, 2022
Basic-api-with-go - A basic api with golang

I am creating my first API with GO. Install go get -u github.com/Yefhem/basic-ap

Yefhem 1 Jan 3, 2022
Go API wrapper for Greenhouse.io API

Greenhouse IO A Go interface to Greenhouse.io's API Useage Creating the Client NewClient accepts: A context; used for any HTTP requests made using the

Grayscale 0 Jan 14, 2022
Flow-based and dataflow programming library for Go (golang)

GoFlow - Dataflow and Flow-based programming library for Go (golang) Status of this branch (WIP) Warning: you are currently on v1 branch of GoFlow. v1

Vladimir Sibirov 1.5k Dec 30, 2022
GObject-introspection based bindings generator

WARNING! This project is no longer maintained. Probably doesn't even compile. GObject-introspection based bindings generator for Go. Work in progress

null 47 Jan 5, 2022
A Go based HTTP Botnet

Second interation of GoBot, https://github.com/SaturnsVoid/GoBot2 GoBot GoBot is a project i am working on as i learn Go. GoBot is a PoC(Proof of Conc

Adam 112 Nov 24, 2022
Generate spreadsheets based on GitHub contributions

pullsheet generates a CSV (comma separated values) & HTML output about GitHub activity across a series of repositories.

Google 63 Oct 17, 2022
Moby Project - a collaborative project for the container ecosystem to assemble container-based systems

The Moby Project Moby is an open-source project created by Docker to enable and accelerate software containerization. It provides a "Lego set" of tool

Moby 64.8k Jan 2, 2023
A modern and intuitive terminal-based text editor

micro is a terminal-based text editor that aims to be easy to use and intuitive, while also taking advantage of the capabilities of modern terminals.

Zachary Yedidia 20.9k Jan 5, 2023