SPIRE is a toolchain of APIs for establishing trust between software systems across a wide variety of hosting platforms

Related tags

Security spire
Overview

SPIRE Logo

CII Best Practices Build Status Coverage Status Go Report Card Slack Status

SPIRE (the SPIFFE Runtime Environment) is a toolchain of APIs for establishing trust between software systems across a wide variety of hosting platforms. SPIRE exposes the SPIFFE Workload API, which can attest running software systems and issue SPIFFE IDs and SVIDs to them. This in turn allows two workloads to establish trust between each other, for example by establishing an mTLS connection or by signing and verifying a JWT token. SPIRE can also enable workloads to securely authenticate to a secret store, a database, or a cloud provider service.

SPIRE is hosted by the Cloud Native Computing Foundation (CNCF) as an incubation-level project. If you are an organization that wants to help shape the evolution of technologies that are container-packaged, dynamically-scheduled and microservices-oriented, consider joining the CNCF.

Get SPIRE

Learn about SPIRE

  • Before trying SPIRE, it's a good idea to learn about its architecture and design goals.
  • Once ready to get started, see the Quickstart Guides for Kubernetes, Linux, and MacOS.
  • There are several examples demonstrating SPIRE usage in the spire-examples and spire-tutorials repositories.
  • Check ADOPTERS.md for a list of production SPIRE adopters, a view of the ecosystem, and use cases.
  • See the SPIRE Roadmap for a list of planned features and enhancements.
  • Join the SPIFFE community on Slack. If you have any questions about how SPIRE works, or how to get it up and running, the best places to ask questions are the SPIFFE Slack channels.
  • Download the free book about SPIFFE and SPIRE, "Solving the Bottom Turtle."

Integrate with SPIRE

For supported integration versions, see Supported Integrations.

Contribute to SPIRE

The SPIFFE community maintains the SPIRE project. Information on the various SIGs and relevant standards can be found in https://github.com/spiffe/spiffe.

Further Reading

  • The Scaling SPIRE guide covers design guidelines, recommendations, and deployment models.
  • For an explanation of how SPIRE compares to related systems such as secret stores, identity providers, authorization policy engines and service meshes see comparisons.

Security

Security Assessments

A third party security firm (Cure53) completed a security audit of SPIFFE and SPIRE in February of 2021. Additionally, the CNCF Technical Advisory Group for Security conducted two assessments on SPIFFE and SPIRE in 2018 and 2020. Please find the reports and supporting material, including the threat model exercise results, below.

Reporting Security Vulnerabilities

If you've found a vulnerability or a potential vulnerability in SPIRE please let us know at [email protected]. We'll send a confirmation email to acknowledge your report, and we'll send an additional email when we've identified the issue positively or negatively.

Comments
  • Add Spiffe ID templating for k8s-workload-registrar

    Add Spiffe ID templating for k8s-workload-registrar

    Currently the k8s-workload-registrar allows for creation of Spiffe IDs using CRDs

    It would be incredibly helpful to be able to have k8s-workload-registrar to allow creating of SpiffeIDs for workloads dynamically using templates defined by a CRD:

    apiVersion: spiffeid.spiffe.io/v1beta1
    kind: SpiffeID
    metadata:
      name: my-spiffe-id
      namespace: my-namespace
    spec:
      spiffeId: spiffe://example.org/{{selector k8s:sa}}/{{selector k8s:pod-uid}}
      parentId: spiffe://example.org/spire/server
    
    opened by edwarnicke 44
  • [RFC] Serverless architecture support

    [RFC] Serverless architecture support

    [RFC] Serverless architecture support

    Co-authored by @MarcosDY.

    Background

    Serverless computing allows to build applications eliminating the need to manage infrastructure. With serverless applications, the cloud service provider automatically provisions, scales, and manages the infrastructure required to run the code, eliminating the need for server software and hardware management by the developer. The current way of workload attestation in SPIRE does not completely fit in this software design pattern, where the execution context is a temporary runtime environment and is not suitable to have SPIRE Agent running to expose the Workload API alongside the serverless function.

    Proposal

    In order to allow the issuance of SVIDs to workloads in a serverless environment, we need to provide a way to issue identities to the workload without using the Workload API to obtain an identity. The workload would attest directly to SPIRE Server to obtain its identity. This means that we would go through an attestation process in a similar fashion than node attestation but without yielding an agent role to the attested serverless (and agentless) environment. The attestation process would proceed similarly as the current AttestAgent server RPC, but performed through a new call that would provide an "agentless" identity instead of a node identity in SPIRE. The renewal process would also proceed similarly to the current RenewAgent RPC, where the caller would present an active "agentless" SVID returned by the attestation call or the most recent one from a previous renewal call. This would allow to avoid going through a complete attestation process when the environment already has a valid SVID that needs to be renewed. The criteria to decide if the SVID should be rotated can be similar to the current criteria adopted in SPIRE, i.e.: rotate the SVID if it has less than half of its lifetime left. The proposed solution should facilitate the issuance of identities in a performant manner, focusing on optimizing the usage of resources, otherwise the advantages of the serverless architecture could be seen reduced by the identity issuance process. To that end, this proposal tries to leverage some of the common features available in the cloud providers that aim to solve performance problems, like reusing the execution context if one is available from a previous function call. The proposed process to obtain an identity in a serverless architecture is as follows:

    • Check if there is already a valid SVID available in the execution context.
      • If there is no valid SVID, call the "agentless" attestation RPC to get an identity.

        • Store the obtained identity in a variable declared outside of the function's handler method so it remains initialized, providing additional optimization when the function is invoked again.
      • If there is already a valid SVID, calculate its lifetime left.

        • if it has more than half of its lifetime left, just use it.
        • if it has less than half of its lifetime left, call the renewal RPC and store the obtained identity in a variable declared outside of the function's handler method.

    Sample implementation

    The following is a description of a sample implementation of the proposed process, including the changes needed in SPIRE and the components required in the serverless environment in order to be able to issue identities without having a SPIRE Agent deployed in the serverless environment.

    SPIRE

    • Add new plugin types to perform the "agentless" attestation in SPIRE Server. Have a new plugin for each provider that has a serverless architecture. For example, there will be a plugin to support AWS Lambda, a plugin for Google Cloud Functions, a plugin for Microsoft Azure Functions and other plugins for any other platform. These are some possible workflows for the implementations:

      • AWS Lambda: the function signs a GetCallerIdentity query for the AWS Security Token Service (STS) using the AWS Signature v4 algorithm and sends it to SPIRE Server. The credentials used to sign the GetCallerIdentity request come from the AWS Lambda runtime environment variables which avoids the need for an operator to manually provision credentials first. To attest the "agentless" workload, SPIRE Server sends the query to the AWS STS service to validate it and issues an SVID with a SPIFFE ID constructed from attributes extracted from the parsed signed query.

      • Google Cloud Functions: the function fetches its identity token using the Compute Metadata Server. The attestor plugin in SPIRE Server validates the token provided and issues an SVID with a SPIFFE ID constructed from attributes extracted from the parsed token.

      • Microsoft Azure: the function obtains its access token from the local token service. The attestor plugin in SPIRE Server validates the token provided and issues an SVID with a SPIFFE ID constructed from attributes extracted from the parsed token.

    • Attestation data structs are usually shared from github.com/spiffe/spire/pkg/common/plugin/<plugin_name>, which would be inconvenient to consume externally. Instead, the types required could be exposed through Protocol Buffers definitions under the proto/spire hierarchy.

    • It would be good to expose a library that can be used to facilitate the attestation process from the serverless environment. This library should expose interfaces to construct the attestation material, call the "agentless" attestation RPC in SPIRE Server and ease the reuse of the issued SVID in case that the state of the environment is preserved in a future invocation. It should also provide functionality to perform the SVID renewal process.

    Serverless environment

    The workload running in the serverless environment needs to be able to be attested without a running SPIRE Agent that exposes the Workload API. Instead, it calls an exposed RPC in SPIRE Server with attestation data that retrieves from the execution runtime. As mentioned above, it would be convenient to have a library that can be consumed in the serverless environment to aid the attestation and identity issuance process. With aim of facilitating the implementation, this proposal recommends implementing a mechanism to externally package dependencies that can be shared across multiple functions. One possible way to achieve this is to have a common interface that can be used to retrieve the identity of the "agentless" workload, that can be called from the running function and is exposed through the runtime environment. For example, in the case of AWS Lambda, the "agentless" attestation functionality can be packaged in a layer. The function that needs to be attested can be configured to use this layer, so it does not need to have it implemented in the function. This layer can also be updated with fixes or improvements without the need of updating the function itself.

    Request for Comments

    This proposal tries to layout changes needed in SPIRE and possible implementation scenarios to provide support to serverless architectures, focusing on providing a solution for AWS Lambda, Google Cloud Functions and Microsoft Azure Functions. Any feedback on the general direction of this proposal, any missing points, suggestions or thoughts in general is greatly appreciated.

    opened by amartinezfayo 41
  • Propose Updated Protos for the SPIRE Server API

    Propose Updated Protos for the SPIRE Server API

    The SPIRE registration API was originally written to allow manipulation of SPIRE registration entries. It has since become the de-facto API for all things administrative. This includes decidedly un-registration-y things such as generating join tokens, minting arbitrary SVIDs, agent management, and bundle management.

    One problem is simple naming - the "registration api" should not include things that are not related to registration. Another problem is the eventual access control that we want to provide... we currently have the "admin" bit you can set to give carte blanche access, however the scope is too wide. Finally, the registration API has grown organically over time to add features such as ListBy* etc... these features were added in a compatible manner, but we now have a chance to take a new approach to supporting this functionality in a more idiomatic way.

    This issue is done when we have loose consensus on the proto(s) for the next iteration of SPIRE management APIs.

    opened by evan2645 39
  • spire-server too high CPU usage

    spire-server too high CPU usage

    • Version: 1.2.0
    • Platform: Linux 5.13.0-30-generic - 20.04.1-Ubuntu SMP x86_64
    • Subsystem: server

    spire-server is running with no agent and without any load on it. CPU consumption is under 1% for ~50-60 minutes, then suddenly it starts consuming ~150-160% CPU and it persists until shutting down the spire-server. The same happens with all replicas. Different environments show different results but the CPU resource consumption is always jumps much higher after similar amount of time spent (~50-60 mins). I observed the above mentioned ~150-160% on a kind cluster, 95-100% on minikube, 100-110% on kvm/qemu and 45-50% on non-virtual k8s environment.

    I tried the same with spire-server 1.0.1, 1.1.0 and 1.2.0 images and got the same results. I also created a configuration without k8s-workload-registrar too, but did not help. Most probably it depends on the configuration I use since I have not managed to reproduce it with the reference configuration.

    Can you please tell me what is wrong in this configuration? What is the culprit and how can I fix it?

    Thanks in advance, Szilard

    opened by szvincze 36
  • Failed to collect all selectors for PID

    Failed to collect all selectors for PID" error="workload attestor \"k8s\" failed: rpc error: code = DeadlineExceeded desc = workloadattestor(k8s): no selectors found after max poll attempts"

    • Version: 1.0.1

    • Platform: k8s + istio

    • Subsystem: spire-server:

    apiVersion: v1
    kind: Namespace
    metadata:
      name: spire
    
    ---
    
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: spire-server
      namespace: spire
    
    ---
    
    apiVersion: v1
    kind: Secret
    metadata:
      name: spire-server
      namespace: spire
    type: Opaque
    data:
      bootstrap.key: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1JR2tBZ0VCQkRBZzJMYnVsWHpRWDFORisyRGkwUkt6TVdmRUdpb0JoaC9mRnB4N3lPRXFrYS8vVHBhZVUzTzUKUUpSWlhkV0hLdWFnQndZRks0RUVBQ0toWkFOaUFBUmFNSDZkSVpMRWhpTE9kdnpqRzdsWVlObVB6U2N2dGJWegpmTi9qeGFITFNacnRqdVlJRXJOOUNTdUFPQzRqaVBSbjdUKzBNZit2eUMwNjBzdXNpbTR6QlllaDdpOXRVRVcxCjdXK1BwZTNwWjRUeVZmQndLOHV6K1p5YTgrcFVyMk09Ci0tLS0tRU5EIEVDIFBSSVZBVEUgS0VZLS0tLS0K
    
    ---
    
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: spire-server
      namespace: spire
    data:
      server.conf: |
        server {
          bind_address = "0.0.0.0"
          bind_port = "8081"
          trust_domain = "example.org"
          data_dir = "/run/spire/data"
          log_level = "DEBUG"
          default_svid_ttl = "1h"
          ca_subject = {
            country = ["US"],
            organization = ["SPIFFE"],
            common_name = "",
          }
        }
    
        plugins {
          DataStore "sql" {
            plugin_data {
              database_type = "sqlite3"
              connection_string = "/run/spire/data/datastore.sqlite3"
            }
          }
    
          NodeAttestor "k8s_sat" {
            plugin_data {
              clusters = {
                "demo-cluster" = {
                  use_token_review_api_validation = true
                  service_account_whitelist = ["spire:spire-agent"]
                }
              }
            }
          }
    
          NodeResolver "noop" {
            plugin_data {}
          }
    
          KeyManager "disk" {
            plugin_data {
              keys_path = "/run/spire/data/keys.json"
            }
          }
    
          UpstreamAuthority "disk" {
            plugin_data {
              key_file_path = "/run/spire/secrets/bootstrap.key"
              cert_file_path = "/run/spire/config/bootstrap.crt"
            }
          }
        }
    
        health_checks {
          listener_enabled = true
          bind_address = "0.0.0.0"
          bind_port = "8080"
          live_path = "/live"
          ready_path = "/ready"
        }
      bootstrap.crt: |
        -----BEGIN CERTIFICATE-----
        MIIBzDCCAVOgAwIBAgIJAJM4DhRH0vmuMAoGCCqGSM49BAMEMB4xCzAJBgNVBAYT
        AlVTMQ8wDQYDVQQKDAZTUElGRkUwHhcNMTgwNTEzMTkzMzQ3WhcNMjMwNTEyMTkz
        MzQ3WjAeMQswCQYDVQQGEwJVUzEPMA0GA1UECgwGU1BJRkZFMHYwEAYHKoZIzj0C
        AQYFK4EEACIDYgAEWjB+nSGSxIYiznb84xu5WGDZj80nL7W1c3zf48Why0ma7Y7m
        CBKzfQkrgDguI4j0Z+0/tDH/r8gtOtLLrIpuMwWHoe4vbVBFte1vj6Xt6WeE8lXw
        cCvLs/mcmvPqVK9jo10wWzAdBgNVHQ4EFgQUh6XzV6LwNazA+GTEVOdu07o5yOgw
        DwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwGQYDVR0RBBIwEIYOc3Bp
        ZmZlOi8vbG9jYWwwCgYIKoZIzj0EAwQDZwAwZAIwE4Me13qMC9i6Fkx0h26y09QZ
        IbuRqA9puLg9AeeAAyo5tBzRl1YL0KNEp02VKSYJAjBdeJvqjJ9wW55OGj1JQwDF
        D7kWeEB6oMlwPbI/5hEY3azJi16I0uN1JSYTSWGSqWc=
        -----END CERTIFICATE-----
    
    ---
    
    apiVersion: apps/v1
    kind: StatefulSet
    metadata:
      name: spire-server
      namespace: spire
      labels:
        app: spire-server
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: spire-server
      serviceName: spire-server
      template:
        metadata:
          namespace: spire
          labels:
            app: spire-server
        spec:
          serviceAccountName: spire-server
          containers:
            - name: spire-server
              image: xxx/spire-server:1.0.1
              args: ["-config", "/run/spire/config/server.conf"]
              ports:
                - containerPort: 8081
              volumeMounts:
                - name: spire-config
                  mountPath: /run/spire/config
                  readOnly: true
                - name: spire-secrets
                  mountPath: /run/spire/secrets
                  readOnly: true
                - name: spire-data
                  mountPath: /run/spire/data
                  readOnly: false
              livenessProbe:
                httpGet:
                  path: /live
                  port: 8080
                failureThreshold: 2
                initialDelaySeconds: 15
                periodSeconds: 60
                timeoutSeconds: 3
              readinessProbe:
                httpGet:
                  path: /ready
                  port: 8080
                initialDelaySeconds: 5
                periodSeconds: 5
          volumes:
            - name: spire-config
              configMap:
                name: spire-server
            - name: spire-secrets
              secret:
                secretName: spire-server
            - name: spire-data
              emptyDir: {}
    ---
    
    apiVersion: v1
    kind: Service
    metadata:
      name: spire-server
      namespace: spire
    spec:
      type: ClusterIP
      ports:
        - name: grpc
          port: 8081
          targetPort: 8081
          protocol: TCP
      selector:
        app: spire-server
    

    spire-agent:

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: spire-agent
      namespace: spire
    
    ---
    
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: spire-agent
      namespace: spire
    data:
      agent.conf: |
        agent {
          data_dir = "/run/spire"
          log_level = "DEBUG"
          server_address = "spire-server"
          server_port = "8081"
          socket_path = "/run/spire/sockets/agent.sock"
          trust_bundle_path = "/run/spire/config/bootstrap.crt"
          trust_domain = "example.org"
        }
    
        plugins {
          NodeAttestor "k8s_sat" {
            plugin_data {
              cluster = "demo-cluster"
            }
          }
    
          KeyManager "memory" {
            plugin_data {
            }
          }
    
          WorkloadAttestor "k8s" {
            plugin_data {
              # Defaults to the secure kubelet port by default.
              # Minikube does not have a cert in the cluster CA bundle that
              # can authenticate the kubelet cert, so skip validation.
              skip_kubelet_verification = "true"
              #kubelet_read_only_port = "10255"
              node_name_env = "MY_NODE_NAME"
            }
          }
    
          WorkloadAttestor "unix" {
              plugin_data {
              }
          }
        }
    
        health_checks {
          listener_enabled = true
          bind_address = "0.0.0.0"
          bind_port = "8080"
          live_path = "/live"
          ready_path = "/ready"
        }
      bootstrap.crt: |
        -----BEGIN CERTIFICATE-----
        MIIBzDCCAVOgAwIBAgIJAJM4DhRH0vmuMAoGCCqGSM49BAMEMB4xCzAJBgNVBAYT
        AlVTMQ8wDQYDVQQKDAZTUElGRkUwHhcNMTgwNTEzMTkzMzQ3WhcNMjMwNTEyMTkz
        MzQ3WjAeMQswCQYDVQQGEwJVUzEPMA0GA1UECgwGU1BJRkZFMHYwEAYHKoZIzj0C
        AQYFK4EEACIDYgAEWjB+nSGSxIYiznb84xu5WGDZj80nL7W1c3zf48Why0ma7Y7m
        CBKzfQkrgDguI4j0Z+0/tDH/r8gtOtLLrIpuMwWHoe4vbVBFte1vj6Xt6WeE8lXw
        cCvLs/mcmvPqVK9jo10wWzAdBgNVHQ4EFgQUh6XzV6LwNazA+GTEVOdu07o5yOgw
        DwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwGQYDVR0RBBIwEIYOc3Bp
        ZmZlOi8vbG9jYWwwCgYIKoZIzj0EAwQDZwAwZAIwE4Me13qMC9i6Fkx0h26y09QZ
        IbuRqA9puLg9AeeAAyo5tBzRl1YL0KNEp02VKSYJAjBdeJvqjJ9wW55OGj1JQwDF
        D7kWeEB6oMlwPbI/5hEY3azJi16I0uN1JSYTSWGSqWc=
        -----END CERTIFICATE-----
    
    ---
    
    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      name: spire-agent
      namespace: spire
      labels:
        app: spire-agent
    spec:
      selector:
        matchLabels:
          app: spire-agent
      template:
        metadata:
          namespace: spire
          labels:
            app: spire-agent
        spec:
          hostPID: true
          hostNetwork: true
          dnsPolicy: ClusterFirstWithHostNet
          serviceAccountName: spire-agent
          initContainers:
            - name: init
              # This is a small image with wait-for-it, choose whatever image
              # you prefer that waits for a service to be up. This image is built
              # from https://github.com/lqhl/wait-for-it
              image: xxx/wait-for-it
              args: ["-t", "30", "spire-server:8081"]
              env:
                - name: MY_NODE_NAME
                  valueFrom:
                    fieldRef:
                      fieldPath: status.podIP
          containers:
            - name: spire-agent
              image: xxx/spire-agent:1.0.1
              args: ["-config", "/run/spire/config/agent.conf"]
              env:
                - name: MY_NODE_NAME
                  valueFrom:
                    fieldRef:
                      fieldPath: status.podIP
              volumeMounts:
                - name: spire-config
                  mountPath: /run/spire/config
                  readOnly: true
                - name: spire-agent-socket
                  mountPath: /run/spire/sockets
                  readOnly: false
              livenessProbe:
                httpGet:
                  path: /live
                  port: 8080
                failureThreshold: 2
                initialDelaySeconds: 15
                periodSeconds: 60
                timeoutSeconds: 3
              readinessProbe:
                httpGet:
                  path: /ready
                  port: 8080
                initialDelaySeconds: 5
                periodSeconds: 5
          volumes:
            - name: spire-config
              configMap:
                name: spire-agent
            - name: spire-agent-socket
              hostPath:
                path: /run/spire/sockets
                type: DirectoryOrCreate
    

    When I get the certificate through spire-agent, I get an error. the logs of spire-agent:

    time="2022-05-18T11:48:58Z" level=warning msg="Container id not found" attempt=7 container_id=d278326d91c8d21ce00c451ddd1dc0602a054ca96978b08fa7fae2b56cc9a676 external=false plugin_name=k8s plugin_type=WorkloadAttestor retry_interval=500ms subsystem_name=catalog
    time="2022-05-18T11:48:58Z" level=warning msg="Container id not found" attempt=8 container_id=d278326d91c8d21ce00c451ddd1dc0602a054ca96978b08fa7fae2b56cc9a676 external=false plugin_name=k8s plugin_type=WorkloadAttestor retry_interval=500ms subsystem_name=catalog
    time="2022-05-18T11:48:59Z" level=warning msg="Container id not found" attempt=9 container_id=d278326d91c8d21ce00c451ddd1dc0602a054ca96978b08fa7fae2b56cc9a676 external=false plugin_name=k8s plugin_type=WorkloadAttestor retry_interval=500ms subsystem_name=catalog
    time="2022-05-18T11:48:59Z" level=warning msg="Container id not found" attempt=10 container_id=d278326d91c8d21ce00c451ddd1dc0602a054ca96978b08fa7fae2b56cc9a676 external=false plugin_name=k8s plugin_type=WorkloadAttestor retry_interval=500ms subsystem_name=catalog
    time="2022-05-18T11:48:59Z" level=error msg="Received error from stream secrets server" error="<nil>" method=StreamSecrets pid=1193 service=SDS.v3 subsystem_name=endpoints
    time="2022-05-18T11:48:59Z" level=error msg="Failed to collect all selectors for PID" error="workload attestor \"k8s\" failed: rpc error: code = Canceled desc = workloadattestor(k8s): context canceled" pid=1193 subsystem_name=workload_attestor
    time="2022-05-18T11:48:59Z" level=debug msg="PID attested to have selectors" pid=1193 selectors="[type:\"unix\" value:\"uid:1337\" type:\"unix\" value:\"gid:1337\"]" subsystem_name=workload_attestor
    time="2022-05-18T11:48:59Z" level=error msg="Failed to attest the workload" error="rpc error: code = Unauthenticated desc = could not verify existence of the original caller: caller is no longer being watched" method=StreamSecrets pid=1193 service=SDS.v3 subsystem_name=endpoints
    

    the logs of spire-server:

    time="2022-05-18T11:07:39Z" level=warning msg="Current umask 0022 is too permissive; setting umask 0027"
    time="2022-05-18T11:07:39Z" level=info msg="Data directory: \"/run/spire/data\""
    time="2022-05-18T11:07:39Z" level=info msg="Opening SQL database" db_type=sqlite3 subsystem_name=sql
    time="2022-05-18T11:07:39Z" level=info msg="Initializing new database" subsystem_name=sql
    time="2022-05-18T11:07:39Z" level=info msg="Connected to SQL database" read_only=false subsystem_name=sql type=sqlite3 version=3.34.0
    time="2022-05-18T11:07:39Z" level=warning msg="The \"noop\" NodeResolver is not required, is deprecated, and will be removed from a future release" subsystem_name=catalog
    time="2022-05-18T11:07:39Z" level=warning msg="The `service_account_whitelist` configurable is deprecated and will be removed in a future release. Please use `service_account_allow_list` instead." external=false plugin_name=k8s_sat plugin_type=NodeAttestor subsystem_name=catalog
    time="2022-05-18T11:07:39Z" level=info msg="Plugin loaded" external=false plugin_name=k8s_sat plugin_type=NodeAttestor subsystem_name=catalog
    time="2022-05-18T11:07:39Z" level=info msg="Plugin loaded" external=false plugin_name=disk plugin_type=KeyManager subsystem_name=catalog
    time="2022-05-18T11:07:39Z" level=info msg="Plugin loaded" external=false plugin_name=disk plugin_type=UpstreamAuthority subsystem_name=catalog
    time="2022-05-18T11:07:39Z" level=debug msg="Loading journal" path=/run/spire/data/journal.pem subsystem_name=ca_manager
    time="2022-05-18T11:07:39Z" level=info msg="Journal loaded" jwt_keys=0 subsystem_name=ca_manager x509_cas=0
    time="2022-05-18T11:07:39Z" level=debug msg="Preparing X509 CA" slot=A subsystem_name=ca_manager
    time="2022-05-18T11:07:39Z" level=info msg="X509 CA prepared" expiration="2022-05-19T11:07:39Z" issued_at="2022-05-18T11:07:39Z" self_signed=false slot=A subsystem_name=ca_manager
    time="2022-05-18T11:07:39Z" level=info msg="X509 CA activated" expiration="2022-05-19T11:07:39Z" issued_at="2022-05-18T11:07:39Z" slot=A subsystem_name=ca_manager
    time="2022-05-18T11:07:39Z" level=debug msg="Successfully rotated X.509 CA" subsystem_name=ca_manager trust_domain_id="spiffe://example.org" ttl=86399.559774583
    time="2022-05-18T11:07:39Z" level=debug msg="Preparing JWT key" slot=A subsystem_name=ca_manager
    time="2022-05-18T11:07:39Z" level=warning msg="UpstreamAuthority plugin does not support JWT-SVIDs. Workloads managed by this server may have trouble communicating with workloads outside this cluster when using JWT-SVIDs." plugin_name=disk subsystem_name=ca_manager
    time="2022-05-18T11:07:39Z" level=info msg="JWT key prepared" expiration="2022-05-19T11:07:39Z" issued_at="2022-05-18T11:07:39Z" slot=A subsystem_name=ca_manager
    time="2022-05-18T11:07:39Z" level=info msg="JWT key activated" expiration="2022-05-19T11:07:39Z" issued_at="2022-05-18T11:07:39Z" slot=A subsystem_name=ca_manager
    time="2022-05-18T11:07:39Z" level=debug msg="Rotating server SVID" subsystem_name=svid_rotator
    time="2022-05-18T11:07:39Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:07:39Z" spiffe_id="spiffe://example.org/spire/server" subsystem_name=ca
    time="2022-05-18T11:07:39Z" level=info msg="Building in-memory entry cache" subsystem_name=endpoints
    time="2022-05-18T11:07:39Z" level=info msg="Completed building in-memory entry cache" subsystem_name=endpoints
    time="2022-05-18T11:07:39Z" level=debug msg="Initializing API endpoints" subsystem_name=endpoints
    time="2022-05-18T11:07:39Z" level=info msg="Starting TCP server" address="[::]:8081" subsystem_name=endpoints
    time="2022-05-18T11:07:39Z" level=info msg="Starting UDS server" address=/tmp/spire-server/private/api.sock subsystem_name=endpoints
    time="2022-05-18T11:07:40Z" level=debug msg="Starting checker" name=catalog.datastore subsystem_name=health
    time="2022-05-18T11:07:40Z" level=debug msg="Starting checker" name=server.ca subsystem_name=health
    time="2022-05-18T11:07:40Z" level=debug msg="Starting checker" name=server.ca.manager subsystem_name=health
    time="2022-05-18T11:07:40Z" level=debug msg="Starting checker" name=server subsystem_name=health
    time="2022-05-18T11:07:40Z" level=info msg="Serving health checks" address="0.0.0.0:8080" subsystem_name=health
    time="2022-05-18T11:07:47Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:07:47Z" spiffe_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/1be11f43-e9e7-41b6-80a4-4dc816b1e889" subsystem_name=ca
    time="2022-05-18T11:07:47Z" level=info msg="Agent attestation request completed" address="192.168.1.7:5204" agent_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/1be11f43-e9e7-41b6-80a4-4dc816b1e889" caller_addr="192.168.1.7:5204" method=AttestAgent node_attestor_type=k8s_sat request_id=7bd55951-34d2-4042-b3ac-18f165a0dc9b service=agent.v1.Agent subsystem_name=api
    time="2022-05-18T11:07:48Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:07:48Z" spiffe_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/64823730-d786-4277-a9bc-106b8eb2f4ff" subsystem_name=ca
    time="2022-05-18T11:07:48Z" level=info msg="Agent attestation request completed" address="192.168.1.6:5510" agent_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/64823730-d786-4277-a9bc-106b8eb2f4ff" caller_addr="192.168.1.6:5510" method=AttestAgent node_attestor_type=k8s_sat request_id=76052c90-a828-4a94-a16a-5d6be221e603 service=agent.v1.Agent subsystem_name=api
    time="2022-05-18T11:11:17Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:11:17Z" spiffe_id="spiffe://example.org/ns/spire/sa/spire-agent" subsystem_name=ca
    time="2022-05-18T11:11:17Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:11:17Z" spiffe_id="spiffe://example.org/ns/istio-system/sa/istio-ingressgateway-service-account" subsystem_name=ca
    time="2022-05-18T11:11:17Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:11:17Z" spiffe_id="spiffe://example.org/ns/default/sa/default" subsystem_name=ca
    time="2022-05-18T11:11:17Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:11:17Z" spiffe_id="spiffe://example.org/ns/spire/sa/spire-agent" subsystem_name=ca
    time="2022-05-18T11:11:17Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:11:17Z" spiffe_id="spiffe://example.org/ns/istio-system/sa/istio-ingressgateway-service-account" subsystem_name=ca
    time="2022-05-18T11:11:17Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:11:17Z" spiffe_id="spiffe://example.org/ns/default/sa/default" subsystem_name=ca
    time="2022-05-18T11:11:32Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:11:32Z" spiffe_id="spiffe://example.org/ns/test/sa/httpbin" subsystem_name=ca
    time="2022-05-18T11:11:32Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:11:32Z" spiffe_id="spiffe://example.org/ns/test/sa/httpbin" subsystem_name=ca
    time="2022-05-18T11:13:55Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:13:55Z" spiffe_id="spiffe://example.org/ns/test/sa/sleep" subsystem_name=ca
    time="2022-05-18T11:13:55Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:13:55Z" spiffe_id="spiffe://example.org/ns/test/sa/sleep" subsystem_name=ca
    time="2022-05-18T11:28:36Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:36Z" spiffe_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/be1300e4-e93d-40f6-b987-6cb42d20e237" subsystem_name=ca
    time="2022-05-18T11:28:36Z" level=info msg="Agent attestation request completed" address="192.168.1.7:26058" agent_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/be1300e4-e93d-40f6-b987-6cb42d20e237" caller_addr="192.168.1.7:26058" method=AttestAgent node_attestor_type=k8s_sat request_id=b00ead65-d001-46e3-89f5-5234442697c9 service=agent.v1.Agent subsystem_name=api
    time="2022-05-18T11:28:42Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:42Z" spiffe_id="spiffe://example.org/ns/test/sa/httpbin" subsystem_name=ca
    time="2022-05-18T11:28:42Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:42Z" spiffe_id="spiffe://example.org/ns/test/sa/sleep" subsystem_name=ca
    time="2022-05-18T11:28:42Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:42Z" spiffe_id="spiffe://example.org/ns/spire/sa/spire-agent" subsystem_name=ca
    time="2022-05-18T11:28:42Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:42Z" spiffe_id="spiffe://example.org/ns/istio-system/sa/istio-ingressgateway-service-account" subsystem_name=ca
    time="2022-05-18T11:28:42Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:42Z" spiffe_id="spiffe://example.org/ns/default/sa/default" subsystem_name=ca
    time="2022-05-18T11:28:42Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:42Z" spiffe_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/bd4ae7d9-4fbf-4ba0-931c-efc4da0b6e08" subsystem_name=ca
    time="2022-05-18T11:28:42Z" level=info msg="Agent attestation request completed" address="192.168.1.6:37898" agent_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/bd4ae7d9-4fbf-4ba0-931c-efc4da0b6e08" caller_addr="192.168.1.6:37898" method=AttestAgent node_attestor_type=k8s_sat request_id=7967a14e-ab5c-4bf9-aaef-21e2d43cf413 service=agent.v1.Agent subsystem_name=api
    time="2022-05-18T11:28:47Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:47Z" spiffe_id="spiffe://example.org/ns/spire/sa/spire-agent" subsystem_name=ca
    time="2022-05-18T11:28:47Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:47Z" spiffe_id="spiffe://example.org/ns/istio-system/sa/istio-ingressgateway-service-account" subsystem_name=ca
    time="2022-05-18T11:28:47Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:47Z" spiffe_id="spiffe://example.org/ns/default/sa/default" subsystem_name=ca
    time="2022-05-18T11:28:47Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:47Z" spiffe_id="spiffe://example.org/ns/test/sa/httpbin" subsystem_name=ca
    time="2022-05-18T11:28:47Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:47Z" spiffe_id="spiffe://example.org/ns/test/sa/sleep" subsystem_name=ca
    time="2022-05-18T11:30:53Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:30:53Z" spiffe_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/93fe2161-95cc-4454-84d4-a510616a5dcf" subsystem_name=ca
    time="2022-05-18T11:30:53Z" level=info msg="Agent attestation request completed" address="192.168.1.6:42279" agent_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/93fe2161-95cc-4454-84d4-a510616a5dcf" caller_addr="192.168.1.6:42279" method=AttestAgent node_attestor_type=k8s_sat request_id=c3bf5aac-0c6b-4ae5-9fb1-86bcd668f8ec service=agent.v1.Agent subsystem_name=api
    time="2022-05-18T11:30:57Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:30:57Z" spiffe_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/ca6a5d08-3517-43a1-8ffc-dc0519a2dea7" subsystem_name=ca
    time="2022-05-18T11:30:57Z" level=info msg="Agent attestation request completed" address="192.168.1.7:51286" agent_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/ca6a5d08-3517-43a1-8ffc-dc0519a2dea7" caller_addr="192.168.1.7:51286" method=AttestAgent node_attestor_type=k8s_sat request_id=9dd8a63e-5ace-4381-8a5a-80e2a1e5f0d1 service=agent.v1.Agent subsystem_name=api
    time="2022-05-18T11:30:58Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:30:58Z" spiffe_id="spiffe://example.org/ns/test/sa/sleep" subsystem_name=ca
    time="2022-05-18T11:30:58Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:30:58Z" spiffe_id="spiffe://example.org/ns/spire/sa/spire-agent" subsystem_name=ca
    time="2022-05-18T11:30:58Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:30:58Z" spiffe_id="spiffe://example.org/ns/istio-system/sa/istio-ingressgateway-service-account" subsystem_name=ca
    time="2022-05-18T11:30:58Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:30:58Z" spiffe_id="spiffe://example.org/ns/default/sa/default" subsystem_name=ca
    time="2022-05-18T11:30:58Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:30:58Z" spiffe_id="spiffe://example.org/ns/test/sa/httpbin" subsystem_name=ca
    time="2022-05-18T11:31:02Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:31:02Z" spiffe_id="spiffe://example.org/ns/test/sa/sleep" subsystem_name=ca
    time="2022-05-18T11:31:02Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:31:02Z" spiffe_id="spiffe://example.org/ns/spire/sa/spire-agent" subsystem_name=ca
    time="2022-05-18T11:31:02Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:31:02Z" spiffe_id="spiffe://example.org/ns/istio-system/sa/istio-ingressgateway-service-account" subsystem_name=ca
    time="2022-05-18T11:31:02Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:31:02Z" spiffe_id="spiffe://example.org/ns/default/sa/default" subsystem_name=ca
    time="2022-05-18T11:31:02Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:31:02Z" spiffe_id="spiffe://example.org/ns/test/sa/httpbin" subsystem_name=ca
    time="2022-05-18T11:32:02Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:02Z" spiffe_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/32c3378f-b8b0-4df6-b6a0-18fb3bf824f7" subsystem_name=ca
    time="2022-05-18T11:32:02Z" level=info msg="Agent attestation request completed" address="192.168.1.7:20470" agent_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/32c3378f-b8b0-4df6-b6a0-18fb3bf824f7" caller_addr="192.168.1.7:20470" method=AttestAgent node_attestor_type=k8s_sat request_id=37844d8e-27f5-42c6-a7be-019bfe6f9c82 service=agent.v1.Agent subsystem_name=api
    time="2022-05-18T11:32:02Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:02Z" spiffe_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/7e068ae9-00f4-4786-9191-2b61cb7f08e9" subsystem_name=ca
    time="2022-05-18T11:32:02Z" level=info msg="Agent attestation request completed" address="192.168.1.6:24604" agent_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/7e068ae9-00f4-4786-9191-2b61cb7f08e9" caller_addr="192.168.1.6:24604" method=AttestAgent node_attestor_type=k8s_sat request_id=60c6024e-eaa6-4f21-8d24-7f4dfff0eb24 service=agent.v1.Agent subsystem_name=api
    time="2022-05-18T11:32:07Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:07Z" spiffe_id="spiffe://example.org/ns/test/sa/httpbin" subsystem_name=ca
    time="2022-05-18T11:32:07Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:07Z" spiffe_id="spiffe://example.org/ns/test/sa/sleep" subsystem_name=ca
    time="2022-05-18T11:32:07Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:07Z" spiffe_id="spiffe://example.org/ns/spire/sa/spire-agent" subsystem_name=ca
    time="2022-05-18T11:32:07Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:07Z" spiffe_id="spiffe://example.org/ns/istio-system/sa/istio-ingressgateway-service-account" subsystem_name=ca
    time="2022-05-18T11:32:07Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:07Z" spiffe_id="spiffe://example.org/ns/default/sa/default" subsystem_name=ca
    time="2022-05-18T11:32:07Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:07Z" spiffe_id="spiffe://example.org/ns/spire/sa/spire-agent" subsystem_name=ca
    time="2022-05-18T11:32:07Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:07Z" spiffe_id="spiffe://example.org/ns/istio-system/sa/istio-ingressgateway-service-account" subsystem_name=ca
    time="2022-05-18T11:32:07Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:07Z" spiffe_id="spiffe://example.org/ns/default/sa/default" subsystem_name=ca
    time="2022-05-18T11:32:07Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:07Z" spiffe_id="spiffe://example.org/ns/test/sa/httpbin" subsystem_name=ca
    time="2022-05-18T11:32:07Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:07Z" spiffe_id="spiffe://example.org/ns/test/sa/sleep" subsystem_name=ca
    time="2022-05-18T11:37:34Z" level=debug msg="Rotating server SVID" subsystem_name=svid_rotator
    time="2022-05-18T11:37:34Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:37:34Z" spiffe_id="spiffe://example.org/spire/server" subsystem_name=ca
    

    the agent list in spire-server.

    /opt/spire/bin # /opt/spire/bin/spire-server agent list
    Found 2 attested agents:
    
    SPIFFE ID         : spiffe://example.org/spire/agent/k8s_sat/demo-cluster/56d7ed58-14ae-4cf5-9880-ffc123b314c1
    Attestation type  : k8s_sat
    Expiration time   : 2022-05-18 13:02:00 +0000 UTC
    Serial number     : 319040273546637254342952962498965679004
    
    SPIFFE ID         : spiffe://example.org/spire/agent/k8s_sat/demo-cluster/63588790-f382-492b-a324-359999d3a394
    Attestation type  : k8s_sat
    Expiration time   : 2022-05-18 13:02:00 +0000 UTC
    Serial number     : 277908708266485423488447531291378329815
    

    the entry in spire-server.

    Found 5 entries
    Entry ID         : 7ec60e3e-147d-444b-b026-a244279a03c6
    SPIFFE ID        : spiffe://example.org/ns/default/sa/default
    Parent ID        : spiffe://example.org/ns/spire/sa/spire-agent
    Revision         : 0
    TTL              : default
    Selector         : k8s:ns:default
    Selector         : k8s:sa:default
    
    Entry ID         : cc99cf5e-cfd7-44ca-966d-b6313a540447
    SPIFFE ID        : spiffe://example.org/ns/istio-system/sa/istio-ingressgateway-service-account
    Parent ID        : spiffe://example.org/ns/spire/sa/spire-agent
    Revision         : 0
    TTL              : default
    Selector         : k8s:ns:istio-system
    Selector         : k8s:sa:istio-ingressgateway-service-account
    
    Entry ID         : a689cef7-6972-46b1-8277-b3212f31c230
    SPIFFE ID        : spiffe://example.org/ns/spire/sa/spire-agent
    Parent ID        : spiffe://example.org/spire/server
    Revision         : 0
    TTL              : default
    Selector         : k8s_sat:agent_ns:spire
    Selector         : k8s_sat:agent_sa:spire-agent
    Selector         : k8s_sat:cluster:demo-cluster
    
    Entry ID         : be9bd775-85cb-4ffa-b914-5493c651d264
    SPIFFE ID        : spiffe://example.org/ns/test/sa/httpbin
    Parent ID        : spiffe://example.org/ns/spire/sa/spire-agent
    Revision         : 0
    TTL              : default
    Selector         : k8s:ns:test
    Selector         : k8s:sa:httpbin
    
    Entry ID         : 3615702c-293e-4d39-9785-16a546c2462b
    SPIFFE ID        : spiffe://example.org/ns/test/sa/sleep
    Parent ID        : spiffe://example.org/ns/spire/sa/spire-agent
    Revision         : 0
    TTL              : default
    Selector         : k8s:ns:test
    Selector         : k8s:sa:sleep
    
    
    priority/backlog 
    opened by tanjunchen 31
  • Spire do not federate between k8s clusters which implements Istio multi-primary on different networks

    Spire do not federate between k8s clusters which implements Istio multi-primary on different networks

    An application is make to federate between two trust domain using spire and Istio. The app being sleep app and helloworld app. The sleep application is able to discover the helloworld service in the other cluster. spire-agent is managing the SDS interface for envoy and its helping to mint the required certs for the envoy. spire is configured with auto attestation of workload.

    A sleep app in cluster is able to reach and connect the helloworld app in the same cluster. however when the same sleep application is trying to connect to the helloworld app in the other cluster it fails with SSL error: CERTIFICATE_VERIFY_FAILED

    Istio is configured for multi-primary with different network ( meaning there is no direct connection between the pods across cluster boundary). An additional east-west ( ew gw ) is installed. This ew gw has a public address for the clusters to reach and it does SNI pass-through for the traffic to directly reach the service hosted inside the cluster. The service is protected with envoy. Envoy validates all connection for mTLS. on successful validation it would allow the communication to get established.

    In case of federation the mTLS will be between two different trust domain and with spire configured correctly it would perform the trust bundle exchange and make the federated CA available to the envoy sitting next to the service.

    Expectation: As the CA and federated CA has made available to the helloworld and sleep service, the connection between the sleep application and helloworld should have gone through. While testing it fails and hence this issue has been raised.

    Topology

    note: entire configuration can be found here: https://github.com/sudeeptoroy/spirefed

    1. brought up two kind clusters: kind-aws-cluster and kind-google-cluster
    2. configured spire for these two clusters and put them in different trust domain: aws.com and google.com
    3. brought up istio on both clusters in multi-primary mode. i have followed this article to bringup istio: https://istio.io/latest/docs/setup/install/multicluster/multi-primary_multi-network/
    4. brought up sample app to rest the federation. 4a. sleep app on aws-cluster 4.b helloworld app on google-cluster 4.c from the sleep app execute curl to helloworld on the other cluster

    here is pictorial representation of the topolgy.

    image

    Result: The curl fails with this error: kubectl exec --context=kind-aws-cluster -n sample -c sleep sleep-95d8696-bk822 -- curl -sS helloworld.sample:5000/hello upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED

    Steps to reproduce: Clone: https://github.com/sudeeptoroy/spirefed Follow the readme

    Observation:

    when i run the helloworld istio-proxy in trace mode i see the following logs:

    2022-10-15T10:39:52.419167Z	debug	envoy filter	original_dst: new connection accepted
    2022-10-15T10:39:52.419262Z	trace	envoy filter	original_dst: set destination to 10.241.1.8:5000
    2022-10-15T10:39:52.419278Z	debug	envoy filter	tls inspector: new connection accepted
    2022-10-15T10:39:52.419291Z	trace	envoy filter	tls inspector: recv: 517
    2022-10-15T10:39:52.419313Z	trace	envoy filter	tls:onALPN(), ALPN: istio-http/1.1,istio,http/1.1
    2022-10-15T10:39:52.419330Z	debug	envoy filter	tls:onServerName(), requestedServerName: outbound_.5000_._.helloworld.sample.svc.cluster.local
    2022-10-15T10:39:52.419399Z	trace	envoy misc	enableTimer called on 0x55e2b1ee4080 for 3600000ms, min is 3600000ms
    2022-10-15T10:39:52.419448Z	debug	envoy conn_handler	[C146] new connection from 10.241.1.6:58350
    2022-10-15T10:39:52.419474Z	trace	envoy connection	[C146] socket event: 3
    2022-10-15T10:39:52.419486Z	trace	envoy connection	[C146] write ready
    2022-10-15T10:39:52.420578Z	trace	envoy connection	[C146] ssl error occurred while read: WANT_READ
    2022-10-15T10:39:52.420705Z	trace	envoy connection	[C146] read ready. dispatch_buffered_data=0
    2022-10-15T10:39:52.420744Z	trace	envoy connection	[C146] ssl error occurred while read: WANT_READ
    2022-10-15T10:39:52.424542Z	trace	envoy connection	[C146] socket event: 3
    2022-10-15T10:39:52.424707Z	trace	envoy connection	[C146] write ready
    2022-10-15T10:39:52.424750Z	trace	envoy connection	[C146] ssl error occurred while read: SSL
    2022-10-15T10:39:52.424767Z	debug	envoy connection	[C146] TLS error: 268436502:SSL routines:**OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN**
    2022-10-15T10:39:52.424780Z	debug	envoy connection	[C146] closing socket: 0
    2022-10-15T10:39:52.424824Z	debug	envoy connection	[C146] TLS error: 268436502:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN
    2022-10-15T10:39:52.424859Z	trace	envoy connection	[C146] raising connection event 0
    2022-10-15T10:39:52.424905Z	trace	envoy conn_handler	[C146] connection on event 0
    2022-10-15T10:39:52.424918Z	debug	envoy conn_handler	[C146] adding to cleanup list
    2022-10-15T10:39:52.424929Z	trace	envoy main	item added to deferred deletion list (size=1)
    2022-10-15T10:39:52.424941Z	trace	envoy main	clearing deferred deletion list (size=1)
    2022-10-15T10:39:52.432818Z	debug	envoy filter	original_dst: new connection accepted
    2022-10-15T10:39:52.432913Z	trace	envoy filter	original_dst: set destination to 10.241.1.8:5000
    2022-10-15T10:39:52.432929Z	debug	envoy filter	tls inspector: new connection accepted
    2022-10-15T10:39:52.432942Z	trace	envoy filter	tls inspector: recv: 517
    2022-10-15T10:39:52.432967Z	trace	envoy filter	tls:onALPN(), ALPN: istio-http/1.1,istio,http/1.1
    2022-10-15T10:39:52.432984Z	debug	envoy filter	tls:onServerName(), requestedServerName: outbound_.5000_._.helloworld.sample.svc.cluster.local
    2022-10-15T10:39:52.433065Z	trace	envoy misc	enableTimer called on 0x55e2b1ee4080 for 3600000ms, min is 3600000ms
    2022-10-15T10:39:52.433086Z	debug	envoy conn_handler	[C147] new connection from 10.241.1.6:58354
    2022-10-15T10:39:52.433166Z	trace	envoy connection	[C147] socket event: 3
    2022-10-15T10:39:52.433209Z	trace	envoy connection	[C147] write ready
    2022-10-15T10:39:52.433493Z	trace	envoy connection	[C147] ssl error occurred while read: WANT_READ
    2022-10-15T10:39:52.433679Z	trace	envoy connection	[C147] read ready. dispatch_buffered_data=0
    2022-10-15T10:39:52.433698Z	trace	envoy connection	[C147] ssl error occurred while read: WANT_READ
    2022-10-15T10:39:52.438301Z	trace	envoy connection	[C147] socket event: 3
    2022-10-15T10:39:52.438685Z	trace	envoy connection	[C147] write ready
    2022-10-15T10:39:52.438821Z	trace	envoy connection	[C147] ssl error occurred while read: SSL
    2022-10-15T10:39:52.438896Z	debug	envoy connection	[C147] TLS error: 268436502:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN
    2022-10-15T10:39:52.439010Z	debug	envoy connection	[C147] closing socket: 0
    2022-10-15T10:39:52.440473Z	debug	envoy connection	[C147] TLS error: 268436502:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN
    2022-10-15T10:39:52.440731Z	trace	envoy connection	[C147] raising connection event 0
    2022-10-15T10:39:52.440776Z	trace	envoy conn_handler	[C147] connection on event 0
    2022-10-15T10:39:52.440874Z	debug	envoy conn_handler	[C147] adding to cleanup list
    2022-10-15T10:39:52.440920Z	trace	envoy main	item added to deferred deletion list (size=1)
    2022-10-15T10:39:52.440939Z	trace	envoy main	clearing deferred deletion list (size=1)
    2022-10-15T10:39:52.464980Z	debug	envoy filter	original_dst: new connection accepted
    2022-10-15T10:39:52.465038Z	trace	envoy filter	original_dst: set destination to 10.241.1.8:5000
    2022-10-15T10:39:52.465045Z	debug	envoy filter	tls inspector: new connection accepted
    2022-10-15T10:39:52.465052Z	trace	envoy filter	tls inspector: recv: 517
    2022-10-15T10:39:52.465073Z	trace	envoy filter	tls:onALPN(), ALPN: istio-http/1.1,istio,http/1.1
    2022-10-15T10:39:52.465083Z	debug	envoy filter	tls:onServerName(), requestedServerName: outbound_.5000_._.helloworld.sample.svc.cluster.local
    2022-10-15T10:39:52.465158Z	trace	envoy misc	enableTimer called on 0x55e2b19a1c80 for 3600000ms, min is 3600000ms
    2022-10-15T10:39:52.465199Z	debug	envoy conn_handler	[C148] new connection from 10.241.1.6:58360
    2022-10-15T10:39:52.465220Z	trace	envoy connection	[C148] socket event: 3
    2022-10-15T10:39:52.465224Z	trace	envoy connection	[C148] write ready
    2022-10-15T10:39:52.465663Z	trace	envoy connection	[C148] ssl error occurred while read: WANT_READ
    2022-10-15T10:39:52.465673Z	trace	envoy connection	[C148] read ready. dispatch_buffered_data=0
    2022-10-15T10:39:52.465678Z	trace	envoy connection	[C148] ssl error occurred while read: WANT_READ
    2022-10-15T10:39:52.473615Z	trace	envoy connection	[C148] socket event: 3
    2022-10-15T10:39:52.473658Z	trace	envoy connection	[C148] write ready
    2022-10-15T10:39:52.473751Z	trace	envoy connection	[C148] ssl error occurred while read: SSL
    2022-10-15T10:39:52.473760Z	debug	envoy connection	[C148] TLS error: 268436502:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN
    2022-10-15T10:39:52.473765Z	debug	envoy connection	[C148] closing socket: 0
    2022-10-15T10:39:52.473908Z	debug	envoy connection	[C148] TLS error: 268436502:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN
    2022-10-15T10:39:52.473934Z	trace	envoy connection	[C148] raising connection event 0
    2022-10-15T10:39:52.473942Z	trace	envoy conn_handler	[C148] connection on event 0
    2022-10-15T10:39:52.473944Z	debug	envoy conn_handler	[C148] adding to cleanup list
    

    From the logs and envoy config at helloworld:

    The first filter at envoy is "original_dst" where tls inspector should route it to "outbound|5000||helloworld.sample.svc.cluster.local". And for some reason this is not accepting the mTLS from the other domain "aws.com".

    Listener dump: check the last section: original_dst

    {
      "name": "virtualInbound",
      "active_state": {
        "version_info": "2022-10-15T10:34:48Z/11",
        "listener": {
          "@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
          "name": "virtualInbound",
          "address": {
            "socket_address": {
              "address": "0.0.0.0",
              "port_value": 15006
            }
          },
          "filter_chains": [
            {
              "filter_chain_match": {
                "destination_port": 15006
              },
              "filters": [
                {
                  "name": "istio.metadata_exchange",
                  "typed_config": {
                    "@type": "type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange",
                    "protocol": "istio-peer-exchange"
                  }
                },
                {
                  "name": "istio.stats",
                  "typed_config": {
                    "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                    "type_url": "type.googleapis.com/envoy.extensions.filters.network.wasm.v3.Wasm",
                    "value": {
                      "config": {
                        "root_id": "stats_inbound",
                        "vm_config": {
                          "vm_id": "tcp_stats_inbound",
                          "runtime": "envoy.wasm.runtime.null",
                          "code": {
                            "local": {
                              "inline_string": "envoy.wasm.stats"
                            }
                          }
                        },
                        "configuration": {
                          "@type": "type.googleapis.com/google.protobuf.StringValue",
                          "value": "{\n  \"debug\": \"false\",\n  \"stat_prefix\": \"istio\",\n  \"metrics\": [\n    {\n      \"dimensions\": {\n        \"destination_cluster\": \"node.metadata['CLUSTER_ID']\",\n        \"source_cluster\": \"downstream_peer.cluster_id\"\n      }\n    }\n  ]\n}\n"
                        }
                      }
                    }
                  }
                },
                {
                  "name": "envoy.filters.network.tcp_proxy",
                  "typed_config": {
                    "@type": "type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy",
                    "stat_prefix": "BlackHoleCluster",
                    "cluster": "BlackHoleCluster"
                  }
                }
              ],
              "name": "virtualInbound-blackhole"
            },
            {
              "filter_chain_match": {
                "prefix_ranges": [
                  {
                    "address_prefix": "0.0.0.0",
                    "prefix_len": 0
                  }
                ],
                "transport_protocol": "tls",
                "application_protocols": [
                  "istio-http/1.0",
                  "istio-http/1.1",
                  "istio-h2"
                ]
              },
              "filters": [
                {
                  "name": "istio.metadata_exchange",
                  "typed_config": {
                    "@type": "type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange",
                    "protocol": "istio-peer-exchange"
                  }
                },
                {
                  "name": "envoy.filters.network.http_connection_manager",
                  "typed_config": {
                    "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
                    "stat_prefix": "InboundPassthroughClusterIpv4",
                    "route_config": {
                      "name": "InboundPassthroughClusterIpv4",
                      "virtual_hosts": [
                        {
                          "name": "inbound|http|0",
                          "domains": [
                            "*"
                          ],
                          "routes": [
                            {
                              "match": {
                                "prefix": "/"
                              },
                              "route": {
                                "cluster": "InboundPassthroughClusterIpv4",
                                "timeout": "0s",
                                "max_stream_duration": {
                                  "max_stream_duration": "0s",
                                  "grpc_timeout_header_max": "0s"
                                }
                              },
                              "decorator": {
                                "operation": ":0/*"
                              },
                              "name": "default"
                            }
                          ]
                        }
                      ],
                      "validate_clusters": false
                    },
                    "http_filters": [
                      {
                        "name": "istio.metadata_exchange",
                        "typed_config": {
                          "@type": "type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm",
                          "config": {
                            "vm_config": {
                              "runtime": "envoy.wasm.runtime.null",
                              "code": {
                                "local": {
                                  "inline_string": "envoy.wasm.metadata_exchange"
                                }
                              }
                            },
                            "configuration": {
                              "@type": "type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange"
                            }
                          }
                        }
                      },
                      {
                        "name": "envoy.filters.http.fault",
                        "typed_config": {
                          "@type": "type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault"
                        }
                      },
                      {
                        "name": "envoy.filters.http.cors",
                        "typed_config": {
                          "@type": "type.googleapis.com/envoy.extensions.filters.http.cors.v3.Cors"
                        }
                      },
                      {
                        "name": "istio.stats",
                        "typed_config": {
                          "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                          "type_url": "type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm",
                          "value": {
                            "config": {
                              "root_id": "stats_inbound",
                              "vm_config": {
                                "vm_id": "stats_inbound",
                                "runtime": "envoy.wasm.runtime.null",
                                "code": {
                                  "local": {
                                    "inline_string": "envoy.wasm.stats"
                                  }
                                }
                              },
                              "configuration": {
                                "@type": "type.googleapis.com/google.protobuf.StringValue",
                                "value": "{\n  \"debug\": \"false\",\n  \"stat_prefix\": \"istio\",\n  \"disable_host_header_fallback\": true,\n  \"metrics\": [\n    {\n      \"dimensions\": {\n        \"destination_cluster\": \"node.metadata['CLUSTER_ID']\",\n        \"source_cluster\": \"downstream_peer.cluster_id\"\n      }\n    }\n  ]\n}\n"
                              }
                            }
                          }
                        }
                      },
                      {
                        "name": "envoy.filters.http.router",
                        "typed_config": {
                          "@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router"
                        }
                      }
                    ],
                    "tracing": {
                      "client_sampling": {
                        "value": 100
                      },
                      "random_sampling": {
                        "value": 1
                      },
                      "overall_sampling": {
                        "value": 100
                      },
                      "custom_tags": [
                        {
                          "tag": "istio.authorization.dry_run.allow_policy.name",
                          "metadata": {
                            "kind": {
                              "request": {}
                            },
                            "metadata_key": {
                              "key": "envoy.filters.http.rbac",
                              "path": [
                                {
                                  "key": "istio_dry_run_allow_shadow_effective_policy_id"
                                }
                              ]
                            }
                          }
                        },
                        {
                          "tag": "istio.authorization.dry_run.allow_policy.result",
                          "metadata": {
                            "kind": {
                              "request": {}
                            },
                            "metadata_key": {
                              "key": "envoy.filters.http.rbac",
                              "path": [
                                {
                                  "key": "istio_dry_run_allow_shadow_engine_result"
                                }
                              ]
                            }
                          }
                        },
                        {
                          "tag": "istio.authorization.dry_run.deny_policy.name",
                          "metadata": {
                            "kind": {
                              "request": {}
                            },
                            "metadata_key": {
                              "key": "envoy.filters.http.rbac",
                              "path": [
                                {
                                  "key": "istio_dry_run_deny_shadow_effective_policy_id"
                                }
                              ]
                            }
                          }
                        },
                        {
                          "tag": "istio.authorization.dry_run.deny_policy.result",
                          "metadata": {
                            "kind": {
                              "request": {}
                            },
                            "metadata_key": {
                              "key": "envoy.filters.http.rbac",
                              "path": [
                                {
                                  "key": "istio_dry_run_deny_shadow_engine_result"
                                }
                              ]
                            }
                          }
                        },
                        {
                          "tag": "istio.canonical_revision",
                          "literal": {
                            "value": "v2"
                          }
                        },
                        {
                          "tag": "istio.canonical_service",
                          "literal": {
                            "value": "helloworld"
                          }
                        },
                        {
                          "tag": "istio.mesh_id",
                          "literal": {
                            "value": "devup-mesh"
                          }
                        },
                        {
                          "tag": "istio.namespace",
                          "literal": {
                            "value": "sample"
                          }
                        }
                      ]
                    },
                    "server_name": "istio-envoy",
                    "use_remote_address": false,
                    "forward_client_cert_details": "APPEND_FORWARD",
                    "set_current_client_cert_details": {
                      "subject": true,
                      "dns": true,
                      "uri": true
                    },
                    "upgrade_configs": [
                      {
                        "upgrade_type": "websocket"
                      }
                    ],
                    "stream_idle_timeout": "0s",
                    "normalize_path": true,
                    "request_id_extension": {
                      "typed_config": {
                        "@type": "type.googleapis.com/envoy.extensions.request_id.uuid.v3.UuidRequestIdConfig",
                        "use_request_id_for_trace_sampling": true
                      }
                    },
                    "path_with_escaped_slashes_action": "KEEP_UNCHANGED"
                  }
                }
              ],
              "transport_socket": {
                "name": "envoy.transport_sockets.tls",
                "typed_config": {
                  "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext",
                  "common_tls_context": {
                    "tls_params": {
                      "tls_minimum_protocol_version": "TLSv1_2",
                      "tls_maximum_protocol_version": "TLSv1_3",
                      "cipher_suites": [
                        "ECDHE-ECDSA-AES256-GCM-SHA384",
                        "ECDHE-RSA-AES256-GCM-SHA384",
                        "ECDHE-ECDSA-AES128-GCM-SHA256",
                        "ECDHE-RSA-AES128-GCM-SHA256",
                        "AES256-GCM-SHA384",
                        "AES128-GCM-SHA256"
                      ]
                    },
                    "alpn_protocols": [
                      "h2",
                      "http/1.1"
                    ],
                    "tls_certificate_sds_secret_configs": [
                      {
                        "name": "default",
                        "sds_config": {
                          "api_config_source": {
                            "api_type": "GRPC",
                            "grpc_services": [
                              {
                                "envoy_grpc": {
                                  "cluster_name": "sds-grpc"
                                }
                              }
                            ],
                            "set_node_on_first_message_only": true,
                            "transport_api_version": "V3"
                          },
                          "initial_fetch_timeout": "0s",
                          "resource_api_version": "V3"
                        }
                      }
                    ],
                    "combined_validation_context": {
                      "default_validation_context": {
                        "match_subject_alt_names": [
                          {
                            "prefix": "spiffe://google.com/"
                          }
                        ]
                      },
                      "validation_context_sds_secret_config": {
                        "name": "ROOTCA",
                        "sds_config": {
                          "api_config_source": {
                            "api_type": "GRPC",
                            "grpc_services": [
                              {
                                "envoy_grpc": {
                                  "cluster_name": "sds-grpc"
                                }
                              }
                            ],
                            "set_node_on_first_message_only": true,
                            "transport_api_version": "V3"
                          },
                          "initial_fetch_timeout": "0s",
                          "resource_api_version": "V3"
                        }
                      }
                    }
                  },
                  "require_client_certificate": true
                }
              },
              "name": "virtualInbound-catchall-http"
            },
            {
              "filter_chain_match": {
                "prefix_ranges": [
                  {
                    "address_prefix": "0.0.0.0",
                    "prefix_len": 0
                  }
                ],
                "transport_protocol": "tls"
              },
              "filters": [
                {
                  "name": "istio.metadata_exchange",
                  "typed_config": {
                    "@type": "type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange",
                    "protocol": "istio-peer-exchange"
                  }
                },
                {
                  "name": "istio.stats",
                  "typed_config": {
                    "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                    "type_url": "type.googleapis.com/envoy.extensions.filters.network.wasm.v3.Wasm",
                    "value": {
                      "config": {
                        "root_id": "stats_inbound",
                        "vm_config": {
                          "vm_id": "tcp_stats_inbound",
                          "runtime": "envoy.wasm.runtime.null",
                          "code": {
                            "local": {
                              "inline_string": "envoy.wasm.stats"
                            }
                          }
                        },
                        "configuration": {
                          "@type": "type.googleapis.com/google.protobuf.StringValue",
                          "value": "{\n  \"debug\": \"false\",\n  \"stat_prefix\": \"istio\",\n  \"metrics\": [\n    {\n      \"dimensions\": {\n        \"destination_cluster\": \"node.metadata['CLUSTER_ID']\",\n        \"source_cluster\": \"downstream_peer.cluster_id\"\n      }\n    }\n  ]\n}\n"
                        }
                      }
                    }
                  }
                },
                {
                  "name": "envoy.filters.network.tcp_proxy",
                  "typed_config": {
                    "@type": "type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy",
                    "stat_prefix": "InboundPassthroughClusterIpv4",
                    "cluster": "InboundPassthroughClusterIpv4"
                  }
                }
              ],
              "transport_socket": {
                "name": "envoy.transport_sockets.tls",
                "typed_config": {
                  "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext",
                  "common_tls_context": {
                    "tls_params": {
                      "tls_minimum_protocol_version": "TLSv1_2",
                      "tls_maximum_protocol_version": "TLSv1_3",
                      "cipher_suites": [
                        "ECDHE-ECDSA-AES256-GCM-SHA384",
                        "ECDHE-RSA-AES256-GCM-SHA384",
                        "ECDHE-ECDSA-AES128-GCM-SHA256",
                        "ECDHE-RSA-AES128-GCM-SHA256",
                        "AES256-GCM-SHA384",
                        "AES128-GCM-SHA256"
                      ]
                    },
                    "alpn_protocols": [
                      "istio-peer-exchange",
                      "h2",
                      "http/1.1"
                    ],
                    "tls_certificate_sds_secret_configs": [
                      {
                        "name": "default",
                        "sds_config": {
                          "api_config_source": {
                            "api_type": "GRPC",
                            "grpc_services": [
                              {
                                "envoy_grpc": {
                                  "cluster_name": "sds-grpc"
                                }
                              }
                            ],
                            "set_node_on_first_message_only": true,
                            "transport_api_version": "V3"
                          },
                          "initial_fetch_timeout": "0s",
                          "resource_api_version": "V3"
                        }
                      }
                    ],
                    "combined_validation_context": {
                      "default_validation_context": {
                        "match_subject_alt_names": [
                          {
                            "prefix": "spiffe://google.com/"
                          }
                        ]
                      },
                      "validation_context_sds_secret_config": {
                        "name": "ROOTCA",
                        "sds_config": {
                          "api_config_source": {
                            "api_type": "GRPC",
                            "grpc_services": [
                              {
                                "envoy_grpc": {
                                  "cluster_name": "sds-grpc"
                                }
                              }
                            ],
                            "set_node_on_first_message_only": true,
                            "transport_api_version": "V3"
                          },
                          "initial_fetch_timeout": "0s",
                          "resource_api_version": "V3"
                        }
                      }
                    }
                  },
                  "require_client_certificate": true
                }
              },
              "name": "virtualInbound"
            },
            {
              "filter_chain_match": {
                "destination_port": 5000,
                "transport_protocol": "tls"
              },
              "filters": [
                {
                  "name": "istio.metadata_exchange",
                  "typed_config": {
                    "@type": "type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange",
                    "protocol": "istio-peer-exchange"
                  }
                },
                {
                  "name": "envoy.filters.network.http_connection_manager",
                  "typed_config": {
                    "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
                    "stat_prefix": "inbound_0.0.0.0_5000",
                    "route_config": {
                      "name": "inbound|5000||",
                      "virtual_hosts": [
                        {
                          "name": "inbound|http|5000",
                          "domains": [
                            "*"
                          ],
                          "routes": [
                            {
                              "match": {
                                "prefix": "/"
                              },
                              "route": {
                                "cluster": "inbound|5000||",
                                "timeout": "0s",
                                "max_stream_duration": {
                                  "max_stream_duration": "0s",
                                  "grpc_timeout_header_max": "0s"
                                }
                              },
                              "decorator": {
                                "operation": "helloworld.sample.svc.cluster.local:5000/*"
                              },
                              "name": "default"
                            }
                          ]
                        }
                      ],
                      "validate_clusters": false
                    },
                    "http_filters": [
                      {
                        "name": "istio.metadata_exchange",
                        "typed_config": {
                          "@type": "type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm",
                          "config": {
                            "vm_config": {
                              "runtime": "envoy.wasm.runtime.null",
                              "code": {
                                "local": {
                                  "inline_string": "envoy.wasm.metadata_exchange"
                                }
                              }
                            },
                            "configuration": {
                              "@type": "type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange"
                            }
                          }
                        }
                      },
                      {
                        "name": "envoy.filters.http.fault",
                        "typed_config": {
                          "@type": "type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault"
                        }
                      },
                      {
                        "name": "envoy.filters.http.cors",
                        "typed_config": {
                          "@type": "type.googleapis.com/envoy.extensions.filters.http.cors.v3.Cors"
                        }
                      },
                      {
                        "name": "istio.stats",
                        "typed_config": {
                          "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                          "type_url": "type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm",
                          "value": {
                            "config": {
                              "root_id": "stats_inbound",
                              "vm_config": {
                                "vm_id": "stats_inbound",
                                "runtime": "envoy.wasm.runtime.null",
                                "code": {
                                  "local": {
                                    "inline_string": "envoy.wasm.stats"
                                  }
                                }
                              },
                              "configuration": {
                                "@type": "type.googleapis.com/google.protobuf.StringValue",
                                "value": "{\n  \"debug\": \"false\",\n  \"stat_prefix\": \"istio\",\n  \"disable_host_header_fallback\": true,\n  \"metrics\": [\n    {\n      \"dimensions\": {\n        \"destination_cluster\": \"node.metadata['CLUSTER_ID']\",\n        \"source_cluster\": \"downstream_peer.cluster_id\"\n      }\n    }\n  ]\n}\n"
                              }
                            }
                          }
                        }
                      },
                      {
                        "name": "envoy.filters.http.router",
                        "typed_config": {
                          "@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router"
                        }
                      }
                    ],
                    "tracing": {
                      "client_sampling": {
                        "value": 100
                      },
                      "random_sampling": {
                        "value": 1
                      },
                      "overall_sampling": {
                        "value": 100
                      },
                      "custom_tags": [
                        {
                          "tag": "istio.authorization.dry_run.allow_policy.name",
                          "metadata": {
                            "kind": {
                              "request": {}
                            },
                            "metadata_key": {
                              "key": "envoy.filters.http.rbac",
                              "path": [
                                {
                                  "key": "istio_dry_run_allow_shadow_effective_policy_id"
                                }
                              ]
                            }
                          }
                        },
                        {
                          "tag": "istio.authorization.dry_run.allow_policy.result",
                          "metadata": {
                            "kind": {
                              "request": {}
                            },
                            "metadata_key": {
                              "key": "envoy.filters.http.rbac",
                              "path": [
                                {
                                  "key": "istio_dry_run_allow_shadow_engine_result"
                                }
                              ]
                            }
                          }
                        },
                        {
                          "tag": "istio.authorization.dry_run.deny_policy.name",
                          "metadata": {
                            "kind": {
                              "request": {}
                            },
                            "metadata_key": {
                              "key": "envoy.filters.http.rbac",
                              "path": [
                                {
                                  "key": "istio_dry_run_deny_shadow_effective_policy_id"
                                }
                              ]
                            }
                          }
                        },
                        {
                          "tag": "istio.authorization.dry_run.deny_policy.result",
                          "metadata": {
                            "kind": {
                              "request": {}
                            },
                            "metadata_key": {
                              "key": "envoy.filters.http.rbac",
                              "path": [
                                {
                                  "key": "istio_dry_run_deny_shadow_engine_result"
                                }
                              ]
                            }
                          }
                        },
                        {
                          "tag": "istio.canonical_revision",
                          "literal": {
                            "value": "v2"
                          }
                        },
                        {
                          "tag": "istio.canonical_service",
                          "literal": {
                            "value": "helloworld"
                          }
                        },
                        {
                          "tag": "istio.mesh_id",
                          "literal": {
                            "value": "devup-mesh"
                          }
                        },
                        {
                          "tag": "istio.namespace",
                          "literal": {
                            "value": "sample"
                          }
                        }
                      ]
                    },
                    "server_name": "istio-envoy",
                    "use_remote_address": false,
                    "forward_client_cert_details": "APPEND_FORWARD",
                    "set_current_client_cert_details": {
                      "subject": true,
                      "dns": true,
                      "uri": true
                    },
                    "upgrade_configs": [
                      {
                        "upgrade_type": "websocket"
                      }
                    ],
                    "stream_idle_timeout": "0s",
                    "normalize_path": true,
                    "request_id_extension": {
                      "typed_config": {
                        "@type": "type.googleapis.com/envoy.extensions.request_id.uuid.v3.UuidRequestIdConfig",
                        "use_request_id_for_trace_sampling": true
                      }
                    },
                    "path_with_escaped_slashes_action": "KEEP_UNCHANGED"
                  }
                }
              ],
              "transport_socket": {
                "name": "envoy.transport_sockets.tls",
                "typed_config": {
                  "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext",
                  "common_tls_context": {
                    "tls_params": {
                      "tls_minimum_protocol_version": "TLSv1_2",
                      "tls_maximum_protocol_version": "TLSv1_3",
                      "cipher_suites": [
                        "ECDHE-ECDSA-AES256-GCM-SHA384",
                        "ECDHE-RSA-AES256-GCM-SHA384",
                        "ECDHE-ECDSA-AES128-GCM-SHA256",
                        "ECDHE-RSA-AES128-GCM-SHA256",
                        "AES256-GCM-SHA384",
                        "AES128-GCM-SHA256"
                      ]
                    },
                    "alpn_protocols": [
                      "h2",
                      "http/1.1"
                    ],
                    "tls_certificate_sds_secret_configs": [
                      {
                        "name": "default",
                        "sds_config": {
                          "api_config_source": {
                            "api_type": "GRPC",
                            "grpc_services": [
                              {
                                "envoy_grpc": {
                                  "cluster_name": "sds-grpc"
                                }
                              }
                            ],
                            "set_node_on_first_message_only": true,
                            "transport_api_version": "V3"
                          },
                          "initial_fetch_timeout": "0s",
                          "resource_api_version": "V3"
                        }
                      }
                    ],
                    "combined_validation_context": {
                      "default_validation_context": {
                        "match_subject_alt_names": [
                          {
                            "prefix": "spiffe://google.com/"
                          }
                        ]
                      },
                      "validation_context_sds_secret_config": {
                        "name": "ROOTCA",
                        "sds_config": {
                          "api_config_source": {
                            "api_type": "GRPC",
                            "grpc_services": [
                              {
                                "envoy_grpc": {
                                  "cluster_name": "sds-grpc"
                                }
                              }
                            ],
                            "set_node_on_first_message_only": true,
                            "transport_api_version": "V3"
                          },
                          "initial_fetch_timeout": "0s",
                          "resource_api_version": "V3"
                        }
                      }
                    }
                  },
                  "require_client_certificate": true
                }
              },
              "name": "0.0.0.0_5000"
            }
          ],
          "listener_filters": [
            {
              "name": "envoy.filters.listener.original_dst",
              "typed_config": {
                "@type": "type.googleapis.com/envoy.extensions.filters.listener.original_dst.v3.OriginalDst"
              }
            },
            {
              "name": "envoy.filters.listener.tls_inspector",
              "typed_config": {
                "@type": "type.googleapis.com/envoy.extensions.filters.listener.tls_inspector.v3.TlsInspector"
              },
              "filter_disabled": {
                "destination_port_range": {
                  "start": 15006,
                  "end": 15007
                }
              }
            }
          ],
          "listener_filters_timeout": "0s",
          "traffic_direction": "INBOUND",
          "continue_on_listener_filters_timeout": true
        },
        "last_updated": "2022-10-15T10:35:30.395Z"
      }
    }
    

    Cluster dump: this is the cluster dump which shows that the SDS validation is with ROOTCA

    "outbound|5000||helloworld.sample.svc.cluster.local"
    {
      "@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
      "name": "outbound|5000||helloworld.sample.svc.cluster.local",
      "type": "EDS",
      "eds_cluster_config": {
        "eds_config": {
          "ads": {},
          "initial_fetch_timeout": "0s",
          "resource_api_version": "V3"
        },
        "service_name": "outbound|5000||helloworld.sample.svc.cluster.local"
      },
      "connect_timeout": "10s",
      "lb_policy": "LEAST_REQUEST",
      "circuit_breakers": {
        "thresholds": [
          {
            "max_connections": 4294967295,
            "max_pending_requests": 4294967295,
            "max_requests": 4294967295,
            "max_retries": 4294967295,
            "track_remaining": true
          }
        ]
      },
      "metadata": {
        "filter_metadata": {
          "istio": {
            "default_original_port": 5000,
            "services": [
              {
                "name": "helloworld",
                "host": "helloworld.sample.svc.cluster.local",
                "namespace": "sample"
              }
            ]
          }
        }
      },
      "common_lb_config": {
        "locality_weighted_lb_config": {}
      },
      "filters": [
        {
          "name": "istio.metadata_exchange",
          "typed_config": {
            "@type": "type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange",
            "protocol": "istio-peer-exchange"
          }
        }
      ],
      "transport_socket_matches": [
        {
          "name": "tlsMode-istio",
          "match": {
            "tlsMode": "istio"
          },
          "transport_socket": {
            "name": "envoy.transport_sockets.tls",
            "typed_config": {
              "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext",
              "common_tls_context": {
                "tls_params": {
                  "tls_minimum_protocol_version": "TLSv1_2",
                  "tls_maximum_protocol_version": "TLSv1_3"
                },
                "alpn_protocols": [
                  "istio-peer-exchange",
                  "istio"
                ],
                "tls_certificate_sds_secret_configs": [
                  {
                    "name": "default",
                    "sds_config": {
                      "api_config_source": {
                        "api_type": "GRPC",
                        "grpc_services": [
                          {
                            "envoy_grpc": {
                              "cluster_name": "sds-grpc"
                            }
                          }
                        ],
                        "set_node_on_first_message_only": true,
                        "transport_api_version": "V3"
                      },
                      "initial_fetch_timeout": "0s",
                      "resource_api_version": "V3"
                    }
                  }
                ],
                "combined_validation_context": {
                  "default_validation_context": {
                    "match_subject_alt_names": [
                      {
                        "exact": "spiffe://google.com/ns/sample/sa/default"
                      }
                    ]
                  },
                  "validation_context_sds_secret_config": {
                    "name": "ROOTCA",
                    "sds_config": {
                      "api_config_source": {
                        "api_type": "GRPC",
                        "grpc_services": [
                          {
                            "envoy_grpc": {
                              "cluster_name": "sds-grpc"
                            }
                          }
                        ],
                        "set_node_on_first_message_only": true,
                        "transport_api_version": "V3"
                      },
                      "initial_fetch_timeout": "0s",
                      "resource_api_version": "V3"
                    }
                  }
                }
              },
              "sni": "outbound_.5000_._.helloworld.sample.svc.cluster.local"
            }
          }
        },
        {
          "name": "tlsMode-disabled",
          "match": {},
          "transport_socket": {
            "name": "envoy.transport_sockets.raw_buffer",
            "typed_config": {
              "@type": "type.googleapis.com/envoy.extensions.transport_sockets.raw_buffer.v3.RawBuffer"
            }
          }
        }
      ]
    }
    

    Secret dump: secret dump which has both the CA configured. upon inspecting you would see that they are minded by spire.

    
      {
       "@type": "type.googleapis.com/envoy.admin.v3.SecretsConfigDump",
       "dynamic_active_secrets": [
        {
         "name": "default",
         "version_info": "2",
         "last_updated": "2022-10-15T10:35:38.847Z",
         "secret": {
          "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret",
          "name": "default",
          "tls_certificate": {
           "certificate_chain": {
            "inline_bytes": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUREakNDQWZhZ0F3SUJBZ0lRR3EwOWliZTVYNDl4T2VZRklVby81akFOQmdrcWhraUc5dzBCQVFzRkFEQWUKTVFzd0NRWURWUVFHRXdKVlV6RVBNQTBHQTFVRUNoTUdVMUJKUmtaRk1CNFhEVEl5TVRBeE5URXdNelV5T0ZvWApEVEl5TVRBeE5URXhNelV6T0Zvd1JURUxNQWtHQTFVRUJoTUNWVk14RGpBTUJnTlZCQW9UQlZOUVNWSkZNU1l3CkpBWURWUVFERXgxb1pXeHNiM2R2Y214a0xYWXlMVFUyTldRM1pHUTNaaTFqY0RnMmFEQlpNQk1HQnlxR1NNNDkKQWdFR0NDcUdTTTQ5QXdFSEEwSUFCRDMreXU0T25oTDVrZW4vVGR5K0FCdWhQRTJvSnVWeGUzWXJtWkhEOXdPQgpyd1BsQjhBNHE0bjhHR1NPVm9qUDJrY3Y1TEJrZGFjeS9JeHNNOUhmbHZLamdlc3dnZWd3RGdZRFZSMFBBUUgvCkJBUURBZ09vTUIwR0ExVWRKUVFXTUJRR0NDc0dBUVVGQndNQkJnZ3JCZ0VGQlFjREFqQU1CZ05WSFJNQkFmOEUKQWpBQU1CMEdBMVVkRGdRV0JCUnlFNVFuZkFGUWVmMlU4enRkVEhDSlVOc0lvREFmQmdOVkhTTUVHREFXZ0JRVgpzK0pMN2dCTzVoZW1GcU11c1c1bnJKUDhEakJwQmdOVkhSRUVZakJnZ2gxb1pXeHNiM2R2Y214a0xYWXlMVFUyCk5XUTNaR1EzWmkxamNEZzJhSUlWYUdWc2JHOTNiM0pzWkM1ellXMXdiR1V1YzNaamhpaHpjR2xtWm1VNkx5OW4KYjI5bmJHVXVZMjl0TDI1ekwzTmhiWEJzWlM5ellTOWtaV1poZFd4ME1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQgpBUUJrRDFnaHVvOFZhTUVBOXJMQ3A0VHM0cnV4QmV5K0FnWnBQMTRvM1I4RHhJSStHWFJDOStHS3RXOGNrQUxWClBrYXRMT1E3NEJCSXk0UHJiaGdQRkVVQ0wvaE9Kb24wazVmS29LbnA0OEFZZTQ2Tk5nckxzZEp3YVhzZzhmdVcKRGE2bExndlZQMkN6ckJxN3FHeHIxbjdJUytzQk1VL3dCaU9BbHlFb3lCTFJIbFg3N3I5dkpIOXNaVW9IckVaMApEMnBIVCtwb3RhNlVMcWQ5OHRIbzN4bnY2R0JxdnZuNXRCUTR3bWVSSnFKTlRTbWZTQ2ZrdmljTjE2VUN3ckNQCm9BeWhWYWJRRUVzcTd4bnFTclZqK2dmcXFQNzlSMldreERra0FOYS9CTUNna0RtM0p5ajJSYmFTYTZCWllhLzgKNnF6YW9wNlFFejJ1RlhoYkxPWW1wYlRoCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"
           },
           "private_key": {
            "inline_bytes": "W3JlZGFjdGVkXQ=="
           }
          }
         }
        },
        {
         "name": "ROOTCA",
         "version_info": "1",
         "last_updated": "2022-10-15T10:35:38.910Z",
         "secret": {
          "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret",
          "name": "ROOTCA",
          "validation_context": {
           "trusted_ca": {},
           "custom_validator_config": {
            "name": "envoy.tls.cert_validator.spiffe",
            "typed_config": {
             "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.SPIFFECertValidatorConfig",
             "trust_domains": [
              {
               "name": "aws.com",
               "trust_bundle": {
                "inline_bytes": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURKVENDQWcyZ0F3SUJBZ0lRQ1NqMGxNQ2IybVhnVUZOMDRiMFBWREFOQmdrcWhraUc5dzBCQVFzRkFEQWUKTVFzd0NRWURWUVFHRXdKVlV6RVBNQTBHQTFVRUNoTUdVMUJKUmtaRk1CNFhEVEl5TVRBeE5URXdNamt5TTFvWApEVEl5TVRBeE5qRXdNamt6TTFvd0hqRUxNQWtHQTFVRUJoTUNWVk14RHpBTkJnTlZCQW9UQmxOUVNVWkdSVENDCkFTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBSzZpUW5ucEsyQ3RXS3VrbTIvT0tJaDIKL1R2NGlTRGlWKzZROFZkWUJRRmQxcmVsUkEzaU0rRjRyQUgvZXFEOHFhSzJGajY5TkJTWGlWNytXdktvZHM4OQpHZW8xUGczV3o0cjl5aE56c0tXN0xuOGprbUFSMmY5RlgzUXAzR2h2QWZDR1V0aU02NkZHSXFsUUZNTmVWNVlCCm9DbzkwVG00TFpwYk1xTXFiZkkxcUlwYm5GUmFZclltU2t1NFB3ZzJVeWlxQUxvbUNEOThjcmQ3VUVUU1Rva1oKUFV5OCsrNXZwclBGSlhQQkg3VzlZdmRrelh5bUt4YXl5N1NPOUtma1JIc2NuWnNxLzhtb1owYndNaFhid2RjMwo2TjVhTFd2WXdDSytxL1ZoY0loRkh2MnJBUWYvcTNzVzh1OEkxdUZ2bzg5Mng3NVArNFpzdFBpMDZjT0Y0UXNDCkF3RUFBYU5mTUYwd0RnWURWUjBQQVFIL0JBUURBZ0VHTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3SFFZRFZSME8KQkJZRUZEQXhST0R2bWNwQTA4UjVLV2RNRFQrczFWNFNNQnNHQTFVZEVRUVVNQktHRUhOd2FXWm1aVG92TDJGMwpjeTVqYjIwd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFFYWVSRytoNG1hUEo5Vm9UanpYSE1nM1dzMVBKMVNSCkF4bzZ2YmszNUhxc0pRbUVnQ1hiZzlsODkvTXBJYUMySzNmQXJBRGdweEhCSU1oRWplZlhJZGFQRnloZDRjYmEKODg3QTF1V0RaOXhFUjNJU0ttek50Y09aRlA2S2dPWTlrMXpLNGp2cTJtdHRYdko2cTZlL1RXRDkrTjhBVS91Uwp1eGw3V3pkVGwrRUlNQ2FyNkRCMlVidUtLY1hQWnBUenVYMWtxSXV4N002Q3kzT1lOTlJ4MzhSKzZ2WWZrdGtuCjF4MlNINnc5blJMeGhlQ3BGeVBhWHc0RFFqOE55RGUvcWpaci9ReFhmRG4zQysyUms0alRQUFdZWTlscExxMGsKNVRxQXlndVpUZ2VLbDNJdXpMcXlBMlJ0Y3p0S3ZoOHF0UmFES0lleHZCWEFTQ054YWRVVUR0Zz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo="
               }
              },
              {
               "name": "google.com",
               "trust_bundle": {
                "inline_bytes": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURLRENDQWhDZ0F3SUJBZ0lRUCtVVFh5SlhRVUFVMlhGU0dhSDBkekFOQmdrcWhraUc5dzBCQVFzRkFEQWUKTVFzd0NRWURWUVFHRXdKVlV6RVBNQTBHQTFVRUNoTUdVMUJKUmtaRk1CNFhEVEl5TVRBeE5URXdNamsxT1ZvWApEVEl5TVRBeE5qRXdNekF3T1Zvd0hqRUxNQWtHQTFVRUJoTUNWVk14RHpBTkJnTlZCQW9UQmxOUVNVWkdSVENDCkFTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTUc4UTBGYlhBUHBweW92UGNvVnlRRDMKdnh2OERpczlSQlc1WE9vSzRPd0FPMTlxYXFMbjAyWHRGUXBEblJ1bzhJVzhPLzJxbjU2Sytzc1BBS2JkTUtMbwpXYXJrZGdERVlTZFJzdTFlYmhlb2cxSmgzTVJ4YTJBUFJtVkpzeVJ6S1FxM284ZVBwM1ZPcEk5OGFSaE9LWXZCCmIwdERkTGtWU0NJaFJZZ0VaQXVoVHI1WlByZGdEYmJTSThtSmFnaTg2cldtR2xCYlR5VFh4ZTVkQitxQmtxc2kKUURpbHRCNU1xUTRBWEg4S2Mxbk5FZFlTL2RlSi9iVDZvM2cwZDhyUFI3bXJ4alVLN1Fhb1Q5eS8wQ1RjbU5ESQpFWVhMTGtGMHBtU3YwaWx6QjlUaU5NNDlyOER0QkxKdk1rWjg2ckpPWWEwQk1IMWJodDZTa1hob2RYRUpYRzhDCkF3RUFBYU5pTUdBd0RnWURWUjBQQVFIL0JBUURBZ0VHTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3SFFZRFZSME8KQkJZRUZCV3o0a3Z1QUU3bUY2WVdveTZ4Ym1lc2svd09NQjRHQTFVZEVRUVhNQldHRTNOd2FXWm1aVG92TDJkdgpiMmRzWlM1amIyMHdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBS2UxT21LQ1pGejE3RklrYkROTUlxVExtNTRRCmt2VlRSZXdSR0FYWUQ0aXJlcDltR01PMjBXeTUxQ1cxSmFyN1JrenV0WENNQjZGYy9WYXdMb3k4bE9CL0xNc2wKRjlicGRJKzgzUmZrcWVsY1hvRWt3dEhvTXdBZHE0RG9yZVpKUVVRbGlZaDQvTDlHUkVIa1JyTDV1WWhmSURsaQpLK3pkenVRYkRzbzlZc1M0SVByYkc3QndsWmpaYVRBT1lmT0V6N0oxeTk2YWp4ZzhZeFdKSG1Kb3ViTTBtclRrCnJtN0lUbnBSZ1h1U3NySy9LSFpqMHdiNTgza1prQVVxL0lldFdiQkt0a3pITVNSK2dDT0FkbUR6b0wralVlWXkKUDFkelhqdmdMQ0RDVWJZSzArUG8zcWtjaTZGYTlKNGxQR2drc1djSFRrTkpuajlqWGU2L1FNbk5SdUE9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"
               }
              }
             ]
            }
           }
          }
         }
        }
       ]
      }
     ]
    }
    

    attaching the entire envoy config of helloworld app

    helloworld-v2-config-dump.json.zip

    triage/in-progress 
    opened by sudeeptoroy 27
  • Enhance AWS node attestor server plugin to validate EC2 instances across multiple AWS account.

    Enhance AWS node attestor server plugin to validate EC2 instances across multiple AWS account.

    We are trying to set up the spire infrastructure in our production environment. The production environment has multiple AWS accounts. We want to deploy a spire server in one of the accounts and attest the EC2 hosts in all other AWS accounts using AWS aws_iid node attestor.

    When both the spire server and the spire agent runs in the same AWS account, the spire server is able to attest to the EC2 host. However when spire server and spire agent are in different AWS account, the node attestation fails with this message: ERRO[0000] agent crashed error=“failed to get SVID: error getting attestation response from SPIRE server: rpc error: code = Internal desc = failed to attest: rpc error: code = Unknown desc = aws-iid: attempted attestation but an error occurred querying AWS via describe-instances: InvalidInstanceID.NotFound:

    opened by sushil-prasad 26
  • SPIRE Agent Windows Support

    SPIRE Agent Windows Support

    The SPIRE Agent should be extended to support attestation of Windows workloads. The current version of Windows supports UDS natively as of Windows 10 as of build 1809. @azdagron Has done a POC using named pipes.

    • What are the introspection capabilities through named pipes/windows UDS?
    • How much does support of Windows users before build 1809 matter?
    • User experience for OS-specific plugins, i.e. UNIX workload attestor.
    • Version: n/a
    • Platform: Windows
    • Subsystem: Agent
    opened by colek42 24
  • Implement upstream authority plugin for GCP

    Implement upstream authority plugin for GCP

    Signed-off-by: ramand.dragcp [email protected]

    Pull Request check list

    • [*] Commit conforms to CONTRIBUTING.md?
    • [*] Proper tests/regressions included?
    • [*] Documentation updated?

    Affected functionality Adds a new plugin for GCP Certificate Authority Service backed upstream authority

    Description of change Thank you for reviewing the PR #2039 by @Jonpez2. I am continuing that PR here. I believe I have incorporated all of the feedback

    A few notes:

    • Changes to go.sum and go.mod happened automatically as part of "make build"
    • We pick all the CAs in GCP CAS that are in enabled state and also matches the specified label key/value pair. The one with the earliest expiration is used to create and sign intermediate CAs
    • I have tried to included relevant documentation links for most of the APIs and types of GCP CAS.

    Which issue this PR fixes

    opened by dragcp 23
  • Safer DB migration strategy in spire server cluster env

    Safer DB migration strategy in spire server cluster env

    Currently if SPIRE Server with DB version X starts up and looks to DB with version < X, SPIRE will auto-migrate the DB to X. Backwards compatibility does not exist.

    Problems:

    • In a distributed environment this is fatal to the other SPIRE Servers. They will be in a < X DB version and fail to start due to not being able to migrate backwards
    • The migration is uncontrolled; in such an environment we not commit to every single Server must upgrade, and hope that the new DB schema is good

    Some (non-mutually exclusive) Solutions:

    • A flag to pass to Server config to not perform auto-migration; without other changes this is fatal to the newer server but lets the older ones continue operations
    • Proper semantic versioning of the DB versions; when starting up and the DB is at a later version than the Server expects, if the new version does not indicate a breaking change then the old Server should be able to continue operations (not perform backwards migration)
    • A method of rolling back the DB version
    • A method of a later Server being backwards compatible for older DB versions
    opened by amoore877 21
  • Implement an Informer strategy for k8s-workload-registrar

    Implement an Informer strategy for k8s-workload-registrar

    The registrar gains an option to use an informer instead of a webhook. In this mode, it watches the k8s API instead of listening for updates from a webhook.

    The controller code is extended so that when entries are added, any outdated entries for the same pod are removed. This means that label changes are now reflected in the registration entries.

    Update client-go version to kubernetes 1.15 to get past API changes. Regenerate mocks to match.

    opened by asuffield 20
  • OIDC Discovery Provider health check endpoint unavailable outside container when using virtual network

    OIDC Discovery Provider health check endpoint unavailable outside container when using virtual network

    The OIDC Discovery Provider exposes its liveness and readiness HTTP APIs over only the local interface by specifying the listen address as ::<port>. Because of this, when the OIDC Discovery Provider is running in a container that is not bound to the host network, it will only listen over 127.0.0.1:<port> inside the container.

    In Kubernetes, for example, this results in the health endpoint being inaccessible outside the container when using virtual pod IPs, which makes it unusable for the purpose of live and ready pod probes.

    See this Stack post for more context.

    Ideally either the OIDC Discovery Provider should accept an address as configuration or default to listening on 0.0.0.0. The latter change is backward compatible, so a new configuration may be less friction.

    priority/backlog 
    opened by rturner3 0
  • Use cli printer server entry commands

    Use cli printer server entry commands

    Pull Request check list

    • [x] Commit conforms to CONTRIBUTING.md?
    • [x] Proper tests/regressions included?
    • [x] Documentation updated?

    Affected functionality 🤔 The following spire-server entry commands are affected:

    • count
    • create
    • delete
    • show
    • update

    Description of change ✍️ Enable output format definition for spire-server entry commands, using cliprinter.

    Which issue this PR fixes ❓ Ongoing work for #1354

    opened by guilhermocc 0
  • Use cli printer server bundle commands

    Use cli printer server bundle commands

    Pull Request check list

    • [x] Commit conforms to CONTRIBUTING.md?
    • [x] Proper tests/regressions included?
    • [x] Documentation updated?

    Affected functionality The following spire-server bundle commands are affected:

    • count
    • delete
    • list
    • set
    • show

    Description of change Enable output format definition for spire-server bundle commands, using cliprinter.

    Which issue this PR fixes Ongoing work for #1354

    opened by guilhermocc 0
  • Re-evaluate SPIRE Server API authorization

    Re-evaluate SPIRE Server API authorization

    Many previous issues and PRs have taken attempts at solving a plethora of authorization pains that the SPIRE Server APIs currently face. These APIs tend to have a number of consumers, and the current authorization framework effectively provides an all-or-nothing authorization posture. This is particularly problematic for registrars which should have control over only a portion of the SPIFFE namespace, or for operators that should have the ability only to e.g. evict agents but not alter entries. These are two of many examples.

    Initially, we tried to implement an OPA Rego-based authorization subsystem, which would allow users to provide their own authorization policies. This has been shipping as an experimental feature for quite some time now, however it suffers from many of its own unique challenges:

    • It must be compatible with the existing authorization system in core, which has complicated the req/resp pattern that OPA expects
    • This complexity is guaranteed to grow as flexibility in core increases, since we're obligated to support users that are not supplying us with OPA Rego
    • Trivial mistakes in the user-provided policy can lead to catastrophic security holes in the SPIRE security posture
    • We've been unable to identify a safe and straightforward way to perform any sanity check on the user supplied policies

    This past Tuesday, SPIRE contributors discussed this issue and concluded that there is no clear path forward in terms of graduating OPA Rego support out of experimental. Rather, the consensus was that the only suitable solution is likely to be a major re-think of the way that SPIRE Server performs API authorization, and that the likely answer is a first-class authorization system based on roles/capabilities/etc.

    This is an unscoped issue meant to explore and capture the possible solutions. This issue is done when we've made a decision on the basic shape of a solution, at which point we can open further issues.

    priority/backlog unscoped 
    opened by evan2645 0
Releases(v1.5.1)
Owner
SPIFFE
Secure Production Identity Framework For Everyone
SPIFFE
Scans and catches callbacks of systems that are impacted by Log4J Log4Shell vulnerability across specific headers.

Log4ShellScanner Scans and catches callbacks of systems that are impacted by Log4J Log4Shell vulnerability across specific headers. Very Beta Warning!

null 56 Jun 17, 2022
Obfuscate Go code by wrapping the Go toolchain

Obfuscate Go code by wrapping the Go toolchain.

null 2.2k Nov 27, 2022
Secret management toolchain

Harp TL;DR. Why harp? Use cases How does it work? Like a Data pipeline but for secret Immutable transformation What can I do? FAQ License Homebrew ins

elastic 133 Nov 9, 2022
zero-trust remote firewall instrumentation

ShieldWall embraces the zero-trust principle and instruments your server firewall to block inbound connections from every IP on any port, by default.

Simone Margaritelli 181 Nov 12, 2022
Update-java-ca-certificates - Small utility to convert the system trust store to a system Java KeyStore

update-java-ca-certificates This small utility takes care of creating a system-w

Swisscom 6 Nov 11, 2022
Find secrets and passwords in container images and file systems

Find secrets and passwords in container images and file systems

null 1.7k Nov 21, 2022
Scanner to send specially crafted requests and catch callbacks of systems that are impacted by Log4J Log4Shell vulnerability (CVE-2021-44228)

scan4log4shell Scanner to send specially crafted requests and catch callbacks of systems that are impacted by Log4J Log4Shell vulnerability CVE-2021-4

Frank Hübner 12 Sep 17, 2022
Volana - Shell command obfuscation to avoid detection systems

volana (moon in malagasy) { Use it ; ??(hide from); ??(detected by) } Shell comm

Ariary 38 Nov 9, 2022
Scan systems and docker images for potential spring4shell vulnerabilities.

Scan systems and docker images for potential spring4shell vulnerabilities. Will detect in-depth (layered archives jar/zip/tar/war and scans for vulnerable Spring4shell versions. Binaries for Windows, Linux and OsX, but can be build on each platform supported by supported Golang.

null 10 Nov 9, 2022
Implementations of the Coconut signing scheme, cross-compatible between Rust and Go.

Coconut Coconut [paper] is a distributed cryptographic signing scheme providing a high degree of privacy for its users. You can find an overview of ho

Nym 20 Jul 24, 2022
Static binary analysis tool to compute shared strings references between binaries and output in JSON, YAML and YARA

StrTwins StrTwins is a binary analysis tool, powered by radare, that is capable to find shared code string references between executables and output i

Anderson 2 May 3, 2022
Gorsair hacks its way into remote docker containers that expose their APIs

Gorsair Gorsair is a penetration testing tool for discovering and remotely accessing Docker APIs from vulnerable Docker containers. Once it has access

Brendan Le Glaunec 773 Nov 22, 2022
Secure software enclave for storage of sensitive information in memory.

MemGuard Software enclave for storage of sensitive information in memory. This package attempts to reduce the likelihood of sensitive data being expos

Awn 2.2k Nov 19, 2022
A software supply chain security inspection tool.

README.md murphysec 一款专注于软件供应链安全的开源工具,包含开源组件依赖分析、漏洞检测及漏洞修复等功能。 安装 macOS 使用Homebrew安装 // TODO Windows 使用scoop安装 scoop bucket add murphysec https://gith

murphysec 38 Feb 20, 2022
Bhojpur Consulting 0 Sep 26, 2022
This is a "simple" game server. Main functionalities are matching and establishing a connection between players

Game Server This is a "simple" game server. Main functionalities are matching and establishing a connection between players How to Run? run the server

eco 8 Aug 28, 2022
Use AWS SQS as a clipboard to copy and paste across different systems and platforms

sqs_clipboard Use AWS SQS as a clipboard to copy and paste across different systems and platforms. Clipboard contents are encrypted in transit and at

John Taylor 22 Oct 16, 2022
Easy to use cryptographic framework for data protection: secure messaging with forward secrecy and secure data storage. Has unified APIs across 14 platforms.

Themis provides strong, usable cryptography for busy people General purpose cryptographic library for storage and messaging for iOS (Swift, Obj-C), An

Cossack Labs 1.6k Nov 25, 2022
Easy to use cryptographic framework for data protection: secure messaging with forward secrecy and secure data storage. Has unified APIs across 14 platforms.

Themis provides strong, usable cryptography for busy people General purpose cryptographic library for storage and messaging for iOS (Swift, Obj-C), An

Cossack Labs 1.6k Nov 21, 2022
Put a web archive (WARC) on an S3 bucket suitable for hosting with S3 Website Hosting.

warc-to-s3 This is a small Go application that consumes a WARC file ( using slyzrc/warc) and puts it on S3 suitable for serving with S3 Website Hostin

Ian Dees 5 Jan 6, 2022