--- title: Kubernetes Ambassador summary: null url: >- https://www.fastly.com/documentation/guides/next-gen-waf/setup-and-configuration/kubernetes/kubernetes-ambassador --- In this example, the Next-Gen WAF is integrated with Ambassador Edge Stack, a cloud native API gateway and ingress controller for Kubernetes, built upon Envoy proxy. ## Integrating the Next-Gen WAF agent The Next-Gen WAF agent can be installed as a sidecar into each pod or as a service for some specialized needs. The recommended way of installing the Next-Gen WAF agent in Kubernetes is by integrating the `sigsci-agent` into a pod as a [sidecar](https://matthewpalmer.net/kubernetes-app-developer/articles/multi-container-pod-design-patterns.html). This means adding the `sigsci-agent` as an additional container to the Kubernetes pod. As a sidecar, the agent will scale with the app/service in the pod instead of having to do this separately. However, in some situations, it may make more sense to install the `sigsci-agent` container as a service and [scale it separately from the application](https://www.fastly.com/documentation/guides/next-gen-waf/setup-and-configuration/kubernetes/kubernetes-agent-scaling). The `sigsci-agent` container can be configured in various ways depending on the installation type and module being used. You can use the [`preStop` container hook](https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks) to slow the pod's shutdown and ensure drain timeouts are met. ```yaml preStop: exec: command: - sleep - "30" ``` By default, the agent prioritizes quick start up and performance readiness for preliminary inspection. However, quick startup isn't always desirable if you only want the agent to inspect traffic after loading your rules and configuration data. If you want to delay agent startup, consider configuring a [startup probe](https://www.fastly.com/documentation/guides/next-gen-waf/setup-and-configuration/kubernetes/kubernetes-startup-probe). ## Getting and updating the agent container image An official `signalsciences/sigsci-agent` container image is available on [Docker Hub](https://hub.docker.com/r/signalsciences/sigsci-agent). Alternatively, if you want to build your own image or need to customize the image, then follow the [sigsci-agent build instructions](https://www.fastly.com/documentation/guides/next-gen-waf/setup-and-configuration/kubernetes/kubernetes-agent/). These instructions reference the `latest` version of the agent with `imagePullPolicy: Always`, which will pull the latest agent version even if one already exist locally. This is so the documentation does not fall out of date and anyone using this will not have an agent that stays stagnant. However, this may not be what if you need to keep installations consistent or on a specific version of the agent. In these cases, you should specify an [agent version](https://www.fastly.com/documentation/reference/changes/ngwaf-agent/). Images on Docker Hub are tagged with their versions and [a list of versions is available on Docker Hub](https://hub.docker.com/r/signalsciences/sigsci-agent/tags). Whether you choose to use the `latest` image or a specific version, there are a few items to consider to keep the agent up-to-date. ### Using the `latest` container image If you do choose to use the `latest` image, then you will want to consider how you will keep the agent up to date. - If you have used the `imagePullPolicy: Always` option, then the latest image will be pulled on each startup and your agent will continue to get updates. - Alternatively, you may instead choose to manually update the local cache by periodically forcing a pull instead of always pulling on startup: ```term copy $ docker pull signalsciences/sigsci-agent:latest ``` Then, use `latest` with `imagePullPolicy: Never` set in the configuration so that pulls are never done on startup (only manually as above): ```yaml - name: sigsci-agent image: signalsciences/sigsci-agent:latest imagePullPolicy: Never ... ``` ### Using a versioned container image To use a specific version of the agent, replace `latest` with the agent version (represented here by `x.xx.x`). You may also want to change `imagePullPolicy: IfNotPresent` in this case as the image should not change. ```yaml - name: sigsci-agent image: signalsciences/sigsci-agent:x.xx.x imagePullPolicy: IfNotPresent ... ``` This will pull the specified agent version and cache it locally. If you use this method, then it is recommended that you parameterize the agent image, using Helm or similar, so that it is easier to update the agent images later on. ### Using a custom tag for the container image It is also possible to apply a custom tag to a local agent image. To do this, pull the agent image (by version or use `latest`), apply a custom tag, then use that custom tag in the configuration. You will need to specify `imagePullPolicy: Never` so local images are only updated manually. After doing so, you will need to periodically update the local image to keep the agent up-to-date. For example: ```term copy nolinenums $ docker pull signalsciences/sigsci-agent:latest $ docker tag signalsciences/sigsci-agent:latest signalsciences/sigsci-agent:testing ``` Then use this image tag in the configuration: ```yaml - name: sigsci-agent image: signalsciences/sigsci-agent:testing imagePullPolicy: Never ... ``` ## Configuring the agent container Agent configuration is normally done via the environment. Most configuration options are available as environment variables. Environment variables names have the configuration option name all capitalized, prefixed with `SIGSCI_` and any dashes (-) changed to underscores (\_). For example, the [max-procs](https://www.fastly.com/documentation/reference/ngwaf/agent-config#agentcfg_max-procs) option would become the `SIGSCI_MAX_PROCS` environment variable. For more details on what options are available, see the [Agent Configuration documentation](https://www.fastly.com/documentation/reference/ngwaf/agent-config). The `sigsci-agent` container has a few required options that need to be configured: - Agent credentials (**Agent Access Key** and **Agent Secret Key**). - A volume to write temporary files. ### Agent credentials The [`sigsci-agent` credentials](https://www.fastly.com/documentation/guides/next-gen-waf/setup-and-configuration/agent-management/accessing-agent-keys) are configured with two environment variables. These variables must be set or the agent will not start. - **SIGSCI_ACCESSKEYID**: The **Agent Access Key** identifies which site (also known as workspace) the agent is configured for. - **SIGSCI_SECRETACCESSKEY**: The **Agent Secret Key** is the shared secret key to authenticate and authorize the agent. Because of the sensitive nature of these values, we recommend you use the built in `secrets` functionality of Kubernetes. With this configuration, the agent will pull the values from the secrets data instead of reading hardcoded values into the deployment configuration. This also makes any desired agent credential rotation easier to manage by having to change them in only one place. Use the `valueFrom` option instead of the `value` option to use the `secrets` functionality. For example: ```yaml env: - name: SIGSCI_ACCESSKEYID valueFrom: secretKeyRef: # Update my-site-name-here to the correct site (workspace) name or similar identifier name: sigsci.my-site-name-here key: accesskeyid - name: SIGSCI_SECRETACCESSKEY valueFrom: secretKeyRef: # Update my-site-name-here to the correct site (workspace) name or similar identifier name: sigsci.my-site-name-here key: secretaccesskey ``` The `secrets` functionality keeps secrets in various stores in Kubernetes. This guide uses the generic secret store in its examples, however any equivalent store can be used. Agent secrets can be added to the generic secret store using YAML similar to the following example: ```yaml apiVersion: v1 kind: Secret metadata: name: sigsci.my-site-name-here stringData: accesskeyid: 12345678-abcd-1234-abcd-1234567890ab secretaccesskey: abcdefg_hijklmn_opqrstuvwxy_z0123456789ABCD ``` This can also be created from the command line with `kubectl` such as with the following example: ```term copy nolinenums $ kubectl create secret generic sigsci.my-site-name-here \ --from-literal=accesskeyid=12345678-abcd-1234-abcd-1234567890ab \ --from-literal=secretaccesskey=abcdefg_hijklmn_opqrstuvwxy_z0123456789ABCD ``` Additional information about Kubernetes `secrets` functionality can be found in the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/secret/). ### Agent temporary volume For added security, we recommended the `sigsci-agent` container be executed with the root filesystem mounted as read only. However, the agent still needs to write some temporary files such as the socket file for RPC communication and some periodically updated files such as geolocation data. To accomplish this with a read only root filesystem, there needs to be a writeable volume mounted. This writeable volume can also be shared to expose the RPC socket file to other containers in the same pod. The recommended way of creating a writeable volume is to use the builtin `emptyDir` volume type. This is typically configured in the `volumes` section of a deployment, as shown in the following example: ```yaml volumes: - name: sigsci-tmp emptyDir: {} ``` Containers will then mount this volume at `/sigsci/tmp`: ```yaml volumeMounts: - name: sigsci-tmp mountPath: /sigsci/tmp ``` The default in the official agent container image is to have the temporary volume mounted at `/sigsci/tmp`. If this needs to be moved for the agent container, then the following agent configuration options should also be changed from their defaults to match the new mount location: - `rpc-address` defaults to `/sigsci/tmp/sigsci.sock` - `shared-cache-dir` defaults to `/sigsci/tmp/cache` ## Integrating the Next-Gen WAF agent into Ambassador Edge Stack (AES) The Next-Gen WAF agent can be integrated with Datawire's Ambassador Edge Stack (AES). This integration uses the underlying Envoy integration built into the agent. The agent is configured with an Envoy gRPC Listener and through AES's Filter, FilterPolicy, and LogService Kubernetes resources. Deployment and configuration is flexible. As such, this guide is designed to provide information that can be applied to your own methods of deployment. Note that the examples in the documentation will refer to installing the latest agent version, but this is only so that the documentation examples do not fall behind. Refer to the [documentation on getting and updating the agent](https://www.fastly.com/documentation/guides/next-gen-waf/setup-and-configuration/kubernetes/kubernetes-envoy/#getting-and-updating-the-agent-container-image) for more details on agent versioning and how to keep the agent up-to-date. ### Namespaces By default, AES is installed into the `ambassador` Kubernetes namespace. The agent and any applications running behind AES do not have to run in this namespace, but you must take care during configuration to use the correct namespaces as this documentation may differ from your configuration. The following namespaces are used in this documentation: #### ambassador - Used for the ambassador install. - Used for all ambassador resources (e.g., Filter, FilterPolicy, LogService, Mapping). - Used for the sigsci-agent when running as a sidecar. #### default - Used for all applications and services running behind AES. - Used for the agent when run in standalone mode. ### Ambassador ID Ambassador Edge Stack supports running multiple Ambassador Edge Stacks in the same cluster, without restricting a given Ambassador Edge Stack to a single namespace. This is done with the `ambassador_id` setting. When running multiple Ambassador Edge Stacks, care must be taken to include the correct `ambassador_id` value within all ambassador resources (e.g., Filter, LogService, Mapping), otherwise the configuration will not be used. AES deployments only running one stack under the `default` ID don't need to set this value. Refer to the [Ambassador ID docs](https://www.getambassador.io/docs/edge-stack/latest/topics/running/running/#ambassador_id) for more information. ### Running the agent as standalone or sidecar The agent can run as a standalone deployment service or as a sidecar container within the AES pod. Either is fine, but running as a sidecar is easier if you are using Helm, as this is directly supported in the Helm values file. Running as a sidecar also has the advantage of scaling with AES, so this is the recommended route if you are using scaling via replica counts or autoscaling. ## Installation Installation involves two tasks: Deploying the agent configured in gRPC mode and Configuring AES to send traffic to the agent. ### Deploying the agent Deploying the agent is done by deploying the `signalsciences/sigsci-agent` container as a sidecar to AES or as a standalone service. The agent must be configured with its Agent Access Key and Agent Secret Key. This is typically done via a Kubernetes secret. One important point about secrets is that the secret must be in the same namespace as the pod using the secret. So, if you are running as a sidecar in the ambassador namespace, then the secret must also reside in that namespace. Refer to the [agent credentials documentation](https://www.fastly.com/documentation/guides/next-gen-waf/setup-and-configuration/kubernetes/kubernetes-envoy/#agent-credentials) for more details. Example Secret in the ambassador namespace: ```yaml apiVersion: v1 kind: Secret metadata: # Edit `my-site-name-here` # and change the namespace to match that which # the agent is to be deployed name: sigsci.my-site-name-here namespace: ambassador stringData: # Edit these `my-agent-*-here` values: accesskeyid: my-agent-access-key-id-here secretaccesskey: my-agent-secret-access-key-here ``` ### Sidecar with Helm Configuring AES with Helm is the easiest way to deploy, as the Ambassador values file already has direct support for this without having to modify an existing deployment YAML file. The [AES Quick start ](https://www.getambassador.io/docs/edge-stack/latest/tutorials/getting-started#installing-ambassador-edge-stack) describes this in detail. To install the agent as a sidecar, you will need to add new configuration lines to your custom values file, then install or upgrade AES with this values file. Refer to the [Ambassador helm chart documentation](https://github.com/datawire/ambassador-chart/tree/master) for a reference on the values file. This will add the container with the correct configuration to the AES pod as a sidecar. Add the following to the values YAML file: ```yaml sidecarContainers: - name: sigsci-agent image: signalsciences/sigsci-agent:latest imagePullPolicy: IfNotPresent # Configure the agent to use Envoy gRPC on port 9999 env: - name: SIGSCI_ACCESSKEYID valueFrom: secretKeyRef: # This secret needs added (see documentation on sigsci secrets) name: sigsci.my-site-name-here key: accesskeyid - name: SIGSCI_SECRETACCESSKEY valueFrom: secretKeyRef: # This secret needs added (see documentation on sigsci secrets) name: sigsci.my-site-name-here key: secretaccesskey # Configure the Envoy to expect response data - name: SIGSCI_ENVOY_EXPECT_RESPONSE_DATA value: "1" # Configure the Envoy gRPC listener address on any unused port - name: SIGSCI_ENVOY_GRPC_ADDRESS value: localhost:9999 ports: - containerPort: 9999 name: grpc securityContext: # The sigsci-agent container should run with its root filesystem read only readOnlyRootFilesystem: true # Ambassador uses user 8888 by default, but the sigsci-agent container # needs to run as sigsci(100) runAsUser: 100 volumeMounts: - name: sigsci-tmp mountPath: /sigsci/tmp volumes: - name: sigsci-tmp emptyDir: {} ``` Example of upgrading AES with helm: ```term copy nolinenums $ helm upgrade ambassador \ --values /path/to/ambassador-sigsci_values.yaml \ --namespace ambassador \ datawire/ambassador ``` Alternatively, you can use Helm to render the manifest files. This makes adding the agent sidecar much easier than manually editing the YAML files. The modified deployment YAML will be in: ```text /ambassador/templates/deployment.yaml ``` Example of rendering the manifests with helm and applying the results: ```term copy nolinenums $ helm template \ --output-dir ./manifests \ --values ./ambassador-sigsci_values.yaml \ --namespace ambassador \ datawire/ambassador $ kubectl apply \ --recursive --filename ./manifests/ambassador ``` ### Sidecar manually Deploying the agent as a sidecar into the AES pod manually requires significantly more work than [using Helm to render the manifests](https://www.fastly.com/documentation/guides/next-gen-waf/setup-and-configuration/kubernetes/kubernetes-ambassador#sidecar-with-helm) and is therefore not recommended. You will need to modify the `aes.yaml` file as described in the [AES Quick Start](https://www.getambassador.io/docs/cloud/latest/service-catalog/quick-start#3-add-additional-metadata-via-yaml). Append the container and volumes as described in the [using Helm instructions](https://www.fastly.com/documentation/guides/next-gen-waf/setup-and-configuration/kubernetes/kubernetes-ambassador#sidecar-with-helm). The [AES Quick start](https://www.getambassador.io/docs/edge-stack/latest/tutorials/getting-started#installing-ambassador-edge-stack) and the [Kubernetes and Envoy documentation](https://www.fastly.com/documentation/guides/next-gen-waf/setup-and-configuration/kubernetes/kubernetes-envoy) provide additional installation details. You will need to modify the following resource: ```yaml apiVersion: apps/v1 kind: Deployment metadata: labels: product: aes name: ambassador namespace: ambassador … containers: … volumes: … ``` The container will need to be added to the `containers` section and the volume to the `volumes` section. ### Standalone To deploy a standalone agent, you only need to add a `Deployment` and `Service` resource for the agent, as shown in the following example: ```yaml apiVersion: v1 kind: Service metadata: name: sigsci-agent # You may want it running in the ambassador namespace #namespace: ambassador labels: service: sigsci-agent spec: type: ClusterIP ports: - name: sigsci-agent port: 9999 targetPort: grpc selector: service: sigsci-agent --- apiVersion: apps/v1 kind: Deployment metadata: name: sigsci-agent # You may want it running in the ambassador namespace #namespace: ambassador spec: replicas: 1 selector: matchLabels: service: sigsci-agent template: metadata: labels: service: sigsci-agent spec: containers: - name: sigsci-agent image: signalsciences/sigsci-agent:latest imagePullPolicy: IfNotPresent # Configure the agent to use Envoy gRPC on port 9999 env: - name: SIGSCI_ACCESSKEYID valueFrom: secretKeyRef: # This secret needs added (see documentation on sigsci secrets) name: sigsci.my-site-name-here key: accesskeyid - name: SIGSCI_SECRETACCESSKEY valueFrom: secretKeyRef: # This secret needs added (see documentation on sigsci secrets) name: sigsci.my-site-name-here key: secretaccesskey # Configure the Envoy to expect response data - name: SIGSCI_ENVOY_EXPECT_RESPONSE_DATA value: "1" # Configure the Envoy gRPC listener address on any unused port - name: SIGSCI_ENVOY_GRPC_ADDRESS value: 0.0.0.0:9999 ports: - containerPort: 9999 name: grpc securityContext: # The sigsci-agent should run with its root filesystem read only readOnlyRootFilesystem: true volumeMounts: - name: sigsci-tmp mountPath: /sigsci/tmp volumes: - name: sigsci-tmp emptyDir: {} ``` For more information, refer to the [Kubernetes and Envoy documentation](https://www.fastly.com/documentation/guides/next-gen-waf/setup-and-configuration/kubernetes/kubernetes-envoy). ## Sending traffic to the agent You will need to configure three Ambassador resources for AES to send data to the agent. Refer to the [Envoy configuration documentation](https://www.fastly.com/documentation/guides/next-gen-waf/setup-and-configuration/reverse-proxy-deployment/configuring-envoy-proxy-deployments/) for more detailed information on what each of these configures in the underlying Envoy install. The following guide uses the example [`quote` service](https://www.getambassador.io/docs/edge-stack/latest/tutorials/getting-started/) included with Ambassador. ### Filter The [Filter resource](https://www.getambassador.io/docs/edge-stack/latest/topics/using/filters/external/) is used to [add the external authorization (`ext_authz`) filter to Envoy](https://www.fastly.com/documentation/guides/next-gen-waf/setup-and-configuration/reverse-proxy-deployment/configuring-envoy-proxy-deployments#adding-the-envoy-external-authorization-http-filter). This will inspect incoming requests that match the FilterPolicy. The Next-Gen WAF agent requires [AuthService](https://www.getambassador.io/docs/latest/topics/running/services/auth-service/) to be defined in the Ambassador configuration, otherwise the agent will not receive request data. AuthService should be enabled by default. If requests are not being received by the agent, check that AuthService is enabled by running `kubectl get authservice`. The namespace used for the `auth_service` configuration is the namespace the agent is deployed to. This guide uses the `ambassador` namespace for sidecar agents and `default` namespace for standalone agents. The format for the `auth_service` URL must be: `agent-hostname[.namespace]:agent-port` Examples: - Sidecar: `auth_service: localhost:9999` - Standalone: `auth_service: sigsci-agent.default:9999` Example Filter YAML: ```yaml # Filter defines an external auth filter to send to the agent kind: Filter apiVersion: getambassador.io/v3alpha1 metadata: name: sigsci namespace: ambassador annotations: getambassador.io/resource-changed: "true" spec: External: # Sidecar agent: auth_service: localhost:9999 # Standalone sigsci-agent service in default namespace: #auth_service: sigsci-agent.default:9999 path_prefix: "" tls: false proto: grpc protocol_version: v3 include_body: max_bytes: 8192 allow_partial: true failure_mode_allow: true timeout_ms: 100000 ``` ### FilterPolicy The [FilterPolicy resource](https://www.getambassador.io/docs/edge-stack/latest/topics/using/filters) maps what paths will be inspected by the agent. You can map this to all traffic (`path: /*`) or subsets (`path: /app1/*`). However, there is a limitation that each subset must map to the same agent. This is due to a limitation on the LogService not having a path based filter like the FilterPolicy. The LogService must route all matching response data to the same agent that handled the request. Example routing all traffic to the agent: ```yaml # FilterPolicy defines which requests go to sigsci kind: FilterPolicy apiVersion: getambassador.io/v3alpha1 metadata: namespace: ambassador name: sigsci-policy annotations: getambassador.io/resource-changed: "true" spec: rules: - host: "*" # All traffic to the sigsci-agent path: "/*" filters: # Use the same name as the Filter above - name: sigsci namespace: ambassador onDeny: break onAllow: continue ifRequestHeader: null arguments: {} ``` You can route subsets of traffic to the agent with multiple rules. However every rule must go to the same agent due to the limitations described above. Example routing subsets of traffic to the agent: ```yaml # FilterPolicy defines which requests go to the sigsci-agent kind: FilterPolicy apiVersion: getambassador.io/v3alpha1 metadata: namespace: ambassador name: sigsci-policy annotations: getambassador.io/resource-changed: "true" spec: rules: # /app1/* and /app2/* to the sigsci-agent - host: "*" path: "/app1/*" filters: # Use the same name as the Filter above - name: sigsci namespace: ambassador onDeny: break onAllow: continue ifRequestHeader: null arguments: {} - host: "*" path: "/app2/*" filters: # Use the same name as the Filter above - name: sigsci namespace: ambassador onDeny: break onAllow: continue ifRequestHeader: null arguments: {} ``` ### LogService The [LogService resource](https://www.getambassador.io/docs/edge-stack/latest/topics/running/services/log-service) is used to [add the gRPC Access Log Service to Envoy](https://www.fastly.com/documentation/guides/next-gen-waf/setup-and-configuration/reverse-proxy-deployment/configuring-envoy-proxy-deployments/#adding-the-envoy-grpc-accesslog-service). This will inspect the outgoing response data and record this data if a signal was detected. It is also used for anomaly signals such as `HTTP_4XX` and `HTTP_5XX`. The namespace used for the `service` configuration is the namespace the agent is deployed to. This guide uses the `ambassador` namespace for sidecar agents and `default` namespace for standalone agents. The format for the `service` URL must be: `agent-hostname[.namespace]:agent-port` Examples: - Sidecar: `service: localhost:9999` - Standalone: `service: sigsci-agent.default:9999` Example: ```yaml # Configure the access log gRPC service for the response # NOTE: There is no policy equiv here, so all requests are sent apiVersion: getambassador.io/v3alpha1 kind: LogService metadata: namespace: ambassador name: sigsci-agent spec: # Sidecar agent service: localhost:9999 # Standalone sigsci-agent service in default namespace: #service: sigsci-agent.default:9999 driver: http driver_config: additional_log_headers: ### Request headers: # Required: - header_name: "x-sigsci-request-id" during_request: true during_response: false during_trailer: false - header_name: "x-sigsci-waf-response" during_request: true during_response: false during_trailer: false # Recommended: - header_name: "accept" during_request: true during_response: false during_trailer: false - header_name: "date" during_request: false during_response: true during_trailer: true - header_name: "server" during_request: false during_response: true during_trailer: true ### Both request/response headers: # Recommended - header_name: "content-type" during_request: true during_response: true during_trailer: true - header_name: "content-length" during_request: true during_response: true during_trailer: true grpc: true protocol_version: v3 ``` ## Related content - [Kubernetes installation overview](https://www.fastly.com/documentation/guides/next-gen-waf/setup-and-configuration/kubernetes/kubernetes-intro) - [Kubernetes agent scaling](https://www.fastly.com/documentation/guides/next-gen-waf/setup-and-configuration/kubernetes/kubernetes-agent-scaling)