Pular para o conteúdo
AvraAvra
Esc
↑↓navegar↵abrir⌘Jpré-visualizar
Nesta página

Install the data layer

Install Avra's ingestion pipeline and agent in your Kubernetes cluster with Helm, verify the installation, and keep it up to date.

The data layer is the part of Avra that runs in your Kubernetes cluster. It holds the ingestion pipeline, which validates and versions your data with Spark, and the Avra agent, which takes the pipeline’s commands from Avra’s control plane. You provide the cluster and the infrastructure around it, and a deployment agent installs and upgrades the rest.

In short: you provide the prerequisites, Avra configures your deployment, you run one install command, and Avra verifies the result. This page covers each step, then how to check the installation, how upgrades work, and the settings your deployment covers.

The layers

Avra packages what runs in your cluster as Helm charts. The charts are grouped in layers, and each layer is versioned and released as a unit. A layer version pins the version of every component inside it, so a release is reproducible.

Layer What it installs
customer-data The Avra agent, the ingestion pipeline, a Spark Connect server, and the Spark Operator
telemetry An OpenTelemetry Collector that forwards operational telemetry to Avra

Install both layers in the same namespace. The agent and the ingestion pipeline send their telemetry to the collector in their own namespace, and the collector reads logs and scrapes metrics only from the pods in the namespace it runs in.

What runs in your cluster

Once both layers are installed, these workloads run in Avra’s namespace:

Workload Comes from What it does
Deployment agent The install command Installs and upgrades the layers’ Helm releases, and reports their status
Avra agent customer-data Polls the control plane, runs the commands it receives, and reports their results. One pod, with a container for each of its tasks
Ingestion pipeline customer-data Runs your pipelines, as a single instance. Only the Avra agent can reach its API
Spark Connect server customer-data Does the pipeline’s processing, and starts executor pods in the namespace for the work
Spark Operator customer-data, optional Manages the Spark Connect server, watching only the namespaces it’s given
Table maintenance customer-data A scheduled job for each pipeline that expires old table snapshots, cleans up after rejected runs, and compacts files. It runs daily at 3:00 AM São Paulo time by default, off-peak, because compaction is heavy
Telemetry collector telemetry Collects metrics and traces from the pods in the namespace, the logs of those on its own node, and resource metrics for that node and every pod on it, and forwards them to Avra

Before you start

Your Avra team works through these prerequisites with you before the installation.

What you provide Example Notes
Kubernetes cluster Amazon EKS, or Google Kubernetes Engine (GKE) Leave room for the Spark executor pods that do the processing
Namespace avra Create it before you run the install command. Both layers install in it
Cluster-scoped permissions Cluster administrator access when you run the install command The layers create a few cluster-scoped resources: the Spark Operator’s custom resource definitions (CRDs) and ClusterRole, and the telemetry collector’s ClusterRole, which lets it read cluster objects and node metrics
NetworkPolicy enforcement A network plugin that enforces NetworkPolicies Inside the namespace, NetworkPolicies are what limit who can reach the Avra agent and the ingestion pipeline
Bucket s3://rio-claro-avra or gs://rio-claro-avra Holds the pipeline’s tables. S3-compatible storage works too, with its endpoint set
PostgreSQL database avra_pipeline on pg.rioclaro.internal:5432 Holds the pipeline’s run state and its table catalog. Dedicate it to this installation, and give it a user that can create tables
Identity for Spark An IAM role for the spark service account on EKS, or Workload Identity on GKE Gives Spark access to your sources and to the pipeline’s bucket. On EKS, Spark also needs a credentials setting to use the role, which your Avra team adds to your values. Where workload identity isn’t available, keys can come from a Kubernetes Secret
Outbound HTTPS See Network requirements Every connection to Avra opens from your cluster
Cluster name rio-claro-prod Identifies your cluster on every poll and every telemetry signal. Use a name specific to you, not a generic one like prod

Two of these deserve care. The database must belong to this installation alone: when the ingestion pipeline starts, it ends every run its database still lists as unfinished, so two installations sharing a database end each other’s runs. And its user must be able to create tables, because the pipeline applies its schema on every start. On PostgreSQL 15 and later, that means granting the user CREATE on schema public, or making it the database owner.

Avra provides the install command and the credentials the installation needs: the agent’s control-plane tokens, the pull credentials for Avra’s images, and the credentials for Avra’s telemetry endpoint.

Network requirements

The data layer only opens outbound connections. Allow these destinations from your cluster:

Destination Port Used by What for
Avra’s control plane 443, HTTPS Avra agent Polls for commands and reports their results
Avra’s telemetry endpoint 443, HTTPS Telemetry collector Forwards operational telemetry
The deployment agent’s endpoint 443, HTTPS Deployment agent Fetches your deployment’s versions and values, and sends the status of each release and its own logs
The registries that serve the layers’ charts and images 443, HTTPS Deployment agent, and your nodes’ container runtime Pulls charts and images
Your sources and the pipeline’s bucket Each service’s own Spark Reads your sources and writes the pipeline’s tables
Your PostgreSQL database 5432 by default Ingestion pipeline, Spark Connect server Keeps run state and the table catalog

Your Avra team gives you the exact hostnames to allow. Nothing outside your cluster connects in, so the data layer needs no inbound rule. Container images are pulled by your nodes’ container runtime, so its proxy and registry mirror settings apply to them.

Inside the namespace, NetworkPolicies limit who can reach the Avra agent and the ingestion pipeline: the ingestion pipeline accepts connections only from the Avra agent, and the agent’s report endpoint only from pods in its namespace. The layers don’t restrict outbound traffic, so control it at your network’s edge, using the destinations in the table.

Install

The install command starts a deployment agent in your namespace. The deployment agent fetches your deployment from outside, installs both layers with Helm, and reports their status. Once the Avra agent is running, it starts polling Avra’s control plane, and Avra takes over from there.

Share your values

Give your Avra team the values your deployment needs: the cluster name, the bucket, the database’s host and name, and the Spark identity. Avra sets up a deployment for your cluster with them, along with the agent’s credentials.

Create the namespace

Create the namespace for Avra:

kubectl create namespace avra

To keep the database credentials out of the deployment’s values, put them in a Secret you manage, and tell your Avra team its name. The keys below are the ones the ingestion pipeline reads by default:

kubectl create secret generic avra-pipeline-db --namespace avra \
  --from-literal=HOST=pg.rioclaro.internal \
  --from-literal=PORT=5432 \
  --from-literal=NAME=avra_pipeline \
  --from-literal=USER=avra \
  --from-file=PASSWORD=./pipeline-db-password

Run the install command

Run the install command Avra sends you. It installs a deployment agent in the namespace:

kubectl apply --namespace avra -f "<install-url>"

The layers install

The deployment agent installs the telemetry and customer-data charts, watches the health of what they create, and reports it.

Avra verifies the installation

The Avra agent starts polling the control plane. Avra checks that the ingestion pipeline can reach each of your sources and destinations, then sets up your first pipeline with you.

Verify the installation

You can check the installation from your side with kubectl and helm, while Avra confirms the rest from the control plane.

Check the pods

kubectl get pods --namespace avra

Look for pods for the deployment agent, the Avra agent, the ingestion pipeline, the Spark Connect server, the Spark Operator, and the telemetry collector, all Running. The ingestion pipeline’s pod shows Init until its database migration finishes.

Check that the Avra agent polls

kubectl logs --namespace avra --selector app.kubernetes.io/name=avra-agent --container poller --tail 50

A healthy agent logs poller started and an identity resolved line for each of its tokens, then polls quietly, logging again only when commands arrive. If its polls fail once it’s running, it logs poller: poll failed, retrying and backs off, doubling the wait between attempts up to a limit.

Check the releases

helm list --namespace avra

Both releases show the status deployed, and the CHART column shows each layer with the version Avra rolled out.

Avra checks access

From the control plane, Avra confirms that your agent’s polls arrive with your cluster name, and asks the ingestion pipeline to check access to each of your sources and destinations.

Troubleshooting

The ingestion pipeline’s pod stays in Init

Its database migration can’t finish. Find the pod with kubectl get pods --namespace avra, then read the migration’s logs:

kubectl logs --namespace avra <pipeline-pod> --container migrate

A refused connection or a failed authentication points at the database credentials or the network path to PostgreSQL. permission denied for schema public means the user can’t create tables: on PostgreSQL 15 and later, grant it CREATE on schema public, or make it the database owner. Kubernetes retries the migration with backoff, so the pod comes up on its own once the cause is fixed. If the error comes from inside a migration, your Avra team rolls the release back or releases one with the migration fixed.

The Avra agent logs poll failed, retrying

The agent can’t reach Avra’s control plane, or the control plane rejected its polls: the error in the log line says which. Check that your cluster allows outbound HTTPS to it, including through any proxy, and if it does, ask your Avra team to check the agent. The agent keeps retrying with a growing wait, and recovers on its own once its polls succeed.

The Avra agent's poller keeps restarting

If its logs show skipping unresolvable identity, the agent couldn’t confirm any of its tokens with the control plane, either because it can’t reach it or because the control plane rejected them. Check the outbound path first, then ask your Avra team to check the tokens.

Pods stay in ImagePullBackOff

Your nodes can’t pull an image. Check that they can reach the registries Avra listed, through your container runtime’s proxy or mirror settings if you use them. If the registry answers that access is denied, ask your Avra team to check the pull credentials in the namespace.

A Helm install stops on a Spark Operator check

When you manage the release yourself, the customer-data chart checks the Spark Operator’s settings before it renders anything. Its jobNamespaces must include the release namespace, and can’t include an empty entry, which would mean every namespace. Its service account needs a name other than default. If your cluster already runs a Spark Operator that watches the namespace, turn the bundled one off instead.

Upgrades and rollback

Avra releases new versions of each layer. To upgrade, Avra moves your deployment to a new version, and the deployment agent upgrades the release. An upgrade that fails is rolled back to the version that was running, and either way the deployment agent reports the outcome.

Because each layer version pins every component inside it, an upgrade moves the whole layer at once, and a rollback returns all of it to a version that worked.

The ingestion pipeline runs as a single instance, so an upgrade replaces its pod instead of rolling it. The new pod applies its database migrations before it serves, then ends any run its database still lists as unfinished. Plan upgrades with your Avra team for a time between runs.

The Spark Operator’s CRDs are cluster-scoped, and they install once and don’t upgrade with the layer. When a new version needs newer CRDs, they’re applied separately, either by hand or by turning on the operator’s CRD upgrade job, which applies them before each install and upgrade. The job is off by default, because a failed job can block upgrades.

Manage the release yourself

If you deploy with your own tooling, such as Helm or a GitOps controller, pull the same charts from Avra’s registry. Your Avra team issues the registry’s address and your credentials, along with a template of the values each layer needs for your environment.

helm registry login <avra-registry-host> --username <username> --password-stdin

helm upgrade --install telemetry \
  oci://<avra-registry>/charts/layers/telemetry \
  --version <version> --namespace avra --values telemetry.yaml

helm upgrade --install customer-data \
  oci://<avra-registry>/charts/layers/customer-data \
  --version <version> --namespace avra --values customer-data.yaml

To review what a layer creates before you apply it, render it locally:

helm template customer-data \
  oci://<avra-registry>/charts/layers/customer-data \
  --version <version> --namespace avra --values customer-data.yaml

Helm keeps each release’s history, so you can roll back to an earlier revision:

helm history customer-data --namespace avra
helm rollback customer-data <revision> --namespace avra

Configuration values

The template your Avra team gives you covers the settings below, and your Avra team fills it in with you. Keys are relative to each component’s section of the layer’s values. Keep secret values out of the files you store: pass them at install time, or point the installation at Kubernetes Secrets you manage. Helm stores the values of every revision in its release history, in Secrets in the namespace.

Avra agent

Setting What it sets
clusterName Required. Your cluster’s name, sent with every poll
controlPlane.url Required. The control plane’s address. Avra sets it
controlPlane.tokens, or controlPlane.existingSecret Required. The agent’s control-plane tokens, which Avra issues. Give them as values, or point to a Secret you manage
controlPlane.timeout Optional. The timeout for requests to the control plane
imagePullSecrets The Secret with the pull credentials for Avra’s images. It must exist in the namespace
networkPolicy.enabled, networkPolicy.ingressPodSelector On by default. Which pods in the namespace can reach the agent’s report endpoint: by default, any of them
resources, nodeSelector, tolerations, affinity, podAnnotations What the agent requests and where it runs. By default, each of its containers requests 50m of CPU and 64 MiB of memory, with a 128 MiB limit

Ingestion pipeline

Setting What it sets
image.tag Required. A pinned version of the pipeline’s image, never a floating tag such as latest
stateDb.host, .port, .name, .user, .password, or stateDb.existingSecret Required. The PostgreSQL database for run state and the table catalog. Prefer a Secret you manage
sparkConnect.catalog.warehouse Required when the layer runs its own Spark Connect server. The bucket path for the pipeline’s tables, such as s3://rio-claro-avra/warehouse
sparkConnect.serviceAccountName The service account Spark runs as. It must match the Spark Operator’s
sparkConnect.create, sparkConnectUrl To use a Spark Connect server you already run, set create to false and give its address in sparkConnectUrl
ingestion.byteCapBytes Required. How many bytes ingestion reads in each micro-batch, 100 MiB in the template. A run still reads everything available. Avra adjusts it to your data volume
drift.topN, drift.epsilon, drift.binCount, drift.binStrategy Required. How the quality gate builds the histograms it compares for drift: the categories kept for each categorical column, the smoothing for empty buckets, and the bins for each numeric column. equal_frequency is the only strategy
anomaly.k Required. How sensitive the anomaly score is
alert.enabled, alert.defaultChannel How Avra routes alerts on failed checks. Avra sets them
maintenance Table maintenance: whether it runs, its settings, and the pipelines it covers
resources, nodeSelector, tolerations, affinity, podAnnotations What the pipeline requests and where it runs. By default, it requests 250m of CPU and 256 MiB of memory, with a 512 MiB limit

Spark

Setting What it sets
enabled Whether the layer installs its own Spark Operator, on by default. Turn it off if your cluster already runs one that watches the namespace
spark.jobNamespaces Required. The namespaces the operator watches. It must include the release namespace, and can’t include an empty entry, which would mean every namespace
spark.serviceAccount.name, .create, .annotations Required. The service account Spark pods run as, created by default as spark. It can’t be default. Its annotations carry workload identity: an IAM role on EKS, or a Google service account on GKE
hook.upgradeCrd Off by default. Turn it on to apply the operator’s CRDs in a job before each install and upgrade

Telemetry collector

Setting What it sets
CLUSTER_NAME Required. Your cluster’s name, attached to every log, metric, and trace the collector forwards. Use a name specific to you, so your telemetry can’t be confused with another cluster’s
The telemetry endpoint and its credentials Where the collector sends telemetry, and how it authenticates. Avra sets them

Security

Connections are outbound only, and nothing connects into your cluster: the Avra agent polls Avra’s control plane for its commands, and the deployment agent fetches your deployment the same way. The deployment agent keeps cluster administrator access after the install, because the layers include cluster-scoped resources, and by default it updates itself.

The Avra agent runs with least privilege. Its role lets it get, create, and patch ConfigMaps, where it keeps its state, and get, create, and update Leases, which keep a single copy of each of its tasks running. The role covers its own namespace and nothing else. The ingestion pipeline doesn’t use the Kubernetes API at all: its service account mounts no token. Both run as a non-root user, with a read-only root filesystem, no privilege escalation, every Linux capability dropped, and the container runtime’s default seccomp profile.

Each layer has its own Avra agent, built with only that layer’s commands, so an agent never carries another layer’s code. The agent doesn’t interpret a command: it passes it to the ingestion pipeline and returns the answer. The data layer’s agent runs these commands:

Command What it does
Validate a pipeline Checks a pipeline definition without starting a run, and optionally checks access to its sources and destinations
Run a pipeline Starts a run of a registered pipeline by name, or of a pipeline sent in full
Get a run’s status Returns where a run stands
Cancel a run Stops a run at its next stage boundary
Restart a component Restarts one ingestion component of a run in progress
Reset a checkpoint Reads one source again from the start, apart from any run

Secrets stay in Kubernetes: the database credentials and the control-plane tokens can come from Secrets you already manage instead of plain values. Keep write access to Avra’s namespace limited, too. Anyone who can create pods there can run them with the service accounts of Avra’s components, the deployment agent’s included, and anyone who can read Secrets there can read the values of every release.

The telemetry layer forwards operational telemetry, labeled with your cluster name: the logs of the pods in Avra’s namespace that run on the collector’s node, the metrics and traces of Avra’s components, and the resource metrics the kubelet reports for the node the collector runs on and the pods on it.

The ingestion pipeline in your environment

Once the data layer is running, Avra’s control plane drives the ingestion pipeline through the Avra agent: it validates pipeline definitions, checks access to your sources and destinations, starts and cancels runs, and follows their status. Spark runs the work in your cluster, the pipeline’s tables stay in your bucket, and its run state stays in your database.

Esta página foi útil?