---
title: Install the data layer
description: Install Avra's ingestion pipeline and agent in your Kubernetes cluster with Helm, verify the installation, and keep it up to date.
---

The data layer is the part of Avra that runs in your Kubernetes cluster. It holds the ingestion pipeline, which validates and versions your data with Spark, and the Avra agent, which takes the pipeline's commands from Avra's control plane. You provide the cluster and the infrastructure around it, and a deployment agent installs and upgrades the rest.

In short: you provide the prerequisites, Avra configures your deployment, you run one install command, and Avra verifies the result. This page covers each step, then how to check the installation, how upgrades work, and the settings your deployment covers.

## The layers

Avra packages what runs in your cluster as Helm charts. The charts are grouped in layers, and each layer is versioned and released as a unit. A layer version pins the version of every component inside it, so a release is reproducible.

| Layer | What it installs |
|-------|------------------|
| **`customer-data`** | The Avra agent, the ingestion pipeline, a Spark Connect server, and the Spark Operator |
| **`telemetry`** | An OpenTelemetry Collector that forwards operational telemetry to Avra |

Install both layers in the same namespace. The agent and the ingestion pipeline send their telemetry to the collector in their own namespace, and the collector reads logs and scrapes metrics only from the pods in the namespace it runs in.

### What runs in your cluster

Once both layers are installed, these workloads run in Avra's namespace:

| Workload | Comes from | What it does |
|----------|------------|--------------|
| **Deployment agent** | The install command | Installs and upgrades the layers' Helm releases, and reports their status |
| **Avra agent** | `customer-data` | Polls the control plane, runs the commands it receives, and reports their results. One pod, with a container for each of its tasks |
| **Ingestion pipeline** | `customer-data` | Runs your pipelines, as a single instance. Only the Avra agent can reach its API |
| **Spark Connect server** | `customer-data` | Does the pipeline's processing, and starts executor pods in the namespace for the work |
| **Spark Operator** | `customer-data`, optional | Manages the Spark Connect server, watching only the namespaces it's given |
| **Table maintenance** | `customer-data` | A scheduled job for each pipeline that expires old table snapshots, cleans up after rejected runs, and compacts files. It runs daily at 3:00 AM São Paulo time by default, off-peak, because compaction is heavy |
| **Telemetry collector** | `telemetry` | Collects metrics and traces from the pods in the namespace, the logs of those on its own node, and resource metrics for that node and every pod on it, and forwards them to Avra |

## Before you start

Your Avra team works through these prerequisites with you before the installation.

| What you provide | Example | Notes |
|------------------|---------|-------|
| **Kubernetes cluster** | Amazon EKS, or Google Kubernetes Engine (GKE) | Leave room for the Spark executor pods that do the processing |
| **Namespace** | `avra` | Create it before you run the install command. Both layers install in it |
| **Cluster-scoped permissions** | Cluster administrator access when you run the install command | The layers create a few cluster-scoped resources: the Spark Operator's custom resource definitions (CRDs) and ClusterRole, and the telemetry collector's ClusterRole, which lets it read cluster objects and node metrics |
| **NetworkPolicy enforcement** | A network plugin that enforces NetworkPolicies | Inside the namespace, NetworkPolicies are what limit who can reach the Avra agent and the ingestion pipeline |
| **Bucket** | `s3://rio-claro-avra` or `gs://rio-claro-avra` | Holds the pipeline's tables. S3-compatible storage works too, with its endpoint set |
| **PostgreSQL database** | `avra_pipeline` on `pg.rioclaro.internal:5432` | Holds the pipeline's run state and its table catalog. Dedicate it to this installation, and give it a user that can create tables |
| **Identity for Spark** | An IAM role for the `spark` service account on EKS, or Workload Identity on GKE | Gives Spark access to your sources and to the pipeline's bucket. On EKS, Spark also needs a credentials setting to use the role, which your Avra team adds to your values. Where workload identity isn't available, keys can come from a Kubernetes Secret |
| **Outbound HTTPS** | See [Network requirements](#network-requirements) | Every connection to Avra opens from your cluster |
| **Cluster name** | `rio-claro-prod` | Identifies your cluster on every poll and every telemetry signal. Use a name specific to you, not a generic one like `prod` |

Two of these deserve care. The database must belong to this installation alone: when the ingestion pipeline starts, it ends every run its database still lists as unfinished, so two installations sharing a database end each other's runs. And its user must be able to create tables, because the pipeline applies its schema on every start. On PostgreSQL 15 and later, that means granting the user `CREATE` on schema `public`, or making it the database owner.

Avra provides the install command and the credentials the installation needs: the agent's control-plane tokens, the pull credentials for Avra's images, and the credentials for Avra's telemetry endpoint.

## Network requirements

The data layer only opens outbound connections. Allow these destinations from your cluster:

| Destination | Port | Used by | What for |
|-------------|------|---------|----------|
| **Avra's control plane** | 443, HTTPS | Avra agent | Polls for commands and reports their results |
| **Avra's telemetry endpoint** | 443, HTTPS | Telemetry collector | Forwards operational telemetry |
| **The deployment agent's endpoint** | 443, HTTPS | Deployment agent | Fetches your deployment's versions and values, and sends the status of each release and its own logs |
| **The registries that serve the layers' charts and images** | 443, HTTPS | Deployment agent, and your nodes' container runtime | Pulls charts and images |
| **Your sources and the pipeline's bucket** | Each service's own | Spark | Reads your sources and writes the pipeline's tables |
| **Your PostgreSQL database** | 5432 by default | Ingestion pipeline, Spark Connect server | Keeps run state and the table catalog |

Your Avra team gives you the exact hostnames to allow. Nothing outside your cluster connects in, so the data layer needs no inbound rule. Container images are pulled by your nodes' container runtime, so its proxy and registry mirror settings apply to them.

Inside the namespace, NetworkPolicies limit who can reach the Avra agent and the ingestion pipeline: the ingestion pipeline accepts connections only from the Avra agent, and the agent's report endpoint only from pods in its namespace. The layers don't restrict outbound traffic, so control it at your network's edge, using the destinations in the table.

## Install

The install command starts a deployment agent in your namespace. The deployment agent fetches your deployment from outside, installs both layers with Helm, and reports their status. Once the Avra agent is running, it starts polling Avra's control plane, and Avra takes over from there.

```mermaid
---
config:
  sequence:
    width: 110
    actorMargin: 30
---
sequenceDiagram
    autonumber
    participant S as Deployment<br/>service
    participant D as Deployment<br/>agent
    participant A as Avra<br/>agent
    participant C as Control<br/>plane
    Note over D: You apply the<br/>install command
    D->>S: Fetch deployment
    D->>D: Install both layers
    D->>S: Report status
    D->>A: Start agent
    A->>C: Poll for commands
    C-->>A: Check access
```

1. **Share your values**

    Give your Avra team the values your deployment needs: the cluster name, the bucket, the database's host and name, and the Spark identity. Avra sets up a deployment for your cluster with them, along with the agent's credentials.

2. **Create the namespace**

    Create the namespace for Avra:

    ```bash
    kubectl create namespace avra
    ```

    To keep the database credentials out of the deployment's values, put them in a Secret you manage, and tell your Avra team its name. The keys below are the ones the ingestion pipeline reads by default:

    ```bash
    kubectl create secret generic avra-pipeline-db --namespace avra \
      --from-literal=HOST=pg.rioclaro.internal \
      --from-literal=PORT=5432 \
      --from-literal=NAME=avra_pipeline \
      --from-literal=USER=avra \
      --from-file=PASSWORD=./pipeline-db-password
    ```

3. **Run the install command**

    Run the install command Avra sends you. It installs a deployment agent in the namespace:

    ```bash
    kubectl apply --namespace avra -f "<install-url>"
    ```

    > **The install command carries a credential**
    >
    > The address in the command ties the deployment agent to your deployment. Treat it as a secret: it can end up in your shell history and in the logs of a proxy that inspects TLS traffic. If it leaks, ask your Avra team for a new one. The old one stops working once a deployment agent connects with the new one.

4. **The layers install**

    The deployment agent installs the `telemetry` and `customer-data` charts, watches the health of what they create, and reports it.

5. **Avra verifies the installation**

    The Avra agent starts polling the control plane. Avra checks that the ingestion pipeline can reach each of your sources and destinations, then sets up your first pipeline with you.

## Verify the installation

You can check the installation from your side with `kubectl` and `helm`, while Avra confirms the rest from the control plane.

1. **Check the pods**

    ```bash
    kubectl get pods --namespace avra
    ```

    Look for pods for the deployment agent, the Avra agent, the ingestion pipeline, the Spark Connect server, the Spark Operator, and the telemetry collector, all `Running`. The ingestion pipeline's pod shows `Init` until its database migration finishes.

2. **Check that the Avra agent polls**

    ```bash
    kubectl logs --namespace avra --selector app.kubernetes.io/name=avra-agent --container poller --tail 50
    ```

    A healthy agent logs `poller started` and an `identity resolved` line for each of its tokens, then polls quietly, logging again only when commands arrive. If its polls fail once it's running, it logs `poller: poll failed, retrying` and backs off, doubling the wait between attempts up to a limit.

3. **Check the releases**

    ```bash
    helm list --namespace avra
    ```

    Both releases show the status `deployed`, and the `CHART` column shows each layer with the version Avra rolled out.

4. **Avra checks access**

    From the control plane, Avra confirms that your agent's polls arrive with your cluster name, and asks the ingestion pipeline to check access to each of your sources and destinations.

## Troubleshooting

**The ingestion pipeline's pod stays in `Init`**

Its database migration can't finish. Find the pod with `kubectl get pods --namespace avra`, then read the migration's logs:

```bash
kubectl logs --namespace avra <pipeline-pod> --container migrate
```

A refused connection or a failed authentication points at the database credentials or the network path to PostgreSQL. `permission denied for schema public` means the user can't create tables: on PostgreSQL 15 and later, grant it `CREATE` on schema `public`, or make it the database owner. Kubernetes retries the migration with backoff, so the pod comes up on its own once the cause is fixed. If the error comes from inside a migration, your Avra team rolls the release back or releases one with the migration fixed.

**The Avra agent logs `poll failed, retrying`**

The agent can't reach Avra's control plane, or the control plane rejected its polls: the error in the log line says which. Check that your cluster allows outbound HTTPS to it, including through any proxy, and if it does, ask your Avra team to check the agent. The agent keeps retrying with a growing wait, and recovers on its own once its polls succeed.

**The Avra agent's poller keeps restarting**

If its logs show `skipping unresolvable identity`, the agent couldn't confirm any of its tokens with the control plane, either because it can't reach it or because the control plane rejected them. Check the outbound path first, then ask your Avra team to check the tokens.

**Pods stay in `ImagePullBackOff`**

Your nodes can't pull an image. Check that they can reach the registries Avra listed, through your container runtime's proxy or mirror settings if you use them. If the registry answers that access is denied, ask your Avra team to check the pull credentials in the namespace.

**A Helm install stops on a Spark Operator check**

When you manage the release yourself, the `customer-data` chart checks the Spark Operator's settings before it renders anything. Its `jobNamespaces` must include the release namespace, and can't include an empty entry, which would mean every namespace. Its service account needs a name other than `default`. If your cluster already runs a Spark Operator that watches the namespace, turn the bundled one off instead.

## Upgrades and rollback

Avra releases new versions of each layer. To upgrade, Avra moves your deployment to a new version, and the deployment agent upgrades the release. An upgrade that fails is rolled back to the version that was running, and either way the deployment agent reports the outcome.

```mermaid
flowchart LR
    A["Avra moves your<br/>deployment to a<br/>new layer version"] --> B["The deployment<br/>agent upgrades<br/>the release"]
    B --> C{"Upgrade<br/>succeeds?"}
    C -- Yes --> D["The new<br/>version runs"]
    C -- No --> E["Rolled back to<br/>the previous<br/>version"]
    classDef accent fill:#0A01FE,stroke:#0A01FE,color:#fff
    class D accent
```

Because each layer version pins every component inside it, an upgrade moves the whole layer at once, and a rollback returns all of it to a version that worked.

The ingestion pipeline runs as a single instance, so an upgrade replaces its pod instead of rolling it. The new pod applies its database migrations before it serves, then ends any run its database still lists as unfinished. Plan upgrades with your Avra team for a time between runs.

The Spark Operator's CRDs are cluster-scoped, and they install once and don't upgrade with the layer. When a new version needs newer CRDs, they're applied separately, either by hand or by turning on the operator's CRD upgrade job, which applies them before each install and upgrade. The job is off by default, because a failed job can block upgrades.

## Manage the release yourself

If you deploy with your own tooling, such as Helm or a GitOps controller, pull the same charts from Avra's registry. Your Avra team issues the registry's address and your credentials, along with a template of the values each layer needs for your environment.

```bash
helm registry login <avra-registry-host> --username <username> --password-stdin

helm upgrade --install telemetry \
  oci://<avra-registry>/charts/layers/telemetry \
  --version <version> --namespace avra --values telemetry.yaml

helm upgrade --install customer-data \
  oci://<avra-registry>/charts/layers/customer-data \
  --version <version> --namespace avra --values customer-data.yaml
```

To review what a layer creates before you apply it, render it locally:

```bash
helm template customer-data \
  oci://<avra-registry>/charts/layers/customer-data \
  --version <version> --namespace avra --values customer-data.yaml
```

Helm keeps each release's history, so you can roll back to an earlier revision:

```bash
helm history customer-data --namespace avra
helm rollback customer-data <revision> --namespace avra
```

:::warning[Choose one way to manage the releases]
Either the deployment agent manages the layers' releases, or you do. Running `helm upgrade` or `helm rollback` on a release the deployment agent manages stops it from applying later versions.
:::

## Configuration values

The template your Avra team gives you covers the settings below, and your Avra team fills it in with you. Keys are relative to each component's section of the layer's values. Keep secret values out of the files you store: pass them at install time, or point the installation at Kubernetes Secrets you manage. Helm stores the values of every revision in its release history, in Secrets in the namespace.

### Avra agent

| Setting | What it sets |
|---------|--------------|
| `clusterName` | Required. Your cluster's name, sent with every poll |
| `controlPlane.url` | Required. The control plane's address. Avra sets it |
| `controlPlane.tokens`, or `controlPlane.existingSecret` | Required. The agent's control-plane tokens, which Avra issues. Give them as values, or point to a Secret you manage |
| `controlPlane.timeout` | Optional. The timeout for requests to the control plane |
| `imagePullSecrets` | The Secret with the pull credentials for Avra's images. It must exist in the namespace |
| `networkPolicy.enabled`, `networkPolicy.ingressPodSelector` | On by default. Which pods in the namespace can reach the agent's report endpoint: by default, any of them |
| `resources`, `nodeSelector`, `tolerations`, `affinity`, `podAnnotations` | What the agent requests and where it runs. By default, each of its containers requests 50m of CPU and 64 MiB of memory, with a 128 MiB limit |

### Ingestion pipeline

| Setting | What it sets |
|---------|--------------|
| `image.tag` | Required. A pinned version of the pipeline's image, never a floating tag such as `latest` |
| `stateDb.host`, `.port`, `.name`, `.user`, `.password`, or `stateDb.existingSecret` | Required. The PostgreSQL database for run state and the table catalog. Prefer a Secret you manage |
| `sparkConnect.catalog.warehouse` | Required when the layer runs its own Spark Connect server. The bucket path for the pipeline's tables, such as `s3://rio-claro-avra/warehouse` |
| `sparkConnect.serviceAccountName` | The service account Spark runs as. It must match the Spark Operator's |
| `sparkConnect.create`, `sparkConnectUrl` | To use a Spark Connect server you already run, set `create` to `false` and give its address in `sparkConnectUrl` |
| `ingestion.byteCapBytes` | Required. How many bytes ingestion reads in each micro-batch, 100 MiB in the template. A run still reads everything available. Avra adjusts it to your data volume |
| `drift.topN`, `drift.epsilon`, `drift.binCount`, `drift.binStrategy` | Required. How the quality gate builds the histograms it compares for drift: the categories kept for each categorical column, the smoothing for empty buckets, and the bins for each numeric column. `equal_frequency` is the only strategy |
| `anomaly.k` | Required. How sensitive the anomaly score is |
| `alert.enabled`, `alert.defaultChannel` | How Avra routes alerts on failed checks. Avra sets them |
| `maintenance` | Table maintenance: whether it runs, its settings, and the pipelines it covers |
| `resources`, `nodeSelector`, `tolerations`, `affinity`, `podAnnotations` | What the pipeline requests and where it runs. By default, it requests 250m of CPU and 256 MiB of memory, with a 512 MiB limit |

### Spark

| Setting | What it sets |
|---------|--------------|
| `enabled` | Whether the layer installs its own Spark Operator, on by default. Turn it off if your cluster already runs one that watches the namespace |
| `spark.jobNamespaces` | Required. The namespaces the operator watches. It must include the release namespace, and can't include an empty entry, which would mean every namespace |
| `spark.serviceAccount.name`, `.create`, `.annotations` | Required. The service account Spark pods run as, created by default as `spark`. It can't be `default`. Its annotations carry workload identity: an IAM role on EKS, or a Google service account on GKE |
| `hook.upgradeCrd` | Off by default. Turn it on to apply the operator's CRDs in a job before each install and upgrade |

### Telemetry collector

| Setting | What it sets |
|---------|--------------|
| `CLUSTER_NAME` | Required. Your cluster's name, attached to every log, metric, and trace the collector forwards. Use a name specific to you, so your telemetry can't be confused with another cluster's |
| The telemetry endpoint and its credentials | Where the collector sends telemetry, and how it authenticates. Avra sets them |

## Security

Connections are outbound only, and nothing connects into your cluster: the Avra agent polls Avra's control plane for its commands, and the deployment agent fetches your deployment the same way. The deployment agent keeps cluster administrator access after the install, because the layers include cluster-scoped resources, and by default it updates itself.

The Avra agent runs with least privilege. Its role lets it get, create, and patch ConfigMaps, where it keeps its state, and get, create, and update Leases, which keep a single copy of each of its tasks running. The role covers its own namespace and nothing else. The ingestion pipeline doesn't use the Kubernetes API at all: its service account mounts no token. Both run as a non-root user, with a read-only root filesystem, no privilege escalation, every Linux capability dropped, and the container runtime's default seccomp profile.

Each layer has its own Avra agent, built with only that layer's commands, so an agent never carries another layer's code. The agent doesn't interpret a command: it passes it to the ingestion pipeline and returns the answer. The data layer's agent runs these commands:

| Command | What it does |
|---------|--------------|
| **Validate a pipeline** | Checks a pipeline definition without starting a run, and optionally checks access to its sources and destinations |
| **Run a pipeline** | Starts a run of a registered pipeline by name, or of a pipeline sent in full |
| **Get a run's status** | Returns where a run stands |
| **Cancel a run** | Stops a run at its next stage boundary |
| **Restart a component** | Restarts one ingestion component of a run in progress |
| **Reset a checkpoint** | Reads one source again from the start, apart from any run |

Secrets stay in Kubernetes: the database credentials and the control-plane tokens can come from Secrets you already manage instead of plain values. Keep write access to Avra's namespace limited, too. Anyone who can create pods there can run them with the service accounts of Avra's components, the deployment agent's included, and anyone who can read Secrets there can read the values of every release.

The telemetry layer forwards operational telemetry, labeled with your cluster name: the logs of the pods in Avra's namespace that run on the collector's node, the metrics and traces of Avra's components, and the resource metrics the kubelet reports for the node the collector runs on and the pods on it.

## The ingestion pipeline in your environment

Once the data layer is running, Avra's control plane drives the ingestion pipeline through the Avra agent: it validates pipeline definitions, checks access to your sources and destinations, starts and cancels runs, and follows their status. Spark runs the work in your cluster, the pipeline's tables stay in your bucket, and its run state stays in your database.

**[Data ingestion](/pt-BR/data-ingestion)**

How a pipeline validates, versions, and publishes your data

**[Avra RFM on AWS SageMaker](/pt-BR/deployment/aws-sagemaker)**

Run the model that reads what the pipeline publishes, in your AWS account
