# Source Hydrator {#source-hydrator} > [!WARNING] > **Beta Feature (Since v3.5.0)** > > This is a [beta-quality](https://github.com/argoproj/argoproj/blob/main/community/feature-status.md#beta) > feature that pushes hydrated manifests to git before syncing them to the cluster. Tools like Helm and Kustomize allow users to express their Kubernetes manifests in a more concise and reusable way (keeping it DRY - Don't Repeat Yourself). However, these tools can obscure the actual Kubernetes manifests that are applied to the cluster. The *rendered manifest pattern* is a feature of Argo CD that allows users to push the hydrated manifests to git before syncing them to the cluster. This allows users to see the actual Kubernetes manifests that are applied to the cluster. ## Enabling the Source Hydrator {#enabling-the-source-hydrator} The source hydrator is disabled by default. To enable the source hydrator, you need to enable the "commit server" component and set the `hydrator.enabled` field in `argocd-cmd-params-cm` ConfigMap to `"true"`. ```yaml apiVersion: v1 kind: ConfigMap metadata: name: argocd-cmd-params-cm namespace: argocd data: hydrator.enabled: "true" ``` > [!IMPORTANT] > **Important** > > After updating the ConfigMap, you must restart the Argo CD controller and API server for the changes to take effect. If you are using one of the `*-install.yaml` manifests to install Argo CD, you can use the `*-install-with-hydrator.yaml` version of that file instead. For example, ``` Without hydrator: https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml With hydrator: https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install-with-hydrator.yaml ``` > [!IMPORTANT] > **Important** > > The `*-install-with-hydrator.yaml` manifests will eventually be removed when the source hydrator is either enabled > by default or removed. The upgrade guide will note if the `install-with-hydrator.yaml` manifests are no longer > available. ## Using the Source Hydrator {#using-the-source-hydrator} To use the source hydrator, you must first install a push and a pull secret. This example uses a GitHub App for authentication, but you can use [any authentication method that Argo CD supports for repository access](https://sourcey-argocd.pages.dev/operator-manual/declarative-setup/index.md#repositories). ```yaml apiVersion: v1 kind: Secret metadata: name: my-push-secret namespace: argocd labels: argocd.argoproj.io/secret-type: repository-write type: Opaque stringData: url: "https://github.com//" type: "git" githubAppID: "" # githubAppInstallationID is optional and will be auto-discovered if omitted githubAppInstallationID: "" # Optional githubAppPrivateKey: | --- apiVersion: v1 kind: Secret metadata: name: my-pull-secret namespace: argocd labels: argocd.argoproj.io/secret-type: repository type: Opaque stringData: url: "https://github.com//" type: "git" githubAppID: "" # githubAppInstallationID is optional and will be auto-discovered if omitted githubAppInstallationID: "" # Optional githubAppPrivateKey: | ``` The only difference between the secrets above, besides the resource name, is that the push secret contains the label `argocd.argoproj.io/secret-type: repository-write`, which causes the Secret to be used for pushing manifests to git instead of pulling from Git. Argo CD requires different secrets for pushing and pulling to provide better isolation. Once your secrets are installed, set the `spec.sourceHydrator` field of the Application. For example: ```yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: my-app spec: sourceHydrator: drySource: repoURL: https://github.com/argoproj/argocd-example-apps path: helm-guestbook targetRevision: HEAD syncSource: targetBranch: environments/dev path: helm-guestbook ``` In this example, the hydrated manifests will be pushed to the `environments/dev` branch of the `argocd-example-apps` repository. The `drySource` field tells Argo CD where your original, unrendered configuration lives. This can be a Helm chart, a Kustomize directory, or plain manifests. Argo CD reads this source, renders the final Kubernetes manifests from it, and then writes those hydrated manifests into the location specified by `syncSource.path`. ### Separate destination repository {#separate-destination-repository} By default, hydrated manifests are written to the same Git repository as the dry source. Set `syncSource.repoURL` to hydrate into a different repository. When `syncSource.repoURL` is omitted, it defaults to `drySource.repoURL`. ```yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: my-app spec: sourceHydrator: drySource: repoURL: https://github.com/my-org/config path: helm-guestbook targetRevision: HEAD syncSource: repoURL: https://github.com/my-org/deployments targetBranch: environments/dev path: helm-guestbook ``` Both repositories must be permitted in the application's AppProject (`spec.sourceRepos`). The hydrator needs read access to the dry source repository and write access to the destination repository (a `repository-write` secret; see the push secret example above). > [!NOTE] > `hydrateTo` inherits its repository and path from `syncSource`. When `syncSource.repoURL` points to a separate > repository, staged manifests are pushed to that repository as well. When using source hydration, the `syncSource.path` field is required and must always point to a non-root directory in the repository. Setting the path to the repository root (for example `"."` or `""`) is not supported. This ensures that hydration is always scoped to a dedicated subdirectory, which avoids unintentionally overwriting or removing files that may exist in the repository root. During each hydration run, Argo CD overwrites the files it generates (such as `manifest.yaml`) in the application's configured path, but it does **not** delete other files already present in that path. Stale files from a previous hydration that are not overwritten will remain in the output directory. Because the generated `manifest.yaml` is fully rewritten on every run, resources that were removed from the dry source disappear from it and are pruned on the next sync (when the `prune` sync option is enabled); however, extra leftover files are not removed automatically. The repository root is never written to, so files such as CI/CD configuration, README files, or other root-level assets remain untouched. If an application’s path changes, the old directory is not removed automatically. Likewise, if an application is deleted, its output path remains in the repository and must be cleaned up manually by the repository owner if desired. This design is intentional: it prevents accidental deletion of files when applications are restructured or removed, and it protects critical files like CI pipelines that may coexist in the repository. > [!NOTE] > The hydrator triggers only when a new commit is detected in the dry source. > Adding or removing Applications does not on its own cause hydration to run. > If the set of Applications changes but the dry-source commit does not, hydration will wait until the next commit. > > This is by design: the hydrator produces one hydration per dry-source commit and maintains atomicity across the > Applications that share a destination. If you add Applications that depend on a dry-source commit that has already > been hydrated (for example, Applications generated by an ApplicationSet), push an empty commit to the dry source so > the new Applications are hydrated. > [!IMPORTANT] > **Important** > > **Project-Scoped Repositories** Repository Secrets may contain a `project` field, making the secret only usable by Applications in that project. The source hydrator only supports project-scoped repositories if all Applications writing to the same repository and branch are in the same project. If Applications in different projects write to the same repository and branch, the source hydrator will not be able to use a project-scoped repository secret and will require a global repository secret. This behavior may change in the future. If there are multiple repository-write Secrets available for a repo, the source hydrator will non-deterministically select one of the matching Secrets and log a warning saying "Found multiple credentials for repoURL". ## Source Configuration Options {#source-configuration-options} The source hydrator supports various source types through inline configuration options in the `drySource` field. This allows you to use Helm charts, Kustomize applications, directories, and plugins with environment-specific configurations. ### Helm Charts {#helm-charts} You can use Helm charts by specifying the `helm` field in the `drySource`: ```yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: my-helm-app spec: sourceHydrator: drySource: repoURL: https://github.com/argoproj/argocd-example-apps path: helm-guestbook targetRevision: HEAD helm: valueFiles: - values-prod.yaml parameters: - name: image.tag value: v1.2.3 releaseName: my-release syncSource: targetBranch: environments/prod path: helm-guestbook-hydrated ``` ### Kustomize Applications {#kustomize-applications} For Kustomize applications, use the `kustomize` field: ```yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: my-kustomize-app spec: sourceHydrator: drySource: repoURL: https://github.com/argoproj/argocd-example-apps path: kustomize-guestbook targetRevision: HEAD kustomize: namePrefix: prod- nameSuffix: -v1 images: - gcr.io/heptio-images/ks-guestbook-demo:0.2 syncSource: targetBranch: environments/prod path: kustomize-guestbook-hydrated ``` ### Directory Applications {#directory-applications} For plain directory applications with specific options, use the `directory` field: ```yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: my-directory-app spec: sourceHydrator: drySource: repoURL: https://github.com/argoproj/argocd-example-apps path: guestbook targetRevision: HEAD directory: recurse: true syncSource: targetBranch: environments/prod path: guestbook-hydrated ``` ### Config Management Plugins {#config-management-plugins} You can also use Config Management Plugins by specifying the `plugin` field: ```yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: my-plugin-app spec: sourceHydrator: drySource: repoURL: https://github.com/argoproj/argocd-example-apps path: my-plugin-app targetRevision: HEAD plugin: name: my-custom-plugin env: - name: ENV_VAR value: prod syncSource: targetBranch: environments/prod path: my-plugin-app-hydrated ``` > [!NOTE] > **Feature Parity** > > The source hydrator supports the same configuration options as the regular Application source field. You can use any combination of these source types with their respective configuration options to match your application's needs. ## Pushing to a "Staging" Branch {#pushing-to-a-staging-branch} The source hydrator can be used to push hydrated manifests to a "staging" branch instead of the `syncSource` branch. This provides a way to prevent the hydrated manifests from being applied to the cluster until some prerequisite conditions are met (in effect, providing a way to handle environment promotion via Pull Requests). To use the source hydrator to push to a "staging" branch, set the `spec.sourceHydrator.hydrateTo` field of the Application. For example: ```yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: my-app spec: project: my-project destination: server: https://kubernetes.default.svc namespace: default sourceHydrator: drySource: repoURL: https://github.com/argoproj/argocd-example-apps path: helm-guestbook targetRevision: HEAD syncSource: targetBranch: environments/dev path: helm-guestbook hydrateTo: targetBranch: environments/dev-next ``` In this example, the hydrated manifests will be pushed to the `environments/dev-next` branch, and Argo CD will not sync the changes until something moves them to the `environments/dev` branch. You could use a CI action to move the hydrated manifests from the `hydrateTo` branch to the `syncSource` branch. To introduce a gating mechanism, you could require a Pull Request to be opened to merge the changes from the `hydrateTo` branch to the `syncSource` branch. Argo CD will only push changes to the `hydrateTo` branch, it will not create a PR or otherwise facilitate moving those changes to the `syncSource` branch. You will need to use your own tooling to move the changes from the `hydrateTo` branch to the `syncSource` branch. ## Commit Tracing {#commit-tracing} It's common for CI or other tooling to push DRY manifest changes after a code change. It's important for users to be able to trace the hydrated commits back to the original code change that caused the hydration. Source Hydrator makes use of some custom git commit trailers to facilitate this tracing. A CI job that builds an image and pushes an image bump to DRY manifests can use the following commit trailers to link the hydrated commit to the code commit. ```shell git commit -m "Bump image to v1.2.3" \ # Must be an RFC 5322 name --trailer "Argocd-reference-commit-author: Author Name " \ # Must be a hex string 5-40 characters long --trailer "Argocd-reference-commit-sha: " \ # The subject is the first line of the commit message. It cannot contain newlines. --trailer "Argocd-reference-commit-subject: Commit message of the code commit" \ # The body must be a valid JSON string, including opening and closing quotes --trailer 'Argocd-reference-commit-body: "Commit message of the code commit\n\nSigned-off-by: Author Name "' \ # The repo URL must be a valid URL --trailer "Argocd-reference-commit-repourl: https://git.example.com/owner/repo" \ # The date must by in ISO 8601 format --trailer "Argocd-reference-commit-date: 2025-06-09T13:50:18-04:00" ``` > [!NOTE] > The commit trailers must not contain newlines. So the full CI script might look something like this: ```shell # Clone code repo git clone https://git.example.com/owner/repo.git cd repo # Build the image and get the new image tag # # Get the commit information author=$(git show -s --format="%an <%ae>") sha=$(git rev-parse HEAD) subject=$(git show -s --format='%s') body=$(git show -s --format='%b') jsonbody=$(jq -n --arg body "$body" '$body') repourl=$(git remote get-url origin) date=$(git show -s --format='%aI') # Clone the dry source repo git clone https://git.example.com/owner/deployment-repo.git cd deployment-repo # Bump the image in the dry manifests # # Commit the changes with the commit trailers git commit -m "Bump image to v1.2.3" \ --trailer "Argocd-reference-commit-author: $author" \ --trailer "Argocd-reference-commit-sha: $sha" \ --trailer "Argocd-reference-commit-subject: $subject" \ --trailer "Argocd-reference-commit-body: $jsonbody" \ --trailer "Argocd-reference-commit-repourl: $repourl" \ --trailer "Argocd-reference-commit-date: $date" ``` The commit metadata will appear in the hydrated commit's root `hydrator.metadata` file: ```json { "author": "CI ", "subject": "chore: bump image to b82add2", "date": "2025-06-09T13:50:08-04:00", "body": "Signed-off-by: CI \n", "drySha": "6cb951525937865dced818bbdd78c89b2d2b3045", "repoURL": "https://git.example.com/owner/manifests-repo", "references": [ { "commit": { "author": { "name": "Author Name", "email": "author@example.com" }, "sha": "b82add298aa045d3672880802d5305c5a8aaa46e", "subject": "chore: make a change", "body": "make a change\n\nSigned-off-by: Author Name ", "repoURL": "https://git.example.com/owner/repo", "date": "2025-06-09T13:50:18-04:00" } } ] } ``` The top-level "body" field contains the commit message of the DRY commit minus the subject line and any `Argocd-reference-commit-*` trailers that were used in `references`. Unrecognized or invalid trailers are preserved in the body. Although `references` is an array, the source hydrator currently only supports a single related commit. If a trailer is specified more than once, the last one will be used. All trailers are optional. If a trailer is not specified, the corresponding field in the metadata will be omitted. ## Commit Message Template {#commit-message-template} The commit message is generated using a [Go text/template](https://pkg.go.dev/text/template), optionally configured by the user via the argocd-cm ConfigMap. The template is rendered using the values from `hydrator.metadata`. The template can be multi-line, allowing users to define a subject line, body and optional trailers. To define the commit message template, you need to set the `sourceHydrator.commitMessageTemplate` field in argocd-cm ConfigMap. The template can invoke functions from the [Sprig function library](https://github.com/Masterminds/sprig). ```yaml apiVersion: v1 kind: ConfigMap metadata: name: argocd-cm namespace: argocd data: sourceHydrator.commitMessageTemplate: | {{.metadata.drySha | trunc 7}}: {{ .metadata.subject }} {{- if .metadata.body }} {{ .metadata.body }} {{- end }} {{ range $ref := .metadata.references }} {{- if and $ref.commit $ref.commit.author }} Co-authored-by: {{ $ref.commit.author }} {{- end }} {{- end }} {{- if .metadata.author }} Co-authored-by: {{ .metadata.author }} {{- end }} ``` ## README Template {#readme-template} The hydration README is generated using a [Go text/template](https://pkg.go.dev/text/template), optionally configured by the user via the `argocd-cm` ConfigMap. The template is rendered using the values from `hydrator.metadata` and can be multi-line to define structured documentation. This allows users to customize how the hydration process and references are documented. To define the README template, set the `sourceHydrator.readmeMessageTemplate` field in the `argocd-cm` ConfigMap. The template may also use functions from the [Sprig function library](https://github.com/Masterminds/sprig). ```yaml apiVersion: v1 kind: ConfigMap metadata: name: argocd-cm namespace: argocd data: sourceHydrator.readmeMessageTemplate: | # Manifest Hydration To hydrate the manifests in this repository, run the following commands: ```shell git clone {{ .RepoURL }} git checkout {{ .DrySHA }} {{ range $command := .Commands }} {{ $command }} {{ end }} ``` {{ if .References }} ## References {{ range $ref := .References }} {{ if $ref.Commit }} * [{{ $ref.Commit.SHA | trunc 7 }}]({{ $ref.Commit.RepoURL }}): {{ $ref.Commit.Subject }} ({{ $ref.Commit.Author }}) {{ end }} {{ end }} {{ end }} ``` ## Commit Author Configuration {#commit-author-configuration} You can customize the git commit author name and email used by the source hydrator when committing hydrated manifests. This is configured via the `argocd-cm` ConfigMap. ```yaml apiVersion: v1 kind: ConfigMap metadata: name: argocd-cm namespace: argocd data: commit.author.name: "GitOps Bot" commit.author.email: "gitops@company.com" ``` **Configuration Keys:** * `commit.author.name`: The git commit author name (defaults to `"Argo CD"` if not set) * `commit.author.email`: The git commit author email (defaults to `"argo-cd@example.com"` if not set) Both values are optional. If only one is configured, the configured value will be used and the other will use its default. ### Credential Templates {#credential-templates} Credential templates allow a single credential to be used for multiple repositories. The source hydrator supports credential templates. For example, if you setup credential templates for the URL prefix `https://github.com/argoproj`, these credentials will be used for all repositories with this URL as prefix (e.g. `https://github.com/argoproj/argocd-example-apps`) that do not have their own credentials configured. For more information, please refer to [Credential templates](https://sourcey-argocd.pages.dev/user-guide/private-repositories/index.md#credential-templates). An example of repo-write-creds secret. ```yaml apiVersion: v1 kind: Secret metadata: name: private-repo namespace: argocd labels: argocd.argoproj.io/secret-type: repo-write-creds stringData: type: git url: https://github.com/argoproj password: my-password username: my-username ``` ## Git Note-based Hydration State Tracking {#git-note-based-hydration-state-tracking} The Source Hydrator does not create a new hydrated commit for a DRY commit if the commit doesn't affect the hydrated manifests. Instead, the hydration state (the DRY SHA last hydrated) is tracked using a [git note](https://git-scm.com/docs/git-notes) in a dedicated `source-hydrator` namespace. On each run, the hydrator: * Checks the git note for the last hydrated DRY SHA. * If manifests have not changed since that SHA, only updates the note. * If manifests have changed, commits the new manifests and updates the note as well. This improves efficiency and reduces commit noise in your repository. If external tooling reads git notes for promotion or coordination (for example, to gate on which dry SHAs have been processed), be aware that `manifest-generate-paths` can prevent note advancement when hydration is skipped. See [Git note attestation and manifest-generate-paths](#git-note-attestation-and-manifest-generate-paths). ## Hydration failures and retries {#hydration-failures-and-retries} When hydration fails, the application remains in the `Failed` phase and the error message is kept on `status.sourceHydrator.currentOperation` so you can diagnose the problem. > [!NOTE] > After a failed hydration, the controller waits about **2 minutes** before automatically retrying > unattended hydration. That interval matches `ARGOCD_RECONCILIATION_TIMEOUT` (the application > controller `--app-resync` period, default **120s**). > > You do not need to wait for that cooldown when: > > - You trigger a **manual or API refresh** (or a **dry-source webhook**), which retries hydration > immediately. > - A **new dry commit** is detected while a dry revision baseline is already recorded in > `status.sourceHydrator.lastComparedDryRevision` (for example, hydration failed at the commit step > after manifests were resolved). > > If the first hydration attempt fails before any dry revision is recorded, automatic detection of a > new commit during the cooldown may be delayed until the cooldown expires or you refresh manually. ## Forcing Hydration with the `hydrate` Annotation {#forcing-hydration-with-the-hydrate-annotation} Argo CD uses the `argocd.argoproj.io/hydrate` annotation to request that an Application's dry source be checked and hydrated (if necessary) outside of the normal commit-detection flow. The application controller consumes and removes the annotation as soon as it has processed it, so it will not normally persist on the Application. The annotation takes one of two values: | Value | Effect | |----------|---------------------------------------------------------------------------------------------------------------------------| | `normal` | Forces a check of whether the dry source has changed since the last hydration. Hydration only runs if a change is found. | | `hard` | Forces hydration unconditionally, even if the dry source revision hasn't changed since the last hydration. | > [!NOTE] > Only the exact value `hard` is special-cased. Any other value is treated as `normal`. Whenever Argo CD sets this annotation itself (see the triggers below), it also sets the companion `argocd.argoproj.io/hydrate-timestamp` annotation to a unique timestamp identifying that specific hydration request. When the controller finishes hydrating, it removes both annotations together, but only if `hydrate-timestamp` still has the value it had when hydration started. If a new hydration request arrived while the previous one was being processed, `hydrate-timestamp` will have a newer value, removal is skipped, and the request is processed instead of being silently dropped. Argo CD sets these annotations automatically in a few situations: * **Manual or API refresh.** `argocd app get --refresh` (and the equivalent API call) sets it to `normal`; `argocd app get --hard-refresh` sets it to `hard`. * **Dry-source webhooks.** A webhook event affecting the `drySource` repository sets it to `normal`, so the dry source is checked right away instead of waiting for the next periodic reconciliation. You can also set the annotation yourself to force a re-hydration, for example to retry sooner than the [failure cooldown](#hydration-failures-and-retries) or after a change that the hydrator wouldn't otherwise detect (such as a change to a [Config Management Plugin](https://sourcey-argocd.pages.dev/operator-manual/config-management-plugins/index.md) that doesn't touch the dry source repository): ```shell kubectl patch application my-app -n argocd --type merge \ -p '{"metadata": {"annotations": {"argocd.argoproj.io/hydrate": "hard"}}}' ``` > [!NOTE] > The annotation only has an effect on Applications with `spec.sourceHydrator` configured, it is ignored otherwise. ## Manifest Generate Paths {#manifest-generate-paths} The source hydrator honors the [`manifest-generate-paths` annotation](https://sourcey-argocd.pages.dev/operator-manual/high_availability/index.md#manifest-paths-annotation) to avoid unnecessary hydration. When the annotation is set, a new dry-source commit that does not touch the annotated paths (resolved relative to the `drySource` path) does not trigger re-hydration. The annotation controls **whether hydration is triggered**, not merely whether hydrated manifests would change. When hydration is skipped, the hydrator does not run and the `hydrator.metadata` git note on the hydrated branch is not advanced. See [Git note attestation and manifest-generate-paths](#git-note-attestation-and-manifest-generate-paths) for implications when external tooling reads git notes. This applies to both webhook-driven and periodically-reconciled refreshes: * A webhook for the **dry source** triggers hydration only when the changed files match the annotated paths. * A webhook for the **sync source** (for example, when an external promotion process merges the `hydrateTo` branch into the `syncSource` branch) triggers a normal refresh and sync of the hydrated manifests. The annotation's path filtering applies to the dry source. The sync source is always watched at its configured `syncSource.path`; the annotation value is not applied to the sync source. ## Limitations {#limitations} ### Repository support {#repository-support} The source hydrator currently supports only Git repositories. `drySource.repoURL` must reference a Git repository. The effective sync repository must also be Git: `syncSource.repoURL` may reference a different Git repository or be omitted to inherit `drySource.repoURL`. OCI repositories (`oci://...`) are supported as [regular Application sources](https://sourcey-argocd.pages.dev/user-guide/oci/index.md), but cannot currently be used as a `drySource` or hydration destination. An OCI dry source may be accepted and resolved to a digest, but hydration will fail when Argo CD attempts to retrieve Git revision metadata. ### Git note attestation and manifest-generate-paths {#git-note-attestation-and-manifest-generate-paths} The hydrator records which dry SHAs it has processed in a git note (`refs/notes/hydrator.metadata`) on the hydrated branch. The note is written by commit-server when hydration runs — including the case where manifests are unchanged and no new hydrated commit is created. When `manifest-generate-paths` causes hydration to be skipped (because changed files fall outside the annotated paths), commit-server is never called and the git note is **not** advanced. Application status may still record that the dry revision was compared, but the note's `drySha` will lag behind. In other words, the note attests only the dry commits for which the hydrator actually ran; it does not advance for dry commits filtered out by `manifest-generate-paths`, even though those commits would produce no manifest change. If you consume `hydrator.metadata` notes, account for this: only dry commits that match an Application's `manifest-generate-paths` (or all dry commits, when the annotation is unset) are attested on its hydrated branch. A planned enhancement ([#28556](https://github.com/argoproj/argo-cd/issues/28556)) will add a lightweight commit-server endpoint to advance the git note without a full hydration run (no repo-server render or manifest disk compare). ### Signature Verification {#signature-verification} *Current Status: Alpha (Since v3.5)* The hydrator enforces the project's `SourceIntegrity` policy (e.g. GPG signature verification) on the DRY revision before producing manifests. If the project requires verification and the DRY commit fails it (or was not verified), hydration is rejected. Verification is opted into per project by configuring `.spec.sourceIntegrity` on the `AppProject`. See [Source Integrity Verification](https://sourcey-argocd.pages.dev/user-guide/source-integrity/index.md) for how to set it up (for example, using [Git GnuPG verification](https://sourcey-argocd.pages.dev/user-guide/source-integrity-git-gpg/index.md)). By default the hydrator **does not** sign the commits it pushes to git, so if signature verification is enabled for the hydrated branch those commits will fail verification when Argo CD attempts to sync the hydrated manifests. See [Signing Hydrated Commits](#signing-hydrated-commits) below for how to opt in. ### Signing Hydrated Commits {#signing-hydrated-commits} *Current Status: Alpha (Since v3.6)* The commit server can optionally GPG-sign every hydrated commit before pushing it. Signing is enabled simply by configuring a signing key path — there is no separate on/off toggle. When a key is configured, the commit server fails fast at startup if the signing key is missing or invalid, and refuses to push any commit it cannot locally verify against the configured key — there is no unsigned fallback. To enable signing: 1. Create a Kubernetes Secret in the Argo CD namespace named `argocd-commit-server-gpg-signing-key` containing the ASCII-armored private key, and (optionally) the passphrase: ```yaml apiVersion: v1 kind: Secret metadata: name: argocd-commit-server-gpg-signing-key namespace: argocd type: Opaque stringData: signingKey: | -----BEGIN PGP PRIVATE KEY BLOCK----- ... -----END PGP PRIVATE KEY BLOCK----- # Optional. Omit entirely if the key has no passphrase. passphrase: "s3cret" ``` 2. Set the relevant params in `argocd-cmd-params-cm`. Setting the key path is what turns signing on: ```yaml # Empty by default (signing off). Point this at the mounted Secret entry to enable signing. commitserver.signing.key.path: "/app/config/gpg/signing/signingKey" # Empty by default. Set this ONLY for a passphrase-protected key, pointing at the # mounted `passphrase` Secret entry. Leave unset for an unprotected key. commitserver.signing.key.passphrase.file: "/app/config/gpg/signing/passphrase" ``` 3. Add the corresponding **public** key to Argo CD's trusted GPG keys (via the UI, CLI, or the `argocd-gpg-keys-cm` ConfigMap) and configure your project's `SourceIntegrity` policy so the hydrated branch is verified at sync time. Without this step Argo CD will accept the signed commit but won't actually enforce that it was signed. Declaratively, add the public key to `argocd-gpg-keys-cm`, keyed by the signing key's ID with the ASCII-armored public key as the value: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: argocd-gpg-keys-cm namespace: argocd labels: app.kubernetes.io/name: argocd-gpg-keys-cm app.kubernetes.io/part-of: argocd data: # The signing key's ID, with its ASCII-armored public key as the value. D56C4FCA57A46444: | -----BEGIN PGP PUBLIC KEY BLOCK----- ... -----END PGP PUBLIC KEY BLOCK----- ``` Then reference that key ID in the project's `SourceIntegrity` policy so the hydrated (`hydrateTo`) branch is verified at sync time: ```yaml apiVersion: argoproj.io/v1alpha1 kind: AppProject metadata: name: my-project namespace: argocd spec: sourceIntegrity: git: policies: - repos: - url: "https://github.com/my-org/my-hydrated-repo.git" gpg: # "strict" verifies the entire branch history; because the commit server signs every hydrated # commit, this holds for a branch signed from the start. Use "head" to verify only the tip commit # instead — useful when the branch already had unsigned commits before signing was enabled. mode: "strict" keys: - "D56C4FCA57A46444" ``` See [Git GnuPG verification](https://sourcey-argocd.pages.dev/user-guide/source-integrity-git-gpg/index.md) for the full policy reference. > [!NOTE] > The configured key is the commit server's single, cluster-wide **default** signing key. Finer-grained signing > keys (for example per-AppProject or per-repository) are not supported yet, but are a planned additive extension — > the existing `commitserver.signing.key.*` params will remain the default and won't be renamed. > [!NOTE] > If your signing key has a dedicated **signing subkey** (common when the primary key is kept certify-only, e.g. a > key extended with `gpg --quick-add-key - sign`), git signs commits with that subkey rather than the primary > key. You only need to list the **primary** key fingerprint in your `SourceIntegrity` policy — verification > resolves the signing subkey back to its primary key, so a subkey signature is accepted as long as the primary is > trusted. The public key you import via `argocd-gpg-keys-cm` already carries its subkeys, so no extra configuration > is required. > [!WARNING] > If you use a tool that auto-merges the hydrated `hydrateTo` branch into the `syncSource` branch — for example > [GitOps Promoter](https://github.com/argoproj-labs/gitops-promoter) or a GitHub "merge pull request" automation — > the live branch HEAD becomes a **merge commit signed by the merger**, not the commit server's signed commit. > Argo CD verifies that merge commit when it picks up the change on the live branch, so verification will fail > unless the merger's public key is also trusted. Add the merger's public key (e.g. GitHub's web-flow key from > `https://github.com/web-flow.gpg`) to `argocd-gpg-keys-cm` and your project's `SourceIntegrity` policy alongside > the commit server's signing key. ### Project-Scoped Push Secrets {#project-scoped-push-secrets} If all the Applications for a given destination repo/branch are under the same project, then the hydrator will use any available project-scoped push secrets. If two Applications for a given repo/branch are in different projects, then the hydrator will not be able to use a project-scoped push secret and will require a global push secret. ### Multiple Argo CD Instances Hydrating to One Branch {#multiple-argo-cd-instances-hydrating-to-one-branch} A hydrated repository and branch should be owned by a single Argo CD instance. Two separate Argo CD instances that share the same `drySource` and write hydrated manifests to the **same** `syncSource` repository and branch are not supported: the hydrator records hydration state in a per-dry-commit git note on the destination branch, so once one instance records a commit, the other instance will short-circuit and leave its manifests stale. Hydrate each instance to its own branch instead. ## Prerequisites {#prerequisites} ### Handle Secrets on the Destination Cluster {#handle-secrets-on-the-destination-cluster} Do not use the source hydrator with any tool that injects secrets into your manifests as part of the hydration process (for example, Helm with SOPS or the Argo CD Vault Plugin). These secrets would be committed to git. Instead, use a secrets operator that populates the secret values on the destination cluster. ## Best Practices {#best-practices} ### Make Hydration Deterministic {#make-hydration-deterministic} The source hydrator should be deterministic. For a given dry source commit, the hydrator should always produce the same hydrated manifests. This means that the hydrator should not rely on an external state or configuration that is not stored in Git. Examples of non-deterministic hydration: * A Helm chart using unpinned dependencies * A Helm chart is using a non-deterministic template function such as `randAlphaNum` or `lookup` * [Config Management Plugins](https://sourcey-argocd.pages.dev/operator-manual/config-management-plugins/index.md) which retrieve non-git state, such as secrets * Kustomize manifests referencing unpinned remote bases ### Enable Branch Protection {#enable-branch-protection} Argo CD should be the only thing pushing hydrated manifests to the hydrated branches. To prevent other tools or users from pushing to the hydrated branches, enable branch protection in your SCM. It is best practice to prefix the hydrated branches with a common prefix, such as `environments/`. This makes it easier to configure branch protection rules on the destination repository. > [!NOTE] > To maintain reproducibility and determinism in the Hydrator’s output, > Argo CD-specific metadata (such as `argocd.argoproj.io/tracking-id`) is > not written to Git during hydration. These annotations are added dynamically > during application sync and comparison. ### Application Path Cleaning Behavior {#application-path-cleaning-behavior} The Source Hydrator does not clean (remove) files from the application's configured output path before writing new manifests. This means that any files previously generated by hydration (or otherwise present) that are not overwritten by the new hydration run will remain in the output directory.