# v3.5 to 3.6 {#v35-to-36} ## Breaking Changes {#breaking-changes} ### Repo-server OTLP headers environment variable renamed {#repo-server-otlp-headers-environment-variable-renamed} The repo-server now reads OpenTelemetry collector headers from `ARGOCD_REPO_SERVER_OTLP_HEADERS` instead of the inconsistently-named `ARGOCD_REPO_OTLP_HEADERS`. The new name follows the `ARGOCD__OTLP_HEADERS` convention used by every other Argo CD component (e.g. `ARGOCD_SERVER_OTLP_HEADERS`, `ARGOCD_APPLICATION_CONTROLLER_OTLP_HEADERS`) and matches the name the shipped repo-server Deployment already injects from the `otlp.headers` config map key. **Impact:** - The default manifests already set `ARGOCD_REPO_SERVER_OTLP_HEADERS`, so most operators are unaffected. - If you set `ARGOCD_REPO_OTLP_HEADERS` directly (for example via a custom Deployment patch or Helm values), rename it to `ARGOCD_REPO_SERVER_OTLP_HEADERS`. The old variable is no longer read. ### Default local include paths now include Helm template helpers and Chart.lock {#default-local-include-paths-now-include-helm-template-helpers-and-chartlock} When using `argocd app diff --local --server-side-generate`, the implicit default for `--local-include` has changed from `["*.yaml", "*.yml", "*.json"]` to `["*.yaml", "*.yml", "*.json", "*.tpl", "Chart.lock"]`. The two new entries cover common Helm files that the old defaults silently excluded: - `*.tpl` — matches any Helm template helper file (e.g. `templates/_helpers.tpl`, `charts/subchart/templates/_helpers.tpl`) at any depth. Without this, `helm template` on the server fails or renders incorrectly when templates call named helpers defined in `_helpers.tpl`. - `Chart.lock` — matches Helm dependency lock files at any depth. Without this, the server resolves dependencies from `Chart.yaml` version constraints instead of the pinned versions in `Chart.lock`, which can produce diffs against the wrong dependency versions. Patterns without a path separator (e.g. `*.yaml`, `*.tpl`) match on the filename alone regardless of directory depth. Patterns containing `/` match against the file's relative path and support `**` to span multiple directories. > [!WARNING] > Specifying `--local-include` on the command line **replaces** the default set entirely — it does not append to it. If you need to add patterns beyond the defaults, you must re-specify the full set you want, including any defaults you wish to keep. > > For example, to also include all files under `charts/`: > > ```bash > argocd app diff my-app \ > --local ./path \ > --server-side-generate \ > --local-include "*.yaml" --local-include "*.yml" --local-include "*.json" \ > --local-include "*.tpl" --local-include "Chart.lock" \ > --local-include "charts/**" > ``` > [!NOTE] > Kustomize apps that reference non-YAML source files in `configMapGenerator` or `secretGenerator` (e.g. `*.env`, `*.properties`) must add those patterns explicitly via `--local-include`, as no generic default can cover all possible file names and extensions. #### Impact {#impact} - If your Helm chart uses `_helpers.tpl` or a `Chart.lock`, server-side diff now works correctly out of the box without extra flags. - If you previously passed a custom `--local-include` list that omitted `*.tpl` or `Chart.lock`, consider whether to add them. #### Mitigations and examples {#mitigations-and-examples} - Restore pre-3.6 behavior (YAML/JSON filename matching only): ```bash argocd app diff my-app \ --local ./path \ --server-side-generate \ --local-include "*.yaml" --local-include "*.yml" --local-include "*.json" ``` - Include ALL files under `charts/` (use when `charts/` contains non-YAML content you need to transmit): ```bash argocd app diff my-app \ --local ./path \ --server-side-generate \ --local-include "*.yaml" --local-include "*.yml" --local-include "*.json" \ --local-include "*.tpl" --local-include "Chart.lock" \ --local-include "charts/**" ``` - Include source files for Kustomize `configMapGenerator` (e.g. `.env` files): ```bash argocd app diff my-app \ --local ./path \ --server-side-generate \ --local-include "*.yaml" --local-include "*.yml" --local-include "*.json" \ --local-include "*.tpl" --local-include "Chart.lock" \ --local-include "*.env" ``` ## Behavioral Improvements / Fixes {#behavioral-improvements--fixes} ### Server-Side Apply now uses Server-Side Diff {#server-side-apply-now-uses-server-side-diff} Applications that enable the `ServerSideApply=true` sync option now calculate diffs using [Server-Side Diff](https://sourcey-argocd.pages.dev/user-guide/diff-strategies/index.md#server-side-diff), which performs a Server-Side Apply dry-run against the kube-apiserver. This replaces the Structured-Merge Diff strategy, which approximated the result locally and has been removed. Server-Side Diff is now used when any of the following is true: - the `controller.diff.server.side` controller flag is enabled, - the `argocd.argoproj.io/compare-options: ServerSideDiff=true` annotation is set, or - the `ServerSideApply=true` sync option is enabled (**new in 3.6**). An explicit `argocd.argoproj.io/compare-options: ServerSideDiff=false` annotation always takes precedence and falls back to the Legacy client-side diff. **Impact:** - Expect additional kube-apiserver load from the dry-run requests, proportional to the number of resources in Applications that use `ServerSideApply=true`. Diff results are cached, so requests are only made on refresh, on a new revision, on an Application spec change, or when a resource's `resourceVersion` changes. - Validating webhooks now participate in the diff stage, so a rejected resource surfaces as a diff error rather than only failing at sync time. No action is required. To keep the Legacy client-side diff for an Application that uses `ServerSideApply=true`, add the opt-out annotation: ```yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: annotations: argocd.argoproj.io/compare-options: ServerSideDiff=false ``` ### GitOps Promoter `PromotionStrategy` and `ChangeTransferPolicy` health status {#gitops-promoter-promotionstrategy-and-changetransferpolicy-health-status} Argo CD's built-in health checks for `promoter.argoproj.io/PromotionStrategy` and `promoter.argoproj.io/ChangeTransferPolicy` now report `Healthy` instead of `Progressing` when a promotion is in progress. **New behavior:** - `Healthy` is returned when a promotion is in progress (proposed commit differs from the active commit), including while commit statuses are pending or failed. - `Progressing` is still returned while waiting for hydration to complete, such as when dry SHAs are not yet populated or environments are temporarily out of sync on the proposed commit. - `Progressing` is also returned for controller-level waits, such as when the resource has no status yet, is not ready, is being deleted, or the spec generation has not been observed. - `Degraded` is unchanged for reconciliation errors and other failure conditions. Health messages are unchanged; only the reported status differs for active promotions. **Impact:** - Applications that include GitOps Promoter resources may show `Healthy` instead of `Progressing` after upgrading while a promotion is underway. - Alerting or automation that treats `Progressing` on these resources as a signal of an active promotion should use the health message or Promoter status fields instead. No action is required for users who do not use GitOps Promoter resources. ### Health transition events now identify the causing resource(s) {#health-transition-events-now-identify-the-causing-resources} The `Updated health status` event emitted when an Application's aggregated health changes now includes the resource(s) that caused the new status. **New behavior:** The event message gains a `(Caused by ...)` suffix listing the offending resource(s), for example: ``` Updated health status: Healthy -> Degraded (Caused by apps/Deployment:default/api) ``` Up to three resources are listed. This makes it easier to investigate flapping application health by pointing directly at the resource(s) driving the change. **Impact:** - If you consume this event message and parse it programmatically, update your parser to tolerate the new `(Caused by ...)` suffix. ### Progressing through steps now relies on annotation in Progressive Sync {#progressing-through-steps-now-relies-on-annotation-in-progressive-sync} With progressive sync enabled, when appset controller identifies a change in any of its owned applications, it adds refresh annotation to all owned applications and waits for all applications to complete reconciling before beginning progressive sync. This ensures that before starting progressive sync, all applications have been reconciled, and not operating on stale data of applications to make progressing decisions. **Impact:** If you're using ApplicationSets with progressive sync, and are managing many applications, you might see some delays in applications progressing through steps. ### Tracing sampler is now parent-based and configurable {#tracing-sampler-is-now-parent-based-and-configurable} Previously, when OpenTelemetry tracing was enabled (`--otlp-address` set), Argo CD sampled **every** span unconditionally (`AlwaysSample`). Argo CD now uses a parent-based head sampler whose root sampling rate is configurable per component via the `--otlp-sample-ratio` flag (or the `ARGOCD__OTLP_SAMPLE_RATIO` environment variable). The flag defaults to `1.0` (sample every trace), so for most deployments behaviour is unchanged. There are two behavioural differences to be aware of: > [!NOTE] > **Sampling is now propagated across services.** A sampling decision made by the > service that starts a trace (for example the application controller) is honored by > every downstream service the trace context flows into (repo-server, commit-server). > This keeps each trace whole instead of being partially sampled across process > boundaries. Set the same `--otlp-sample-ratio` on all components to get a consistent > rate. > [!WARNING] > **An inbound "not sampled" decision is now honored.** Because the sampler is > parent-based, a request that already carries a W3C `traceparent` marked *not sampled* > is no longer recorded, even with `--otlp-sample-ratio=1.0`. Under the old > `AlwaysSample` behaviour such requests were recorded regardless. This only affects > deployments where an upstream proxy, gateway, or client injects trace context into > requests to `argocd-server`; if nothing upstream propagates `traceparent`, every > trace is still rooted in Argo CD and sampled at the configured ratio. Values supplied to `--otlp-sample-ratio` outside the `[0.0, 1.0]` range (or `NaN`) are **rejected at startup** with an error, so a misconfigured flag fails fast rather than silently disabling or exceeding sampling. When the ratio is set via the `ARGOCD__OTLP_SAMPLE_RATIO` environment variable instead, an invalid or out-of-range value is ignored and the default (`1.0`) is used (a warning is logged). ### Open Telemetry Upgraded {#open-telemetry-upgraded} Argo CD v3.6 upgrades the `otelgrpc` and `otelhttp` tracing instrumentation to `v0.70.0`, which bundles semantic conventions [v1.42.0](https://github.com/open-telemetry/semantic-conventions/releases/tag/v1.42.0) and [v1.43.0](https://github.com/open-telemetry/semantic-conventions/releases/tag/v1.43.0). Please see those release notes for a list of changed/dropped span attribute names, which may require changes to Grafana dashboards/alerts based on Argo CD OTEL traces. One specific change to be aware of: `otelgrpc` now sets the `server.address`/`server.port` span attributes from the gRPC dial target rather than the resolved peer IP. If a dashboard or alert keys on `server.address` expecting a literal pod/service IP for internal Argo CD gRPC calls (`argocd-server` ↔ `repo-server` ↔ `cmp-server`), it may need updating. ### Custom Healthchecks Added {#custom-healthchecks-added} - A bundled health check for CloudNativePG's `postgresql.cnpg.io/Database` resource. It reports `Healthy` only once the operator has applied the current generation (`status.applied` with a matching `status.observedGeneration`), so a sync wave acts as a real dependency barrier for it instead of treating the not-yet-applied Database as healthy. - A bundled health check for Argo CD Image Updater's `argocd-image-updater.argoproj.io/ImageUpdater` resource. It derives health from the standard Kubernetes conditions set by the operator: `Healthy` when the `Ready` condition is `True`, `Degraded` when the `Error` condition is `True` or `Ready` is `False`, and `Progressing` while the controller is actively reconciling or has not yet observed the current spec generation. ### Health Checks During Deletion {#health-checks-during-deletion} In the past, Argo CD always reported "Progressing" / "Pending deletion" for resources being deleted (i.e. `metadata.deletionTimestamp` set). Argo CD 3.6 brings two improvements: 1) The default health check lists any pending finalizers 2) Health checks (built in or custom) may return an optional `deletionMessage` value - if set, the default behavior is overridden This new field allows health checks to report more detailed information about the resource being deleted. The main use case is adding helpful context about pending finalizers. If a resource is pending deletion for a long time, the health message can explain to the user the purpose of the pending finalizers and warn about any risks of manually removing them. See the [health check documentation](https://sourcey-argocd.pages.dev/operator-manual/health/index.md#terminating-resources) for more information about how to use the feature. ## API Changes {#api-changes} ### Cluster `configHash` field {#cluster-confighash-field} A new ephemeral `configHash` field has been added to the `Cluster` object to help reliably detect configuration changes. The field is not present in the underlying cluster secret and is dynamically computed during queries. This hash should be treated as an opaque value and no assumptions should be made about the mechanism of its generation as it is not a part of the API contract and may change in the future. ### Gitops Engine Changes {#gitops-engine-changes} The `pkg/utils/kube/KubectlOptionsRunner.Create()` function now ignores its third argument (cmd *cobra.Command) because the underlying function `k8s.io/kubectl/pkg/cmd/create.CreateOptions.RunCreate()`in kubernetes client has changed, it does not accept this parameter since v0.37.0. This argument will be removed in the next major Argo CD release.