Upgrading

v3.5 to 3.6

Breaking Changes

Repo-server OTLP headers environment variable renamed

The repo-server now reads OpenTelemetry collector headers from ARGOCD_REPO_SERVER_OTLP_HEADERS instead of the inconsistently-named ARGOCD_REPO_OTLP_HEADERS. The new name follows the ARGOCD_<COMPONENT>_OTLP_HEADERS convention used by every other Argo CD component (e.g. ARGOCD_SERVER_OTLP_HEADERS, ARGOCD_APPLICATION_CONTROLLER_OTLP_HEADERS) and matches the name the shipped repo-server Deployment already injects from the otlp.headers config map key.

Impact:

  • The default manifests already set ARGOCD_REPO_SERVER_OTLP_HEADERS, so most operators are unaffected.
  • If you set ARGOCD_REPO_OTLP_HEADERS directly (for example via a custom Deployment patch or Helm values), rename it to ARGOCD_REPO_SERVER_OTLP_HEADERS. The old variable is no longer read.

Default local include paths now include Helm template helpers and Chart.lock

When using argocd app diff --local --server-side-generate, the implicit default for --local-include has changed from ["*.yaml", "*.yml", "*.json"] to ["*.yaml", "*.yml", "*.json", "*.tpl", "Chart.lock"].

The two new entries cover common Helm files that the old defaults silently excluded:

  • *.tpl — matches any Helm template helper file (e.g. templates/_helpers.tpl, charts/subchart/templates/_helpers.tpl) at any depth. Without this, helm template on the server fails or renders incorrectly when templates call named helpers defined in _helpers.tpl.
  • Chart.lock — matches Helm dependency lock files at any depth. Without this, the server resolves dependencies from Chart.yaml version constraints instead of the pinned versions in Chart.lock, which can produce diffs against the wrong dependency versions.

Patterns without a path separator (e.g. *.yaml, *.tpl) match on the filename alone regardless of directory depth. Patterns containing / match against the file's relative path and support ** to span multiple directories.

Warning

Specifying --local-include on the command line replaces the default set entirely — it does not append to it. If you need to add patterns beyond the defaults, you must re-specify the full set you want, including any defaults you wish to keep.

For example, to also include all files under charts/:

argocd app diff my-app \
  --local ./path \
  --server-side-generate \
  --local-include "*.yaml" --local-include "*.yml" --local-include "*.json" \
  --local-include "*.tpl" --local-include "Chart.lock" \
  --local-include "charts/**"
Note

Kustomize apps that reference non-YAML source files in configMapGenerator or secretGenerator (e.g. *.env, *.properties) must add those patterns explicitly via --local-include, as no generic default can cover all possible file names and extensions.

Impact

  • If your Helm chart uses _helpers.tpl or a Chart.lock, server-side diff now works correctly out of the box without extra flags.
  • If you previously passed a custom --local-include list that omitted *.tpl or Chart.lock, consider whether to add them.

Mitigations and examples

  • Restore pre-3.6 behavior (YAML/JSON filename matching only):

    argocd app diff my-app \
      --local ./path \
      --server-side-generate \
      --local-include "*.yaml" --local-include "*.yml" --local-include "*.json"
  • Include ALL files under charts/ (use when charts/ contains non-YAML content you need to transmit):

    argocd app diff my-app \
      --local ./path \
      --server-side-generate \
      --local-include "*.yaml" --local-include "*.yml" --local-include "*.json" \
      --local-include "*.tpl" --local-include "Chart.lock" \
      --local-include "charts/**"
  • Include source files for Kustomize configMapGenerator (e.g. .env files):

    argocd app diff my-app \
      --local ./path \
      --server-side-generate \
      --local-include "*.yaml" --local-include "*.yml" --local-include "*.json" \
      --local-include "*.tpl" --local-include "Chart.lock" \
      --local-include "*.env"

Behavioral Improvements / Fixes

Server-Side Apply now uses Server-Side Diff

Applications that enable the ServerSideApply=true sync option now calculate diffs using Server-Side Diff, which performs a Server-Side Apply dry-run against the kube-apiserver. This replaces the Structured-Merge Diff strategy, which approximated the result locally and has been removed.

Server-Side Diff is now used when any of the following is true:

  • the controller.diff.server.side controller flag is enabled,
  • the argocd.argoproj.io/compare-options: ServerSideDiff=true annotation is set, or
  • the ServerSideApply=true sync option is enabled (new in 3.6).

An explicit argocd.argoproj.io/compare-options: ServerSideDiff=false annotation always takes precedence and falls back to the Legacy client-side diff.

Impact:

  • Expect additional kube-apiserver load from the dry-run requests, proportional to the number of resources in Applications that use ServerSideApply=true. Diff results are cached, so requests are only made on refresh, on a new revision, on an Application spec change, or when a resource's resourceVersion changes.
  • Validating webhooks now participate in the diff stage, so a rejected resource surfaces as a diff error rather than only failing at sync time.

No action is required. To keep the Legacy client-side diff for an Application that uses ServerSideApply=true, add the opt-out annotation:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd.argoproj.io/compare-options: ServerSideDiff=false

GitOps Promoter PromotionStrategy and ChangeTransferPolicy health status

Argo CD's built-in health checks for promoter.argoproj.io/PromotionStrategy and promoter.argoproj.io/ChangeTransferPolicy now report Healthy instead of Progressing when a promotion is in progress.

New behavior:

  • Healthy is returned when a promotion is in progress (proposed commit differs from the active commit), including while commit statuses are pending or failed.
  • Progressing is still returned while waiting for hydration to complete, such as when dry SHAs are not yet populated or environments are temporarily out of sync on the proposed commit.
  • Progressing is also returned for controller-level waits, such as when the resource has no status yet, is not ready, is being deleted, or the spec generation has not been observed.
  • Degraded is unchanged for reconciliation errors and other failure conditions.

Health messages are unchanged; only the reported status differs for active promotions.

Impact:

  • Applications that include GitOps Promoter resources may show Healthy instead of Progressing after upgrading while a promotion is underway.
  • Alerting or automation that treats Progressing on these resources as a signal of an active promotion should use the health message or Promoter status fields instead.

No action is required for users who do not use GitOps Promoter resources.

Health transition events now identify the causing resource(s)

The Updated health status event emitted when an Application's aggregated health changes now includes the resource(s) that caused the new status.

New behavior:

The event message gains a (Caused by ...) suffix listing the offending resource(s), for example:

Updated health status: Healthy -> Degraded (Caused by apps/Deployment:default/api)

Up to three resources are listed. This makes it easier to investigate flapping application health by pointing directly at the resource(s) driving the change.

Impact:

  • If you consume this event message and parse it programmatically, update your parser to tolerate the new (Caused by ...) suffix.

Progressing through steps now relies on annotation in Progressive Sync

With progressive sync enabled, when appset controller identifies a change in any of its owned applications, it adds refresh annotation to all owned applications and waits for all applications to complete reconciling before beginning progressive sync. This ensures that before starting progressive sync, all applications have been reconciled, and not operating on stale data of applications to make progressing decisions.

Impact: If you're using ApplicationSets with progressive sync, and are managing many applications, you might see some delays in applications progressing through steps.

Tracing sampler is now parent-based and configurable

Previously, when OpenTelemetry tracing was enabled (--otlp-address set), Argo CD sampled every span unconditionally (AlwaysSample). Argo CD now uses a parent-based head sampler whose root sampling rate is configurable per component via the --otlp-sample-ratio flag (or the ARGOCD_<COMPONENT>_OTLP_SAMPLE_RATIO environment variable). The flag defaults to 1.0 (sample every trace), so for most deployments behaviour is unchanged.

There are two behavioural differences to be aware of:

Note

Sampling is now propagated across services. A sampling decision made by the service that starts a trace (for example the application controller) is honored by every downstream service the trace context flows into (repo-server, commit-server). This keeps each trace whole instead of being partially sampled across process boundaries. Set the same --otlp-sample-ratio on all components to get a consistent rate.

Warning

An inbound "not sampled" decision is now honored. Because the sampler is parent-based, a request that already carries a W3C traceparent marked not sampled is no longer recorded, even with --otlp-sample-ratio=1.0. Under the old AlwaysSample behaviour such requests were recorded regardless. This only affects deployments where an upstream proxy, gateway, or client injects trace context into requests to argocd-server; if nothing upstream propagates traceparent, every trace is still rooted in Argo CD and sampled at the configured ratio.

Values supplied to --otlp-sample-ratio outside the [0.0, 1.0] range (or NaN) are rejected at startup with an error, so a misconfigured flag fails fast rather than silently disabling or exceeding sampling. When the ratio is set via the ARGOCD_<COMPONENT>_OTLP_SAMPLE_RATIO environment variable instead, an invalid or out-of-range value is ignored and the default (1.0) is used (a warning is logged).

Open Telemetry Upgraded

Argo CD v3.6 upgrades the otelgrpc and otelhttp tracing instrumentation to v0.70.0, which bundles semantic conventions v1.42.0 and v1.43.0. Please see those release notes for a list of changed/dropped span attribute names, which may require changes to Grafana dashboards/alerts based on Argo CD OTEL traces.

One specific change to be aware of: otelgrpc now sets the server.address/server.port span attributes from the gRPC dial target rather than the resolved peer IP. If a dashboard or alert keys on server.address expecting a literal pod/service IP for internal Argo CD gRPC calls (argocd-server ↔ repo-server ↔ cmp-server), it may need updating.

Custom Healthchecks Added

  • A bundled health check for CloudNativePG's postgresql.cnpg.io/Database resource. It reports Healthy only once the operator has applied the current generation (status.applied with a matching status.observedGeneration), so a sync wave acts as a real dependency barrier for it instead of treating the not-yet-applied Database as healthy.

  • A bundled health check for Argo CD Image Updater's argocd-image-updater.argoproj.io/ImageUpdater resource. It derives health from the standard Kubernetes conditions set by the operator: Healthy when the Ready condition is True, Degraded when the Error condition is True or Ready is False, and Progressing while the controller is actively reconciling or has not yet observed the current spec generation.

Health Checks During Deletion

In the past, Argo CD always reported "Progressing" / "Pending deletion" for resources being deleted (i.e. metadata.deletionTimestamp set). Argo CD 3.6 brings two improvements:

  1. The default health check lists any pending finalizers
  2. Health checks (built in or custom) may return an optional deletionMessage value - if set, the default behavior is overridden

This new field allows health checks to report more detailed information about the resource being deleted. The main use case is adding helpful context about pending finalizers. If a resource is pending deletion for a long time, the health message can explain to the user the purpose of the pending finalizers and warn about any risks of manually removing them.

See the health check documentation for more information about how to use the feature.

API Changes

Cluster configHash field

A new ephemeral configHash field has been added to the Cluster object to help reliably detect configuration changes.

The field is not present in the underlying cluster secret and is dynamically computed during queries.

This hash should be treated as an opaque value and no assumptions should be made about the mechanism of its generation as it is not a part of the API contract and may change in the future.

Gitops Engine Changes

The pkg/utils/kube/KubectlOptionsRunner.Create() function now ignores its third argument (cmd *cobra.Command) because the underlying function k8s.io/kubectl/pkg/cmd/create.CreateOptions.RunCreate()in kubernetes client has changed, it does not accept this parameter since v0.37.0. This argument will be removed in the next major Argo CD release.