v3.5 to 3.6
Breaking Changes
Repo-server OTLP headers environment variable renamed
The repo-server now reads OpenTelemetry collector headers from ARGOCD_REPO_SERVER_OTLP_HEADERS
instead of the inconsistently-named ARGOCD_REPO_OTLP_HEADERS. The new name follows the
ARGOCD_<COMPONENT>_OTLP_HEADERS convention used by every other Argo CD component (e.g.
ARGOCD_SERVER_OTLP_HEADERS, ARGOCD_APPLICATION_CONTROLLER_OTLP_HEADERS) and matches the name the
shipped repo-server Deployment already injects from the otlp.headers config map key.
Impact:
- The default manifests already set
ARGOCD_REPO_SERVER_OTLP_HEADERS, so most operators are unaffected. - If you set
ARGOCD_REPO_OTLP_HEADERSdirectly (for example via a custom Deployment patch or Helm values), rename it toARGOCD_REPO_SERVER_OTLP_HEADERS. The old variable is no longer read.
Default local include paths now include Helm template helpers and Chart.lock
When using argocd app diff --local --server-side-generate, the implicit default for --local-include has changed from ["*.yaml", "*.yml", "*.json"] to ["*.yaml", "*.yml", "*.json", "*.tpl", "Chart.lock"].
The two new entries cover common Helm files that the old defaults silently excluded:
*.tpl— matches any Helm template helper file (e.g.templates/_helpers.tpl,charts/subchart/templates/_helpers.tpl) at any depth. Without this,helm templateon the server fails or renders incorrectly when templates call named helpers defined in_helpers.tpl.Chart.lock— matches Helm dependency lock files at any depth. Without this, the server resolves dependencies fromChart.yamlversion constraints instead of the pinned versions inChart.lock, which can produce diffs against the wrong dependency versions.
Patterns without a path separator (e.g. *.yaml, *.tpl) match on the filename alone regardless of directory depth. Patterns containing / match against the file's relative path and support ** to span multiple directories.
Specifying --local-include on the command line replaces the default set entirely — it does not append to it. If you need to add patterns beyond the defaults, you must re-specify the full set you want, including any defaults you wish to keep.
For example, to also include all files under charts/:
argocd app diff my-app \
--local ./path \
--server-side-generate \
--local-include "*.yaml" --local-include "*.yml" --local-include "*.json" \
--local-include "*.tpl" --local-include "Chart.lock" \
--local-include "charts/**"Kustomize apps that reference non-YAML source files in configMapGenerator or secretGenerator (e.g. *.env, *.properties) must add those patterns explicitly via --local-include, as no generic default can cover all possible file names and extensions.
Impact
- If your Helm chart uses
_helpers.tplor aChart.lock, server-side diff now works correctly out of the box without extra flags. - If you previously passed a custom
--local-includelist that omitted*.tplorChart.lock, consider whether to add them.
Mitigations and examples
Restore pre-3.6 behavior (YAML/JSON filename matching only):
argocd app diff my-app \ --local ./path \ --server-side-generate \ --local-include "*.yaml" --local-include "*.yml" --local-include "*.json"Include ALL files under
charts/(use whencharts/contains non-YAML content you need to transmit):argocd app diff my-app \ --local ./path \ --server-side-generate \ --local-include "*.yaml" --local-include "*.yml" --local-include "*.json" \ --local-include "*.tpl" --local-include "Chart.lock" \ --local-include "charts/**"Include source files for Kustomize
configMapGenerator(e.g..envfiles):argocd app diff my-app \ --local ./path \ --server-side-generate \ --local-include "*.yaml" --local-include "*.yml" --local-include "*.json" \ --local-include "*.tpl" --local-include "Chart.lock" \ --local-include "*.env"
Behavioral Improvements / Fixes
Server-Side Apply now uses Server-Side Diff
Applications that enable the ServerSideApply=true sync option now calculate diffs using
Server-Side Diff, which performs a
Server-Side Apply dry-run against the kube-apiserver. This replaces the Structured-Merge Diff
strategy, which approximated the result locally and has been removed.
Server-Side Diff is now used when any of the following is true:
- the
controller.diff.server.sidecontroller flag is enabled, - the
argocd.argoproj.io/compare-options: ServerSideDiff=trueannotation is set, or - the
ServerSideApply=truesync option is enabled (new in 3.6).
An explicit argocd.argoproj.io/compare-options: ServerSideDiff=false annotation always takes
precedence and falls back to the Legacy client-side diff.
Impact:
- Expect additional kube-apiserver load from the dry-run requests, proportional to the number of
resources in Applications that use
ServerSideApply=true. Diff results are cached, so requests are only made on refresh, on a new revision, on an Application spec change, or when a resource'sresourceVersionchanges. - Validating webhooks now participate in the diff stage, so a rejected resource surfaces as a diff error rather than only failing at sync time.
No action is required. To keep the Legacy client-side diff for an Application that uses
ServerSideApply=true, add the opt-out annotation:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
annotations:
argocd.argoproj.io/compare-options: ServerSideDiff=falseGitOps Promoter PromotionStrategy and ChangeTransferPolicy health status
Argo CD's built-in health checks for promoter.argoproj.io/PromotionStrategy and
promoter.argoproj.io/ChangeTransferPolicy now report Healthy instead of
Progressing when a promotion is in progress.
New behavior:
Healthyis returned when a promotion is in progress (proposed commit differs from the active commit), including while commit statuses are pending or failed.Progressingis still returned while waiting for hydration to complete, such as when dry SHAs are not yet populated or environments are temporarily out of sync on the proposed commit.Progressingis also returned for controller-level waits, such as when the resource has no status yet, is not ready, is being deleted, or the spec generation has not been observed.Degradedis unchanged for reconciliation errors and other failure conditions.
Health messages are unchanged; only the reported status differs for active promotions.
Impact:
- Applications that include GitOps Promoter resources may show
Healthyinstead ofProgressingafter upgrading while a promotion is underway. - Alerting or automation that treats
Progressingon these resources as a signal of an active promotion should use the health message or Promoter status fields instead.
No action is required for users who do not use GitOps Promoter resources.
Health transition events now identify the causing resource(s)
The Updated health status event emitted when an Application's aggregated health changes now includes the resource(s) that caused the new status.
New behavior:
The event message gains a (Caused by ...) suffix listing the offending resource(s), for example:
Updated health status: Healthy -> Degraded (Caused by apps/Deployment:default/api)Up to three resources are listed. This makes it easier to investigate flapping application health by pointing directly at the resource(s) driving the change.
Impact:
- If you consume this event message and parse it programmatically, update your parser to tolerate the new
(Caused by ...)suffix.
Progressing through steps now relies on annotation in Progressive Sync
With progressive sync enabled, when appset controller identifies a change in any of its owned applications, it adds refresh annotation to all owned applications and waits for all applications to complete reconciling before beginning progressive sync. This ensures that before starting progressive sync, all applications have been reconciled, and not operating on stale data of applications to make progressing decisions.
Impact: If you're using ApplicationSets with progressive sync, and are managing many applications, you might see some delays in applications progressing through steps.
Tracing sampler is now parent-based and configurable
Previously, when OpenTelemetry tracing was enabled (--otlp-address set), Argo CD
sampled every span unconditionally (AlwaysSample). Argo CD now uses a
parent-based head sampler whose root sampling rate is configurable per component via
the --otlp-sample-ratio flag (or the ARGOCD_<COMPONENT>_OTLP_SAMPLE_RATIO
environment variable). The flag defaults to 1.0 (sample every trace), so for most
deployments behaviour is unchanged.
There are two behavioural differences to be aware of:
Sampling is now propagated across services. A sampling decision made by the
service that starts a trace (for example the application controller) is honored by
every downstream service the trace context flows into (repo-server, commit-server).
This keeps each trace whole instead of being partially sampled across process
boundaries. Set the same --otlp-sample-ratio on all components to get a consistent
rate.
An inbound "not sampled" decision is now honored. Because the sampler is
parent-based, a request that already carries a W3C traceparent marked not sampled
is no longer recorded, even with --otlp-sample-ratio=1.0. Under the old
AlwaysSample behaviour such requests were recorded regardless. This only affects
deployments where an upstream proxy, gateway, or client injects trace context into
requests to argocd-server; if nothing upstream propagates traceparent, every
trace is still rooted in Argo CD and sampled at the configured ratio.
Values supplied to --otlp-sample-ratio outside the [0.0, 1.0] range (or NaN) are
rejected at startup with an error, so a misconfigured flag fails fast rather than
silently disabling or exceeding sampling. When the ratio is set via the
ARGOCD_<COMPONENT>_OTLP_SAMPLE_RATIO environment variable instead, an invalid or
out-of-range value is ignored and the default (1.0) is used (a warning is logged).
Open Telemetry Upgraded
Argo CD v3.6 upgrades the otelgrpc and otelhttp tracing instrumentation to v0.70.0, which
bundles semantic conventions v1.42.0
and v1.43.0. Please
see those release notes for a list of changed/dropped span attribute names, which may require
changes to Grafana dashboards/alerts based on Argo CD OTEL traces.
One specific change to be aware of: otelgrpc now sets the server.address/server.port span
attributes from the gRPC dial target rather than the resolved peer IP. If a dashboard or alert
keys on server.address expecting a literal pod/service IP for internal Argo CD gRPC calls
(argocd-server ↔ repo-server ↔ cmp-server), it may need updating.
Custom Healthchecks Added
A bundled health check for CloudNativePG's
postgresql.cnpg.io/Databaseresource. It reportsHealthyonly once the operator has applied the current generation (status.appliedwith a matchingstatus.observedGeneration), so a sync wave acts as a real dependency barrier for it instead of treating the not-yet-applied Database as healthy.A bundled health check for Argo CD Image Updater's
argocd-image-updater.argoproj.io/ImageUpdaterresource. It derives health from the standard Kubernetes conditions set by the operator:Healthywhen theReadycondition isTrue,Degradedwhen theErrorcondition isTrueorReadyisFalse, andProgressingwhile the controller is actively reconciling or has not yet observed the current spec generation.
Health Checks During Deletion
In the past, Argo CD always reported "Progressing" / "Pending deletion" for resources being deleted
(i.e. metadata.deletionTimestamp set). Argo CD 3.6 brings two improvements:
- The default health check lists any pending finalizers
- Health checks (built in or custom) may return an optional
deletionMessagevalue - if set, the default behavior is overridden
This new field allows health checks to report more detailed information about the resource being deleted. The main use case is adding helpful context about pending finalizers. If a resource is pending deletion for a long time, the health message can explain to the user the purpose of the pending finalizers and warn about any risks of manually removing them.
See the health check documentation for more information about how to use the feature.
API Changes
Cluster configHash field
A new ephemeral configHash field has been added to the Cluster object to help reliably detect configuration changes.
The field is not present in the underlying cluster secret and is dynamically computed during queries.
This hash should be treated as an opaque value and no assumptions should be made about the mechanism of its generation as it is not a part of the API contract and may change in the future.
Gitops Engine Changes
The pkg/utils/kube/KubectlOptionsRunner.Create() function now ignores its third argument (cmd *cobra.Command) because
the underlying function k8s.io/kubectl/pkg/cmd/create.CreateOptions.RunCreate()in kubernetes client has changed, it
does not accept this parameter since v0.37.0. This argument will be removed in the next major Argo CD release.
