This document describes the observability layer added to vtex.rewriter to monitor the health of the external dependencies used during binding discovery (License Manager and the Catalog sales-channel endpoint), introduced together with the Tenant API → License Manager + Catalog migration.

The goal is to answer, at a glance: are our upstream dependencies healthy, how fast are they, and which path resolved each request's binding?

File	Purpose
node/observability/dependencyHealth.ts	Generic, app-agnostic layer (instrumentDependency).
node/observability/metrics.ts	Metric primitives (recordCounter, emitCall) and metric names.
node/observability/index.ts	Public export surface.
node/utils/bindings/	Wires License Manager + Catalog calls through the layer (dependencies.ts) and emits rewriter_binding_resolution_total (bindings.ts).

The Grafana dashboard and alert rules are not kept in this repository; they are maintained as Grafana-as-code assets outside it. The reference queries, panel layout and alert thresholds below are the source of truth for reproducing them.

The layer only depends on a logger, an account, and the diagnosticsMetrics global, so it can later be extracted into a shared library and reused by other services.

Metrics go through global.diagnosticsMetrics — the @vtex/diagnostics-nodejs / OpenTelemetry pipeline introduced in @vtex/api 7.x — which is what lands in the telemetry.metrics_distributed ClickHouse table. The legacy metrics.batch (MetricsAccumulator) pipeline is not used for these metrics, since it does not reach that table.

Following OpenTelemetry best practices we keep a small, fixed set of metric names and differentiate with low-cardinality attributes instead of encoding values into the metric name. Request-scoped base attributes (account, workspace, production, …) are merged automatically by the runtime, so we don't pass them.

All emission is guarded: if diagnosticsMetrics is unavailable (older runtimes, tests) instrumentation degrades to a no-op and never breaks the wrapped call.

One increment per dependency call.

Attribute	Values
dependency	license-manager, catalog
operation	listBindings, getSalesChannel
outcome	success, error
status_class	2xx, 4xx, 5xx, unknown (also 1xx/3xx)

Latency is recorded into the runtime's shared latency histogram (created by DiagnosticsMetrics), tagged with dependency, operation and status_class. Filter by Attributes['dependency'] to isolate our dependency calls.

Which strategy resolved the request's binding. Attribute strategy: id_match (explicit id), address_match (explicit __bindingAddress), url_match (host + path match), fallback (first storefront binding). A spike in fallback usually means a binding/address mismatch after a config change.

On failure, instrumentDependency logs an error before re-throwing (so the caller keeps control of the failure):

Counters are stored as aggregates; read them with sumMerge(Sum). Example — dependency request volume split by outcome and status class (adapt app, workspace, tenant, and the time bounds to your dashboard):

The rewriter_* metric names are already unique to this app, so the app column filter is intentionally omitted (it can return zero rows if the exact app@version string does not match). The shared histogram io_app_operation_duration_milliseconds panel does keep the app filter, since that metric is emitted by every app.

This is the exact query backing the "LM + Catalog requests" panel of the Grafana dashboard. The $account textbox variable defaults to __all__ (every account); type a quoted account such as 'storecomponents' to scope it.

Error ratio per dependency:

Latency uses the shared histogram io_app_operation_duration_milliseconds filtered by Attributes['dependency']. The exact quantile read depends on how histograms are materialized in your ClickHouse schema (bucket columns vs. quantilesMerge); see the dashboard panel for the template and adjust.

The dependency-health dashboard (maintained as Grafana-as-code outside this repo) is a single, multi-app Grafana dashboard backed by a ClickHouse datasource. It is organized in three sections:

• Saúde — está tudo bem? at-a-glance stat tiles (overall + per dependency) that read OK / Atenção / Crítico from the error ratio, plus a total request volume sanity tile.
• Latência friendly average-latency tiles per dependency (green < 200ms, orange < 500ms, red ≥ 500ms) with sparklines, plus a latency time series.
• Tráfego e erros request rate by outcome, HTTP status class, error ratio and the rewriter-specific binding-resolution mix. The per-dependency panels repeat over $dependency, so they cover whatever dependencies the selected app emits — no panel is tied to a specific dependency or operation.

Variable	Type	Purpose
datasource	datasource	ClickHouse datasource.
app	custom	App to inspect. The value is the metric prefix (e.g. vtex.rewriter → rewriter), which drives the per-app metric name ${app}_dependency_requests_total. Add one vtex.<app> : <prefix> option as each app is migrated.
appVersion	query	Concrete app@x.x (e.g. vtex.rewriter@1.x), auto-discovered from the data for the selected app. Filters the app column; keep All to span every running version.
dependency	query (multi)	Auto-discovered from the data for the selected app (SELECT DISTINCT Attributes['dependency'] …). Drives the repeated per-dependency panels — nothing is hardcoded to catalog / license-manager / a specific operation.
workspace, tenant, production	textbox	Scope filters (ClickHouse columns).
account	textbox	__all__ for every account, or quoted account(s) such as 'storecomponents'.

The per-dependency panels (error rate tile, latency tile, success-vs-error, HTTP status class) use Grafana panel repeat over $dependency: one panel is rendered per dependency the selected app actually emits, so the dashboard adapts to any app without editing queries. The cross-dependency comparison panels (latency over time, error ratio) are data-driven too — they GROUP BY Attributes['dependency'] and produce one series per dependency.

The dependency counters are app-prefixed (${app}_dependency_requests_total, ${app}_binding_resolution_total); the latency histogram is the shared io_app_operation_duration_milliseconds, scoped to the selected app via the app column and Attributes['dependency']. The binding-resolution panel is rewriter-specific and stays empty for apps that do not emit that metric.

The accompanying alert rules (also kept outside this repo) define starting-point rules, all segregated per dependency (each rule does GROUP BY dependency, so Grafana raises one independent alert instance per dependency, labeled dependency):

Rule	Source	Default condition
rewriter-dep-error-rate	rewriter_dependency_requests_total	error/total > 5% for 5m (warning)
rewriter-dep-5xx	status_class = '5xx'	> 5 responses in 5m (critical)
rewriter-dep-4xx	status_class = '4xx'	> 10 responses in 5m (warning)
rewriter-dep-latency	io_app_operation_duration_milliseconds	avg > 1500ms for 5m (warning)

The metric and attribute names above are the stable contract; tune thresholds, the app / appVersion / workspace variables and time windows to your environment before enabling. Conditional aggregation uses sumMergeIf(...) (not the SQL FILTER clause, which is unreliable with -Merge over ClickHouse AggregateFunction columns).