It deployed fine. Tests passed, the pipeline went green, ECS rolled out the new task definition without a hiccup. Then three days later a customer opens a ticket saying checkout is “slow sometimes,” and you’re staring at a CloudWatch Logs console with console.log output scattered across a dozen containers, no way to tell which request belongs to which log line, and zero idea whether the slowness is your API, the database, or the payment provider's API three hops downstream.
This is the gap between “it’s deployed” and “I know what it’s doing.” I’ve shipped enough Node.js services on ECS/Fargate to know you don’t get observability for free — you have to build it in, the same way you build in auth or error handling. This guide walks through the three pillars (logs, metrics, traces), wires them into an Express API with real TypeScript, and ships everything to CloudWatch and OpenTelemetry without blowing your AWS bill. By the end you’ll have structured logs with correlation IDs, custom metrics via Embedded Metric Format, distributed traces across services, a working dashboard and alarm, and a sampling strategy that keeps cost sane.
What to Instrument: Logs, Metrics, Traces
These three answer different questions, and you need all three because none of them answers the other two’s question well.
- Logs tell you what happened — a specific event, with context. “User 4471 failed checkout: card declined.” Great for debugging one request, expensive at scale if you log everything.
- Metrics tell you how much / how often — aggregated numbers over time. “p99 latency on /checkout is 850ms." Cheap to store, great for dashboards and alarms, but they don't tell you why.
- Traces tell you where time went across a request’s lifecycle — which service, which downstream call, which DB query ate the 800ms. Essential the moment you have more than one service talking to another.
A slow endpoint shows up first in metrics (alarm fires), gets explained by a trace (which span is slow), and gets root-caused by logs (the actual error message on that span). Skip any one of the three and you’re guessing at production incidents instead of diagnosing them.
Structured Logging in Express with Correlation IDs
console.log("something happened") is not a logging strategy. You want JSON logs — one object per line — with a request ID that ties every log line from a single request together, and that ID needs to propagate downstream if you call other services.
Install a fast structured logger and a UUID generator:
npm install pino pino-http uuid
npm install -D @types/uuid
Set up the logger and the correlation ID middleware:
// src/observability/logger.ts
import pino from "pino";
// In production we want plain JSON (CloudWatch parses this natively).
// Locally, pretty-print for readability.
export const logger = pino({
level: process.env.LOG_LEVEL ?? "info",
base: {
service: process.env.SERVICE_NAME ?? "checkout-api",
env: process.env.NODE_ENV ?? "development",
},
timestamp: pino.stdTimeFunctions.isoTime,
transport:
process.env.NODE_ENV === "development"
? { target: "pino-pretty", options: { colorize: true } }
: undefined,
});
// src/middleware/requestContext.ts
import { randomUUID } from "crypto";
import { AsyncLocalStorage } from "async_hooks";
import type { NextFunction, Request, Response } from "express";
import { logger } from "../observability/logger";
interface RequestContext {
requestId: string;
userId?: string;
}// AsyncLocalStorage lets any function down the call stack read the
// current request's context without threading it through every argument.
export const requestContextStorage = new AsyncLocalStorage<RequestContext>();
export function requestContextMiddleware(
req: Request,
res: Response,
next: NextFunction
) {
// Reuse an incoming correlation ID (e.g. from an upstream service or
// API Gateway) if present, otherwise mint a new one.
const requestId =
(req.headers["x-request-id"] as string | undefined) ?? randomUUID();
res.setHeader("x-request-id", requestId); const context: RequestContext = { requestId }; requestContextStorage.run(context, () => {
const start = Date.now();// Child logger carries requestId on every line automatically.
req.log = logger.child({ requestId });
res.on("finish", () => {
req.log.info(
{
method: req.method,
path: req.route?.path ?? req.path,
statusCode: res.statusCode,
durationMs: Date.now() - start,
},
"request completed"
);
});next();
});
}
export function getRequestId(): string | undefined {
return requestContextStorage.getStore()?.requestId;
}Wire it in before your routes, and extend Express’s Request type so req.log type-checks:
// src/types/express.d.ts
import type { Logger } from "pino";
declare global {
namespace Express {
interface Request {
log: Logger;
}
}
}// src/app.ts
import express from "express";
import { requestContextMiddleware } from "./middleware/requestContext";
const app = express();
app.use(requestContextMiddleware);
app.get("/checkout/:orderId", async (req, res) => {
req.log.info({ orderId: req.params.orderId }, "processing checkout");
// ... business logic, using req.log everywhere instead of console.log
res.json({ status: "ok" });
});export default app;
Every line is now JSON with requestId, service, env, and a timestamp. Ship this to CloudWatch Logs and every query becomes fields @message | filter requestId = "abc-123" — one query, the entire request's story, across every log group if you use the same field name consistently across services.
Sending Metrics to CloudWatch (Custom Metrics vs EMF)
You have two ways to get numbers into CloudWatch from Node.js: calling PutMetricData directly via the SDK, or emitting Embedded Metric Format (EMF) log lines that CloudWatch parses automatically. Use EMF. It's free (no extra API calls, no extra IAM permissions beyond writing to logs you already write to), it's async (no blocking your request on an AWS API call), and it batches automatically.
Direct PutMetricData for comparison — this is what you're avoiding:
// The naive approach — an extra network call per metric, on your request path.
import { CloudWatchClient, PutMetricDataCommand } from "@aws-sdk/client-cloudwatch";
const cw = new CloudWatchClient({});async function recordLatency(durationMs: number) {
await cw.send(
new PutMetricDataCommand({
Namespace: "CheckoutAPI",
MetricData: [
{ MetricName: "RequestLatency", Value: durationMs, Unit: "Milliseconds" },
],
})
);
}That’s a synchronous AWS call in your request path for every metric, every request. It works, but it adds latency and cost. EMF instead: you just log a specially-shaped JSON object, and the CloudWatch agent/Lambda extension/Fargate FireLens sidecar extracts metrics from it in the background.
npm install aws-embedded-metrics
// src/observability/metrics.ts
import { metricScope, Unit } from "aws-embedded-metrics";
import type { MetricsLogger } from "aws-embedded-metrics";
// metricScope wraps a function and gives it a MetricsLogger that flushes
// an EMF-formatted log line when the function returns.
export const recordCheckoutMetrics = metricScope(
(metrics: MetricsLogger) =>
(durationMs: number, statusCode: number, orderValueCents: number) => {
metrics.setNamespace("CheckoutAPI");
// Dimensions let you slice metrics in the console (by route, by env).
metrics.setDimensions({ Route: "/checkout", Environment: process.env.NODE_ENV ?? "production" });
metrics.putMetric("RequestLatency", durationMs, Unit.Milliseconds);
metrics.putMetric("RequestCount", 1, Unit.Count);
metrics.putMetric(
statusCode >= 500 ? "ServerErrorCount" : "SuccessCount",
1,
Unit.Count
);// Business metrics belong here too, not just infra metrics.
metrics.putMetric("OrderValueCents", orderValueCents, Unit.None);
}
);
Use it in the route:
// src/routes/checkout.ts
import { Router } from "express";
import { recordCheckoutMetrics } from "../observability/metrics";
export const checkoutRouter = Router();
checkoutRouter.post("/checkout", async (req, res) => {
const start = Date.now();
try {
const order = await processOrder(req.body); // your existing logic
res.json(order);
recordCheckoutMetrics(Date.now() - start, res.statusCode, order.totalCents);
} catch (err) {
req.log.error({ err }, "checkout failed");
res.status(500).json({ error: "checkout_failed" });
recordCheckoutMetrics(Date.now() - start, 500, 0);
}
});On Fargate, EMF log lines need to reach CloudWatch Logs, which they already do if your container logs to stdout and your task definition uses the awslogs driver — no sidecar required. CloudWatch automatically detects the EMF schema in any log line shipped to a log group and creates the metrics for you, no extra config beyond making sure the log group exists.
Distributed Tracing with OpenTelemetry Across Microservices
Once you have more than one service, “which service is slow” stops being answerable from logs alone. OpenTelemetry (OTel) gives you vendor-neutral traces you can ship to AWS X-Ray (or anywhere else later — that portability is the whole point of using OTel instead of the X-Ray SDK directly).
npm install @opentelemetry/api @opentelemetry/sdk-node \
@opentelemetry/auto-instrumentations-node \
@opentelemetry/exporter-trace-otlp-http \
@opentelemetry/resources @opentelemetry/semantic-conventions
Set up the SDK in a file loaded before your app code (this matters — instrumentation needs to patch modules before they’re required):
// src/observability/tracing.ts
import { NodeSDK } from "@opentelemetry/sdk-node";
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { Resource } from "@opentelemetry/resources";
import { SemanticResourceAttributes as ResAttr } from "@opentelemetry/semantic-conventions";
const sdk = new NodeSDK({
resource: new Resource({
[ResAttr.SERVICE_NAME]: process.env.SERVICE_NAME ?? "checkout-api",
[ResAttr.SERVICE_VERSION]: process.env.SERVICE_VERSION ?? "unknown",
// Ties every span emitted from this process back to a specific
// running ECS task — critical for step 6 below.
"aws.ecs.task.id": process.env.ECS_TASK_ID,
"aws.ecs.cluster": process.env.ECS_CLUSTER_NAME,
}),
// Ships spans to the ADOT (AWS Distro for OpenTelemetry) Collector
// running as a sidecar container, which forwards them to X-Ray.
traceExporter: new OTLPTraceExporter({
url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT ?? "http://localhost:4318/v1/traces",
}),
instrumentations: [
getNodeAutoInstrumentations({
// Auto-instruments express, http, pg, aws-sdk, and more out of the box.
"@opentelemetry/instrumentation-fs": { enabled: false }, // too noisy
}),
],
});sdk.start();
process.on("SIGTERM", () => {
sdk.shutdown().finally(() => process.exit(0));
});Load it first via Node’s --require, or preload flag, before your entrypoint:
// package.json (relevant script)
{
"scripts": {
"start": "node --require ./dist/observability/tracing.js dist/index.js"
}
}
For calls that aren’t auto-instrumented — say, an internal call to another microservice via fetch where you want a custom span with business context — create spans manually:
// src/services/inventoryClient.ts
import { trace, SpanStatusCode } from "@opentelemetry/api";
const tracer = trace.getTracer("checkout-api");export async function reserveInventory(sku: string, qty: number) {
return tracer.startActiveSpan("inventory.reserve", async (span) => {
span.setAttribute("sku", sku);
span.setAttribute("qty", qty);
try {
const res = await fetch(`${process.env.INVENTORY_URL}/reserve`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ sku, qty }),
});
if (!res.ok) throw new Error(`inventory service returned ${res.status}`);
span.setStatus({ code: SpanStatusCode.OK });
return res.json();
} catch (err) {
span.recordException(err as Error);
span.setStatus({ code: SpanStatusCode.ERROR });
throw err;
} finally {
span.end();
}
});
}The HTTP auto-instrumentation propagates the trace context (traceparent header) automatically, so the inventory service — if it's also running OTel — picks up the same trace and its spans nest under this one. That's what lets you open one trace in X-Ray and see the full request across both services, with actual timing per hop.
Run the ADOT Collector as a sidecar in your ECS task definition so the app never talks to AWS APIs directly for tracing:
{
"containerDefinitions": [
{
"name": "checkout-api",
"environment": [
{ "name": "OTEL_EXPORTER_OTLP_ENDPOINT", "value": "http://localhost:4318" }
]
},
{
"name": "aws-otel-collector",
"image": "public.ecr.aws/aws-observability/aws-otel-collector:latest",
"command": ["--config=/etc/ecs/ecs-xray.yaml"]
}
]
}Building a Basic CloudWatch Dashboard and Alarm
Console-clicking dashboards doesn’t survive a redeploy of your infra. Define it as code. Here’s a CDK stack for the dashboard and an alarm on error rate, matching the EMF metrics from earlier:
// infra/observability-stack.ts
import * as cdk from "aws-cdk-lib";
import { Construct } from "constructs";
import * as cloudwatch from "aws-cdk-lib/aws-cloudwatch";
import * as actions from "aws-cdk-lib/aws-cloudwatch-actions";
import * as sns from "aws-cdk-lib/aws-sns";
import * as subs from "aws-cdk-lib/aws-sns-subscriptions";
export class ObservabilityStack extends cdk.Stack {
constructor(scope: Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);const alertTopic = new sns.Topic(this, "CheckoutAlerts");
alertTopic.addSubscription(new subs.EmailSubscription("oncall@example.com"));
const namespace = "CheckoutAPI";
const latency = new cloudwatch.Metric({
namespace,
metricName: "RequestLatency",
statistic: "p99",
period: cdk.Duration.minutes(1),
});
const errors = new cloudwatch.Metric({
namespace,
metricName: "ServerErrorCount",
statistic: "Sum",
period: cdk.Duration.minutes(1),
});
const requests = new cloudwatch.Metric({
namespace,
metricName: "RequestCount",
statistic: "Sum",
period: cdk.Duration.minutes(1),
});
// Error rate as a math expression: errors / total requests.
const errorRate = new cloudwatch.MathExpression({
expression: "(errors / requests) * 100",
usingMetrics: { errors, requests },
label: "Error Rate (%)",
});
new cloudwatch.Alarm(this, "HighErrorRateAlarm", {
metric: errorRate,
threshold: 5, // percent
evaluationPeriods: 3,
datapointsToAlarm: 2,
treatMissingData: cloudwatch.TreatMissingData.NOT_BREACHING,
alarmDescription: "Checkout API error rate above 5% for 2 of 3 minutes",
}).addAlarmAction(new actions.SnsAction(alertTopic)); new cloudwatch.Alarm(this, "HighLatencyAlarm", {
metric: latency,
threshold: 1000, // ms
evaluationPeriods: 5,
datapointsToAlarm: 3,
treatMissingData: cloudwatch.TreatMissingData.NOT_BREACHING,
}).addAlarmAction(new actions.SnsAction(alertTopic)); new cloudwatch.Dashboard(this, "CheckoutDashboard", {
dashboardName: "checkout-api-production",
widgets: [
[
new cloudwatch.GraphWidget({ title: "p99 Latency (ms)", left: [latency], width: 12 }),
new cloudwatch.GraphWidget({ title: "Requests / Errors", left: [requests, errors], width: 12 }),
],
],
});
}
}Deploy with cdk deploy ObservabilityStack. You now have an alarm that pages someone the moment error rate crosses 5% for two out of three minutes — tight enough to catch real incidents, loose enough to not page on a single blip.
Tying Traces Back to Specific ECS Tasks/Containers
When an alarm fires, “the checkout service is slow” isn’t enough — you need to know which task, because on Fargate you might have 8 tasks running and only one is thrashing (bad host, noisy neighbor, memory leak in that specific process). That’s why the resource attributes in the tracing setup above included aws.ecs.task.id and aws.ecs.cluster — every span carries them.
Pull the task ID from the ECS metadata endpoint at container startup rather than hardcoding it:
// src/observability/ecsMetadata.ts
export async function getEcsTaskId(): Promise<string | undefined> {
const metadataUri = process.env.ECS_CONTAINER_METADATA_URI_V4;
if (!metadataUri) return undefined; // not running on ECS (e.g. local dev)
try {
const res = await fetch(`${metadataUri}/task`);
const data = (await res.json()) as { TaskARN: string };
return data.TaskARN.split("/").pop();
} catch {
return undefined; // don't fail startup over metadata being unavailable
}
}Set ECS_TASK_ID as an environment variable at boot, before tracing.ts reads it:
// src/index.ts
import { getEcsTaskId } from "./observability/ecsMetadata";
async function bootstrap() {
process.env.ECS_TASK_ID = (await getEcsTaskId()) ?? "local";
// Now safe to import app/tracing — but since tracing.ts is loaded via
// --require before this file runs, prefer reading task metadata
// synchronously from the sidecar's env injection instead when possible.
}In practice, the cleanest path is to inject ECS_TASK_ID from the ECS task definition itself using the container's built-in metadata rather than fetching it in-process — but the fetch approach above works when you need it available before the container's env is finalized. Either way, once aws.ecs.task.id is on every span, you can filter X-Ray traces or CloudWatch Logs Insights queries down to one misbehaving task:
fields @timestamp, @message
| filter ecsTaskId = "a1b2c3d4e5f6..."
| filter statusCode >= 500
| sort @timestamp desc
Pair that with a Container Insights dashboard (enable it on the ECS cluster — one checkbox in the console or containerInsights: true in CDK's Cluster construct) and you get per-task CPU/memory graphs right next to the trace data, so "is this task starved for CPU" is a 10-second check instead of a guess.
Cost-Conscious Observability
Full tracing and unlimited log retention on every request will absolutely show up on your AWS bill at scale. Three levers:
1. Trace sampling. You don’t need 100% of traces — you need enough to catch problems and enough of every error to debug them. Use a parent-based sampler with a ratio, but always sample errors:
// src/observability/tracing.ts (sampler addition)
import { ParentBasedSampler, TraceIdRatioBasedSampler } from "@opentelemetry/sdk-trace-base";
const sdk = new NodeSDK({
// ...
sampler: new ParentBasedSampler({
// Sample 10% of root traces; always respect an upstream decision
// to sample if this request is part of an already-sampled trace.
root: new TraceIdRatioBasedSampler(0.1),
}),
// ...
});For guaranteed error visibility despite sampling, force-sample manually on error paths in your span code (span.setAttribute("sampling.priority", 1) pattern, or emit an error metric via EMF regardless of trace sampling — metrics aren't sampled, so your error-rate alarm stays accurate even at 10% trace sampling).
2. Log retention and level. Set an explicit retention on every log group — the default is “never expire,” which quietly costs money forever:
// infra/log-groups.ts
import * as logs from "aws-cdk-lib/aws-logs";
new logs.LogGroup(this, "CheckoutApiLogs", {
logGroupName: "/ecs/checkout-api",
retention: logs.RetentionDays.TWO_WEEKS, // adjust per compliance needs
removalPolicy: cdk.RemovalPolicy.DESTROY,
});Run at info level in production, not debug — gate verbose logs behind LOG_LEVEL=debug you flip temporarily during an incident, not permanently.
3. Metric cardinality. Every unique combination of dimension values on a custom metric is a separate CloudWatch metric, billed separately. Never put unbounded values (user ID, order ID, request ID) in an EMF dimension — that’s what logs are for. Keep dimensions to a small, bounded set: route, environment, status class. This one mistake is the single most common cause of a surprise CloudWatch bill I’ve seen on client projects.
Wrapping Up
You now have a working loop: structured JSON logs with a correlation ID that ties every line of a request together, EMF-based custom metrics that cost nothing extra to emit, OpenTelemetry traces that follow a request across service boundaries and back to the exact ECS task that handled it, a CDK-defined dashboard and alarm that pages you before a customer files a ticket, and sampling/retention settings that keep the bill under control.
Next steps worth doing before you call this done: add the same requestId propagation to any queue-based (SQS/SNS) workflows, since correlation breaks the moment a request leaves HTTP; wire X-Ray trace IDs into your error tracker (Sentry, Bugsnag) so a stack trace and a distributed trace are one click apart; and run a load test against your /checkout route specifically to confirm your p99 alarm threshold is realistic before you rely on it in an actual incident. Observability you haven't tested under load is a guess with extra steps.
