Building an AI-Powered Feature in Node.js with the OpenAI API

Calling openai.chat.completions.create() and piping the result back to your frontend takes about ten minutes. Shipping that same feature to production takes a lot longer, and almost none of the extra work is about the AI part.

The gap is everything a real feature needs that a demo doesn’t: a user shouldn’t stare at a spinner for eight seconds waiting for a full response, one abusive client shouldn’t be able to run up a four-figure OpenAI bill overnight, the OpenAI API will time out or 500 on you in production, and if ten users ask the same question in the same hour, you’re paying for the same completion ten times. None of this shows up in the “hello world” example. All of it shows up in your first week of real traffic.

In this guide I’ll build a production-grade AI-assisted search/summarization endpoint in a TypeScript + Express API: SDK setup, a real endpoint, streaming with Server-Sent Events, rate limiting, timeout/retry handling, response caching, and a deployment path for both ECS and Lambda. Every code block is something you can drop into a real project.

Setting up the OpenAI SDK in a TypeScript Express project

Start with a clean project and the pieces we’ll actually use — the OpenAI SDK, Express, Zod for validation, and a couple of small utilities for rate limiting and caching.

mkdir ai-feature-api && cd ai-feature-api
npm init -y
npm install express openai zod dotenv lru-cache express-rate-limit
npm install -D typescript @types/express @types/node tsx
npx tsc --init

.env — never hardcode the API key, and never let it reach client code:

OPENAI_API_KEY=sk-your-key-here
OPENAI_TIMEOUT_MS=15000
PORT=3000

src/openaiClient.ts — a single, shared client instance with sane defaults baked in. This is the file every route imports from; don't instantiate OpenAI ad hoc inside handlers.

// src/openaiClient.ts
import OpenAI from "openai";
import "dotenv/config";

if (!process.env.OPENAI_API_KEY) {
  throw new Error("OPENAI_API_KEY is not set. Check your .env file.");
}

export const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
  // The SDK's own retry/timeout handling is a starting point — we'll
  // layer our own retry logic on top for finer control later.
  timeout: Number(process.env.OPENAI_TIMEOUT_MS ?? 15_000),
  maxRetries: 0, // we handle retries ourselves, see errorHandling.ts
});

// Centralize model choice so a model bump is a one-line change.
export const MODEL = "gpt-4o-mini";

Setting maxRetries: 0 is deliberate. The SDK's built-in retry is fine for scripts, but in a request/response API you want to control when to retry, how many times, and what happens to the client while you do it — that's its own section below.

src/index.ts — the app skeleton:

// src/index.ts
import express from "express";
import "dotenv/config";
import { summarizeRouter } from "./routes/summarize";

const app = express();
app.use(express.json({ limit: "1mb" }));

app.use("/api/summarize", summarizeRouter);

app.get("/health", (_req, res) => res.json({ ok: true }));

const port = Number(process.env.PORT ?? 3000);
app.listen(port, () => console.log(`Listening on :${port}`));

Designing an endpoint for a real feature

Let’s build something concrete: a summarization endpoint that takes a block of text (or a search query plus a set of documents) and returns an AI-generated summary. This is a common real-world shape — support ticket summarization, doc search assist, changelog digesting.

The design decisions that matter here, before we write a line of prompt code:

• Validate input strictly. Don’t let arbitrary-length text hit the model unchecked — that’s a cost and latency risk.
• Cap max_tokens. An unbounded response is an unbounded bill.
• Keep the system prompt server-side. Never let the client influence the system prompt directly.
• Return a request ID. You’ll want it in logs when a user reports a bad output.

// src/routes/summarize.ts
import { Router } from "express";
import { z } from "zod";
import { randomUUID } from "crypto";
import { openai, MODEL } from "../openaiClient";
import { callWithRetry } from "../lib/errorHandling";
import { getCached, setCached, cacheKeyFor } from "../lib/cache";
import { rateLimiter } from "../lib/rateLimiter";

export const summarizeRouter = Router();

const requestSchema = z.object({
  text: z.string().min(20, "Text is too short to summarize").max(12_000),
  style: z.enum(["brief", "detailed", "bullets"]).default("brief"),
});

const STYLE_INSTRUCTIONS: Record<string, string> = {
  brief: "Summarize in 2-3 sentences.",
  detailed: "Summarize in a thorough paragraph, keeping key facts and numbers.",
  bullets: "Summarize as 4-6 concise bullet points.",
};

summarizeRouter.post("/", rateLimiter, async (req, res) => {
  const requestId = randomUUID();
  const parsed = requestSchema.safeParse(req.body);

  if (!parsed.success) {
    return res.status(400).json({
      requestId,
      error: "Invalid request",
      details: parsed.error.flatten(),
    });
  }

  const { text, style } = parsed.data;
  const cacheKey = cacheKeyFor({ text, style });

  const cached = getCached(cacheKey);
  if (cached) {
    return res.json({ requestId, summary: cached, cached: true });
  }

  try {
    const completion = await callWithRetry(() =>
      openai.chat.completions.create({
        model: MODEL,
        max_tokens: 300,
        temperature: 0.3,
        messages: [
          {
            role: "system",
            content:
              "You are a precise summarization assistant. Never invent facts not present in the source text.",
          },
          { role: "user", content: `${STYLE_INSTRUCTIONS[style]}\n\nText:\n${text}` },
        ],
      })
    );

    const summary = completion.choices[0]?.message?.content ?? "";
    setCached(cacheKey, summary);

    res.json({ requestId, summary, cached: false });
  } catch (err) {
    console.error(`[${requestId}] summarize failed:`, err);
    res.status(502).json({ requestId, error: "Summarization service unavailable" });
  }
});

This is already miles ahead of the naive version — validated input, capped output, a request ID for support, and hooks for caching and retries. Now let’s make it feel fast.

Streaming responses to the client (SSE vs chunked responses)

A 300-token summary can take 3–6 seconds to generate in full. Streaming it token-by-token makes the feature feel instant, because the user sees output within a few hundred milliseconds instead of staring at nothing.

You have two realistic options in Node/Express:

• Server-Sent Events (SSE) — a long-lived HTTP response over text/event-stream, one-directional (server → client), reconnects automatically in the browser via EventSource. This is the right default for streaming AI text to a browser.
• Plain chunked responses — you write raw chunks to the response stream yourself (res.write()) without the SSE framing. Simpler, but you lose automatic reconnect, event typing, and most frontend libraries expect SSE framing for AI streams anyway.

Use SSE unless you have a specific reason not to (e.g., streaming to a non-browser consumer where you fully control both ends — then raw chunked res.write() is fine and slightly leaner).

// src/routes/summarizeStream.ts
import { Router } from "express";
import { z } from "zod";
import { randomUUID } from "crypto";
import { openai, MODEL } from "../openaiClient";
import { rateLimiter } from "../lib/rateLimiter";

export const summarizeStreamRouter = Router();

const requestSchema = z.object({
  text: z.string().min(20).max(12_000),
});

summarizeStreamRouter.post("/", rateLimiter, async (req, res) => {
  const requestId = randomUUID();
  const parsed = requestSchema.safeParse(req.body);
  if (!parsed.success) {
    return res.status(400).json({ requestId, error: "Invalid request" });
  }

  // SSE headers — set before writing anything else.
  res.setHeader("Content-Type", "text/event-stream");
  res.setHeader("Cache-Control", "no-cache, no-transform");
  res.setHeader("Connection", "keep-alive");
  res.flushHeaders();

  const send = (event: string, data: unknown) => {
    res.write(`event: ${event}\n`);
    res.write(`data: ${JSON.stringify(data)}\n\n`);
  };

  // Clean up if the client disconnects mid-stream (closed tab, etc).
  const abortController = new AbortController();
  req.on("close", () => abortController.abort());

  try {
    const stream = await openai.chat.completions.create(
      {
        model: MODEL,
        max_tokens: 300,
        temperature: 0.3,
        stream: true,
        messages: [
          { role: "system", content: "Summarize the text in 2-3 sentences." },
          { role: "user", content: parsed.data.text },
        ],
      },
      { signal: abortController.signal }
    );

    for await (const chunk of stream) {
      const token = chunk.choices[0]?.delta?.content;
      if (token) send("token", { token });
    }

    send("done", { requestId });
    res.end();
  } catch (err) {
    console.error(`[${requestId}] stream failed:`, err);
    send("error", { requestId, message: "Stream interrupted" });
    res.end();
  }
});

On the frontend, this is consumed with plain fetch and a stream reader (SSE via EventSource only supports GET, so for a POST-initiated stream, read the body manually):

// client-side (browser), for reference
async function streamSummary(text: string) {
  const res = await fetch("/api/summarize/stream", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ text }),
  });

  const reader = res.body!.getReader();
  const decoder = new TextDecoder();
  let output = "";

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    const chunk = decoder.decode(value);
    for (const line of chunk.split("\n\n")) {
      if (line.startsWith("data: ")) {
        const payload = JSON.parse(line.slice(6));
        if (payload.token) {
          output += payload.token;
          renderPartial(output); // update UI incrementally
        }
      }
    }
  }
}

The important detail in the server code: req.on("close", ...) aborts the OpenAI request when the client disconnects. Without it, a user closing their tab mid-summary still costs you the full completion.

Rate limiting and cost control

AI endpoints need rate limiting more than almost any other kind of route, because the cost-per-request is orders of magnitude higher than a typical CRUD call. express-rate-limit handles the basic per-IP case; layer a per-user token budget on top for real cost control.

// src/lib/rateLimiter.ts
import rateLimit from "express-rate-limit";
import type { Request, Response, NextFunction } from "express";

// Coarse layer: caps requests per IP, protects against abuse/bots.
export const rateLimiter = rateLimit({
  windowMs: 60 * 1000, // 1 minute
  max: 10, // 10 AI requests per minute per IP
  standardHeaders: true,
  legacyHeaders: false,
  message: { error: "Too many requests. Please slow down." },
});

// Finer layer: caps token spend per authenticated user per day.
// Swap the in-memory Map for Redis in a multi-instance deployment.
const dailyTokenUsage = new Map<string, { tokens: number; resetAt: number }>();
const DAILY_TOKEN_BUDGET = 50_000;

export function tokenBudgetGuard(req: Request, res: Response, next: NextFunction) {
  const userId = (req as any).userId ?? req.ip;
  const now = Date.now();
  const entry = dailyTokenUsage.get(userId);

  if (!entry || entry.resetAt < now) {
    dailyTokenUsage.set(userId, { tokens: 0, resetAt: now + 24 * 60 * 60 * 1000 });
    return next();
  }

  if (entry.tokens >= DAILY_TOKEN_BUDGET) {
    return res.status(429).json({
      error: "Daily AI usage limit reached. Resets at midnight UTC.",
    });
  }

  next();
}

// Call this after a completion returns, using completion.usage.total_tokens.
export function recordTokenUsage(userId: string, tokens: number) {
  const entry = dailyTokenUsage.get(userId);
  if (entry) entry.tokens += tokens;
}

Wire the budget guard in alongside the request-rate limiter, and record actual usage after each completion:

// inside summarizeRouter.post handler, after a successful completion:
recordTokenUsage(requestId, completion.usage?.total_tokens ?? 0);

Two more cost levers worth setting from day one: pin max_tokens on every request (done above), and log completion.usage for every call to a metrics sink (CloudWatch, Datadog, whatever you use) so you can graph spend per route, not just guess at it from the OpenAI dashboard at month's end.

Error handling for provider timeouts/failures

The OpenAI API will occasionally time out, rate-limit you (429), or return a 5xx. Your endpoint needs to retry transient failures with backoff and fail fast on non-retryable ones — not surface a raw stack trace to the client.

// src/lib/errorHandling.ts
import { APIError } from "openai";

const RETRYABLE_STATUS_CODES = new Set([429, 500, 502, 503, 504]);
const MAX_RETRIES = 3;
const BASE_DELAY_MS = 500;

function isRetryable(err: unknown): boolean {
  if (err instanceof APIError && err.status) {
    return RETRYABLE_STATUS_CODES.has(err.status);
  }
  // Network-level errors (timeouts, DNS, connection resets) are worth retrying too.
  if (err instanceof Error && /timeout|ECONNRESET|ETIMEDOUT/i.test(err.message)) {
    return true;
  }
  return false;
}

function delay(ms: number) {
  return new Promise((resolve) => setTimeout(resolve, ms));
}

/**
 * Wraps an OpenAI call with exponential backoff + jitter.
 * Retries only on transient failures; rethrows immediately otherwise.
 */
export async function callWithRetry<T>(
  fn: () => Promise<T>,
  maxRetries = MAX_RETRIES
): Promise<T> {
  let attempt = 0;

  while (true) {
    try {
      return await fn();
    } catch (err) {
      attempt++;
      if (attempt > maxRetries || !isRetryable(err)) {
        throw err;
      }
      const backoff = BASE_DELAY_MS * 2 ** (attempt - 1);
      const jitter = Math.random() * 200;
      console.warn(`Retrying OpenAI call (attempt ${attempt}/${maxRetries}) after error:`, err);
      await delay(backoff + jitter);
    }
  }
}

Pair this with a timeout at the client level (already set via timeout in openaiClient.ts), and make sure your route handler always returns a clean error shape rather than letting an unhandled exception crash the response:

// A shared error responder, used in every AI route's catch block.
export function respondWithProviderError(res: Response, requestId: string, err: unknown) {
  if (err instanceof APIError && err.status === 429) {
    return res.status(503).json({
      requestId,
      error: "AI service is temporarily overloaded. Please try again shortly.",
    });
  }
  return res.status(502).json({
    requestId,
    error: "AI service is currently unavailable.",
  });
}

Never forward err.message from the OpenAI SDK directly to the client — it can leak internal details you don't want exposed, and it's rarely useful to an end user anyway. Log the full error server-side, return a generic message with the requestId client-side.

Caching repeat queries

Summarization and search-assist queries repeat a lot in practice — the same support doc, the same FAQ, the same popular search term. Caching identical requests avoids paying for and waiting on the same completion twice.

An in-memory LRU cache is enough for a single-instance deployment or as an L1 in front of Redis:

// src/lib/cache.ts
import { LRUCache } from "lru-cache";
import { createHash } from "crypto";

const cache = new LRUCache<string, string>({
  max: 500, // max entries
  ttl: 1000 * 60 * 30, // 30 minutes
});

export function cacheKeyFor(input: Record<string, unknown>): string {
  const normalized = JSON.stringify(input, Object.keys(input).sort());
  return createHash("sha256").update(normalized).digest("hex");
}

export function getCached(key: string): string | undefined {
  return cache.get(key);
}

export function setCached(key: string, value: string): void {
  if (value) cache.set(key, value);
}

For a multi-instance deployment (ECS with 3+ tasks, or Lambda with concurrent invocations), an in-process cache gets you cheap per-instance hits but not shared hits across instances. Swap in Redis for a shared cache with one interface change:

// src/lib/cacheRedis.ts — drop-in replacement for multi-instance deployments
import { createClient } from "redis";

const redis = createClient({ url: process.env.REDIS_URL });
await redis.connect();

export async function getCached(key: string): Promise<string | null> {
  return redis.get(`summary:${key}`);
}

export async function setCached(key: string, value: string): Promise<void> {
  await redis.set(`summary:${key}`, value, { EX: 60 * 30 }); // 30 min TTL
}

Only cache deterministic-enough requests — a temperature: 0.3 summarization of the same input text is a good candidate; a creative, high-temperature chat response usually isn't, since the client expects variation on repeat calls.

Deploying it on ECS/Lambda

Both are viable; the choice mostly comes down to your streaming requirements and traffic shape.

ECS (Fargate) is the better fit if SSE streaming is core to the feature. Long-lived connections behave predictably behind an Application Load Balancer, and you’re not fighting a platform’s execution-time or response-buffering model.

Key settings for the task definition:

{
  "containerDefinitions": [
    {
      "name": "ai-feature-api",
      "image": "<your-ecr-repo>:latest",
      "portMappings": [{ "containerPort": 3000 }],
      "environment": [{ "name": "PORT", "value": "3000" }],
      "secrets": [
        {
          "name": "OPENAI_API_KEY",
          "valueFrom": "arn:aws:secretsmanager:...:secret:openai-api-key"
        }
      ],
      "healthCheck": {
        "command": ["CMD-SHELL", "curl -f http://localhost:3000/health || exit 1"],
        "interval": 30,
        "timeout": 5,
        "retries": 3
      }
    }
  ]
}

Two ALB-specific gotchas for SSE: set the ALB idle timeout above your longest expected stream duration (e.g., 120s), and disable response buffering isn’t needed on ALB itself, but make sure any CloudFront distribution in front of it has streaming/Cache-Control: no-transform respected — CloudFront will otherwise buffer the whole SSE response before forwarding it, defeating the purpose.

Lambda works well for the non-streaming JSON endpoint (the first summarizeRouter above) fronted by API Gateway or a Lambda Function URL, and it's the cheaper option for spiky, low-volume traffic since you pay per invocation. Wrap the Express app with a Lambda adapter rather than rewriting routes as separate handlers:

// src/lambda.ts
import serverlessExpress from "@vendia/serverless-express";
import express from "express";
import { summarizeRouter } from "./routes/summarize";

const app = express();
app.use(express.json({ limit: "1mb" }));
app.use("/api/summarize", summarizeRouter);

export const handler = serverlessExpress({ app });

Set the function timeout above your OpenAI call’s max duration plus retry backoff — 30 seconds is a safe floor given the retry logic above. Avoid Lambda for the streaming route: Lambda Function URLs support response streaming now, but API Gateway does not, and the added complexity usually isn’t worth it unless streaming is a hard requirement on a serverless-only stack. My default recommendation: run the streaming endpoint on ECS/Fargate, and if you want the rest of your API on Lambda, that’s a fine split — they can share the same OpenAI client and cache modules.

Wrapping Up

What we built: a validated, cost-capped summarization endpoint; a streaming SSE variant that makes the feature feel instant; per-IP and per-user rate limiting; retry-with-backoff around provider failures; a cache layer that avoids paying twice for the same question; and a deployment path for both container and serverless targets.

A few things worth doing next if you’re taking this into production:

• Add structured logging of completion.usage per request, tagged by route and user, so you can build a real cost dashboard instead of checking the OpenAI billing page.
• Add a circuit breaker (a library like opossum works fine) in front of callWithRetry so that during a full OpenAI outage, you fail fast instead of queuing up retries that will all fail anyway.
• Test streaming under a flaky network — kill the connection mid-stream in a local test and confirm the AbortController actually cancels the upstream OpenAI call; it's easy to leak in-flight requests if you get the plumbing wrong.
• Consider a fallback model — if gpt-4o-mini is degraded, having a secondary model or provider to fall back to keeps the feature up even during a partial OpenAI outage.

None of this is exotic engineering — it’s the same discipline you’d apply to any external dependency with variable latency and a real cost per call. The AI part is the easy 10%; this is the other 90%.