Microservices — Practical

Microservices — Practical patterns

Outbox table & relay (PostgreSQL example)

CREATE TABLE outbox (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  aggregate_type TEXT NOT NULL,
  aggregate_id   TEXT NOT NULL,
  event_type     TEXT NOT NULL,
  payload        JSONB NOT NULL,
  created_at     TIMESTAMPTZ NOT NULL DEFAULT now(),
  published_at   TIMESTAMPTZ
);
CREATE INDEX outbox_unpublished ON outbox (created_at)
  WHERE published_at IS NULL;

await db.tx(async t => {
  await t.users.update(id, patch);
  await t.outbox.insert({
    aggregate_type: 'User', aggregate_id: id,
    event_type: 'UserUpdated', payload: patch,
  });
});

Relay loop (or CDC via Debezium reading WAL):

async function publishOutbox() {
  const rows = await db.query(
    `SELECT * FROM outbox WHERE published_at IS NULL ORDER BY created_at LIMIT 100 FOR UPDATE SKIP LOCKED`);
  for (const r of rows) {
    await broker.publish(r.event_type, r.payload, { messageId: r.id });
    await db.query(`UPDATE outbox SET published_at=now() WHERE id=$1`, [r.id]);
  }
}

Idempotent consumer (inbox)

CREATE TABLE inbox (
  message_id UUID PRIMARY KEY,
  processed_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

async function handle(msg: Msg) {
  await db.tx(async t => {
    const ok = await t.query(
      `INSERT INTO inbox (message_id) VALUES ($1) ON CONFLICT DO NOTHING RETURNING message_id`,
      [msg.id]);
    if (!ok.rowCount) return;            // duplicate
    await applyEffect(t, msg.payload);
  });
  await msg.ack();
}

Idempotency-Key middleware

app.post('/payments', async (req, res) => {
  const key = req.header('Idempotency-Key');
  if (!key) return res.status(400).end();
  const cached = await redis.get(`idem:${key}`);
  if (cached) return res.json(JSON.parse(cached));

  const result = await processPayment(req.body);
  await redis.set(`idem:${key}`, JSON.stringify(result), 'EX', 86400);
  res.json(result);
});

Circuit breaker (opossum, Node)

import CircuitBreaker from 'opossum';

const breaker = new CircuitBreaker(callDownstream, {
  timeout: 1500,
  errorThresholdPercentage: 50,
  resetTimeout: 30000,
});
breaker.on('open', () => log.warn('breaker open'));
breaker.fallback(() => ({ degraded: true }));

app.get('/x', async (_, res) => res.json(await breaker.fire()));

Retry with jitter (universal pattern)

async function retry<T>(fn: () => Promise<T>, max = 4, base = 100) {
  for (let i = 0; ; i++) {
    try { return await fn(); }
    catch (e) {
      if (i >= max || !isRetryable(e)) throw e;
      const sleep = base * 2 ** i + Math.random() * base;
      await new Promise(r => setTimeout(r, sleep));
    }
  }
}

Saga orchestration sketch (Temporal-style)

async function bookTrip(input: TripInput) {
  const flight = await activities.bookFlight(input);
  try {
    const hotel = await activities.bookHotel(input);
    try {
      const car = await activities.bookCar(input);
      return { flight, hotel, car };
    } catch (e) {
      await activities.cancelHotel(hotel);
      throw e;
    }
  } catch (e) {
    await activities.cancelFlight(flight);
    throw e;
  }
}

In Temporal, durable workflow handles retries, compensation, and replay automatically.

Dead letter queue (RabbitMQ example)

ch.assertExchange('orders', 'topic');
ch.assertExchange('orders.dlx', 'topic');
ch.assertQueue('orders.q', {
  arguments: { 'x-dead-letter-exchange': 'orders.dlx' },
});
ch.assertQueue('orders.dlq', {});
ch.bindQueue('orders.dlq', 'orders.dlx', '#');

Health & readiness endpoints (K8s)

app.get('/healthz', (_, res) => res.json({ status: 'ok' }));        // liveness
app.get('/readyz', async (_, res) => {                              // readiness
  const ok = await db.ping().then(() => true).catch(() => false);
  res.status(ok ? 200 : 503).json({ db: ok });
});

Distributed tracing (OpenTelemetry, Node)

import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';

new NodeSDK({
  traceExporter: new OTLPTraceExporter({ url: process.env.OTEL_ENDPOINT }),
  instrumentations: [require('@opentelemetry/auto-instrumentations-node').getNodeAutoInstrumentations()],
}).start();

Every log includes trace_id + span_id so logs join with traces.

Service-to-service auth

• mTLS via service mesh — zero-trust, strong identity.
• JWT signed by internal IdP — pass between services in headers; verify signature and audience.
• SPIFFE / SPIRE — workload identity standard.

Useful tooling

• Temporal / Cadence — durable workflows.
• Debezium — CDC outbox alternative.
• gRPC + buf — schema-first, generated clients.
• OpenAPI / Swagger — REST contracts.
• Pact — consumer-driven contract tests.
• K6 / Locust — load test.
• Linkerd / Istio — service mesh.
• Jaeger / Tempo / Honeycomb — tracing backend.

---

Deep — Outbox + Inbox SQL pattern

-- Producer side: business write + outbox in same transaction
BEGIN;
  INSERT INTO orders(id, customer_id, total, status)
  VALUES ('o-123', 'c-9', 199.00, 'PENDING');

  INSERT INTO outbox(id, aggregate_type, aggregate_id, event_type, payload, created_at)
  VALUES (gen_random_uuid(), 'Order', 'o-123', 'OrderCreated',
          '{"orderId":"o-123","total":199.00}'::jsonb, now());
COMMIT;
-- Debezium tails WAL → publishes to Kafka topic "orders.events"

-- Consumer side (Inbox): unique constraint dedupes retries
BEGIN;
  INSERT INTO processed_messages(consumer, message_id) VALUES ('billing', :msgId);
  -- ↑ unique violation if duplicate → rollback whole tx, ack message
  UPDATE invoices SET status='PAID' WHERE order_id=:orderId;
COMMIT;

Deep — derived read model via outbox

-- B owns `customers`. A needs customer name on the orders page.
-- B's outbox emits CustomerUpdated; A maintains a local projection.
CREATE TABLE customer_projection (
  customer_id uuid PRIMARY KEY,
  display_name text NOT NULL,
  tier text,
  updated_at timestamptz NOT NULL
);

-- A's consumer (idempotent inbox + upsert)
BEGIN;
  INSERT INTO processed_messages(consumer, message_id) VALUES ('orders', :msgId);
  INSERT INTO customer_projection (customer_id, display_name, tier, updated_at)
  VALUES (:id, :name, :tier, :ts)
  ON CONFLICT (customer_id) DO UPDATE
    SET display_name = EXCLUDED.display_name,
        tier         = EXCLUDED.tier,
        updated_at   = EXCLUDED.updated_at
    WHERE customer_projection.updated_at < EXCLUDED.updated_at;
COMMIT;

Deep — Kubernetes Service for discovery

apiVersion: v1
kind: Service
metadata: { name: orders, namespace: prod }
spec:
  type: ClusterIP            # internal VIP
  selector: { app: orders }
  ports: [{ port: 80, targetPort: 8080 }]
---
# DNS: orders.prod.svc.cluster.local
# Backed by EndpointSlice auto-managed from Pod readiness

Deep — Protobuf evolution

message Order {
  reserved 4, 7;             // formerly `discount_code`, `legacy_status`
  reserved "discount_code";
  string id           = 1;
  string customer_id  = 2;
  int64  total_cents  = 3;
  optional string note = 5;  // additive — old clients ignore unknown
}

Deep — OpenAPI snippet

paths:
  /orders/{id}:
    get:
      operationId: getOrder
      parameters: [{ name: id, in: path, required: true, schema: { type: string }}]
      responses:
        "200": { content: { application/json: { schema: { $ref: '#/components/schemas/Order' }}}}
        "404": { description: Not found }

Deep — Pact consumer test + can-i-deploy gate

# Consumer test (pseudo)
pact.given('order o-123 exists').
     upon_receiving('a get order').
     with(method: :get, path: '/orders/o-123').
     will_respond_with(status: 200, body: { id: 'o-123', total: 199.0 })
# → generates web-app__orders-svc.pact.json, uploaded to broker

# CI gate
pact-broker can-i-deploy --pacticipant orders-svc --version $GIT_SHA --to-environment production

Deep — Kafka transactions (Kafka-only EOS)

// Kafka transactions — exactly-once *within* Kafka (multi-topic atomic write)
await producer.initTransactions();
await producer.beginTransaction();
try {
  await producer.send({ topic: 'orders', messages: [{ key: orderId, value: orderJson }] });
  await producer.send({ topic: 'audit',  messages: [{ key: orderId, value: auditJson }] });
  await producer.sendOffsets({ consumerGroupId: 'orders-svc', topics: offsets });
  await producer.commitTransaction();
} catch (e) {
  await producer.abortTransaction();
  throw e;
}
// NOTE: this protects Kafka→Kafka only. Postgres still needs outbox+inbox.

Deep — Fitness functions in CI

// JUnit 5 + ArchUnit: domain modules must not import each other
@ArchTest
static final ArchRule domains_independent =
    noClasses().that().resideInAPackage("..domain.orders..")
               .should().dependOnClassesThat().resideInAPackage("..domain.billing..");

# CI gate using dependency-cruiser (Node ecosystem)
npx depcruise --validate .dependency-cruiser.cjs src/
# fails the pipeline if any forbidden import sneaks in

Deep — ADR template (Nygard)

# ADR-0007: Use Temporal for saga orchestration

## Status
Accepted (2026-04-12)

## Context
We have 5 services participating in the order workflow with branching logic
and 12-hour timeouts. Choreography led to debugging pain (no single workflow view).

## Decision
Adopt Temporal as the orchestrator. Workflows live in a `workflows/` module
in each owning service. Activities are idempotent.

## Consequences
+ Single source of truth for workflow state; durable retries; replay debugging.
- New runtime (Temporal cluster) to operate; team must learn the SDK.

## Alternatives considered
- AWS Step Functions: rejected — vendor lock-in incompatible with policy.
- Custom state machine in Postgres: rejected — reinvents retries/timeouts poorly.

Deep — Sunset header (RFC 8594)

HTTP/1.1 200 OK
Deprecation: Sun, 01 Jun 2026 00:00:00 GMT
Sunset:      Sun, 01 Jun 2027 00:00:00 GMT
Link: <https://api.example.com/docs/migrate-v2>; rel="sunset"