Temporal Patterns: Scatter-Gather for Parallel Queries

In my previous post on Temporal, we explored the Saga pattern for distributed transactions with compensating actions. Now let’s look at another classic from Enterprise Integration Patterns: Scatter-Gather.

The Pattern

When you need to query multiple sources and aggregate results, the Scatter-Gather pattern is your friend. Send requests to multiple services in parallel (scatter), wait for responses, then combine them (gather).

In Night City, we query data brokers to find the best deal for moving sensitive information.

The Domain: Data Courier Services

Each broker has different characteristics:

The Workflow

First, configure the activities with retry policies. Since brokers are flaky, we want quick retries but don’t want to wait forever:

const {
  getAfterlifeQuote,
  getNetWatchBlackMarketQuote,
  getArasakaServicesQuote,
  getVoodooBoysQuote,
  getMilitechQuote,
  aggregateDataBrokerQuotes
} = proxyActivities<typeof activities>({
  startToCloseTimeout: '15 seconds',
  retry: {
    initialInterval: '500ms',
    backoffCoefficient: 2,
    maximumAttempts: 2,  // Quick retry, then move on
    maximumInterval: '5 seconds'
  }
});

The workflow itself builds an array of activity calls (without awaiting) and then uses Promise.all():

export async function dataBrokerScatterGather(
  request: DataCourierRequest
): Promise<DataBrokerAggregation> {
  // SCATTER: Build array of activity calls
  const brokerPromises = [
    safeQuote('Afterlife', () => getAfterlifeQuote(request)),
    safeQuote('NetWatch', () => getNetWatchBlackMarketQuote(request)),
    safeQuote('Arasaka', () => getArasakaServicesQuote(request)),
    safeQuote('Voodoo Boys', () => getVoodooBoysQuote(request)),
    safeQuote('Militech', () => getMilitechQuote(request)),
  ];

  // Execute all queries simultaneously
  const quotes = await Promise.all(brokerPromises);

  // GATHER: Aggregate and analyze
  return await aggregateDataBrokerQuotes(request, quotes);
}

// Wrapper that returns null instead of throwing
async function safeQuote<T>(
  brokerName: string,
  quoteFn: () => Promise<T | null>
): Promise<T | null> {
  try {
    return await quoteFn();
  } catch (error) {
    console.log(`[${brokerName}] Error: ${error}`);
    return null;  // Broker didn't respond - continue with others
  }
}

Full code: data-broker-scatter-gather.ts

How Temporal Handles This

The magic here is in how Temporal’s TypeScript SDK implements proxyActivities(). When you call an activity function like getAfterlifeQuote(request), the SDK doesn’t execute the activity immediately. Instead, it returns a Promise and pushes a scheduleActivity command to an internal queue. Looking at the SDK source, when you call an activity:

// Simplified from sdk-typescript workflow.ts
function scheduleActivityNextHandler({ options, args, seq, activityType }) {
  return new Promise((resolve, reject) => {
    activator.pushCommand({
      scheduleActivity: {
        seq,
        activityType,
        arguments: toPayloads(...args),
        retryPolicy: compileRetryPolicy(options.retry),
        startToCloseTimeout: msOptionalToTs(options.startToCloseTimeout),
        // ... other options
      }
    });
    activator.completions.activity.set(seq, { resolve, reject });
  });
}

What we find here is that the activities are scheduled lazily. The workflow builds a batch of commands without blocking. When you await Promise.all(brokerPromises), all five scheduleActivity commands get sent to the Temporal server in a single workflow task completion.

Task Queue Dispatch

The Temporal server places each activity task in the task queue. Workers poll this queue via synchronous RPC, picking up tasks only when they have capacity. This pull-based model means:

• Load balancing: Any worker listening to the queue can pick up any task
• No overload: Workers only request work when ready
• Horizontal scaling: Add more workers to process more activities in parallel

Temporal supports eager dispatch: when a worker completes a workflow task that schedules activities on the same task queue, the server hands those activity tasks directly back to the worker instead of going through the matching service. This is enabled by default via the allowEagerDispatch option and reduces latency for collocated workflows and activities.

Per-Activity Guarantees

Each of the five broker activities operates independently:

The safeQuote wrapper catches failures after retries are exhausted. If NetWatch times out twice, we get null for that broker but still have quotes from the other four. This is possible because Promise.all() works exactly as it does in standard JavaScript. Temporal’s workflow runtime supports all Promise static methods natively.

Why Not Promise.allSettled()?

You could use Promise.allSettled() instead of the safeQuote wrapper. The tradeoff: allSettled always waits for all promises, even if one fails early. With safeQuote wrapping each activity, failures are already converted to successful null results, so Promise.all() gives equivalent semantics with slightly cleaner result types.

In the Temporal UI, you’ll see all five activities scheduled at the same time:

The workflow history shows all five broker queries dispatched simultaneously. Each has its own ActivityTaskScheduled event at nearly the same timestamp.

The timeline visualization makes it even clearer:

Activities run in parallel, completing independently. Total time is roughly the slowest broker, not the sum of all.

If each broker takes 2 seconds, parallel execution takes ~2 seconds total instead of 10. And if one broker fails after retries, the others continue unaffected.

Sample Output

═══════════════════════════════════════════════════════════════════════
DATA BROKER SCATTER-GATHER
Package: PKG-JOHNNY-2023 | 80GB | military encryption | HOT
═══════════════════════════════════════════════════════════════════════

▶ SCATTER: Querying data broker network...
[AFTERLIFE] Quote: €$8,240, 36h delivery, 95% success
[NETWATCH BLACK MARKET] Connection terminated unexpectedly. Trace detected.
[ARASAKA SERVICES] Quote: €$12,480, 72h delivery, 98% success
[VOODOO BOYS] Quote: €$3,920, 28h delivery, 88% success
[MILITECH ACQ] Quote: €$5,880, 16h delivery, 85% success

✓ SCATTER complete: 4 responded, 1 unavailable

▶ GATHER: Aggregating broker responses...

┌─────────────────────────────────┬──────────┬──────────┬─────────┐
│ Broker                          │ Price    │ Delivery │ Success │
├─────────────────────────────────┼──────────┼──────────┼─────────┤
│ Voodoo Boys Data Haven          │ €$3920   │ 28h      │ 88%     │
│ Militech Acquisitions           │ €$5880   │ 16h      │ 85%     │
│ Afterlife Connections           │ €$8240   │ 36h      │ 95%     │
│ Arasaka External Services       │ €$12480  │ 72h      │ 98%     │
└─────────────────────────────────┴──────────┴──────────┴─────────┘

🏆 RECOMMENDATIONS:
  💰 Best Price: Voodoo Boys Data Haven (€$3920)
  ⚡ Fastest: Militech Acquisitions (16h)
  🎯 Most Reliable: Arasaka External Services (98%)
  ⭐ Overall Best: Voodoo Boys Data Haven

What Happens When a Broker Fails?

Understanding failure modes is crucial. There are two distinct failure types:

Type 1: Graceful “No Quote” (Activity Returns null)

Each broker has its own personality. NetWatch has a 15% chance of terminating the connection (it’s probably a sting):

// In getNetWatchBlackMarketQuote activity
if (Math.random() < 0.15) {
  console.log(`[NETWATCH] Connection terminated. Trace detected.`);
  return null;  // Activity succeeds but returns null
}

When an activity returns null, it’s a successful completion. Temporal records ActivityTaskCompleted in the history. No retries occur because there’s no error.

Type 2: Failures That Trigger Retries

If an activity throws an Error, Temporal’s worker catches it and applies the retry policy:

// Example: Network timeout throws
async function getArasakaServicesQuote(request: DataCourierRequest) {
  const response = await fetch('https://arasaka.corp/quotes', { ... });
  if (!response.ok) {
    throw new Error('Arasaka services unavailable');  // Temporal will retry
  }
  return response.json();
}

The retry mechanism is built into the SDK. Looking at ActivityOptions, the retry field accepts a RetryPolicy with:

• initialInterval: Wait time before first retry (500ms)
• backoffCoefficient: Multiplier for subsequent retries (2x)
• maximumAttempts: Total attempts including the first (2)
• maximumInterval: Cap on retry delay (5s)

After maximumAttempts is exhausted, the error propagates to the workflow. That’s where safeQuote catches it:

async function safeQuote<T>(brokerName: string, quoteFn: () => Promise<T | null>) {
  try {
    return await quoteFn();
  } catch (error) {
    // All retries exhausted - activity failed permanently
    console.log(`[${brokerName}] Error after retries: ${error}`);
    return null;  // Convert failure to partial success
  }
}

Non-Retryable Errors

Some errors shouldn’t be retried. For example, authentication failures or invalid requests. Use ApplicationFailure.nonRetryable() to signal this:

import { ApplicationFailure } from '@temporalio/common';

async function getVoodooBoysQuote(request: DataCourierRequest) {
  if (!request.dataPackage.classification) {
    throw ApplicationFailure.nonRetryable(
      'Missing data classification',
      'VALIDATION_ERROR'
    );
  }
  // ... continue with quote
}

This error bypasses the retry policy entirely. It’s immediately reported as a failure, and safeQuote converts it to null.

The Result

Either way, the workflow continues with partial results. The Scatter-Gather pattern explicitly tolerates broker unavailability. You get quotes from whoever responds, and make the best decision with available information.

Further Reading

• Temporal TypeScript SDK: workflow namespace - Full API reference for workflow functions
• Activity Execution - How activities are scheduled and executed
• Task Queues - The pull-based dispatch mechanism
• Retry Policies - Configuring automatic retries
• Detecting Activity Failures - Timeout types and failure detection
• SDK Source: workflow.ts - See scheduleActivityNextHandler and proxyActivities implementation
• Community Discussion: Promise.all in workflows - Confirmation that all Promise static methods work natively

Try It Yourself

git clone https://github.com/jamescarr/night-city-services.git
cd night-city-services
docker compose up -d
pnpm install
pnpm run worker  # Terminal 1
pnpm run scatter # Terminal 2

Up Next

In the next post, we’ll explore the Process Manager pattern, a state machine that responds to external signals. We’ll coordinate a heist against Arasaka Tower, with the ability to abort mid-operation when things go sideways.