Skip to content

Revise backpressure examples in streams documentation#90

Open
baskakov wants to merge 1 commit into
nodejs:mainfrom
baskakov:patch-1
Open

Revise backpressure examples in streams documentation#90
baskakov wants to merge 1 commit into
nodejs:mainfrom
baskakov:patch-1

Conversation

@baskakov

Copy link
Copy Markdown

Discussed here: https://stackoverflow.com/questions/79821132/node-js-back-pressure-documentation-whats-the-relevance-of-their-example Original backpressure example contains following oddities: 1) Incorrect zip example 2) Incorrect assumption that zip console command loads full file into memory. Updated examples to demonstrate file compression using Node.js streams with backpressure handling.

Discussed here: https://stackoverflow.com/questions/79821132/node-js-back-pressure-documentation-whats-the-relevance-of-their-example
Original backpressure example contains following oddities:
1) Incorrect zip example 2) Incorrect assumption that zip console command loads full file into memory.
Updated examples to demonstrate file compression using Node.js streams with backpressure handling.

Signed-off-by: Dmitry Baskakov <dmitry@bask.ws>
Copilot AI review requested due to automatic review settings June 14, 2026 09:26
@vercel

vercel Bot commented Jun 14, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
nodejs-learn Ready Ready Preview Jun 14, 2026 9:26am

Request Review

@cursor

cursor Bot commented Jun 14, 2026

Copy link
Copy Markdown

PR Summary

Low Risk
Documentation-only edits to examples and explanatory text; no runtime or API changes.

Overview
Replaces the opening backpressure demo in backpressuring-in-streams.md so it no longer contrasts a shell zip run with a Node gzip pipeline.

The section now walks through three Node-only steps on the same ~9 GB file: readFileSync + gzipSync (buffer/heap limits), manual data handlers that ignore write() backpressure (unbounded buffers / OOM), then inp.pipe(gzip).pipe(out) with prose on pause, false from write(), and 'drain'.

The removed narrative that compared a “corrupt” zip output to a successful stream decompress is gone; the existing pipe / pump / pipeline notes are unchanged.

Reviewed by Cursor Bugbot for commit 7ce0776. Bugbot is set up for automated code reviews on this repo. Configure here.

@github-actions

Copy link
Copy Markdown

👋 Codeowner Review Request

The following codeowners have been identified for the changed files:

Team reviewers: @nodejs/streams

Please review the changes when you have a chance. Thank you! 🙏

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Updates the backpressure module to better illustrate memory/throughput implications of buffering vs streaming in Node.js compression examples.

Changes:

  • Replaces the zip(1) example with fs.readFileSync + zlib.gzipSync examples (CJS + ESM).
  • Adds an explicit “streams without backpressure” example plus an explanation of why it can OOM.
  • Removes the prior note about comparing resulting archives.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.


In one scenario, we will take a large file (approximately ~9 GB) and compress it
using the familiar [`zip(1)`][] tool.
In one scenario, we will read a large file (approximately ~9 GB) using `fs.readFileSync`: and compress it
writeFileSync('The.Matrix.1080p.mkv.gz', compressed);
```

This fails on two separate limits, whichever you hit first: the buffer size cap or heap exhaustion. Lets rewite it another way, using Node.js' [`Stream`][] but without backpressure:
While that will take a few minutes to complete, in another shell we may run
a script that takes Node.js' module [`zlib`][], that wraps around another
compression tool, [`gzip(1)`][].
Neither of two writes respects backpressure. Stage 1 keeps pushing into gzip even after gzip.write() returns false, and stage 2 keeps pushing into out even after out.write() returns false. Both internal buffers can grow without bound, so this is very prone to running out of memory. On a large compressible file both numbers climb fast and it heads for `JavaScript heap out of memory`.
inp.on('data', (chunk) => gzip.write(chunk));
inp.on('end', () => gzip.end());
gzip.on('data', (chunk) => out.write(chunk));
gzip.on('end', () => out.end());

```cjs
const { createReadStream, createWriteStream } = require('node:fs');
const gzip = require('node:zlib').createGzip();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
const gzip = require('node:zlib').createGzip();
const { createGzip } = require('node:zlib');
const gzip = createGzip();

While that will take a few minutes to complete, in another shell we may run
a script that takes Node.js' module [`zlib`][], that wraps around another
compression tool, [`gzip(1)`][].
Neither of two writes respects backpressure. Stage 1 keeps pushing into gzip even after gzip.write() returns false, and stage 2 keeps pushing into out even after out.write() returns false. Both internal buffers can grow without bound, so this is very prone to running out of memory. On a large compressible file both numbers climb fast and it heads for `JavaScript heap out of memory`.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Neither of two writes respects backpressure. Stage 1 keeps pushing into gzip even after gzip.write() returns false, and stage 2 keeps pushing into out even after out.write() returns false. Both internal buffers can grow without bound, so this is very prone to running out of memory. On a large compressible file both numbers climb fast and it heads for `JavaScript heap out of memory`.
Neither of two writes respects backpressure. Stage 1 keeps pushing into gzip even after `gzip.write()` returns false, and stage 2 keeps pushing into out even after `out.write()` returns false. Both internal buffers can grow without bound, so this is very prone to running out of memory. On a large compressible file both numbers climb fast and it heads for `JavaScript heap out of memory`.

compression tool, [`gzip(1)`][].
Neither of two writes respects backpressure. Stage 1 keeps pushing into gzip even after gzip.write() returns false, and stage 2 keeps pushing into out even after out.write() returns false. Both internal buffers can grow without bound, so this is very prone to running out of memory. On a large compressible file both numbers climb fast and it heads for `JavaScript heap out of memory`.

To resolve this, we may use pipe, which pauses the read when write() returns false. When `gzip.write()` returns `false`, `pipe` calls `pause()` on the read stream, halting disk reads. Once gzip works through its backlog and the buffer empties, it emits a `'drain'` event, and `pipe` calls `resume()` to start reading again.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
To resolve this, we may use pipe, which pauses the read when write() returns false. When `gzip.write()` returns `false`, `pipe` calls `pause()` on the read stream, halting disk reads. Once gzip works through its backlog and the buffer empties, it emits a `'drain'` event, and `pipe` calls `resume()` to start reading again.
To resolve this, we may use pipe, which pauses the read when `write()` returns false. When `gzip.write()` returns `false`, `pipe` calls `pause()` on the read stream, halting disk reads. Once gzip works through its backlog and the buffer empties, it emits a `'drain'` event, and `pipe` calls `resume()` to start reading again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants