Node.js Streams và Buffer

1. Why use Streams instead of reading entire files

• fs.readFile() reads the entire file content into RAM at once before returning. For very large files (e.g. 5GB), Node must allocate an equivalent amount of RAM, easily causing JavaScript heap out of memory or slowing down the whole server if RAM is insufficient.
• fs.createReadStream() reads the file in small chunks (default 64KB), processing each chunk before reading the next, without holding the entire file content in RAM at once.
• The second advantage of streams (besides saving RAM): users receive the first byte almost immediately, without having to wait for the server to read the entire file before starting to send. If the user cancels midway, the server can also stop reading early.

2. Backpressure, the mechanism to regulate read/write speed

• When .pipe() a readable stream (reading a file) to a writable stream (e.g. response), if the write speed is slower than the read speed, data risks piling up in RAM.
• Node solves this with the backpressure mechanism: the writable stream has an internal buffer with a size limit (highWaterMark, default 64KB). When .write() into a full buffer, the function returns false.
• .pipe() automatically listens to this false signal, pauses further reading from the readable stream, until the writable emits the drain event (buffer has free space) before continuing to read. Thanks to this, the read speed always self-regulates to match the write speed.
• If manually reading with stream.on('data', chunk => writable.write(chunk)) instead of using .pipe(), this automatic backpressure mechanism no longer works: the readable still emits data events continuously according to the disk read speed, regardless of whether the writable can keep up, and data can still pile up in RAM.
• If you really need to process each chunk manually (not just forwarding directly), you must write the logic yourself: check if .write() returns false then call stream.pause(), listen for drain to stream.resume(). Therefore, .pipe() (or pipeline() from the stream module, which handles errors better) is always recommended over writing by hand, unless custom logic is needed in between.