How to stream to an HTTP response

This article is based on Node v16.14.0.

A Node.js stream can help process large files, larger than the free memory of your computer, since it processes the data in small chunks.

💰 The Pragmatic Programmer: journey to mastery. 💰 One of the best books in software development, sold over 200,000 times.

Streams are a built-in feature in Node.js and represent asynchronous flow of data. This article explains how to stream a large file to an HTTP response in Node.js.

Streams in Node.js

• What is a Stream in Node.js?
• Connect streams with the pipe method
• Handle stream errors
• Connect streams with the pipeline method
• How to use streams to ETL data
• Stream to an HTTP response (this article)

Send a stream back to the client as a response to an HTTP request

This article is using Node v16.14.0 and ExpressJS 4.17.2.

In previous articles I have covered a lot of basic stream handling and theoretical background. Now, let's look at some implementation. In this tutorial we are going to use streams to efficiently send a file as an HTTP response and download it.

The big advantage of streams is that you can process large files (bigger than the available memory). In general, reading a large file into memory is an inefficient use of resources.

We are going to use ExpressJS with some CSV sample data from a previous article, How to use streams to ETL data.

Overview

1. Initialize project and install dependencies
2. Create a readable stream from a file
3. Error handling and set headers for downloading file

1. Initialize project and install dependencies

We start with creating a folder for the project

mkdir streams-http
cd streams-http

We are going to use npm packages, hence, we have to initialize the project to get a package.json Initialize empty project to install dependencies, add -y flag to agree to everything.

npm init -y

Install ExpressJS.

npm i express

Create a folder for sample data and add the CSV data.

mkdir data
cd data
touch sample-data.csv

Copy all sample data into sample-data.csv and save it. Use copy+paste or fs.writeFile in the REPL or with the -p flag in the terminal.

id,firstName,lastName,email,email2,randomized
100,Jobi,Taam,[email protected],[email protected],Z lsmDLjL
101,Dacia,Elephus,[email protected],[email protected],Za jfPaJof
102,Arlina,Bibi,[email protected],[email protected],zmzlfER
103,Lindie,Torray,[email protected],[email protected],ibVggFEh
104,Modestia,Leonard,[email protected],[email protected]," Tit KCrdh"
105,Karlee,Cornelia,[email protected],[email protected],PkQCUXzq
106,Netty,Travax,[email protected],[email protected],psJKWDBrXm
107,Dede,Romelda,[email protected],[email protected],heUrfT
108,Sissy,Crudden,[email protected],[email protected],cDJxC
109,Sherrie,Sekofski,[email protected],[email protected],dvYHUJ
110,Sarette,Maryanne,[email protected],[email protected],rskGIJNF
111,Selia,Waite,[email protected],[email protected],DOPBe
112,Karly,Tjon,[email protected],[email protected],zzef nCMVL
113,Sherrie,Berriman,[email protected],[email protected],rQqmjw
114,Nadine,Greenwald,[email protected],[email protected],JZsmKafeIf
115,Antonietta,Gino,[email protected],[email protected],IyuCBqwlj
116,June,Dorothy,[email protected],[email protected],vyCTyOjt
117,Belva,Merriott,[email protected],[email protected],MwwiGEjDfR
118,Robinia,Hollingsworth,[email protected],[email protected],wCaIu
119,Dorthy,Pozzy,[email protected],[email protected],fmWOUCIM
120,Barbi,Buffum,[email protected],[email protected],VOZEKSqrZa
121,Priscilla,Hourigan,[email protected],[email protected],XouVGeWwJ
122,Tarra,Hunfredo,[email protected],[email protected],NVzIduxd
123,Madalyn,Westphal,[email protected],[email protected],XIDAOx
124,Ruthe,McAdams,[email protected],[email protected],iwVelLKZH
125,Maryellen,Brotherson,[email protected],[email protected],nfoiVBjjqw
126,Shirlee,Mike,[email protected],[email protected],MnTkBSFDfo
127,Orsola,Giule,[email protected],[email protected],VPrfEYJi
128,Linzy,Bennie,[email protected],[email protected],ZHctp
129,Vanessa,Cohdwell,[email protected],[email protected],RvUcbJihHf
130,Jaclyn,Salvidor,[email protected],[email protected],gbbIxz
131,Mildrid,Pettiford,[email protected],[email protected],snyeV
132,Carol-Jean,Eliathas,[email protected],[email protected],EAAjYHiij
133,Susette,Ogren,[email protected],[email protected]," BhYgr"
134,Farrah,Suanne,[email protected],[email protected],hYZbZIc
135,Cissiee,Idelia,[email protected],[email protected],PNuxbvjx
136,Alleen,Clara,[email protected],[email protected],YkonJWtV
137,Merry,Letsou,[email protected],[email protected],sLfCumcwco
138,Fanny,Clywd,[email protected],[email protected],Go kx
139,Trixi,Pascia,[email protected],[email protected],lipLcqRAHr
140,Sandie,Quinn,[email protected],[email protected],KrGazhI
141,Dania,Wenda,[email protected],[email protected],CXzs kDv
142,Kellen,Vivle,[email protected],[email protected],RrKPYqq
143,Jany,Whittaker,[email protected],[email protected],XAIufn
144,Lusa,Fillbert,[email protected],[email protected],FBFQnPm
145,Farrah,Edee,[email protected],[email protected],TrCwKb
146,Felice,Peonir,[email protected],[email protected],YtVZywf
147,Starla,Juan,[email protected],[email protected],aUTvjVNyw
148,Briney,Elvyn,[email protected],[email protected],tCEvgeUbwF
149,Marcelline,Ricarda,[email protected],[email protected],sDwIlLckbd
150,Mureil,Rubie,[email protected],[email protected],HbcfbKd
151,Nollie,Dudley,[email protected],[email protected],EzjjrNwVUm
152,Yolane,Melony,[email protected],[email protected],wfqSgpgL
153,Brena,Reidar,[email protected],[email protected],iTlvaS
154,Glenda,Sabella,[email protected],[email protected],zzaWxeI
155,Paola,Virgin,[email protected],[email protected],gJO hXTWZl
156,Aryn,Erich,[email protected],[email protected],qUoLwH
157,Tiffie,Borrell,[email protected],[email protected],cIYuVMHwF
158,Anestassia,Daniele,[email protected],[email protected],JsDbQbc
159,Ira,Glovsky,[email protected],[email protected],zKITnYXyhC
160,Sara-Ann,Dannye,[email protected],[email protected],wPClmU
161,Modestia,Zina,[email protected],[email protected],YRwcMqPK
162,Kelly,Poll,[email protected],[email protected],zgklmO
163,Ernesta,Swanhildas,[email protected],[email protected],tWafP
164,Giustina,Erminia,[email protected],[email protected],XgOKKAps
165,Jerry,Kravits,[email protected],[email protected],olzBzS
166,Magdalena,Khorma,[email protected],[email protected],BBKPB
167,Lory,Pacorro,[email protected],[email protected],YmWQB
168,Carilyn,Ethban,[email protected],[email protected],KUXenrJh
169,Tierney,Swigart,[email protected],[email protected],iQCQJ
170,Beverley,Stacy,[email protected],[email protected],NMrS Zpa f
171,Ida,Dex,[email protected],[email protected],hiIgOCxNg
172,Sam,Hieronymus,[email protected],[email protected],dLSkVe
173,Lonnie,Colyer,[email protected],[email protected],ZeDosRy
174,Rori,Ethban,[email protected],[email protected],SXFZQmX
175,Lelah,Niles,[email protected],[email protected],NwxvCXeszl
176,Kathi,Hepsibah,[email protected],[email protected],SOcAOSn
177,Dominga,Cyrie,[email protected],[email protected],IkjDyuqK
178,Pearline,Bakerman,[email protected],[email protected],vHVCkQ
179,Selma,Gillan,[email protected],[email protected],hSZgpBNsw
180,Bernardine,Muriel,[email protected],[email protected],AnSDTDa U
181,Ermengarde,Hollingsworth,[email protected],[email protected],IYQZ Nmv
182,Marguerite,Newell,[email protected],[email protected],kSaD uaHH
183,Albertina,Nisbet,[email protected],[email protected],Y jHyluB
184,Chere,Torray,[email protected],[email protected],loElYdo
185,Vevay,O'Neill,Vevay.O'[email protected],Vevay.O'[email protected],uLZSdatVn
186,Ann-Marie,Gladstone,[email protected],[email protected],fwKlEksI
187,Donnie,Lymann,[email protected],[email protected],deBrqXyyjf
188,Myriam,Posner,[email protected],[email protected],gEMZo
189,Dale,Pitt,[email protected],[email protected],OeMdG
190,Cindelyn,Thornburg,[email protected],[email protected],kvhFmKGoMZ
191,Maisey,Hertzfeld,[email protected],[email protected],OajjJ
192,Corina,Heisel,[email protected],[email protected],luoDJeHo
193,Susette,Marcellus,[email protected],[email protected],AXHtR AyV
194,Lanae,Sekofski,[email protected],[email protected],FgToedU
195,Linet,Beebe,[email protected],[email protected],DYGfRP
196,Emilia,Screens,[email protected],[email protected],LXUcleSs
197,Tierney,Avi,[email protected],[email protected],VegzbHH
198,Pollyanna,Thar,[email protected],[email protected],GjYeEGK
199,Darci,Elephus,[email protected],[email protected],DaQNdN

Create an index.js file (in root folder), which will be the main file for our code.

cd .. # if you are in the data folder
touch index.js

2. Create a readable stream from a file

First, we are going to create a basic express server, which listens on port 3000. Open index.js in your IDE and add the following code.

const express = require('express');

const app = express();
const PORT = 3000;

app.listen(PORT, () =>
  console.log(`Server listening on port ${PORT}`),
);

Start the node server with running node index.js in the project root folder. You should see Server listening on port 3000 in your terminal. Terminate the server with CTRL+C.

Let's make another route to download the csv file. Add a GET handler with the route /get-data to index.js.

app.get('/get-data', (req, res, next) => {
  // TBD
});

Now we have a route on which we are going to download the file in the end. We can proceed to create a readable stream to read the file. For creating a stream we have to import the fs module.

const fs = require('fs');

Create a stream to read file sample-data.csv.

app.get('/dl', (req, res, next) => {
  const fileStream = fs.createReadStream(
    `${__dirname}/data/sample-data.csv`,
  );
});

The constant fileStream represents the data stream from the file. This stream we directly pipe into the response.

app.get('/dl', (req, res, next) => {
  const fileStream = fs.createReadStream(
    `${__dirname}/data/sample-data.csv`,
  );
  fileStream.pipe(res);
});

Start the server again node index.js. And open a web browser with http://localhost:3000/get-data. You should see the csv file.

3. Error handling and set headers for downloading file

Now we are sending a file as a stream, but it should download. Let's make it happen. As always, we have to think of error handling first. What could go wrong? The file could not exist.

To handle this, we have to listen to the open event on the readStream to check, if the file exists and only pipe the stream if it does.

app.get('/get-data', (req, res, next) => {
  const fileStream = fs.createReadStream(
    `${__dirname}/data/sample-data.csv`,
  );
  fileStream.on('open', () => {
    fileStream.pipe(res);
  });
});

And if the file doesn't exist, we return the error as response.

app.get('/get-data', (req, res, next) => {
  const fileStream = fs.createReadStream(
    `${__dirname}/data/sample-data.csv`,
  );
  fileStream.on('open', () => {
    fileStream.pipe(res);
  });
  fileStream.on('error', err => {
    next(err);
  });
});

In most cases the pipeline method should be used, but pipeline destroys streams when an error occurs, and we would not be able to send a response back. Hence, for this use case manually error handling is acceptable.

At the moment the browser is displaying the file inline, it's loading the file in the browser. To tell the browser to download the file, we have to set:

• a Content-Type header on the response to specify what file we are sending, and
• the Content-Disposition header to attachment with a file name.

Express has a method for this attachment("FILENAME"). It sets the HTTP response Content-Disposition header field to “attachment”, and if a filename is given, then it sets the Content-Type based on the extension name via res.type(), and sets the Content-Disposition “filename=” parameter.

When using res.attachment('streamed-sample-data'), the content-type header will be set to text/csv and the content-disposition to the streamed-sample-data.csv.

app.get('/get-data', (req, res, next) => {
  const fileStream = fs.createReadStream(
    `${__dirname}/data/sample-data.csv`,
  );
  fileStream.on('open', () => {
    res.attachment('streamed-sample-data.csv');
    fileStream.pipe(res);
  });
  fileStream.on('error', err => {
    next(err);
  });
});

Restart your node server and go to http://localhost:3000/get-data. The file streamed-sample-data.csv should be downloaded.

TL;DR

• Error handling has to be done always, especially when working with streams.
• Error handling should be done in most cases with pipeline, only if the stream should not be destroyed, manually error handling is necessary.
• The express method attachment() is used to add Content-Type and Content-Disposition headers to a response.
• For the future, the Express framework has a method for sending files via stream sendFile().

Thanks for reading and if you have any questions, use the comment function or send me a message @mariokandut.

If you want to know more about Node, have a look at these Node Tutorials.

References (and Big thanks):

ExpressJS, HeyNode, Node.js - Streams, MDN - Streams, MDN - HTTP