How to correctly send to the server and save the buffer of the uploaded file?

Question

Express.js

Hello.
I've been struggling with a problem for many hours now:
I want to send a ZIP file from my computer using NodeJS. I send the file buffer using the PUT method. And the data is being transferred. However, the file that is written to the server does not match the size of the original file.

Here's how I send a file in NodeJS :

const buffer = fs.readFileSync(`../source/archive.zip`);
const options = {
    hostname: url.hostname,
    port: 443,
    path: url.pathname,
    method: 'PUT',
    headers: {
        'Content-Type': 'application/zip',
        'Content-Length': buffer.length
    }
};
const req = https.request(options, res => {
    res.on('error', err => console.log('ошибка:', err));
    res.on('end', () => console.log('отправлено'));
});

req.write(buffer);
req.end();

And this is how I receive the file on the server using NodeJS + Express :

router.put('/', (req, res, next) => {

    const pathToFile = path.join(__dirname, '..', 'public', 'uploads', 'archive.zip');

    let chunks = '';

    req.on('data', chunk => {
        chunks += chunk;
    });

    req.on('end', () => {
        fs.writeFile(pathToFile, chunks, { encoding: 'binary' } , (err) => {
            if (err) throw err;
            res.send('файл сохранён');
        });
    });

});

As I wrote above: the saved file is always different in size from the original. At the same time, if writeFileI specify the encoding in the function 'binary', then the size of the saved file is slightly smaller than the original one. And if I do not specify the encoding, then the size is approximately twice the size of the original. In both cases, the archive is broken and cannot be unzipped.

Tell me what am I doing wrong? I 've already rummaged through the entire Internet ...

Another thing that confuses me is that the meta-data of the file is not transferred with the data buffer. Or they are still sent, but I don’t know how to check it.

Thanks in advance!

Answer 1

Firstly, strings in JS are stored in memory as utf-16, which means that saving an arbitrary buffer to a string can break the data.
This is where your data breaks down:

let chunks = '';

req.on('data', chunk => {
    chunks += chunk;
});

Secondly, you should not read entire files into memory. Memory is not rubber, and one day it may simply not be enough.
Thirdly, there is no need to transfer bytes from one stream to another by hand, there is a pipe that does this much more efficiently.
Shipping:

const filePath = `../source/archive.zip`;
const reader = fs.createReadStream(filePath);
const options = {
    hostname: url.hostname,
    port: 443,
    path: url.pathname,
    method: 'PUT',
    headers: {
        'Content-Type': 'application/zip',
        'Content-Length': fs.statSync(filePath).size
    }
};
const req = https.request(options, res => {
    res.on('error', err => console.log('ошибка:', err));
    res.on('end', () => console.log('отправлено'));
});
reader.pipe(req);

Receipt:

router.put('/', (req, res, next) => {
    const pathToFile = path.join(__dirname, '../public/uploads/archive.zip');
    const writer = fs.createWriteStream(pathToFile);
    req.pipe(writer).once('error', next).once('finish', () => {
        res.send('файл сохранён');
    });
});