Content of form data does not persist leading to socket hang up

[sunknudsen]

Describe the bug

• Node.js version: v12.18.3
• OS & version: macOS Catalina 10.15.7

I am using the afterResponse hook to get access token when API returns 401 and retry request.

When using form-data, the content of _streams becomes [] when request is retried leading to socket hang up error.

Initial run:

_streams: [
  '----------------------------529798752574445155234154\r\n' +
    'Content-Disposition: form-data; name="title"\r\n' +
    '\r\n',
  'Big Buck Bunny',
  [Function: bound ],
  '----------------------------529798752574445155234154\r\n' +
    'Content-Disposition: form-data; name="description"\r\n' +
    '\r\n',
  'This is a test...',
  [Function: bound ]
],

Retry run:

_streams: [],

Actual behavior

The content of _streams is [].

Expected behavior

The content of _streams should persist between retries.

Code to reproduce

The following code is not reproducible "as-is" because it requires credentials, but illustrates the issue.

"use strict"

import got, { Options } from "got"
import formData from "form-data"

const test = async function () {
  const form = new formData()
  form.append("title", "Big Buck Bunny")
  form.append("description", "This is a test...")
  const initialOptions: Options = {
    headers: {
      authorization: "Bearer {redactedInvalidAccessToken}",
    },
    body: form,
    responseType: "json",
    throwHttpErrors: false,
  }
  console.log(initialOptions)
  const initialVideoResponse: any = await got.put(
    `https://peertube.sunknudsen.com/api/v1/videos/f79b7b72-817f-4ba1-9c58-d59d63d64366`,
    initialOptions
  )
  console.log(initialVideoResponse.statusCode, initialVideoResponse.body)
  const tokenResponse: any = await got.post(
    `https://peertube.sunknudsen.com/api/v1/users/token`,
    {
      form: {
        client_id: "{redactedClientId}",
        client_secret: "{redactedClientSecret}",
        refresh_token: "{redactedRefreshToken}",
        grant_type: "refresh_token",
      },
      responseType: "json",
      throwHttpErrors: false,
    }
  )
  console.log(tokenResponse.statusCode, tokenResponse.body)
  const retryOptions: Options = {
    headers: {
      authorization: `Bearer ${tokenResponse.body.access_token}`,
    },
    body: form,
    responseType: "json",
    throwHttpErrors: false,
  }
  console.log(retryOptions)
  // The following request hangs and eventuality throws socket hang up error
  const retryVideoResponse: any = await got.put(
    `https://peertube.sunknudsen.com/api/v1/videos/f79b7b72-817f-4ba1-9c58-d59d63d64366`,
    retryOptions
  )
  console.log(retryVideoResponse.statusCode, retryVideoResponse.body)
}

test()

Checklist

• [ x ] I have read the documentation.
• [ x ] I have tried my code with the latest version of Node.js and Got.

[Giotino]

This is an expected behavior.
FormData is a stream, when it's read (and finished) you cannot get your bytes back.

If you want to make a request with the same FormData body you should recreate it.

Example:

const got = require('got');
const FormData = require('form-data');

const genBody = () => {
  const form = new FormData();
  form.append("title", "Big Buck Bunny");
  form.append("description", "This is a test...");
  return form;
};

const firstRequest = got('URL', {
  body: genBody()
});

const secondRequest = got('URL', {
  body: genBody()
});

[sunknudsen]

Hi @Giotino, what about when using hooks or retries? The same request is replayed right?

[Giotino]

No, it still not works because the form data can be read only one time.

If you're uploading a lot of data, such as files, you should do something like this:

const got = require('got');
const FormData = require('form-data');

const genBody = () => {
  const form = new FormData();
  form.append('A', 'B');
  return form;
};

got.post('URL', {
  body: genBody(),
  hooks: {
    beforeRetry: [
      options => {
        const form = genBody();
        options.body = form;
        options.headers['content-type'] = `multipart/form-data; boundary=${form.getBoundary()}`;
      }
    ]
  }
});

Or, if you're sending a little form (like your example) you can simply do this (because it's using a buffer as body):

const got = require('got');
const FormData = require('form-data');
const getStream = require('get-stream');

const form = new FormData();
form.append('A', 'B');
const bufferBody = getStream(form);

got.post('URL', {
  body: bufferBody,
  headers: {
    'content-type': `multipart/form-data; boundary=${form.getBoundary()}`
  }
});

Both solution works fine in both cases, but, the first one is (IMO) overkill for small forms, and the second one is going to store the entire form (files included) in RAM before sending it (without taking advantage of the Node.js streams) which is not optimal if you're working with files.