payload icon indicating copy to clipboard operation
payload copied to clipboard

chore: adds filters

Open kendelljoseph opened this issue 7 months ago • 1 comments

kendelljoseph avatar May 30 '25 16:05 kendelljoseph

A few questions/notes:

  • Can we validate that this function call is actually doing anything? Feels like we should have a test. The library doesn't appear to have any types, which makes me a little more uneasy.
  • From reading, looks like this lib throws an error in the scenario where the URL is blocked. We should catch and rethrow an APIError, so it is handled nicely.

denolfe avatar May 30 '25 18:05 denolfe

Installing node-fetch saddens me deeply. But it indeed doesn't seem possible to do this using node-native fetch without installing undici, which is huge. See:

  • https://github.com/nodejs/node/issues/48977
  • https://github.com/nodejs/undici/discussions/2371

I don't know if there is a library that's leaner than node-fetch. I think we should add a comment next to the import explaining why it's needed, and a TODO: to re-visit this in the future once node-native fetch hopefully exposes an API to do this

AlessioGr avatar Jun 09 '25 22:06 AlessioGr

This PR has been updated to use undici, which is built into Node.js with a custom interceptor for url filtering. It was the only viable alternative that allowed both overriding agent/dispatch and also credentials: include.

denolfe avatar Jun 10 '25 19:06 denolfe

🚀 This is included in version v3.43.0

github-actions[bot] avatar Jun 16 '25 20:06 github-actions[bot]

After upgrading to v3.43.0, I can't execute any pnpm payload xxx command. And it seems this PR may have led to it. I may be wrong...

This is the issue:

> payload

/Users/xxxxxx/Documents/Projects/cnc-backend-cms/node_modules/.pnpm/[email protected]/node_modules/undici/lib/web/fetch/webidl.js:27
  return new TypeError(`${message.header}: ${message.message}`)
         ^

TypeError: TypeError: Illegal constructor
    at webidl.errors.exception (/Users/xxxxxx/Documents/Projects/cnc-backend-cms/node_modules/.pnpm/[email protected]/node_modules/undici/lib/web/fetch/webidl.js:27:10)
    at webidl.illegalConstructor (/Users/xxxxxx/Documents/Projects/cnc-backend-cms/node_modules/.pnpm/[email protected]/node_modules/undici/lib/web/fetch/webidl.js:81:23)
    at new CacheStorage (/Users/xxxxxx/Documents/Projects/cnc-backend-cms/node_modules/.pnpm/[email protected]/node_modules/undici/lib/web/cache/cachestorage.js:17:14)
    at Object.<anonymous> (/Users/xxxxxx/Documents/Projects/cnc-backend-cms/node_modules/.pnpm/[email protected]/node_modules/undici/index.js:144:25)
    at Module._compile (node:internal/modules/cjs/loader:1529:14)
    at Object.transformer (file:///Users/xxxxxx/Documents/Projects/cnc-backend-cms/node_modules/.pnpm/[email protected]/node_modules/tsx/dist/register-C3TE0KFF.mjs:2:1115)
    at Module.load (node:internal/modules/cjs/loader:1275:32)
    at Module._load (node:internal/modules/cjs/loader:1096:12)
    at Module.require (node:internal/modules/cjs/loader:1298:19)
    at require (node:internal/modules/helpers:182:18)

Node.js v20.19.0

@denolfe

sam-gab avatar Jun 17 '25 06:06 sam-gab

After upgrading to v3.43.0, I can't execute any pnpm payload xxx command. And it seems this PR may have led to it. I may be wrong...

Thanks @sam-gab . Let us take a look.

denolfe avatar Jun 17 '25 10:06 denolfe

Facing the same issue like @sam-gab.

khnn avatar Jun 19 '25 07:06 khnn

On version 3.43.0, when I try to run payload.jobs.run() from a custom CLI command, I get this error message:

26 |             throw new Error(`Blocked unsafe attempt to ${url}`);
27 |         }
28 |         return dispatch(opts, handler);
29 |     };
30 | };
31 | const safeDispatcher = new Agent().compose(ssrfFilterInterceptor);
                                        ^
TypeError: new Agent().compose is not a function. (In 'new Agent().compose(ssrfFilterInterceptor)', 'new Agent().compose' is undefined)
      at /home/node/app/node_modules/payload/dist/uploads/safeFetch.js:31:36

It's working fine 3.42.0.

bertrandkhe avatar Jun 19 '25 07:06 bertrandkhe

I also face similar issue when I try to edit Document Upload from using Drawer I get:

fetch("http://localhost:3000/api/research-documents/6853e79ec74ff5e6453aa6b7?depth=0&fallback-locale=null", {
  ---,
  "referrer": "http://localhost:3000/admin/collections/meetings/6853cd951fb29a5e4f860490",
  "referrerPolicy": "strict-origin-when-cross-origin",
  "body": "------WebKitFormBoundaryOBc6sCM7NuGdHyLh\r\nContent-Disposition: form-data; name=\"_payload\"\r\n\r\n{\"url\":\"http://localhost:3000/api/research-documents/file/Can_you_translate_this_document_to_hebre-1.pdf\",\"creator\":\"tsemachh\",\"filename\":\"Can_you_translate_this_document_to_hebre-1.pdf\",\"mimeType\":\"application/pdf\",\"filesize\":139204,\"companies\":[\"6848050ca0c94d84bb5555f5\",\"6848050ca0c94d84bb5555fd\"],\"createdAt\":\"2025-06-19T10:34:06.077Z\",\"updatedAt\":\"2025-06-19T10:34:06.077Z\",\"uploadedAt\":\"2025-06-19T10:33:41.865Z\",\"thumbnailURL\":null}\r\n------WebKitFormBoundaryOBc6sCM7NuGdHyLh--\r\n",
  "method": "PATCH",
  "mode": "cors",
  "credentials": "include"
});

I get next error in console:

{
  "level": 50,
  "time": 1750333310047,
  "user": "tsemachhadad",
  "hostname": "Tsemachs-MacBook-Pro.local",
  "env": "production",
  "err": {
    "type": "z",
    "message": "A problem occurred while uploading the file. Blocked unsafe attempt to http://localhost:3000/api/research-documents/file/Can_you_translate_this_document_to_hebre-1.pdf",
    "stack": [
      "z: A problem occurred while uploading the file. Blocked unsafe attempt to http://localhost:3000/api/research-documents/file/Can_you_translate_this_document_to_hebre-1.pdf",
      "    at rg (/Users/tsemachhadad/devemada/emda-dev/.next/server/chunks/940.js:188:79825)",
      "    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)",
      "    at async B (/Users/tsemachhadad/devemada/emda-dev/.next/server/chunks/940.js:190:27905)",
      "    at async handler (/Users/tsemachhadad/devemada/emda-dev/.next/server/chunks/940.js:413:604780)",
      "    at async g (/Users/tsemachhadad/devemada/emda-dev/.next/server/chunks/6109.js:1:4473)",
      "    at async /Users/tsemachhadad/devemada/emda-dev/.next/server/chunks/6109.js:1:7150",
      "    at async to.do (/Users/tsemachhadad/devemada/emda-dev/node_modules/next/dist/compiled/next-server/app-route.runtime.prod.js:18:18605)",
      "    at async to.handle (/Users/tsemachhadad/devemada/emda-dev/node_modules/next/dist/compiled/next-server/app-route.runtime.prod.js:18:23632)",
      "    at async doRender (/Users/tsemachhadad/devemada/emda-dev/node_modules/next/dist/server/base-server.js:1513:42)",
      "    at async NextNodeServer.renderToResponseWithComponentsImpl (/Users/tsemachhadad/devemada/emda-dev/node_modules/next/dist/server/base-server.js:1915:28)"
    ],
    "data": null,
    "isOperational": true,
    "isPublic": false,
    "status": 500,
    "name": "z"
  },
  "msg": "A problem occurred while uploading the file. Blocked unsafe attempt to http://localhost:3000/api/research-documents/file/Can_you_translate_this_document_to_hebre-1.pdf"
}

This happens also on previous payload versions at leat 3.39 , didn't check backwards though @denolfe - see the mentioned issue it's related as well , it's actually blocking both localhost and central env on this flow

tsemachh avatar Jun 19 '25 11:06 tsemachh

On version 3.43.0, when I try to run payload.jobs.run() from a custom CLI command, I get this error message:

26 |             throw new Error(`Blocked unsafe attempt to ${url}`);
27 |         }
28 |         return dispatch(opts, handler);
29 |     };
30 | };
31 | const safeDispatcher = new Agent().compose(ssrfFilterInterceptor);
                                        ^
TypeError: new Agent().compose is not a function. (In 'new Agent().compose(ssrfFilterInterceptor)', 'new Agent().compose' is undefined)
      at /home/node/app/node_modules/payload/dist/uploads/safeFetch.js:31:36

It's working fine 3.42.0.

same issue using next custom server, working fine in 3.42.0 for me

geminigeek avatar Jun 21 '25 12:06 geminigeek

Facing the same issue like @sam-gab.

Hi @khnn, noticed you were the only one who seemed to be facing the same issue I was.

Just an update, I realized switching from commonJs to ES Modules by adding "type": "module", to the package.json solved the issue. I was sticking with commonjs prior to v3.43.0 because I was using some packages that don't fully support ES Modules. cc: @denolfe

sam-gab avatar Jun 27 '25 05:06 sam-gab

Facing the same issue like @sam-gab.

Hi @khnn, noticed you were the only one who seemed to be facing the same issue I was.

Just an update, I realized switching from commonJs to ES Modules by adding "type": "module", to the package.json solved the issue.

I was sticking with commonjs prior to v3.43.0 because I was using some packages that don't fully support ES Modules.

cc: @denolfe

@sam-gab Thanks for pointing that out. I'll try that.

khnn avatar Jun 27 '25 13:06 khnn

I'm having the same issue as @geminigeek. Are you using Bun?

AlTavares avatar Jul 01 '25 17:07 AlTavares

I'm having the same issue as @geminigeek. Are you using Bun?

yes, what version of payload you have? i am stuck on 3.38 , waiting for a PR, you can try with latest payload!

geminigeek avatar Jul 01 '25 18:07 geminigeek

3.42 is the latest one I got working. This seems related.

AlTavares avatar Jul 01 '25 18:07 AlTavares

3.42 is the latest one I got working. This seems related.

i created a PR to temporarily fix this issue https://github.com/payloadcms/payload/pull/13032

geminigeek avatar Jul 03 '25 17:07 geminigeek

As of version 3.49.1, I still can't run any payload command via npm. Getting the same "TypeError: Illegal constructor" message

UPDATE: I was able to get it working by adding "type": "module" to my package.json and converting a config file to module syntax.

cameronrdecker avatar Jul 30 '25 15:07 cameronrdecker

Same situation here. Any plans for a fix, besides switching from 'commonjs' to 'module'?

BoavidaGuerra avatar Sep 03 '25 08:09 BoavidaGuerra