Module not found: Can't resolve 'fs' in 'node_modules/@aws-sdk/lib-storage/dist/es

[fanatic75]

Describe the bug

Angular project not building because of FS error, is there no way to upload big files to s3 via Angular 11 with v3 sdk ?

Your environment

Angular - 11, Win 10

SDK version number

3.20.0

Is the issue in the browser/Node.js/ReactNative?

Browser

Details of the browser/Node.js/ReactNative version

Paste output of npx envinfo --browsers or node -v or react-native -v v14.15.1

Steps to reproduce

 const uploadParams = {
      Bucket: this.bucketName,
      Key: `podcasts/${this.userInfo.identityId}/${file.name}`,
      Body: file
    };
     try {
      const parallelUploads3 = new Upload({
        client: this.s3,
        params: uploadParams,
      });

      parallelUploads3.on("httpUploadProgress",progressFunc);

      await parallelUploads3.done();

    } catch (err) {
      return console.error(err);
    } 

Observed behavior

Angular app crashes

Expected behavior

Use the library

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

duplicate of #2136

[grothem]

I was able to get it working by using this fix cross posted in webpack issues. I had to use the @angular-builders/custom-webpack package to make it work.

[grothem]

ahh scratch that. The build succeeds, but now it's failing at run time 😕

[fanatic75]

For now I have moved to v2 till this is sorted out.

[ajredniwja]

Hi @fanatic75 thanks for bringing this one, I wasn't able to reproduce it. This was initially because fs is not available in browser and the SDK tries to load it, but I made a fix to that. I only see a warning but no errors. Can you setup a repo with your example that duplicates the error?

[grothem]

@ajredniwja Here's a repo that reproduces the issue. I used Angular CLI to generate a new v11 project, then added @aws-sdk/lib-storage and @aws-sdk/client-s3, then just pasted in the example from the lib-storage repo.

[ajredniwja]

@grothem thanks, I'll try to work with it, would be some time.

[ajredniwja]

Hi there, it turns out to be a dependency issue just add:

  "browser": {
    "crypto": false,
    "fs": false,
    "path": false,
    "os": false,
    "net": false,
    "stream": false,
    "tls": false
  }

In your package.json and it should work as expected.

[fanatic75]

Hey @ajredniwja , I added this object in package.json but didn't work. Thanks

[ajredniwja]

@fanatic75, let me share my package.json

[ajredniwja]

{
  "name": "aws-lib-storage",
  "version": "0.0.0",
  "scripts": {
    "ng": "ng",
    "start": "ng serve",
    "build": "ng build",
    "test": "ng test",
    "lint": "ng lint",
    "e2e": "ng e2e"
  },
  "browser": {
    "crypto": false,
    "fs": false,
    "path": false,
    "os": false,
    "net": false,
    "stream": false,
    "tls": false
  },
  "private": true,

  "dependencies": {
    "@angular/animations": "~11.2.14",
    "@angular/common": "~11.2.14",
    "@angular/compiler": "~11.2.14",
    "@angular/core": "~11.2.14",
    "@angular/forms": "~11.2.14",
    "@angular/platform-browser": "~11.2.14",
    "@angular/platform-browser-dynamic": "~11.2.14",
    "@angular/router": "~11.2.14",
    "ng": "0.0.0",
    "rxjs": "~6.6.0",
    "tslib": "^2.0.0",
    "zone.js": "~0.11.3",
    "@aws-sdk/client-cognito-identity": "^3.15.0",
    "@aws-sdk/client-lex-runtime-service": "^3.15.0",
    "@aws-sdk/client-s3": "^3.21.0",
    "@aws-sdk/client-sqs": "^3.21.0",
    "@aws-sdk/credential-provider-cognito-identity": "^3.7.0",
    "@aws-sdk/lib-storage": "3.21.0",
    "@aws-sdk/s3-request-presigner": "^3.16.0",
    "@testing-library/jest-dom": "^5.11.4",
    "@testing-library/user-event": "^12.1.10",
    "axios": "^0.21.1",
    "fs": "0.0.1-security",
    "web-vitals": "^1.0.1"
  },
  "devDependencies": {
    "@angular-devkit/build-angular": "~0.1102.13",
    "@angular/cli": "~11.2.14",
    "@angular/compiler-cli": "~11.2.14",
    "@types/jasmine": "~3.6.0",
    "@types/node": "^12.11.1",
    "codelyzer": "^6.0.0",
    "jasmine-core": "~3.6.0",
    "jasmine-spec-reporter": "~5.0.0",
    "karma": "~6.1.0",
    "karma-chrome-launcher": "~3.1.0",
    "karma-coverage": "~2.0.3",
    "karma-jasmine": "~4.0.0",
    "karma-jasmine-html-reporter": "~1.5.0",
    "protractor": "~7.0.0",
    "ts-node": "~8.3.0",
    "tslint": "~6.1.0",
    "typescript": "~4.1.5"
  }
}

This is what I used, some of which was copied from the react project I had and the angular app ran perfectly uploading stuff on S3.

[grothem]

Thanks @ajredniwja, that did fix the build errors for me. Is this something that could be added to the @aws-sdk/lib-storage module package.json, instead of having the end users specify this config?

[ajredniwja]

@grothem yes, would need to find which package it is.

[ignlg]

After applying those fixes, it still fails with Angular 12, that uses Webpack v5.

This is the core of the package.json used:

{
  ...
  "dependencies": {
    "@angular/animations": "^12.1.3",
    "@angular/common": "^12.1.3",
    "@angular/compiler": "^12.1.3",
    "@angular/core": "^12.1.3",
    "@angular/forms": "^12.1.3",
    "@angular/localize": "^12.1.3",
    "@angular/platform-browser": "^12.1.3",
    "@angular/platform-browser-dynamic": "^12.1.3",
    "@angular/router": "^12.1.3",
    "@aws-sdk/client-s3": "^3.22.0",
    "@aws-sdk/lib-storage": "^3.23.0",
    "fs": "^0.0.1-security",
    "zone.js": "^0.11.3"
  },
  "devDependencies": {
    "@angular-devkit/build-angular": "~12.1.3",
    "@angular-devkit/core": "12.1.3",
    "@angular/cli": "^12.1.3",
    "@angular/compiler-cli": "^12.1.3",
    "@angular/language-service": "^12.1.3",
    "typescript": "4.3.5"
  },
  "browser": {
    "crypto": false,
    "fs": false,
    "net": false,
    "os": false,
    "path": false,
    "stream": false,
    "tls": false
  },
  ...
}

Also tsconfig is configured with:

    "lib": ["ES2020", "ES2019", "ESNext", "DOM"],

Any ideas?

[brett-vendia]

Failing in a React app using Next 11 (which is also Webpack 5), even after adding the "browser" entry

[grothem]

@ajredniwja just wondering if there was any update on this. I'm now running into issues using v2 of aws-sdk and angular 12 (see this issue)

[ajredniwja]

testing this now, working in react app using the latest packages, the test included uploading some stuff to S3

[ajredniwja]

The code I used:

export const s3Test = async () => {
    // debugger;

    const creds = fromCognitoIdentityPool({
        client: new CognitoIdentityClient({ region }),
        identityPoolId: idPool,
    });

    const client = new S3Client({
        region,
        credentials: credentials
    });

    const Key = `${Date.now()}-new-key`;
    let upload = new Upload({
        client,
        tags: [{
            Key: "my-tag-key",
            Value: "my-tag-value"
        }],
        params: { Key, Bucket,
            Body: "hello world!"
        }
    }); 

    upload.on("httpUploadProgress", (progress ) => {
        console.log(progress);
    });

    const uploadResult = await upload.done();
    
    // debugger;
    console.log("done uploadingg", uploadResult);

}

[ajredniwja]

For Angular using web-pack 5 it is using the node config to build which causes this error, which uses fs as seen here, but it needs to use the browser config as which doesn't use fs. I believe, it should be a web-pack issue since it is not able to load the correct config. Not sure how to achieve that in an Angular project but working on it, will discuss it with the team to find more on the issue.

[WIStudent]

A webpack team member explained the cause of this issue here: https://github.com/webpack/webpack/issues/13498#issuecomment-853815100

Basicly, the browser field inside the package.json is ignored because the module field takes precedence. Instead they recommend defining package exports with different taget environments.

[robec1982]

ahh scratch that. The build succeeds, but now it's failing at run time 😕

@grothem did you solve this. I'm experiencing the same issue. Webpack is creating the bundle with 3 WARNINGS (no errors) but the resulting bundle doesn't work. It launches an error like this:

webpack://orders-importer-bundle/./node_modules/amazon-sp-api/lib/endpoints.js?:9
    ...readdirSync(__dirname + '/resources').reduce((eps, ep) => {
       ^

TypeError: readdirSync is not a function

[grothem]

@robec1982 It's working for me now with the latest version of the sdk