Test phrase passed even though Test cases failed

[andychoww]

Before opening, please confirm:

• [X] I have searched for duplicate or closed issues and discussions.
• [X] I have read the guide for submitting bug reports.
• [X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

JavaScript Framework

React

Amplify APIs

Not applicable

Amplify Categories

hosting

Environment information

I am using Jest as the Test runner

Describe the bug

The Jest test cases fail on Test Phrase in Amplify pipeline, but it still continue to proceed instead of stopping.

Expected behavior

The Test Phrase should be failed and stop proceeding to Deploy Phrase

Reproduction steps

1. commit the changes in branch and push to remote
2. The pipelines in Amplify runs
3. The Jest test cases fail in Test Phrase, but the pipeline still continues
4. The application deploys successfully

Code Snippet

// Put your code below this line.

Jest.config.ts

export default {
  preset: 'ts-jest',
  coverageProvider: 'v8',
  globals: {
    'ts-jest': {
      isolatedModules: true,
    },
  },
  testEnvironment: 'node',
  transform: {
    'node_modules/variables/.+\\.(j|t)sx?$': 'ts-jest',
  },
  transformIgnorePatterns: ['node_modules/(?!variables/.*)'],
};

amplify.yml

version: 2
frontend:
  phases:
    preBuild:
      commands:
        - npm install
    build:
      commands:
        - npm run build
  artifacts:
    baseDirectory: build
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*
test:
  phases:
    preTest:
      commands:
        - npm install
    test:
      commands:
        - 'npm test'
  artifacts:
    baseDirectory: ./
    configFilePath: '**/mochawesome.json'
    files:
      - '**/*'

Log output

// Put your logs below this line
2022-05-10T09:14:52.261Z [WARNING]: FAIL src/utils/formatters/formatters.test.ts
2022-05-10T09:14:52.286Z [WARNING]: ● formatters test › decimalPlacesRegex test › should return true - when the decimal of the string equals to decimalPlaces
                                    expect(received).toEqual(expected) // deep equality
                                    Expected: false
                                    Received: true
                                    28 |     it('should return true - when the decimal of the string equals to decimalPlaces', () => {
                                    29 |       const result = decimalPlacesRegex(DEFAULT_AMOUNT, DEFAULT_DECIMAL);
                                    > 30 |       expect(result).toEqual(false);
                                    |                      ^
                                    31 |     });
                                    32 |     it('should return false - when the decimal of the string is different from decimalPlaces', () => {
                                    33 |       const result = decimalPlacesRegex(DEFAULT_AMOUNT, 2);
                                    at Object.<anonymous> (src/utils/formatters/formatters.test.ts:30:22)
2022-05-10T09:14:52.301Z [WARNING]: Test Suites: 1 failed, 1 passed, 2 total
                                    Tests:       1 failed, 16 passed, 17 total
                                    Snapshots:   0 total
                                    Time:        1.309 s
                                    Ran all test suites.
2022-05-10T09:14:52.376Z [WARNING]: npm
2022-05-10T09:14:52.377Z [WARNING]: ERR! Test failed.  See above for more details.
2022-05-10T09:14:52.380Z [INFO]: # Completed phase: test
2022-05-10T09:14:52.380Z [INFO]: ## Tests completed successfully
2022-05-10T09:14:52.382Z [INFO]: # Starting test artifact upload process...
2022-05-10T09:14:59.382Z [INFO]: # Uploading build artifact 'testConfig.json'...
2022-05-10T09:14:59.950Z [INFO]: # Build artifact is: 6MB
2022-05-10T09:14:59.951Z [INFO]: # Uploading build artifact 'testArtifacts.zip'...
2022-05-10T09:15:00.172Z [INFO]: # Test artifact upload completed
2022-05-10T09:15:00.175Z [INFO]: No custom headers found.
2022-05-10T09:15:00.194Z [INFO]: # Starting build artifact upload process...
2022-05-10T09:15:00.667Z [INFO]: # Build artifact is: 4MB
2022-05-10T09:15:00.668Z [INFO]: # Uploading build artifact '__artifacts.zip'...
2022-05-10T09:15:00.686Z [INFO]: # Build artifact is: 4MB
2022-05-10T09:15:00.686Z [INFO]: # Uploading build artifact '__artifactsHash.zip'...
2022-05-10T09:15:00.849Z [INFO]: # Build artifact upload completed
2022-05-10T09:15:00.849Z [INFO]: # Starting environment caching...
2022-05-10T09:15:00.849Z [INFO]: # Uploading environment cache artifact...
2022-05-10T09:15:00.928Z [INFO]: # Environment caching completed
Terminating logging...

[chrisbonifacio]

Hi @andychoww 👋 thanks for raising this issue. This seems to be related to Amplify Hosting's CI/CD pipeline. I will transfer this over to the hosting team's repo for better support and feedback.

[ghost]

Hi @andychoww 👋🏽 can you try implementing the workaround suggested here please: https://github.com/aws-amplify/amplify-hosting/issues/2409#issuecomment-1027401956

[andychoww]

Hi @hloriana tried but the test phrase still passes

version: 4
frontend:
  phases:
    preBuild:
      commands:
        - npm install
    build:
      commands:
        - npm run build
  artifacts:
    baseDirectory: build
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*
test:
  phases:
    preTest:
      commands:
        - npm install
        - npm install pm2
    test:
      commands:
        - 'npm test'
    postTest:
      commands:
        - npx pm2 kill
  artifacts:
    baseDirectory: ./
    configFilePath: '**/mochawesome.json'
    files:
      - '**/*'

[andychoww]

I even tried below but still not working

version: 5
frontend:
  phases:
    preBuild:
      commands:
        - npm install
    build:
      commands:
        - npm run build
  artifacts:
    baseDirectory: build
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*
test:
  phases:
    preTest:
      commands:
        - npm install
        - npm install pm2
    test:
      commands:
        - CI=true npm test
    postTest:
      commands:
        - npx pm2 kill
  artifacts:
    baseDirectory: ./
    configFilePath: '**/mochawesome.json'
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*

[ghost]

@andychoww please add - npx pm2 start npm -- start under preTest like in this example for running Cypress tests:

test:
  phases:
    preTest:
      commands:
        - npm ci
        - npm install wait-on
        - npm install pm2
        - npm install [email protected] mochawesome mochawesome-merge mochawesome-report-generator
        - npx pm2 start npm -- start
        - 'npx wait-on --timeout 60 http://localhost:3000'
    test:
      commands:
        - 'npx cypress run --reporter mochawesome --reporter-options "reportDir=cypress/report/mochawesome-report,overwrite=false,html=false,json=true,timestamp=mmddyyyy_HHMMss"'
    postTest:
      commands:
        - npx mochawesome-merge cypress/report/mochawesome-report/mochawesome*.json > cypress/report/mochawesome.json
        - npx pm2 kill
  artifacts:
    baseDirectory: cypress
    configFilePath: '**/mochawesome.json'
    files:
      - '**/*.png'
      - '**/*.mp4'

[andychoww]

@hloriana I have added that in preTest but still not working

version: 6
frontend:
  phases:
    preBuild:
      commands:
        - npm install
    build:
      commands:
        - npm run build
  artifacts:
    baseDirectory: build
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*
test:
  phases:
    preTest:
      commands:
        - npm install
        - npm install pm2
        - npx pm2 start npm -- start
    test:
      commands:
        - npm test
    postTest:
      commands:
        - npx pm2 kill
  artifacts:
    baseDirectory: ./
    configFilePath: '**/mochawesome.json'
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*

[ghost]

Thank you for testing that out @andychoww. We are tracking this as a bug and will update you once it's been fixed. In the mean time, we are still investigating a possible workaround. Thank you for your patience!

[aleksey-moravio]

Solution from @hloriana is about Cypress test, but @andychoww report about Jest tests. That's why all previous conversation is useless.

[ghost]

Amplify Hosting only supports testing with Cypress at this time (see here). Please follow the feature request to support testing with the Jest framework on issue https://github.com/aws-amplify/amplify-hosting/issues/464.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.