Private URLs for Test Artifact do not display uploaded Playwright Traces

[ethan-bm]

Describe the bug If:

• "Private URLs for Test Artifact" for Test Artifacts is toggled on
• "Share credentials with Testmot.io Reporter" is toggled on (i.e. not passed in via env variable, but entered in Testomat settings)
• Backblaze is used for the artifact storage
• Playwright test is run via GitHub Action
  • TESTOMATIO_PRIVATE_ARTIFACTS=1 is passed in as an env variable

Then:

• When the test is viewed from the associated run, in Testomat
  • When the attachments are viewed
  • The trace can be downloaded but not viewed

To Reproduce Steps to reproduce the behavior:

1. Run Playwright via GitHub action with trace as a saved artifact, testomat.io reporter integrated as described above
2. Once the GitHub action completes, go to Testomat and find the run
3. On the run, view the test and its attachments
4. Observe the trace file is available. When the trace is clicked, an error message appears, but the file can be downloaded

Expected behavior In step 4 from "To Reproduce" above, I would expect the trace file to be viewable in the embedded trace viewer.

Screenshots

Desktop (please complete the following information):

• OS: Widows 11
• Browser: Edge
• Browser Version: 128.0.2739.42
• Application: Web App, Development Environment

Additional context N/A

[poliarush]

more details here https://app.crisp.chat/website/4192ab4e-8b76-4b7e-a8c4-ea66dea249e7/inbox/session_e9696553-76b8-4d3b-948e-0808c9701a6d/

[ethan-bm]

Update:

I received feedback to update Backblaze's CORS policy like so:

This enabled the trace to be viewed (as long as TESTOMATIO_PRIVATE_ARTIFACTS=1 is used). However, there is a catch. The trace is only viewable/downloadable if a single test is in the run. If multiple tests are uploaded in the same run, it does not work.

I have updated my project since submitting the issue and am at:

• Playwright, v1.47.0
• Testomat Reporter, v1.5.0

[DavertMik]

@AZANIR could you connect with a client?

[AZANIR]

@ethan-bm Hi can we meet?

[ethan-bm]

@AZANIR Yes, I am available to talk. What hours and form of communication do you prefer? I am based out of the East Coast in the US (ET time zone, GMT -4).

[AZANIR]

We've had a lot of artifact updates, if the issue is relevant we'd like to make a call and look into the problem in more detail. We can schedule a call at your convenience.

[poliarush]

@AZANIR https://app.crisp.chat/website/4192ab4e-8b76-4b7e-a8c4-ea66dea249e7/inbox/session_e9696553-76b8-4d3b-948e-0808c9701a6d/

[ethan-bm]

@AZANIR I am available today from 9am-12pm ET if you would like to hop on a call.

[ethan-bm]

Current environment details

Valid as of: 28Oct2024

Local Development

• I use Docker Desktop with the Docker image Provided by Playwright:
  • mcr.microsoft.com/playwright:v1.48.0-noble
• Node.js: v20.18.0

Testomat Run

I run my Playwright tests for testomat via GitHub Action. Here are the config details:

• Docker image: playwright:v1.48.0-noble
  • Same base docker image as local docker environment.
• Testomat Reporter: v1.5.3
• dotenv: v16.4.5
• @types/node: v22.7.5

Here is what my GitHub Action run command looks like:

 run: TESTOMATIO_ENV="${{ env.TESTOMATIO_ENVIRONMENT }}" TESTOMATIO_TITLE="${{ env.REPORT_NAME }}" TESTOMATIO_PRIVATE_ARTIFACTS=1 npx playwright test ${{ github.event.inputs.Target_Item }} --config=playwright.testomatWithMoreArtifacts.config.ts
        env:
          RAM_PASSWORD: ${{ secrets.RAM_PASSWORD }}
          RAM_PASSWORD2: ${{ secrets.RAM_PASSWORD2 }}
          RAM_URL: ${{ github.event.inputs.URL }}
          TESTOMATIO_KEY: ${{ secrets.TESTOMATIO_KEY }}
          TESTOMATIO_ENVIRONMENT: ${{ github.event.inputs.Testomat_Environment}}
          REPORT_NAME: ${{ env.DATE }}_${{ github.event.inputs.URL }}_${{ github.event.inputs.Target_Item }}

I do not pass in backblaze credentials via GitHub Action or locally in my docker environment. I have them saved in my Testomat Project's Artifact Settings.

Here is the config file that is used for Playwright when run via GitHub Action:

import { defineConfig, devices } from '@playwright/test';
import process from 'process';
import dotenv from 'dotenv';

/**
 * Read environment variables from file.
 * https://github.com/motdotla/dotenv
 */
dotenv.config();

let reporter;

if (process.env.CI) {
    reporter = [
        ['@testomatio/reporter/lib/adapter/playwright.js',
            {
                apiKey: process.env.TESTOMATIO_KEY,
            },
        ],
        ['line'],
        ['github'],
    ];
}
else {
    //Local environment
    reporter = [
        ['list'],
        ['html']
    ];
}

/**
 * See https://playwright.dev/docs/test-configuration.
 */
export default defineConfig({
    testDir: './tests',
    /* Run tests in files in parallel */
    fullyParallel: true,
    /* Fail the build on CI if you accidentally left test.only in the source code. */
    forbidOnly: !!process.env.CI,
    /* Retry on CI only */
    retries: process.env.CI ? 0 : 0,
    /* Opt out of parallel tests on CI. */
    workers: process.env.CI ? 2 : 3,
    /* Reporter to use. See https://playwright.dev/docs/test-reporters */
    reporter: reporter,
    /* Shared settings for all the projects below. See https://playwright.dev/docs/api/class-testoptions. */
    use: {
        /* Base URL to use in actions like `await page.goto('/')`. */
        // baseURL: 'http://127.0.0.1:3000',

        /* Collect trace when retrying the failed test. See https://playwright.dev/docs/trace-viewer */
        trace: process.env.CI ? 'on': 'retain-on-failure',
        video: process.env.CI ? 'retain-on-failure': 'off',
        screenshot: process.env.CI ? 'on': 'off',
    },
    expect: {
        //Increase the default timeout for assert actions
        timeout: process.env.CI ? 5 * 60 * 1000 : 1 * 60 * 1000,
        //Set the max pixel difference for pixels and/or ratio
        toHaveScreenshot: { maxDiffPixelRatio: 0.1 },
    },
    //Increase the global timeout
    timeout: process.env.CI ? 5 * 60 * 1000 : 3 * 60 * 1000,

    /* Configure projects for major browsers */
    projects: [
        {
            name: 'chromium',
            use: { ...devices['Desktop Chrome'] },
        },

        // {
        //   name: 'firefox',
        //   use: { ...devices['Desktop Firefox'] },
        // },

        // {
        //   name: 'webkit',
        //   use: { ...devices['Desktop Safari'] },
        // },

        /* Test against mobile viewports. */
        // {
        //   name: 'Mobile Chrome',
        //   use: { ...devices['Pixel 5'] },
        // },
        // {
        //   name: 'Mobile Safari',
        //   use: { ...devices['iPhone 12'] },
        // },

        /* Test against branded browsers. */
        // {
        //   name: 'Microsoft Edge',
        //   use: { ...devices['Desktop Edge'], channel: 'msedge' },
        // },
        // {
        //   name: 'Google Chrome',
        //   use: { ...devices['Desktop Chrome'], channel: 'chrome' },
        // },
    ],

    /* Run your local dev server before starting the tests */
    // webServer: {
    //   command: 'npm run start',
    //   url: 'http://127.0.0.1:3000',
    //   reuseExistingServer: !process.env.CI,
    // },
});

[DavertMik]

@ethan-bm Is there a chance you could try another provider than Backblaze? Maybe AWS S3 or DigitalOcean S3 or Cloudflare R2 (which is the cheapest option).

If it will work there, then we can say this issue is specific to Backblaze

In this case we can proceed working on it

[ethan-bm]

I will try DigitalOcean S3. My company requires submitting an acquisition form for purchasing any software that requires a card on file, which I will have to submit and get approved. Please give 5-10 business days to complete the internal paperwork and setup/test the new provider.

[ethan-bm]

Update: Still waiting on approval from my company to purchase credits for Digital Ocean. The software acquistion form has been submitted and is being reviewed.

[ethan-bm]

Update: Digital Ocean has been denied for software to use by my company. They have instructed me to use AWS instead. Re-submitting a software acquistion request form to get an AWS trial.

[ethan-bm]

Update: AWS has been denied. At this time, I am only approved to use products in the Azure suite for artifact storage. As such, I will disable artifact uploads & integrations to testomat until Azure integrations are a working feature. I plan to append a guid to test runs which I will use to trace back to the relevant artifact as stored in Blob Storage or some other Azure storage option.