dd-trace-js

dd-trace-js copied to clipboard

Context: I'm discussing this DD product throughout: https://docs.datadoghq.com/continuous_integration/setup_tests/javascript/

This is a feature request to add first-class support for @playwright/test (https://github.com/microsoft/playwright) an amazing Microsoft project for end-to-end browser testing (including its own fabulous test runner).

Currently, DD and Playwright users can use the JUnit report option to take advantage of DD reporting and tools, but it would be nice to have a formal integration (at least for me, as a DD and Playwright user) like there is for Cypress/Mocha/Jest.

If this is something that the DD team would be willing to add to its set of existing plugins, I'd be happy to help contribute it. (Briefly looked at some of the other DD plugins which mostly make sense but would need a few pointers as to overall how they work on the DD side of things.)

Disclaimer: I'm only an external contributor to Playwright (https://github.com/microsoft/playwright/pulls?q=author%3Arwoll) and have no formal relationship with Microsoft.

We definitely welcome contributions! However, since our CI product is at a very early stage, I'm not sure if we accept additional integrations yet, so I'll defer to @juan-fernandez to confirm whether this would be possible at this point in time.

hi @rwoll! Sadly we don't have the bandwidth right now to work on new integrations. This said, contributions are more than welcome, as Roch mentioned, and we'll help you out. A good point to start understanding the test instrumentations might be the cucumber plugin, which is one of the simpler ones: https://github.com/DataDog/dd-trace-js/blob/master/packages/datadog-plugin-cucumber/src/index.js.

Thanks @juan-fernandez! I will try to give it a pass at some point in the next few weeks!

@juan-fernandez I noticed Cypress DD plugin (and many of the others), runtime patch Cypress via module.exports, but Cypress (like Playwright) supports Custom Reporters (Cypress Reporters Doc and Playwright Custom Reporters Docs.

Is the runtime patch (versus a Custom Reporter) used so DD gets more specific runtime details than can be traced with a Custom Cypress Reporter or can it be implemented as a Custom Cypress reporter and collect the same data?

(I'm wondering not to critique the Cypress plugin, but am trying to understand for Playwright if I will want to runtime patch the Test Runner itself or just implement a Custom Reporter that invokes the relevant DD bits. Custom Reporter API is stable and public in Playwright so if that can be used over runtime patching I think that is preferred at first reading.)

Thanks!

Custom Reporter API is stable and public in Playwright so if that can be used over runtime patching I think that is preferred at first rea

Hi @rwoll, that's a great question. Normally a runtime patch approach is preferred because you get a much deeper visibility and the full benefits of distributed tracing.

If a report is good enough (say you just want to know if your test passed or failed), we actually support uploading JUnit XML reports: https://docs.datadoghq.com/continuous_integration/setup_tests/junit_upload/#uploading-test-reports

@rwoll Did you end up pursuing the runtime patch approach suggested here and have anything public to share? My team is looking to get the deeper visibility of our playwright tests with DD

@jmichel17 No, I was using DD at my old job, but am no longer there.

@juan-fernandez Playwright is widely used. Might consider prioritizing this in your backlog

For others: got some tracing in DD CI Test monitoring by utilizing Playwright's Reporter class like this.

Warnings to others attempting this:

1. I had a tremendously hard time opening a span in onTestBegin and then attempting to finish it in onTestEnd. The dd-trace library's Span object can't be used across methods like you would expect, even if you maintain a reference to it. I tried several methods to do this and unfortunately nothing worked.
2. I found I had to initialize a new tracer in every fresh call to onTestEnd. Like the library's Span object, the tracer doesn't seem to behave correctly across method calls. So I squeezed all reporting into this single callback instead of using Reporter's other finer-grain callbacks.

import { Reporter, TestCase, TestResult } from '@playwright/test/reporter'
import ddTrace from 'dd-trace'
import { SpanOptions } from 'opentracing/lib/tracer'

const {
  TEST_STATUS,
  CI_APP_ORIGIN,
  getTestEnvironmentMetadata,
  getTestCommonTags,
  getTestParentSpan,
} = require('dd-trace/packages/dd-trace/src/plugins/util/test')
const { ORIGIN_KEY } = require('dd-trace/packages/dd-trace/src/constants')

class DataDogTestReporter implements Reporter {
  private testEnvironmentMetadata: Record<string, any>

  onBegin(): void {
    this.testEnvironmentMetadata = getTestEnvironmentMetadata('playwright')
  }

  onTestEnd(test: TestCase, result: TestResult) {
    const localTracer = ddTrace.init({
      startupLogs: false,
    })
    const childOf = getTestParentSpan(localTracer)
    const testFile = test.parent.location.file.substring(
      test.parent.location.file.indexOf('__tests__') + '__tests__/e2e/'.length,
    )
    const spanOptions: SpanOptions = {
      childOf,
      tags: {
        [ORIGIN_KEY]: CI_APP_ORIGIN,
        ...getTestCommonTags(
          test.title,
          `${testFile} / ${test.parent.title}`,
          (localTracer as any)._tracer._version,
        ),
        ...this.testEnvironmentMetadata,
      },
      startTime: result.startTime.getTime(),
    }
    const span = localTracer.startSpan('playwright.test', spanOptions)
    result.steps.forEach((step) => {
      const childSpan = localTracer.startSpan(step.title, {
        childOf: span,
        startTime: step.startTime.getTime(),
      })
      childSpan.finish(step.startTime.getTime() + step.duration)
    })
    span.setTag(TEST_STATUS, PLAYWRIGHT_STATUS_TO_TEST_STATUS[result.status])
    span.finish(result.startTime.getTime() + result.duration)
    ;(localTracer as any)._tracer._exporter._writer.flush()
  }
}

export default DataDogTestReporter

const PLAYWRIGHT_STATUS_TO_TEST_STATUS = {
  passed: 'pass',
  failed: 'fail',
  timedOut: 'fail',
  skipped: 'skip',
}

Could this ticket be reopened? I believe the CI aspect of Datadog is no longer in beta so it would be great to see this reopened as Playwright gets more of the testing marketshare. @juan-fernandez

@rochdev Hi Roch, I see you reopened this feature-reuqest, can you share any update for this? When will datadog support Playwright framework in Test Service Setup?

@rwoll Hi rwoll, I have a question about your discussion. You said Currently, DD and Playwright users can use the JUnit report option to take advantage of DD reporting and tools Did that mean I can 't choose Jest to replace Playwright in https://app.datadoghq.com/ci/setup/test ,it will fail for playwright integration tests. And Your previous solution was to write a customer reporter like @jmichel17 pasted above, it can push some metrics to datadog to show a dashboard. Can it looks like RUM(real user monitor) and give a deeper visibility to the integration test case.?

@xbhoneybee I don't think there are any plans right now to support Playwright, I mostly reopened so that we can track the feature request. @juan-fernandez would have more information about support of any additional test framework.

@xbhoneybee Using my custom Reporter above will get you pretty far with metrics in the DD testing dashboard w/ Playwright tests

@rochdev Just checked, Playwright actually has more Github stars than Cypress. I'll just reiterate once more that it is definitely a much-used, mature testing framework. Would be great to see an officially supported DD integration.

hey! We haven't considered @playwright/test yet but I'll have a look to check how difficult its instrumentation is. Thanks for the feedback!

This would be super helpful for us too!

@jmichel17 I've been trying to use your reporter above and it seems to work flawlessly on my machine locally (windows with datadog agent locally). However when trying to run it in a Bitbucket pipeline the traces appear to be missing the Git information, the test runs are present but no Git details/branches etc.

By any random chance do you happen to know why?

@declancarrollcb This is probably unrelated to the custom reporter. You need to ensure your CI environment has the proper Datadog variables set. See this page: https://docs.datadoghq.com/continuous_integration/troubleshooting/

In my company's CI we ensure the following are set when running our Playwright tests: DD_GIT_BRANCH DD_GIT_REPOSITORY_URL DD_GIT_COMMIT_AUTHOR_NAME DD_GIT_COMMIT_AUTHOR_EMAIL DD_GIT_COMMIT_AUTHOR_DATE DD_GIT_COMMIT_MESSAGE DD_GIT_COMMIT_COMMITTER_NAME DD_GIT_COMMIT_COMMITTER_EMAIL DD_GIT_COMMIT_COMMITTER_DATE DD_GIT_COMMIT_SHA DD_GIT_REPOSITORY_URL DD_TRACE_ENABLED=true

The dd-trace library will pick these up and associate this information w/ the test tracing that the custom Reporter generates.

My company utilizes Github actions for our CI and it provides all these to us as default env variables. We execute a shell script before running our tests to translate the Github provided environment variables to the Datadog specified variables. I imagine Bitbucket probably provides a similar mechanism for you to do the same.

Thanks for the detailed reply @jmichel17. I was previously using the Mocha as the test runner and all the variables seemed to be set automatically. So I thought it was the dd-trace call that set that up. I'll create a script to set the variables and hopefully, it's all good from there.

For anyone who is having the same issues I was having above, it's due to a bug on the Atlassian side. The git commit hash is different in a Bitbucket PR pipeline vs a Bitbucket scheduled pipeline.

https://jira.atlassian.com/browse/BCLOUD-21201

You can fix it by assigning it with the correct commit value in the pipeline:

export DD_GIT_COMMIT_SHA=$(git log -1 --format="%H" $BITBUCKET_COMMIT 2>/dev/null)

Hi All, I have been following this thread for a while. I am looking for DataDog Support for @playwright/test

Please let us know if this will be taken up by DataDog in near future ?

Hi All, I have been following this thread for a while. I am looking for DataDog Support for @playwright/test

Please let us know if this will be taken up by DataDog in near future ?

hi! This is definitely useful feedback so thanks for that. I did have a look at the implementation and it's not straightforward, so this will need planning 😄 . I'm sharing this thread with the team though, so we can gauge the priority of supporting this library.

an update: we're planning to support this in the near future. I'm already working on it in https://github.com/DataDog/dd-trace-js/pull/2708. It'll still take a bit to complete but progress has been made!

Thanks to @jmichel17 btw! Your snippet was useful 😄

@juan-fernandez please let us know if there will be some documentation provided as well along with release ?

@juan-fernandez please let us know if there will be some documentation provided as well along with release ?

yes of course 😄. I'll update this with the changes to the docs site

playwright support has been released in https://github.com/DataDog/dd-trace-js/releases/tag/v3.13.0 and https://github.com/DataDog/dd-trace-js/releases/tag/v2.26.0 🎉 . Documentation updates are coming soon 😄

playwright support has been released in https://github.com/DataDog/dd-trace-js/releases/tag/v3.13.0 and https://github.com/DataDog/dd-trace-js/releases/tag/v2.26.0 🎉 . Documentation updates are coming soon 😄

you can see the instructions to setup playwright in https://docs.datadoghq.com/continuous_integration/tests/javascript/?tab=playwright#instrument-your-tests. Support is from @playwright/test>=1.18.0

Update: not an issue - I just misconfigured the env variables.

@juan-fernandez , thank you for your work! I've just tried it and it doesn't seem to work: https://github.com/DataDog/dd-trace-js/issues/2766 Maybe I'm missing something? Otherwise, can you recommend me a way to get more logs out of it?