Support logging from web/service workers

[bcaudan]

Converting #387 to follow this topic.

This behavior is not requested by many customers so it is not in our top priorities right now. It is definitely something that we would want to integrate but it could take some time before we make progress on that.

[MrLemur]

I would like to leave my support for this issue. We are testing using the library in an extension to log metrics, but seem to be running in to an issue where the library stops sending anything even after it is initialised.

[Sporradik]

+1

[feimosi]

Is there any ETA for this? It's been over a year already.

[bcaudan]

Still a low priority, so no ETA for now.

[feimosi]

Are there any workarounds? Even if I mock document and window objects and no error is thrown, I still don't see my logs in Datadog. It's set up correctly and works on the main page.

[jonesetc]

Ran into this issue unexpectedly when trying to set up logging on an internal chrome extension. All of the external calls happen in the background service worker because of the way ManifestV3 extensions work, so being able to have DD logger in place there would make catching errors much simpler and more reliable.

[itayadler]

I managed to make this work with the extension I work on @ work, the workarounds:

I used happy-dom to polyfill window and document, notice you'll also need to polyfill it prior to your bundle requiring any other module, you can do this with vite/rollup using the intro feature.

happy-dom uses a naiive polyfill for document.cookie, I needed to replace it with tough-cookie, similar to what JSDOM uses. I didn't manage to make other things work with JSDOM, which is why I went with happy-dom. notice that document.cookie is critical to get working properly, otherwise nothing will be sent to datadog

perhaps in your case you don't need the DOM APIs like I did, so you might get off with a more crude polyfill, but for datadog its important you polyfill window.location, I set it to our app URL, and it managed to work with tough-cookie.

[prestonp]

I didn't want to pull in full dependency so I'm doing a smaller polyfill. I created this polyfills.ts and import it early in our worker code.

// @ts-expect-error partial polyfill for @datadog/browser-logs
globalThis.document = {
  referrer: `chrome://${chrome.runtime.id}`,
};

globalThis.window = {
  // @ts-expect-error partial polyfill for @datadog/browser-logs
  location: {
    href: `chrome://${chrome.runtime.id}`,
  },
};

[rzec-allma]

Any update on this, Manifest v2 is not really going to be support pretty soon : https://developer.chrome.com/docs/extensions/mv3/mv2-sunset/ : so that make datadog nearly un-usable for web extensions

[bcaudan]

Still no official support and no plan to move forward on this topic.

[johuder33]

Hey community as many of you I was suffering and thinking how we can make DD working on Service Worker env, especially for whose using DD in manifest V3 extensions.

Please this is a patch solution to get datadog working into service worker, but it's not an official solution from DD team.

Since this ticket was created long ago, it looks like Datadog team has not plan to move forward with this.

So I went through the dd package code, and then I realized that we can make some changes in our extension / service worker side in order to make DD working into a service worker.

First at all this comment helped me a lot to find my patch to make dd working, so thanks @prestonp for shared it.

When I went through the code I realized that DD is using some window / dom apis into the package, apis like:

• document.readyState
• document.getElementsByTagName
• document.location
• document.location.hostname,
• document.location.referrer
• document.location.href
• cookie getter
• cookie setter
• window.TextEncoder luckily this TextEncoder is supported by Web / Service Workers
• window.fetch
• window.location
• window.location.hostname
• window.location.href
• XMLHttpRequest object that is definitely not supported by Web / Service Workers

Once we already know which window / dom apis DD package uses, it's time to patch all the window / dom apis, and make the needed implementation with apis that Web / Service Workers support.

for all the window / document apis accesses, I used the following:

// please defined an URL as you want , or simply you can use "chrome.getURL('')" to use the "chrome-extension://*" protocol
const url = `https://hostname.com/service-worker/${chrome.runtime.id}`;

globalThis.document = {
  readyState: 'completed',
  getElementsByTagName: () => ([]),
  location: {
    referrer: url,
    hostname: 'hostname.com',
    href: chrome.runtime.getURL('')
  },
  // I didn't implement the cookie to use chrome.cookies extension api under the hood, because the usage for document.cookie is sync but the extension api to get cookies is async, so it could not work as expected.
  get cookie() {
    return undefined;
  },
  set cookie(cookie) {
  }
};

// we patch the TextEncoder into window patched object to use the TextEncoder from the Service Worker environment, so we can use self. in order to be the same for window environment and service worker environment

// the same above for fetch patching
globalThis.window = {
  TextEncoder: self.TextEncoder,
  fetch: self.fetch,
  location: {
    hostname: 'hostname.com',
    href: chrome.runtime.getURL('')
  }
};

with above code we patched all the window / dom apis accesses and now we only miss the XMLHttpRequest implementation.

So let's patch it

class XMLHttpRequestWithFetch {
  constructor() {
    this.method = null;
    this.url = null;
    this.status = 500;
    this.listeners = {};
  }

  open(method, url) {
    this.method = method;
    this.url = url;
  }

  addEventListener(eventName, cb) {
    if (!this.listeners[eventName]) {
      this.listeners[eventName] = [];
    }
    this.listeners[eventName].push(cb);
  }

  removeEventListener(eventName, cb) {
    const handlers = this.listeners[eventName];
    if (handlers) {
      const restOfHandlers = handlers.filter(callback => callback !== cb);
      if (restOfHandlers && restOfHandlers.length) {
        this.listeners[eventName] = restOfHandlers;
      } else {
        delete this.listeners[eventName];
      }
    }
  }

  send(data) {
    let _body = data;
    if (typeof data === 'object') {
      _body = JSON.stringify(data);
    }
    fetch(this.url, {
      method: this.method,
      body: _body
    })
      .then((response) => {
        this.status = response.status;
        // notify all listeners that we're done
        Object.keys(this.listeners).forEach((event) => {
          this.listeners[event].forEach((handler) => handler(response));
        });
      })
      .catch(() => {
        // @TODO: probably we should handle the failing case.
      });
  }
}

globalThis.XMLHttpRequest = XMLHttpRequestWithFetch;

so the last important thing we need to do is to import this "Datadog polyfill" file just right before we start using the dd package.

import './datadog-polyfill-for-service-worker.js'; // always import the polyfill right before datadog usage
import { datadogLogs } from '@datadog/browser-logs';
....

So then build / install your extension Service Worker, and you will see how logs are sent to the datadog server.

[UncleFirefox]

Hi,

I'm also trying to use datadogRum to use addAction from a worker thread. Obviously I can't pass the object straight away from the main thread unless I serialize it. I need it that way because I need to be able to correlate the session that was started on the main thread.

Is there a way to force a session id if I create the datadogRum instance from the worker thread? Or would I need to go through the serialization path? I tried some serializers but I keep getting some ugly ReferenceError: callMonitored is not defined I guess I have to smart about how to serialize...

Any ideas?

BTW my use case is simple: I want to spin up a worker to check UI responsiveness. If the main thread does not respond to the worker in 10-15 seconds (most likely due to a long running operation or a freeze) I want to be able to log into DD with my corresponding user session that there was a freeze to later check what actions happened through that provoked it.

[orriborri]

It would be great if rum tracing/logging worked on web/service workers. Is there an official way of getting this heard on datadogs side?

[davidswinegar]

Our application makes fetch requests inside a web worker for performance reasons, and we'd like to be able to use RUM to trace these requests as well. We'd love for this feature or at least a way to get Datadog headers that we could use to track the request that we could send to the server.

[dbjorge]

We've run into this multiple times now (mostly in manifest v3 extension background workers), so I looked into how much work it would be to upstream a fix instead of maintaining a polyfill in several places. To get it to cooperate with just logs (not rum, which I think you'd want to track separately since it's a bunch of additional work), I think this is an idea of the scope of work required:

1. Throughout packages/core and packages/logs, replace window.location and document.location with getGlobalObject<WorkerGlobalScope | Window>().location (maybe give getGlobalOption's type parameter T a default value of WorkerGlobalScope | Window to facilitate this)
2. packages/core/src/browser/cookie.ts: Update areCookiesAuthorized's existing early-exit case for document.cookie === undefined to instead use document?.cookie === undefined. Consumers of this file already handle falling back to local storage if this function returns false, so it's okay that the rest of the file assumes the availability of document.cookie
3. packages/core/src/browser/fetchObservable.ts: window -> getGlobalObject()
4. packages/core/src/browser/pageExitObservable: Add an early check if window === undefined that causes it to produce an observable which never fires (there is no "unload" event or similar for a worker)
5. packages/core/src/browser/runOnReadyState.ts: Not used by logs
6. packages/core/src/browser/xhrObservable.ts: add early exit path for getGlobalObject().XMLHttpRequest === undefined that creates a no-op observable
7. packages/core/src/domain/configuration/configuration.ts: Unless someone has a bright idea for replacing pageExitObservable, the flush timeout probably needs to be decreased by default in worker contexts
8. packages/core/src/domain/report/reportObservable: window.ReportingObserver -> window?.ReportingObserver in initial early exit check in createReportObserver to no-op in the non-DOM context
9. packages/core/src/domain/session/sessionManager.ts: Make trackVisibility no-op if document isn't available (workers are never considered visible)
10. packages/core/src/domain/tracekit/tracekit.ts: window -> getGlobalObject() (note that onunhandledrejection is technically not required by the standard to be present in workers, but it is generally available in practice - MDN notes that it "may" be present in a worker)
11. packages/core/src/tools/utils/browserDetection.ts: window -> getGlobalObject(), (document as any).documentMode -> (document as any)?.documentMode
12. packages/core/src/tools/utils/byteUtils.ts: window.TextEncoder -> getGlobalObject().TextEncoder
13. packages/core/src/tools/utils/polyfills.ts: window.CSS -> getGlobalObject<Window>().CSS, allow the polyfill to handle worker context
14. packages/core/src/tools/utils/timeUtils.ts: performance.timing.navigationStart is deprecated and probably needs to change regardless of this issue, but both it and PerformanceNavigationTiming are additionally unavailable in a worker context; probably fall back to pinning timeStampNow() as of logger init if navigation timings are unavailable
15. packages/core/src/tools/utils/urlPolyfill.ts: probably ok to assume that a worker context has supported URL impl
16. packages/core/src/transport/httpRequest.ts: update fallback logic of beacon and fetch strategies to only fall back to XHR if it is actually available, + window.Request -> getGlobalObject().Request
17. packages/logs/src/boot/logsPublicApi.ts: document.referrer -> document?.referrer, window.location.href -> getGlobalObject().location.href

This is a lengthy-looking list, but most of these changes are individually pretty simple. I think the most complex/"interesting" parts are:

• disallowing httpRequest.ts falling back to XHR
• deciding how to handle not having a good page exit observer implementation, and whether the flush timeout needs to go way down to compensate
• working out a test strategy, especially e2e tests but also a concise way of simulating a worker environment for non-e2e tests

@bcaudan , is this scope of work something your team would be willing to accept as a contribution? (not committing to it yet, but considering it)

[guygrip]

Hi all!

We've recently documented our journey of enabling logging within Chrome extensions and streaming these logs to Datadog. It provides insights into the unique challenges faced and the solutions we crafted. I think you'll find the solution very cool and easy to implement!

Check it out here: Enabling Chrome Extension Logging with Datadog: A Journey of Trial and Error

Any feedback is appreciated. Thanks!

[dbjorge]

@bcaudan , just wanted to give a gentle ping on the question in this comment above - we'd like to understand whether the work described in that comment something your team would be willing to accept as a contribution, so we can avoid having to maintain a huge and fragile polyfill that does the same work as a workaround. Are you the right person to ask?

[bcaudan]

Hi @dbjorge,

We would be glad to receive this kind of contribution but given the scope of the change and some of the unknowns, it may take a lot of back and forth and given the current priority of this topic for us, we may not be as reactive as we would want to be 😕

[mnholtz]

Hey everyone! For folks running into this in the browser extension space, I wanted to share an alternative workaround to those already presented that allows you to continue to use the Datadog Browser SDK as normal.

Our approach leverages the offscreen document api.

In this approach, we have the service worker create an offscreen document, and moved all of our Datadog SDK imports and calls to the offscreen document js. We then have the service worker message the offscreen document via the messenger api with the log (in our case, error logs) and all other necessary information, and the offscreen document responds to that message by forwarding the log to Datadog using the browser SDK methods.

Here is the PR that we made for reference: https://github.com/pixiebrix/pixiebrix-extension/pull/8276

Note that the only potential caveat with this approach has to do with the Reason passed to offscreen.createDocument, as no Reason exposed by the offscreen api really fits the use case of forwarding logs. We were worried that this would result in a rejection from CWS. This was not the case for us (CWS approved that release) but just wanted to flag that as a potential risk just in case 🙂

Hope this is helpful!

[ecefuel]

Another workaround option is to use webpacks imports-loader or ProvidePlugin. You may need to customize the mocked values for your use case.

dom-shims.ts

export const document = {
  addEventListener: function() {},
  cookie: '',
  referrer: '',
  readyState: 'complete',
  visibilityState: 'hidden'
}

export const window = {
  addEventListener: function() {},
  fetch,
  document,
  location,
  TextEncoder,
  navigator,
  Request,
  // @ts-ignore
  ReportingObserver
}

const loaded = Date.now()
const start = loaded - 500

export const performance = {
  now: () => Date.now(),
  timing: {
    connectStart: start,
    navigationStart: start,
    secureConnectionStart: start,
    fetchStart: start,
    domContentLoadedEventStart: start,
    responseStart: start,
    domInteractive: loaded,
    domainLookupEnd: loaded,
    responseEnd: loaded,
    redirectStart: 0,
    requestStart: start,
    unloadEventEnd: 0,
    unloadEventStart: 0,
    domLoading: loaded,
    domComplete: loaded,
    domainLookupStart: start,
    loadEventStart: start,
    domContentLoadedEventEnd: loaded,
    loadEventEnd: loaded,
    redirectEnd: 0,
    connectEnd: loaded
  }
}

// imports-loader options
{
          test: /@datadog/,
          use: [
            {
              loader: 'imports-loader',
              options: {
                type: 'commonjs',
                imports: [
                  {
                    syntax: 'multiple',
                    moduleName: path.resolve('src', 'dom-shims.ts'),
                    name: 'window'
                  },
                  {
                    syntax: 'multiple',
                    moduleName: path.resolve('src', 'dom-shims.ts'),
                    name: 'document'
                  },
                  {
                    syntax: 'multiple',
                    moduleName: path.resolve('src', 'dom-shims.ts'),
                    name: 'performance'
                  }
                ]
              }
            }
          ]
        }

// ProvidePlugin
new webpack.ProvidePlugin({
          window: [path.resolve('src', 'dom-shims.ts'), 'window'],
          document: [path.resolve('src', 'dom-shims.ts'), 'document'],
          performance: [path.resolve('src', 'dom-shims.ts'), 'performance']
        }),

[pavelstudeny]

Now that browser extensions are required by Google to use manifest v3, the logging does not work for browser extensions' background script altogether. Perhaps this would be an argument to increase the priority.