chrome tabCapture example

[ddenis1994]

hi am tiring to use the tabCapture API without success for a few days now

what I tried running the tab capture in the content - results in undefined to the API running the tab capture inside an iframe that the src is HTML that accessible to the extension (iFrame.src = chrome.runtime.getURL("gif.html");) -result in an error when the user clicks on a button to initiate the recorded Error starting tab capture

maybe someone can share code example for using the tab capture API in manifest version 3

thanks

[guest271314]

You should be able to open an <iframe> with URL set to chrome-extension://<id>/file.html by specifying the file in "web_accessible_resources" then use navigator.mediaDevices.getUserMedia() and navigator.mediaDevices.getDisplayMedia().

[guest271314]

The MV2 version does capture video, however suffers from MediaStreamTrack of kind video muting and unmuting due to unspecified Chrome implementation https://bugs.chromium.org/p/chromium/issues/detail?id=931033, https://bugs.chromium.org/p/chromium/issues/detail?id=1099280, https://bugs.chromium.org/p/chromium/issues/detail?id=1100746, https://github.com/w3c/mediacapture-screen-share/issues/141 resulting in the video not playing at HTML <video> element, rather is a freeze frame of the last time the user focused on the tab being shared.

What are you trying to achieve?

Can you post the code you are testing here?

[ddenis1994]

I am trying to capture the tab video for later edit share and upload in the end, I solved it by creating streamID inside the popup page and pass it to the content script for capturing .

[kwertop]

HI @ddenis1994, @guest271314 Could you please post any sample code for fetching mediaStream in content script. I've managed to get streamId from popup.html but when I'm doing:

navigator.mediaDevices.getUserMedia({ audio: { mandatory: { chromeMediaSource: 'desktop', //also tried by setting this to 'tab' chromeMediaSourceId: streamId } } }, (tabStream) => { // })

in the content script, I'm getting DOMException: Requested device not found error for chromeMediaSource = desktop while DOMException: Error starting tab capture error for chromeMediaSource = tab .

[guest271314]

@rahul0106 What specific media content are you trying to capture?

[kwertop]

@guest271314 I'm trying to capture audio in any live web based meeting conference app playing in a chrome tab. I'm able to capture user's audio input but I also need to capture the system audio.

[guest271314]

You can use getDisplayMedia() select Tab capture at UI to capture audio output of a tab. Chrome does not provide a means to capture entire system audio output, only audio output by a Chrome tab. On *nix you can utilize parec https://github.com/guest271314/captureSystemAudio/tree/master/native_messaging/capture_system_audio.

[guest271314]

@rahul0106 See also https://github.com/Lightcord/Lightcord/issues/31.

[kwertop]

@guest271314 By system audio, I meant the output audio of the tab. Like a speaker in a zoom meeting.

[kwertop]

I tried using getDisplayMedia(). It throws the following error: Failed to execute 'getDisplayMedia' on 'MediaDevices': Access to the feature "display-capture" is disallowed by permission policy desktopCapture permission is present in the manifest.json file.

[guest271314]

I meant the output audio of the tab.

Is audio playing in the tab at an <audio> element, or using AudioContext?

Can you post the code you are trying here?

[kwertop]

There isn't any <audio> element in the page source of a zoom call (not sure about AudioContext). This is what I'm trying:

navigator.mediaDevices.getUserMedia({
  audio: {
    mandatory: {
      chromeMediaSource: 'tab',
      chromeMediaSourceId: streamId // this is passed from popup.html using chrome.runtime.sendmessage
    }
  }
}, (tabStream) => {
    // do something with tabStream
});

[guest271314]

There isn't any <audio> element in the page source of a zoom call (not sure about AudioContext). This is what I'm trying:

navigator.mediaDevices.getUserMedia({
  audio: {
    mandatory: {
      chromeMediaSource: 'tab',
      chromeMediaSourceId: streamId // this is passed from popup.html using chrome.runtime.sendmessage
    }
  }
}, (tabStream) => {
    // do something with tabStream
});

Chrome only captures audio when the audio is playing in the tab, that is, <audio> and AudioContext audio destination node. For example, Chromium does not capture window.speechSynthesis.speak() when Google voices are not used, where speech dispatcher technically plays audio at system outside of the browser https://bugs.chromium.org/p/chromium/issues/detail?id=1185527, why this answer https://stackoverflow.com/questions/45003548/how-to-capture-generated-audio-from-window-speechsynthesis-speak-call/70665493#70665493 does not produce expected result.

The signature used in the code navigator.getUserMedia(constraints, successCallback, errorCallback); is deprecated https://developer.mozilla.org/en-US/docs/Web/API/Navigator/getUserMedia, see https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia.

As indicated in the comments linked

@rahul0106 See also Lightcord/Lightcord#31.

specifically note the constraints https://github.com/Lightcord/Lightcord/issues/31#issuecomment-922553830.

And the flags used in https://bugs.chromium.org/p/chromium/issues/detail?id=1195881#c19 for headless capture, e.g.,

--auto-select-desktop-capture-source="Entire screen"

manifest.json

{
  "name": "Tab audio capture",
  "version": "1.0",
  "manifest_version": 3,
  "permissions": [ 
    "tabs", 
    "activeTab", 
    "desktopCapture", 
    "scripting" 
  ],
  "background": {
    "service_worker": "background.js"
  },
  "action": {}
}

background.js

async function captureStream(streamId) {
  const stream = await navigator.mediaDevices.getUserMedia({
    video: {
      mandatory: {
        chromeMediaSource: 'screen',
        chromeMediaSourceId: streamId,
      },
    },
    audio: {
      mandatory: {
        chromeMediaSource: 'desktop',
        chromeMediaSourceId: streamId,
      },
    },
  });
  stream.removeTrack(stream.getVideoTracks()[0]);
  console.log(stream, stream.getTracks()[0]);
  const recorder = new MediaRecorder(stream);
  recorder.start();
  return new Promise((resolve) => {
    setTimeout(() => recorder.stop(), 1000 * 10);
    recorder.ondataavailable = (e) => {
      const blobURL = URL.createObjectURL(e.data);
      resolve(blobURL);
      console.log(blobURL);
    };
  });
}

chrome.action.onClicked.addListener(async (tab) => {
  console.log(tab);
  const { streamId, options } = await new Promise((resolve) => {
    chrome.desktopCapture.chooseDesktopMedia(
      ['tab', 'audio'],
      tab,
      async (streamId, options) => {
        resolve({ streamId, options });
      }
    );
  }).catch((err) => console.error(err));
  console.log(streamId, options);
  const [{frameId, result}] = await chrome.scripting.executeScript({
    target: {
      tabId: tab.id,
    },
    world: 'MAIN',
    args: [streamId],
    func: captureStream,
  });
  console.log(frameId, result);
});

audio-capture.zip

[LiuShuaiyi]

Is there anyway to start capturing the current tab directly (video only)? If I understand correctly navigator.mediaDevices.getUserMedia always pops up a window for users to select what to capture. What if I don't want that window? What if I just want to start capture the current active tab right away?

[guest271314]

They have designed the API to limit that capability. Should be possible using a local application.

[guest271314]

@LiuShuaiyi You can try with this flag https://peter.sh/experiments/chromium-command-line-switches/#auto-select-desktop-capture-source --auto-select-desktop-capture-source="Tab".

[LiuShuaiyi]

Okay I finally got a working prototype with the same approach as said in OP's comment: https://github.com/GoogleChrome/chrome-extensions-samples/issues/627#issuecomment-899104997.

• Use chrome.tabCapture.getMediaStreamId to get stream id in popup.
• Pass the stream id to content script with chrome.tabs.sendMessage.
• In content script's message handler, use navigator.mediaDevices.getUserMedia to start capture.

With this approach, capture starts immediately on the active tab without the media select popup window.

[donething]

Okay I finally got a working prototype with the same approach as said in OP's comment: #627 (comment).

• Use chrome.tabCapture.getMediaStreamId to get stream id in popup.
• Pass the stream id to content script with chrome.tabs.sendMessage.
• In content script's message handler, use navigator.mediaDevices.getUserMedia to start capture.

With this approach, capture starts immediately on the active tab without the media select popup window.

In the third step, Why I can not use navigator.mediaDevices.getUserMedia to start capture in content script's message handler.

The Chrome hint "DOMException: Error starting tab capture".

[guest271314]

Can you post your code here?

[donething]

Can you post your code here?

let streamer;
let audioCtx;
let source;
let gainNode;

chrome.runtime.onConnect.addListener(port => {
  port.onMessage.addListener(async msg => {
    audioCtx = new AudioContext();
    try {
      // The Chrome hint "DOMException: Error starting tab capture".
      streamer = await navigator.mediaDevices.getUserMedia({
        audio: {
          mandatory: {
            chromeMediaSource: 'tab',
            // this is passed from popup.html using chrome.runtime.sendmessage
            chromeMediaSourceId: msg.streamID
          }
        }
      });
    } catch (e) {
      console.log(e);
    }
    
    // bind audiocontext...
  });
});

[guest271314]

@donething The code in https://github.com/GoogleChrome/chrome-extensions-samples/issues/627#issuecomment-1013604912 works as expected.

[guest271314]

@donething How does the user activate the popup? Can you post you complete code here?

[donething]

@donething How does the user activate the popup? Can you post you complete code here?

I'm sorry, I had deleted those code, I cannot post them now.

I had test https://github.com/GoogleChrome/chrome-extensions-samples/issues/627#issuecomment-1013604912 on my brower, It works.

Thank you.

[zhw2590582]

@LiuShuaiyi @ddenis1994 After I use the method you mentioned, I can successfully get the audio, but the page will become silent, I want it to continue to play the sound, do you have a way to do this

[zhw2590582]

@guest271314 @ddenis1994 @donething

test-the-audio.zip You can download and test it, I don't know where I'm doing wrong, can you help me take a look, thanks a lot:

manifest.json

{
  "name": "Test the audio",
  "version": "1.0.0",
  "description": "Sound can be recorded, but the tab is muted",
  "manifest_version": 3,
  "permissions": ["tabs", "activeTab", "scripting", "tabCapture"],
  "action": {
    "default_popup": "popup.html"
  }
}

popup.js

// Get the current id
chrome.tabs.query(
  {
    active: true,
    currentWindow: true,
  },
  (tabs) => {
    const tabId = tabs[0].id;
    // Get the streamId
    chrome.tabCapture.getMediaStreamId(
      {
        consumerTabId: tabId,
      },
      (streamId) => {
        // Load the content.js
        chrome.scripting.executeScript(
          {
            target: { tabId },
            files: ["content.js"],
          },
          () => {
            // Send the streamId to the tab
            chrome.tabs.sendMessage(tabId, streamId);
          }
        );
      }
    );
  }
);

chrome.runtime.onMessage.addListener(async (streamId) => {
  console.log("streamId--->", streamId);

  // Get the stream
  const stream = await navigator.mediaDevices.getUserMedia({
    audio: {
      mandatory: {
        chromeMediaSource: "tab",
        chromeMediaSourceId: streamId,
        echoCancellation: true,
      },
    },
  });

  console.log("stream--->", stream);

  // Now I successfully recorded the sound, but the page is also muted
  // -------------------------------------------------------------------
  // Connected to an audio player, but still no sound, why ?????????????
  // -------------------------------------------------------------------

  const audioContext = new AudioContext();
  const mediaStream = audioContext.createMediaStreamSource(stream);
  mediaStream.connect(audioContext.destination);
});

[guest271314]

@guest271314 @ddenis1994 @donething

test-the-audio.zip You can download and test it, I don't know where I'm doing wrong, can you help me take a look, thanks a lot:

manifest.json

{
  "name": "Test the audio",
  "version": "1.0.0",
  "description": "Sound can be recorded, but the tab is muted",
  "manifest_version": 3,
  "permissions": ["tabs", "activeTab", "scripting", "tabCapture"],
  "action": {
    "default_popup": "popup.html"
  }
}

popup.js

// Get the current id
chrome.tabs.query(
  {
    active: true,
    currentWindow: true,
  },
  (tabs) => {
    const tabId = tabs[0].id;
    // Get the streamId
    chrome.tabCapture.getMediaStreamId(
      {
        consumerTabId: tabId,
      },
      (streamId) => {
        // Load the content.js
        chrome.scripting.executeScript(
          {
            target: { tabId },
            files: ["content.js"],
          },
          () => {
            // Send the streamId to the tab
            chrome.tabs.sendMessage(tabId, streamId);
          }
        );
      }
    );
  }
);

chrome.runtime.onMessage.addListener(async (streamId) => {
  console.log("streamId--->", streamId);

  // Get the stream
  const stream = await navigator.mediaDevices.getUserMedia({
    audio: {
      mandatory: {
        chromeMediaSource: "tab",
        chromeMediaSourceId: streamId,
        echoCancellation: true,
      },
    },
  });

  console.log("stream--->", stream);

  // Now I successfully recorded the sound, but the page is also muted
  // -------------------------------------------------------------------
  // Connected to an audio player, but still no sound, why ?????????????
  // -------------------------------------------------------------------

  const audioContext = new AudioContext();
  const mediaStream = audioContext.createMediaStreamSource(stream);
  mediaStream.connect(audioContext.destination);
});

I tested on Chromium 102.

I reproduced the tab muting when when getUserMedia() is executed.

Looks like a bug to me https://crbug.com.

The tab audio should not be muted when getUserMedia() is called.

Recording the audio does capture sound.

I re-tested the code at https://github.com/GoogleChrome/chrome-extensions-samples/issues/627#issuecomment-1013604912. The tab is not muted, the recording is garbled.

This error is reported at chrome://extensions:

Uncaught (in promise) Error: The message port closed before a response was received.

[zhw2590582]

@guest271314

The following error can be ignored

Uncaught (in promise) Error: The message port closed before a response was received.

test-the-audio-v2.zip

I use the code you said, the audio can be successfully recorded, and the page can play the sound normally

But I don't want the browser to evoke the tab selection popup again, I want to record directly on the current page

{
  "name": "Test the audio",
  "version": "1.0.0",
  "description": "Sound can be recorded, but the tab is muted",
  "manifest_version": 3,
  "permissions": ["tabs", "activeTab", "scripting", "desktopCapture"],
  "action": {
    "default_popup": "popup.html"
  }
}

popup.js

// Get the current id
chrome.tabs.query(
  {
    active: true,
    currentWindow: true,
  },
  (tabs) => {
    const tab = tabs[0];
    const tabId = tab.id;

    // Get the streamId from desktopCapture
    chrome.desktopCapture.chooseDesktopMedia(
      ["tab", "audio"],
      tab,
      (streamId) => {
        // Load the content.js
        chrome.scripting.executeScript(
          {
            target: { tabId },
            files: ["content.js"],
          },
          () => {
            // Send the streamId to the tab
            chrome.tabs.sendMessage(tabId, streamId);
          }
        );
      }
    );
  }
);

content.js

const audioContext = new AudioContext();

chrome.runtime.onMessage.addListener(async (streamId) => {
  console.log("streamId--->", streamId);

  // Get the stream
  const stream = await navigator.mediaDevices.getUserMedia({
    video: {
      mandatory: {
        chromeMediaSource: "screen",
        chromeMediaSourceId: streamId,
      },
    },
    audio: {
      mandatory: {
        chromeMediaSource: "desktop",
        chromeMediaSourceId: streamId,
      },
    },
  });

  stream.removeTrack(stream.getVideoTracks()[0]);

  console.log("stream--->", stream);

  // Here you can successfully record the sound, and the page will not mute
  // -------------------------------------------
  // If I don't create the recorder node, the recording is garbled, it's weird
  // -------------------------------------------
 
  const mediaStream = audioContext.createMediaStreamSource(stream);

  const recorder = audioContext.createScriptProcessor(0, 1, 1);
  recorder.onaudioprocess = (event) => {
    const inputData = event.inputBuffer.getChannelData(0);
    console.log("audio--->", inputData);
  };

  mediaStream.connect(recorder);
  recorder.connect(audioContext.destination);
});

[guest271314]

But I don't want the browser to evoke the tab selection popup again, I want to record directly on the current page

What do you mean by

again

?

You can try the following flag on your own machine.

@LiuShuaiyi You can try with this flag https://peter.sh/experiments/chromium-command-line-switches/#auto-select-desktop-capture-source --auto-select-desktop-capture-source="Tab".

There are other similar flags.

I don't think you can bypass the capture UI using extension code alone.

This code from your first example

  // Get the stream
  const stream = await navigator.mediaDevices.getUserMedia({
    audio: {
      mandatory: {
        chromeMediaSource: "tab",
        chromeMediaSourceId: streamId,
        echoCancellation: true,
      },
    },
  });

that mutes the audio output at the tab is a bug I suggest you file an issue about at https://crbug.com.

[zhw2590582]

@guest271314 ok, thanks a lot for your help

[dima11221122]

Is it possible to capture tab including navigation. Aformentioned approach can be used only if a webpage doesn't refresh.