brozzler

brozzler copied to clipboard

I am running brozzler on Ubuntu 17.10. It's working quite well, except that all screenshots shown in the brozzler dashboard web UI are completely black. Is this a known issue? I am using Chromium Version 65.0.3325.181 (Official Build) Built on Ubuntu , running on Ubuntu 17.10 (64-bit)

Ugh. To be clear, there are thumbnail images showing up in the dashboard, it's just that the images are completely black? What about the full size screenshots, are they also black? If so, can you try running brozzle-page http://example.com/... it should log something like this:

2018-04-06 11:06:38,311 90093 INFO MainThread root.on_screenshot(cli.py:192) wrote screenshot to /tmp/http___example_com_-20180406110638.png

Is that image also black?

Yes, both the thumbnail and the full size screenshot files are black. I did notice after writing this issue though that sometimes the contents of the browser window itself of Chromium is black while the page is loading. Perhaps the screenshot is being taken too soon, before the window has updated with the rendered page?

Here is the output of running brozzle-page http://example.com/:

2018-04-06 21:13:22,416 32392 INFO MainThread brozzler.chrome.Chrome.start(chrome.py:141) running: 'chromium-browser --remote-debugging-port=9222 --use-mock-keychain --user-data-dir=/tmp/tmpni3k2umg/chrome-user-data --disable-background-networking --disable-web-sockets --disable-cache --window-size=1100,900 --no-default-browser-check --disable-first-run-ui --no-first-run --homepage=about:blank --disable-direct-npapi-requests --disable-web-security --disable-notifications --disable-extensions --disable-save-password-bubble about:blank'
2018-04-06 21:13:22,419 32392 INFO MainThread brozzler.chrome.Chrome.start(chrome.py:150) chrome running, pid 32398
2018-04-06 21:13:35,496 32392 INFO MainThread brozzler.chrome.Chrome._websocket_url(chrome.py:172) got chrome window websocket debug url ws://localhost:9222/devtools/page/2A6DEFB09EEA7B6250BA1AFE8B56281C from http://localhost:9222/json
2018-04-06 21:13:36,501 32392 INFO MainThread brozzler.worker.BrozzlerWorker.brozzle_page(worker.py:355) brozzling Page({"id":"e91e3b619453591798bb2cc1a4edf5a3d8bf5dcf","url":"http://example.com/",...})
2018-04-06 21:13:36,506 32392 INFO MainThread brozzler.worker.BrozzlerWorker._try_youtube_dl(worker.py:296) trying youtube-dl on Page({"id":"e91e3b619453591798bb2cc1a4edf5a3d8bf5dcf","url":"http://example.com/",...})
2018-04-06 21:13:37,142 32392 WARNING MainThread youtube_dl.report_warning(YoutubeDL.py:592) Falling back on generic information extractor.
2018-04-06 21:13:37,427 32392 ERROR MainThread youtube_dl.to_stderr(YoutubeDL.py:515) ERROR: Unsupported URL: http://example.com/
2018-04-06 21:13:37,427 32392 INFO MainThread brozzler.worker.BrozzlerWorker.brozzle_page(worker.py:383) needs browsing: Page({"id":"e91e3b619453591798bb2cc1a4edf5a3d8bf5dcf","url":"http://example.com/",...})
2018-04-06 21:13:37,928 32392 INFO MainThread brozzler.browser.Browser.navigate_to_page(browser.py:520) navigating to page http://example.com/
2018-04-06 21:13:38,429 32392 INFO MainThread brozzler.browser.Browser.screenshot(browser.py:552) taking screenshot
2018-04-06 21:13:38,931 32392 INFO MainThread root.on_screenshot(cli.py:191) wrote screenshot to /tmp/http___example_com_-20180406211338.png
2018-04-06 21:13:39,006 32392 INFO MainThread root.behavior_script(__init__.py:103) using template='umbraBehavior.js.j2' populated with parameters={'actions': [{'selector': 'button.sc-button-play, .playButton, div.soundItem, .jwlist>a'}]} for 'http://example.com/'
2018-04-06 21:13:54,010 32392 INFO MainThread brozzler.browser.Browser.run_behavior(browser.py:607) behavior decided it has finished
2018-04-06 21:13:54,010 32392 INFO MainThread brozzler.browser.Browser.extract_outlinks(browser.py:528) extracting outlinks
2018-04-06 21:13:55,516 32392 INFO MainThread root.brozzle_page(cli.py:199) outlinks:
	http://www.iana.org/domains/example
2018-04-06 21:13:55,517 32392 INFO MainThread brozzler.browser.Browser.stop(browser.py:352) shutting down websocket connection
2018-04-06 21:13:55,518 32392 INFO MainThread brozzler.chrome.Chrome.stop(chrome.py:246) terminating chrome pgid 32398
2018-04-06 21:13:56,019 32392 INFO MainThread brozzler.chrome.Chrome.stop(chrome.py:258) chrome pid 32398 exited normally

As for the image that brozzle-page http://example.com/ was supposed to create, where do I find it? I don't see any new capture in the Brozzler dashboard at http://127.0.0.1:8000/ and I can't find the capture at http://localhost:8880/brozzler/*/http://example.com/ either, and I don't see any new files in the directory I was standing in when I ran brozzle-page http://example.com/.

I see now that the output said where it put the screenshot, heh.

Still black yes.

No luck reproducing the problem so far. Tried on a mac, Chromium 65.0.3325.181 Built on Ubuntu , running on Ubuntu 16.04 (using Xvnc) and Chromium 64.0.3282.119 built on Debian 9.3, running on Debian 9.4 (using normal X with a graphics card and physical screen)

Can you apply this change

diff --git a/brozzler/browser.py b/brozzler/browser.py
index ea0063cbeb..f1814ccf70 100644
--- a/brozzler/browser.py
+++ b/brozzler/browser.py
@@ -461,7 +461,9 @@ class Browser:
                             page_url)
                         self.navigate_to_page(page_url, timeout=page_timeout)
                 if on_screenshot:
-                    self._try_screenshot(on_screenshot)
+                    for i in range(10):
+                        self._try_screenshot(on_screenshot)
+                        time.sleep(1)
                 behavior_script = brozzler.behavior_script(
                         page_url, behavior_parameters,
                         behaviors_dir=behaviors_dir)

run brozzle-page http://example.com/ again, and look at the 10 screenshots produced, to see if they're all black?