Chrome Driver for Fetching Web Content

[pedramamini]

What do you need?

This is a feature request with example code that should be able to be dropped right into installer/client/cli/:

https://gist.github.com/pedramamini/e1f7f9dc6013734fca44961cca4e7890

CLI tool and library for fetching content via Chrome driven by Selenium. Has some (rudimentary) tricks up its sleeve to evade mechanized browser detection.

Requirements

pip install selenium
pip install webdriver_manager

Usage

Usage:
    usage: chrome_fetch.py [-h] [--sleep SLEEP] [--headless] [--debug] [--referrer [REFERRER]]
                           [--human]
                           url
    Fetch the inner text of a webpage.
    positional arguments:
      url                   URL of the webpage to fetch
    options:
      -h, --help            show this help message and exit
      --sleep SLEEP         Time to wait in-between operations
      --headless            Run in headless mode.
      --debug               Enable debug mode.
      --referrer [REFERRER]
                            Referrer URL to start from (default: https://www.google.com).
      --human               Mimick human behavior with mouse

Example

$ chrome_fetch.py https://inquest.net/blog/around-we-go-planet-stealer-emerges/ | summarize
ONE SENTENCE SUMMARY:
Planet Stealer, a new information-stealing trojan targeting sensitive data, highlights the evolving threat landscape and the importance of cybersecurity vigilance.

MAIN POINTS:
1. Planet Stealer is an emerging information-stealing trojan recently documented and sold in underground forums.
2. Implemented in Go, it aims to collect and exfiltrate sensitive information from compromised hosts.
3. It's part of the malware-as-a-service ecosystem, appealing to adversaries for data theft and sale.
4. The malware targets browser information, cryptocurrency wallets, and messenger credentials among others.
5. Features include sandbox evasion and data exfiltration via Telegram, indicating sophisticated capabilities.
6. Distributed as EXE files, often via loader trojans, with active command & control servers noted.
7. Communication with C2 servers uses HTTP API with JSON data, suggesting modern backend infrastructure.
8. Samples of Planet Stealer have been observed in the wild, packed with UPX for obfuscation.
9. Countermeasures include network-based detection systems and real-time threat intelligence application.
10. InQuest credits open-source intelligence for disclosing details about Planet Stealer, emphasizing community collaboration in threat intelligence.

TAKEAWAYS:
1. The emergence of Planet Stealer underscores the continuous innovation in malware development and distribution.
2. Information stealers remain a significant part of the cybercrime ecosystem due to their lucrative potential.
3. Effective cybersecurity measures require comprehensive network-based detection and real-time threat intelligence.
4. Collaboration and sharing of threat intelligence within the cybersecurity community are crucial for timely identification and mitigation of new threats.
5. Enterprises should enhance their security posture to protect against sophisticated threats like Planet Stealer through advanced detection capabilities and informed threat intelligence.

[nicolas-g]

👍

[joaomorossini]

That would be great!

[matigumma]

i think there are a better approach.. using jina.ai reader api. its fast, easy and not need to run any additional code.

curl https://r.jina.ai/https://inquest.net/blog/around-we-go-planet-stealer-emerges/ | fabric -p summarize

iv tested it with jina.ai website curl https://r.jina.ai/https://jina.ai/ | fabric -c -p summarize im using -c to add context that translate output for me to spanish ;)

enjoy

[pedramamini]

@matigumma love it! I've added this to my .zshrc and prefer to use it since, by nature, we're grabbing public resources here anyway:

jf ()
{
    curl -s https://r.jina.ai/$1
}

[joaomorossini]

Thanks for the suggestion, @matigumma . I found Jina Reader to be very useful. One thing that intrigues me, though, is that I have not logged in to the service, nor have I set the API key anywhere, but still Jina AI kind of knows that I'm the one making the cools via my Terminal. Do you guys know how their authentication works? I don't remember setting up any credentials

[matigumma]

Thanks for the suggestion, @matigumma . I found Jina Reader to be very useful. One thing that intrigues me, though, is that I have not logged in to the service, nor have I set the API key anywhere, but still Jina AI kind of knows that I'm the one making the cools via my Terminal. Do you guys know how their authentication works? I don't remember setting up any credentials

docs:

https://jina.ai/reader

i think u can do with a bash script curl 'https://r.jina.ai/https://example.com' -H "Authorization: Bearer jina_api_key"

[eugeis]

You can use

-u, --scrape_url= Scrape website URL to markdown using Jina AI

[fabioscarsi]

What about paywalls? Does Jina work for pages protected by paywalls (which are really a lot of our readings)?