Webscraping the site has been futile, is this on purpose ?

[joshdgeek]

i have been building a little project where i gather information about market movers across different online platform, i wanted to use nitter to webscrape daily tweets of a few personnel but as soon as i run the script i get some error stating that an element i clearly identified in dev tools when i inspect elements does not exists, i thought of putting a timeout but it doesn't matter if it's up to a minute, it will always not locate it. I switched to setting headless to false {Puppeteer js) and as soon as the page opens it closes instantly..

is this from my end or nitter's:

original script:

`

const getNews = async () => {
    const browser = await puppeteer.launch({ headless: false });
    const page = await browser.newPage();

    await page.goto("https://nitter.net/xyz", {
        waitUntil: "networkidle2",
    })

    await page.waitForSelector(".timeline");

    const tweetData = await page.$$eval(".timeline", (statusEls) => {
        return statusEls.slice(0, 1).map(tweet => {
            const tweetEL = tweet.querySelector(".tweet-body > .tweet-content")
            const title = tweetEL ? tweetEL.textContent : null;

            //IMAGE FILE
            const tweetImageEl = tweet.querySelector(".attachments > img");
            const image = tweetImageEl ? tweetImageEl.getAttribute("src") : null;

            return ({ title, image, link: null, sourceName: "X" })
        })
    })

    await browser.close();

    const nitterTweet = tweetData.map(truth => ({
        ...tweetData,
        time: timeAndDate.dateTime()
    }))

    return nitterTweet
}

`

[zedeus]

Yes, I have put a lot of effort into preventing scraping of nitter.net, do NOT scrape any public instances. Host your own if you need to do this, or use any of the available libraries like twikit.

[animegrafmays]

yes, as zedeus said. we all have spent a lot of time making sure the nitter services we host is available for all to use as the software was intended. you will likely catch automated bans from most public instances listed on the wiki/status page