wait_for doesn't work as expected

[Wadehl]

I'm trying to scrape the information related to YouTube authors. I need to click the "…more" button to display the content of a pop-up window.

After executing the JavaScript code in the browser, I've confirmed that the pop-up window can be displayed normally. And I also found that "#links-section" is one of the newly added DOM nodes after the pop-up window is displayed asynchronously.

Based on this logic, I wrote the following Config. However, in the end, I found that the obtained HTML doesn't contain the content I expected. I really hope someone can help me and give me some directions on how to fix this problem.

Here is my code:

base_url = "https://www.youtube.com/@MrBeast" # homepage of any YouTube user.

async def main(url):
    async with AsyncWebCrawler() as crawler:
        config = CrawlerRunConfig(
            js_code="document.querySelector('button.truncated-text-wiz__absolute-button').click()",
            wait_for="""
              js:() => {
                const linksSection = document.querySelector('#links-section');
                return !!(linksSection && linksSection.textContent.trim() !== '');
              }
            """,
            simulate_user=True,
            magic=True,
            )
        
        result = await crawler.arun(
            url=url,
            config=config,
        )
        # print(result.html)