signal only works in main thread

[rosendyakov]

Hello, I'm currently developing a very simple Flask app, running only locally and I wanted to scrape some Reddit posts, using your API. I followed the example, as it's specified in the documentation, however whenever I run my script, I get the following error:

ValueError: signal only works in main thread

I read that Flask-SocketIO package causes this, but I saw that this project uses Websocket-client, which is a different package.

Would really appreciate your input.

[mattpodolak]

hey @rosendyakov can you provide the following info, and the minimum amount of code needed to re-create the issue? This will help me as I look into this further:

python version: flask version: pmaw version:

[SeifReda30]

I have the same issue in deploying a web application using pmaw

File "/app/adam-radar/Python-Scripts/User Specified Scripts/Discussion Platforms/Reddit/reddit_submissions_by_keywords.py", line 111, in reddit_submissions api_request_generator = api.search_submissions(q=keyword,after=start_time,before=end_time) File "/home/appuser/venv/lib/python3.9/site-packages/pmaw/PushshiftAPI.py", line 77, in search_submissions return self._search(kind="submission", **kwargs) File "/home/appuser/venv/lib/python3.9/site-packages/pmaw/PushshiftAPIBase.py", line 304, in _search self.req.check_sigs() File "/home/appuser/venv/lib/python3.9/site-packages/pmaw/Request.py", line 110, in check_sigs signal.signal(getattr(signal, "SIG" + sig), self._exit) File "/usr/local/lib/python3.9/signal.py", line 56, in signal handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))

[mike-mo]

I am hitting the same symptom, though my setup is a little bit more involved (Azure Durable Functions), so in order to make up for the added complexity I published my repro to https://github.com/mike-mo/azure-durable-pmaw

Python version: 3.10.10 Azure Functions Core Tools Version: 4.0.5030 Azure Functions Runtime Version: 4.15.2.20177 pmaw version: 3.0.0

Same Python version and pmaw version work fine to run a basic script that fetches the information. It must be something to do with how threading is handled by these frameworks.

[CryptoRahino]

having the same issue here, i'm using multiprocessing.pool.ThreadPool to call the api function `def run_download(subreddits: list, start_date: int, end_date: int, additional_args: dict, working_dir: Path = None) -> DataFrame: logger.info(f"Starting Download from Reddit using subreddits {subreddits}") all_df = DataFrame()

with ThreadPool() as pool:
    query = {'start_date': start_date,
             'end_date': end_date,
             "working_dir": working_dir,
             **additional_args}
    lst_df =pool.starmap(_get_subreddit, [(start_date,end_date,  subreddit) for subreddit in subreddits])

    for df in lst_df:
        if df.empty:
            continue
        all_df = concat([df, all_df], axis=0)

def _get_subreddit(self, start_time: int, end_time: int, subreddit_name=None, **kwargs) -> DataFrame:
    params = self.params.copy()
    params.update(kwargs)
    subreddit_name = subreddit_name or self.subreddit_name
    df = DataFrame(
        self.api.search_submissions(subreddit=subreddit_name, since=start_time, until=end_time, **params))
    return df.drop_duplicates('id')
        
        `

i have tried this with async function but it also didn't help. ValueError: signal only works in main thread of the main interpreter

[simoninnyc]

Did you manage to solve this? Running into a similar issue using an Azure Durable Function with Scrapy and signal