Ivan Kozik

I suspect this might happen when a huge binary file gets parsed as HTML, but I need to test it.

The URL would be helpful, if you can share it. If this happens with just wpull as well, it's probably worth filing a wpull bug for chfoo to look at....

Also, I never see this on a machine with 32GB of RAM, so it would be helpful to know how little RAM is needed to reproduce the bug for a...

That is going to be pretty hopeless in general because wpull/grab-site processes start out using ~55MB but typically have heap growth and end up using ~140MB-400MB RAM per crawl. (The...

I would certainly be happy if someone took on that project, as I have no time to maintain grab-site at the moment. I forked to ludios_wpull when it looked like...

I use an ssh tunnel for this: ``` autossh -f -C -L 127.0.0.1:29000:127.0.0.1:29000 user@hostname -N ``` Remember to use `GRAB_SITE_INTERFACE=127.0.0.1` as well when running `gs-server` to avoid leaking the dashboard...

Hmm, actually, an nginx setup might be tricky because you would also have to reverse-proxy the WebSocket endpoint: http://nginx.org/en/docs/http/websocket.html - which I have not tested at all.

It shouldn't be too difficult to add SSL support to `gs-server`. It looks like [create_server here](https://github.com/ludios/grab-site/blob/987893eeff0d7603f5a5a958d1da3e251f2cc28d/libgrabsite/server.py#L104) just needs to get passed an SSL context: https://docs.python.org/3/library/asyncio-eventloop.html#creating-listening-connections This would not provide as...

That sounds about right. Users should be able to have `gs-server` listen on both TCP and SSL since they might want to avoid doing SSL between `grab-site` instances and `gs-server`,...

I never meant that the TCP and SSL listener have to be on the same port, just that it should be possible to have both of them running on different...