benoit74

Task requested on farm.zimit.kiwix.org with too much memory

8

We have a task on farm.zimit.kiwix.org which requests more memory than mwoffliner workers have available:   I removed the task on the farm. Is it an issue to fix...

Use `docker compose` instead of `docker-compose`

3

Fix documentation, `docker-compose` command does not exists anymore now that compose is part of docker, one has to use `docker compose`

Keep logs of letsencrypt and reverse-proxy container

1

In https://github.com/openzim/wp1/blob/main/docker-compose.yml (prod configuration), logs of the reverse-proxy and letsencrypt containers are set to `logging.driver: 'none'`, i.e. we immediately drop the logs. During https://github.com/kiwix/operations/issues/211, this prevented me to diagnose an...

Impossible to ZIM files from upload.wikimedia.org

2

See https://github.com/openzim/zim-requests/issues/272#issuecomment-2466938768 Since files are hosted on upload.wikimedia.org, we must comply with their User-Agent policy at https://meta.wikimedia.org/wiki/User-Agent_policy I suggest we add a CLI option to pass a custom User-Agent to...

test_indexing_item_not_front test is failing

2

I had to skip this test which is failing for now with most recent libzim: https://github.com/openzim/python-scraperlib/blob/fef63f81fdb9dd6d2a5e17d9c8785e3fd22665e9/tests/zim/test_indexing.py#L114-L144 This looks like an upstream issue, hopefully only at read time: https://github.com/openzim/libzim/issues/981

This issue serves as a checklist for the release event. - [ ] Secure the CI is green on git `main` - [ ] Check that dependencies ranges are ok,...

[next major] Do not shadow Python standard-library module

2

Ruff / Flake8 has a new rule `A005`: https://docs.astral.sh/ruff/rules/stdlib-module-shadowing/ It is recommended to not shadow Python standard-library modules. Currently, we have 5 issues: ``` src/zimscraperlib/html.py:1:1: A005 Module `html` shadows a...

Pass proper user-agent in `stream_file` when host is `upload.wikimedia.org`

1

For files hosted on upload.wikimedia.org, we must comply with their User-Agent policy at https://meta.wikimedia.org/wiki/User-Agent_policy Doing so at scraperlib level in `stream_file` (main methods using in many scraper to download files...

Add executor to zimscraperlib

2

This PR enrich the scraperlib with a `ScraperExecutor`. This class is capable to process tasks in parallel, with a given number of worker **threads**. This executor is mainly inspired from...

MDN zimit 2 ZIM has issue with demo output iframe on Android

13

**Describe the bug** I have a test ZIM with an `iframe` with has an `srcdoc` attribute instead of the classical `src` (i.e. code is "inline"). The iframe stays blank. This...