tdruez

You can try to desactivate the multiprocessing with `SCANCODEIO_PROCESSES=-1` in your local .env file, and re-run the pipeline. You also want to provide your code in a branch with instructions...

@AvishrantsSh any success with disabling multiprocessing?

> But I don't think its a problem with multiprocessing, because rest of the pipelines are working smoothly but still load_inventory pipeline is unable to load the JSON generated by...

> ut there seems to be some problem that I can't get hold off. Sometimes insertion into the database would suddenly fail at get_or_create. In my case, it occured when...

@AvishrantsSh Are you still having this issue?

From 1.76 GB to 713M for the compressed tarball with current changes in the https://github.com/nexB/scancode.io/tree/350-docker-optimizations branch Also, the build is much faster and leverage caching.

``` Step 8/9 : RUN --mount=type=cache,target=/root/.cache/pip pip install . ERROR: Service 'web' failed to build : the --mount option requires BuildKit. Refer to https://docs.docker.com/go/buildkit/ to learn how to build images...

> Not creating these as packages may mean we skip on useful things. They cannot be created as packages (only resources) at the DB level since those are missing mandatory...

@PlabanKr @Pratyush1606 any changes we can make in the codebase to fix this issue for good?

A warning about the crlf issue was added in the documentation https://github.com/nexB/scancode.io/commit/37d325a03c02dc5ae9205d4b16eaf729253523ee#diff-d9b149498982c0663c3b7170398773361ed5678f1a627e9c2fd8d2c955c563dbR38-R41