Xueqin Cui
Xueqin Cui
we probably need `go.mod` as well
Dependabot does [support](https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#package-ecosystem) git submodule - I just updated my experimental [dependabot.yml](https://github.com/cuixq/osv.dev/blob/dependabot/.github/dependabot.yml) to see if dependabot can help with this.
I would like to keep this open to remind us to get rid of the code completely in v2.
@michaelkedar shall we review this PR to make sure everything is up to date?
Cloud Profiler is set for test instance: https://github.com/google/osv.dev/pull/2941 And `build-essential` is required as per [documentation](https://cloud.google.com/profiler/docs/profiling-python): https://github.com/google/osv.dev/pull/2944
@spencerschrock thanks for the profiling - I will take a look.
it seems [`json.Unmarshal`](https://github.com/google/osv-scalibr/blob/main/clients/datasource/pypi_registry.go#L92) is quite expensive (which is a bit surprise to me). one potential improvement I can think of is to cache the marshalled struct instead of the response...
> we also observed a huge increase in memory usage (from 6GB to 40GB). Does the cache ever get emptied? The cache probably is not emptied until a full run...
> Perhaps you and I should try the jsonv2 experiment? Yes I am also looking at other `json` packages - e.g. [json-iterator/go](https://github.com/json-iterator/go) and [goccy/go-json](https://github.com/goccy/go-json)
I think that is our plan to have more fine-grain control over which plugins (including enrichers) to use in osv-scanner.