cdx_toolkit icon indicating copy to clipboard operation
cdx_toolkit copied to clipboard
verified publisher icon cocrawler

feat: Adding settings, utils, write to S3, and writer close

Open malteos opened this issue 1 month ago • 3 comments

This PR integrates a couple of general changes from the EOT PR (https://github.com/cocrawler/cdx_toolkit/pull/54):

  • Settings variables are loaded from environment variables in settings.py
  • Common CLI methods are moved to utils.py
  • Read and write to S3 (via fsspec).
  • Missing writer.close() statement is added to CLI and example.

malteos avatar Nov 19 '25 11:11 malteos