DRAFT: tool for sharding fastqs

[tfenne]

[tfenne]

Looks like Rust I would have written, which may be a terrible thing, or a great thing!

Would you humor me and allow me to install CodeRabbit into this repo so I can see what that AI would say?

Go for it!

[clintval]

@coderabbitai review

[coderabbitai[bot]]

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai[bot]]

Walkthrough

This PR introduces a new Shard command that splits input FASTQ files into multiple sharded output sets using round-robin record distribution. The implementation includes reader and writer pool builders, progress tracking via ProgLog, and BGZF compression support. Dependencies are updated to newer patch and minor versions across multiple crates (ahash, bstr, enum_dispatch, env_logger, flate2, itertools, log, mimalloc, read-structure, rstest, seq_io, thiserror, and proglog).

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

• build_readers and build_writer_pool logic: verify error handling and resource cleanup paths, especially for file I/O and pooled writer initialization
• Round-robin record distribution: confirm correctness of shard assignment loop and that all records map to expected outputs
• Writer pool lifecycle: ensure proper graceful shutdown of writers and pool, check for potential deadlocks or resource leaks
• Unit tests: validate test helpers generate correct synthetic data and assertions cover edge cases (empty files, single vs. multiple inputs)
• Dependency updates: spot-check compatibility, particularly proglog's new pretty_counts feature usage and flate2 version jump (1.0.25 → 1.1.4)

Pre-merge checks and finishing touches

❌ Failed checks (1 inconclusive)

Check name	Status	Explanation	Resolution
Description Check	❓ Inconclusive	No pull request description was provided by the author. Since there is no descriptive content to evaluate, there is insufficient information to determine whether a description would be related to or off-topic from the changeset. While the check is intentionally lenient, a completely absent description prevents evaluation of relevance.	The author should provide a brief description of the changes, even if minimal. This could include context on what the shard command does, why it was added, or any relevant implementation notes that would help reviewers understand the intent.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title Check	✅ Passed	The title "DRAFT: tool for sharding fastqs" clearly summarizes the main change in the pull request: the addition of a new Shard command that implements FASTQ file sharding functionality. The title is concise, specific, and directly reflects the primary purpose of the changeset—introducing a tool to split FASTQ files into sharded output sets across multiple threads and compression formats. The DRAFT prefix appropriately indicates the work-in-progress status.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✨ Finishing touches

• [ ] 📝 Generate docstrings

🧪 Generate unit tests (beta)

• [ ] Create PR with unit tests
• [ ] Post copyable unit tests in a comment
• [ ] Commit unit tests in branch tf_shard

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[clintval]

I just noticed CodeRabbit otherwise would have duplicated three of my comments and provides that report in one of the Markdown drop downs. Pretty cool.