Generate subsampling config with a script

[victorlin]

Description of proposed changes

Shift to having one subsampling configuration YAML per build.

Motivations:

1. The current approach of copy/pasting is getting unwieldy for making additions/modifications to subsampling config.
2. There have been recent efforts to design a generic subsampling tool + config schema. If we are to use such a tool in this workflow, the {region} templating must be lifted out of Snakemake so that a complete YAML config file can be given to the subsampling tool. Without a script, this would result in even more copy/pasting.

Checklist

• [x] Start using augur subsample here?
• [ ] Consider pulling out other duplicated config (builds, traits, frequencies)?
• [ ] Test with trial builds
• [ ] Changelog?
• [ ] Release new version of workflow?

---

scratch

dropped:

• a8f66bb0bef78db9fad4c3b1e6df3ca3aa164814
• 52501d66ba2aaa3774245a8711cad2c323269747

[trvrb]

I'm afraid I'm confused as to the push here. If you were to implement weighted subsampling in augur (per https://github.com/nextstrain/augur/issues/1318#issuecomment-2013653668), then we could take the existing YAML files like https://github.com/nextstrain/ncov/blob/master/nextstrain_profiles/nextstrain-gisaid/builds.yaml#L672 and slim them down considerably. I find the pattern of script generated intermediate files confusing and only want them when really necessary. Again, I'm confused here as it would seem that any work to templatize this would want to wait until augur subsample exists (otherwise we're just spinning our wheels).

[trvrb]

Okay... maybe I'm catching up. The current YAML files generated by nextstrain_profiles/nextstrain-gisaid/generate-subsampling-config.py like nextstrain_profiles/nextstrain-gisaid/subsampling/global_6m.yaml are meant as just a refactor? And that these current YAML files would be replaced by the version that would be compatible with a future augur subsample? I have to admit, it's still seeming like work that doesn't need to be done yet, but I don't have a great sense for how you're planning to stage things.

[victorlin]

The current YAML files … are meant as just a refactor? And that these current YAML files would be replaced by the version that would be compatible with a future augur subsample?

Yes, I'm planning to bridge the gap between current setup and augur subsample with a translation to individual subsampling config YAMLs which will then be modified directly to see the difference in config schema.

I have to admit, it's still seeming like work that doesn't need to be done yet, but I don't have a great sense for how you're planning to stage things.

Fair point. Before seeing your comment (was offline on an airplane), I continued and expanded the script to cover all profiles/builds. By then, I also started thinking that the scoping of this PR isn't right.

I'm not sure how where to take this PR - maybe it should wait and augur subsample be added here directly. So far, it's been useful exploring these ideas:

1. Decoupling subsampling config from Snakemake config. This is necessary in order to test augur subsample on this workflow.
2. Deduplicating subsampling config among profiles. Identical subsampling config can continue to exist separately across profiles such as gisaid/gisaid-21L/open, but now is a good opportunity to get rid of that duplication and define subsampling config in a place that's more shared, especially if we'll need to bulk update config schema for augur subsample.
3. Dissecting subsampling logic. I made the script dynamically calculate sample sizes using weights, which is helpful in thinking about how to implement weighted subsampling even if we don't end up merging the script here.

[victorlin]

Update: I've moved 3b60b825f7d0c3f94d40f202180c9ab492dca623 over to #1106. Currently focusing on that instead of the larger subsampling changes in this PR.

Closing this for now.