aces
[redcap] bulk importer
Brief summary of changes
This PR adds a new tool to bulk import data from REDCap linked instances/projects. It will be mostly useful for retrospective data, or to do targeted batch import.
Multiple caveats and todos:
- ~arguments need to be defined. There are in the script for now.~
- difficult for me to test as is, I need to make some edits/overrides for ibis to actually use it (difference in instrument names between LORIS/REDCap).
- I purposefully let some code as comments, these snippets are already tested and might be useful if we decide to wrangle on event/instruments names.
- REDCap has a event-instrument mapping endpoint, that conveniently has the unique_event_name and the form_name . I used the LORIS corresponding class to get these two, but again, that only works if the names are exactly the same on both ends (LORIS and REDCap).
- Participant IDs/Record IDs will also need mappings I guess if they are not clearly the same on both ends.
- we can add restrictive arguments later.
Link(s) to related issue(s)
No related issue.
Okay, my importer is now ready-to-review (see https://github.com/aces/Loris/pull/9905), and I finished my re-review of this PR.
Generally, I think my PR is more mature. This PR seems to have many TO-DOs and commented code. Also, this PR uses a "hack" in sending notifications to LORIS to import the REDCap records, whereas my PR imports the REDCap records directly without using a notification, which I think is cleaner and more maintainable.
Other than that, it seems to me that we can easily merge your importer into mine, as there are only two significant differences between our two importers:
- Your importer gets records for all REDCap projects in the configuration, while my importer only targets a single REDCap project. In C-BIG, we need to be able to run our REDCap scripts on a per-project basis. We can either let things as-is in my importer, or add a
--alloption if you really need it for your project. - Your importer gets the records to import by querying the LORIS database, whereas my importer gets the records to import by reading the configuration and querying the REDCap API, which yield different results. I think this should be an option in the importer. Something like
--select-records=query-redcap(my option) or--select-records=fill-sessions(your option), not sure about the names though.
If you agree to merge your importer into mine, we can either merge my importer first and then do a second PR to add yours, or add yours directly into my PR. I have no preference.
