[DL4H] Add New Example and Task for MIMIC-3 Discharge Notes Linked to Top-Level ICD-9 Categories

[frbrigagao]

Student: Fabricio Brigagao ([email protected]) Contribution: Adds a new example and a new custom task for MIMIC-3 discharge notes. Paper Title: Classifying Unstructured Clinical Notes via Automatic Weak Supervision (KeyClass) Paper Link: https://arxiv.org/abs/2206.12088

Adds a new task and example on how to create a dataset from MIMIC-III discharge summaries, following the procedure described in “Classifying Unstructured Clinical Notes via Automatic Weak Supervision”.

Highlights

• Uses a new custom task created for to solve this problem: MIMIC3DischargeNotesICD9Coding.
• Each discharge note is mapped to zero or more of the 19 top-level ICD-9 categories based on the ICD-9 codes assigned to it in the MIMIC-3 database.
• Generates train/test splits and writes files in the layout expected by KeyClass:
  • labels.txt: Names of the 19 ICD-9 top-level categories
  • train.txt: One discharge note per line (training)
  • train_labels.txt: 19-dim binary vectors for the training set
  • test.txt: One discharge note per line (test)
  • test_labels.txt: 19-dim binary vectors for the test set.

Limitation: TF-IDF filtering is not applied to the discharge note's text since the paper was unclear on this aspect and did not provide the respective source code (possible future improvement).

Files to Review

• pyhealth/datasets/configs/mimic3.yaml: Added needed fields from the NOTEEVENTS and DIAGNOSES_ICD tables.
• pyhealth/tasks/discharge_notes_coding.py: New task to generate the dataset.
• pyhealth/tasks/__init__.py: Added import for the new task.
• examples/discharge_notes_top_level_icd9_cats.py: New example using the new task to generate the final files.
• requirements.txt: Used latest dependencies provided by the instructor (https://github.com/sunlabuiuc/PyHealth/pull/380).

Testing

The new task and dataset generation can be tested by running the examples/discharge_notes_top_level_icd9_cats.py.

Both were tested on the full MIMIC database and generated the same number of records for train (39,541) and test (13,181) splits as listed on Table 2 of the paper.