(Backend) Push pfdcm container fs data into ChRIS swift fs

[mairin]

Currently the ChRIS pacs service (pfdcm?) collects data from PACS and it resides inside its container. We need a backend dev to implement a data push from that container's fs into ChRIS' swift filesystem. (Could set up orthanc to act like a pacs to mimic for dev/testing.)

[aravindhp]

Continued working with @rudolphpienaar on this. We now have a development environment on Azure setup where we can bring up pfdcm and orthan. Now when querying pfdcm <==> orthanc using, pfurl, we are running into the following issue:

aravindh@aravindh-chris:~/FNNDSC/pfdcm$ docker run --rm --name pfurl fnndsc/pfurl --verb POST                             \
>         --raw                                   \
>         --http ${HOST_IP}:4055/api/v1/cmd       \
>         --httpResponseBodyParse                 \
>         --jsonwrapper 'payload'                 \
>         --msg '{  
>             "action": "PACSinteract",          
>             "meta": {
>                 "do":  "query",
>                 "on" : {
>                    "PatientID": "LILLA-9731"
>                 },
>                 "PACS" : "orthanc"
>             }
>         }'
{
    "msg": "Error in Class 'DCMhandler' calling method 'PACSinteract_process' on:\n(<class 'TypeError'>, TypeError('must be str, not int',), <traceback object at 0x7f34645a0808>)\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.6/dist-packages/pfstorage/pfstorage.py\", line 950, in do_POST_actionParse\n    d_actionResult      = method(request = d_msg)\n  File \"/usr/local/pfdcm/bin/../pfdcm/pfdcm.py\", line 727, in PACSinteract_process\n    d_ret = self.dcm.PACSinteract_process(*args, **kwargs)\n  File \"/usr/local/pfdcm/bin/../pfdcm/pfdcm.py\", line 642, in PACSinteract_process\n    d_ret       = pypx.find({**d_service, **d_on})\n  File \"/usr/local/lib/python3.6/dist-packages/pypx/__init__.py\", line 10, in find\n    return Find(opt).run(opt)\n  File \"/usr/local/lib/python3.6/dist-packages/pypx/find.py\", line 474, in run\n    'QueryRetrieveLevel':   'STUDY'\n  File \"/usr/local/lib/python3.6/dist-packages/pypx/base.py\", line 96, in systemlevel_run\n    str_cmd         = f_commandGen(opt)\n  File \"/usr/local/lib/python3.6/dist-packages/pypx/find.py\", line 450, in findscu_command\n    self.commandSuffix()\n  File \"/usr/local/lib/python3.6/dist-packages/pypx/base.py\", line 113, in commandSuffix\n    command_suffix +=   ' '         + self.serverPort\nTypeError: must be str, not int\n",
    "status": false
}

On the pfdcm container output we see:

2020-11-06 19:27:34  |    84029d0f8d24 |        pfstorage.py:pfdcm.do_POST_dataParse() | 
<----
Headers received = 
Host: 10.1.2.10:4055
User-Agent: PycURL/7.43.0.6 libcurl/7.68.0 OpenSSL/1.1.1f zlib/1.2.11 brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.2.0) libssh/0.9.3/openssl/zlib nghttp2/1.40.0 librtmp/2.3
Accept: */*
Mode: control
Content-Length: 118
Content-Type: application/x-www-form-urlencoded
<----
2020-11-06 19:27:34  |    84029d0f8d24 |        pfstorage.py:pfdcm.do_POST_dataParse() | Parsing JSON data...
2020-11-06 19:27:34  |    84029d0f8d24 |                  pfstorage.py:pfdcm.do_POST() | d_msg = 
{
    "action": "PACSinteract",
    "meta": {
        "do": "query",
        "on": {
            "PatientID": "LILLA-9731"
        },
        "PACS": "orthanc"
    }
}
2020-11-06 19:27:34  |    84029d0f8d24 |      pfstorage.py:pfdcm.do_POST_actionParse() | verb: PACSinteract detected.
2020-11-06 19:27:34  |    84029d0f8d24 |      pfstorage.py:pfdcm.do_POST_actionParse() | method to call: PACSinteract_process(request = d_msg) 
2020-11-06 19:27:34  |    84029d0f8d24 |      pfstorage.py:pfdcm.do_POST_actionParse() | Error in Class 'DCMhandler' calling method 'PACSinteract_process' on:
(<class 'TypeError'>, TypeError('must be str, not int',), <traceback object at 0x7f34645a0808>)
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/pfstorage/pfstorage.py", line 950, in do_POST_actionParse
    d_actionResult      = method(request = d_msg)
  File "/usr/local/pfdcm/bin/../pfdcm/pfdcm.py", line 727, in PACSinteract_process
    d_ret = self.dcm.PACSinteract_process(*args, **kwargs)
  File "/usr/local/pfdcm/bin/../pfdcm/pfdcm.py", line 642, in PACSinteract_process
    d_ret       = pypx.find({**d_service, **d_on})
  File "/usr/local/lib/python3.6/dist-packages/pypx/__init__.py", line 10, in find
    return Find(opt).run(opt)
  File "/usr/local/lib/python3.6/dist-packages/pypx/find.py", line 474, in run
    'QueryRetrieveLevel':   'STUDY'
  File "/usr/local/lib/python3.6/dist-packages/pypx/base.py", line 96, in systemlevel_run
    str_cmd         = f_commandGen(opt)
  File "/usr/local/lib/python3.6/dist-packages/pypx/find.py", line 450, in findscu_command
    self.commandSuffix()
  File "/usr/local/lib/python3.6/dist-packages/pypx/base.py", line 113, in commandSuffix
    command_suffix +=   ' '         + self.serverPort
TypeError: must be str, not int
2020-11-06 19:27:34  |    84029d0f8d24 |      pfstorage.py:pfdcm.do_POST_actionParse() | 
---->
{
    "status": false,
    "msg": "Error in Class 'DCMhandler' calling method 'PACSinteract_process' on:\n(<class 'TypeError'>, TypeError('must be str, not int',), <traceback object at 0x7f34645a0808>)\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.6/dist-packages/pfstorage/pfstorage.py\", line 950, in do_POST_actionParse\n    d_actionResult      = method(request = d_msg)\n  File \"/usr/local/pfdcm/bin/../pfdcm/pfdcm.py\", line 727, in PACSinteract_process\n    d_ret = self.dcm.PACSinteract_process(*args, **kwargs)\n  File \"/usr/local/pfdcm/bin/../pfdcm/pfdcm.py\", line 642, in PACSinteract_process\n    d_ret       = pypx.find({**d_service, **d_on})\n  File \"/usr/local/lib/python3.6/dist-packages/pypx/__init__.py\", line 10, in find\n    return Find(opt).run(opt)\n  File \"/usr/local/lib/python3.6/dist-packages/pypx/find.py\", line 474, in run\n    'QueryRetrieveLevel':   'STUDY'\n  File \"/usr/local/lib/python3.6/dist-packages/pypx/base.py\", line 96, in systemlevel_run\n    str_cmd         = f_commandGen(opt)\n  File \"/usr/local/lib/python3.6/dist-packages/pypx/find.py\", line 450, in findscu_command\n    self.commandSuffix()\n  File \"/usr/local/lib/python3.6/dist-packages/pypx/base.py\", line 113, in commandSuffix\n    command_suffix +=   ' '         + self.serverPort\nTypeError: must be str, not int\n"
}
---->
172.20.0.1 - - [06/Nov/2020 19:27:34] "POST /api/v1/cmd HTTP/1.1" 200 -

@rudolphpienaar suspects that changes in pypx is causing this.

[junaruga]

@aravindhp Could you explain more about "Azure setup"? Do we have the development environment on Azure cloud?

[aravindhp]

@junaruga I have an Ubuntu VM on Azure where I am doing this development as I did not want to change anything on my local dev machine. There is nothing special about the VM.

[junaruga]

@aravindhp OK. I understood it. Thanks for the explanation.

[aravindhp]

After applying https://github.com/FNNDSC/pypx/pull/11, I ran into the following problem:

aravindh@aravindh-chris:~/FNNDSC/pypx$ export HOST_IP=10.1.2.10
aravindh@aravindh-chris:~/FNNDSC/pypx$ docker run --rm --name pfurl fnndsc/pfurl --verb POST --raw --http ${HOST_IP}:4055/api/v1/cmd --httpResponseBodyParse --jsonwrapper 'payload' --msg '{"action": "PACSinteract", "meta": { "do":  "query", "on" : { "PatientID": "4780041" }, "PACS" : "orthanc" }}' -v 5
<snip>
{
    "msg": "Error in Class 'DCMhandler' calling method 'PACSinteract_process' on:\n(<class 'TypeError'>, TypeError('string indices must be integers'), <traceback object at 0x7f5952b32a40>)\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.8/dist-packages/pfstorage/pfstorage.py\", line 950, in do_POST_actionParse\n    d_actionResult      = method(request = d_msg)\n  File \"/usr/local/pfdcm/bin/../pfdcm/pfdcm.py\", line 727, in PACSinteract_process\n    d_ret = self.dcm.PACSinteract_process(*args, **kwargs)\n  File \"/usr/local/pfdcm/bin/../pfdcm/pfdcm.py\", line 642, in PACSinteract_process\n    d_ret       = pypx.find({**d_service, **d_on})\n  File \"/src/pypx/pypx/__init__.py\", line 10, in find\n    return Find(opt).run(opt)\n  File \"/src/pypx/pypx/find.py\", line 495, in run\n    'StudyInstanceUID':     study['StudyInstanceUID']['value']\nTypeError: string indices must be integers\n",
    "status": false
}
<snip>

There seems to be couple more problem with pypx. For one, systemlevel_run() is not checking if a command executing successfully or not. Opened https://github.com/FNNDSC/pypx/issues/12 to tack this issue. Further debugging revealed that findscu command is trying to talk to 127.0.0.1:4242. Example:

/usr/bin/findscu -xi -S  -k AccessionNumber -k AcquisitionProtocolDescription -k AcquisitionProtocolName -k InstanceNumber -k ModalitiesInStudy -k Modality -k NumberOfSeriesRelatedInstances -k PatientAge -k PatientBirthDate -k "PatientID=4780041" -k PatientName -k PatientSex -k PerformedStationAETitle -k ProtocolName -k "QueryRetrieveLevel=STUDY" -k SeriesDate -k SeriesDescription -k SeriesInstanceUID -k StudyDate -k StudyDescription -k StudyInstanceUID  -aec ORTHANC -aet CHIPS 127.0.0.1 4242

even though the input specified is different. I suspect that the input is getting clobbered during initialization.