PhenoFront and PlantCV

[AdamDimech]

This query is actually about PhenoFront, which is a 'partner' to PlantCV for those of us using LemnaTec Scanalyser systems.

I am interested in using it, but wanted to know whether it is still current? It hasn't been updated since 2014, so before embarking on a process of installing it and testing, I'd like to know whether this endeavour is worth it (or is there another, better way to directly access a LemnaTec database now?). In particular, is the code still current?

I really love PlantCV so want to extend its use to what it does best - analyse LemnaTec output data!

[nfahlgren]

Hi @AdamDimech!

I would not recommend using PhenoFront anymore, we have discontinued it locally. It just was not maintained after the developer left and eventually became problematic when things needed to be updated.

What we use now is simpler, albeit more minimal: https://github.com/danforthcenter/data-science-tools

I am working on reorganizing the repository above, so apologies for the mess. The two files that matter at the moment are LT-db-extractor.py and database.example-config. In principle, you would make a copy of database.example-config and fill in the relevant information for your database, experiment, and camera dimensions. Then you run the script with that config file as input and specify an output directory. The script connects to the database server to query out image and experimental metadata. It identifies the raw images and transfers them to the local machine via SFTP, then it converts them to PNGs.

I have only tested this on our system though, so no doubt we would need to update some things to be more flexible, in terms of camera types, raw formats, image naming, database structure, etc.

Happy to help if this sounds interesting.

[AdamDimech]

Hi @nfahlgren,

I have spent the past day working on this code to make some 'improvements which I really hope will make the script useful universally.

I am currently working on a Windows machine (I will move to Unix for production later) but the script broke because of file path errors. I also took the opportunity to make some other changes to the code, which include:

• Adding a new input flag -e for experiment, so that this doesn't need to be hard-coded into the JSON config file (and therefore a config file doesn't need to be written for each data query).
• Cleaner error reporting if the local target directory already exists
• Removal of the U flag when the database is being read, as this triggers depreciation warnings with my version of Python (DeprecationWarning: 'U' mode is deprecated).
• Change the code so that images can have "VIS", "vis" or "TV" as labels (our systems use that for the top view).
• Added a timestamp feature so that output CSV files are named in the format Experiment_SnapshotInfo_YYYYMMDDHHSS.csv. This way the CSV contains some useful metadata about the analysis run.
• Added code so that slashes are all changed to forward-slashes if the OS is Windows (therefore making the code compatible over all operating systems).
• Added a new parameter to the config file called "path" so that the home path on the LemnaTec server can be set. (Your default is /data/pgftp but mine is /data0/pgftp so it seemed useful to have this specified in a config file rather than hard-coded).
• I had to remove os.remove(local_file) as it was generating the following errors on my system: PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'T:/Output/Deleteme/snapshot4702001/blob191682'. To resolve this, I added a def to the end that contains code that recursively deletes all files containing "blob" in their names. It's the same outcome without the error.

I will soon generate a Git pull request so that my code can be reviewed by you. I have never done this before, so please be patient if I make some sort of mistake. I will use the PlantCV tutorial as a guide.

What I have not done (just yet):

• Check whether PlantCV can work with the CSV and image files.

I will update this thread as I progress.

[nfahlgren]

Hi @AdamDimech, that's awesome! I sent you an invitation to join that repository as a collaborator. That will allow you to write directly to that repository. The advantage is that you can clone it, create a new branch, commit your changes there, then push the branch back to our repo and generate the pull request directly there. It's a bit easier than doing it using a Fork as our documentation in PlantCV suggests (though that's totally fine too).

No worries about learning the pull request system, it's a bit of a process and we're happy to help.

[AdamDimech]

Hi @nfahlgren,

Thanks for the invite, which I have accepted. I have persisted with the PlantCV instructions for now as it seemed easier whilst I get my head around the mechanics of Git.

I have made a further change to those listed above. I noticed that your default OpenCV colour conversion algorithm is COLOR_BAYER_RG2BGR, whereas I had to use COLOR_BayerGR2RGB. It seemed probable that other LemnaTec systems may vary so I moved this to the JSON file as a configuration setting. Unfortunately I cannot get the names to parse into cv2.cvtColor so we have to use the int code values instead. I have added details to the README.md file.

I'd be very keen to know how this runs on your system. Hopefully nothing breaks.

Please note that I have not yet checked whether PlantCV can work with the CSV and image files. That is my next task.

[AdamDimech]

Hi @nfahlgren:

I can confirm that PlantCV has worked well with the modified code.

I have also added the ability to input a date range for the snapshots that should be downloaded (using -a and -z flags).

[nfahlgren]

Hi @AdamDimech,

Sorry for the silence on this, once I catch up on a few things early next week I will revisit the updates you have worked on. Thanks!

[AdamDimech]

I was wondering whether you'd had time to take a look at this just yet?

[nfahlgren]

Hi @AdamDimech, sorry for the delay, have just been slammed with other stuff lately. I will try to get this looked over next week. Thanks!

[dschneiderch]

@nfahlgren I picked up @AdamDimech's fork and ran with it. Perhaps most importantly, I modified the SQL query so that it checks for the blob path in image_file_table rather than assuming the timestamp date. That assumption doesn't work if the image was transferred to the database at a later date. That actually also allows the entire code to be simplified. The only downside is you'd have to write a separate query for watering data that does not have image data associated with it (easy enough though) I also generalized a few more things:

1. rotate image any way based on the saved code.
2. 8 bit PSII images based on the dataformat code
3. updated psycopg code to accept arguments per documentation
4. added flags for specific cameralabel and to append images into existing directory
5. split ssh hostname and db hostname in config file so you can run it locally

Then I changed the file structure for my own purposes and removed snapshot# directory format (but that can be added back in)

TO DO:

1. if this is to be a useful tool for multiple people, we should confirm rotate_flip_codes, data_format codes are the same between all of us.
2. give option to create snapshot# subdirectory file structure that you have
3. remove height/width from config file and read it from the sql table

anyway, happy to engage further if interested. unfortunately my rudimentary git(hub) skills may not allow for simple pull requests because my commits may be grouped into too large a branch but we could try.

[nfahlgren]

Oh wow, cool! Yeah, if we can make it into something that most people with the same kind of database can use I think that would be useful.

I'm not set on the snapshot# folder names in anyway. A previous developer came up with that format and I had built PlantCV around parsing it so it was just easier to leave that format as is when I made this program. But it's easy enough to change PlantCV to read a different format if it makes more sense to store the data another way.

One issue in the back of my mind recently is last time I looked one of the packages we are using (paramiko I think?) was only available in Python 2. Ideally we could move this tool to Python 3 since 2 is reaching end of life.

[dschneiderch]

I only use python 3 and it's no problem. my fork (EDIT: fixed link)

[nfahlgren]

Okay, cool!

[dschneiderch]

I'm not set on the snapshot# folder names in anyway. A previous developer came up with that format and I had built PlantCV around parsing it so it was just easier to leave that format as is when I made this program. But it's easy enough to change PlantCV to read a different format if it makes more sense to store the data another way.

good thing plantcv works really well with filename metadata now :-D

[nfahlgren]

haha, exactly! :)