ioHub
ioHub copied to clipboard
isolver
Specify absolute paths in experiment_config.yaml
Currently, paths to experiment resources and results are relative to the experiment directory. It would be nice if we could specify absolute paths if someone wanted to store data or pull resources from outside of the working directory.
- Check that dir exists. If it does not, show dog and ask if it could be created or cancel exp.
- I'd hdf5 file exists, check it is for current exp, else give dialog asking whether to replace or exit exp.
Here is a proposed new format for specifying the experiments virtual folder structure. This would be in the experiment_config.yaml:
directories:
- experiment:
root:
type: [r,]
file_types:
- .yaml
- results:
root: D:/results/iohub_experiments/<code>
type: [w,]
file_types:
- .hdf5
directories:
- session:
root: <session_defaults.code>
type: [w,]
file_types:
- .xls
- .edf
- resources:
root: resources
directories:
- condition_files:
root: condition_files
type: [r,]
file_types:
- .xls
- images:
root: images
type: [r,]
file_types:
- .png
- .jpeg
- audio:
root: audio
type: [r,]
file_types:
- .wav
- .mp4
Where:
- Each element of the list under 'directories' specifies a virtual directory with the name of the element, so in this example the three 'top level' virtual directories are 'experiment', results', and 'resources'.
- A virtual directory can itself specify 'directories', that would become child folders of that virtual dir.
- The 'root' attribute defines the physical system directory path for the virtual folder:
- If root is empty, the directory script directory is used.
- If root is an absolute path, it gets used as such. For example the 'results' v. folder gives an absolute path.
- If root is a not an absolute path or empty, then it is treated and add the parent path + the current root gives the virtual folders system directory. For example, 'images' would have an expected system path of <experiment_script_path>/resources/images.
- If a path string contains a <....>, this is replaced by the runtime value of the associated experiment_config parameter. So for example, if the experiment_config.yaml defined the code: parameter to be 'my_exp_1', the results virtual folder would have a path of [experiment_script_path]/results/my_exp_1
- The type attribute tells the experiment if the folder can contain files that it may want to 'read' from or 'write' to.
- The file_types attribute specifies the file extensions expected to be in the given virtual folder.
- When the experiment looks for possible locations to find an image called 'picture.png' for example, then it would look in any virtual folder path path is type 'r' (read) and has '.png' in the list of possible file types.
- Within your experiment, you can specify a virtual folder by using io.vfolders.[folder_name].[folder_name] .... For example, using the above v. folder definition, to get to the images v. folder you could go images_vfolder = io.vfolders.resources.images. You could then say images_vfolder.getFiles() to get all the files in that vfolder, or images_vfolder.getSystemPath() to get the physical path of the v. folder.
- Within your experiment you can also get all the vfolders that a given file type can be in: io.file_vfolders.png would return [images,], since only 1 vfolder is defined as a possible .png file location. If 2 vfolders had been defined to contain png's, the result would be [images1,images2 for example].
- Within your experiment you can also get all the system paths that a given file type can be in: io.file_dirs.png would return ['<experiment_script_path>/resources/images',] for example.
- Finally, all the files for a given type can be retrieved as the absolute path to every file that exists of that type by: io.file_paths.png. If the image folder had 3 images in it, called 'p1.png', 'p2.png','p3.png', then the result of that reference would be: ['<experiment_script_path>/resources/images/p1.png','<experiment_script_path>/resources/images/p2.png','<experiment_script_path>/resources/images/p3.png']
Let me know if you see any issues with this, ways to make it simpler, or if you do not think it will meet your requirements. Once I know I will plan implementation.
This will also address the bug about 'session folders being empty' and the 'hdf5 file is split to 2 locations' bug.
The string representation of the above directories config is the following based on the current prototype code:
Processing File: D:\Dropbox\DEV\ioHub\examples\ioHubAccessDelayTest\experiment_Config.yaml
Virtual Folder:
Name: vfolders
System Path: D:\ioHub\examples\ioHubAccessDelayTest
File Type Filter: []
Type: ['r', 'w']
Virtual Folder:
Name: experiment
System Path: D:\Dropbox\DEV\ioHub\examples\ioHubAccessDelayTest\
File Type Filter: ['.yaml']
Type: ['r']
Virtual Folder:
Name: results
System Path: D:/results/iohub_experiments/<code>
File Type Filter: ['.hdf5']
Type: ['w']
Virtual Folder:
Name: session
System Path: D:/results/iohub_experiments/<code>\<session_defaults.code>
File Type Filter: ['.xls', '.edf']
Type: ['w']
Virtual Folder:
Name: resources
System Path: D:\Dropbox\DEV\ioHub\examples\ioHubAccessDelayTest\resources
File Type Filter: []
Type: ['r', 'w']
Virtual Folder:
Name: condition_files
System Path: D:\Dropbox\DEV\ioHub\examples\ioHubAccessDelayTest\resources\condition_files
File Type Filter: ['.xls']
Type: ['r']
Virtual Folder:
Name: images
System Path: D:\Dropbox\DEV\ioHub\examples\ioHubAccessDelayTest\resources\images
File Type Filter: ['.png', '.jpeg']
Type: ['r']
Virtual Folder:
Name: audio
System Path: D:\Dropbox\DEV\ioHub\examples\ioHubAccessDelayTest\resources\audio
File Type Filter: ['.wav', '.mp4']
Type: ['r']
It definitely satisfies the original request. I like the idea of passing runtime vars in to separate data based on session ids or experiment versions. Though specifying folder privileges and file_types is really neat I'm imagining those would stick with the defaults for most uses (or mine at least). If you're looking to simplify, that would be my suggestion.
…………………… Pierce Edmiston [email protected] sapir.psych.wisc.edu
On Apr 16, 2013, at 2:43 PM, Sol Simpson wrote:
Here is a proposed new format for specifying the experiments virtual folder structure. This would be in the experiment_config.yaml:
directories: - experiment: root: type: [r,] file_types: - .yaml - results: root: D:/results/iohub_experiments/
type: [w,] file_types: - .hdf5
directories: - session: root: <session_defaults.code> type: [w,] file_types: - .xls - .edf
- resources: root: resources directories: - condition_files: root: condition_files type: [r,] file_types: - .xls
- images: root: images type: [r,] file_types: - .png - .jpeg - audio: root: audio type: [r,] file_types: - .wav - .mp4 Where:#. Each element of the list under 'directories' specifies a virtual directory with the name of the element, so in this example the three 'top level' virtual directories are 'experiment', results', and 'resources'. #. A virtual directory can itself specify 'directories', that would become child folders of that virtual dir. #. The 'root' attribute defines the physical system directory path for the virtual folder: #. If root is empty, the directory script directory is used. #. If root is an absolute path, it gets used as such. For example the 'results' v. folder gives an absolute path. #. If root is a not an absolute path or empty, then it is treated and add the parent path + the current root gives the virtual folders system directory. For example, 'images' would have an expected system path of /resources/images. #. If a path string contains a <....>, this is replaced by the runtime value of the associated experiment_config parameter. So for example, if the experiment_config.yaml defined the code: parameter to be 'my_exp_1', the results virtual folder would have a path of [experiment_script_path]/results/my_exp_1 #. The type attribute tells the experiment if the folder can contain files that it may want to 'read' from or 'write' to. #. The file_types attribute specifies the file extensions expected to be in the given virtual folder. #. When the experiment looks for possible locations to find an image called 'picture.png' for example, then it would look in any virtual folder path path is type 'r' (read) and has '.png' in the list of possible file types.
Within your experiment, you can specify a virtual folder by using io.vfolders.[folder_name].[folder_name] .... For example, using the above v. folder definition, to get to the images v. folder you could go images_vfolder = io.vfolders.resources.images. You could then say images_vfolder.getFiles() to get all the files in that vfolder, or images_vfolder.getSystemPath() to get the physical path of the v. folder.
Within your experiment you can also get all the vfolders that a given file type can be in: io.file_vfolders.png would return [images,], since only 1 vfolder is defined as a possible .png file location. If 2 vfolders had been defined to contain png's, the result would be [images1,images2 for example].
Within your experiment you can also get all the system paths that a given file type can be in: io.file_dirs.png would return ['/resources/images',] for example.
Finally, all the files for a given type can be retrieved as the absolute path to every file that exists of that type by:
io.file_paths.png. If the image folder had 3 images in it, called 'p1.png', 'p2.png','p3.png', then the result of that reference would be: ['/resources/images/p1.png','/resources/images/p2.png','/resources/images/p3.png']
Let me know if you see any issues with this, ways to make it simpler, or if you do not think it will meet your requirements. Once I know I will plan implementation.
This will also address the bug about 'session folders being empty' and the 'hdf5 file is split to 2 locations' bug.
The string representation of the above directories config is the following based on the current prototype code:
Processing File: D:\Dropbox\DEV\OpenPsycho\DEV\OpenPsycho2.7.3.1-32bit\my-code\ioHub\examples\ioHubAccessDelayTest\experiment_Config.yaml
Virtual Folder: Name: vfolders System Path: D:\ioHub\examples\ioHubAccessDelayTest File Type Filter: [] Type: ['r', 'w']
Virtual Folder: Name: experiment System Path: D:\Dropbox\DEV\ioHub\examples\ioHubAccessDelayTest\ File Type Filter: ['.yaml'] Type: ['r'] Virtual Folder: Name: results System Path: D:/results/iohub_experiments/<code> File Type Filter: ['.hdf5'] Type: ['w'] Virtual Folder: Name: session System Path: D:/results/iohub_experiments/<code>\<session_defaults.code> File Type Filter: ['.xls', '.edf'] Type: ['w'] Virtual Folder: Name: resources System Path: D:\Dropbox\DEV\ioHub\examples\ioHubAccessDelayTest\resources File Type Filter: [] Type: ['r', 'w'] Virtual Folder: Name: condition_files System Path: D:\Dropbox\DEV\ioHub\examples\ioHubAccessDelayTest\resources\condition_files File Type Filter: ['.xls'] Type: ['r'] Virtual Folder: Name: images System Path: D:\Dropbox\DEV\ioHub\examples\ioHubAccessDelayTest\resources\images File Type Filter: ['.png', '.jpeg'] Type: ['r'] Virtual Folder: Name: audio System Path: D:\Dropbox\DEV\ioHub\examples\ioHubAccessDelayTest\resources\audio File Type Filter: ['.wav', '.mp4'] Type: ['r']— Reply to this email directly or view it on GitHub.