BIDSLayout unable to interpret the input path on Windows 10

[snezakg]

Hi,

I am running the following commands through a Jupyter Notebook from Anaconda 4.6.8 on Windows 10:

bids_dir = "C:/Users/myself/DevArea/ds-supermri/bids" layout = BIDSLayout(bids_dir)

The command is crashing with:

error: incomplete escape \U at position 2

I tried changing slashes to single and double back-slashes, and also putting an r in front of the string, but BIDSLayout doesn't seem to be interpreted any of these.

Is there any explanation to this behaviour or a suggestion how to fix it?

Provided below:

1. Complete error message
2. Complete listing of the conda environment used

I am happy to provide more details if needed. Thanks in advance.

Cheers, Snežana

===========================================================

1. Complete error message

---

error Traceback (most recent call last) in 4 subject = 'eyesVD46a' 5 ----> 6 layout = BIDSLayout(bids_dir) 7 print(layout)

~\AppData\Local\Continuum\anaconda3\envs\supermri-env\lib\site-packages\bids\layout\layout.py in init(self, root, validate, index_associated, absolute_paths, derivatives, config, sources, ignore, force_index, config_filename, regex_search) 183 config = [Config.load(c) for c in listify(config)] 184 self.config = {c.name: c for c in config} --> 185 self.root_node = BIDSRootNode(self.root, config, self) 186 187 # Consolidate entities into master list. Note: no conflicts occur b/c

~\AppData\Local\Continuum\anaconda3\envs\supermri-env\lib\site-packages\bids\layout\core.py in init(self, path, config, layout, force_index) 544 self._layout = layout 545 super(BIDSRootNode, self).init(path, config, --> 546 force_index=force_index) 547 548 def _setup(self):

~\AppData\Local\Continuum\anaconda3\envs\supermri-env\lib\site-packages\bids\layout\core.py in init(self, path, config, root, parent, force_index) 366 367 # Index files and create child nodes --> 368 self.index() 369 370 def getitem(self, key):

~\AppData\Local\Continuum\anaconda3\envs\supermri-env\lib\site-packages\bids\layout\core.py in index(self) 501 continue 502 --> 503 child_class = self._get_child_class(d) 504 # TODO: filter the config files based on include/exclude rules 505 child = child_class(d, config_list, root_node, self,

~\AppData\Local\Continuum\anaconda3\envs\supermri-env\lib\site-packages\bids\layout\core.py in _get_child_class(self, path) 412 template = template.replace('{%s}' % ent, patt) 413 template += r'[^%s]*$' % os.path.sep --> 414 if re.match(template, path): 415 return listify(self._child_class)[i] 416

~\AppData\Local\Continuum\anaconda3\envs\supermri-env\lib\re.py in match(pattern, string, flags) 171 """Try to apply the pattern at the start of the string, returning 172 a Match object, or None if no match was found.""" --> 173 return _compile(pattern, flags).match(string) 174 175 def fullmatch(pattern, string, flags=0):

~\AppData\Local\Continuum\anaconda3\envs\supermri-env\lib\re.py in _compile(pattern, flags) 284 if not sre_compile.isstring(pattern): 285 raise TypeError("first argument must be string or compiled pattern") --> 286 p = sre_compile.compile(pattern, flags) 287 if not (flags & DEBUG): 288 if len(_cache) >= _MAXCACHE:

~\AppData\Local\Continuum\anaconda3\envs\supermri-env\lib\sre_compile.py in compile(p, flags) 762 if isstring(p): 763 pattern = p --> 764 p = sre_parse.parse(p, flags) 765 else: 766 pattern = None

~\AppData\Local\Continuum\anaconda3\envs\supermri-env\lib\sre_parse.py in parse(str, flags, pattern) 928 929 try: --> 930 p = _parse_sub(source, pattern, flags & SRE_FLAG_VERBOSE, 0) 931 except Verbose: 932 # the VERBOSE flag was switched on inside the pattern. to be

~\AppData\Local\Continuum\anaconda3\envs\supermri-env\lib\sre_parse.py in _parse_sub(source, state, verbose, nested) 424 while True: 425 itemsappend(_parse(source, state, verbose, nested + 1, --> 426 not nested and not items)) 427 if not sourcematch("|"): 428 break

~\AppData\Local\Continuum\anaconda3\envs\supermri-env\lib\sre_parse.py in _parse(source, state, verbose, nested, first) 505 506 if this[0] == "\": --> 507 code = _escape(source, this, state) 508 subpatternappend(code) 509

~\AppData\Local\Continuum\anaconda3\envs\supermri-env\lib\sre_parse.py in _escape(source, escape, state) 367 escape += source.getwhile(8, HEXDIGITS) 368 if len(escape) != 10: --> 369 raise source.error("incomplete escape %s" % escape, len(escape)) 370 c = int(escape[2:], 16) 371 chr(c) # raise ValueError for invalid code

error: incomplete escape \U at position 2

=========================================================== 2) Complete listing of the conda environment used

Name Version Build Channel

attrs 19.1.0 py37_1 backcall 0.1.0 py37_0 bids 0.0 pypi_0 pypi bids-validator 1.2.2 pypi_0 pypi blas 1.0 mkl bleach 3.1.0 py37_0 bz2file 0.98 py_0 conda-forge ca-certificates 2019.1.23 0 certifi 2019.3.9 py37_0 chardet 3.0.4 pypi_0 pypi citeproc-py 0.4.0 pypi_0 pypi click 7.0 py_0 conda-forge colorama 0.4.1 py37_0 cycler 0.10.0 py_1 conda-forge decorator 4.4.0 py_0 conda-forge defusedxml 0.5.0 py37_1 docopt 0.6.2 pypi_0 pypi duecredit 0.7.0 pypi_0 pypi entrypoints 0.3 py37_0 freetype 2.10.0 h5db478b_0 conda-forge funcsigs 1.0.2 py_3 conda-forge future 0.17.1 py37_1000 conda-forge graphviz 2.38.0 h6538335_1011 conda-forge h5py 2.9.0 nompi_py37h3cb27cb_1102 conda-forge hdf5 1.10.4 nompi_hcc15c50_1105 conda-forge html5lib 1.0.1 py_0 conda-forge icc_rt 2019.0.0 h0cc432a_1 icu 58.2 ha66f8fd_1 idna 2.8 pypi_0 pypi intel-openmp 2019.3 203 ipykernel 5.1.0 py37h39e3cac_0 ipython 7.3.0 py37h39e3cac_0 ipython_genutils 0.2.0 py37_0 ipywidgets 7.4.2 py37_0 isodate 0.6.0 py_1 conda-forge jedi 0.13.3 py37_0 jinja2 2.10 py37_0 jpeg 9c hfa6e2cd_1001 conda-forge jsonschema 3.0.1 py37_0 jupyter 1.0.0 py37_7 jupyter_client 5.2.4 py37_0 jupyter_console 6.0.0 py37_0 jupyter_core 4.4.0 py37_0 keepalive 0.5 py_1 conda-forge kiwisolver 1.0.1 py37he980bc4_1002 conda-forge libblas 3.8.0 4_mkl conda-forge libcblas 3.8.0 4_mkl conda-forge libiconv 1.15 hfa6e2cd_1005 conda-forge liblapack 3.8.0 4_mkl conda-forge libpng 1.6.36 h7602738_1000 conda-forge libsodium 1.0.16 h9d3ae62_0 libxml2 2.9.8 h9ce36c8_1005 conda-forge libxslt 1.1.32 heafd4d3_1002 conda-forge lxml 4.3.2 py37heafd4d3_0 conda-forge m2w64-gcc-libgfortran 5.3.0 6 m2w64-gcc-libs 5.3.0 7 m2w64-gcc-libs-core 5.3.0 7 m2w64-gmp 6.1.0 2 m2w64-libwinpthread-git 5.0.0.4634.697f757 2 markupsafe 1.1.1 py37he774522_0 matplotlib 3.0.3 py37_0 conda-forge matplotlib-base 3.0.3 py37h3e3dc42_0 conda-forge mistune 0.8.4 py37he774522_0 mkl 2019.1 144 msys2-conda-epoch 20160418 1 nbconvert 5.4.1 py37_3 nbformat 4.4.0 py37_0 networkx 2.2 py_1 conda-forge neurdflib 5.0.0.post1 py_0 conda-forge nibabel 2.3.3 py_1 conda-forge nipype 1.1.8 py37_0 conda-forge notebook 5.7.6 py37_0 num2words 0.5.9 pypi_0 pypi numpy 1.16.2 py37h8078771_1 conda-forge openssl 1.1.1b he774522_1 packaging 19.0 py_0 conda-forge pandas 0.24.2 pypi_0 pypi pandoc 2.2.3.2 0 pandocfilters 1.4.2 py37_1 parso 0.3.4 py37_0 patsy 0.5.1 pypi_0 pypi pickleshare 0.7.5 py37_0 pip 19.0.3 py37_0 conda-forge prometheus_client 0.6.0 py37_0 prompt_toolkit 2.0.9 py37_0 prov 1.5.3 py_1 conda-forge psutil 5.6.1 py37hfa6e2cd_0 conda-forge pybids 0.8.0 pypi_0 pypi pydicom 1.2.2 py_0 conda-forge pydot 1.4.1 py37_1000 conda-forge pydotplus 2.0.2 py_2 conda-forge pyface 6.0.0 py_1 conda-forge pygments 2.3.1 py_0 conda-forge pyparsing 2.3.1 py_0 conda-forge pyqt 5.9.2 py37h6538335_2 pyreadline 2.1 py37_1000 conda-forge pyrsistent 0.14.11 py37he774522_0 python 3.7.2 hb12ca83_0 conda-forge python-dateutil 2.8.0 py_0 conda-forge pytz 2018.9 pypi_0 pypi pywin32 224 py37hfa6e2cd_1000 conda-forge pywinpty 0.5.5 py37_1000 pyzmq 18.0.0 py37ha925a31_0 qt 5.9.7 vc14h73c81de_0 qtconsole 4.4.3 py37_0 rdflib 4.2.2 py37_1000 conda-forge requests 2.21.0 pypi_0 pypi scipy 1.2.1 py37h29ff71c_0 send2trash 1.5.0 py37_0 setuptools 40.8.0 py37_0 conda-forge simplejson 3.16.0 py37hfa6e2cd_1002 conda-forge sip 4.19.8 py37h6538335_1000 conda-forge six 1.12.0 py37_1000 conda-forge sparqlwrapper 1.8.2 py37_1000 conda-forge sqlite 3.26.0 hfa6e2cd_1001 conda-forge terminado 0.8.1 py37_1 testpath 0.4.2 py37_0 tornado 6.0.1 py37hfa6e2cd_0 conda-forge traitlets 4.3.2 py37_0 traits 5.0.0 py37hfa6e2cd_0 conda-forge traitsui 6.0.0 py_1 conda-forge urllib3 1.24.1 pypi_0 pypi vc 14.1 h0510ff6_4 vs2015_runtime 14.15.26706 h3a45250_0 wcwidth 0.1.7 py37_0 webencodings 0.5.1 py_1 conda-forge wheel 0.33.1 py37_0 conda-forge widgetsnbextension 3.4.2 py37_0 wincertstore 0.2 py37_1002 conda-forge winpty 0.4.3 4 zeromq 4.3.1 h33f27b4_3 zlib 1.2.11 h2fa13f4_1004 conda-forge

[tyarkoni]

Thanks for the bug report, @snezakg. We don't officially support Windows at the moment, unfortunately. Fixing this will require someone more familiar with the platform to take a look. I'll leave this open for the time being. Sorry!

[effigies]

The basic problem is that the path is canonicalizes /U... to \U..., which is then further interpreted as a unicode escape in a second-level interpolation.

Still would need a Windows box to test fixes.

[crocodoyle]

I have a different Windows 10 error, BIDSLayout, doesn't raise an error when I give it a path to a BIDS dataset (freshly downloaded from OpenNeuro), but doesn't find any of the subjects/sessions or anything

[tyarkoni]

Does it work in the bash subsystem? Just trying to gauge whether it's specific to one dataset, or is a general Windows issue. (Qualifier above still applies: I'm probably not going to be able to fix any Windows-specific issues myself, but would happily accept PRs.)

[crocodoyle]

OMG IT WORKS! I was not expecting this at all, they've definitely made some improvements

[joewlambeth]

OMG IT WORKS! I was not expecting this at all, they've definitely made some improvements

I'm still having issues with this, unfortunately. When working with both my own dataset, and the tutorial dataset provided, I'm getting 0 subjects, sessions, and runs.

EDIT: Upon further inspection, it seems that it's a problem with bids-validator, not pybids. The top-level regexes (or any of them for that matter) aren't matching with Windows directories or files, for whatever reason. Also, when I pass validate=False in BIDSLayout(), the issue goes away.

[kumarvinodhg]

Although using validate = False solves the problem of BIDSLayout not finding the subjects. However, it is still unable to find the sessions.

[effigies]

@kumarvinodhg Can you provide any further information about your problem? Have you validated the dataset?

[kalenkovich]

I am also having a problem on a Windows machine. When I run the following lines from the tutorial

# Here we're using an example BIDS dataset that's bundled with the pybids tests
data_path = os.path.join(get_test_data_path(), '7t_trt')

# Initialize the layout
layout = BIDSLayout(data_path)

# Print some basic information about the layout
layout

I get

BIDS Layout: ...ackages\bids\tests\data\7t_trt | Subjects: 0 | Sessions: 0 | Runs: 0

Fixing this will require someone more familiar with the platform to take a look.

I would love to help.

[kalenkovich]

(Qualifier above still applies: I'm probably not going to be able to fix any Windows-specific issues myself, but would happily accept PRs.)

@tyarkoni, I submitted a pr #627.

The problem was was with line 84 in bids.layout.index.BIDS._validate_file:

https://github.com/bids-standard/pybids/blob/01aac40f51a0b2ce9112b5e8c307e98b33cfc255/bids/layout/index.py#L83-L85

A Windows path such as 'a\b.json' would get '/' prepended resulting in '/a\b.json' which is not a valid path. If we switch from '/' to os.path.sep like this:

to_check = os.path.relpath(f, self.root) 
to_check = os.path.join(os.path.sep, to_check)
return self.validator.is_bids(to_check) 

the result would become '\a\b.json' on Windows. On a POSIX system,'a/b.json' would still become '/a/b.json' so the path would be valid on any OS. The problem is that bids-validator does not need a valid path, it needs a POSIX path. To work around this on the pybids side, I converted the path to POSIX using pathlib.Path.as_posix:

from pathlib import Path
...
to_check = os.path.relpath(f, self.root) 
to_check = os.path.join(os.path.sep, to_check)
to_check = Path(to_check).as_posix()  # bids-validator works with posix paths only
return self.validator.is_bids(to_check) 

This seems unusual to me that a package works differently with Windows and POSIX paths. Should I submit an issue with bids-validator?