hard-linking fails if source and tmp folder are on different partitions or devices

[nvictus]

import_file() will raise an OSError: [Errno 18] Invalid cross-device link.

There should be an alternative if hard-linking is not possible (e.g. copying to tmp).

[annashcherbina]

Is there support for this now by any chance? Would be great to visualize tracks hosted on a different server without having to download them locally.

[pkerpedjiev]

Are the tracks accessible by http?

[annashcherbina]

Yes, they are hosted on a public web server and accessible by http.

[pkerpedjiev]

I presume that this is related to https://github.com/higlass/higlass-manage/issues/48? If so, I propose we continue the discussion there so that there aren't two threads to follow?

[maharshi14]

Hi! I'm having the same problem. I'm running HiGlass on an Ubuntu server, I have a hg-data folder but no hg-tmp, it runs on my mac but not on the server. I get the error OSError: [Errno 18] Invalid cross-device link: 'Growing.Rep2_hicpro.mcool' -> '/tmp/higlass-docker/Growing.Rep2_hicpro.mcool'

[pkerpedjiev]

May I suggest creating a folder that you'll use for your data and letting higlass-manage know about it with --media-dir? Then you should be able to ingest datasets using --no-upload.

[maharshi14]

Thanks a lot for the quick reply!

I added the --media-dir as follows:

higlass-manage start -m /home/rishi/higlass_data

Then I tried to ingest it as follows:

higlass-manage ingest higlass_data/Growing.Rep2_hicpro.mcool --no-upload

I got the error:

state True Inferred filetype: cooler Inferred datatype: matrix state True name_text: hg_name: default command: python higlass-server/manage.py ingest_tileset --filename higlass_data/Growing.Rep2_hicpro.mcool --filetype cooler --datatype matrix --no-upload --uid MvVDCY9LTd-gOSc_NZdfIA ERROR: CommandError: File does not exist under media root

I realised that since I've already specified --media-dir as higlass_data, I tried without it:

higlass-manage ingest Growing.Rep2_hicpro.mcool --no-upload

Then I got no error and it stopped at:

state True Inferred filetype: cooler Inferred datatype: matrix state True name_text: hg_name: default command: python higlass-server/manage.py ingest_tileset --filename Growing.Rep2_hicpro.mcool --filetype cooler --datatype matrix --no-upload --uid VTDNwwYLTLCOU6FBVUUo5g

When I tried to view the dataset, I got this error:

higlass-manage view Growing.Rep2_hicpro.mcool

state True temp_dir: /tmp/higlass-docker Inferred filetype: cooler Inferred datatype: matrix Traceback (most recent call last): File "/home/rishi/.local/bin/higlass-manage", line 11, in sys.exit(cli()) File "/home/rishi/.local/lib/python3.6/site-packages/click/core.py", line 764, in call return self.main(*args, **kwargs) File "/home/rishi/.local/lib/python3.6/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/home/rishi/.local/lib/python3.6/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/rishi/.local/lib/python3.6/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/rishi/.local/lib/python3.6/site-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/home/rishi/.local/lib/python3.6/site-packages/higlass_manage/view.py", line 95, in view checksum1 = md5(tileset_path) File "/home/rishi/.local/lib/python3.6/site-packages/higlass_manage/common.py", line 15, in md5 with open(fname, "rb") as f: FileNotFoundError: [Errno 2] No such file or directory: '/home/rishi/hg-data/media/media/Growing.Rep2_hicpro.mcool'

[maharshi14]

Hi,

I tried changing both the --data-dir and --media-dir to my data folder, and it gave me this error:

state True temp_dir: /tmp/higlass-docker Inferred filetype: cooler Inferred datatype: matrix Traceback (most recent call last): File "/home/rishi/.local/bin/higlass-manage", line 11, in sys.exit(cli()) File "/home/rishi/.local/lib/python3.6/site-packages/click/core.py", line 764, in call return self.main(*args, **kwargs) File "/home/rishi/.local/lib/python3.6/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/home/rishi/.local/lib/python3.6/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/rishi/.local/lib/python3.6/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/rishi/.local/lib/python3.6/site-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/home/rishi/.local/lib/python3.6/site-packages/higlass_manage/view.py", line 95, in view checksum1 = md5(tileset_path) File "/home/rishi/.local/lib/python3.6/site-packages/higlass_manage/common.py", line 15, in md5 with open(fname, "rb") as f: PermissionError: [Errno 13] Permission denied: '/var/lib/docker/volumes/ebc90bbe11e5a8288e30325b613cf431db46dc0773b6b977764f6fbf820645a0/_data/media/media/Growing.Rep2_hicpro.mcool'

Which is weird because I'm already in the sudoers list. I tried sudo higlass-manage view Growing.Rep2_hicpro.mcool, but it gave me:

sudo: higlass-manage: command not found

[pkerpedjiev]

Hey, sorry, this is poorly documented on our part.

There are a couple caveats:

1. The media directory needs to be an absolute directory
2. The file path needs to be relative to the media directory

This worked for me on OS X:

cp my_bigwig.bw data/
higlass-manage start --no-public-data --media-dir $(pwd)/data
higlass-manage ingest --no-upload my_bigwig.bw

[pkerpedjiev]

I just created a PR to update the docs for the --no-upload option and to not require an absolute path for the media dir: https://github.com/higlass/higlass-manage/pull/52/

[maharshi14]

Hi!

I still get the following error:

FileNotFoundError: [Errno 2] No such file or directory: '/home/rishi/hg-data/media/media/Growing.Rep2_hicpro.mcool'

I am not sure why the file path has media/media, because this is not a directory that I created :/

[maharshi14]

Okay, I don't know why this happened, but I just manually created media/media directories and now it works!

[jakevc]

I have experienced the same OSError: [Errno 18] Invalid cross-device link: error when hg-tmp is on a different filesystem, and the commit referenced above is how I fixed it (works on Ubuntu 18.04). I'm not sure if the limitations of this approach are appropriate, or align with the desired behavior for higlass-manage ingest.

I suppose this is also indirectly related to #48