Tileset info not found with custom data

[yangmu3]

Hi,

I tried the example code from https://docs.higlass.io/jupyter.html#numpy-matrix.

import numpy as np

dim = 2000
I, J = np.indices((dim, dim))
data = (
    -(J + 47) * np.sin(np.sqrt(np.abs(I / 2 + (J + 47))))
    - I * np.sin(np.sqrt(np.abs(I - (J + 47))))
)

from  clodius.tiles import npmatrix
from higlass.tilesets import Tileset

ts = Tileset(
    tileset_info=lambda: npmatrix.tileset_info(data),
    tiles=lambda tids: npmatrix.tiles_wrapper(data, tids)
)

display, server, viewconf = higlass.display([
    View([
        Track(track_type='top-axis', position='top'),
        Track(track_type='left-axis', position='left'),
        Track(track_type='heatmap',
              position='center',
              tileset=ts,
              height=250,
              options={ 'valueScaleMax': 0.5 }),

    ])
])
display

And I got this And this is when I try ts.tileset_info() Could you give me some suggestions on how to solve this? Thank you!

[pkerpedjiev]

Hi there. Are you per chance behind a firewall? The higlass.display function will start a server in the background and it looks like the client is having trouble accessing it.

I just tried the exact same example and it worked fine.

[yangmu3]

Yes, that might be the reason. What should I do if I happen to be behind a firewall?

[pkerpedjiev]

Hmmm, to be honest I'm not sure. HiGlass needs a server component to access data. You could open up a port on your firewall and pass that port number to the display function:

display, server, viewconf = higlass.display([
    View([
        Track(track_type='top-axis', position='top'),
        Track(track_type='left-axis', position='left'),
        Track(track_type='heatmap',
              position='center',
              tileset=ts,
              height=250,
              options={ 'valueScaleMax': 0.5 }),

    ])
], server_port=8061)

[Shiyuanjun001225]

Hi, I have the same question, have you solve this question?

[tcege]

Hi everyone, I am encountering the same problem while trying to visualize a .mcool file in HiGlass-Python. My goal is to load and display a Hi-C heatmap using Jupyter Notebook and the higlass-python library. I am not using the HiGlass server or Docker—just the Python library in a local environment. I am using a Conda environment specifically set up for it, with Python 3.9 and all required dependencies installed in this environment.

Setup: Environment: Conda environment (Python 3.9) Installed Libraries: higlass-python (latest version) higlass-widget higlass-schema All dependencies such as ipywidgets, notebook<7.0, etc., are installed and working. Notebook: Jupyter Notebook with widgetsnbextension enabled.

Code is: import higlass as hg tileset = hg.cooler("/home/ltao/DATA2/tege/MicroC_cochlea/merge/merge_matrix_10kb.mcool::/resolutions/10000") track = tileset.track("heatmap") view = hg.view(track, width=12) view

When running the code, the Jupyter Notebook widget renders, but I get this error:

Tileset info not found. Server: [http://localhost:39279/tilesets/api/v1/] tilesetUid:

The heatmap widget appears, but no data is shown.

I have verified that the .mcool file exists at the specified path and is accessible. I have checked that the resolution path (/resolutions/10000) in the .mcool file is valid and available. I have ensured that Jupyter Notebook is running in the correct environment, with all necessary extensions enabled.

Does anyone know what could be causing this issue or if there's a step I might be missing? Any help would be greatly appreciated.

[pkerpedjiev]

Hi there, can you try without the ::/resolutions/10000 in the filename? So this:

import higlass as hg
tileset = hg.cooler("/home/ltao/DATA2/tege/MicroC_cochlea/merge/merge_matrix_10kb.mcool")
track = tileset.track("heatmap")
view = hg.view(track, width=12)
view

I just tried this with a local file and the latest higlass-python and it worked for me:

[tcege]

Thank you for the suggestion! Here's what I did step by step, including how I activated my environment, launched Jupyter Notebook, and tested your proposed code:

conda activate higlass_2 # activated my conda environment

jupyter notebook --no-browser --NotebookApp.token='' --ip=0.0.0.0 --port=8888 # launched Jupyter Notebook

http://alpha:8888/ #opened the Notebook in my browser, my data is stored on a server (not on my local machine)

I tried your suggested code:

Unfortunately, I still encountered the same problem. Thank you again for your help!

[pkerpedjiev]

That's odd. Could you open your browser developer tools console and post the output of that?

[tcege]

Thank you for your help in troubleshooting this issue. Here is the output :

[tcege]

[nvictus]

The hg server is running on the same machine as your notebook server but a different machine than your notebook frontend. The higlass client in your browser doesn't know that -- you can see it is trying to fetch tiles from localhost.

There is an undocumented solution to turn on jupyter-server-proxy:

import higlass as hg
hg.server.enable_proxy()

But it may not work for a variety of reasons. The alternative workaround is to configure your tilesets to declare their endpoint on the remote higlass server (assuming the server is accessible), but it currently requires some massaging:

import functools
import hashlib
import pathlib
from higlass.tilesets import LocalTileset

# Fix a port number you know is exposed and free, different from the port jupyter is running on
HIGLASS_PORT = 8889

# We currently need this to bypass the automatic port setting by hg.cooler()
def cooler_tileset(filepath: str):
    try:
        from clodius.tiles.cooler import tiles, tileset_info
    except ImportError:
        raise ImportError(
            'You must have `clodius` installed to use "matrix" data-server.'
        )
        
    abspath = pathlib.Path(filepath).absolute()
    uid = hashlib.md5(str(abspath).encode()).hexdigest()

    return LocalTileset(
        datatype="matrix",
        tiles=functools.partial(tiles, filepath),
        info=functools.partial(tileset_info, filepath),
        uid=uid,
    )

tileset = cooler_tileset(path)
hg.server.add(tileset, port=HIGLASS_PORT)

track = tileset.track("heatmap")
view = hg.view(track)
view

[tcege]

Thank you for your detailed suggestions! I tried your recommended solutions and pasted the results with screenshots.

[nvictus]

Doesn't help much if I can't see the full traceback, but I think I know what I forgot:

tileset = cooler_tileset(path)
tileset_resource = hg.server.add(tileset, port=HIGLASS_PORT)

track = tileset_resource.track("heatmap")

[nvictus]

Sigh. I just remembered, there is no exposed API to set the host name before the local server starts, so this only works if you are port forwarding your jupyter server connection over localhost to a local port. You will also have to forward the remote port of the higlass server to (the same) local port.

The short of it is, if hg.server.enable_proxy() doesn't work, this setup isn't officially supported right now. The plan is to drop the need to spawn a separate server from Jupyter entirely: https://github.com/higlass/higlass-python/issues/144

cc @manzt

[tcege]

Thank you for your suggestion! I tried using hg.server.enable_proxy() and port forwarding, but I still have issues. The problem seems related to the remote server setup.

Thank you again for your help!

[manzt]

Please have a look at higlass-python v1.3.0. We have removed the hg.server in preference of a mechanism for sending tile data over Jupyter's internal comms. This should hopefully enable more robust usage of HiGlass Python across Jupyter-based systems.

There should not be any ports or server config as we now piggyback on Jupyter's websocket to send messages between the server and client.

[tcege]

Thank you for the update! I’ll check it out.

[tcege]

Thank you for the update on HiGlass-Python v1.3.0, but I am encountering the same issues displaying .mcool files. Here’s what I did:

1. Activate the Conda environment: (bash) conda activate higlass

2. Started Jupyter Notebook: (bash) jupyter notebook --no-browser --NotebookApp.token='' --ip=0.0.0.0 --port=8888

3. I manually opened my browser and ran the following code: (python)

import higlass as hg tileset = hg.cooler("/home/ltao/DATA2/tege/MicroC_cochlea/new_merge/new_merge_matrix_10kb.mcool::/resolutions/10000") track = tileset.track("heatmap") view = hg.view(track, width=12) view

Instead of displaying the HiGlass visualization, I got this error:Tileset info not found. Server: [jupyter] tilesetUid: [hg_7f1a37d9730]

 Is there an additional configuration step required in HiGlass v1.3.0 to properly display .mcool files? Thank you!

[manzt]

Hi there, I think this is an issue with the usage of hg.cooler. I think it is silently failing due to passing in resolutions (clearly we need better errors when someone goes wrong!).

The helper just expects an .mcool path (no ::/resolutions/ suffix):

tileset = hg.cooler("/home/ltao/DATA2/tege/MicroC_cochlea/new_merge/new_merge_matrix_10kb.mcool")

[tcege]

Thank you for your suggestion!

I tried removing the ::/resolutions/10000 suffix and just passed the .mcool file path as you suggested

However, I still get the same "Tileset info not found" error.

Thank you again!

[nvictus]

Can you confirm that that path and the resolutions in the file can be accessed without errors from your notebook if you point cooler to it. e.g.

import cooler

path = "/home/ltao/DATA2/tege/MicroC_cochlea/new_merge/new_merge_matrix_10kb.mcool"

for group_path in cooler.file_ops.list_coolers(path):
    print(group_path)
    cooler.Cooler(f"{path}::{group_path}")

[tcege]

Thank you for the suggestion! I ran the code, and all resolutions in the .mcool file opened successfully without errors. Here’s the output:

[pkerpedjiev]

@tcege any chance you'd be willing to share the cooler file you're trying to open?

Happy to provide you an email address if you're not comfortable sharing it publicly.

Apologies if you've already shared it somewhere in the thread above.

[tcege]

https://drive.google.com/file/d/1FAgMdpL815Q4w3eMAS8mTUIoP4rjfliv/view?usp=drive_link

So sorry for the late reply! I've shared the link to my .mcool file. Let me know if you have any trouble accessing it!