How do i prepare the data for Visualisation?

[dittops]

I'm trying to use the data visualization using meerkat. The viz/main.py visualization is a sample data of Git Hub. Is there a script with which I can expand to other datasets?

[mauriceweber]

I think you can follow the steps described in the viz Readme https://github.com/togethercomputer/RedPajama-Data/tree/main/viz#reproducing-the-data-processing-pipeline to expand to the full github or also other slices of the dataset?

[dittops]

I have tried following the steps. It's creating the pca32.faisis file. But as per the main.py file, it's loading the data from "https://huggingface.co/datasets/meerkat-ml/lemma/resolve/main/filtered_08cdfa755e6d4d89b673d5bd1acee5f6.mk.tar.gz". How do I create that data which can be passed?