Get experiment logged texts and metadata in custom panel

[george-gca]

Before Asking:

• [X] I have searched the Issue Tracker.
• [X] I have searched the Documentation.

What is your question related to?

• [ ] Comet Python SDK
• [X] Comet UI
• [ ] Third Party Integrations (Huggingface, TensorboardX, Pytorch Lightning etc.)

What is your question?

Is there any way that I can get all my experiment's logged texts, along with its metadata, when creating a custom panel?

The reason is that I am working with question and answer, and I'd like to create some visualizations with filters related to its metadata. For example, if I log some metadata to each question regarding the type of question, the size of the answer, and if it was answered correctly, I could create some kind of visualization showing the percentage of corrected answers per question type and per question size.

Currently, the only way that I can think of doing this is by adding, for example, the question type in each metric's name, but if I want to add some other useful information this gets out of hand.

[DN6]

Hello @george-gca. Are you trying to use Python or JS Panels to visualize your data?

[george-gca]

JS Panels, but only because python panels currently don't support plotly.

[DN6]

Hi @george-gca. Metadata is accessible in the logged asset. Here is a short snippet of an example panel that fetches the text assets from an experiment and returns a list of dictionaries containing the text and metadata. Hope this helps.

class MyPanel extends Comet.Panel {
    setup() {
        this.options = {
            experimentKey: null,
            layout: {
                showlegend: false,
                legend: {
                    orientation: "v"
                },
                title: {
                    text: "Text Data"
                }
            }
        };
    }

    fetchTextAssets(experimentKey) {
        return this.api.experimentAssets(experimentKey, "text-sample");
    }

    fetchTextContent(experimentKey, fileName) {
        return this.api.experimentAssetByName(experimentKey, fileName);
    }

    async draw(experimentKeys, projectId) {
        if (this.options.experimentKey !== null) {
            this.drawOne(this.options.experimentKey);
        } else if (experimentKeys.length > 0) {
            this.select("Select an experiment: ", experimentKeys);
            this.drawOne(experimentKeys[0]);
        };
    }

    async drawOne(experimentKey) {
        // Fetch all text assets for the experiment
        const textAssets = await this.fetchTextAssets(experimentKey);

        // Fetch the text content for each asset
        const data = textAssets.map(textAsset => {
            return this.fetchTextContent(experimentKey, textAsset.fileName).then(
                textContent => {
                    var output = {
                        text: textContent,
                        // Metadata is stored in the asset dictionary
                        metadata: textAsset.metadata
                    };
                    return output;
                }
            );
        });
        const results = await Promise.all(data);
        console.log(results);
    }
}

[george-gca]

@DN6 I believe there is a bug regarding the returning textContent from experimentAssetByName function. It is returning always the same content from the 1st text data, despite all other information coming as it should. Basically I took your code and added a few console.log.

// Fetch the text content for each asset
const data = textAssets.map(textAsset => {
    return this.fetchTextContent(experimentKey, textAsset.fileName).then(
        textContent => {
            var output = {
                text: textContent,
                // Metadata is stored in the asset dictionary
                metadata: textAsset.metadata
            };
            console.log(textContent);
            console.log(textAsset.fileName);
            console.log(JSON.parse(textAsset.metadata).question_id);
            return output;
        }
    );
});

The result for the first text data comes as:

"Question: can you see a tie on the top?
Expected: yes
Answer: <unk>"

"text-sample.txt"

50031

which is correct, but then for the 2nd text comes as:

"Question: can you see a tie on the top?
Expected: yes
Answer: <unk>"

"text-sample.txt"

8189

which is wrong. The question 8189 has as text:

Question: can you see a tie on the top part of the pic?
Expected: yes
Answer: yes

Also there is a mispelling on the docs:

> api.experimentAssetByName('experimentKey1', 'file.py').then(result => {
    console.log(results); // should be result in here
  });

[DN6]

Hi @george-gca taking another look at this. Thanks for flagging the typo!

[DN6]

Hi @george-gca . The issue was due to the fact that all text samples from experiment.log_text share the same filename. Causing the same content to be returned. Here is an updated snippet that uses the assetId of the text. I believe this will fix your problem. Would you mind testing and confirming?

class MyPanel extends Comet.Panel {
    setup() {
        this.options = {
            experimentKey: null,
            layout: {
                showlegend: false,
                legend: {
                    orientation: "v"
                },
                title: {
                    text: "Text Data"
                }
            }
        };
    }

    fetchTextAssets(experimentKey) {
        return this.api.experimentAssets(experimentKey, "text-sample");
    }

    fetchTextContent(experimentKey, assetId) {
        return this.api.experimentAsset(experimentKey, assetId);
    }

    async draw(experimentKeys, projectId) {
        if (this.options.experimentKey !== null) {
            this.drawOne(this.options.experimentKey);
        } else if (experimentKeys.length > 0) {
            this.select("Select an experiment: ", experimentKeys);
            this.drawOne(experimentKeys[0]);
        };
    }

    async drawOne(experimentKey) {
        // Fetch all text assets for the experiment
        const textAssets = await this.fetchTextAssets(experimentKey);

        // Fetch the text content for each asset
        const data = textAssets.map(textAsset => {
            return this.fetchTextContent(experimentKey, textAsset.assetId).then(
                textContent => {
                    var output = {
                        text: textContent,
                        // Metadata is stored in the asset dictionary
                        metadata: textAsset.metadata
                    };
                    return output;
                }
            );
        });
        const results = await Promise.all(data);
        console.log(results);
    }
}

[george-gca]

@DN6 it worked. Thanks a lot!