[knn classifier] KNN Classification classify() returning results with labels that has been changed to numbers

[hlp-pls]

Dear community, I am experiencing problems with the KNNClassifier() classify() function. In short, I am loading a custom trained data model JSON file, and trying to access the label names from the model, but the labels aren't what I added to the model using the addExample() function. I'm new to ml5 and not quite sure whether this is an issue that should be fixed, or I did something foolish as usual.

https://github.com/CodingTrain/website/tree/master/Courses/beginner_ml5/07_knn_classifier/P5

https://youtu.be/JWsKay58Z2g

--> I am currently following the tutorial from above, and loaded the KNN classifier custom model JSON file that I had saved by following the tutorial. I have uploaded the project files in a zip below, but here is the code of the javascript file that I've been working on.

let features;
let video;
let knn;
let labelP;
let label = "";
let ready = false;

function modelReady(){
	console.log('MobileNet loaded!');	
	knn = ml5.KNNClassifier();	
	knn.load("model.json", function(){
		console.log("KNN data loaded!")
		goClassify();
	});	
}

function videoReady(){
	console.log('Video is ready!');
}

function setup(){
	createCanvas(640,480);
	video = createCapture(VIDEO,videoReady);
	video.size(640,480);
	video.hide();
	features = ml5.featureExtractor('MobileNet',modelReady);
	//knn = ml5.KNNClassifier();
	labelP = createP("Need training data.");
}

function goClassify(){
	const logits = features.infer(video);
	knn.classify(logits, function (error, result){	
		if(error){
			console.error(error);
		}else{
			label = result.label;
			console.log(result);
			labelP.html(label);
			goClassify();
		}
	});
}

function keyPressed(){
	const logits = features.infer(video);
	if(key=='l'){
		knn.addExample(logits,"left");
		console.log("left");
	}else if(key=='r'){
		knn.addExample(logits,"right");
		console.log("right");
	}else if(key=='u'){
		knn.addExample(logits,"up");
		console.log("up");
	}else if(key=='d'){
		knn.addExample(logits,"down");
		console.log("down");
	}else if(key=='s'){
		save(knn,'model.json');
	}
}

function draw(){
	push();
	translate(width,0); // move to far corner
  	scale(-1.0,1.0);    // flip x-axis backwards
	image(video,0,0);
	pop();
}

--> The tutorial was in ml5js version 0.2.1, and seemed to have problems with saving the JSON file that exceeds certain size. But I used version 0.4.3 and had no problem saving it, and no problem when loading the saved JSON file (which was named "model.json").

--> Previously, when I was following the tutorial that trains the KNNClassifier in real time, the result from classify() returned labels with strings that I had named and added using the addExample() function. But when I loaded a model that I had previously saved, the labels were changed to numbers like 0,1,2,3.

--> I compared my JSON file with the tutorial's which can be seen for about a split second, but it seemed to have the same structure that looks something like below;

{
        "dataset": {
                 "some number": {
                            "label": string, ...
                 }
        },
        "tensors": {
                 ..."some interger string": some float value between 0 and 1...
        }
}

--> when I print the results from classify() to console, the labels are changed to the number inside the "dataset" instead of the "label" inside of the number! But when I use the "confidencesByLabel", I can only get the confidence value if I put the strings I originally assigned in the bracket like this:

results.confidencesByLabel["the string name that i used"];

--> So in conclusion, I presume that the JSON files does have the label that I want, but I can't access it when I try to get it using the classify() function and get the label from the result. This problem does not occur when I am not loading my own model.

--> information about my setup!

• **currently using a local server in the project folder directory. (localhost:8000)
• Web browser & version: chrome 79.0.3945.88
• Operating System: macOS Mojave
• ml5 version you're using: 0.4.3
• Any additional notes: I copied and pasted the ml5js library from the link below. https://unpkg.com/[email protected]/dist/ml5.min.js

--> The zip file below containes the project folder that I have been working with. KNN_classification_load_update.zip

Thank you for reading.

[sobreira]

Hi, did you find a solution for this - Same issue for me.

Returning dataset instead of Label

Thanks!

Edson

[sobreira]

Last time it was valid were in the version 0.31, all the other version looks the same. Sometimes we record some data with 1 (the first data), then ml5js it´s returning the dataset, which came with a different data - 1, for instance.

Thanks!

[sobreira]

I did a new test and I got the following: I have training a model with label 9, hackeduca and 0.

So result.label (from knnClassifier.classify() returned all label correctly and check the result. Adding the following line of code : console.log('Classify - Result : ' + JSON.stringify(result)); we receive the following result: Classify - Result : {"classIndex":0,"label":"9","confidences":{"0":1,"1":0,"2":0},"confidencesByLabel":{"0":0,"9":1,"hackeduca":0}}

Saving this model and open it again the result is different it´s missing the first argument ("classIndex":

Classify - Result : {"label":"1", "confidences":{"0":0,"1":1,"2":0},"confidencesByLabel":{"0":0,"9":0,"hackeduca":1}}

From this point on the "old ClassIndex" became the "new label"

And the result.label shows up a wrong data.

[sobreira]

Important to mention that the issue is not only change label to numbers. the label is completely ignored (even if we record as number). The only valid thing is the Index which is recorded in a sequence starting from 0. The retrieve the value we should know the sequence of record.

[LyuboslavLyubenov]

Can confirm that i also experience similar behaviour. I will try to investigate why that happens

[harrykwan]

knnClassifier.load return this error

index.js:218 Uncaught (in promise) TypeError: Cannot read property 'shape' of undefined

In load function: const { dataset, tensors } = data; this.mapStringToIndex = Object.keys(dataset).map(key => dataset[key].label); const tensorsData = tensors .map((tensor, i) => { if (tensor) { const values = Object.keys(tensor).map(v => tensor[v]); return tf.tensor(values, dataset[i].shape, dataset[i].dtype); } return null; }) .reduce((acc, cur, j) => { acc[j] = cur; return acc; }, {}); this.knnClassifier.setClassifierDataset(tensorsData);

[CarolCPR]

I have an identical problem!

Using the function

console.log('Classify - Result : ' + JSON.stringify(result));

, returns:

Real data:

[CarolCPR]

My code:

let video;
let features;
let knn;
let labelP;
let ready = false;
let label = 'nothing';

function setup() {
  createCanvas(320, 240);
  video = createCapture(VIDEO);
  video.size(320, 240);
  video.hide();
  features = ml5.featureExtractor('MobileNet', modelReady);
  //knn = ml5.KNNClassifier();
  labelP = createP('need training data');
  labelP.style('font-size', '32pt');
}

function goClassify(){
  const logits = features.infer(video);
    knn.classify(logits, function (error, result){
        if(error){
            console.error(error);
        }else {
            label = result.label;
            labelP.html(result.label);
            console.log('Classify - Result : ' + JSON.stringify(result));
            goClassify();
        }
    });
}

function keyPressed(){
  const logits = features.infer(video);
  if(key == '+') knn.save("model.json");
  else if(key.length==1)
    knn.addExample(logits, key);
    console.log(key);
}

function modelReady(){
  console.log("MobileNet loaded!");
  knn = ml5.KNNClassifier();
  knn.load("model_3.json", function(){
    console.log('KNN Data Loaded');
    goClassify();
    });
}

function draw() {
  image(video, 0, 0);
  /*if(!ready && knn.getNumLabels() > 0){
    goClassify();
    ready = true;
  }*/
  noStroke()
  fill(0, 0, 200, 100);
  rect(0, 100, 120, 240);
}

[vennsoh]

Have the same problem as well! Labels are shown as numbers instead of the string that I gave. Any workarounds?

[benkei-kuruma]

Greetings all! Thanks so much for putting together this awesome library.

I'm going to be teaching (virtually) beginner ML with the ml5.js library for the first time this summer to a group of high school students, and I've also hit this snag while going through the library and watching Dan's videos.

Here's a temporary workaround I wrote, which uses the keys from the confidencesByLabel property of the result to get the label strings. I'm curious if there is a better way to do this in the meantime. All I do here is check to see which of the labels has the highest confidence value, and return that.

function getLabel(result) {
  const entries = Object.entries(result.confidencesByLabel);
  let greatestConfidence = entries[0];
  for(let i = 0; i < entries.length; i++) {
    if(entries[i][1] > greatestConfidence[1]) {
      greatestConfidence = entries[i];
    }
  }
  return greatestConfidence[0];
}

So essentially, call getLabel(result) where you would otherwise use result.label (which is a number and not a label).

[tangmingsh]

@ahob85 Thanks. It works. I also teach ml5.js to high school students.

[yashchaudhari008]

Greetings all! Thanks so much for putting together this awesome library.

I'm going to be teaching (virtually) beginner ML with the ml5.js library for the first time this summer to a group of high school students, and I've also hit this snag while going through the library and watching Dan's videos.

Here's a temporary workaround I wrote, which uses the keys from the confidencesByLabel property of the result to get the label strings. I'm curious if there is a better way to do this in the meantime. All I do here is check to see which of the labels has the highest confidence value, and return that.

function getLabel(result) {
  const entries = Object.entries(result.confidencesByLabel);
  let greatestConfidence = entries[0];
  for(let i = 0; i < entries.length; i++) {
    if(entries[i][1] > greatestConfidence[1]) {
      greatestConfidence = entries[i];
    }
  }
  return greatestConfidence[0];
}

So essentially, call getLabel(result) where you would otherwise use result.label (which is a number and not a label).

One liner:- Object.entries(result.confidencesByLabel).find(([k,v]) => v==1)[0]