map facial point to 3d model

[nephilimboy]

Hi, thanks for your great work, is there any way to map the output of "ctracker.getCurrentPosition();" into the 3D model? if there is can you give me some hit? Thanks in advance

[auduno]

Hi! clmtrackr is using a 2D face model, so it's unfortunately not possible to get a mapping to a 3d model :(

[ryansit]

I don't know if this is what you mean, but you can do videoctrack.draw(videooverlay, undefined, 'vertices'); with the vertices option to draw the output more 3D looking.

[bobbigmac]

I've kludged this to roughly translate an event array to 3d transforms. It's not quite 'right' but it might point someone in the right direction.

function mapEventTo3dTransforms(event) {
	if(event) {
		var p2 = event[33], p1 = event[7];
		var angleDeg = (-90 - (Math.atan2(p2[1] - p1[1], p2[0] - p1[0]) * 180 / Math.PI));

		// turn angle in degrees
		var t2 = event[62], t1 = [(event[13][0] - event[1][0]) / 2, (event[13][1] - event[1][1]) / 2];
		var turnDeg = (-90 - (Math.atan2(p2[1] - p1[1], p2[0] - p1[0]) * 180 / Math.PI));

		// tilt angle in degrees
		var tops = ((event[0][1] + event[14][1]) / 2);
		var bottoms = ((event[6][1] + event[8][1]) / 2);
		var middle = bottoms + ((tops - bottoms) / 2);

		//TODO: This just isn't right, but it's close
		var tiltDeg = -(event[37][1] / middle);

		var dx = event[14][0] - event[0][0];
		var dy = event[14][1] - event[0][1];
		var hyp = Math.sqrt((dx * dx) + (dy * dy));

		return {
			x: tiltDeg,
			y: turnDeg,
			z: angleDeg,
			hyp: hyp
		};
	}
}

[ghost]

I played around with this a little bit. The biggest problem is the clmtrackr... When I turn and tilt my head the clmtrackr does not follow... this is very unfortunate, but it is a 2D mapper after all...

[bobbigmac]

I got really decent tracking with it, I figure 30 degrees X, Y and Z. Mileage does vary with backgrounds though (if I'm in front of my bookshelf it struggles but with even lighting and fairly clear background I've been really impressed).

[ghost]

Then my lightning and camera seem to be the problem.

[bobbigmac]

This is what I ended up using. It's not 'great', I think some form of vector-based linear regression would be much more accurate, but it works pretty well :)

https://gist.github.com/bobbigmac/08b6f341385fc4445fe545a80aa3d15f

Here's a video using the output to render via css transform (the glasses) a 3d-space transform with an .obj (the skull and halo) rendered through three.js and estimating screen co-ordinates for a simple interactive UI (i.e. using my head pose to interact with screen elements).