librekinect icon indicating copy to clipboard operation
librekinect copied to clipboard

Published 20 hours ago • verified publisher icon xxorde

librekinect on beaglebone black: almost there? Or not really ....?

Open stinkytiger opened this issue 10 years ago • 5 comments

Hi,

I am running:

Linux version 3.14.4-bone4.1 (papa@calvin2) (gcc version 4.8.3 20140401 (prerelease) (crosstool-NG linaro-1.13.1-4.8-2014.04 - Linaro GCC 4.8-2014.04) ) #1 SMP Sun Jun 8 15:16:41 EDT 2014 [ 0.000000] CPU: ARMv7 Processor [413fc082] revision 2 (ARMv7), cr=10c5387d [ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache [ 0.000000] Machine model: TI AM335x BeagleBone [ 0.000000] cma: CMA: reserved 16 MiB at 9e800000 [ 0.000000] Memory policy: Data cache writeback [ 0.000000] On node 0 totalpages: 130816 [ 0.000000] free_area_init_node: node 0, pgdat c0ad1e80, node_mem_map dfaf2000 [ 0.000000] Normal zone: 1024 pages used for memmap [ 0.000000] Normal zone: 0 pages reserved [ 0.000000] Normal zone: 130816 pages, LIFO batch:31 [ 0.000000] CPU: All CPU(s) started in SVC mode. [ 0.000000] AM335X ES2.1 (sgx neon )

from Robert C Nelson. And I compiled the librekinect as a module and I think is seems to have loaded in correctly because i see in dmesg:

[ 3.603352] usbcore: registered new interface driver kinect_depth [ 3.622175] usb 2-1.1: new high-speed USB device number 4 using musb-hdrc [ 3.723861] usb 2-1.1: device v045e p02ad is not supported [ 3.729638] usb 2-1.1: New USB device found, idVendor=045e, idProduct=02ad [ 3.729648] usb 2-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3 [ 3.729657] usb 2-1.1: Product: Xbox Kinect Audio, \xffffffc2\xffffffa9\xffffffa9 2011 Microsoft Corporation. All rights reserved. [ 3.729666] usb 2-1.1: Manufacturer: Microsoft [ 3.729674] usb 2-1.1: SerialNumber: A44884X01190210A [ 4.039768] omap_rtc 44e3e000.rtc: rtc core: registered 44e3e000.rtc as rtc0 [ 4.511987] random: nonblocking pool is initialized [ 4.734481] omap-sham 53100000.sham: hw accel on OMAP rev 4.3 [ 4.934042] omap-aes 53500000.aes: OMAP AES hw accel rev: 3.2 [ 5.162182] usb 2-1.3: new high-speed USB device number 5 using musb-hdrc [ 5.265488] usb 2-1.3: device v045e p02ae is not supported [ 5.274579] usb 2-1.3: New USB device found, idVendor=045e, idProduct=02ae [ 5.274593] usb 2-1.3: New USB device strings: Mfr=2, Product=1, SerialNumber=3 [ 5.274602] usb 2-1.3: Product: Xbox NUI Camera [ 5.274610] usb 2-1.3: Manufacturer: Microsoft [ 5.274618] usb 2-1.3: SerialNumber: A00363W01604210A [ 5.305430] gspca: kinect_depth-2.14.0 probing 045e:02ae [ 6.256680] net eth0: initializing cpsw version 1.12 (0) [ 6.332402] net eth0: phy found : id is : 0x7c0f1 [ 6.332525] libphy: PHY 4a101000.mdio:01 not found [ 6.337560] net eth0: phy 4a101000.mdio:01 not found on slave 1

and I see in /dev ===>

root@woofieone:/dev# root@woofieone:/dev# ls -l /dev/video0 crw-rw---- 1 root video 81, 0 Jun 15 00:45 /dev/video0

So that looks good. If I just use:

cat /dev/video0

It thinks for a while then I get:

root@woofieone:/dev# time cat /dev/video0 cat: /dev/video0: No such device

real 0m5.381s user 0m0.000s sys 0m0.940s root@woofieone:/dev#

So it is doing something I think. And in the DMESG I see:

[ 2245.353096] kinect_depth 2-1.3:1.0: urb status: -71 [ 2245.358099] gspca: usb_submit_urb() ret -19 [ 2248.233111] usb 2-1.3: new high-speed USB device number 7 using musb-hdrc [ 2248.336423] usb 2-1.3: device v045e p02ae is not supported [ 2248.342230] usb 2-1.3: New USB device found, idVendor=045e, idProduct=02ae [ 2248.342265] usb 2-1.3: New USB device strings: Mfr=2, Product=1, SerialNumber=3 [ 2248.342293] usb 2-1.3: Product: Xbox NUI Camera [ 2248.342320] usb 2-1.3: Manufacturer: Microsoft [ 2248.342347] usb 2-1.3: SerialNumber: A00363W01604210A [ 2248.349988] gspca: kinect_depth-2.14.0 probing 045e:02ae [ 2256.986620] kinect: send_cmd: Bad tag 0004 != 0005 [ 2256.991791] kinect: send_cmd: Output control transfer failed (-71) [ 2256.998482] kinect_depth 2-1.3:1.0: urb status: -71 [ 2257.003498] kinect_depth 2-1.3:1.0: urb status: -71kinect_depth 2-1.3:1.0: urb status: -71 [ 2257.012346] kinect_depth 2-1.3:1.0: urb status: -71kinect_depth 2-1.3:1.0: urb status: -71 [ 2257.021223] kinect_depth 2-1.3:1.0: urb status: -71kinect_depth 2-1.3:1.0: urb status: -71 [ 2257.030098] kinect_depth 2-1.3:1.0: urb status: -71kinect_depth 2-1.3:1.0: urb status: -71 [ 2257.038971] kinect_depth 2-1.3:1.0: urb status: -71kinect_depth 2-1.3:1.0: urb status: -71 [ 2257.047848] kinect_depth 2-1.3:1.0: urb status: -71kinect_depth 2-1.3:1.0: urb status: -71 [ 2257.056734] kinect_depth 2-1.3:1.0: urb status: -71kinect_depth 2-1.3:1.0: urb status: -71 [ 2257.065614] kinect_depth 2-1.3:1.0: urb status: -71kinect_depth 2-1.3:1.0: urb status: -71 [ 2257.074484] kinect_depth 2-1.3:1.0: urb status: -71kinect_depth 2-1.3:1.0: urb status: -71 [ 2257.083383] kinect_depth 2-1.3:1.0: urb status: -71kinect_depth 2-1.3:1.0: urb status: -71

So my questions are:

  1. would cat /dev/video0 work? I.E. I would get some junk on the screen. Right now I see nothing ....

  2. Do I need to write a small C code to read /dev/video0. If so do you have an example one handy? All I need is to read and dump to a file. One frame would do. The main thing is what is the correct buffer size to use in bytes? Is it with header or without header? I.E. should I use 422400 bytes for 11 bits?

  3. I tried to change to 11 bits instead of 10 bits so .... in kinect.c

    /* Depth Stream Format 0x03: 11 bit stream | 0x02: 10 bit */
    //write_register(gspca_dev, 0x12, 0x02);
    write_register(gspca_dev, 0x12, 0x03);

and also:

static const struct v4l2_pix_format video_camera_mode[] = { {640, 480, V4L2_PIX_FMT_Y10BPACK, V4L2_FIELD_NONE, .bytesperline = 640 * 11 / 8, .sizeimage = 640 * 480 * 11 / 8, .colorspace = V4L2_COLORSPACE_SRGB, .priv = MODE_640x480 | FORMAT_Y10B | FPS_HIGH}, };

I kept FORMAT_Y10B and V4L2_PIX_FMT_Y10BPACK the same, mainly because I could not see a 11 bit format that seemed to fit. Is this correct? All I think I did was change the buffer size in bytesperline and sizeimage. Do you have a code snipper Alexander that I should put here?

The other bit of info I have is:

oot@woofieone:/dev# lsusb Bus 002 Device 008: ID 045e:02ae Microsoft Corp. Xbox NUI Camera Bus 002 Device 003: ID 045e:02b0 Microsoft Corp. Xbox NUI Motor Bus 002 Device 004: ID 045e:02ad Microsoft Corp. Xbox NUI Audio Bus 002 Device 002: ID 0409:005a NEC Corp. HighSpeed Hub Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

And if I do the lsusb -v that shows I think the correct stuff too but it is a bit long so I will not paste it here.

And finally lsmod shows:

root@woofieone:/dev# lsmod Module Size Used by kinect 3907 0 gspca 22986 1 kinect omap_aes 12590 0 omap_sham 17873 0 ti_am335x_adc 5138 0 kfifo_buf 2580 1 ti_am335x_adc industrialio 48147 2 ti_am335x_adc,kfifo_buf rtc_omap 5026 0 uio_pdrv_genirq 3469 0 uio 8749 1 uio_pdrv_genirq

Many thanks Alexander.

Warmest regards, Mike.

stinkytiger avatar Jun 15 '14 01:06 stinkytiger

Hi,

OK I think I am getting no data at all. So does not work right now.

In Dmesg I see:

[15174.152039] usb 2-1.3: new high-speed USB device number 21 using musb-hdrc [15174.255523] usb 2-1.3: device v045e p02ae is not supported [15174.261339] usb 2-1.3: New USB device found, idVendor=045e, idProduct=02ae [15174.261374] usb 2-1.3: New USB device strings: Mfr=2, Product=1, SerialNumber=3 [15174.261403] usb 2-1.3: Product: Xbox NUI Camera [15174.261430] usb 2-1.3: Manufacturer: Microsoft [15174.261458] usb 2-1.3: SerialNumber: A00363W01604210A [15174.270163] gspca: kinect_depth-2.14.0 probing 045e:02ae [19824.912233] usbcore: deregistering interface driver kinect_depth [19837.077670] gspca: Unknown symbol GLOBAL_OFFSET_TABLE (err 0) [19993.927702] gspca: v2.14.0 registered [19999.221586] gspca: kinect_depth-2.14.0 probing 045e:02ae [19999.226726] usbcore: registered new interface driver kinect_depth [21150.215431] kinect: send_cmd: Bad tag 0004 != 0005 [21150.220600] kinect: send_cmd: Output control transfer failed (-71) [21150.227306] kinect_depth 2-1.3:1.0: urb status: -71 [21150.232412] kinect_depth 2-1.3:1.0: urb status: -71kinect_depth 2-1.3:1.0: urb status: -71 [21150.241280] kinect_depth 2-1.3:1.0: urb status: -71kinect_depth 2-1.3:1.0: urb status: -71 [21150.250203] kinect_depth 2-1.3:1.0: urb status: -71kinect_depth 2-1.3:1.0: urb status: -71 [21150.259051] kinect_depth 2-1.3:1.0: urb status: -71kinect_depth 2-1.3:1.0: urb status: -71 [21150.267911] kinect_depth 2-1.3:1.0: urb status: -71kinect_depth 2-1.3:1.0: urb status: -71 [21150.276779] kinect_depth 2-1.3:1.0: urb status: -71kinect_depth 2-1.3:1.0: urb status: -71 [21150.285650] kinect_depth 2-1.3:1.0: urb status: -71kinect_depth 2-1.3:1.0: urb status: -71 [21150.294662] kinect_depth 2-1.3:1.0: urb status: -71kinect_depth 2-1.3:1.0: urb status: -71 [21150.303506] gspca: usb_submit_urb() ret -19 [21150.308212] usb 2-1.3: USB disconnect, device number 21 [21150.308273] kinect_depth 2-1.3:1.0: urb status: -71 [21150.313268] gspca: usb_submit_urb() ret -19 [21150.318025] kinect_depth 2-1.3:1.0: urb status: -71 [21150.323021] gspca: usb_submit_urb() ret -19 [21153.282038] usb 2-1.3: new high-speed USB device number 22 using musb-hdrc [21153.385368] usb 2-1.3: device v045e p02ae is not supported [21153.391178] usb 2-1.3: New USB device found, idVendor=045e, idProduct=02ae [21153.391215] usb 2-1.3: New USB device strings: Mfr=2, Product=1, SerialNumber=3

Hmm ... not sure what is happening.

Looks like the camera is never initalized. I.E. the Green LED keeps on flashing. Waiting for a command.

Best, Mike,

stinkytiger avatar Jun 15 '14 17:06 stinkytiger

Hi,

I think I found the issue at least on a strace: open("grab.raw", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3 open("/dev/video0", O_RDONLY) = 4 fstat64(4, {st_mode=S_IFCHR|0660, st_rdev=makedev(81, 0), ...}) = 0 ioctl(4, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 0xbe8c754c) = -1 ENOTTY (Inappropriate ioctl for device) mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb6f8a000 read(4, 0xbe8c76a0, 307200) = -1 ENODEV (No such device) read(4, 0xbe8c76a0, 307200) = -1 ENODEV (No such device) fstat64(3, {st_mode=S_IFREG|0664, st_size=0, ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb6f72000 write(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 307200) = 307200 close(3) = 0

from the source code:

#include <stdio.h> #include #include <stdlib.h> #include <string.h> #include "kinectgo.h"

int main() { FILE *camera, *grab; grab = fopen("grab.raw", "wd"); camera = fopen("/dev/video0", "rb"); uint8_t framebuffer[FRAMEBUFSZ]; memset(framebuffer, NULL, FRAMEBUFSZ * sizeof(uint8_t)); fread(framebuffer, sizeof(uint8_t), FRAMEBUFSZ, camera); fread(framebuffer, sizeof(uint8_t), FRAMEBUFSZ, camera); fwrite(framebuffer, sizeof(uint8_t), FRAMEBUFSZ, grab); fclose(grab); fclose(camera); return 0; }

So I guess although the device is there, it is a dead device for some reason. Even though it shows on lsusb

stinkytiger avatar Jun 15 '14 18:06 stinkytiger

Have some data with a strace I see:

read(4, "\344\271.K\222\344y\36G\221\3449\16C\220\3449\16C\220\343\370\376?\216\343\270\356;\216\343\370"..., 258048) = 7884 read(4, "\377\371.G\220\3449\36G\221\344y\36G\221\344y\36C\220\343\370\376;\216\343\370\2767\215\343\270"..., 249856) = 46308 read(4, "\377\371\36K\222\344y.K\221\344y\16G\220\3449\16C\221\3449\16C\220\343\370\356?\217\3449"..., 204800) = 7340 read(4, "\377\371\36G\221\3449\16C\220\3449\16C\220\3449\16C\221\3

I think there is some sort of dead-lock bug somewhere. To get it to go I need to do a dmesg. Then it is happy-ish. I.E. Yes I get data, but drop loads of frames. Hence I am seeing lots of:

[ 2748.162911] kinect: [Stream 70] Invalid magic 0f43 [ 2748.168085] kinect: [Stream 70] Invalid magic e97a [ 2748.173104] kinect: [Stream 70] Invalid magic ffff [ 2748.178123] kinect: [Stream 70] Invalid magic 033f [ 2748.183142] kinect: [Stream 70] Invalid magic ff3f [ 2748.188314] kinect: [Stream 70] Invalid magic f73e [ 2748.193334] kinect: [Stream 70] Invalid magic ff3f

in the dmesg. Presumably the code is waiting for the End of Frame code to come by. However because of all the dropped frames, the magic number does not appear in the right place and hence the code complains. The rate that sort of runs at is a Frame a second, which is not great. And I suspect it is some sort of mish-mash of frames. I.E. I print a "Frame Done" after I get 307,200 bytes. But this must be some sort of random assortment of data. Oh well progress. At least it is not all zeros and hence no data ... :)

Best, Mike.

stinkytiger avatar Jun 15 '14 23:06 stinkytiger

Hi,

I do not know your setup and can only guess here.

So lets try to assume its the easiest to fix problem. :)

I had similar problems when the Kinect was under powered, I even could grep a single frame sometimes.

Have you tried to connect your Kinect through a active powered USB-Hub?

This did the trick in some cases.

Let me know!

xxorde avatar Aug 11 '14 08:08 xxorde

Hi Alex,

Thanks for getting back to me. I did make a bit more progress since I posted. Your driver works-ish on the BeagleBone Black. The problem is not your driver but the BeagleBone Black USB driver. In particular the MUSB CPPI41 dma driver. For a long time that driver was broken because of a ARM TI hardware bug. The interrupt is sent on the USB when the data starts not when it is ready to be read at the end of a receive. So the fudge fix has been (a) to not use DMA which equals slow programmed reads instead, or (b) put in a time delay hack to delay the interrupt dependent on the USB message length. Neither is that great. Lots of E-mail on the Linux ARM kernel chain.

The practical upshot of this is that your diver works well for the first bunch of transfers on a video frame. Eventually the BeagleBone black looses the keep up battle of speed and drops a transfer. Unfortunately that means the depth video frame goes all squiffy. I have been trying to fix if in adding to your driver some of the Sync code from LibFreenect. I.E. if a frame is dropped, it fills in with some "blank" data so that the pixels that you do get are correctly placed.

Have been on my project for a while so may not be able to get back to you with a "result" any time soon.

Power wise I did try a powered USB, a Dlink and an EBay special. Neither changed things. I am powering my BBB with a 5v PSU not via a USB lead, so I think I have good Amps in there.

Thanks again for your kind help.

Warmest regards, Mike.

On Mon, 2014-08-11 at 01:06 -0700, Alexander wrote:

Hi,

I do not know your setup and can only guess here.

So lets try to assume its the easiest to fix problem. :)

I had similar problems when the Kinect was under powered, I even could grep a single frame sometimes.

Have you tried to connect your Kinect through a active powered USB-Hub?

This did the trick in some cases.

Let me know!

— Reply to this email directly or view it on GitHub.

stinkytiger avatar Aug 11 '14 22:08 stinkytiger