mono_dataset_code Vignette calibration problem

Vignette calibration problem

Open fairf4x opened this issue 7 years ago • 53 comments

I am running vignetteCalib on my own dataset - just 30 images to test the approach. The program produces only black vignette and following output:

Load Dataset samples_vignette3/: found 30 files in folder /images; assuming that all images are there.
Input resolution: 856 480
Input Calibration (fx fy cx cy): 338.125397 194.108505 269.381378 83.858086 1.000000
NO RECTIFICATION
Reading Photometric Calibration from file samples_vignette3/pcalib.txt
Reading Vignette Image from samples_vignette3/vignette.png
PhotometricUndistorter: Invalid vignette image size! got 0 x 0, expected 856 x 480. Set vignette to 1.
Dataset samples_vignette3/: Got 30 files!
SEQUENCE NAME: samples_vignette3/!
plane image values 10000000000.000000 - -10000000000.000000!
0.000000 residual terms => -nan
0.000000 residual terms => -nan
plane image values 10000000000.000000 - -10000000000.000000!
0.000000 residual terms => -nan
0.000000 residual terms => -nan
...

the output is in the same fashion until the end.

Any idea what might be wrong with my dataset? Is it possible that it does not work because it is not composed from video capture? (I can only get exposure times for the third column of times.txt when shooting images one by one with the camera)

Feb 28 '17 21:02 fairf4x

Hey @fairf4x , Did you end up finding the solution for this? I am also trying to do vignettecalib and pcalib for my camera. For vignette calib, is there a certain way that you need to move the camera or is it just arbitrary? For pcalib, I have no idea where to even get started. Do you know what exactly to do for this? Thanks in advance.

Apr 07 '17 08:04 vannem95

I didn't manage to produce vignettecalib using vignetteCalib yet, but I was able to produce a pcalib.txt file from my own dataset using responseCalib. I will try to explain the process.

Suppose you have a directory with your dataset called sweepData and you want to use responseCalib to create pcalib.txt for you.

the directory sweepData should have following content:

sweepData:
   camera.txt
   images.zip
   times.txt

camera.txt should contain camera calibration as described here My camera.txt look like this:

0.39500631 0.40439271 0.31469786 0.17470434 1.0
856 480
none
856 480

It can be written by hand on condition you know the intrinsic parameters for your camera.

images.zip should be an archive of jpg files with names in format: XXXX.jpg where XXXX is imageID (leading zeroes starting with 0000)
times.txt is a text file with three whitespace separated columns: imageID timestamp exposureTime Where timestamp is a POSIX time and exposureTime is a real number denoting exposure time in ms

Calling responseCalib like this:

./responseCalib /path/to/data/sweepData/

should produce directory with pcalib.txt

Apr 11 '17 22:04 fairf4x

Hey @fairf4x , First off thanks for the response. I understand camera.txt. As per images, i looked at a sample dataset here (https://vision.in.tum.de/data/datasets/mono-dataset?redirect=1) which had a constant image with increasing exposures. What did you use? Also, how did you get the times.txt file or the corresponding data in it? Thanks in advance.

Edit1: I understand what everything is but i am not sure how to get the data. I can only change the exposure of my camera using v4l2 thru terminal and can't using videocapture(opencv). How did you get the data for times.txt? Thanks

Edit2: I got the images with increasing exposures. I changed the absolute exposure to get them. But I don't know what the values are in msec. Do you know how to get that?

Apr 12 '17 18:04 vannem95

Hey @fairf4x , So, I used videocapture and v4l2 to get the images with different absolute exposure values. But I am not sure how to get the exposure in msec. I looked at the difference between POSIX time of any consecutive frames and it is always constant(0.67sec (1/15th of a s)as my camera only does 15 fps). Did you have different msec exposure values or about the same?

Thanks in advance.

Apr 18 '17 06:04 vannem95

Hi, I did not capture video. I am experimenting with drone camera so my control over the camera settings is limited. For "sweep" dataset I was using a simple ROS node that automatically shoots a picture after some interval (i.e. 1.5s) and increases exposure of the camera between -3.0 and +3.0 with step 0.5. The values -/+ 3.0 are bounds for the camera exposure given by the drone API. I wasn't able to figure out units used here.

My only idea about how the dataset should look like comes from investigation of the sample datasets. Framerate of "sweep" datasets seems to be 10 fps and exposure values are varying from as low as 0.0597894751 to 19.2336845398. This might be in ms. However when I tried to change scale of exposure times in my own dataset (e.g. by multiplying with 100) it did not seem to have any effect on the result of responseCalib. I suspect that the units are not important here.

Apr 18 '17 21:04 fairf4x

Thanks for that, @fairf4x . So, I tried out some sample videos/images. I used different sample sizes and scenarios (for the camera). All of them had 12-14 "-nan"s in the front and then regular numbers in the pcalib.txt. In general the numbers are increasing but the distribution of the numbers was not like 0 to 255 like they had in the example. My numbers started from around 60 and went to 255. What were your pcalib.txt numbers like? Did you get any "-nan"s at all? What did your image/video have? Do you have any suggestions that would give me a better pcalib?

Also, did you figure out how to do the vignette calib?

Thanks for your help. I appreciate it.

Apr 20 '17 08:04 vannem95

@vannem95 Inverse of gamma is captured as G(x) which is dumped as '.png' images as the optimization progresses. If there are no images with very low exposure times, they are set to 0 (and hence the inverse of the value is Not a Number). You could solve this by adding images with very low exposure times (preferably, lots of them)

@fairf4x were the aruco markers detected and displayed when you ran the calibration? Between these two steps? SEQUENCE NAME: samples_vignette3/! plane image values 10000000000.000000 - -10000000000.000000!

For instance, my values start with an initialization of : plane image values 1.264622 - 131.889465! 100646408.000000 residual terms => 7213.918457 and coverage from there.

Is it possible that it does not work because it is not composed from video capture? (I can only get exposure times for the third column of times.txt when shooting images one by one with the camera)

Regarding your original question. This shouldn't matter; The authors too have used a similar setup.

Have you set the exposure to a constant or reading values calculated by auto exposure for this step? My setup was the latter.

Apr 21 '17 05:04 anilprasadmn

@anilprasadmn Thanks. I will try that out. By the way, do you know how to perform vignettecalib?

Apr 21 '17 05:04 vannem95

@vannem95 Yes, I got the vignette calibration working.

Apr 21 '17 06:04 anilprasadmn

Do you see a "vignettecalib" executable in you bin? Because when i did cmake and make, it didnt get built. @anilprasadmn

Apr 21 '17 06:04 vannem95

You need the pcalib.txt from the responsecalib as an input for vignettecalib. The steps are as described by @fairf4x . Create a folder with images.zip (or just a folder by the name 'images') of the aruco marker images along with pcalib.txt, camera.txt and times.txt. (similar to the folder structure for responsecalib with the addition of pcalib.txt) Once you have this, you can run the vignettecalib on this dataset (mind the trailing slash)

Apr 21 '17 06:04 anilprasadmn

@anilprasadmn Also are we supposed to move the camera around like they did in the narrow_vignette video?

Apr 21 '17 06:04 vannem95

Yes, you need to tile all of the visible portion of your image view with the marker grid (which is 5 times the markers size, since this region is used for calculation. make sure its a non-glossy white plane)

Apr 21 '17 06:04 anilprasadmn

@anilprasadmn Cool. Thank you so much for helping out. One last thing, when i did cmake and make, the vignettecalib executable didn't get built in the bin folder like playdataset and responsecalib. Did it get built for you?

Apr 21 '17 06:04 vannem95

You need to have aruco installed for the vignettecalib to be compiled. (It is included in the Thirdparty folder and instructions can be found in the mono dataset github description page)

Apr 21 '17 06:04 anilprasadmn

@anilprasadmn Great. Thank you so much for your help, anil. You are a lifesaver.

Apr 21 '17 06:04 vannem95

@anilprasadmn Hey anil, sorry to bother you. but i still keep getting at least 10 nans(Got 12-14 before). Attached is the terminal output i am getting:

Load Dataset /home/vivek/sweepData/: found no in folder /images; assuming that images are zipped. got 9999 entries and 9999 files from zipfile! Input resolution: 640 480 Input Calibration (fx fy cx cy): 363357.156250 271843.906250 193337.625000 110752.367188 0.089228 Out: Crop Output resolution: 640 480 new K: 362791.531250 271664.718750 193053.156250 110654.179688 old K: 363357.156250 271843.906250 193337.125000 110751.867188

FOV Undistorter: Warning! Image has black pixels.

Reading Photometric Calibration from file /home/vivek/sweepData/pcalib.txt PhotometricUndistorter: Could not open file! Dataset /home/vivek/sweepData/: Got 9999 files! loaded 9999 images init RMSE = 127948.423747! Irradiance 17.220422 - 28.188419 init done optG RMSE = 30114.734982! Inv. Response 28233.615821 - 275309.873451 OptE RMSE = 27962.417690! Irradiance 8.876896 - 28.295072 resc RMSE = 30.205212! rescale with 0.001080! optG RMSE = 29.293719! Inv. Response 21.915380 - 297.203783 OptE RMSE = 28.536765! Irradiance 0.008563 - 0.029328 resc RMSE = 25.127685! rescale with 0.880537! optG RMSE = 24.489668! Inv. Response 17.835236 - 250.958528 OptE RMSE = 23.868996! Irradiance 0.007119 - 0.024561 resc RMSE = 24.928692! rescale with 1.044396! optG RMSE = 24.297015! Inv. Response 17.647475 - 249.256340 OptE RMSE = 23.681357! Irradiance 0.007059 - 0.024371 resc RMSE = 24.908544! rescale with 1.051821! optG RMSE = 24.277392! Inv. Response 17.628362 - 249.083994 OptE RMSE = 23.662233! Irradiance 0.007053 - 0.024351 resc RMSE = 24.906394! rescale with 1.052580! optG RMSE = 24.275297! Inv. Response 17.626335 - 249.065543 OptE RMSE = 23.660191! Irradiance 0.007052 - 0.024349 resc RMSE = 24.906168! rescale with 1.052661! optG RMSE = 24.275076! Inv. Response 17.626123 - 249.063598 OptE RMSE = 23.659976! Irradiance 0.007052 - 0.024349 resc RMSE = 24.906144! rescale with 1.052670! optG RMSE = 24.275053! Inv. Response 17.626100 - 249.063394 OptE RMSE = 23.659953! Irradiance 0.007052 - 0.024349 resc RMSE = 24.906142! rescale with 1.052671! optG RMSE = 24.275051! Inv. Response 17.626098 - 249.063373 OptE RMSE = 23.659951! Irradiance 0.007052 - 0.024349 resc RMSE = 24.906142! rescale with 1.052671! optG RMSE = 24.275051! Inv. Response 17.626098 - 249.063371 OptE RMSE = 23.659951! Irradiance 0.007052 - 0.024349 resc RMSE = 24.906142! rescale with 1.052671!

I am using a 15fps logitech 9000 quickcam pro on which i am changing the absolute exposure from 1 to 10000 using uvc drivers. My images is a bent cardboard covering the glossy floor. The exposure in my times.txt is 1000/(absolute exposure){just a guess. don't know how to get the exposure in msec}. Do you see any obvious mistakes i might be doing [or] any suggestions to solve this issue? Thanks in advance

Apr 21 '17 10:04 vannem95

Can you also provide the times.txt with a sample image? The value in the 3rd column should be exposure in ms. [shutter speed/ 1000 if you are using v4l2. I read exposure through the exif data since had issues with v4l2] #4

Also, I hope the white balance etc is set appropriately.

Apr 21 '17 11:04 anilprasadmn

Here is the times.txt:

1492802212.15 0000 0.100010 1492802212.21 0001 0.100020 1492802212.28 0002 0.100030 1492802212.35 0003 0.100040 1492802212.41 0004 0.100050 1492802212.48 0005 0.100060 1492802212.55 0006 0.100070 1492802212.61 0007 0.100080 1492802212.68 0008 0.100090 1492802212.75 0009 0.100100 1492802212.81 0010 0.100110 1492802212.88 0011 0.100120 1492802212.95 0012 0.100130 1492802213.01 0013 0.100140 1492802213.08 0014 0.100150 .. .. ..

1492802886.62 9983 62.500000 1492802886.82 9984 66.666667 1492802887.02 9985 71.428571 1492802887.22 9986 76.923077 1492802887.42 9987 83.333333 1492802887.62 9988 90.909091 1492802887.82 9989 100.000000 1492802888.02 9990 111.111111 1492802888.22 9991 125.000000 1492802888.42 9992 142.857143 1492802888.62 9993 166.666667 1492802888.82 9994 200.000000 1492802889.02 9995 250.000000 1492802889.22 9996 333.333333 1492802889.42 9997 500.000000 1492802889.62 9998 1000.000000

I do not have control over exposure in msec. Here are my options on v4l2:

vivek-Precision-T1700: v4l2-ctl --all Driver Info (not using libv4l2): Driver name : uvcvideo Card type : UVC Camera (046d:0990) Bus info : usb-0000:00:1d.0-1.3 Driver version: 4.4.49 Capabilities : 0x84200001 Video Capture Streaming Extended Pix Format Device Capabilities Device Caps : 0x04200001 Video Capture Streaming Extended Pix Format Priority: 2 Video input : 0 (Camera 1: ok) Format Video Capture: Width/Height : 640/480 Pixel Format : 'YUYV' Field : None Bytes per Line : 1280 Size Image : 614400 Colorspace : sRGB Transfer Function : Default YCbCr Encoding : Default Quantization : Default Flags : Crop Capability Video Capture: Bounds : Left 0, Top 0, Width 640, Height 480 Default : Left 0, Top 0, Width 640, Height 480 Pixel Aspect: 1/1 Selection: crop_default, Left 0, Top 0, Width 640, Height 480 Selection: crop_bounds, Left 0, Top 0, Width 640, Height 480 Streaming Parameters Video Capture: Capabilities : timeperframe Frames per second: 15.000 (15/1) Read buffers : 0 brightness (int) : min=0 max=255 step=1 default=128 value=128 contrast (int) : min=0 max=255 step=1 default=32 value=32 saturation (int) : min=0 max=255 step=1 default=32 value=32 white_balance_temperature_auto (bool) : default=1 value=1 gain (int) : min=0 max=255 step=1 default=0 value=206 power_line_frequency (menu) : min=0 max=2 default=2 value=2 white_balance_temperature (int) : min=0 max=10000 step=10 default=4000 value=0 flags=inactive sharpness (int) : min=0 max=255 step=1 default=224 value=224 backlight_compensation (int) : min=0 max=2 step=1 default=1 value=1 exposure_auto (menu) : min=0 max=3 default=3 value=1 exposure_absolute (int) : min=1 max=10000 step=1 default=166 value=10000 exposure_auto_priority (bool) : default=0 value=1 focus (int) : min=0 max=255 step=1 default=0 value=0 led1_mode (menu) : min=0 max=3 default=3 value=3 led1_frequency (int) : min=0 max=255 step=1 default=0 value=0 disable_video_processing (bool) : default=0 value=0 raw_bits_per_pixel (int) : min=0 max=1 step=1 default=0 value=0

Apr 21 '17 19:04 vannem95

Here is why i went with that formula(exposure in msec = 1000/absolute exposure).

https://buildyourown.wordpress.com/2009/07/16/skype-logitech-quickcam-pro-9000/

The author of this post said the exposure range is 1/1000 to 1/5 of a sec and earlier versions had 1/10000 to 1/10 of a sec. The range seems to be 5 to 1000 and 10 to 10000 and the range i see on absolute exposure is 1 to 10000. So, i went with 1/10000 to 1/1 of a sec and i have no idea if it's wrong or right. I couldn't find a way to set the shutter speed.

Apr 21 '17 19:04 vannem95

Here is the responsecalib output when i run this dataset:

FOV Undistorter: Warning! Image has black pixels.

Reading Photometric Calibration from file /home/vivek/sweepData/pcalib.txt PhotometricUndistorter: Could not open file! Dataset /home/vivek/sweepData/: Got 9999 files! loaded 9999 images init RMSE = 126138.141890! Irradiance 17.052605 - 28.258626 init done optG RMSE = 29638.453328! Inv. Response 29438.359313 - 296847.513951 OptE RMSE = 27373.314080! Irradiance 8.413489 - 28.229015 resc RMSE = 23.514413! rescale with 0.000859! optG RMSE = 22.805304! Inv. Response 17.747669 - 256.516364 OptE RMSE = 22.224789! Irradiance 0.006380 - 0.023291 resc RMSE = 22.093410! rescale with 0.994089! optG RMSE = 21.540971! Inv. Response 16.254162 - 244.782224 OptE RMSE = 21.003272! Irradiance 0.005987 - 0.022036 resc RMSE = 21.879997! rescale with 1.041742! optG RMSE = 21.333924! Inv. Response 16.050825 - 242.683461 OptE RMSE = 20.801489! Irradiance 0.005926 - 0.021826 resc RMSE = 21.857195! rescale with 1.050751! optG RMSE = 21.311701! Inv. Response 16.029667 - 242.455629 OptE RMSE = 20.779821! Irradiance 0.005919 - 0.021804 resc RMSE = 21.854945! rescale with 1.051739! optG RMSE = 21.309507! Inv. Response 16.027596 - 242.433079 OptE RMSE = 20.777682! Irradiance 0.005919 - 0.021801 resc RMSE = 21.854728! rescale with 1.051837! optG RMSE = 21.309296! Inv. Response 16.027396 - 242.430903 OptE RMSE = 20.777476! Irradiance 0.005919 - 0.021801 resc RMSE = 21.854707! rescale with 1.051846! optG RMSE = 21.309276! Inv. Response 16.027377 - 242.430694 OptE RMSE = 20.777456! Irradiance 0.005919 - 0.021801 resc RMSE = 21.854705! rescale with 1.051847! optG RMSE = 21.309274! Inv. Response 16.027375 - 242.430674 OptE RMSE = 20.777454! Irradiance 0.005919 - 0.021801 resc RMSE = 21.854705! rescale with 1.051847! optG RMSE = 21.309273! Inv. Response 16.027375 - 242.430672 OptE RMSE = 20.777454! Irradiance 0.005919 - 0.021801 resc RMSE = 21.854705! rescale with 1.051847!

Apr 21 '17 21:04 vannem95

9998

Apr 24 '17 23:04 vannem95

Here is the image with the highest exposure.

Apr 24 '17 23:04 vannem95

@anilprasadmn Thoughts?

Apr 25 '17 03:04 vannem95

@vannem95 Your times.txt is not apropriate:

1492802212.15 0000 0.100010 It should have been in the format of the datasets provided. Have a look at them, it will help you figure out the best practice to capture data. ie., framenumber, timestamp,exposuretime

Make sure that there are no change in the scene (such as moving people casting shadows on the setup) when you are capturing the data. The setup looks fine (but could be brighter; again, best practice is to mimic the authors setup)

Apr 25 '17 06:04 anilprasadmn

Good catch. Can't believe i missed that. Thanks a lot. appreciate it.

Apr 25 '17 06:04 vannem95

hey @anilprasadmn , so that change didn't affect the result. i tried the calibs on a logitech c920. the calibs were very similar to the 9000 cam and i kept getting nans. but using a vignette image i found in one of the dso/dso-ros forums i got dso-ros to work barely. it fails if i dont move it very slowly. does it work fine for you? also do you know by chance how to output the pose of the camera? thanks in advance

May 11 '17 05:05 vannem95

@vannem95 Not all of the datasets I captured succeeded in converging and returning a valid result. [nor was I able to reproduce it with enough confidence.] So give it a couple of tries, read the paper multiple times and you need some luck as well. Prune your input data and scene to get better convergence. [look at the partial results that are dumped to get an idea of what might going wrong]

Vignette are specific for the lens. Since, they are not very critical (errors in it don't necessarily change the position of the energy functional) you could make do with a generic one. (for instance, my camera has an drastically different response to the authors, the images are brighter near the edge, in which replacing one with the other it will be worse than not giving it in)

I too face a similar problem and was hoping that @fairf4x would revert back with his results (whose thread we seem to have hijacked). I am using a Omnivision OV5674 sensor with a wide angle sunnex lens. I would like to see datasets with global shutter (so that I know what to blame for the errors)

Pose of the camera is dumped in results.txt

May 11 '17 10:05 anilprasadmn

I see that results.txt is a by product of dso. Also, when i run dso_ros, it either crashes or i quit it and i don't get any outputs other than the pangolin window. do i have to remove pangolin to see the results.txt file? @anilprasadmn

May 11 '17 21:05 vannem95

I needed to solve some issue with vignetteCalibration. I am using exposure times from EXIF information for my dataset images multiplied by 1000 (in order to obtain time in ms as suggested by @anilprasadmn).

Still it produces the same output:

plane image values 10000000000.000000 - -10000000000.000000!
0.000000 residual terms => -nan
0.000000 residual terms => -nan

Is there some simple way to check whether the aruco markers are recognized?

May 11 '17 21:05 fairf4x

mono_dataset_code mono_dataset_code copied to clipboard

Vignette calibration problem

mono_dataset_code
mono_dataset_code copied to clipboard