cbird icon indicating copy to clipboard operation
cbird copied to clipboard
verified publisher icon scrubbbbs

Speed up indexing process on windows? How to use gpu? How to use skip-loop-filter?

Open PeterCodar opened this issue 1 year ago • 4 comments

For windows users, it is a little bit difficult to get started.

I have downloaded cbird-windows-0.7.0-x86_64.zip and extracted it locally. Then I found this minimalistic working command line

cbird -use \\192.168.1.2\share\folder\subfolder -update

Unfortunately, indexing a single video file on my NAS seems to take about 30 seconds to finish. With about 27'500 video files in different formats and sizes, this would need about 229 hours...

What can I do to speed up this indexing process?

I read something about skip-loop-filter but can't find the working syntax. How exactly do I use skip-loop-filter?

I read something about gpu <bool> Enable gpu video decoding (Nvidia) [false] but can't find the working syntax. How exactly do I use gpu? In the CMD line help there is a section "Index Parameters (for -update)" but the examples talk about -p.types. Should that not be -i.types and i.gpu? Whatever I try to enable my Nvidia RTX gpu, it does not seem to work no hw codec, trying sw and does not speed up the index process:

win32 console detected mode=0x3

[D][Database::Database] "\\\\192.168.1.2\\Test\\_Diverse\\t1"
[D][Database::connect] thread:0x21b084b42e0 sqlite_0_0 //192.168.1.2/Test/_Diverse/t1/_index/media0.db
[D][Database::connect] thread:0x21b084b42e0 sqlite_1_1 //192.168.1.2/Test/_Diverse/t1/_index/media1.db
[D][Database::connect] thread:0x21b084b42e0 sqlite_2_2 //192.168.1.2/Test/_Diverse/t1/_index/media2.db
[D][Database::connect] thread:0x21b084b42e0 sqlite_3_3 //192.168.1.2/Test/_Diverse/t1/_index/media3.db
[W][Database::lastAdded] missing timestamp file, -update may ignore modified files
//192.168.1.2/Test/_Diverse/t1 i:0 v:0 ign:0 mod:0 ok:0                                                                                                                                      //192.168.1.2/Test/_Diverse/t1 i:0 v:1 ign:0 mod:0 ok:0                                                                                                                                      
[D][Scanner::scanDirectory] estimate cost=291505504256.00 path=//192.168.1.2/Test/_Diverse/t1/2,5 GB test video file.mp4
[I][Scanner::scanDirectory] scan completed, indexing 1 additions...
[D][VideoContext::open{2,5 GB test video file.mp4}] trying hw codec : h264_cuvid
[D][VideoContext::open{2,5 GB test video file.mp4}] no hw codec, trying sw
queued:image=0,video=0:batch=1,threadpool:gpu=0,video=1,global=0
[D][VideoContext::convertFrame{2,5 GB test video file.mp4}] yuv420p "@1280x720" => yuv420p "@128x128" area fast
queued:image=0,video=0:batch=1,threadpool:gpu=0,video=1,global=0
[D][Media::makeVideoIndex{2,5 GB test video file.mp4}] 1280x720 128px 7:1 CPU(16) 3922fps 6%
queued:image=0,video=0:batch=1,threadpool:gpu=0,video=1,global=0
[D][Media::makeVideoIndex{2,5 GB test video file.mp4}] 1280x720 128px 7:1 CPU(16) 3742fps 12%
queued:image=0,video=0:batch=1,threadpool:gpu=0,video=1,global=0
[D][Media::makeVideoIndex{2,5 GB test video file.mp4}] 1280x720 128px 7:1 CPU(16) 3970fps 18%
queued:image=0,video=0:batch=1,threadpool:gpu=0,video=1,global=0
[C][Media::makeVideoIndex{2,5 GB test video file.mp4}] "//192.168.1.2/Test/_Diverse/t1/2,5 GB test video file.mp4" greater than 64k frames unsupported, quitting
[D][Media::makeVideoIndex{2,5 GB test video file.mp4}] //192.168.1.2/Test/_Diverse/t1/2,5 GB test video file.mp4 nframes=65536 near=57136 filt=0 corrupt=0
[D][Scanner::processVideo{2,5 GB test video file.mp4}] perf codec=h264 bitrate=1813136 pixels/ms=17164546.0
[D][Database::add] count=1 write=3+8+0+6=17 ms
[I][Scanner::processFinished] indexing completed

I also think that saving the database and *.vdx files locally on a fast internal SSD would speed up the process (instead of writing the _index folder and the files over LAN and on a "slow" NAS hard drive.) How can I tell cbird to create the database and index files locally?

Do you know any other CMD line hints that would speed up the process with files over 1 Gbit/s-LAN saved on a NAS?

PeterCodar avatar Apr 25 '24 06:04 PeterCodar

i have the same issues when i use -i.gpu true, and when this error vdx appear i have to start from the beginning, gpu the options is bugged a lot, for normal mode you can speed up the process a little, use link to the local folder "mklink" and use -i.links true anyway i think this project is abandoned sad. its my command .\cbird.exe -p.alg video -p.types v -p.eg 1 -i.algos 16 -p.vtrim 3700 -i.links true -p.vfm 200 -p.dht 3 -use Y:\lotsdupes -similar -show

the _index folder is on disc c:\ (ssd)

tehMArs avatar Apr 30 '24 10:04 tehMArs

Thanks for the feedback. Unfortunately video search needs to decode all video frames, it is the most reliable method I know of. Probably something to use remote indexers so multiple machines can get involved would be good. At the moment you would have to divide up your files into clusters you want to manage separately and index each one on a different machine.

  • skip_loop_filter is enabled starting in v0.7
  • -i.gpu 1 enables gpu decoding for NVidia only, -i.gputhr 2 says to use at most 2 parallel gpu decoders, using more is usually needed to saturate the gpu codecs, but it depends on the video dimensions/codec.
  • using gpu doesn't disable cpu decoding, but the gpu takes priority IIRC (haven't tested this lately). Each video decoder consumes an index thread -i.idxthr so you can reduce the cpu load that way.
  • I didn't explicitly compile FFmpeg hardware codec support into the windows binary package, it might not be there - also it hasn't been tested. You can check the included ffplay.exe -codecs and look for codecs with "cuvid" in the name. It is probably not there but could be made to work.
  • -i.decthr 1 will give one thread per decoder/job, for a total of 16 inflight jobs in your case. This should be more efficient, but you lose 16 jobs worth of work if you cancel indexing to do something else. If you run out of RAM or the network can't keep up (it probably can for 16 jobs), increase in multiples of two.
  • having more inflight jobs should mask any latency provided you have the bandwidth. You can control this with -i.idxthr <n> , the default is the cpu core count

scrubbbbs avatar May 16 '24 13:05 scrubbbbs

Thank you for helpful answer!

Just a short comment: There is no ffplay.exe included in the cbird-windows-0.7.0-x86_64.zip

What about my question "how to create the database and index files locally"?

PeterCodar avatar May 16 '24 13:05 PeterCodar

Sorry this is late, I am currently mopping up issues for a release sometime soon.

I didn't explicitly compile FFmpeg hardware codec support into the windows binary package, it might not be there - also it hasn't been tested. You can check the included ffplay.exe -codecs and look for codecs with "cuvid" in the name. It is probably not there but could be made to work.

Cuvid codecs will be available in the next Windows binary release, it was an issue with the build recipe. I haven't been able to test it yet (all my windows "machines" are VMs w/o gpus).

There is no ffplay.exe included in the cbird-windows-0.7.0-x86_64.zip

This will be included in the future.

What about my question "how to create the database and index files locally"?

You have to use symlinks with the Windows mklink command. You can either link the _index folder to somewhere on an SSD or link the content folder(s) on the ssd, then use -i.links true -update

scrubbbbs avatar Feb 19 '25 14:02 scrubbbbs