dcscn-super-resolution
dcscn-super-resolution copied to clipboard
jiny2001
Multi threaded batch generator and loader
The following code allows the Build_Batch option to be multi threaded for large (or small) datasets. This increases the speed of generating batch images as well as loading already generated batch images.
Hi, thank you so much for sending the request!!
Sorry for being silent for a while. Since I need to review and double check with the running code, it's taking time. So far from the quick look, I'm thinking 1) if we can make 8 thread objects into array so that the code can be much shorter and also thinking 2) it might be good to separate batch building function from main training function so that there will be a big benefit in terms of memory consumption and code readability.
I will try your code hopefully in this weekend and give you a feedback. Thx!!
I think I can make that happen. Good idea about putting the threads in an array, I never really considered that. I re-updated the code today, so if you're still using an older version I don't think it will work.
On Thu, Sep 5, 2019, 4:34 PM Jin Yamanaka [email protected] wrote:
Hi, thank you so much for sending the request!!
Sorry for being silent for a while. Since I need to review and double check with the running code, it's taking time. So far from the quick look, I'm thinking 1) if we can make 8 thread objects into array so that the code can be much shorter and also thinking 2) it might be good to separate batch building function from main training function so that there will be a big benefit in terms of memory consumption and code readability.
I will try your code hopefully in this weekend and give you a feedback. Thx!!
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jiny2001/dcscn-super-resolution/pull/97?email_source=notifications&email_token=AFZDDIVBEZ5MXOWYJ3YTSGLQIF3NPA5CNFSM4ITKRB72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6A4JYY#issuecomment-528598243, or mute the thread https://github.com/notifications/unsubscribe-auth/AFZDDIQ3EHZKQW2JF74HRBTQIF3NPANCNFSM4ITKRB7Q .
That's great, thank you! I will work on the code and will put some comments or suggestions after you updated. Thx!
done and done. It also no longer has a limit to the total number of threads you can launch.
I can move the code to it's own file, but i think it would probably be better if you do that. I'm not really sure what code you want or where you want it moved to.
It should load the correct data for each thread now. Is there anything else i can help with?
Hi ereish64! Sorry for not responding long time... Currently I am having a long business trip and very busy. But will try to review and merge the code on this weekend. Thx!!
No worries!
On Sat, Mar 7, 2020, 10:11 PM Jin Yamanaka [email protected] wrote:
Hi ereish64! Sorry for not responding long time... Currently I am having a long business trip and very busy. But will try to review and merge the code on this weekend. Thx!!
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jiny2001/dcscn-super-resolution/pull/97?email_source=notifications&email_token=AFZDDIX4GW3VTCPUAU4KV4TRGMSG7A5CNFSM4ITKRB72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOEMXMY#issuecomment-596167603, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFZDDITOLTKH45NO63N27EDRGMSG7ANCNFSM4ITKRB7Q .