Stop multiple plots copying over to the HDD at the same time.

[ptyx11]

Often when plots finish right after another and they are being copied over to the HDD the process slows to a halt. I think the plotman should do this one at a time to avoid issues with write speeds to the HDD.

[ilovepancakes95]

Copying to the DST location slowed to a halt today for me too and I am guessing it was because too many copies were happening at once, but even when one finished, the others were all messed up and still transferred very slowly.

[randellhodges]

If you are only running 1 instance of plotman, you can set the dst to be the same as tmp and then setup the archive process and use rsync (even if you are staying on the same machine). Plotman currently only allows 1 archive process to run so you can use that to serialize copying from the completed plot out of tmp to the archive location.

[ilovepancakes95]

setup the archive process and use rsync

I would but it seems overly complicated to setup and run archive via rsync compared to simply mounting an SMB share to my TrueNAS instance.

[ptyx11]

If you are only running 1 instance of plotman, you can set the dst to be the same as tmp and then setup the archive process and use rsync (even if you are staying on the same machine). Plotman currently only allows 1 archive process to run so you can use that to serialize copying from the completed plot out of tmp to the archive location.

can we just add a function where the plotman looks for 2 plots on step 4 and sets one on suspend while the first one completes and then resumes? that way we don't have to mess with archiving?

[randellhodges]

If you are only running 1 instance of plotman, you can set the dst to be the same as tmp and then setup the archive process and use rsync (even if you are staying on the same machine). Plotman currently only allows 1 archive process to run so you can use that to serialize copying from the completed plot out of tmp to the archive location.

can we just add a function where the plotman looks for 2 plots on step 4 and sets one on suspend while the first one completes and then resumes? that way we don't have to mess with archiving?

Anything is possible. If someone authors and submits a pull request, at some point in the future, it might get integrated. The alternative I mentioned above, if the caveats apply, can be implemented immediately.

[randellhodges]

setup the archive process and use rsync

I would but it seems overly complicated to setup and run archive via rsync compared to simply mounting an SMB share to my TrueNAS instance.

I thought that as well. I am not familiar with TrueNAS, but I followed the directions with https://github.com/ericaltendorf/plotman/wiki/Archiving and it was quick to setup and get working. The only issue might be getting the SSH keys copied over if you are not used to that.

[ptyx11]

If you are only running 1 instance of plotman, you can set the dst to be the same as tmp and then setup the archive process and use rsync (even if you are staying on the same machine). Plotman currently only allows 1 archive process to run so you can use that to serialize copying from the completed plot out of tmp to the archive location.

can we just add a function where the plotman looks for 2 plots on step 4 and sets one on suspend while the first one completes and then resumes? that way we don't have to mess with archiving?

Anything is possible. If someone authors and submits a pull request, at some point in the future, it might get integrated. The alternative I mentioned above, if the caveats apply, can be implemented immediately.

I'm going to try it, i'm a bit confused though. So do I change the dst directory to my SSD paths instead of the HDD path and activate Archival to my remote farm? I already setup the rsync and archive but it's not clear how I remove the final dst to just leave them in the SSD until archival

[randellhodges]

Assuming you are running the main branch of plotman and you are using a single tmp (example here, it is mounted at /mnt/chia/tmp). This trick only works if you have a single tmp on the current main/release code.

directories:
        tmp:
                - /mnt/chia/tmp
        dst:
                - /mnt/chia/tmp

My understanding is that, when chia create moves it to the dst, since it is the same, the move will actually just result in a rename. You'll then have a finished .plot file sitting in your tmp folder.

The archive process will then come along, rsync it out of tmp and then delete it from tmp.

Note: There is work being done in development (not sure if it was yet merged) that all you had to do was not define a dst and that would cause plotman to leave it in tmp. That will help if you have multiple tmps.

[ptyx11]

ok thanks, i'm trying to find a python dev that can work on the solution i proposed. 3 of them already turned me down though.