chia-plotter
chia-plotter copied to clipboard
madMAx43v3r
Observations on ramdisk for both drives
Was able to put together a test server with 384GB of ram and run some tests. Specs are:
Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz x 2 384GB DDR4 2 x Dell MZ7LH480HAHQ0D3 in raid 0 (Hooked up to the motherboard at 6Gbps Ramdisk 1 = 112GB Ramdisk 2 = 225GB buckets = 256
Phase 1: 620s give or take Phase 2: 310s Phase 3: 525s Phase 4: 75s
Total time: 1550s
I have 56 threads and there isn't much of a difference with 56 threads vs 40 threads. Where it gets really interesting is when I don't use ramdisk2 and use the SSD. My times are very similar using the SSDs. When monitoring the process I never really see more than 40 threads active at any point. This machine is dedicated to this testing so if you need debug logs or something let me know.
I am using 3990x with 64C 128T and 256GB RAM SSD: Intel p4610 Ramdisk: 110GB
result show me that, using 128 threads might fall in the trap of context switching of HyperThreads System using 33% in P1, 50 - 60% in P2 So you may try with -r 28, maybe it will give you same result
E5-2680 v4 has only 14 cores so you won't see any improvement over 28 "threads" in the plotter
@fiveangle There are 2 of them. I am going to disable HT and see what happens.
@fiveangle There are 2 of them
Yes, that is why I said 28thr is max perf you will get (2x 14cores). I've found keeping HT enabled allows the system to function well without contention, running basically off the scraps of cycles left over from the plot jobs, so leave HT enabled, just drop your threads count to match the number of physical cores you have.
I have 2x E5-2670 v2 (10-core w/ht) and generate a plot in ~2200s on flash while assigning 20-thr. Threads above that offers no improvement, and in fact degrades every so slightly.
What will be the minimum ramdisk space needed? I used size=110G and it created 118.1GB disk. it seems the command uses Gibibyte and not Gigabyte.
I want to set at at the maximum needed, so there is more system memory left for more threads.
Also how many threads would be optimal for 5950X and 5900X
I use 5900 (-r 12) with 2 pcie 4 nvme drives and my plots finish in 17XX seconds
E5-2680 v4 has only 14 cores so you won't see any improvement over 28 "threads" in the plotter
I use 5900 (-r 12) with 2 pcie 4 nvme drives and my plots finish in 17XX seconds
are you doing 2 job -r 12 at the same time? or just raid them
neither, each drive is plot temp (drive 1 = plot temp 1, drive 2 = plot temp 2)
Was able to put together a test server with 384GB of ram and run some tests. Specs are:
Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz x 2 384GB DDR4 2 x Dell MZ7LH480HAHQ0D3 in raid 0 (Hooked up to the motherboard at 6Gbps Ramdisk 1 = 112GB Ramdisk 2 = 225GB buckets = 256
Phase 1: 620s give or take Phase 2: 310s Phase 3: 525s Phase 4: 75s
Total time: 1550s
I have 56 threads and there isn't much of a difference with 56 threads vs 40 threads. Where it gets really interesting is when I don't use ramdisk2 and use the SSD. My times are very similar using the SSDs. When monitoring the process I never really see more than 40 threads active at any point. This machine is dedicated to this testing so if you need debug logs or something let me know.
I wonder if you would try with tmpdir1 = tmpdir2 (i.e. don't specify tmpdir2 at all). Max says peak usage will not exceed 256gb. Just create one 256gb ramdisk.
I wonder if you would try with tmpdir1 = tmpdir2 (i.e. don't specify tmpdir2 at all). Max says peak usage will not exceed 256gb. Just create one 256gb ramdisk.
Yeah, but if all you have is 256gb of ram, then you can't do anything else... or even create such a drive. I've found the actual usage to be about 250gb on a ramdisk, but you have to have your destination directory be an NVME drive anyway. If you don't move it from the ramdisk to another drive fast enough, the next plot will fill up the remainder of the ramdrive and the plotter will error out.
I wonder if you would try with tmpdir1 = tmpdir2 (i.e. don't specify tmpdir2 at all). Max says peak usage will not exceed 256gb. Just create one 256gb ramdisk.
Yeah, but if all you have is 256gb of ram, then you can't do anything else... or even create such a drive. I've found the actual usage to be about 250gb on a ramdisk, but you have to have your destination directory be an NVME drive anyway. If you don't move it from the ramdisk to another drive fast enough, the next plot will fill up the remainder of the ramdrive and the plotter will error out.
Yep I know - I got a system with lots of ram and I tried. I had 288gb, so an extra 32gb for other stuff. I also tried different types of ram drives on Linux, and Bcachefs with ramdrive caching and stuff. In the end I just stuck to using 8x enterprise sata ssd in raid0 for tmpdir2
Thing is, its so rare to have 256gb to begin with, let alone a motherboard that can handle it. Then to have to take it a step further is even more annoying. I have 256gb on my motherboard right now and my board isn't even rated for it, but its still taking it like a champ, go figure. But upping it to 288gb for that extra 32gb for other stuff... I just don't see that happening without buying some insanely expensive MB to replace my current one.
Thing is, its so rare to have 256gb to begin with, let alone a motherboard that can handle it. Then to have to take it a step further is even more annoying. I have 256gb on my motherboard right now and my board isn't even rated for it, but its still taking it like a champ, go figure. But upping it to 288gb for that extra 32gb for other stuff... I just don't see that happening without buying some insanely expensive MB to replace my current one.
That's because you're using consumer grade hardware. I picked up a PowerEdge R630 with 256gb for about £975. This machine has 24 DIMM slots and will take up to 1.5TB of RAM. DDR4. The extra 32gb was about £80 I think. I very quickly realised that putting together a PC for ramdisk plotting was going to be stupidly expensive.
I'm hoping I'll use the Poweredge for something else useful now that the price of XCH is crap and farming wins are so rare :-)
Yeah, but my computer is my pre-existing main computer. Only extra cash I had to spend was for the ram, though I did pay more than you did just for the RAM. 256gb of ram (8x32gb) was like $1200. So it sounds like you got one hell of a deal that is unlikely to be duplicated by others. I really wish this thing could run, fluently, on 256gb of ram (even including other programs). According to the normal Chia coin its only supposed to take 239gb, so... that seems weird. But perhaps the tradeoff we have to accept for this plotter.
256gb of ram (8x32gb) was like $1200
Again, you're not getting it. You're toying with consumer-grade stuff. 16gb sticks of ecc can be had for less than $40ea on ebay. A (relatively) cheap 24-slot R620 can hold 384GB. Just because your gamer system can't cut it doesn't mean no one else can. My plotting systems have 192GB and 128GB respectively, and that's only because I've filled up most of my drives and am holding on anymore, but I'd pull the trigger on more RAM in a heartbeat at these prices if it would help. Consumer grade stuff can stick with the GM plotter if it makes more sense for them, but it doesn't make sense to hamstring everyone else with large scale-up enterprise hardware systems. That's the whole point of MM :)