pcircle
pcircle copied to clipboard
Published
20 hours ago •
fwang2
fwang2
Fprof summary histogram shows invalid size value
After looking at the histogram at the end of one of our fprof runs, we noticed the final "Size" value for files >= 4TiB was ~ 25EB, when the file system is ~4.9PB.
`Fileset Histogram
Buckets Num of Files Size %(Files) %(Size)
<= 4.00 KiB 328,144,430 342.24 GiB 27.61% 0.00%
<= 8.00 KiB 44,787,106 247.42 GiB 3.77% 0.00%
<= 16.00 KiB 61,569,693 693.72 GiB 5.18% 0.00%
<= 32.00 KiB 64,078,449 1.37 TiB 5.39% 0.00%
<= 64.00 KiB 61,463,893 2.83 TiB 5.17% 0.00%
<= 128.00 KiB 167,418,095 13.07 TiB 14.09% 0.00%
<= 256.00 KiB 50,463,212 8.79 TiB 4.25% 0.00%
<= 512.00 KiB 30,603,947 10.61 TiB 2.58% 0.00%
<= 1.00 MiB 158,232,430 118.15 TiB 13.31% 0.00%
<= 2.00 MiB 99,770,567 122.61 TiB 8.40% 0.00%
<= 4.00 MiB 42,683,406 116.86 TiB 3.59% 0.00%
<= 16.00 MiB 37,221,345 315.87 TiB 3.13% 0.00%
<= 32.00 MiB 13,372,951 295.73 TiB 1.13% 0.00%
<= 64.00 MiB 11,957,960 526.20 TiB 1.01% 0.00%
<= 128.00 MiB 8,472,459 697.83 TiB 0.71% 0.00%
<= 256.00 MiB 3,533,639 626.71 TiB 0.30% 0.00%
<= 512.00 MiB 2,643,028 913.37 TiB 0.22% 0.00%
<= 1.00 GiB 1,179,467 739.37 TiB 0.10% 0.00%
<= 4.00 GiB 699,953 1199.34 TiB 0.06% 0.00%
<= 64.00 GiB 80,872 681.69 TiB 0.01% 0.00%
<= 128.00 GiB 560 47.54 TiB 0.00% 0.00%
<= 256.00 GiB 246 41.74 TiB 0.00% 0.00%
<= 512.00 GiB 68 23.36 TiB 0.00% 0.00%
<= 1.00 TiB 52 37.00 TiB 0.00% 0.00%
<= 4.00 TiB 70 106.76 TiB 0.00% 0.00%
> 4.00 TiB 20 24963302.02 TiB 0.00% 99.97%
`
Can you use --top N option to show 10 or even 20 largest files? I am suspecting one or more files are not reporting the correct size, this will help me to track down the problem Thanks.
I've started a job on the same file system tracking the top 20. I only have one node to run it on so it may take several days to finish.
On 04/26/2017 06:46 PM, Feiyi Wang wrote:
Can you use |--top N| option to show 10 or even 20 largest files? I am suspecting one or more files are not reporting the correct size, this will help me to track down the problem Thanks.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/olcf/pcircle/issues/45#issuecomment-297588400, or mute the thread https://github.com/notifications/unsubscribe-auth/AOSsQmMEmwlC3fUFqHxpCFmes7H3eG1iks5rz_OOgaJpZM4NJDmU.
Here's output from a new run with a new section at the bottom containing real sizes of the top 20 using du. `` Fprof epilogue:
Directory count: 44,184,657
Sym links count: 10,164,895
Hard linked files: 236,312
File count: 1,317,244,368
Skipped count: 734
Total file size: 24970290.02 TiB
Avg file size: 19.41 GiB
Max files within dir: 26,546,573
Tree walk time: 20h 56m
Scanning rate: 18194/s
Fprof loads: [18550170, 17214510, 16844360, 17312665, 16983137, 17079680, 16636943, 15676405, 17106085, 14521525, 13732338, 18359136, 16362823, 16688090, 18385281, 16556060, 17246359, 16693450, 18465103, 15907590, 16090472, 16140300, 14414622, 15809160, 15789829, 20261550, 18188142, 17319292, 18116553, 15166256, 17058365, 17854555, 19163461, 19849091, 18700651, 17930858, 17600331, 16557183, 15483314, 17902728, 16539349, 18727124, 17456022, 17452242, 19828551, 15327368, 17044878, 17386641, 14878072, 17228891, 17384523, 16988016, 17569467, 15239875, 14382679, 16325946, 17464731, 18378861, 17745446, 15325307, 16000979, 17271417, 17714741, 17231450, 14722610, 15146347, 18065341, 16472641, 19332010, 16531165, 16592650, 14691231, 19735541, 16889645, 17940473, 18294434, 16472103, 17319629, 17893667, 16718539]
Fileset Histogram
Buckets Num of Files Size %(Files) %(Size)
<= 4.00 KiB 368,680,518 395.84 GiB 27.99% 0.00%
<= 8.00 KiB 53,341,714 293.89 GiB 4.05% 0.00%
<= 16.00 KiB 68,499,583 771.22 GiB 5.20% 0.00%
<= 32.00 KiB 75,583,031 1.62 TiB 5.74% 0.00%
<= 64.00 KiB 72,890,867 3.33 TiB 5.53% 0.00%
<= 128.00 KiB 174,242,112 13.66 TiB 13.23% 0.00%
<= 256.00 KiB 53,165,161 9.24 TiB 4.04% 0.00%
<= 512.00 KiB 33,034,071 11.48 TiB 2.51% 0.00%
<= 1.00 MiB 168,773,762 125.65 TiB 12.81% 0.00%
<= 2.00 MiB 109,494,082 136.60 TiB 8.31% 0.00%
<= 4.00 MiB 48,081,177 129.51 TiB 3.65% 0.00%
<= 16.00 MiB 46,112,481 372.19 TiB 3.50% 0.00%
<= 32.00 MiB 14,826,120 330.75 TiB 1.13% 0.00%
<= 64.00 MiB 12,995,749 568.39 TiB 0.99% 0.00%
<= 128.00 MiB 9,064,198 747.67 TiB 0.69% 0.00%
<= 256.00 MiB 3,727,121 657.46 TiB 0.28% 0.00%
<= 512.00 MiB 2,727,897 941.35 TiB 0.21% 0.00%
<= 1.00 GiB 1,224,302 763.66 TiB 0.09% 0.00%
<= 4.00 GiB 693,763 1188.01 TiB 0.05% 0.00%
<= 64.00 GiB 85,597 720.22 TiB 0.01% 0.00%
<= 128.00 GiB 574 48.74 TiB 0.00% 0.00%
<= 256.00 GiB 266 45.09 TiB 0.00% 0.00%
<= 512.00 GiB 77 26.40 TiB 0.00% 0.00%
<= 1.00 TiB 55 38.81 TiB 0.00% 0.00%
<= 4.00 TiB 70 106.76 TiB 0.00% 0.00%
> 4.00 TiB 20 24963302.02 TiB 0.00% 99.97%
Top File Report:
1: /p/lscratche/<replaced>/IM1_11.53706_0.0_air_58.silo (4183585.56 TiB)
2: /p/lscratche/<replaced>/IM1_11.53706_0.0_air_62.silo (4175215.36 TiB)
3: /p/lscratche/<replaced>/IM1_11.53706_0.0_air_57.silo (4167055.85 TiB)
4: /p/lscratche/<replaced>/IM1_11.53706_0.0_air_59.silo (4162526.12 TiB)
5: /p/lscratche/<replaced>/IM1_11.53706_0.0_air_52.silo (4150532.10 TiB)
6: /p/lscratche/<replaced>/IM1_11.53706_0.0_air_47.silo (4123891.33 TiB)
7: /p/lscratche/<replaced>/np.out (128.00 TiB)
8: /p/lscratche/<replaced>/RVE_28S.5 (42.98 TiB)
9: /p/lscratche/<replaced>/RVE_28S.13 (42.77 TiB)
10: /p/lscratche/<replaced>/IM1_11.53706_0.0_air_54.silo (42.67 TiB)
11: /p/lscratche/<replaced>/RVE_28S.16 (42.67 TiB)
12: /p/lscratche/<replaced>/IM1_11.53706_0.0_air_63.silo (42.67 TiB)
13: /p/lscratche/<replaced>/IM1_11.53706_0.0_air_56.silo (42.67 TiB)
14: /p/lscratche/<replaced>/25TiB_file (25.00 TiB)
15: /p/lscratche/<replaced>/25TiB_file (25.00 TiB)
16: /p/lscratche/<replaced>/MS2-2015-07-22.tar (14.31 TiB)
17: /p/lscratche/<replaced>/OTIS.tar (14.31 TiB)
18: /p/lscratche/<replaced>/3D-2014-09-15.tar (13.52 TiB)
19: /p/lscratche/<replaced>/dpf20160318 (12.11 TiB)
20: /p/lscratche/<replaced>/SPL-2014-09-15.tar (7.04 TiB)
Actual:
1. /p/lscratche/<replaced>/IM1_11.53706_0.0_air_58.silo (16M)
2. /p/lscratche/<replaced>/IM1_11.53706_0.0_air_62.silo (7.1M)
3. /p/lscratche/<replaced>/IM1_11.53706_0.0_air_57.silo (16M)
4. /p/lscratche/<replaced>/IM1_11.53706_0.0_air_59.silo (16M)
5. /p/lscratche/<replaced>/IM1_11.53706_0.0_air_52.silo (27M)
6. /p/lscratche/<replaced>/IM1_11.53706_0.0_air_47.silo (36M)
7. /p/lscratche/<replaced>/np.out (1.2M)
8. /p/lscratche/<replaced>/RVE_28S.5 (425K)
9. /p/lscratche/<replaced>/RVE_28S.13 (387K)
10. /p/lscratche/<replaced>/IM1_11.53706_0.0_air_54.silo (27M)
11. /p/lscratche/<replaced>/RVE_28S.16 (431K)
12. /p/lscratche/<replaced>/IM1_11.53706_0.0_air_63.silo (7.1M)
13. /p/lscratche/<replaced>/IM1_11.53706_0.0_air_56.silo (16M)
14. /p/lscratche/<replace>/25TiB_file (13T)
15. /p/lscratche/<replace>/25TiB_file (13T)
16. /p/lscratche/<replaced/MS2-2015-07-22.tar (8.7T)
17. /p/lscratche/<replaced>/OTIS.tar (8.7T)
18. /p/lscratche/<replaced>/3D-2014-09-15.tar (2.6T)
19. /p/lscratche/<replaced>/dpf20160318 (8.0T)
20. /p/lscratche/<replaced>/SPL-2014-09-15.tar (3.0T)
The latest code appears to work well: `Running Parameters:
fprof version: 0.16+51.gc8daab4
Full rev id: c8daab41a327c9d45242c21fa6adaf331532bb14
Num of hosts: 12
Num of processes: 84
Syslog report: no
Stripe analysis: no
Root path: ['/p/lscratche']
...
Fprof epilogue:
Directory count: 45,265,268
Sym links count: 10,430,723
Hard linked files: 309,219
File count: 1,342,586,738
Sparse files: 824,561,829
Skipped count: 2,977
Total file size: 3621.73 TiB
Avg file size: 2.83 MiB
Max files within dir: 26,546,573
Tree walk time: 16h 23m
Scanning rate: 23688/s
Fileset Histogram
Buckets Num of Files Size %(Files) %(Size)
<= 4.00 KiB 405,101,078 478.02 GiB 30.17% 0.01%
<= 8.00 KiB 125,265,283 711.78 GiB 9.33% 0.02%
<= 16.00 KiB 64,625,194 699.03 GiB 4.81% 0.02%
<= 32.00 KiB 60,576,954 1.33 TiB 4.51% 0.04%
<= 64.00 KiB 95,345,635 4.36 TiB 7.10% 0.12%
<= 128.00 KiB 132,993,524 10.06 TiB 9.91% 0.28%
<= 256.00 KiB 47,006,551 8.13 TiB 3.50% 0.22%
<= 512.00 KiB 113,780,368 42.45 TiB 8.47% 1.17%
<= 1.00 MiB 134,780,790 87.91 TiB 10.04% 2.43%
<= 2.00 MiB 72,754,593 95.37 TiB 5.42% 2.63%
<= 4.00 MiB 24,239,190 64.34 TiB 1.81% 1.78%
<= 16.00 MiB 40,482,662 316.71 TiB 3.02% 8.74%
<= 32.00 MiB 9,258,891 199.41 TiB 0.69% 5.51%
<= 64.00 MiB 9,478,784 386.65 TiB 0.71% 10.68%
<= 128.00 MiB 3,659,133 304.91 TiB 0.27% 8.42%
<= 256.00 MiB 1,654,034 289.96 TiB 0.12% 8.01%
<= 512.00 MiB 803,875 270.47 TiB 0.06% 7.47%
<= 1.00 GiB 381,657 259.78 TiB 0.03% 7.17%
<= 4.00 GiB 340,887 619.90 TiB 0.03% 17.12%
<= 64.00 GiB 57,012 449.67 TiB 0.00% 12.42%
<= 128.00 GiB 381 33.33 TiB 0.00% 0.92%
<= 256.00 GiB 105 18.75 TiB 0.00% 0.52%
<= 512.00 GiB 66 23.15 TiB 0.00% 0.64%
<= 1.00 TiB 65 46.65 TiB 0.00% 1.29%
<= 4.00 TiB 21 35.73 TiB 0.00% 0.99%
> 4.00 TiB 5 50.87 TiB 0.00% 1.40%
Top File Report:
1: /p/lscratche/.../bigtest/25TiB_file (12.77 TiB)
2: /p/lscratche/.../25TiB_file (12.77 TiB)
3: /p/lscratche/.../MS2-2015-07-22.tar (8.68 TiB)
4: /p/lscratche/.../OTIS.tar (8.68 TiB)
5: /p/lscratche/.../dpf20160318 (7.96 TiB)
6: /p/lscratche/.../SPL-2014-09-15.tar (2.98 TiB)
7: /p/lscratche/.../3D-2014-09-15.tar (2.56 TiB)
8: /p/lscratche/.../Artie-01-23-15.tar (2.47 TiB)
9: /p/lscratche/.../pmovie1.p4 (2.03 TiB)
10: /p/lscratche/.../pmovie2.p4 (2.01 TiB)
11: /p/lscratche/.../pmovie2.p4 (2.00 TiB)
12: /p/lscratche/.../pmovie1.p4 (1.94 TiB)
13: /p/lscratche/.../SSG_Good solution.simh (1.78 TiB)
14: /p/lscratche/.../pmovie1.p4 (1.71 TiB)
15: /p/lscratche/.../pmovie1.p4 (1.68 TiB)
16: /p/lscratche/.../pmovie1.p4 (1.53 TiB)
17: /p/lscratche/.../pmovie1.p4 (1.52 TiB)
18: /p/lscratche/.../powersharingpaperresults.tar.gz (1.44 TiB)
19: /p/lscratche/.../pmovie5.p4 (1.39 TiB)
20: /p/lscratche/.../pmovie1.p4 (1.32 TiB)
`