thanos
thanos copied to clipboard
Published
20 hours ago •
thanos-io
thanos-io
receive: does not cleanup old data on disk, but tries to load it on boot, causing OOM down the line
trafficstars
Thanos, Prometheus and Golang version used: quay.io/thanos/thanos:v0.21.1
Object Storage Provider: GCS
What happened: Running with:
--tsdb.retention=1h
--tsdb.path=/thanos-data
--tsdb.allow-overlapping-blocks
--shipper.allow-out-of-order-uploads
Several days old data is left on disk, with no attempts of shipping it to storage. Not necessarily days worth of data, tho I have not really checked that.
Some WAL segments are left forever on disk, eventually causing OOM on boot, as memory needs becomes ever growing. Giving it enough memory to load all segments appears to make no difference, as even when loaded no attempts to ship them are made, even after several days.
Manual removal of the affected segments on disk appears to resolve it, as nothing is logged about data inconsistency after doing so.
What you expected to happen:
Old data is removed from disk and not attempted to be loaded forever.
How to reproduce it (as minimally and precisely as possible): Run a receiver and ingest data to it, observe data being left on disk. It may need to run for a few days, and restart a few times, maybe be shutdown for sometime in-between too, not entirely sure.
Full logs to relevant components:
Looking in the WAL directory (captured a few weeks ago when first encountering it)
/ # ls -lah /thanos-data/default-tenant/wal/
total 127M
drwxr-xr-x 104 root root 12.0K Jun 18 09:05 .
drwxr-xr-x 6 root root 4.0K Jun 18 09:05 ..
-rw-r--r-- 1 root root 3.2M Jun 15 09:12 00000739
-rw-r--r-- 1 root root 96.0K Jun 15 11:16 00000740
-rw-r--r-- 1 root root 10.5K Jun 15 11:16 00000741
-rw-r--r-- 1 root root 0 Jun 15 13:03 00000742
-rw-r--r-- 1 root root 0 Jun 15 13:11 00000743
-rw-r--r-- 1 root root 64.0K Jun 15 15:15 00000744
-rw-r--r-- 1 root root 15.3M Jun 15 15:20 00000745
-rw-r--r-- 1 root root 4.6M Jun 15 15:20 00000746
-rw-r--r-- 1 root root 29.4M Jun 15 16:12 00000747
-rw-r--r-- 1 root root 7.6M Jun 15 16:13 00000748
-rw-r--r-- 1 root root 0 Jun 15 17:01 00000749
-rw-r--r-- 1 root root 15.3M Jun 15 17:26 00000750
-rw-r--r-- 1 root root 0 Jun 15 17:26 00000751
-rw-r--r-- 1 root root 45.3M Jun 15 19:20 00000752
-rw-r--r-- 1 root root 5.6M Jun 15 19:20 00000753
-rw-r--r-- 1 root root 0 Jun 15 21:05 00000754
-rw-r--r-- 1 root root 0 Jun 15 21:07 00000755
-rw-r--r-- 1 root root 0 Jun 15 23:06 00000756
-rw-r--r-- 1 root root 0 Jun 16 01:05 00000757
-rw-r--r-- 1 root root 0 Jun 16 01:07 00000758
-rw-r--r-- 1 root root 0 Jun 16 03:06 00000759
-rw-r--r-- 1 root root 0 Jun 16 03:07 00000760
-rw-r--r-- 1 root root 0 Jun 16 05:05 00000761
-rw-r--r-- 1 root root 0 Jun 16 06:49 00000762
-rw-r--r-- 1 root root 0 Jun 16 06:50 00000763
-rw-r--r-- 1 root root 0 Jun 16 06:52 00000764
-rw-r--r-- 1 root root 0 Jun 16 07:03 00000765
-rw-r--r-- 1 root root 0 Jun 16 07:05 00000766
-rw-r--r-- 1 root root 0 Jun 16 07:07 00000767
-rw-r--r-- 1 root root 0 Jun 16 09:03 00000768
-rw-r--r-- 1 root root 0 Jun 16 09:04 00000769
-rw-r--r-- 1 root root 0 Jun 16 09:06 00000770
-rw-r--r-- 1 root root 0 Jun 16 09:08 00000771
-rw-r--r-- 1 root root 0 Jun 16 09:10 00000772
-rw-r--r-- 1 root root 0 Jun 16 09:13 00000773
-rw-r--r-- 1 root root 0 Jun 16 09:17 00000774
-rw-r--r-- 1 root root 0 Jun 16 09:24 00000775
-rw-r--r-- 1 root root 0 Jun 16 10:23 00000776
-rw-r--r-- 1 root root 0 Jun 16 10:25 00000777
-rw-r--r-- 1 root root 0 Jun 16 10:27 00000778
-rw-r--r-- 1 root root 0 Jun 16 10:28 00000779
-rw-r--r-- 1 root root 0 Jun 16 10:31 00000780
-rw-r--r-- 1 root root 0 Jun 16 10:34 00000781
-rw-r--r-- 1 root root 0 Jun 16 10:38 00000782
-rw-r--r-- 1 root root 0 Jun 16 10:44 00000783
-rw-r--r-- 1 root root 0 Jun 16 10:51 00000784
-rw-r--r-- 1 root root 0 Jun 16 10:57 00000785
-rw-r--r-- 1 root root 0 Jun 16 11:04 00000786
-rw-r--r-- 1 root root 0 Jun 16 11:11 00000787
-rw-r--r-- 1 root root 0 Jun 16 11:17 00000788
-rw-r--r-- 1 root root 0 Jun 16 11:24 00000789
-rw-r--r-- 1 root root 0 Jun 16 11:30 00000790
-rw-r--r-- 1 root root 0 Jun 16 11:37 00000791
-rw-r--r-- 1 root root 0 Jun 16 11:54 00000792
-rw-r--r-- 1 root root 0 Jun 16 11:55 00000793
-rw-r--r-- 1 root root 0 Jun 16 11:57 00000794
-rw-r--r-- 1 root root 0 Jun 16 11:59 00000795
-rw-r--r-- 1 root root 0 Jun 16 12:01 00000796
-rw-r--r-- 1 root root 0 Jun 16 12:03 00000797
-rw-r--r-- 1 root root 0 Jun 16 12:07 00000798
-rw-r--r-- 1 root root 0 Jun 16 12:14 00000799
-rw-r--r-- 1 root root 0 Jun 16 12:21 00000800
-rw-r--r-- 1 root root 0 Jun 16 12:27 00000801
-rw-r--r-- 1 root root 0 Jun 16 12:33 00000802
-rw-r--r-- 1 root root 0 Jun 16 12:34 00000803
-rw-r--r-- 1 root root 0 Jun 16 12:36 00000804
-rw-r--r-- 1 root root 0 Jun 16 12:38 00000805
-rw-r--r-- 1 root root 0 Jun 16 12:40 00000806
-rw-r--r-- 1 root root 0 Jun 16 12:43 00000807
-rw-r--r-- 1 root root 0 Jun 16 12:48 00000808
-rw-r--r-- 1 root root 0 Jun 16 12:54 00000809
-rw-r--r-- 1 root root 0 Jun 16 13:01 00000810
-rw-r--r-- 1 root root 0 Jun 16 13:07 00000811
-rw-r--r-- 1 root root 0 Jun 16 13:14 00000812
-rw-r--r-- 1 root root 0 Jun 16 13:20 00000813
-rw-r--r-- 1 root root 0 Jun 16 13:27 00000814
-rw-r--r-- 1 root root 0 Jun 16 14:38 00000815
-rw-r--r-- 1 root root 0 Jun 16 14:39 00000816
-rw-r--r-- 1 root root 0 Jun 16 14:41 00000817
-rw-r--r-- 1 root root 0 Jun 16 14:43 00000818
-rw-r--r-- 1 root root 0 Jun 16 14:45 00000819
-rw-r--r-- 1 root root 0 Jun 16 14:48 00000820
-rw-r--r-- 1 root root 0 Jun 16 14:52 00000821
-rw-r--r-- 1 root root 0 Jun 16 14:59 00000822
-rw-r--r-- 1 root root 0 Jun 16 15:05 00000823
-rw-r--r-- 1 root root 0 Jun 16 15:12 00000824
-rw-r--r-- 1 root root 0 Jun 16 15:18 00000825
-rw-r--r-- 1 root root 0 Jun 16 16:40 00000826
-rw-r--r-- 1 root root 0 Jun 16 16:42 00000827
-rw-r--r-- 1 root root 0 Jun 16 16:43 00000828
-rw-r--r-- 1 root root 0 Jun 16 16:45 00000829
-rw-r--r-- 1 root root 0 Jun 16 16:47 00000830
-rw-r--r-- 1 root root 0 Jun 16 16:50 00000831
-rw-r--r-- 1 root root 0 Jun 16 16:54 00000832
-rw-r--r-- 1 root root 0 Jun 16 17:01 00000833
-rw-r--r-- 1 root root 0 Jun 16 17:07 00000834
-rw-r--r-- 1 root root 0 Jun 16 19:04 00000835
-rw-r--r-- 1 root root 0 Jun 16 19:05 00000836
-rw-r--r-- 1 root root 0 Jun 16 19:07 00000837
-rw-r--r-- 1 root root 0 Jun 16 19:09 00000838
-rw-r--r-- 1 root root 0 Jun 16 19:11 00000839
-rw-r--r-- 1 root root 0 Jun 16 19:14 00000840
-rw-r--r-- 1 root root 0 Jun 16 19:18 00000841
-rw-r--r-- 1 root root 0 Jun 16 19:24 00000842
-rw-r--r-- 1 root root 0 Jun 16 21:02 00000843
-rw-r--r-- 1 root root 0 Jun 16 21:04 00000844
-rw-r--r-- 1 root root 0 Jun 16 21:06 00000845
-rw-r--r-- 1 root root 0 Jun 16 23:03 00000846
-rw-r--r-- 1 root root 0 Jun 16 23:05 00000847
-rw-r--r-- 1 root root 0 Jun 16 23:07 00000848
-rw-r--r-- 1 root root 0 Jun 16 23:09 00000849
-rw-r--r-- 1 root root 0 Jun 17 01:06 00000850
-rw-r--r-- 1 root root 0 Jun 17 01:07 00000851
-rw-r--r-- 1 root root 0 Jun 17 03:05 00000852
-rw-r--r-- 1 root root 0 Jun 17 03:07 00000853
-rw-r--r-- 1 root root 0 Jun 17 03:08 00000854
-rw-r--r-- 1 root root 0 Jun 17 05:06 00000855
-rw-r--r-- 1 root root 0 Jun 17 05:07 00000856
-rw-r--r-- 1 root root 0 Jun 17 07:01 00000857
-rw-r--r-- 1 root root 0 Jun 17 07:03 00000858
-rw-r--r-- 1 root root 0 Jun 17 07:05 00000859
-rw-r--r-- 1 root root 0 Jun 17 07:08 00000860
-rw-r--r-- 1 root root 0 Jun 17 07:10 00000861
-rw-r--r-- 1 root root 0 Jun 17 07:26 00000862
-rw-r--r-- 1 root root 0 Jun 17 07:28 00000863
-rw-r--r-- 1 root root 0 Jun 17 07:29 00000864
-rw-r--r-- 1 root root 0 Jun 17 07:31 00000865
-rw-r--r-- 1 root root 0 Jun 17 09:01 00000866
-rw-r--r-- 1 root root 0 Jun 17 09:03 00000867
-rw-r--r-- 1 root root 0 Jun 17 09:04 00000868
-rw-r--r-- 1 root root 0 Jun 17 09:06 00000869
-rw-r--r-- 1 root root 0 Jun 17 09:08 00000870
-rw-r--r-- 1 root root 0 Jun 17 09:11 00000871
-rw-r--r-- 1 root root 0 Jun 17 09:15 00000872
-rw-r--r-- 1 root root 0 Jun 17 09:21 00000873
-rw-r--r-- 1 root root 0 Jun 17 09:28 00000874
-rw-r--r-- 1 root root 0 Jun 17 09:34 00000875
-rw-r--r-- 1 root root 0 Jun 17 09:41 00000876
-rw-r--r-- 1 root root 0 Jun 17 09:47 00000877
-rw-r--r-- 1 root root 0 Jun 17 09:53 00000878
-rw-r--r-- 1 root root 0 Jun 17 10:00 00000879
-rw-r--r-- 1 root root 0 Jun 17 10:06 00000880
-rw-r--r-- 1 root root 0 Jun 17 10:13 00000881
-rw-r--r-- 1 root root 0 Jun 17 10:19 00000882
-rw-r--r-- 1 root root 0 Jun 17 10:26 00000883
-rw-r--r-- 1 root root 0 Jun 17 10:32 00000884
-rw-r--r-- 1 root root 0 Jun 17 10:39 00000885
-rw-r--r-- 1 root root 0 Jun 17 10:45 00000886
-rw-r--r-- 1 root root 0 Jun 17 10:52 00000887
-rw-r--r-- 1 root root 0 Jun 17 10:59 00000888
-rw-r--r-- 1 root root 0 Jun 17 11:05 00000889
-rw-r--r-- 1 root root 0 Jun 17 11:12 00000890
-rw-r--r-- 1 root root 0 Jun 17 11:18 00000891
-rw-r--r-- 1 root root 0 Jun 17 11:25 00000892
-rw-r--r-- 1 root root 0 Jun 17 11:31 00000893
-rw-r--r-- 1 root root 0 Jun 17 11:38 00000894
-rw-r--r-- 1 root root 0 Jun 17 11:44 00000895
-rw-r--r-- 1 root root 0 Jun 17 13:01 00000896
-rw-r--r-- 1 root root 0 Jun 17 13:03 00000897
-rw-r--r-- 1 root root 0 Jun 17 13:05 00000898
-rw-r--r-- 1 root root 0 Jun 17 13:07 00000899
-rw-r--r-- 1 root root 0 Jun 17 13:09 00000900
-rw-r--r-- 1 root root 0 Jun 17 13:12 00000901
-rw-r--r-- 1 root root 0 Jun 17 15:01 00000902
-rw-r--r-- 1 root root 0 Jun 17 15:03 00000903
-rw-r--r-- 1 root root 0 Jun 17 15:04 00000904
-rw-r--r-- 1 root root 0 Jun 17 15:06 00000905
-rw-r--r-- 1 root root 0 Jun 17 15:09 00000906
-rw-r--r-- 1 root root 0 Jun 17 17:01 00000907
-rw-r--r-- 1 root root 0 Jun 17 17:03 00000908
-rw-r--r-- 1 root root 0 Jun 17 17:04 00000909
-rw-r--r-- 1 root root 0 Jun 17 17:06 00000910
-rw-r--r-- 1 root root 0 Jun 17 17:08 00000911
-rw-r--r-- 1 root root 0 Jun 17 19:04 00000912
-rw-r--r-- 1 root root 0 Jun 17 19:05 00000913
-rw-r--r-- 1 root root 0 Jun 17 19:07 00000914
-rw-r--r-- 1 root root 0 Jun 17 19:09 00000915
-rw-r--r-- 1 root root 0 Jun 17 19:11 00000916
-rw-r--r-- 1 root root 0 Jun 17 19:14 00000917
-rw-r--r-- 1 root root 0 Jun 17 21:04 00000918
-rw-r--r-- 1 root root 0 Jun 17 21:06 00000919
-rw-r--r-- 1 root root 0 Jun 17 21:07 00000920
-rw-r--r-- 1 root root 0 Jun 17 21:09 00000921
-rw-r--r-- 1 root root 0 Jun 17 21:12 00000922
-rw-r--r-- 1 root root 0 Jun 17 23:04 00000923
-rw-r--r-- 1 root root 0 Jun 17 23:05 00000924
-rw-r--r-- 1 root root 0 Jun 17 23:07 00000925
-rw-r--r-- 1 root root 0 Jun 17 23:09 00000926
-rw-r--r-- 1 root root 0 Jun 17 23:12 00000927
-rw-r--r-- 1 root root 0 Jun 17 23:15 00000928
-rw-r--r-- 1 root root 0 Jun 17 23:19 00000929
-rw-r--r-- 1 root root 0 Jun 18 01:04 00000930
-rw-r--r-- 1 root root 0 Jun 18 01:06 00000931
-rw-r--r-- 1 root root 0 Jun 18 01:08 00000932
-rw-r--r-- 1 root root 0 Jun 18 01:09 00000933
-rw-r--r-- 1 root root 0 Jun 18 01:12 00000934
-rw-r--r-- 1 root root 0 Jun 18 03:04 00000935
-rw-r--r-- 1 root root 0 Jun 18 03:06 00000936
-rw-r--r-- 1 root root 0 Jun 18 03:07 00000937
-rw-r--r-- 1 root root 0 Jun 18 03:09 00000938
-rw-r--r-- 1 root root 0 Jun 18 03:12 00000939
-rw-r--r-- 1 root root 0 Jun 18 03:15 00000940
-rw-r--r-- 1 root root 0 Jun 18 05:04 00000941
-rw-r--r-- 1 root root 0 Jun 18 05:05 00000942
-rw-r--r-- 1 root root 0 Jun 18 05:07 00000943
-rw-r--r-- 1 root root 0 Jun 18 05:09 00000944
-rw-r--r-- 1 root root 0 Jun 18 05:11 00000945
-rw-r--r-- 1 root root 0 Jun 18 05:14 00000946
-rw-r--r-- 1 root root 0 Jun 18 07:05 00000947
-rw-r--r-- 1 root root 0 Jun 18 07:06 00000948
-rw-r--r-- 1 root root 0 Jun 18 07:08 00000949
-rw-r--r-- 1 root root 0 Jun 18 07:09 00000950
-rw-r--r-- 1 root root 0 Jun 18 07:11 00000951
-rw-r--r-- 1 root root 0 Jun 18 07:14 00000952
-rw-r--r-- 1 root root 0 Jun 18 08:54 00000953
-rw-r--r-- 1 root root 0 Jun 18 08:56 00000954
-rw-r--r-- 1 root root 0 Jun 18 08:58 00000955
-rw-r--r-- 1 root root 0 Jun 18 09:00 00000956
-rw-r--r-- 1 root root 0 Jun 18 09:02 00000957
-rw-r--r-- 1 root root 0 Jun 18 09:05 00000958
drwxr-xr-x 2 root root 4.0K May 13 13:09 checkpoint.00000034.tmp
drwxr-xr-x 2 root root 4.0K May 18 03:16 checkpoint.00000168.tmp
drwxr-xr-x 2 root root 4.0K May 19 19:19 checkpoint.00000237.tmp
drwxr-xr-x 2 root root 4.0K May 27 09:07 checkpoint.00000465.tmp
drwxr-xr-x 2 root root 4.0K May 28 13:22 checkpoint.00000481.tmp
drwxr-xr-x 2 root root 4.0K May 31 09:11 checkpoint.00000488.tmp
drwxr-xr-x 2 root root 4.0K May 31 13:12 checkpoint.00000492.tmp
drwxr-xr-x 2 root root 4.0K May 31 15:16 checkpoint.00000493.tmp
drwxr-xr-x 2 root root 4.0K Jun 1 11:10 checkpoint.00000516.tmp
drwxr-xr-x 2 root root 4.0K Jun 1 13:07 checkpoint.00000517.tmp
drwxr-xr-x 2 root root 4.0K Jun 1 19:07 checkpoint.00000525.tmp
drwxr-xr-x 2 root root 4.0K Jun 1 21:12 checkpoint.00000528.tmp
drwxr-xr-x 2 root root 4.0K Jun 2 01:07 checkpoint.00000532.tmp
drwxr-xr-x 2 root root 4.0K Jun 2 03:06 checkpoint.00000533.tmp
drwxr-xr-x 2 root root 4.0K Jun 2 09:07 checkpoint.00000540.tmp
drwxr-xr-x 2 root root 4.0K Jun 2 15:15 checkpoint.00000547.tmp
drwxr-xr-x 2 root root 4.0K Jun 2 17:17 checkpoint.00000548.tmp
drwxr-xr-x 2 root root 4.0K Jun 2 19:08 checkpoint.00000549.tmp
drwxr-xr-x 2 root root 4.0K Jun 2 21:07 checkpoint.00000551.tmp
drwxr-xr-x 2 root root 4.0K Jun 2 23:07 checkpoint.00000552.tmp
drwxr-xr-x 2 root root 4.0K Jun 3 03:06 checkpoint.00000554.tmp
drwxr-xr-x 2 root root 4.0K Jun 3 05:07 checkpoint.00000555.tmp
drwxr-xr-x 2 root root 4.0K Jun 3 07:07 checkpoint.00000557.tmp
drwxr-xr-x 2 root root 4.0K Jun 3 09:07 checkpoint.00000558.tmp
drwxr-xr-x 2 root root 4.0K Jun 3 17:15 checkpoint.00000573.tmp
drwxr-xr-x 2 root root 4.0K Jun 3 19:20 checkpoint.00000574.tmp
drwxr-xr-x 2 root root 4.0K Jun 3 21:07 checkpoint.00000576.tmp
drwxr-xr-x 2 root root 4.0K Jun 3 23:07 checkpoint.00000577.tmp
drwxr-xr-x 2 root root 4.0K Jun 4 01:07 checkpoint.00000578.tmp
drwxr-xr-x 2 root root 4.0K Jun 4 03:06 checkpoint.00000580.tmp
drwxr-xr-x 2 root root 4.0K Jun 4 05:06 checkpoint.00000581.tmp
drwxr-xr-x 2 root root 4.0K Jun 4 07:07 checkpoint.00000582.tmp
drwxr-xr-x 2 root root 4.0K Jun 4 09:17 checkpoint.00000584.tmp
drwxr-xr-x 2 root root 4.0K Jun 4 11:18 checkpoint.00000590.tmp
drwxr-xr-x 2 root root 4.0K Jun 4 13:18 checkpoint.00000592.tmp
drwxr-xr-x 2 root root 4.0K Jun 4 15:13 checkpoint.00000593.tmp
drwxr-xr-x 2 root root 4.0K Jun 4 17:07 checkpoint.00000594.tmp
drwxr-xr-x 2 root root 4.0K Jun 4 21:07 checkpoint.00000596.tmp
drwxr-xr-x 2 root root 4.0K Jun 4 23:07 checkpoint.00000597.tmp
drwxr-xr-x 2 root root 4.0K Jun 5 01:06 checkpoint.00000598.tmp
drwxr-xr-x 2 root root 4.0K Jun 5 03:07 checkpoint.00000600.tmp
drwxr-xr-x 2 root root 4.0K Jun 5 05:06 checkpoint.00000601.tmp
drwxr-xr-x 2 root root 4.0K Jun 5 07:06 checkpoint.00000602.tmp
drwxr-xr-x 2 root root 4.0K Jun 5 09:06 checkpoint.00000604.tmp
drwxr-xr-x 2 root root 4.0K Jun 5 11:06 checkpoint.00000605.tmp
drwxr-xr-x 2 root root 4.0K Jun 5 15:06 checkpoint.00000607.tmp
drwxr-xr-x 2 root root 4.0K Jun 6 11:06 checkpoint.00000618.tmp
drwxr-xr-x 2 root root 4.0K Jun 6 13:06 checkpoint.00000619.tmp
drwxr-xr-x 2 root root 4.0K Jun 6 17:06 checkpoint.00000621.tmp
drwxr-xr-x 2 root root 4.0K Jun 6 21:06 checkpoint.00000622.tmp
drwxr-xr-x 2 root root 4.0K Jun 7 03:06 checkpoint.00000624.tmp
drwxr-xr-x 2 root root 4.0K Jun 7 05:06 checkpoint.00000625.tmp
drwxr-xr-x 2 root root 4.0K Jun 7 07:06 checkpoint.00000627.tmp
drwxr-xr-x 2 root root 4.0K Jun 7 09:13 checkpoint.00000639.tmp
drwxr-xr-x 2 root root 4.0K Jun 7 17:08 checkpoint.00000641.tmp
drwxr-xr-x 2 root root 4.0K Jun 8 11:19 checkpoint.00000650.tmp
drwxr-xr-x 2 root root 4.0K Jun 8 13:18 checkpoint.00000651.tmp
drwxr-xr-x 2 root root 4.0K Jun 8 15:17 checkpoint.00000653.tmp
drwxr-xr-x 2 root root 4.0K Jun 8 17:21 checkpoint.00000654.tmp
drwxr-xr-x 2 root root 4.0K Jun 8 19:21 checkpoint.00000655.tmp
drwxr-xr-x 2 root root 4.0K Jun 8 21:19 checkpoint.00000657.tmp
drwxr-xr-x 2 root root 4.0K Jun 8 23:16 checkpoint.00000658.tmp
drwxr-xr-x 2 root root 4.0K Jun 9 03:13 checkpoint.00000659.tmp
drwxr-xr-x 2 root root 4.0K Jun 9 05:13 checkpoint.00000661.tmp
drwxr-xr-x 2 root root 4.0K Jun 9 07:14 checkpoint.00000668.tmp
drwxr-xr-x 2 root root 4.0K Jun 9 09:24 checkpoint.00000669.tmp
drwxr-xr-x 2 root root 4.0K Jun 9 11:19 checkpoint.00000671.tmp
drwxr-xr-x 2 root root 4.0K Jun 9 13:26 checkpoint.00000672.tmp
drwxr-xr-x 2 root root 4.0K Jun 9 15:20 checkpoint.00000673.tmp
drwxr-xr-x 2 root root 4.0K Jun 9 15:29 checkpoint.00000675.tmp
drwxr-xr-x 2 root root 4.0K Jun 9 17:24 checkpoint.00000676.tmp
drwxr-xr-x 2 root root 4.0K Jun 9 19:25 checkpoint.00000677.tmp
drwxr-xr-x 2 root root 4.0K Jun 9 21:22 checkpoint.00000679.tmp
drwxr-xr-x 2 root root 4.0K Jun 9 23:21 checkpoint.00000680.tmp
drwxr-xr-x 2 root root 4.0K Jun 10 01:21 checkpoint.00000681.tmp
drwxr-xr-x 2 root root 4.0K Jun 10 03:26 checkpoint.00000690.tmp
drwxr-xr-x 2 root root 4.0K Jun 10 05:26 checkpoint.00000692.tmp
drwxr-xr-x 2 root root 4.0K Jun 10 07:35 checkpoint.00000693.tmp
drwxr-xr-x 2 root root 4.0K Jun 10 09:25 checkpoint.00000694.tmp
drwxr-xr-x 2 root root 4.0K Jun 10 11:27 checkpoint.00000696.tmp
drwxr-xr-x 2 root root 4.0K Jun 10 12:38 checkpoint.00000697.tmp
drwxr-xr-x 2 root root 4.0K Jun 10 15:20 checkpoint.00000700.tmp
drwxr-xr-x 2 root root 4.0K Jun 14 11:07 checkpoint.00000701.tmp
drwxr-xr-x 2 root root 4.0K Jun 14 13:15 checkpoint.00000709.tmp
drwxr-xr-x 2 root root 4.0K Jun 14 14:08 checkpoint.00000711.tmp
drwxr-xr-x 2 root root 4.0K Jun 14 15:12 checkpoint.00000712.tmp
drwxr-xr-x 2 root root 4.0K Jun 14 17:17 checkpoint.00000713.tmp
drwxr-xr-x 2 root root 4.0K Jun 14 19:08 checkpoint.00000715.tmp
drwxr-xr-x 2 root root 4.0K Jun 14 21:12 checkpoint.00000721.tmp
drwxr-xr-x 2 root root 4.0K Jun 14 23:07 checkpoint.00000723.tmp
drwxr-xr-x 2 root root 4.0K Jun 15 01:08 checkpoint.00000724.tmp
drwxr-xr-x 2 root root 4.0K Jun 15 03:07 checkpoint.00000725.tmp
drwxr-xr-x 2 root root 4.0K Jun 15 05:07 checkpoint.00000727.tmp
drwxr-xr-x 2 root root 4.0K Jun 15 07:07 checkpoint.00000728.tmp
drwxr-xr-x 2 root root 4.0K Jun 15 09:12 checkpoint.00000734.tmp
drwxr-xr-x 2 root root 4.0K Jun 15 11:16 checkpoint.00000736.tmp
drwxr-xr-x 2 root root 4.0K Jun 15 13:12 checkpoint.00000737.tmp
drwxr-xr-x 2 root root 4.0K Jun 15 15:15 checkpoint.00000738
drwxr-xr-x 2 root root 4.0K Jun 15 15:20 checkpoint.00000742.tmp
drwxr-xr-x 2 root root 4.0K Jun 15 16:12 checkpoint.00000743.tmp
drwxr-xr-x 2 root root 4.0K Jun 15 17:26 checkpoint.00000745.tmp
drwxr-xr-x 2 root root 4.0K Jun 15 19:20 checkpoint.00000747.tmp
Thanos Receive 0.22: We experienced the same behavior once so far. Had set --tsdb.retention=6h which was usually good for 12Gi worth of TSDB data. Then it started to "forget" deleting old WAL segments which led to 187Gi in the wal sub-folder of the thanos receive data folder.
Found this related Prometheus issue, which I believe to be the root cause: https://github.com/prometheus/prometheus/issues/7955
If I'm correct according to: https://github.com/thanos-io/thanos/blob/v0.22.0/.bingo/prometheus.mod
... Thanos 0.22 is based on: https://pkg.go.dev/github.com/prometheus/[email protected]+incompatible
This does not include the fix mentioned here: https://github.com/prometheus/prometheus/issues/7955#issuecomment-696022050
That would require at least Prometheus v2.21.
@bwplotka Are there plans to pull Thanos' Prometheus dependency at least up to this version? Or can you suggest how to mitigate this with the now current 0.22 release?
@bwplotka
We had the same issue today again. Receiver did not flush expired WAL files from disk until the disk was full. It had retention set to 6h yet kept WALs of the last 5 days around.
The way I still see it this issue might be fixed by the information I left in my previous comment. I be happy to read your thoughts on that.
Thanks!
Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.
![stale[bot] avatar](https://avatars.githubusercontent.com/in/1724?v=4 "Published by a pub.dev verified publisher"){kind=small}
As far as I can see, this issue is still present and waiting for feedback from the maintainers.
Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.
Closing for now as promised, let us know if you need this to be reopened! 🤗
@bwplotka @GiedriusS - can we get this reopened? This issue just bit us and caused our ruler to balloon from ~2GB of memory to ~60GB of memory and cause all kinds of havoc while we troubleshooted the root cause.
Sure - can you help figuring out what happens here? What version are you running? I think WAL should be managed by the embedded Prometheus libraries
Sure - can you help figuring out what happens here? What version are you running? I think WAL should be managed by the embedded Prometheus libraries
I'll ask a team-member to pop in here and fill in some details - but we're on v0.37.2.
Could you paste the goroutine dump when this happens? You can do it via /debug/pprof
This is a follow-up to @diranged comment.
We are running Thanos v0.37.2 with the following configuration:
Args:
rule
--enable-auto-gomemlimit
--web.external-prefix=/ruler
--web.route-prefix=/ruler
--log.level=info
--data-dir=/thanos/data
--eval-interval=1m
--label=cluster="test"
--label=app_env="prod"
--label=app_group="test"
--query=http://prometheus:9090
--query.http-method=GET
--rule-file=/etc/thanos/rules/*/*.yaml
We have two remote write configurations:
- Local AMP
- Centralized AMP
Issue Details
Typically, the WAL size in similar environments is around 300-400MB, but in this environment, it had grown to ~5GB. Upon investigation, we found several old .tmp files and WAL files from March 14 that had not been cleaned up.
/thanos/data/wal $ du -sh
5.7G .
drwxrwsr-x 2 thanos thanos 4.0K Dec 22 02:19 checkpoint.00001456.tmp
drwxrwsr-x 2 thanos thanos 4.0K Dec 23 12:35 checkpoint.00001778.tmp
drwxrwsr-x 2 thanos thanos 4.0K Dec 24 01:45 checkpoint.00002171.tmp
...
drwxrwsr-x 2 thanos thanos 4.0K Mar 17 17:39 checkpoint.00007444.tmp
13.1M 00007815
14.4M 00007816
10.2M 00007817
113.7M 00007818
272.0K 00007819
57.0M 00007820
207.1M checkpoint.00001456.tmp
508.9M checkpoint.00001778.tmp
504.3M checkpoint.00001788.tmp
473.4M checkpoint.00002171.tmp
488.2M checkpoint.00002344.tmp
1013.4M checkpoint.00006699
858.8M checkpoint.00006711.tmp
661.1M checkpoint.00007444.tmp
Thanos Ruler failed to complete the WAL checkpointing process, leading to continuous replaying from an old WAL checkpoint.
ts=2025-03-14T08:36:40.635034277Z caller=checkpoint.go:101 level=info msg="Creating checkpoint" from_segment=6700 to_segment=6711 mint=1741941080000
ts=2025-03-14T08:37:08.897002311Z caller=main.go:182 level=info msg="caught signal. Exiting." signal=terminated
Once the checkpointing process failed, Thanos did not clean up the old WAL files, causing excessive WAL accumulation.
/thanos/data/wal $ ls -lthr
-rw-rw-r-- 1 thanos thanos 0 Mar 13 16:54 00006700
-rw-rw-r-- 1 thanos thanos 0 Mar 13 16:55 00006701
-rw-rw-r-- 1 thanos thanos 0 Mar 13 16:56 00006702
-rw-rw-r-- 1 thanos thanos 0 Mar 13 16:59 00006703
-rw-rw-r-- 1 thanos thanos 0 Mar 13 17:00 00006704
-rw-rw-r-- 1 thanos thanos 64.1M Mar 13 18:00 00006705
-rw-rw-r-- 1 thanos thanos 116.4M Mar 13 20:01 00006706
-rw-rw-r-- 1 thanos thanos 6.5M Mar 13 20:05 00006707
-rw-rw-r-- 1 thanos thanos 127 Mar 13 20:08 00006708
-rw-rw-r-- 1 thanos thanos 65.6M Mar 13 21:10 00006709
-rw-rw-r-- 1 thanos thanos 127.9M Mar 13 23:11 00006710
-rw-rw-r-- 1 thanos thanos 70.9M Mar 14 00:15 00006711
-rw-rw-r-- 1 thanos thanos 2.0M Mar 14 00:18 00006712
-rw-rw-r-- 1 thanos thanos 0 Mar 14 00:21 00006713
-rw-rw-r-- 1 thanos thanos 105.4M Mar 14 02:25 00006714
-rw-rw-r-- 1 thanos thanos 45.4M Mar 14 03:25 00006715
-rw-rw-r-- 1 thanos thanos 69.3M Mar 14 05:26 00006716
drwxrwsr-x 2 thanos thanos 4.0K Mar 14 05:26 checkpoint.00006699
-rw-rw-r-- 1 thanos thanos 21.7M Mar 14 06:35 00006717
-rw-rw-r-- 1 thanos thanos 25.2M Mar 14 08:36 00006718
-rw-rw-r-- 1 thanos thanos 143.7K Mar 14 08:37 00006719
-rw-rw-r-- 1 thanos thanos 0 Mar 14 08:37 00006720
-rw-rw-r-- 1 thanos thanos 0 Mar 14 08:40 00006721
-rw-rw-r-- 1 thanos thanos 12.5M Mar 14 10:05 00006722
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:05 00006723
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:06 00006724
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:07 00006725
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:10 00006726
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:12 00006727
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:12 00006728
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:13 00006729
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:14 00006730
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:15 00006731
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:16 00006732
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:18 00006733
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:19 00006734
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:20 00006735
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:21 00006736
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:22 00006737
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:23 00006738
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:24 00006739
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:26 00006740
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:27 00006741
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:28 00006742
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:29 00006743
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:30 00006744
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:31 00006745
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:32 00006746
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:34 00006747
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:35 00006748
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:36 00006749
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:40 00006750
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:41 00006751
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:44 00006752
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:47 00006753
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:48 00006754
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:48 00006755
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:50 00006756
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:53 00006757
-rw-rw-r-- 1 thanos thanos 0 Mar 14 10:56 00006758
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:00 00006759
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:03 00006760
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:04 00006761
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:06 00006762
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:07 00006763
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:08 00006764
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:09 00006765
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:12 00006766
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:15 00006767
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:16 00006768
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:18 00006769
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:19 00006770
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:21 00006771
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:22 00006772
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:23 00006773
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:24 00006774
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:26 00006775
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:26 00006776
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:28 00006777
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:28 00006778
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:29 00006779
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:30 00006780
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:33 00006781
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:34 00006782
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:35 00006783
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:36 00006784
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:37 00006785
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:41 00006786
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:42 00006787
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:43 00006788
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:46 00006789
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:47 00006790
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:48 00006791
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:49 00006792
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:50 00006793
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:52 00006794
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:53 00006795
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:54 00006796
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:55 00006797
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:56 00006798
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:56 00006799
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:58 00006800
-rw-rw-r-- 1 thanos thanos 0 Mar 14 11:59 00006801
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:00 00006802
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:01 00006803
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:02 00006804
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:03 00006805
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:04 00006806
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:05 00006807
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:06 00006808
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:07 00006809
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:10 00006810
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:11 00006811
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:13 00006812
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:14 00006813
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:15 00006814
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:16 00006815
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:17 00006816
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:18 00006817
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:19 00006818
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:19 00006819
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:20 00006820
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:21 00006821
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:22 00006822
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:25 00006823
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:26 00006824
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:28 00006825
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:29 00006826
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:31 00006827
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:32 00006828
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:35 00006829
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:36 00006830
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:37 00006831
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:40 00006832
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:41 00006833
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:42 00006834
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:43 00006835
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:46 00006836
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:47 00006837
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:49 00006838
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:50 00006839
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:51 00006840
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:52 00006841
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:53 00006842
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:55 00006843
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:57 00006844
-rw-rw-r-- 1 thanos thanos 0 Mar 14 12:58 00006845
-rw-rw-r-- 1 thanos thanos 0 Mar 14 13:00 00006846
-rw-rw-r-- 1 thanos thanos 0 Mar 14 13:01 00006847
-rw-rw-r-- 1 thanos thanos 0 Mar 14 13:02 00006848
-rw-rw-r-- 1 thanos thanos 0 Mar 14 13:03 00006849
-rw-rw-r-- 1 thanos thanos 0 Mar 14 13:04 00006850
-rw-rw-r-- 1 thanos thanos 0 Mar 14 13:05 00006851
-rw-rw-r-- 1 thanos thanos 0 Mar 14 13:06 00006852
-rw-rw-r-- 1 thanos thanos 0 Mar 14 13:07 00006853
-rw-rw-r-- 1 thanos thanos 0 Mar 14 13:08 00006854
-rw-rw-r-- 1 thanos thanos 0 Mar 14 13:09 00006855
-rw-rw-r-- 1 thanos thanos 0 Mar 14 13:10 00006856
-rw-rw-r-- 1 thanos thanos 0 Mar 14 13:11 00006857
-rw-rw-r-- 1 thanos thanos 59.7M Mar 14 14:32 00006858
-rw-rw-r-- 1 thanos thanos 0 Mar 14 14:32 00006859
-rw-rw-r-- 1 thanos thanos 0 Mar 14 14:33 00006860
-rw-rw-r-- 1 thanos thanos 0 Mar 14 14:36 00006861
-rw-rw-r-- 1 thanos thanos 0 Mar 14 14:39 00006862
-rw-rw-r-- 1 thanos thanos 0 Mar 14 14:40 00006863
-rw-rw-r-- 1 thanos thanos 0 Mar 14 14:42 00006864
-rw-rw-r-- 1 thanos thanos 0 Mar 14 14:43 00006865
-rw-rw-r-- 1 thanos thanos 0 Mar 14 14:45 00006866
-rw-rw-r-- 1 thanos thanos 0 Mar 14 14:47 00006867
-rw-rw-r-- 1 thanos thanos 0 Mar 14 14:48 00006868
-rw-rw-r-- 1 thanos thanos 0 Mar 14 14:49 00006869
-rw-rw-r-- 1 thanos thanos 0 Mar 14 14:50 00006870
-rw-rw-r-- 1 thanos thanos 0 Mar 14 14:51 00006871
-rw-rw-r-- 1 thanos thanos 0 Mar 14 14:52 00006872
-rw-rw-r-- 1 thanos thanos 0 Mar 14 14:53 00006873
-rw-rw-r-- 1 thanos thanos 0 Mar 14 14:54 00006874
-rw-rw-r-- 1 thanos thanos 0 Mar 14 14:55 00006875
-rw-rw-r-- 1 thanos thanos 0 Mar 14 14:55 00006876
-rw-rw-r-- 1 thanos thanos 0 Mar 14 14:57 00006877
-rw-rw-r-- 1 thanos thanos 0 Mar 14 14:58 00006878
-rw-rw-r-- 1 thanos thanos 0 Mar 14 14:58 00006879
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:00 00006880
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:01 00006881
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:01 00006882
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:04 00006883
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:06 00006884
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:07 00006885
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:08 00006886
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:08 00006887
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:10 00006888
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:10 00006889
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:11 00006890
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:13 00006891
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:13 00006892
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:14 00006893
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:16 00006894
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:16 00006895
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:17 00006896
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:21 00006897
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:22 00006898
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:23 00006899
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:26 00006900
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:27 00006901
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:28 00006902
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:29 00006903
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:30 00006904
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:32 00006905
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:33 00006906
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:34 00006907
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:35 00006908
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:35 00006909
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:39 00006910
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:41 00006911
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:41 00006912
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:42 00006913
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:45 00006914
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:47 00006915
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:48 00006916
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:51 00006917
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:52 00006918
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:52 00006919
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:54 00006920
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:54 00006921
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:57 00006922
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:58 00006923
-rw-rw-r-- 1 thanos thanos 0 Mar 14 15:59 00006924
-rw-rw-r-- 1 thanos thanos 0 Mar 14 16:00 00006925
-rw-rw-r-- 1 thanos thanos 48.0M Mar 14 16:45 00006926
-rw-rw-r-- 1 thanos thanos 0 Mar 14 16:46 00006927
-rw-rw-r-- 1 thanos thanos 0 Mar 14 16:47 00006928
-rw-rw-r-- 1 thanos thanos 0 Mar 14 16:51 00006929
-rw-rw-r-- 1 thanos thanos 0 Mar 14 16:51 00006930
-rw-rw-r-- 1 thanos thanos 0 Mar 14 16:52 00006931
-rw-rw-r-- 1 thanos thanos 0 Mar 14 16:53 00006932
-rw-rw-r-- 1 thanos thanos 0 Mar 14 16:55 00006933
-rw-rw-r-- 1 thanos thanos 0 Mar 14 16:55 00006934
-rw-rw-r-- 1 thanos thanos 0 Mar 14 16:56 00006935
-rw-rw-r-- 1 thanos thanos 0 Mar 14 16:57 00006936
-rw-rw-r-- 1 thanos thanos 0 Mar 14 16:58 00006937
-rw-rw-r-- 1 thanos thanos 0 Mar 14 16:59 00006938
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:00 00006939
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:01 00006940
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:03 00006941
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:04 00006942
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:05 00006943
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:06 00006944
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:07 00006945
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:08 00006946
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:09 00006947
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:10 00006948
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:14 00006949
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:16 00006950
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:17 00006951
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:20 00006952
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:21 00006953
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:23 00006954
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:23 00006955
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:26 00006956
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:28 00006957
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:28 00006958
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:29 00006959
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:31 00006960
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:31 00006961
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:32 00006962
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:36 00006963
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:37 00006964
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:37 00006965
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:41 00006966
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:42 00006967
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:44 00006968
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:45 00006969
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:49 00006970
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:50 00006971
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:51 00006972
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:54 00006973
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:55 00006974
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:57 00006975
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:57 00006976
-rw-rw-r-- 1 thanos thanos 0 Mar 14 17:58 00006977
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:00 00006978
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:01 00006979
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:02 00006980
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:04 00006981
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:05 00006982
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:07 00006983
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:07 00006984
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:08 00006985
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:12 00006986
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:13 00006987
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:14 00006988
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:17 00006989
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:18 00006990
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:19 00006991
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:20 00006992
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:23 00006993
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:24 00006994
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:25 00006995
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:26 00006996
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:28 00006997
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:30 00006998
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:32 00006999
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:33 00007000
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:34 00007001
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:35 00007002
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:36 00007003
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:39 00007004
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:39 00007005
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:41 00007006
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:42 00007007
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:42 00007008
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:45 00007009
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:47 00007010
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:47 00007011
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:49 00007012
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:50 00007013
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:51 00007014
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:52 00007015
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:53 00007016
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:54 00007017
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:56 00007018
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:56 00007019
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:57 00007020
-rw-rw-r-- 1 thanos thanos 0 Mar 14 18:58 00007021
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:00 00007022
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:03 00007023
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:05 00007024
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:06 00007025
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:07 00007026
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:08 00007027
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:12 00007028
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:13 00007029
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:15 00007030
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:18 00007031
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:19 00007032
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:20 00007033
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:21 00007034
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:24 00007035
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:25 00007036
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:26 00007037
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:27 00007038
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:28 00007039
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:29 00007040
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:31 00007041
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:32 00007042
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:34 00007043
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:36 00007044
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:37 00007045
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:38 00007046
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:39 00007047
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:40 00007048
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:41 00007049
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:45 00007050
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:46 00007051
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:46 00007052
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:50 00007053
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:51 00007054
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:52 00007055
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:54 00007056
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:57 00007057
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:58 00007058
-rw-rw-r-- 1 thanos thanos 0 Mar 14 19:59 00007059
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:02 00007060
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:03 00007061
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:04 00007062
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:05 00007063
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:06 00007064
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:08 00007065
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:09 00007066
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:12 00007067
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:12 00007068
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:13 00007069
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:14 00007070
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:16 00007071
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:19 00007072
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:20 00007073
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:20 00007074
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:22 00007075
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:24 00007076
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:27 00007077
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:27 00007078
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:29 00007079
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:30 00007080
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:31 00007081
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:32 00007082
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:33 00007083
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:35 00007084
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:38 00007085
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:41 00007086
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:42 00007087
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:43 00007088
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:44 00007089
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:45 00007090
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:46 00007091
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:47 00007092
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:50 00007093
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:51 00007094
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:54 00007095
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:57 00007096
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:58 00007097
-rw-rw-r-- 1 thanos thanos 0 Mar 14 20:59 00007098
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:00 00007099
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:03 00007100
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:04 00007101
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:05 00007102
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:06 00007103
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:07 00007104
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:09 00007105
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:10 00007106
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:11 00007107
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:12 00007108
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:13 00007109
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:14 00007110
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:15 00007111
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:16 00007112
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:17 00007113
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:18 00007114
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:18 00007115
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:21 00007116
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:23 00007117
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:24 00007118
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:27 00007119
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:28 00007120
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:29 00007121
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:30 00007122
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:31 00007123
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:32 00007124
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:33 00007125
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:34 00007126
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:35 00007127
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:35 00007128
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:37 00007129
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:37 00007130
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:38 00007131
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:40 00007132
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:40 00007133
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:41 00007134
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:42 00007135
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:45 00007136
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:46 00007137
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:49 00007138
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:51 00007139
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:52 00007140
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:55 00007141
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:56 00007142
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:57 00007143
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:58 00007144
-rw-rw-r-- 1 thanos thanos 0 Mar 14 21:59 00007145
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:00 00007146
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:03 00007147
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:04 00007148
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:06 00007149
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:07 00007150
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:08 00007151
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:10 00007152
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:13 00007153
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:14 00007154
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:14 00007155
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:16 00007156
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:17 00007157
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:17 00007158
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:20 00007159
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:21 00007160
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:22 00007161
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:24 00007162
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:26 00007163
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:27 00007164
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:28 00007165
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:29 00007166
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:31 00007167
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:31 00007168
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:32 00007169
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:35 00007170
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:37 00007171
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:38 00007172
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:39 00007173
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:40 00007174
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:41 00007175
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:44 00007176
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:46 00007177
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:47 00007178
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:48 00007179
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:49 00007180
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:50 00007181
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:51 00007182
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:53 00007183
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:54 00007184
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:55 00007185
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:56 00007186
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:57 00007187
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:58 00007188
-rw-rw-r-- 1 thanos thanos 0 Mar 14 22:59 00007189
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:03 00007190
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:04 00007191
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:05 00007192
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:06 00007193
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:09 00007194
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:10 00007195
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:11 00007196
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:12 00007197
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:15 00007198
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:16 00007199
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:17 00007200
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:18 00007201
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:22 00007202
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:24 00007203
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:25 00007204
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:26 00007205
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:28 00007206
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:29 00007207
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:31 00007208
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:32 00007209
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:33 00007210
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:34 00007211
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:35 00007212
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:36 00007213
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:37 00007214
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:40 00007215
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:40 00007216
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:41 00007217
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:44 00007218
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:46 00007219
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:47 00007220
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:48 00007221
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:49 00007222
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:50 00007223
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:53 00007224
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:56 00007225
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:57 00007226
-rw-rw-r-- 1 thanos thanos 0 Mar 14 23:58 00007227
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:02 00007228
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:03 00007229
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:04 00007230
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:05 00007231
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:08 00007232
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:11 00007233
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:12 00007234
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:13 00007235
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:14 00007236
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:15 00007237
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:16 00007238
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:19 00007239
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:20 00007240
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:21 00007241
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:24 00007242
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:26 00007243
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:27 00007244
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:29 00007245
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:30 00007246
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:31 00007247
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:34 00007248
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:35 00007249
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:36 00007250
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:37 00007251
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:38 00007252
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:39 00007253
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:42 00007254
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:43 00007255
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:44 00007256
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:47 00007257
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:48 00007258
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:50 00007259
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:51 00007260
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:52 00007261
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:52 00007262
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:54 00007263
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:57 00007264
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:58 00007265
-rw-rw-r-- 1 thanos thanos 0 Mar 15 00:59 00007266
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:00 00007267
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:01 00007268
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:03 00007269
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:04 00007270
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:05 00007271
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:06 00007272
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:09 00007273
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:10 00007274
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:13 00007275
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:14 00007276
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:15 00007277
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:16 00007278
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:17 00007279
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:20 00007280
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:20 00007281
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:21 00007282
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:22 00007283
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:25 00007284
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:26 00007285
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:28 00007286
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:29 00007287
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:30 00007288
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:33 00007289
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:36 00007290
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:37 00007291
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:40 00007292
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:41 00007293
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:42 00007294
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:43 00007295
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:44 00007296
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:47 00007297
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:48 00007298
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:49 00007299
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:50 00007300
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:53 00007301
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:56 00007302
-rw-rw-r-- 1 thanos thanos 0 Mar 15 01:59 00007303
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:00 00007304
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:01 00007305
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:02 00007306
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:04 00007307
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:05 00007308
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:06 00007309
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:08 00007310
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:10 00007311
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:13 00007312
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:14 00007313
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:17 00007314
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:18 00007315
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:18 00007316
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:20 00007317
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:21 00007318
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:23 00007319
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:24 00007320
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:27 00007321
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:28 00007322
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:29 00007323
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:30 00007324
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:31 00007325
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:32 00007326
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:35 00007327
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:36 00007328
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:37 00007329
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:38 00007330
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:39 00007331
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:41 00007332
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:42 00007333
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:43 00007334
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:45 00007335
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:45 00007336
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:47 00007337
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:49 00007338
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:53 00007339
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:55 00007340
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:56 00007341
-rw-rw-r-- 1 thanos thanos 0 Mar 15 02:59 00007342
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:00 00007343
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:01 00007344
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:03 00007345
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:03 00007346
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:05 00007347
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:06 00007348
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:09 00007349
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:11 00007350
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:12 00007351
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:15 00007352
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:16 00007353
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:17 00007354
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:18 00007355
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:19 00007356
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:21 00007357
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:22 00007358
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:23 00007359
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:24 00007360
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:27 00007361
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:28 00007362
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:29 00007363
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:30 00007364
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:33 00007365
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:34 00007366
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:36 00007367
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:37 00007368
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:38 00007369
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:39 00007370
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:40 00007371
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:41 00007372
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:42 00007373
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:45 00007374
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:47 00007375
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:48 00007376
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:51 00007377
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:52 00007378
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:54 00007379
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:55 00007380
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:56 00007381
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:57 00007382
-rw-rw-r-- 1 thanos thanos 0 Mar 15 03:58 00007383
-rw-rw-r-- 1 thanos thanos 0 Mar 15 04:01 00007384
-rw-rw-r-- 1 thanos thanos 0 Mar 15 04:02 00007385
-rw-rw-r-- 1 thanos thanos 0 Mar 15 04:03 00007386
-rw-rw-r-- 1 thanos thanos 0 Mar 15 04:04 00007387
-rw-rw-r-- 1 thanos thanos 0 Mar 15 04:07 00007388
-rw-rw-r-- 1 thanos thanos 0 Mar 15 04:08 00007389
-rw-rw-r-- 1 thanos thanos 0 Mar 15 04:09 00007390
-rw-rw-r-- 1 thanos thanos 0 Mar 15 04:10 00007391
-rw-rw-r-- 1 thanos thanos 0 Mar 15 04:11 00007392
-rw-rw-r-- 1 thanos thanos 0 Mar 15 04:12 00007393
-rw-rw-r-- 1 thanos thanos 0 Mar 15 04:13 00007394
-rw-rw-r-- 1 thanos thanos 0 Mar 15 04:15 00007395
-rw-rw-r-- 1 thanos thanos 0 Mar 15 04:16 00007396
-rw-rw-r-- 1 thanos thanos 0 Mar 15 04:20 00007397
-rw-rw-r-- 1 thanos thanos 0 Mar 15 04:21 00007398
-rw-rw-r-- 1 thanos thanos 31.1M Mar 15 05:02 00007399
-rw-rw-r-- 1 thanos thanos 0 Mar 15 05:03 00007400
-rw-rw-r-- 1 thanos thanos 0 Mar 15 05:06 00007401
-rw-rw-r-- 1 thanos thanos 0 Mar 15 05:08 00007402
-rw-rw-r-- 1 thanos thanos 0 Mar 15 05:09 00007403
-rw-rw-r-- 1 thanos thanos 0 Mar 15 05:12 00007404
-rw-rw-r-- 1 thanos thanos 0 Mar 15 05:13 00007405
-rw-rw-r-- 1 thanos thanos 0 Mar 15 05:14 00007406
-rw-rw-r-- 1 thanos thanos 0 Mar 15 05:16 00007407
-rw-rw-r-- 1 thanos thanos 0 Mar 15 05:19 00007408
-rw-rw-r-- 1 thanos thanos 0 Mar 15 05:21 00007409
-rw-rw-r-- 1 thanos thanos 0 Mar 15 05:22 00007410
-rw-rw-r-- 1 thanos thanos 0 Mar 15 05:23 00007411
-rw-rw-r-- 1 thanos thanos 0 Mar 15 05:26 00007412
-rw-rw-r-- 1 thanos thanos 0 Mar 15 05:27 00007413
-rw-rw-r-- 1 thanos thanos 0 Mar 15 05:28 00007414
-rw-rw-r-- 1 thanos thanos 0 Mar 15 05:31 00007415
-rw-rw-r-- 1 thanos thanos 0 Mar 15 05:32 00007416
-rw-rw-r-- 1 thanos thanos 0 Mar 15 05:33 00007417
-rw-rw-r-- 1 thanos thanos 0 Mar 15 05:34 00007418
-rw-rw-r-- 1 thanos thanos 0 Mar 15 05:37 00007419
-rw-rw-r-- 1 thanos thanos 0 Mar 15 05:38 00007420
-rw-rw-r-- 1 thanos thanos 0 Mar 15 05:39 00007421
-rw-rw-r-- 1 thanos thanos 0 Mar 15 05:40 00007422
-rw-rw-r-- 1 thanos thanos 0 Mar 15 05:40 00007423
-rw-rw-r-- 1 thanos thanos 0 Mar 15 05:43 00007424
-rw-rw-r-- 1 thanos thanos 0 Mar 15 05:46 00007425
-rw-rw-r-- 1 thanos thanos 0 Mar 15 05:47 00007426
-rw-rw-r-- 1 thanos thanos 0 Mar 15 05:49 00007427
-rw-rw-r-- 1 thanos thanos 0 Mar 15 05:50 00007428
-rw-rw-r-- 1 thanos thanos 0 Mar 15 05:51 00007429
-rw-rw-r-- 1 thanos thanos 0 Mar 15 05:53 00007430
-rw-rw-r-- 1 thanos thanos 0 Mar 15 05:54 00007431
-rw-rw-r-- 1 thanos thanos 0 Mar 15 05:56 00007432
-rw-rw-r-- 1 thanos thanos 0 Mar 15 05:58 00007433
-rw-rw-r-- 1 thanos thanos 0 Mar 15 05:59 00007434
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:00 00007435
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:01 00007436
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:02 00007437
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:05 00007438
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:05 00007439
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:07 00007440
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:08 00007441
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:11 00007442
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:12 00007443
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:14 00007444
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:15 00007445
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:16 00007446
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:17 00007447
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:18 00007448
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:20 00007449
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:21 00007450
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:24 00007451
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:27 00007452
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:28 00007453
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:29 00007454
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:30 00007455
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:31 00007456
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:33 00007457
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:34 00007458
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:37 00007459
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:39 00007460
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:40 00007461
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:41 00007462
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:43 00007463
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:44 00007464
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:45 00007465
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:46 00007466
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:49 00007467
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:50 00007468
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:51 00007469
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:53 00007470
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:55 00007471
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:56 00007472
-rw-rw-r-- 1 thanos thanos 0 Mar 15 06:59 00007473
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:00 00007474
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:01 00007475
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:02 00007476
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:05 00007477
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:06 00007478
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:07 00007479
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:08 00007480
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:11 00007481
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:12 00007482
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:13 00007483
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:14 00007484
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:17 00007485
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:18 00007486
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:19 00007487
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:20 00007488
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:21 00007489
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:24 00007490
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:27 00007491
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:28 00007492
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:30 00007493
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:31 00007494
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:33 00007495
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:36 00007496
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:38 00007497
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:39 00007498
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:40 00007499
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:41 00007500
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:44 00007501
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:45 00007502
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:47 00007503
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:48 00007504
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:50 00007505
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:51 00007506
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:52 00007507
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:53 00007508
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:54 00007509
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:55 00007510
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:56 00007511
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:57 00007512
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:58 00007513
-rw-rw-r-- 1 thanos thanos 0 Mar 15 07:59 00007514
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:02 00007515
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:03 00007516
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:05 00007517
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:06 00007518
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:08 00007519
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:09 00007520
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:10 00007521
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:11 00007522
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:13 00007523
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:14 00007524
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:16 00007525
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:18 00007526
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:19 00007527
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:20 00007528
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:21 00007529
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:24 00007530
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:26 00007531
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:27 00007532
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:29 00007533
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:31 00007534
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:31 00007535
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:33 00007536
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:34 00007537
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:36 00007538
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:36 00007539
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:40 00007540
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:41 00007541
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:42 00007542
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:42 00007543
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:43 00007544
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:44 00007545
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:48 00007546
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:50 00007547
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:51 00007548
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:54 00007549
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:55 00007550
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:56 00007551
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:57 00007552
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:58 00007553
-rw-rw-r-- 1 thanos thanos 0 Mar 15 08:59 00007554
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:01 00007555
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:02 00007556
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:03 00007557
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:06 00007558
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:08 00007559
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:10 00007560
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:10 00007561
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:11 00007562
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:15 00007563
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:17 00007564
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:20 00007565
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:22 00007566
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:25 00007567
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:26 00007568
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:27 00007569
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:28 00007570
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:29 00007571
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:31 00007572
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:32 00007573
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:34 00007574
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:36 00007575
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:37 00007576
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:38 00007577
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:41 00007578
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:42 00007579
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:44 00007580
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:45 00007581
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:47 00007582
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:49 00007583
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:49 00007584
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:51 00007585
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:52 00007586
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:54 00007587
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:54 00007588
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:57 00007589
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:58 00007590
-rw-rw-r-- 1 thanos thanos 0 Mar 15 09:59 00007591
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:00 00007592
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:01 00007593
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:02 00007594
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:04 00007595
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:05 00007596
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:06 00007597
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:06 00007598
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:07 00007599
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:11 00007600
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:12 00007601
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:13 00007602
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:14 00007603
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:15 00007604
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:17 00007605
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:18 00007606
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:21 00007607
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:22 00007608
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:24 00007609
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:25 00007610
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:26 00007611
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:26 00007612
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:30 00007613
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:32 00007614
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:36 00007615
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:37 00007616
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:39 00007617
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:40 00007618
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:43 00007619
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:44 00007620
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:45 00007621
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:46 00007622
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:47 00007623
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:48 00007624
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:51 00007625
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:52 00007626
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:53 00007627
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:54 00007628
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:55 00007629
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:56 00007630
-rw-rw-r-- 1 thanos thanos 0 Mar 15 10:57 00007631
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:00 00007632
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:02 00007633
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:03 00007634
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:05 00007635
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:06 00007636
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:09 00007637
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:09 00007638
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:11 00007639
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:12 00007640
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:13 00007641
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:14 00007642
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:16 00007643
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:17 00007644
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:18 00007645
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:19 00007646
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:22 00007647
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:23 00007648
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:24 00007649
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:25 00007650
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:26 00007651
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:28 00007652
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:29 00007653
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:32 00007654
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:32 00007655
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:33 00007656
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:35 00007657
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:36 00007658
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:38 00007659
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:39 00007660
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:40 00007661
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:41 00007662
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:42 00007663
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:43 00007664
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:45 00007665
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:45 00007666
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:48 00007667
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:51 00007668
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:51 00007669
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:54 00007670
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:55 00007671
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:55 00007672
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:56 00007673
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:57 00007674
-rw-rw-r-- 1 thanos thanos 0 Mar 15 11:58 00007675
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:01 00007676
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:02 00007677
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:03 00007678
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:03 00007679
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:05 00007680
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:05 00007681
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:08 00007682
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:09 00007683
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:10 00007684
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:11 00007685
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:12 00007686
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:15 00007687
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:17 00007688
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:18 00007689
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:19 00007690
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:20 00007691
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:20 00007692
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:21 00007693
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:22 00007694
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:25 00007695
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:26 00007696
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:27 00007697
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:31 00007698
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:32 00007699
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:33 00007700
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:34 00007701
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:35 00007702
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:37 00007703
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:38 00007704
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:39 00007705
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:40 00007706
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:41 00007707
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:41 00007708
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:42 00007709
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:46 00007710
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:47 00007711
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:48 00007712
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:49 00007713
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:52 00007714
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:53 00007715
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:53 00007716
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:54 00007717
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:55 00007718
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:58 00007719
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:58 00007720
-rw-rw-r-- 1 thanos thanos 0 Mar 15 12:59 00007721
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:00 00007722
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:01 00007723
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:04 00007724
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:05 00007725
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:06 00007726
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:08 00007727
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:09 00007728
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:11 00007729
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:12 00007730
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:13 00007731
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:14 00007732
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:18 00007733
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:19 00007734
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:20 00007735
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:21 00007736
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:24 00007737
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:25 00007738
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:26 00007739
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:27 00007740
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:28 00007741
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:31 00007742
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:34 00007743
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:35 00007744
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:37 00007745
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:38 00007746
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:39 00007747
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:40 00007748
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:44 00007749
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:45 00007750
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:46 00007751
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:47 00007752
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:50 00007753
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:51 00007754
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:52 00007755
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:55 00007756
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:56 00007757
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:57 00007758
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:58 00007759
-rw-rw-r-- 1 thanos thanos 0 Mar 15 13:59 00007760
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:00 00007761
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:01 00007762
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:03 00007763
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:04 00007764
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:05 00007765
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:06 00007766
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:07 00007767
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:08 00007768
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:09 00007769
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:09 00007770
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:12 00007771
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:14 00007772
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:15 00007773
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:16 00007774
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:17 00007775
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:18 00007776
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:21 00007777
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:24 00007778
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:25 00007779
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:26 00007780
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:29 00007781
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:30 00007782
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:32 00007783
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:32 00007784
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:33 00007785
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:35 00007786
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:36 00007787
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:37 00007788
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:38 00007789
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:40 00007790
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:40 00007791
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:41 00007792
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:45 00007793
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:48 00007794
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:49 00007795
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:52 00007796
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:53 00007797
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:56 00007798
-rw-rw-r-- 1 thanos thanos 0 Mar 15 14:58 00007799
-rw-rw-r-- 1 thanos thanos 0 Mar 15 15:00 00007800
-rw-rw-r-- 1 thanos thanos 0 Mar 15 15:01 00007801
-rw-rw-r-- 1 thanos thanos 0 Mar 15 15:03 00007802
-rw-rw-r-- 1 thanos thanos 0 Mar 15 15:06 00007803
-rw-rw-r-- 1 thanos thanos 0 Mar 15 15:07 00007804
-rw-rw-r-- 1 thanos thanos 0 Mar 15 15:08 00007805
-rw-rw-r-- 1 thanos thanos 0 Mar 15 15:10 00007806
-rw-rw-r-- 1 thanos thanos 0 Mar 15 15:11 00007807
-rw-rw-r-- 1 thanos thanos 0 Mar 15 15:12 00007808
-rw-rw-r-- 1 thanos thanos 0 Mar 15 15:13 00007809
-rw-rw-r-- 1 thanos thanos 0 Mar 15 15:16 00007810
-rw-rw-r-- 1 thanos thanos 0 Mar 15 15:18 00007811
-rw-rw-r-- 1 thanos thanos 0 Mar 15 15:18 00007812
-rw-rw-r-- 1 thanos thanos 0 Mar 15 15:19 00007813
-rw-rw-r-- 1 thanos thanos 0 Mar 15 15:20 00007814
-rw-rw-r-- 1 thanos thanos 13.1M Mar 17 15:19 00007815
-rw-rw-r-- 1 thanos thanos 14.4M Mar 17 15:23 00007816
-rw-rw-r-- 1 thanos thanos 10.2M Mar 17 15:37 00007817
-rw-rw-r-- 1 thanos thanos 113.7M Mar 17 17:39 00007818
-rw-rw-r-- 1 thanos thanos 268.3K Mar 17 17:39 00007819
drwxrwsr-x 2 thanos thanos 4.0K Mar 17 17:39 checkpoint.00007444.tmp
-rw-rw-r-- 1 thanos thanos 112.7M Mar 17 19:43 00007820
-rw-rw-r-- 1 thanos thanos 194.2K Mar 17 19:43 00007821
drwxrwsr-x 2 thanos thanos 4.0K Mar 17 19:43 checkpoint.00007446.tmp
-rw-rw-r-- 1 thanos thanos 0 Mar 17 19:44 00007822
-rw-rw-r-- 1 thanos thanos 7.5M Mar 17 19:55 00007823
-rw-rw-r-- 1 thanos thanos 8.8M Mar 17 20:07 00007824
-rw-rw-r-- 1 thanos thanos 2.7M Mar 17 20:13 00007825
-rw-rw-r-- 1 thanos thanos 2.1M Mar 17 20:18 00007826
-rw-rw-r-- 1 thanos thanos 116.6K Mar 17 20:20 00007827
-rw-rw-r-- 1 thanos thanos 1.3M Mar 17 20:23 00007828
-rw-rw-r-- 1 thanos thanos 14.6M Mar 17 20:45 00007829
When the Pod restarted, the large WAL replay resulted in high memory consumption, eventually triggering node eviction before the replay could finish. At one point memory usage went to 64GB and Pod was running nearly for 2 hours. During that timeframe we saw messages in logs like WAL replaying done but still memory usage didn't come down and eventually Pod was evicted by node due to node low on memory.
45s Warning Evicted pod/thanos-ruler-default-0 The node was low on resource: memory.
Threshold quantity: 1574201572, available: 799516Ki. Container thanos-ruler was using 26306460Ki, request is 24Gi, has larger
consumption of memory.
Here is the memory usage (check after 13th)
At one point pod was in a crashloopbackoff state due to no space left on device. So we increased PVC size from 5GB to 10GB.
ts=2025-03-17T14:18:02.328721664Z caller=main.go:145 level=error err="creating WAL: open /thanos/data/wal/00008772: no space left on device..."
Resolution
We manually cleaned up old WAL files. Immediately after this, the Pod’s memory usage decreased significantly, and it stabilized without further evictions.
Here is the memory usage after cleanup up the WAL files
Thanks for the info! I saw one situation where the OOM killer killed some processes, and it killed some fan-out goroutines, and after that, the TSDB got stuck waiting for all readers to finish before continuing the checkpointing process. I'm not 100% sure about the details, but I wonder whether you might've encountered the same issue. Do you have a goroutine dump perhaps when checkpointing stops working?
~~Here are a couple goroutine dumps (port-forwarded the pod at port 10902 and hit /debug/pprof/goroutine) I have for a similar situation (old WALs not being cleaned up, so Pod is in high memory usage). This is before I delete the WAL dir to mitigate~~:
Ran at time N
goroutine profile: total 915
856 @ 0x47b70e 0x4581f7 0x176d06a 0x1776646 0x482f61
# 0x176d069 github.com/prometheus/prometheus/rules.(*Group).run+0x9a9 /go/pkg/mod/github.com/prometheus/[email protected]/rules/group.go:296
# 0x1776645 github.com/prometheus/prometheus/rules.(*Manager).Update.func1+0xa5 /go/pkg/mod/github.com/prometheus/[email protected]/rules/manager.go:246
7 @ 0x47b70e 0x4581f7 0x7eafe7 0x482f61
# 0x7eafe6 net/http.(*persistConn).writeLoop+0xe6 /go/pkg/mod/golang.org/[email protected]/src/net/http/transport.go:2590
5 @ 0x47b70e 0x43e3d7 0x47a925 0x502627 0x50391a 0x503908 0x5bbf45 0x5cd8a5 0x7e8bc7 0x58f623 0x58f753 0x7e96d2 0x482f61
# 0x47a924 internal/poll.runtime_pollWait+0x84 /go/pkg/mod/golang.org/[email protected]/src/runtime/netpoll.go:351
# 0x502626 internal/poll.(*pollDesc).wait+0x26 /go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_poll_runtime.go:84
# 0x503919 internal/poll.(*pollDesc).waitRead+0x279 /go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_poll_runtime.go:89
# 0x503907 internal/poll.(*FD).Read+0x267 /go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_unix.go:165
# 0x5bbf44 net.(*netFD).Read+0x24 /go/pkg/mod/golang.org/[email protected]/src/net/fd_posix.go:55
# 0x5cd8a4 net.(*conn).Read+0x44 /go/pkg/mod/golang.org/[email protected]/src/net/net.go:194
# 0x7e8bc6 net/http.(*persistConn).Read+0x46 /go/pkg/mod/golang.org/[email protected]/src/net/http/transport.go:2122
# 0x58f622 bufio.(*Reader).fill+0x102 /go/pkg/mod/golang.org/[email protected]/src/bufio/bufio.go:113
# 0x58f752 bufio.(*Reader).Peek+0x52 /go/pkg/mod/golang.org/[email protected]/src/bufio/bufio.go:152
# 0x7e96d1 net/http.(*persistConn).readLoop+0x171 /go/pkg/mod/golang.org/[email protected]/src/net/http/transport.go:2275
3 @ 0x47b70e 0x4581f7 0x7ebec5 0x7e009a 0x7ef393 0x15d0ee3 0x15d0599 0x15e9888 0x15d0599 0x15d091c 0x15d0599 0x15e9128 0x15d0599 0x1ee4e24 0x1628cd2 0x77b0c2 0x77a971 0x77cf89 0x1ee55f7 0x1ee55ef 0x1f0c430 0x1f0fb2c 0x1f10765 0x253e885 0x1768738 0x1770137 0x176f7ba 0x1775285 0x176cfbc 0x1776646 0x482f61
# 0x7ebec4 net/http.(*persistConn).roundTrip+0x784 /go/pkg/mod/golang.org/[email protected]/src/net/http/transport.go:2826
# 0x7e0099 net/http.(*Transport).roundTrip+0xb99 /go/pkg/mod/golang.org/[email protected]/src/net/http/transport.go:696
# 0x7ef392 net/http.(*Transport).RoundTrip+0x12 /go/pkg/mod/golang.org/[email protected]/src/net/http/roundtrip.go:30
# 0x15d0ee2 github.com/prometheus/client_golang/prometheus/promhttp.InstrumentRoundTripperDuration.func1+0xa2 /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:119
# 0x15d0598 github.com/prometheus/client_golang/prometheus/promhttp.RoundTripperFunc.RoundTrip+0x18 /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:32
# 0x15e9887 github.com/thanos-io/thanos/pkg/extprom/http.InstrumentedRoundTripper.InstrumentRoundTripperTrace.func5+0x687 /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:247
# 0x15d0598 github.com/prometheus/client_golang/prometheus/promhttp.RoundTripperFunc.RoundTrip+0x18 /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:32
# 0x15d091b github.com/prometheus/client_golang/prometheus/promhttp.InstrumentRoundTripperCounter.func1+0x5b /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:75
# 0x15d0598 github.com/prometheus/client_golang/prometheus/promhttp.RoundTripperFunc.RoundTrip+0x18 /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:32
# 0x15e9127 github.com/thanos-io/thanos/pkg/extprom/http.InstrumentedRoundTripper.InstrumentRoundTripperInFlight.func6+0xa7 /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:44
# 0x15d0598 github.com/prometheus/client_golang/prometheus/promhttp.RoundTripperFunc.RoundTrip+0x18 /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:32
# 0x1ee4e23 github.com/thanos-io/thanos/pkg/clientconfig.userAgentRoundTripper.RoundTrip+0x163 /app/pkg/clientconfig/http.go:299
# 0x1628cd1 github.com/thanos-io/thanos/pkg/tracing.(*tripperware).RoundTrip+0x471 /app/pkg/tracing/http.go:112
# 0x77b0c1 net/http.send+0x5e1 /go/pkg/mod/golang.org/[email protected]/src/net/http/client.go:259
# 0x77a970 net/http.(*Client).send+0x90 /go/pkg/mod/golang.org/[email protected]/src/net/http/client.go:180
# 0x77cf88 net/http.(*Client).do+0x988 /go/pkg/mod/golang.org/[email protected]/src/net/http/client.go:728
# 0x1ee55f6 net/http.(*Client).Do+0x16 /go/pkg/mod/golang.org/[email protected]/src/net/http/client.go:587
# 0x1ee55ee github.com/thanos-io/thanos/pkg/clientconfig.(*HTTPClient).Do+0xe /app/pkg/clientconfig/http.go:387
# 0x1f0c42f github.com/thanos-io/thanos/pkg/promclient.(*Client).req2xx+0x4cf /app/pkg/promclient/promclient.go:132
# 0x1f0fb2b github.com/thanos-io/thanos/pkg/promclient.(*Client).QueryInstant+0x5eb /app/pkg/promclient/promclient.go:450
# 0x1f10764 github.com/thanos-io/thanos/pkg/promclient.(*Client).PromqlQueryInstant+0x84 /app/pkg/promclient/promclient.go:503
# 0x253e884 main.runRule.queryFuncCreator.func24.1+0xdc4 /app/cmd/thanos/rule.go:917
# 0x1768737 github.com/prometheus/prometheus/rules.(*AlertingRule).Eval+0x1f7 /go/pkg/mod/github.com/prometheus/[email protected]/rules/alerting.go:343
# 0x1770136 github.com/prometheus/prometheus/rules.(*Group).Eval.func1+0x776 /go/pkg/mod/github.com/prometheus/[email protected]/rules/group.go:540
# 0x176f7b9 github.com/prometheus/prometheus/rules.(*Group).Eval+0x379 /go/pkg/mod/github.com/prometheus/[email protected]/rules/group.go:665
# 0x1775284 github.com/prometheus/prometheus/rules.DefaultEvalIterationFunc+0xe4 /go/pkg/mod/github.com/prometheus/[email protected]/rules/manager.go:81
# 0x176cfbb github.com/prometheus/prometheus/rules.(*Group).run+0x8fb /go/pkg/mod/github.com/prometheus/[email protected]/rules/group.go:307
# 0x1776645 github.com/prometheus/prometheus/rules.(*Manager).Update.func1+0xa5 /go/pkg/mod/github.com/prometheus/[email protected]/rules/manager.go:246
2 @ 0x47b70e 0x411d85 0x411912 0x1f800ae 0x482f61
# 0x1f800ad github.com/prometheus/prometheus/rules.(*Manager).Run+0x6d /go/pkg/mod/github.com/prometheus/[email protected]/rules/manager.go:171
2 @ 0x47b70e 0x43e3d7 0x47a925 0x502627 0x50391a 0x503908 0x5bbf45 0x5cd8a5 0x6fed5b 0x54c1d8 0x6fef3e 0x6fc04f 0x702825 0x702826 0x7e8bc7 0x58f623 0x58f753 0x7e96d2 0x482f61
# 0x47a924 internal/poll.runtime_pollWait+0x84 /go/pkg/mod/golang.org/[email protected]/src/runtime/netpoll.go:351
# 0x502626 internal/poll.(*pollDesc).wait+0x26 /go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_poll_runtime.go:84
# 0x503919 internal/poll.(*pollDesc).waitRead+0x279 /go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_poll_runtime.go:89
# 0x503907 internal/poll.(*FD).Read+0x267 /go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_unix.go:165
# 0x5bbf44 net.(*netFD).Read+0x24 /go/pkg/mod/golang.org/[email protected]/src/net/fd_posix.go:55
# 0x5cd8a4 net.(*conn).Read+0x44 /go/pkg/mod/golang.org/[email protected]/src/net/net.go:194
# 0x6fed5a crypto/tls.(*atLeastReader).Read+0x3a /go/pkg/mod/golang.org/[email protected]/src/crypto/tls/conn.go:809
# 0x54c1d7 bytes.(*Buffer).ReadFrom+0x97 /go/pkg/mod/golang.org/[email protected]/src/bytes/buffer.go:211
# 0x6fef3d crypto/tls.(*Conn).readFromUntil+0xdd /go/pkg/mod/golang.org/[email protected]/src/crypto/tls/conn.go:831
# 0x6fc04e crypto/tls.(*Conn).readRecordOrCCS+0x3ce /go/pkg/mod/golang.org/[email protected]/src/crypto/tls/conn.go:629
# 0x702824 crypto/tls.(*Conn).readRecord+0x144 /go/pkg/mod/golang.org/[email protected]/src/crypto/tls/conn.go:591
# 0x702825 crypto/tls.(*Conn).Read+0x145 /go/pkg/mod/golang.org/[email protected]/src/crypto/tls/conn.go:1385
# 0x7e8bc6 net/http.(*persistConn).Read+0x46 /go/pkg/mod/golang.org/[email protected]/src/net/http/transport.go:2122
# 0x58f622 bufio.(*Reader).fill+0x102 /go/pkg/mod/golang.org/[email protected]/src/bufio/bufio.go:113
# 0x58f752 bufio.(*Reader).Peek+0x52 /go/pkg/mod/golang.org/[email protected]/src/bufio/bufio.go:152
# 0x7e96d1 net/http.(*persistConn).readLoop+0x171 /go/pkg/mod/golang.org/[email protected]/src/net/http/transport.go:2275
2 @ 0x47b70e 0x4581f7 0x176cafc 0x1776646 0x482f61
# 0x176cafb github.com/prometheus/prometheus/rules.(*Group).run+0x43b /go/pkg/mod/github.com/prometheus/[email protected]/rules/group.go:270
# 0x1776645 github.com/prometheus/prometheus/rules.(*Manager).Update.func1+0xa5 /go/pkg/mod/github.com/prometheus/[email protected]/rules/manager.go:246
2 @ 0x47b70e 0x4581f7 0x1915505 0x482f61
# 0x1915504 github.com/prometheus/prometheus/storage/remote.(*QueueManager).updateShardsLoop+0x104 /go/pkg/mod/github.com/prometheus/[email protected]/storage/remote/queue_manager.go:1062
2 @ 0x47b70e 0x4581f7 0x191644a 0x482f61
# 0x1916449 github.com/prometheus/prometheus/storage/remote.(*QueueManager).reshardLoop+0xe9 /go/pkg/mod/github.com/prometheus/[email protected]/storage/remote/queue_manager.go:1189
2 @ 0x47b70e 0x4581f7 0x191859b 0x482f61
# 0x191859a github.com/prometheus/prometheus/storage/remote.(*shards).runShard+0x51a /go/pkg/mod/github.com/prometheus/[email protected]/storage/remote/queue_manager.go:1556
2 @ 0x47b70e 0x4581f7 0x1934ff5 0x253f874 0x6631c9 0x482f61
# 0x1934ff4 github.com/thanos-io/thanos/pkg/runutil.RepeatInfinitely+0x1f4 /app/pkg/runutil/runutil.go:76
# 0x253f873 main.addDiscoveryGroups.func3+0x93 /app/cmd/thanos/rule.go:990
# 0x6631c8 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
2 @ 0x47b70e 0x4581f7 0x1ee5a45 0x1ee59cf 0x253f91f 0x6631c9 0x482f61
# 0x1ee5a44 github.com/thanos-io/thanos/pkg/clientconfig.(*HTTPClient).Discover.func2+0x204 /app/pkg/clientconfig/http.go:420
# 0x1ee59ce github.com/thanos-io/thanos/pkg/clientconfig.(*HTTPClient).Discover+0x18e /app/pkg/clientconfig/http.go:431
# 0x253f91e main.addDiscoveryGroups.func1+0x1e /app/cmd/thanos/rule.go:983
# 0x6631c8 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
2 @ 0x47b70e 0x4581f7 0xef717c 0xef685a 0xef6125 0x482f61
# 0xef717b github.com/prometheus/prometheus/tsdb/wlog.(*Watcher).watch+0x43b /go/pkg/mod/github.com/prometheus/[email protected]/tsdb/wlog/watcher.go:389
# 0xef6859 github.com/prometheus/prometheus/tsdb/wlog.(*Watcher).Run+0x559 /go/pkg/mod/github.com/prometheus/[email protected]/tsdb/wlog/watcher.go:304
# 0xef6124 github.com/prometheus/prometheus/tsdb/wlog.(*Watcher).loop+0x124 /go/pkg/mod/github.com/prometheus/[email protected]/tsdb/wlog/watcher.go:253
1 @ 0x4180e9 0x47d949 0x662d93 0x482f61
# 0x47d948 os/signal.signal_recv+0x28 /go/pkg/mod/golang.org/[email protected]/src/runtime/sigqueue.go:152
# 0x662d92 os/signal.loop+0x12 /go/pkg/mod/golang.org/[email protected]/src/os/signal/signal_unix.go:23
1 @ 0x43a9d1 0x47a51d 0x1a04071 0x1a03ea5 0x1a00ccb 0x1a1a1cf 0x1a1ac7e 0x7cf0e9 0x7d0fe4 0x7efc2e 0x7cd5e5 0x482f61
# 0x1a04070 runtime/pprof.writeRuntimeProfile+0xb0 /go/pkg/mod/golang.org/[email protected]/src/runtime/pprof/pprof.go:796
# 0x1a03ea4 runtime/pprof.writeGoroutine+0x44 /go/pkg/mod/golang.org/[email protected]/src/runtime/pprof/pprof.go:755
# 0x1a00cca runtime/pprof.(*Profile).WriteTo+0x14a /go/pkg/mod/golang.org/[email protected]/src/runtime/pprof/pprof.go:377
# 0x1a1a1ce net/http/pprof.handler.ServeHTTP+0x52e /go/pkg/mod/golang.org/[email protected]/src/net/http/pprof/pprof.go:272
# 0x1a1ac7d net/http/pprof.Index+0xdd /go/pkg/mod/golang.org/[email protected]/src/net/http/pprof/pprof.go:389
# 0x7cf0e8 net/http.HandlerFunc.ServeHTTP+0x28 /go/pkg/mod/golang.org/[email protected]/src/net/http/server.go:2294
# 0x7d0fe3 net/http.(*ServeMux).ServeHTTP+0x1c3 /go/pkg/mod/golang.org/[email protected]/src/net/http/server.go:2822
# 0x7efc2d net/http.serverHandler.ServeHTTP+0x8d /go/pkg/mod/golang.org/[email protected]/src/net/http/server.go:3301
# 0x7cd5e4 net/http.(*conn).serve+0x624 /go/pkg/mod/golang.org/[email protected]/src/net/http/server.go:2102
1 @ 0x47b70e 0x411d85 0x411912 0x2517c0d 0x6631c9 0x482f61
# 0x2517c0c main.main.func2+0x2c /app/cmd/thanos/main.go:129
# 0x6631c8 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x47b70e 0x411d85 0x411912 0x253d7e5 0x6631c9 0x482f61
# 0x253d7e4 main.runRule.func6+0x44 /app/cmd/thanos/rule.go:646
# 0x6631c8 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x47b70e 0x411d85 0x411912 0x5b9e25 0x5b9146 0x5ccb4f 0x5acdb7 0x5c9a97 0x5abed4 0x482f61
# 0x5b9e24 net.(*Resolver).goLookupIPCNAMEOrder.func4+0x64 /go/pkg/mod/golang.org/[email protected]/src/net/dnsclient_unix.go:673
# 0x5b9145 net.(*Resolver).goLookupIPCNAMEOrder+0xbe5 /go/pkg/mod/golang.org/[email protected]/src/net/dnsclient_unix.go:683
# 0x5ccb4e net.(*Resolver).lookupIP+0xee /go/pkg/mod/golang.org/[email protected]/src/net/lookup_unix.go:68
# 0x5acdb6 net.init.func1+0x36 /go/pkg/mod/golang.org/[email protected]/src/net/hook.go:21
# 0x5c9a96 net.(*Resolver).lookupIPAddr.func1+0x36 /go/pkg/mod/golang.org/[email protected]/src/net/lookup.go:334
# 0x5abed3 internal/singleflight.(*Group).doCall+0x33 /go/pkg/mod/golang.org/[email protected]/src/internal/singleflight/singleflight.go:93
1 @ 0x47b70e 0x411d85 0x411912 0x663085 0x2517638 0x4459eb 0x482f61
# 0x663084 github.com/oklog/run.(*Group).Run+0x144 /go/pkg/mod/github.com/oklog/[email protected]/group.go:43
# 0x2517637 main.main+0x1977 /app/cmd/thanos/main.go:169
# 0x4459ea runtime.main+0x28a /go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:283
1 @ 0x47b70e 0x43e3d7 0x47a925 0x502627 0x50391a 0x503908 0x5bbf45 0x5cd8a5 0x7c7959 0x58f623 0x58f753 0x7cd745 0x482f61
# 0x47a924 internal/poll.runtime_pollWait+0x84 /go/pkg/mod/golang.org/[email protected]/src/runtime/netpoll.go:351
# 0x502626 internal/poll.(*pollDesc).wait+0x26 /go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_poll_runtime.go:84
# 0x503919 internal/poll.(*pollDesc).waitRead+0x279 /go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_poll_runtime.go:89
# 0x503907 internal/poll.(*FD).Read+0x267 /go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_unix.go:165
# 0x5bbf44 net.(*netFD).Read+0x24 /go/pkg/mod/golang.org/[email protected]/src/net/fd_posix.go:55
# 0x5cd8a4 net.(*conn).Read+0x44 /go/pkg/mod/golang.org/[email protected]/src/net/net.go:194
# 0x7c7958 net/http.(*connReader).Read+0x158 /go/pkg/mod/golang.org/[email protected]/src/net/http/server.go:798
# 0x58f622 bufio.(*Reader).fill+0x102 /go/pkg/mod/golang.org/[email protected]/src/bufio/bufio.go:113
# 0x58f752 bufio.(*Reader).Peek+0x52 /go/pkg/mod/golang.org/[email protected]/src/bufio/bufio.go:152
# 0x7cd744 net/http.(*conn).serve+0x784 /go/pkg/mod/golang.org/[email protected]/src/net/http/server.go:2137
1 @ 0x47b70e 0x43e3d7 0x47a925 0x502627 0x5079f5 0x5079e3 0x5bdee9 0x5d72db 0x5d64d0 0x7d2b8c 0x1a502d6 0x1a4f5e7 0x9e7870 0x482f61
# 0x47a924 internal/poll.runtime_pollWait+0x84 /go/pkg/mod/golang.org/[email protected]/src/runtime/netpoll.go:351
# 0x502626 internal/poll.(*pollDesc).wait+0x26 /go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_poll_runtime.go:84
# 0x5079f4 internal/poll.(*pollDesc).waitRead+0x294 /go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_poll_runtime.go:89
# 0x5079e2 internal/poll.(*FD).Accept+0x282 /go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_unix.go:620
# 0x5bdee8 net.(*netFD).accept+0x28 /go/pkg/mod/golang.org/[email protected]/src/net/fd_unix.go:172
# 0x5d72da net.(*TCPListener).accept+0x1a /go/pkg/mod/golang.org/[email protected]/src/net/tcpsock_posix.go:159
# 0x5d64cf net.(*TCPListener).Accept+0x2f /go/pkg/mod/golang.org/[email protected]/src/net/tcpsock.go:380
# 0x7d2b8b net/http.(*Server).Serve+0x30b /go/pkg/mod/golang.org/[email protected]/src/net/http/server.go:3424
# 0x1a502d5 github.com/prometheus/exporter-toolkit/web.Serve+0x755 /go/pkg/mod/github.com/prometheus/[email protected]/web/tls_config.go:351
# 0x1a4f5e6 github.com/prometheus/exporter-toolkit/web.ServeMultiple.func1+0x26 /go/pkg/mod/github.com/prometheus/[email protected]/web/tls_config.go:274
# 0x9e786f golang.org/x/sync/errgroup.(*Group).Go.func1+0x4f /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78
1 @ 0x47b70e 0x43e3d7 0x47a925 0x502627 0x5079f5 0x5079e3 0x5bdee9 0x5d72db 0x5d64d0 0xd963d3 0x203b9aa 0x253c89a 0x6631c9 0x482f61
# 0x47a924 internal/poll.runtime_pollWait+0x84 /go/pkg/mod/golang.org/[email protected]/src/runtime/netpoll.go:351
# 0x502626 internal/poll.(*pollDesc).wait+0x26 /go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_poll_runtime.go:84
# 0x5079f4 internal/poll.(*pollDesc).waitRead+0x294 /go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_poll_runtime.go:89
# 0x5079e2 internal/poll.(*FD).Accept+0x282 /go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_unix.go:620
# 0x5bdee8 net.(*netFD).accept+0x28 /go/pkg/mod/golang.org/[email protected]/src/net/fd_unix.go:172
# 0x5d72da net.(*TCPListener).accept+0x1a /go/pkg/mod/golang.org/[email protected]/src/net/tcpsock_posix.go:159
# 0x5d64cf net.(*TCPListener).Accept+0x2f /go/pkg/mod/golang.org/[email protected]/src/net/tcpsock.go:380
# 0xd963d2 google.golang.org/grpc.(*Server).Serve+0x472 /go/pkg/mod/google.golang.org/[email protected]/server.go:875
# 0x203b9a9 github.com/thanos-io/thanos/pkg/server/grpc.(*Server).ListenAndServe+0x189 /app/pkg/server/grpc/grpc.go:168
# 0x253c899 main.runRule.func14+0x39 /app/cmd/thanos/rule.go:762
# 0x6631c8 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x47b70e 0x4581f7 0x1003a25 0x482f61
# 0x1003a24 github.com/baidubce/bce-sdk-go/util/log.NewLogger.func1+0xa4 /go/pkg/mod/github.com/baidubce/[email protected]/util/log/logger.go:375
1 @ 0x47b70e 0x4581f7 0x17c6625 0x482f61
# 0x17c6624 github.com/prometheus/prometheus/util/logging.(*Deduper).run+0xe4 /go/pkg/mod/github.com/prometheus/[email protected]/util/logging/dedupe.go:122
1 @ 0x47b70e 0x4581f7 0x19245fe 0x482f61
# 0x19245fd github.com/prometheus/prometheus/storage/remote.(*WriteStorage).run+0xbd /go/pkg/mod/github.com/prometheus/[email protected]/storage/remote/write.go:119
1 @ 0x47b70e 0x4581f7 0x1ac227f 0x482f61
# 0x1ac227e go.opencensus.io/stats/view.(*worker).start+0x9e /go/pkg/mod/[email protected]/stats/view/worker.go:292
1 @ 0x47b70e 0x4581f7 0x24a0672 0x482f61
# 0x24a0671 github.com/prometheus/prometheus/tsdb/agent.(*DB).run+0xd1 /go/pkg/mod/github.com/prometheus/[email protected]/tsdb/agent/db.go:594
1 @ 0x47b70e 0x4581f7 0x24a8fea 0x253d6a5 0x1629489 0x253d605 0x6631c9 0x482f61
# 0x24a8fe9 github.com/thanos-io/thanos/pkg/alert.(*Queue).Pop+0x89 /app/pkg/alert/alert.go:118
# 0x253d6a4 main.runRule.func8.1+0x44 /app/cmd/thanos/rule.go:663
# 0x1629488 github.com/thanos-io/thanos/pkg/tracing.DoInSpan+0x88 /app/pkg/tracing/tracing.go:95
# 0x253d604 main.runRule.func8+0x84 /app/cmd/thanos/rule.go:662
# 0x6631c8 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x47b70e 0x4581f7 0x251804e 0x2517a3f 0x6631c9 0x482f61
# 0x251804d main.interrupt+0xed /app/cmd/thanos/main.go:180
# 0x2517a3e main.main.func4+0x1e /app/cmd/thanos/main.go:153
# 0x6631c8 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x47b70e 0x4581f7 0x2518305 0x25179c5 0x6631c9 0x482f61
# 0x2518304 main.reload+0xe4 /app/cmd/thanos/main.go:193
# 0x25179c4 main.main.func6+0x24 /app/cmd/thanos/main.go:163
# 0x6631c8 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x47b70e 0x4581f7 0x253d1c5 0x6631c9 0x482f61
# 0x253d1c4 main.runRule.func10+0x284 /app/cmd/thanos/rule.go:688
# 0x6631c8 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x47b70e 0x4581f7 0x5c91d9 0x5c6c65 0x5afd05 0x5b0d28 0xa0d849 0xa0d60c 0x7e3372 0x7e6345 0x7e4e58 0x7e4c95 0x482f61
# 0x5c91d8 net.(*Resolver).lookupIPAddr+0x3d8 /go/pkg/mod/golang.org/[email protected]/src/net/lookup.go:342
# 0x5c6c64 net.(*Resolver).internetAddrList+0x4c4 /go/pkg/mod/golang.org/[email protected]/src/net/ipsock.go:289
# 0x5afd04 net.(*Resolver).resolveAddrList+0x3e4 /go/pkg/mod/golang.org/[email protected]/src/net/dial.go:353
# 0x5b0d27 net.(*Dialer).DialContext+0x407 /go/pkg/mod/golang.org/[email protected]/src/net/dial.go:560
# 0xa0d848 github.com/mwitkow/go-conntrack.dialClientConnTracker+0x1e8 /go/pkg/mod/github.com/mwitkow/[email protected]/dialer_wrapper.go:123
# 0xa0d60b github.com/mwitkow/go-conntrack.NewDialContextFunc.func1+0xeb /go/pkg/mod/github.com/mwitkow/[email protected]/dialer_wrapper.go:94
# 0x7e3371 net/http.(*Transport).dial+0xd1 /go/pkg/mod/golang.org/[email protected]/src/net/http/transport.go:1278
# 0x7e6344 net/http.(*Transport).dialConn+0x7e4 /go/pkg/mod/golang.org/[email protected]/src/net/http/transport.go:1780
# 0x7e4e57 net/http.(*Transport).dialConnFor+0xb7 /go/pkg/mod/golang.org/[email protected]/src/net/http/transport.go:1615
# 0x7e4c94 net/http.(*Transport).startDialConnForLocked.func1+0x34 /go/pkg/mod/golang.org/[email protected]/src/net/http/transport.go:1597
1 @ 0x47b70e 0x4581f7 0x7e432e 0x7e0057 0x7ef393 0x15d0ee3 0x15d0599 0x15e9888 0x15d0599 0x15d091c 0x15d0599 0x15e9128 0x15d0599 0x1ee4e24 0x1628cd2 0x77b0c2 0x77a971 0x77cf89 0x1ee55f7 0x1ee55ef 0x1f0c430 0x1f0fb2c 0x1f10765 0x253e885 0x1768738 0x1770137 0x176f7ba 0x1775285 0x176cfbc 0x1776646 0x482f61
# 0x7e432d net/http.(*Transport).getConn+0x4ad /go/pkg/mod/golang.org/[email protected]/src/net/http/transport.go:1520
# 0x7e0056 net/http.(*Transport).roundTrip+0xb56 /go/pkg/mod/golang.org/[email protected]/src/net/http/transport.go:685
# 0x7ef392 net/http.(*Transport).RoundTrip+0x12 /go/pkg/mod/golang.org/[email protected]/src/net/http/roundtrip.go:30
# 0x15d0ee2 github.com/prometheus/client_golang/prometheus/promhttp.InstrumentRoundTripperDuration.func1+0xa2 /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:119
# 0x15d0598 github.com/prometheus/client_golang/prometheus/promhttp.RoundTripperFunc.RoundTrip+0x18 /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:32
# 0x15e9887 github.com/thanos-io/thanos/pkg/extprom/http.InstrumentedRoundTripper.InstrumentRoundTripperTrace.func5+0x687 /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:247
# 0x15d0598 github.com/prometheus/client_golang/prometheus/promhttp.RoundTripperFunc.RoundTrip+0x18 /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:32
# 0x15d091b github.com/prometheus/client_golang/prometheus/promhttp.InstrumentRoundTripperCounter.func1+0x5b /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:75
# 0x15d0598 github.com/prometheus/client_golang/prometheus/promhttp.RoundTripperFunc.RoundTrip+0x18 /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:32
# 0x15e9127 github.com/thanos-io/thanos/pkg/extprom/http.InstrumentedRoundTripper.InstrumentRoundTripperInFlight.func6+0xa7 /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:44
# 0x15d0598 github.com/prometheus/client_golang/prometheus/promhttp.RoundTripperFunc.RoundTrip+0x18 /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:32
# 0x1ee4e23 github.com/thanos-io/thanos/pkg/clientconfig.userAgentRoundTripper.RoundTrip+0x163 /app/pkg/clientconfig/http.go:299
# 0x1628cd1 github.com/thanos-io/thanos/pkg/tracing.(*tripperware).RoundTrip+0x471 /app/pkg/tracing/http.go:112
# 0x77b0c1 net/http.send+0x5e1 /go/pkg/mod/golang.org/[email protected]/src/net/http/client.go:259
# 0x77a970 net/http.(*Client).send+0x90 /go/pkg/mod/golang.org/[email protected]/src/net/http/client.go:180
# 0x77cf88 net/http.(*Client).do+0x988 /go/pkg/mod/golang.org/[email protected]/src/net/http/client.go:728
# 0x1ee55f6 net/http.(*Client).Do+0x16 /go/pkg/mod/golang.org/[email protected]/src/net/http/client.go:587
# 0x1ee55ee github.com/thanos-io/thanos/pkg/clientconfig.(*HTTPClient).Do+0xe /app/pkg/clientconfig/http.go:387
# 0x1f0c42f github.com/thanos-io/thanos/pkg/promclient.(*Client).req2xx+0x4cf /app/pkg/promclient/promclient.go:132
# 0x1f0fb2b github.com/thanos-io/thanos/pkg/promclient.(*Client).QueryInstant+0x5eb /app/pkg/promclient/promclient.go:450
# 0x1f10764 github.com/thanos-io/thanos/pkg/promclient.(*Client).PromqlQueryInstant+0x84 /app/pkg/promclient/promclient.go:503
# 0x253e884 main.runRule.queryFuncCreator.func24.1+0xdc4 /app/cmd/thanos/rule.go:917
# 0x1768737 github.com/prometheus/prometheus/rules.(*AlertingRule).Eval+0x1f7 /go/pkg/mod/github.com/prometheus/[email protected]/rules/alerting.go:343
# 0x1770136 github.com/prometheus/prometheus/rules.(*Group).Eval.func1+0x776 /go/pkg/mod/github.com/prometheus/[email protected]/rules/group.go:540
# 0x176f7b9 github.com/prometheus/prometheus/rules.(*Group).Eval+0x379 /go/pkg/mod/github.com/prometheus/[email protected]/rules/group.go:665
# 0x1775284 github.com/prometheus/prometheus/rules.DefaultEvalIterationFunc+0xe4 /go/pkg/mod/github.com/prometheus/[email protected]/rules/manager.go:81
# 0x176cfbb github.com/prometheus/prometheus/rules.(*Group).run+0x8fb /go/pkg/mod/github.com/prometheus/[email protected]/rules/group.go:307
# 0x1776645 github.com/prometheus/prometheus/rules.(*Manager).Update.func1+0xa5 /go/pkg/mod/github.com/prometheus/[email protected]/rules/manager.go:246
1 @ 0x47b70e 0x4581f7 0x7ebec5 0x7e009a 0x7ef393 0x15d0ee3 0x15d0599 0x15e9888 0x15d0599 0x15d091c 0x15d0599 0x15e9128 0x15d0599 0x1ee4e24 0x1628cd2 0x77b0c2 0x77a971 0x77cf89 0x1ee55f7 0x1ee55ef 0x1f0c430 0x1f0fb2c 0x1f10765 0x253e885 0x17783ff 0x1770137 0x176f7ba 0x1775285 0x176cfbc 0x1776646 0x482f61
# 0x7ebec4 net/http.(*persistConn).roundTrip+0x784 /go/pkg/mod/golang.org/[email protected]/src/net/http/transport.go:2826
# 0x7e0099 net/http.(*Transport).roundTrip+0xb99 /go/pkg/mod/golang.org/[email protected]/src/net/http/transport.go:696
# 0x7ef392 net/http.(*Transport).RoundTrip+0x12 /go/pkg/mod/golang.org/[email protected]/src/net/http/roundtrip.go:30
# 0x15d0ee2 github.com/prometheus/client_golang/prometheus/promhttp.InstrumentRoundTripperDuration.func1+0xa2 /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:119
# 0x15d0598 github.com/prometheus/client_golang/prometheus/promhttp.RoundTripperFunc.RoundTrip+0x18 /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:32
# 0x15e9887 github.com/thanos-io/thanos/pkg/extprom/http.InstrumentedRoundTripper.InstrumentRoundTripperTrace.func5+0x687 /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:247
# 0x15d0598 github.com/prometheus/client_golang/prometheus/promhttp.RoundTripperFunc.RoundTrip+0x18 /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:32
# 0x15d091b github.com/prometheus/client_golang/prometheus/promhttp.InstrumentRoundTripperCounter.func1+0x5b /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:75
# 0x15d0598 github.com/prometheus/client_golang/prometheus/promhttp.RoundTripperFunc.RoundTrip+0x18 /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:32
# 0x15e9127 github.com/thanos-io/thanos/pkg/extprom/http.InstrumentedRoundTripper.InstrumentRoundTripperInFlight.func6+0xa7 /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:44
# 0x15d0598 github.com/prometheus/client_golang/prometheus/promhttp.RoundTripperFunc.RoundTrip+0x18 /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:32
# 0x1ee4e23 github.com/thanos-io/thanos/pkg/clientconfig.userAgentRoundTripper.RoundTrip+0x163 /app/pkg/clientconfig/http.go:299
# 0x1628cd1 github.com/thanos-io/thanos/pkg/tracing.(*tripperware).RoundTrip+0x471 /app/pkg/tracing/http.go:112
# 0x77b0c1 net/http.send+0x5e1 /go/pkg/mod/golang.org/[email protected]/src/net/http/client.go:259
# 0x77a970 net/http.(*Client).send+0x90 /go/pkg/mod/golang.org/[email protected]/src/net/http/client.go:180
# 0x77cf88 net/http.(*Client).do+0x988 /go/pkg/mod/golang.org/[email protected]/src/net/http/client.go:728
# 0x1ee55f6 net/http.(*Client).Do+0x16 /go/pkg/mod/golang.org/[email protected]/src/net/http/client.go:587
# 0x1ee55ee github.com/thanos-io/thanos/pkg/clientconfig.(*HTTPClient).Do+0xe /app/pkg/clientconfig/http.go:387
# 0x1f0c42f github.com/thanos-io/thanos/pkg/promclient.(*Client).req2xx+0x4cf /app/pkg/promclient/promclient.go:132
# 0x1f0fb2b github.com/thanos-io/thanos/pkg/promclient.(*Client).QueryInstant+0x5eb /app/pkg/promclient/promclient.go:450
# 0x1f10764 github.com/thanos-io/thanos/pkg/promclient.(*Client).PromqlQueryInstant+0x84 /app/pkg/promclient/promclient.go:503
# 0x253e884 main.runRule.queryFuncCreator.func24.1+0xdc4 /app/cmd/thanos/rule.go:917
# 0x17783fe github.com/prometheus/prometheus/rules.(*RecordingRule).Eval+0x19e /go/pkg/mod/github.com/prometheus/[email protected]/rules/recording.go:83
# 0x1770136 github.com/prometheus/prometheus/rules.(*Group).Eval.func1+0x776 /go/pkg/mod/github.com/prometheus/[email protected]/rules/group.go:540
# 0x176f7b9 github.com/prometheus/prometheus/rules.(*Group).Eval+0x379 /go/pkg/mod/github.com/prometheus/[email protected]/rules/group.go:665
# 0x1775284 github.com/prometheus/prometheus/rules.DefaultEvalIterationFunc+0xe4 /go/pkg/mod/github.com/prometheus/[email protected]/rules/manager.go:81
# 0x176cfbb github.com/prometheus/prometheus/rules.(*Group).run+0x8fb /go/pkg/mod/github.com/prometheus/[email protected]/rules/group.go:307
# 0x1776645 github.com/prometheus/prometheus/rules.(*Manager).Update.func1+0xa5 /go/pkg/mod/github.com/prometheus/[email protected]/rules/manager.go:246
1 @ 0x47b70e 0x4581f7 0xefb876 0x482f61
# 0xefb875 github.com/prometheus/prometheus/tsdb/wlog.(*WL).run+0x95 /go/pkg/mod/github.com/prometheus/[email protected]/tsdb/wlog/wlog.go:416
1 @ 0x47b70e 0x45929d 0x459274 0x47ce65 0x48ea68 0x9e771e 0x1a4f557 0x1a4fa12 0x1a53de5 0x253c53a 0x6631c9 0x482f61
# 0x47ce64 sync.runtime_SemacquireWaitGroup+0x24 /go/pkg/mod/golang.org/[email protected]/src/runtime/sema.go:110
# 0x48ea67 sync.(*WaitGroup).Wait+0x47 /go/pkg/mod/golang.org/[email protected]/src/sync/waitgroup.go:118
# 0x9e771d golang.org/x/sync/errgroup.(*Group).Wait+0x1d /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:56
# 0x1a4f556 github.com/prometheus/exporter-toolkit/web.ServeMultiple+0x116 /go/pkg/mod/github.com/prometheus/[email protected]/web/tls_config.go:277
# 0x1a4fa11 github.com/prometheus/exporter-toolkit/web.ListenAndServe+0x411 /go/pkg/mod/github.com/prometheus/[email protected]/web/tls_config.go:325
# 0x1a53de4 github.com/thanos-io/thanos/pkg/server/http.(*Server).ListenAndServe+0x244 /app/pkg/server/http/http.go:86
# 0x253c539 main.runRule.func18+0x39 /app/cmd/thanos/rule.go:812
# 0x6631c8 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x482f61
1 @ 0x4a18c5 0x49f9d8 0x50394e 0x503936 0x5037d1 0x5bbf45 0x5cd8a5 0x7c74b7 0x482f61
# 0x4a18c4 syscall.Syscall+0x24 /go/pkg/mod/golang.org/[email protected]/src/syscall/syscall_linux.go:73
# 0x49f9d7 syscall.read+0x37 /go/pkg/mod/golang.org/[email protected]/src/syscall/zsyscall_linux_amd64.go:736
# 0x50394d syscall.Read+0x2ad /go/pkg/mod/golang.org/[email protected]/src/syscall/syscall_unix.go:183
# 0x503935 internal/poll.ignoringEINTRIO+0x295 /go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_unix.go:745
# 0x5037d0 internal/poll.(*FD).Read+0x130 /go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_unix.go:161
# 0x5bbf44 net.(*netFD).Read+0x24 /go/pkg/mod/golang.org/[email protected]/src/net/fd_posix.go:55
# 0x5cd8a4 net.(*conn).Read+0x44 /go/pkg/mod/golang.org/[email protected]/src/net/net.go:194
# 0x7c74b6 net/http.(*connReader).backgroundRead+0x36 /go/pkg/mod/golang.org/[email protected]/src/net/http/server.go:690
1 @ 0x5a8626 0x5b4550 0x5b561b 0x5b6c26 0x5ba105 0x482f61
# 0x5a8625 vendor/golang.org/x/net/dns/dnsmessage.NewBuilder+0x265 /go/pkg/mod/golang.org/[email protected]/src/vendor/golang.org/x/net/dns/dnsmessage/message.go:1244
# 0x5b454f net.newRequest+0x8f /go/pkg/mod/golang.org/[email protected]/src/net/dnsclient_unix.go:61
# 0x5b561a net.(*Resolver).exchange+0xda /go/pkg/mod/golang.org/[email protected]/src/net/dnsclient_unix.go:171
# 0x5b6c25 net.(*Resolver).tryOneName+0x4c5 /go/pkg/mod/golang.org/[email protected]/src/net/dnsclient_unix.go:316
# 0x5ba104 net.(*Resolver).goLookupIPCNAMEOrder.func3.1+0x84 /go/pkg/mod/golang.org/[email protected]/src/net/dnsclient_unix.go:667
Ran at time N + (2 minutes)
goroutine profile: total 909
859 @ 0x47b70e 0x4581f7 0x176d06a 0x1776646 0x482f61
# 0x176d069 github.com/prometheus/prometheus/rules.(*Group).run+0x9a9 /go/pkg/mod/github.com/prometheus/[email protected]/rules/group.go:296
# 0x1776645 github.com/prometheus/prometheus/rules.(*Manager).Update.func1+0xa5 /go/pkg/mod/github.com/prometheus/[email protected]/rules/manager.go:246
6 @ 0x47b70e 0x4581f7 0x7eafe7 0x482f61
# 0x7eafe6 net/http.(*persistConn).writeLoop+0xe6 /go/pkg/mod/golang.org/[email protected]/src/net/http/transport.go:2590
4 @ 0x47b70e 0x43e3d7 0x47a925 0x502627 0x50391a 0x503908 0x5bbf45 0x5cd8a5 0x7e8bc7 0x58f623 0x58f753 0x7e96d2 0x482f61
# 0x47a924 internal/poll.runtime_pollWait+0x84 /go/pkg/mod/golang.org/[email protected]/src/runtime/netpoll.go:351
# 0x502626 internal/poll.(*pollDesc).wait+0x26 /go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_poll_runtime.go:84
# 0x503919 internal/poll.(*pollDesc).waitRead+0x279 /go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_poll_runtime.go:89
# 0x503907 internal/poll.(*FD).Read+0x267 /go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_unix.go:165
# 0x5bbf44 net.(*netFD).Read+0x24 /go/pkg/mod/golang.org/[email protected]/src/net/fd_posix.go:55
# 0x5cd8a4 net.(*conn).Read+0x44 /go/pkg/mod/golang.org/[email protected]/src/net/net.go:194
# 0x7e8bc6 net/http.(*persistConn).Read+0x46 /go/pkg/mod/golang.org/[email protected]/src/net/http/transport.go:2122
# 0x58f622 bufio.(*Reader).fill+0x102 /go/pkg/mod/golang.org/[email protected]/src/bufio/bufio.go:113
# 0x58f752 bufio.(*Reader).Peek+0x52 /go/pkg/mod/golang.org/[email protected]/src/bufio/bufio.go:152
# 0x7e96d1 net/http.(*persistConn).readLoop+0x171 /go/pkg/mod/golang.org/[email protected]/src/net/http/transport.go:2275
2 @ 0x47b70e 0x411d85 0x411912 0x1f800ae 0x482f61
# 0x1f800ad github.com/prometheus/prometheus/rules.(*Manager).Run+0x6d /go/pkg/mod/github.com/prometheus/[email protected]/rules/manager.go:171
2 @ 0x47b70e 0x43e3d7 0x47a925 0x502627 0x50391a 0x503908 0x5bbf45 0x5cd8a5 0x6fed5b 0x54c1d8 0x6fef3e 0x6fc04f 0x702825 0x702826 0x7e8bc7 0x58f623 0x58f753 0x7e96d2 0x482f61
# 0x47a924 internal/poll.runtime_pollWait+0x84 /go/pkg/mod/golang.org/[email protected]/src/runtime/netpoll.go:351
# 0x502626 internal/poll.(*pollDesc).wait+0x26 /go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_poll_runtime.go:84
# 0x503919 internal/poll.(*pollDesc).waitRead+0x279 /go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_poll_runtime.go:89
# 0x503907 internal/poll.(*FD).Read+0x267 /go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_unix.go:165
# 0x5bbf44 net.(*netFD).Read+0x24 /go/pkg/mod/golang.org/[email protected]/src/net/fd_posix.go:55
# 0x5cd8a4 net.(*conn).Read+0x44 /go/pkg/mod/golang.org/[email protected]/src/net/net.go:194
# 0x6fed5a crypto/tls.(*atLeastReader).Read+0x3a /go/pkg/mod/golang.org/[email protected]/src/crypto/tls/conn.go:809
# 0x54c1d7 bytes.(*Buffer).ReadFrom+0x97 /go/pkg/mod/golang.org/[email protected]/src/bytes/buffer.go:211
# 0x6fef3d crypto/tls.(*Conn).readFromUntil+0xdd /go/pkg/mod/golang.org/[email protected]/src/crypto/tls/conn.go:831
# 0x6fc04e crypto/tls.(*Conn).readRecordOrCCS+0x3ce /go/pkg/mod/golang.org/[email protected]/src/crypto/tls/conn.go:629
# 0x702824 crypto/tls.(*Conn).readRecord+0x144 /go/pkg/mod/golang.org/[email protected]/src/crypto/tls/conn.go:591
# 0x702825 crypto/tls.(*Conn).Read+0x145 /go/pkg/mod/golang.org/[email protected]/src/crypto/tls/conn.go:1385
# 0x7e8bc6 net/http.(*persistConn).Read+0x46 /go/pkg/mod/golang.org/[email protected]/src/net/http/transport.go:2122
# 0x58f622 bufio.(*Reader).fill+0x102 /go/pkg/mod/golang.org/[email protected]/src/bufio/bufio.go:113
# 0x58f752 bufio.(*Reader).Peek+0x52 /go/pkg/mod/golang.org/[email protected]/src/bufio/bufio.go:152
# 0x7e96d1 net/http.(*persistConn).readLoop+0x171 /go/pkg/mod/golang.org/[email protected]/src/net/http/transport.go:2275
2 @ 0x47b70e 0x4581f7 0x176cafc 0x1776646 0x482f61
# 0x176cafb github.com/prometheus/prometheus/rules.(*Group).run+0x43b /go/pkg/mod/github.com/prometheus/[email protected]/rules/group.go:270
# 0x1776645 github.com/prometheus/prometheus/rules.(*Manager).Update.func1+0xa5 /go/pkg/mod/github.com/prometheus/[email protected]/rules/manager.go:246
2 @ 0x47b70e 0x4581f7 0x1915505 0x482f61
# 0x1915504 github.com/prometheus/prometheus/storage/remote.(*QueueManager).updateShardsLoop+0x104 /go/pkg/mod/github.com/prometheus/[email protected]/storage/remote/queue_manager.go:1062
2 @ 0x47b70e 0x4581f7 0x191644a 0x482f61
# 0x1916449 github.com/prometheus/prometheus/storage/remote.(*QueueManager).reshardLoop+0xe9 /go/pkg/mod/github.com/prometheus/[email protected]/storage/remote/queue_manager.go:1189
2 @ 0x47b70e 0x4581f7 0x191859b 0x482f61
# 0x191859a github.com/prometheus/prometheus/storage/remote.(*shards).runShard+0x51a /go/pkg/mod/github.com/prometheus/[email protected]/storage/remote/queue_manager.go:1556
2 @ 0x47b70e 0x4581f7 0x1934ff5 0x253f874 0x6631c9 0x482f61
# 0x1934ff4 github.com/thanos-io/thanos/pkg/runutil.RepeatInfinitely+0x1f4 /app/pkg/runutil/runutil.go:76
# 0x253f873 main.addDiscoveryGroups.func3+0x93 /app/cmd/thanos/rule.go:990
# 0x6631c8 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
2 @ 0x47b70e 0x4581f7 0x1ee5a45 0x1ee59cf 0x253f91f 0x6631c9 0x482f61
# 0x1ee5a44 github.com/thanos-io/thanos/pkg/clientconfig.(*HTTPClient).Discover.func2+0x204 /app/pkg/clientconfig/http.go:420
# 0x1ee59ce github.com/thanos-io/thanos/pkg/clientconfig.(*HTTPClient).Discover+0x18e /app/pkg/clientconfig/http.go:431
# 0x253f91e main.addDiscoveryGroups.func1+0x1e /app/cmd/thanos/rule.go:983
# 0x6631c8 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
2 @ 0x47b70e 0x4581f7 0xef717c 0xef685a 0xef6125 0x482f61
# 0xef717b github.com/prometheus/prometheus/tsdb/wlog.(*Watcher).watch+0x43b /go/pkg/mod/github.com/prometheus/[email protected]/tsdb/wlog/watcher.go:389
# 0xef6859 github.com/prometheus/prometheus/tsdb/wlog.(*Watcher).Run+0x559 /go/pkg/mod/github.com/prometheus/[email protected]/tsdb/wlog/watcher.go:304
# 0xef6124 github.com/prometheus/prometheus/tsdb/wlog.(*Watcher).loop+0x124 /go/pkg/mod/github.com/prometheus/[email protected]/tsdb/wlog/watcher.go:253
1 @ 0x4180e9 0x47d949 0x662d93 0x482f61
# 0x47d948 os/signal.signal_recv+0x28 /go/pkg/mod/golang.org/[email protected]/src/runtime/sigqueue.go:152
# 0x662d92 os/signal.loop+0x12 /go/pkg/mod/golang.org/[email protected]/src/os/signal/signal_unix.go:23
1 @ 0x43a9d1 0x47a51d 0x1a04071 0x1a03ea5 0x1a00ccb 0x1a1a1cf 0x1a1ac7e 0x7cf0e9 0x7d0fe4 0x7efc2e 0x7cd5e5 0x482f61
# 0x1a04070 runtime/pprof.writeRuntimeProfile+0xb0 /go/pkg/mod/golang.org/[email protected]/src/runtime/pprof/pprof.go:796
# 0x1a03ea4 runtime/pprof.writeGoroutine+0x44 /go/pkg/mod/golang.org/[email protected]/src/runtime/pprof/pprof.go:755
# 0x1a00cca runtime/pprof.(*Profile).WriteTo+0x14a /go/pkg/mod/golang.org/[email protected]/src/runtime/pprof/pprof.go:377
# 0x1a1a1ce net/http/pprof.handler.ServeHTTP+0x52e /go/pkg/mod/golang.org/[email protected]/src/net/http/pprof/pprof.go:272
# 0x1a1ac7d net/http/pprof.Index+0xdd /go/pkg/mod/golang.org/[email protected]/src/net/http/pprof/pprof.go:389
# 0x7cf0e8 net/http.HandlerFunc.ServeHTTP+0x28 /go/pkg/mod/golang.org/[email protected]/src/net/http/server.go:2294
# 0x7d0fe3 net/http.(*ServeMux).ServeHTTP+0x1c3 /go/pkg/mod/golang.org/[email protected]/src/net/http/server.go:2822
# 0x7efc2d net/http.serverHandler.ServeHTTP+0x8d /go/pkg/mod/golang.org/[email protected]/src/net/http/server.go:3301
# 0x7cd5e4 net/http.(*conn).serve+0x624 /go/pkg/mod/golang.org/[email protected]/src/net/http/server.go:2102
1 @ 0x47b70e 0x411d85 0x411912 0x2517c0d 0x6631c9 0x482f61
# 0x2517c0c main.main.func2+0x2c /app/cmd/thanos/main.go:129
# 0x6631c8 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x47b70e 0x411d85 0x411912 0x253d7e5 0x6631c9 0x482f61
# 0x253d7e4 main.runRule.func6+0x44 /app/cmd/thanos/rule.go:646
# 0x6631c8 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x47b70e 0x411d85 0x411912 0x663085 0x2517638 0x4459eb 0x482f61
# 0x663084 github.com/oklog/run.(*Group).Run+0x144 /go/pkg/mod/github.com/oklog/[email protected]/group.go:43
# 0x2517637 main.main+0x1977 /app/cmd/thanos/main.go:169
# 0x4459ea runtime.main+0x28a /go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:283
1 @ 0x47b70e 0x43e3d7 0x47a925 0x502627 0x50391a 0x503908 0x5bbf45 0x5cd8a5 0x7c7959 0x58f623 0x58f753 0x7cd745 0x482f61
# 0x47a924 internal/poll.runtime_pollWait+0x84 /go/pkg/mod/golang.org/[email protected]/src/runtime/netpoll.go:351
# 0x502626 internal/poll.(*pollDesc).wait+0x26 /go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_poll_runtime.go:84
# 0x503919 internal/poll.(*pollDesc).waitRead+0x279 /go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_poll_runtime.go:89
# 0x503907 internal/poll.(*FD).Read+0x267 /go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_unix.go:165
# 0x5bbf44 net.(*netFD).Read+0x24 /go/pkg/mod/golang.org/[email protected]/src/net/fd_posix.go:55
# 0x5cd8a4 net.(*conn).Read+0x44 /go/pkg/mod/golang.org/[email protected]/src/net/net.go:194
# 0x7c7958 net/http.(*connReader).Read+0x158 /go/pkg/mod/golang.org/[email protected]/src/net/http/server.go:798
# 0x58f622 bufio.(*Reader).fill+0x102 /go/pkg/mod/golang.org/[email protected]/src/bufio/bufio.go:113
# 0x58f752 bufio.(*Reader).Peek+0x52 /go/pkg/mod/golang.org/[email protected]/src/bufio/bufio.go:152
# 0x7cd744 net/http.(*conn).serve+0x784 /go/pkg/mod/golang.org/[email protected]/src/net/http/server.go:2137
1 @ 0x47b70e 0x43e3d7 0x47a925 0x502627 0x5079f5 0x5079e3 0x5bdee9 0x5d72db 0x5d64d0 0x7d2b8c 0x1a502d6 0x1a4f5e7 0x9e7870 0x482f61
# 0x47a924 internal/poll.runtime_pollWait+0x84 /go/pkg/mod/golang.org/[email protected]/src/runtime/netpoll.go:351
# 0x502626 internal/poll.(*pollDesc).wait+0x26 /go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_poll_runtime.go:84
# 0x5079f4 internal/poll.(*pollDesc).waitRead+0x294 /go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_poll_runtime.go:89
# 0x5079e2 internal/poll.(*FD).Accept+0x282 /go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_unix.go:620
# 0x5bdee8 net.(*netFD).accept+0x28 /go/pkg/mod/golang.org/[email protected]/src/net/fd_unix.go:172
# 0x5d72da net.(*TCPListener).accept+0x1a /go/pkg/mod/golang.org/[email protected]/src/net/tcpsock_posix.go:159
# 0x5d64cf net.(*TCPListener).Accept+0x2f /go/pkg/mod/golang.org/[email protected]/src/net/tcpsock.go:380
# 0x7d2b8b net/http.(*Server).Serve+0x30b /go/pkg/mod/golang.org/[email protected]/src/net/http/server.go:3424
# 0x1a502d5 github.com/prometheus/exporter-toolkit/web.Serve+0x755 /go/pkg/mod/github.com/prometheus/[email protected]/web/tls_config.go:351
# 0x1a4f5e6 github.com/prometheus/exporter-toolkit/web.ServeMultiple.func1+0x26 /go/pkg/mod/github.com/prometheus/[email protected]/web/tls_config.go:274
# 0x9e786f golang.org/x/sync/errgroup.(*Group).Go.func1+0x4f /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78
1 @ 0x47b70e 0x43e3d7 0x47a925 0x502627 0x5079f5 0x5079e3 0x5bdee9 0x5d72db 0x5d64d0 0xd963d3 0x203b9aa 0x253c89a 0x6631c9 0x482f61
# 0x47a924 internal/poll.runtime_pollWait+0x84 /go/pkg/mod/golang.org/[email protected]/src/runtime/netpoll.go:351
# 0x502626 internal/poll.(*pollDesc).wait+0x26 /go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_poll_runtime.go:84
# 0x5079f4 internal/poll.(*pollDesc).waitRead+0x294 /go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_poll_runtime.go:89
# 0x5079e2 internal/poll.(*FD).Accept+0x282 /go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_unix.go:620
# 0x5bdee8 net.(*netFD).accept+0x28 /go/pkg/mod/golang.org/[email protected]/src/net/fd_unix.go:172
# 0x5d72da net.(*TCPListener).accept+0x1a /go/pkg/mod/golang.org/[email protected]/src/net/tcpsock_posix.go:159
# 0x5d64cf net.(*TCPListener).Accept+0x2f /go/pkg/mod/golang.org/[email protected]/src/net/tcpsock.go:380
# 0xd963d2 google.golang.org/grpc.(*Server).Serve+0x472 /go/pkg/mod/google.golang.org/[email protected]/server.go:875
# 0x203b9a9 github.com/thanos-io/thanos/pkg/server/grpc.(*Server).ListenAndServe+0x189 /app/pkg/server/grpc/grpc.go:168
# 0x253c899 main.runRule.func14+0x39 /app/cmd/thanos/rule.go:762
# 0x6631c8 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x47b70e 0x4581f7 0x1003a25 0x482f61
# 0x1003a24 github.com/baidubce/bce-sdk-go/util/log.NewLogger.func1+0xa4 /go/pkg/mod/github.com/baidubce/[email protected]/util/log/logger.go:375
1 @ 0x47b70e 0x4581f7 0x17c6625 0x482f61
# 0x17c6624 github.com/prometheus/prometheus/util/logging.(*Deduper).run+0xe4 /go/pkg/mod/github.com/prometheus/[email protected]/util/logging/dedupe.go:122
1 @ 0x47b70e 0x4581f7 0x19245fe 0x482f61
# 0x19245fd github.com/prometheus/prometheus/storage/remote.(*WriteStorage).run+0xbd /go/pkg/mod/github.com/prometheus/[email protected]/storage/remote/write.go:119
1 @ 0x47b70e 0x4581f7 0x1ac227f 0x482f61
# 0x1ac227e go.opencensus.io/stats/view.(*worker).start+0x9e /go/pkg/mod/[email protected]/stats/view/worker.go:292
1 @ 0x47b70e 0x4581f7 0x24a0672 0x482f61
# 0x24a0671 github.com/prometheus/prometheus/tsdb/agent.(*DB).run+0xd1 /go/pkg/mod/github.com/prometheus/[email protected]/tsdb/agent/db.go:594
1 @ 0x47b70e 0x4581f7 0x24a8fea 0x253d6a5 0x1629489 0x253d605 0x6631c9 0x482f61
# 0x24a8fe9 github.com/thanos-io/thanos/pkg/alert.(*Queue).Pop+0x89 /app/pkg/alert/alert.go:118
# 0x253d6a4 main.runRule.func8.1+0x44 /app/cmd/thanos/rule.go:663
# 0x1629488 github.com/thanos-io/thanos/pkg/tracing.DoInSpan+0x88 /app/pkg/tracing/tracing.go:95
# 0x253d604 main.runRule.func8+0x84 /app/cmd/thanos/rule.go:662
# 0x6631c8 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x47b70e 0x4581f7 0x251804e 0x2517a3f 0x6631c9 0x482f61
# 0x251804d main.interrupt+0xed /app/cmd/thanos/main.go:180
# 0x2517a3e main.main.func4+0x1e /app/cmd/thanos/main.go:153
# 0x6631c8 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x47b70e 0x4581f7 0x2518305 0x25179c5 0x6631c9 0x482f61
# 0x2518304 main.reload+0xe4 /app/cmd/thanos/main.go:193
# 0x25179c4 main.main.func6+0x24 /app/cmd/thanos/main.go:163
# 0x6631c8 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x47b70e 0x4581f7 0x253d1c5 0x6631c9 0x482f61
# 0x253d1c4 main.runRule.func10+0x284 /app/cmd/thanos/rule.go:688
# 0x6631c8 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x47b70e 0x4581f7 0x7ebec5 0x7e009a 0x7ef393 0x15d0ee3 0x15d0599 0x15e9888 0x15d0599 0x15d091c 0x15d0599 0x15e9128 0x15d0599 0x1ee4e24 0x1628cd2 0x77b0c2 0x77a971 0x77cf89 0x1ee55f7 0x1ee55ef 0x1f0c430 0x1f0fb2c 0x1f10765 0x253e885 0x1768738 0x1770137 0x176f7ba 0x1775285 0x176cfbc 0x1776646 0x482f61
# 0x7ebec4 net/http.(*persistConn).roundTrip+0x784 /go/pkg/mod/golang.org/[email protected]/src/net/http/transport.go:2826
# 0x7e0099 net/http.(*Transport).roundTrip+0xb99 /go/pkg/mod/golang.org/[email protected]/src/net/http/transport.go:696
# 0x7ef392 net/http.(*Transport).RoundTrip+0x12 /go/pkg/mod/golang.org/[email protected]/src/net/http/roundtrip.go:30
# 0x15d0ee2 github.com/prometheus/client_golang/prometheus/promhttp.InstrumentRoundTripperDuration.func1+0xa2 /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:119
# 0x15d0598 github.com/prometheus/client_golang/prometheus/promhttp.RoundTripperFunc.RoundTrip+0x18 /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:32
# 0x15e9887 github.com/thanos-io/thanos/pkg/extprom/http.InstrumentedRoundTripper.InstrumentRoundTripperTrace.func5+0x687 /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:247
# 0x15d0598 github.com/prometheus/client_golang/prometheus/promhttp.RoundTripperFunc.RoundTrip+0x18 /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:32
# 0x15d091b github.com/prometheus/client_golang/prometheus/promhttp.InstrumentRoundTripperCounter.func1+0x5b /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:75
# 0x15d0598 github.com/prometheus/client_golang/prometheus/promhttp.RoundTripperFunc.RoundTrip+0x18 /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:32
# 0x15e9127 github.com/thanos-io/thanos/pkg/extprom/http.InstrumentedRoundTripper.InstrumentRoundTripperInFlight.func6+0xa7 /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:44
# 0x15d0598 github.com/prometheus/client_golang/prometheus/promhttp.RoundTripperFunc.RoundTrip+0x18 /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:32
# 0x1ee4e23 github.com/thanos-io/thanos/pkg/clientconfig.userAgentRoundTripper.RoundTrip+0x163 /app/pkg/clientconfig/http.go:299
# 0x1628cd1 github.com/thanos-io/thanos/pkg/tracing.(*tripperware).RoundTrip+0x471 /app/pkg/tracing/http.go:112
# 0x77b0c1 net/http.send+0x5e1 /go/pkg/mod/golang.org/[email protected]/src/net/http/client.go:259
# 0x77a970 net/http.(*Client).send+0x90 /go/pkg/mod/golang.org/[email protected]/src/net/http/client.go:180
# 0x77cf88 net/http.(*Client).do+0x988 /go/pkg/mod/golang.org/[email protected]/src/net/http/client.go:728
# 0x1ee55f6 net/http.(*Client).Do+0x16 /go/pkg/mod/golang.org/[email protected]/src/net/http/client.go:587
# 0x1ee55ee github.com/thanos-io/thanos/pkg/clientconfig.(*HTTPClient).Do+0xe /app/pkg/clientconfig/http.go:387
# 0x1f0c42f github.com/thanos-io/thanos/pkg/promclient.(*Client).req2xx+0x4cf /app/pkg/promclient/promclient.go:132
# 0x1f0fb2b github.com/thanos-io/thanos/pkg/promclient.(*Client).QueryInstant+0x5eb /app/pkg/promclient/promclient.go:450
# 0x1f10764 github.com/thanos-io/thanos/pkg/promclient.(*Client).PromqlQueryInstant+0x84 /app/pkg/promclient/promclient.go:503
# 0x253e884 main.runRule.queryFuncCreator.func24.1+0xdc4 /app/cmd/thanos/rule.go:917
# 0x1768737 github.com/prometheus/prometheus/rules.(*AlertingRule).Eval+0x1f7 /go/pkg/mod/github.com/prometheus/[email protected]/rules/alerting.go:343
# 0x1770136 github.com/prometheus/prometheus/rules.(*Group).Eval.func1+0x776 /go/pkg/mod/github.com/prometheus/[email protected]/rules/group.go:540
# 0x176f7b9 github.com/prometheus/prometheus/rules.(*Group).Eval+0x379 /go/pkg/mod/github.com/prometheus/[email protected]/rules/group.go:665
# 0x1775284 github.com/prometheus/prometheus/rules.DefaultEvalIterationFunc+0xe4 /go/pkg/mod/github.com/prometheus/[email protected]/rules/manager.go:81
# 0x176cfbb github.com/prometheus/prometheus/rules.(*Group).run+0x8fb /go/pkg/mod/github.com/prometheus/[email protected]/rules/group.go:307
# 0x1776645 github.com/prometheus/prometheus/rules.(*Manager).Update.func1+0xa5 /go/pkg/mod/github.com/prometheus/[email protected]/rules/manager.go:246
1 @ 0x47b70e 0x4581f7 0x7ebec5 0x7e009a 0x7ef393 0x15d0ee3 0x15d0599 0x15e9888 0x15d0599 0x15d091c 0x15d0599 0x15e9128 0x15d0599 0x1ee4e24 0x1628cd2 0x77b0c2 0x77a971 0x77cf89 0x1ee55f7 0x1ee55ef 0x1f0c430 0x1f0fb2c 0x1f10765 0x253e885 0x17783ff 0x1770137 0x176f7ba 0x1775285 0x176cfbc 0x1776646 0x482f61
# 0x7ebec4 net/http.(*persistConn).roundTrip+0x784 /go/pkg/mod/golang.org/[email protected]/src/net/http/transport.go:2826
# 0x7e0099 net/http.(*Transport).roundTrip+0xb99 /go/pkg/mod/golang.org/[email protected]/src/net/http/transport.go:696
# 0x7ef392 net/http.(*Transport).RoundTrip+0x12 /go/pkg/mod/golang.org/[email protected]/src/net/http/roundtrip.go:30
# 0x15d0ee2 github.com/prometheus/client_golang/prometheus/promhttp.InstrumentRoundTripperDuration.func1+0xa2 /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:119
# 0x15d0598 github.com/prometheus/client_golang/prometheus/promhttp.RoundTripperFunc.RoundTrip+0x18 /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:32
# 0x15e9887 github.com/thanos-io/thanos/pkg/extprom/http.InstrumentedRoundTripper.InstrumentRoundTripperTrace.func5+0x687 /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:247
# 0x15d0598 github.com/prometheus/client_golang/prometheus/promhttp.RoundTripperFunc.RoundTrip+0x18 /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:32
# 0x15d091b github.com/prometheus/client_golang/prometheus/promhttp.InstrumentRoundTripperCounter.func1+0x5b /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:75
# 0x15d0598 github.com/prometheus/client_golang/prometheus/promhttp.RoundTripperFunc.RoundTrip+0x18 /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:32
# 0x15e9127 github.com/thanos-io/thanos/pkg/extprom/http.InstrumentedRoundTripper.InstrumentRoundTripperInFlight.func6+0xa7 /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:44
# 0x15d0598 github.com/prometheus/client_golang/prometheus/promhttp.RoundTripperFunc.RoundTrip+0x18 /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_client.go:32
# 0x1ee4e23 github.com/thanos-io/thanos/pkg/clientconfig.userAgentRoundTripper.RoundTrip+0x163 /app/pkg/clientconfig/http.go:299
# 0x1628cd1 github.com/thanos-io/thanos/pkg/tracing.(*tripperware).RoundTrip+0x471 /app/pkg/tracing/http.go:112
# 0x77b0c1 net/http.send+0x5e1 /go/pkg/mod/golang.org/[email protected]/src/net/http/client.go:259
# 0x77a970 net/http.(*Client).send+0x90 /go/pkg/mod/golang.org/[email protected]/src/net/http/client.go:180
# 0x77cf88 net/http.(*Client).do+0x988 /go/pkg/mod/golang.org/[email protected]/src/net/http/client.go:728
# 0x1ee55f6 net/http.(*Client).Do+0x16 /go/pkg/mod/golang.org/[email protected]/src/net/http/client.go:587
# 0x1ee55ee github.com/thanos-io/thanos/pkg/clientconfig.(*HTTPClient).Do+0xe /app/pkg/clientconfig/http.go:387
# 0x1f0c42f github.com/thanos-io/thanos/pkg/promclient.(*Client).req2xx+0x4cf /app/pkg/promclient/promclient.go:132
# 0x1f0fb2b github.com/thanos-io/thanos/pkg/promclient.(*Client).QueryInstant+0x5eb /app/pkg/promclient/promclient.go:450
# 0x1f10764 github.com/thanos-io/thanos/pkg/promclient.(*Client).PromqlQueryInstant+0x84 /app/pkg/promclient/promclient.go:503
# 0x253e884 main.runRule.queryFuncCreator.func24.1+0xdc4 /app/cmd/thanos/rule.go:917
# 0x17783fe github.com/prometheus/prometheus/rules.(*RecordingRule).Eval+0x19e /go/pkg/mod/github.com/prometheus/[email protected]/rules/recording.go:83
# 0x1770136 github.com/prometheus/prometheus/rules.(*Group).Eval.func1+0x776 /go/pkg/mod/github.com/prometheus/[email protected]/rules/group.go:540
# 0x176f7b9 github.com/prometheus/prometheus/rules.(*Group).Eval+0x379 /go/pkg/mod/github.com/prometheus/[email protected]/rules/group.go:665
# 0x1775284 github.com/prometheus/prometheus/rules.DefaultEvalIterationFunc+0xe4 /go/pkg/mod/github.com/prometheus/[email protected]/rules/manager.go:81
# 0x176cfbb github.com/prometheus/prometheus/rules.(*Group).run+0x8fb /go/pkg/mod/github.com/prometheus/[email protected]/rules/group.go:307
# 0x1776645 github.com/prometheus/prometheus/rules.(*Manager).Update.func1+0xa5 /go/pkg/mod/github.com/prometheus/[email protected]/rules/manager.go:246
1 @ 0x47b70e 0x4581f7 0xefb876 0x482f61
# 0xefb875 github.com/prometheus/prometheus/tsdb/wlog.(*WL).run+0x95 /go/pkg/mod/github.com/prometheus/[email protected]/tsdb/wlog/wlog.go:416
1 @ 0x47b70e 0x45929d 0x459274 0x47ce65 0x48ea68 0x9e771e 0x1a4f557 0x1a4fa12 0x1a53de5 0x253c53a 0x6631c9 0x482f61
# 0x47ce64 sync.runtime_SemacquireWaitGroup+0x24 /go/pkg/mod/golang.org/[email protected]/src/runtime/sema.go:110
# 0x48ea67 sync.(*WaitGroup).Wait+0x47 /go/pkg/mod/golang.org/[email protected]/src/sync/waitgroup.go:118
# 0x9e771d golang.org/x/sync/errgroup.(*Group).Wait+0x1d /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:56
# 0x1a4f556 github.com/prometheus/exporter-toolkit/web.ServeMultiple+0x116 /go/pkg/mod/github.com/prometheus/[email protected]/web/tls_config.go:277
# 0x1a4fa11 github.com/prometheus/exporter-toolkit/web.ListenAndServe+0x411 /go/pkg/mod/github.com/prometheus/[email protected]/web/tls_config.go:325
# 0x1a53de4 github.com/thanos-io/thanos/pkg/server/http.(*Server).ListenAndServe+0x244 /app/pkg/server/http/http.go:86
# 0x253c539 main.runRule.func18+0x39 /app/cmd/thanos/rule.go:812
# 0x6631c8 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x4a18c5 0x49f9d8 0x50394e 0x503936 0x5037d1 0x5bbf45 0x5cd8a5 0x7c74b7 0x482f61
# 0x4a18c4 syscall.Syscall+0x24 /go/pkg/mod/golang.org/[email protected]/src/syscall/syscall_linux.go:73
# 0x49f9d7 syscall.read+0x37 /go/pkg/mod/golang.org/[email protected]/src/syscall/zsyscall_linux_amd64.go:736
# 0x50394d syscall.Read+0x2ad /go/pkg/mod/golang.org/[email protected]/src/syscall/syscall_unix.go:183
# 0x503935 internal/poll.ignoringEINTRIO+0x295 /go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_unix.go:745
# 0x5037d0 internal/poll.(*FD).Read+0x130 /go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_unix.go:161
# 0x5bbf44 net.(*netFD).Read+0x24 /go/pkg/mod/golang.org/[email protected]/src/net/fd_posix.go:55
# 0x5cd8a4 net.(*conn).Read+0x44 /go/pkg/mod/golang.org/[email protected]/src/net/net.go:194
# 0x7c74b6 net/http.(*connReader).backgroundRead+0x36 /go/pkg/mod/golang.org/[email protected]/src/net/http/server.go:690
~~(Click each to expand)~~
EDIT: Ignore above. That was actually from a healthy thanos ruler. I'm unable to do the same on the ruler I described as it causes it to crash (because already high memory usage on limited h/w). I'll try to see if I can capture it next time my thanos-ruler exhibits this.
