LTX corruption

[tjheeta]

After failover testing done by fly restart vm:

2022-08-25T03:43:05Z app[8cd3b300] sea [info]cannot open store: open databases: open database: db=00000001 err=recover ltx: read ltx file header (0000000000000062-0000000000000062.ltx): unmarshal header: invalid LTX file

As far as I know, there was no activity for probably 10-30 seconds before the restart. Possibly longer. I'm attaching the entire directory as a base64 in hope that it is of some use to you.

root@8cd3b300:~/db/00000001/ltx# ls -latr
total 236
-rw-r--r-- 1 root root 8320 Aug 25 03:41 0000000000000050-0000000000000050.ltx
-rw-r--r-- 1 root root 8320 Aug 25 03:41 0000000000000051-0000000000000051.ltx
-rw-r--r-- 1 root root 8320 Aug 25 03:41 0000000000000052-0000000000000052.ltx
-rw-r--r-- 1 root root 8320 Aug 25 03:41 0000000000000053-0000000000000053.ltx
-rw-r--r-- 1 root root 8320 Aug 25 03:41 0000000000000054-0000000000000054.ltx
-rw-r--r-- 1 root root 8320 Aug 25 03:41 0000000000000055-0000000000000055.ltx
-rw-r--r-- 1 root root 8320 Aug 25 03:41 0000000000000056-0000000000000056.ltx
-rw-r--r-- 1 root root 8320 Aug 25 03:41 0000000000000057-0000000000000057.ltx
-rw-r--r-- 1 root root 8320 Aug 25 03:41 0000000000000058-0000000000000058.ltx
-rw-r--r-- 1 root root 8320 Aug 25 03:42 0000000000000059-0000000000000059.ltx
-rw-r--r-- 1 root root 8320 Aug 25 03:42 000000000000005a-000000000000005a.ltx
-rw-r--r-- 1 root root 8320 Aug 25 03:42 000000000000005b-000000000000005b.ltx
-rw-r--r-- 1 root root 8320 Aug 25 03:42 000000000000005c-000000000000005c.ltx
-rw-r--r-- 1 root root 8320 Aug 25 03:42 000000000000005d-000000000000005d.ltx
-rw-r--r-- 1 root root 8320 Aug 25 03:42 000000000000005e-000000000000005e.ltx
-rw-r--r-- 1 root root 8320 Aug 25 03:42 000000000000005f-000000000000005f.ltx
-rw-r--r-- 1 root root 8320 Aug 25 03:42 0000000000000060-0000000000000060.ltx
-rw-r--r-- 1 root root 8320 Aug 25 03:42 0000000000000061-0000000000000061.ltx
-rw-r--r-- 1 root root 8320 Aug 25 03:42 0000000000000062-0000000000000062.ltx

$ base64 ltx.tar.gz 
H4sIAAAAAAAAA+3dcWxUdwHA8d+1TMAxD1mAIIrHKUhnr7z37t4dbBCHs4FGttD2gLa0tO/ag5XQ
NvRu1CUkBEKmkf4BmLS6RMymfzANMGOAhYxW0dhlY0khyBBHNFoSp53RZMPUwfTeXdvAQ37bvVvf
3ct9P0B/l+v9+rsfb/+wfO/3FCVNXS6mjrlARNdFZinFOqYfq6GgourhSDioCkVVQiFN+PQpfE+T
nkkkjW6fT3R3dSVlr/uo77uUMnH9dya/NVX/DWR//VVNV7n+Trjr+it3CasB6xMVqVfZWyMcCt3n
+mupX3de/9Tzqq6GdOFTPvnt3qvIr//6aJ2aeTTb/FKa+uMZ/5ZhGUVf5ORLu3sPnbKxjKe2en17
Mu7b1tXdYSR9wdRqHo94PPX3nvn5pXe8tiT1Z9qdcz/GzzdERfPnHjInez8lvLtSXwAAAAAAyJv2
kunzFyzw7AskjdjOeGvXM53JROZr6RM1lWuilb7omq+vr/T5M0/6fcv87W1+X9VT0cq1lTW+DTVV
T66pqfd9s7K+3Ofvjm9v7+r0+6KVddHy8SmTry3bN8szff7y5Z59a9NrJVqfjncYzR3t27uNZGpa
4p4nSu5+B/d833wzu+PdifSa//8dtXcm4t3JeFuzkcy8reZvpH5itOrJyrLU5kvNf58/4L0ovDe9
73pHvH9IPQQAAAAAAIVuxgPTxMOejnZj0fRpqUczjY7E9NKJByXjD2Z4zBcl4ka6MxjZ89s3jwcO
fO1I59u+3UNXD+R7C8VO0v/obQHrEw70P1rY7H+Capj+xwmS/qfJMoqbz3/w+oPBwYiNZWT9T5PI
vf9pov8BAAAAABQO+h/6HwAAAAAAXCa7/idgTjlw8ciJ4SM/O3fse77a9cc65+R7C8VO1v9ss/Y/
25zofyLp/kcL0v84QdL/NFtG8eKsstJXd/X5bSwj63+aRe79TzP9DwAAAACgcND/0P8AAAAAAOAy
2fU/6duLDa99fPv24f3H/nN69qk5Twe0fG+h2Mn6n1Zr/9Pq4Pk/IfofJ0j6n0bLKMZ23HhvZnR4
v41lZP1Po8i9/2mk/wEAAAAAFA76H/ofAAAAAABcJrv+p9ycMnEPoauDn/nL787+9Ev53kKxk/U/
qrX/Uae+/9FD6fN/VF2j/3GCpP+ptoxioHfu3lcGq9bZWEbW/1SL3PufavofAAAAAEDhoP+h/wEA
AAAAwGWy63/Src+FoT0LWxc1nX9nVfn1X998c26+t1DsZP1P3Nr/xJ07/0dT6H+cIOl/tlrGyXv3
2VhG1v9sFbn3P1vpfwAAAAAAhYP+h/4HAAAAAACXya7/qTCnvDirrPTVXX3+kfdLd5z/9+If5XsL
xU7S/4SVgPUJJ/qf8fN/VPofJ0j6nxbLOHnvPhvLyPqfFpF7/9NC/wMAAAAAKBz0P/Q/AAAAAAC4
THb9Tzrn6IucfGl376FTfaMHj/krL9/K9xaKnez8n6D1/J+gA/f/0oOZ/idI/+MESf9TaxnF6LWh
097bLU02lpH1P7Ui9/6nlv4HAAAAAFA46H/ofwAAAAAAcJns+p8l5pSjh09cutyYfOvgh5/v/+z8
xXbOEsEnSNb/hK39T9iJ/kdP9z9h+h9HSPqfTZZR3Pjb77/z0PPev9pYRtb/bBK59z+b6H8AAAAA
AIWD/of+BwAAAAAAl8mu/1lmTnmtJ7bEeFbr+fmHyox/PHXtVL63UOxk/Y9u7X90J/qfUKb/Ueh/
nCDpfzZaRtH/wsujVx5bGLOxjKz/2Shy73820v8AAAAAAAoH/Q/9DwAAAAAALpNd//MVc8rEGSLf
/f7DaldgLJLvLRQ7Wf8Ts/Y/sanvf8Ja2Ox/gqpG/+MESf+zxTKKg4t7D116/zczbCwj63+2iNz7
ny30PwAAAACAwkH/Q/8DAAAAAIDLZNf/fNWcMrbjxnszo8P7fzBnef3ALxbMzvcWip2s/zGs/Y/h
RP+Tvv9XUInQ/zhB0v80WEZx/FZk9cC8sX4by8j6nwaRe//TQP8DAAAAACgc9D/0PwAAAAAAuEx2
/c8j5pSJM0S+Xb3quc7yuefyvYViJ+t/FGv/ozhw/69Q+vwfNUT/4whJ/7PBMoozF688cv1Xr9/Z
6nxcsv5ng8i9/9lA/wMAAAAAKBz0P/Q/AAAAAAC4THb9j9+cMtA7d+8rg1Xr3r4+Uv/Bc/9tzfcW
ip2s/9Gs/Y/mQP+jK5n+J0z/4wRJ/1NjGcWFoT0LWxc1nbexjKz/qRG59z819D8AAAAAgMJB/0P/
AwAAAACAy2TX/3zZnDJ6bei093ZL0xsN3pF158I/zPcWip2s/1lp7X9WOnH/r1C6/wnT/zhC0v/U
W0bxxzNb6+Nn3v2CjWVk/U+9yL3/qaf/AQAAAAAUDvof+h8AAAAAAFwmu/7H/H8A4vityOqBeWP9
b0S+OPinRIMjiQfuT9b/rLD2PyscOP8nomb6H53+xwmS/qfOMoqzqx5cc/bq0ts2lpH1P3Ui9/6n
jv4HAAAAAFA46H/ofwAAAAAAcJns+p9l5pSJM0T+9eO/HyrpaazI9xaKnaz/iVj7n4gT/U/m/l9h
jf7HCZL+Z7NlFK/1xJYYz2o9NpaR9T+bRe79z2b6HwAAAABA4aD/of8BAAAAAMBlbJz/M3GGyPmy
vYGTlw//M99bKHaS/iesBaxPOHD/r2Dm/B9dof8BAAAAAAAAAAAAAAAAPors/J+Q9fyfkAPn/+ih
TP8Tpv9xguT8n6hlFEcPn7h0uTH5lo1lZOf/REXu5/9EOf8HAAAAAFA4OP+H838AAAAAAHCZ7M7/
WWpO6X/h5dErjy2M/aR2dWDvp5cezfcWit1k/9NmJI2YkYhP0Rr373+C6cd3n/+jqRr3/3KEpMuJ
idy7nBhdDgAAAACgcBRvl2P+29zj/bNI/QYAAAAAAK4178CFX555Z+iJUhHUFE0LKCsCmh5VtEcV
/VFtJZ/JAQAAAADAhbL7TI42+fmPTqNjKj77YZJ//sNcXR///IceCqrm+a+qFgny+Q8nJOOJZFus
IrFrZ3tyqq4/AAAAAAAAgPz4H9qMJFcA+AIA