Konstantin Lebedev

@ldapomni Maybe mc has a logging mode for debugging? In aws s3, there is no concept of folders, but only keys, to remove which you must explicitly pass a list...

``` ================== WARNING: DATA RACE Write at 0x00c000822818 by goroutine 893: github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).Version() /go/src/github.com/seaweedfs/seaweedfs/weed/storage/volume.go:104 +0x137 github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).ToVolumeInformationMessage() /go/src/github.com/seaweedfs/seaweedfs/weed/storage/volume.go:322 +0x136 github.com/seaweedfs/seaweedfs/weed/storage.(*Store).CollectHeartbeat() /go/src/github.com/seaweedfs/seaweedfs/weed/storage/store.go:237 +0xf07 github.com/seaweedfs/seaweedfs/weed/server.(*VolumeServer).doHeartbeat() /go/src/github.com/seaweedfs/seaweedfs/weed/server/volume_grpc_client_to_master.go:215 +0x1c3a github.com/seaweedfs/seaweedfs/weed/server.(*VolumeServer).heartbeat() /go/src/github.com/seaweedfs/seaweedfs/weed/server/volume_grpc_client_to_master.go:69 +0x449 github.com/seaweedfs/seaweedfs/weed/server.NewVolumeServer.func1() /go/src/github.com/seaweedfs/seaweedfs/weed/server/volume_server.go:123 +0x39 Previous...

write line `v.SuperBlock.Version = needle.Version(v.volumeInfo.Version)` ``` func (v *Volume) Version() needle.Version { v.superBlockAccessLock.Lock() defer v.superBlockAccessLock.Unlock() if v.volumeInfo.Version != 0 { v.SuperBlock.Version = needle.Version(v.volumeInfo.Version) } return v.SuperBlock.Version } ``` read line...

> Moving some variables for atomic counting may cause atomic operation to fail on 32bit OS. Can then for mass structures it is possible to through `//+build 386` ?

> The atomic error is a runtime error reported in several github issues. I am not aware any way to check it during build time. Like this is it https://github.com/golang/go/issues/36606

Possibly issue is https://github.com/seaweedfs/seaweedfs/issues/3565

On version 3.26 the problem is no reproduced ``` Sep 9, 2022 @ 13:52:50.144 | I0909 08:52:50.144723 meta_aggregator.go:100 stop subscribing remote 10.103.178.73:9090 meta change Sep 9, 2022 @ 13:52:50.144 |...

@chrislusf Reproduces an issue where all master and filler pods are restarted at the same time

filler do not reconnect to leader ``` E0117 10:10:16 1 s3api_object_handlers.go:421] upload to filer error: rpc error: code = Unknown desc = raft.Server: Not current leader ```

@chrislusf Can you suggest where to look for the problem? I added metrics that monitor reconnects to the master https://github.com/chrislusf/seaweedfs/pull/2613 and updated to 2.87 but I haven't figured out how...