seaweedfs-csi-driver icon indicating copy to clipboard operation
seaweedfs-csi-driver copied to clipboard

Published 20 hours ago • verified publisher icon seaweedfs

Docker Swarm compatibility

Open pedroetb opened this issue 2 years ago • 7 comments

There is any way to make this CSI driver work for Docker Swarm volumes?

pedroetb avatar Dec 29 '22 12:12 pedroetb

Leaving my vote here as well. Docker 23.0.0 is finally out, and having a SeaweedFS CSI driver would be useful.

Link to the changelog:

https://github.com/moby/moby/releases/tag/v23.0.0 https://docs.docker.com/engine/release-notes/23.0/

s4ke avatar Feb 04 '23 00:02 s4ke

Yes Docker Swarm supports CSI interface starting on 23.0.0 but CSI drivers still need to be packages as Docker plugins. Official documentation about that process exists on https://github.com/moby/moby/blob/master/docs/cluster_volumes.md and I have build some example scripts and guidance to https://github.com/olljanat/csi-plugins-for-docker-swarm

olljanat avatar Feb 05 '23 09:02 olljanat

Is there anything this repo can do to smooth the whole process? @olljanat @s4ke

chrislusf avatar Feb 05 '23 21:02 chrislusf

@chrislusf preferably someone who have existing SeaweedFS environment and understanding what are requirements of this CSI driver should:

  1. Take copy of one of those examples which I have shared (there is three now).
  2. Modify it scripts and config on way that it works with this CSI plugin.
  3. Eventually came as pull request here which adds those as part of new version release process.

olljanat avatar Feb 05 '23 22:02 olljanat

Judging by the fact that this driver already implements a monolith for both the controller and node server, this plugin should be quite easy to repackage. All we would have to do is repackage the image as a docker plugin, see the following example:

https://github.com/hetznercloud/csi-driver/tree/main/deploy/docker-swarm/pkg

s4ke avatar Feb 18 '23 00:02 s4ke

Has anyone tried to package this up yet for Docker?

prologic avatar Mar 02 '24 02:03 prologic

Sorry, currently busy with other things. If anyone needs guidance on how to do this, let me know. I am happy to help but just dont have the time to take care of all of it.

s4ke avatar Mar 02 '24 10:03 s4ke

I want this plugin, so I guess I will do it then. @s4ke Can you point me in the right direction to get started?

Gradlon avatar Jul 11 '24 11:07 Gradlon

For the Hetzner plugin I ported, you can look here: https://github.com/hetznercloud/csi-driver/blob/main/cmd/aio/main.go

Packaging the plugin is done here: https://github.com/hetznercloud/csi-driver/blob/main/deploy/docker-swarm/pkg/README.md

You can find more details here: https://github.com/olljanat/csi-plugins-for-docker-swarm

Also, feel free to reach out to me via discord over at devops.fan. I'd be happy to jump onto a call if you'd like as well.

s4ke avatar Jul 11 '24 12:07 s4ke

@s4ke I gave it a go. As far as I understand it, it should be straightforward. For the moment I am stuck at the point of creating the Socket, somehow Docker can't connect to it. The code can be found here https://github.com/Gradlon/swarm-csi-swaweedfs Any suggestions on what I am missing?

Gradlon avatar Jul 11 '24 16:07 Gradlon

Do you have some docker daemon logs for me? For me it was quite helpful to enable "debug": true on the docker daemon. Also, if you need to control the containerd container that is used for the driver, you should look into using the containerd ctr command. It has similar options as the docker command.

s4ke avatar Jul 11 '24 17:07 s4ke

I gave it another go.

I believe that tit fails on

Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="F0711 18:53:44.301927 server.go:61 invalid endpoint: /var/run/csi.sock" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102

But I could be wrong, not used to reading go Error's yet ;)

Jul 11 20:53:43 builder dockerd[4764]: time="2024-07-11T20:53:43.967351563+02:00" level=debug msg="Calling POST /v1.46/plugins/gradlon/swarm-csi-swaweedfs:v1.2.0/enable?timeout=30"
Jul 11 20:53:43 builder dockerd[4764]: time="2024-07-11T20:53:43.969385385+02:00" level=debug msg="bundle dir created" bundle=/run/docker/plugins/985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102 module=libcontainerd namespace=plugins.moby root=/var/lib/docker/plugins/985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102/rootfs
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44.102223488+02:00" level=debug msg=event module=libcontainerd namespace=plugins.moby topic=/tasks/create
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44.117858083+02:00" level=debug msg=event module=libcontainerd namespace=plugins.moby topic=/tasks/start
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.138077 main.go:73 will run node: true, controller: true, attacher: true" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.138293 main.go:79 connect to filer 10.0.1.1:8888" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.138304 driver.go:53 Driver: seaweedfs-csi-driver version: 1.0.0" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.138595 driver.go:127 Enabling volume access mode: MULTI_NODE_MULTI_WRITER" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.138603 driver.go:127 Enabling volume access mode: SINGLE_NODE_WRITER" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.138607 driver.go:127 Enabling volume access mode: SINGLE_NODE_MULTI_WRITER" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.138610 driver.go:127 Enabling volume access mode: SINGLE_NODE_SINGLE_WRITER" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.138614 driver.go:134 Enabling controller service capability: CREATE_DELETE_VOLUME" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.138619 driver.go:134 Enabling controller service capability: SINGLE_NODE_MULTI_WRITER" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.138622 driver.go:134 Enabling controller service capability: EXPAND_VOLUME" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.138626 driver.go:134 Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.138636 driver.go:90 starting" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="F0711 18:53:44.138815 server.go:61 invalid endpoint: /var/run/csi.sock" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="goroutine 55 [running]:" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="github.com/seaweedfs/seaweedfs/weed/glog.stacks(0x0)" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="\t/go/pkg/mod/github.com/seaweedfs/[email protected]/weed/glog/glog.go:768 +0x85" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="github.com/seaweedfs/seaweedfs/weed/glog.(*loggingT).output(0x2905bc0, 0x3, 0xc0003fc540, {0x1fda19b?, 0x1?}, 0xc00005b530?, 0x0)" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="\t/go/pkg/mod/github.com/seaweedfs/[email protected]/weed/glog/glog.go:719 +0x38a" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="github.com/seaweedfs/seaweedfs/weed/glog.(*loggingT).printDepth(0x2905bc0, 0x3, 0x0?, {0xc00008bf38, 0x1, 0x1})" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="\t/go/pkg/mod/github.com/seaweedfs/[email protected]/weed/glog/glog.go:648 +0xea" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="github.com/seaweedfs/seaweedfs/weed/glog.(*loggingT).print(...)" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="\t/go/pkg/mod/github.com/seaweedfs/[email protected]/weed/glog/glog.go:639" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="github.com/seaweedfs/seaweedfs/weed/glog.Fatal(...)" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="\t/go/pkg/mod/github.com/seaweedfs/[email protected]/weed/glog/glog.go:1134" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="github.com/seaweedfs/seaweedfs-csi-driver/pkg/driver.(*nonBlockingGRPCServer).serve(0xc000421518, {0x7fff74e29ef0?, 0x0?}, {0x1be7b28, 0xc0004b63e8}, {0x1bffe08, 0xc0004b63e0}, {0x1bfca78, 0xc00011fdd0})" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="\t/go/src/github.com/seaweedfs/seaweedfs-csi-driver/pkg/driver/server.go:61 +0xd4" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="github.com/seaweedfs/seaweedfs-csi-driver/pkg/driver.(*nonBlockingGRPCServer).Start.func1()" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="\t/go/src/github.com/seaweedfs/seaweedfs-csi-driver/pkg/driver/server.go:41 +0x6f" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="created by github.com/seaweedfs/seaweedfs-csi-driver/pkg/driver.(*nonBlockingGRPCServer).Start in goroutine 1" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="\t/go/src/github.com/seaweedfs/seaweedfs-csi-driver/pkg/driver/server.go:39 +0xf3" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44.149726373+02:00" level=debug msg=event module=libcontainerd namespace=plugins.moby topic=/tasks/exit
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44.175998440+02:00" level=debug msg=event module=libcontainerd namespace=plugins.moby topic=/tasks/delete
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44.176031563+02:00" level=info msg="ignoring event" container=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102 module=libcontainerd namespace=plugins.moby topic=/tasks/delete type="*events.TaskDelete"
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44.197385259+02:00" level=debug msg="bundle dir created" bundle=/run/docker/plugins/985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102 module=libcontainerd namespace=plugins.moby root=/var/lib/docker/plugins/985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102/rootfs
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44.262861122+02:00" level=debug msg=event module=libcontainerd namespace=plugins.moby topic=/tasks/create
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44.283055724+02:00" level=debug msg=event module=libcontainerd namespace=plugins.moby topic=/tasks/start
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.300898 main.go:73 will run node: true, controller: true, attacher: true" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.301158 main.go:79 connect to filer 10.0.1.1:8888" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.301166 driver.go:53 Driver: seaweedfs-csi-driver version: 1.0.0" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.301542 driver.go:127 Enabling volume access mode: MULTI_NODE_MULTI_WRITER" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.301555 driver.go:127 Enabling volume access mode: SINGLE_NODE_WRITER" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.301558 driver.go:127 Enabling volume access mode: SINGLE_NODE_MULTI_WRITER" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.301562 driver.go:127 Enabling volume access mode: SINGLE_NODE_SINGLE_WRITER" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.301581 driver.go:134 Enabling controller service capability: CREATE_DELETE_VOLUME" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.301586 driver.go:134 Enabling controller service capability: SINGLE_NODE_MULTI_WRITER" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.301590 driver.go:134 Enabling controller service capability: EXPAND_VOLUME" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.301594 driver.go:134 Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="I0711 18:53:44.301601 driver.go:90 starting" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="F0711 18:53:44.301927 server.go:61 invalid endpoint: /var/run/csi.sock" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="goroutine 53 [running]:" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="github.com/seaweedfs/seaweedfs/weed/glog.stacks(0x0)" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="\t/go/pkg/mod/github.com/seaweedfs/[email protected]/weed/glog/glog.go:768 +0x85" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="github.com/seaweedfs/seaweedfs/weed/glog.(*loggingT).output(0x2905bc0, 0x3, 0xc00016c310, {0x1fda19b?, 0x1?}, 0xc000103760?, 0x0)" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="\t/go/pkg/mod/github.com/seaweedfs/[email protected]/weed/glog/glog.go:719 +0x38a" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="github.com/seaweedfs/seaweedfs/weed/glog.(*loggingT).printDepth(0x2905bc0, 0x3, 0x0?, {0xc000088f38, 0x1, 0x1})" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="\t/go/pkg/mod/github.com/seaweedfs/[email protected]/weed/glog/glog.go:648 +0xea" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="github.com/seaweedfs/seaweedfs/weed/glog.(*loggingT).print(...)" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="\t/go/pkg/mod/github.com/seaweedfs/[email protected]/weed/glog/glog.go:639" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="github.com/seaweedfs/seaweedfs/weed/glog.Fatal(...)" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="\t/go/pkg/mod/github.com/seaweedfs/[email protected]/weed/glog/glog.go:1134" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="github.com/seaweedfs/seaweedfs-csi-driver/pkg/driver.(*nonBlockingGRPCServer).serve(0xc0001346a8, {0x7ffe5e1b0ef0?, 0x0?}, {0x1be7b28, 0xc00012a1c0}, {0x1bffe08, 0xc00012a1b8}, {0x1bfca78, 0xc0001d3da0})" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="\t/go/src/github.com/seaweedfs/seaweedfs-csi-driver/pkg/driver/server.go:61 +0xd4" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="github.com/seaweedfs/seaweedfs-csi-driver/pkg/driver.(*nonBlockingGRPCServer).Start.func1()" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="\t/go/src/github.com/seaweedfs/seaweedfs-csi-driver/pkg/driver/server.go:41 +0x6f" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="created by github.com/seaweedfs/seaweedfs-csi-driver/pkg/driver.(*nonBlockingGRPCServer).Start in goroutine 1" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44+02:00" level=error msg="\t/go/src/github.com/seaweedfs/seaweedfs-csi-driver/pkg/driver/server.go:39 +0xf3" plugin=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102
Jul 11 20:53:44 builder dockerd[4764]: time="2024-07-11T20:53:44.314400485+02:00" level=debug msg=event module=libcontainerd namespace=plugins.moby topic=/tasks/exit
Jul 11 20:53:56 builder dockerd[4764]: time="2024-07-11T20:53:56.621472118+02:00" level=debug msg="error net dialing plugin: dial unix /run/docker/plugins/985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102/csi/csi.sock: connect: no such file or directory"
Jul 11 20:53:56 builder dockerd[4764]: time="2024-07-11T20:53:56.640165673+02:00" level=error msg="Sending SIGTERM to plugin failed with error: NotFound: process already finished: not found"
Jul 11 20:53:56 builder dockerd[4764]: time="2024-07-11T20:53:56.640338522+02:00" level=debug msg="FIXME: Got an API for which error does not match any expected type!!!" error="dial unix /run/docker/plugins/985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102/csi/csi.sock: connect: no such file or directory" error_type="*net.OpError" module=api
Jul 11 20:53:56 builder dockerd[4764]: time="2024-07-11T20:53:56.640441142+02:00" level=error msg="Handler for POST /v1.46/plugins/gradlon/swarm-csi-swaweedfs:v1.2.0/enable returned error: dial unix /run/docker/plugins/985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102/csi/csi.sock: connect: no such file or directory"
Jul 11 20:53:56 builder dockerd[4764]: time="2024-07-11T20:53:56.786914522+02:00" level=debug msg="error net dialing plugin: dial unix /run/docker/plugins/985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102/csi/csi.sock: connect: no such file or directory"
Jul 11 20:53:56 builder dockerd[4764]: time="2024-07-11T20:53:56.801264468+02:00" level=error msg="Sending SIGTERM to plugin failed with error: NotFound: process already finished: not found"
Jul 11 20:53:56 builder dockerd[4764]: time="2024-07-11T20:53:56.827514808+02:00" level=debug msg=event module=libcontainerd namespace=plugins.moby topic=/tasks/delete
Jul 11 20:53:56 builder dockerd[4764]: time="2024-07-11T20:53:56.827621777+02:00" level=info msg="ignoring event" container=985fc05f71364389edee9051907919eb204012d2056af5cb62f167f2ec90e102 module=libcontainerd namespace=plugins.moby topic=/tasks/delete type="*events.TaskDelete"

Gradlon avatar Jul 11 '24 18:07 Gradlon

Have you debugged the logs of the plugin with ctr already? could be that the socket is never created. Maybe the process is crashing?

s4ke avatar Jul 11 '24 21:07 s4ke

Running

docker run -it --rm --privileged --pid=host -v /proc:/host/proc -v /var/run/docker/netns:/var/run/docker/netns -v /var/lib/docker/plugins/<ID>/rootfs:/plugin-root alpine chroot plugin-root sh

./seaweedfs-csi-driver -nodeid=$HOSTNAME -filer=10.0.1.1:8888

I can see that the process works in this context at least. The socket is created at /tmp/seaweedfs-csi.sock

the problem is that docker is looking for the socket at /run/docker/plugins/<Plugin ID> but there i only get the <ID>stderr/stdout

After some testing, I figured out that "-endpoint=unix://run/docker/plugins/seaweed.sock" make the socket available on the host.

Gradlon avatar Jul 12 '24 08:07 Gradlon

After some testing, I figured out that "-endpoint=unix://run/docker/plugins/seaweed.sock" make the socket available on the host.

Looking at https://github.com/hetznercloud/csi-driver/blob/b6cf8a8764dd2c05bc290dec68adf2ae5b2fcc69/deploy/docker-swarm/pkg/config.json#L16 this looks about right.

Have you already checked out how to debug the plugin using ctr?

Looks like you are unblocked, though, right?

s4ke avatar Jul 12 '24 08:07 s4ke

No, did not understand how I should be doing this Currently, I am trying to pass arguments to the CSI. Socket connection now works, now I want to make it configurable.

Gradlon avatar Jul 12 '24 09:07 Gradlon

Figured it out.

docker plugin install --disable --alias seaweedfs-csi:swarm --grant-all-permissions gradlon/swarm-csi-swaweedfs:v1.2.0 docker plugin set seaweedfs-csi:swarm FILER=IP1:8888,IP2:8888 docker plugin set seaweedfs-csi:swarm CACHE_SIZE=512 docker plugin enable seaweedfs-csi:swarm docker volume create --driver gradlon/swarm-csi-swaweedfs:v1.2.0 --availability active --scope single --sharing none --type mount test-volume

This works to the point that I get a cluster Volume but it is stuck.

test-volume             seaweedfs-csi:swarm   active         pending creation

Gradlon avatar Jul 12 '24 10:07 Gradlon

@chrislusf @s4ke

I can't figure out why it is failing. Any ideas? Code is here https://github.com/Gradlon/swarm-csi-swaweedfs

Jul 12 15:00:02 builder dockerd[840]: time="2024-07-12T15:00:02.384813636+02:00" level=debug msg="Calling POST /v1.45/volumes/create" spanID=0c2d0a4015fe5925 traceID=d17b16c9a9bcb320497e2c716d3a1dcd
Jul 12 15:00:02 builder dockerd[840]: time="2024-07-12T15:00:02.385015174+02:00" level=debug msg="form data: {\"ClusterVolumeSpec\":{\"AccessMode\":{\"MountVolume\":{},\"Scope\":\"single\",\"Sharing\":\"all\"},\"AccessibilityRequirements\":{},\"Availability\":\"active\",\"CapacityRange\":{\"LimitBytes\":0,\"RequiredBytes\":0}},\"Driver\":\"gradlon/swarm-csi-swaweedfs:v1.2.0\",\"DriverOpts\":{\"path\":\"/data\"},\"Name\":\"testVolume\"}" spanID=0c2d0a4015fe5925 traceID=d17b16c9a9bcb320497e2c716d3a1dcd
Jul 12 15:00:02 builder dockerd[840]: time="2024-07-12T15:00:02.385158055+02:00" level=debug msg="using cluster volume" spanID=0c2d0a4015fe5925 traceID=d17b16c9a9bcb320497e2c716d3a1dcd


Jul 12 15:00:26 builder dockerd[840]: time="2024-07-12T15:00:26.912339604+02:00" level=debug msg="2024/07/12 15:00:26 WARNING: [core] [Channel #12 SubChannel #13] grpc: addrConn.createTransport failed to connect to {Addr: \"/run/docker/plugins/f3306c1f2f0db70878a674d9073dd19b1d0109c7b6a70c7b648d9756a69757df/seaweed.sock\", ServerName: \"localhost\", Attributes: {\"<%!p(networktype.keyType=grpc.internal.transport.networktype)>\": \"unix\" }, }. Err: connection error: desc = \"transport: Error while dialing: dial unix /run/docker/plugins/f3306c1f2f0db70878a674d9073dd19b1d0109c7b6a70c7b648d9756a69757df/seaweed.sock: connect: no such file or directory\"" library=grpc

Gradlon avatar Jul 12 '24 13:07 Gradlon

seems no error messages from seaweed itself.

chrislusf avatar Jul 12 '24 17:07 chrislusf

I found the issue. There was an outdated driver version that was blocking the process. Probably related to this: https://github.com/moby/swarmkit/issues/3120

Now the volume is created(I see a directory in the Filer). But I have two issues.

  1. Path argument is not applied, the volume is always created in /buckets/<VolumeName>
  2. And I am unable to mount the volume onto a Container.
Jul 13 13:10:45 builder dockerd[69345]: time="2024-07-13T13:10:45.313975523+02:00" level=debug msg="form data: {\"ClusterVolumeSpec\":{\"AccessMode\":{\"MountVolume\":{},\"Scope\":\"single\",\"Sharing\":\"all\"},\"AccessibilityRequirements\":{},\"Availability\":\"active\",\"CapacityRange\":{\"LimitBytes\":0,\"RequiredBytes\":0}},\"Driver\":\"gradlon/swarm-csi-swaweedfs:v1.2.0\",\"DriverOpts\":{\"path\":\"/docker/volumes\"},\"Name\":\"testVolume\"}" spanID=217124d632e3265b traceID=beab043ad2e94e70f1416a5dd0a32067
Jul 13 13:10:45 builder dockerd[69345]: time="2024-07-13T13:10:45.314104200+02:00" level=debug msg="using cluster volume" spanID=217124d632e3265b traceID=beab043ad2e94e70f1416a5dd0a32067
Jul 13 13:10:45 builder dockerd[69345]: time="2024-07-13T13:10:45.316060868+02:00" level=info msg="creating volume" attempt=0 driver="gradlon/swarm-csi-swaweedfs:v1.2.0" module=csi/manager node.id=xajfahck1rppdkh1s88haz400 volume.id=goezb38l6wk050isx2qwfkxqv
Jul 13 13:10:45 builder dockerd[69345]: time="2024-07-13T13:10:45+02:00" level=error msg="I0713 11:10:45.316983 controllerserver.go:27 create volume req: testVolume" plugin=321af86c8b8cb7d10d5d7e43dd08e6a10022635f3fab43c2a4eab791ee25c5ab
Jul 13 13:10:45 builder dockerd[69345]: time="2024-07-13T13:10:45.324127667+02:00" level=debug msg="updated volume" module=scheduler node.id=xajfahck1rppdkh1s88haz400 volume.id=goezb38l6wk050isx2qwfkxqv

Wehen trying to mount the volume i get this error

Jul 13 13:21:08 builder dockerd[69345]: time="2024-07-13T13:21:08.741832071+02:00" level=info msg="attempting to publish volume" attempt=7 module=node/agent/csi volume.id=tpw9rgtlofwp7i1pt1j4r4ak2
Jul 13 13:21:08 builder dockerd[69345]: time="2024-07-13T13:21:08+02:00" level=error msg="I0713 11:21:08.746976 nodeserver.go:34 node stage volume testvolume to /data/staged/tpw9rgtlofwp7i1pt1j4r4ak2" plugin=321af86c8b8cb7d10d5d7e43dd08e6a10022635f3fab43c2a4eab791ee25c5ab
Jul 13 13:21:08 builder dockerd[69345]: time="2024-07-13T13:21:08+02:00" level=error msg="I0713 11:21:08.747216 mounter_seaweedfs.go:45 mounting [10.0.0.3:8888] /buckets/testvolume to /data/staged/tpw9rgtlofwp7i1pt1j4r4ak2" plugin=321af86c8b8cb7d10d5d7e43dd08e6a10022635f3fab43c2a4eab791ee25c5ab
Jul 13 13:21:08 builder dockerd[69345]: time="2024-07-13T13:21:08+02:00" level=error msg="W0713 11:21:08.747300 mounter_seaweedfs.go:139 VolumeContext 'path' ignored" plugin=321af86c8b8cb7d10d5d7e43dd08e6a10022635f3fab43c2a4eab791ee25c5ab
Jul 13 13:21:08 builder dockerd[69345]: time="2024-07-13T13:21:08+02:00" level=error msg="I0713 11:21:08.747384 mounter.go:53 Mounting fuse with command: weed and args: [-logtostderr=true mount -dirAutoCreate=true -umask=000 -dir=/data/staged/tpw9rgtlofwp7i1pt1j4r4ak2 -localSocket=/tmp/seaweedfs-mount-196015100.sock -cacheDir=/tmp/testvolume -collection=testvolume -filer.path=/buckets/testvolume -concurrentWriters=32 -filer=10.0.0.3:8888 -cacheCapacityMB=0]" plugin=321af86c8b8cb7d10d5d7e43dd08e6a10022635f3fab43c2a4eab791ee25c5ab
Jul 13 13:21:08 builder dockerd[69345]: time="2024-07-13T13:21:08+02:00" level=info msg="mount point owner uid=0 gid=0 mode=drwxrwxrwx" plugin=321af86c8b8cb7d10d5d7e43dd08e6a10022635f3fab43c2a4eab791ee25c5ab
Jul 13 13:21:08 builder dockerd[69345]: time="2024-07-13T13:21:08+02:00" level=info msg="current uid=0 gid=0" plugin=321af86c8b8cb7d10d5d7e43dd08e6a10022635f3fab43c2a4eab791ee25c5ab
Jul 13 13:21:08 builder dockerd[69345]: time="2024-07-13T13:21:08+02:00" level=error msg="I0713 11:21:08.809603 leveldb_store.go:47 filer store dir: /tmp/testvolume/774824a5/meta" plugin=321af86c8b8cb7d10d5d7e43dd08e6a10022635f3fab43c2a4eab791ee25c5ab
Jul 13 13:21:08 builder dockerd[69345]: time="2024-07-13T13:21:08+02:00" level=error msg="I0713 11:21:08.809678 file_util.go:27 Folder /tmp/testvolume/774824a5/meta Permission: -rwxr-xr-x" plugin=321af86c8b8cb7d10d5d7e43dd08e6a10022635f3fab43c2a4eab791ee25c5ab
Jul 13 13:21:08 builder dockerd[69345]: time="2024-07-13T13:21:08+02:00" level=error msg="/bin/fusermount: fuse device not found, try 'modprobe fuse' first" plugin=321af86c8b8cb7d10d5d7e43dd08e6a10022635f3fab43c2a4eab791ee25c5ab
Jul 13 13:21:08 builder dockerd[69345]: time="2024-07-13T13:21:08+02:00" level=error msg="F0713 11:21:08.820251 mount_std.go:257 Mount fail: fusermount exited with code 256" plugin=321af86c8b8cb7d10d5d7e43dd08e6a10022635f3fab43c2a4eab791ee25c5ab
Jul 13 13:21:08 builder dockerd[69345]: time="2024-07-13T13:21:08+02:00" level=error msg="E0713 11:21:08.822759 mounter.go:75 weed mount exit, pid: 104, path: /data/staged/tpw9rgtlofwp7i1pt1j4r4ak2, error: exit status 255" plugin=321af86c8b8cb7d10d5d7e43dd08e6a10022635f3fab43c2a4eab791ee25c5ab

Gradlon avatar Jul 13 '24 11:07 Gradlon

@s4ke The CSI Plugin is working!

I mapped most of the supported Driver options. As you can see in the config.

I did no excessive testing yet (Works on my machine;)

There is one thing that bothers me is that the target directory on seaweed is always /buckets/{VolumeName} @chrislusf After reading more into the mounter code, I can see that is configurable, can you help me on this one?

Also, do you want to integrate my code into this repo, should I create a pull request?

Gradlon avatar Jul 13 '24 12:07 Gradlon

/buckets/ was originally configuration, but later removed because of complexity.

Yes. This PR is welcome!

chrislusf avatar Jul 13 '24 16:07 chrislusf

Do you have any idea how I can pass this as an argument? Somehow i can't find out how to make it work.

Gradlon avatar Jul 13 '24 17:07 Gradlon

/buckets/ is not configurable.

chrislusf avatar Jul 13 '24 17:07 chrislusf

How hard would it be to make it configurable?

Gradlon avatar Jul 13 '24 18:07 Gradlon

fairly hard.

chrislusf avatar Jul 13 '24 18:07 chrislusf

Ok, that is unfortunate.

I wunder, when looking at the code

	if params == nil {
		params = make(map[string]string)
	}
	glog.V(4).Infof("params:%v", params)
	capacity := req.GetCapacityRange().GetRequiredBytes()

	if err := filer_pb.Mkdir(cs.Driver, "/buckets", volumeId, nil); err != nil {
		return nil, fmt.Errorf("error setting bucket metadata: %v", err)
	}

It looks feasible to me. But also I don't have the entire picture and a rather naive approach to it. params = make(map[string]string) those should be the parameters passed with the -opt flag while creating the volume, right?

Please excuse me for insisting, for me, it makes not that much sense to have my docker volumes stored in folder called buckets :/

Would you be willing to invest the time to implement this?

Gradlon avatar Jul 13 '24 18:07 Gradlon

Pull request is ready. The plugin works as expected. I had some issues using it, but they are probably more related to some issue with the CSI implementation itself. https://github.com/seaweedfs/seaweedfs-csi-driver/pull/173

Gradlon avatar Jul 17 '24 13:07 Gradlon