fn icon indicating copy to clipboard operation
fn copied to clipboard

Published 20 hours ago • verified publisher icon fnproject

API error (500): invalid mode: /tmp/iofs

Open Daniel15 opened this issue 5 years ago • 16 comments

Description Unable to execute any functions :(

Steps to reproduce the issue:

  1. Downloaded latest version of fn.exe CLI
  2. Ran fn start
  3. Created new function as per quickstart guide (tried both Node.js and Python)
  4. Tried executing function using fn invoke

Describe the results you received:

λ fn invoke helloapp hellopy
Error invoking function. status: 500 message: internal server error

Error:

time="2019-03-30T15:49:51Z" level=info msg="Fn serving on `:8080`" type=full
time="2019-03-30T15:50:19Z" level=error msg="Could not create container" app_id=01D77KZNGHNG8G00GZJ0000001 cpus= error="API error (500): invalid mode: /tmp/iofs" fn_id=01D77MHK5YNG8G00GZJ0000003 id=01D77MPSYDNG8G00GZJ0000002 idle_timeout=30 image="hellopy:0.0.2" memory=256 stack=CreateContainer
time="2019-03-30T15:50:19Z" level=error msg="internal server error" action="server.handleFnInvokeCall)-fm" error="API error (500): invalid mode: /tmp/iofs" fn_id=01D77MHK5YNG8G00GZJ0000003 stack="goroutine 127 [running]:
runtime/debug.Stack(0xc42047fce0, 0x1297ca0, 0xc4204ab960)
	/usr/local/go/src/runtime/debug/stack.go:24 +0xa7
github.com/fnproject/fn/api/server.HandleErrorResponse(0x12a3fc0, 0xc4201a96e0, 0x7f5dad00e7c0, 0xc4200e82c0, 0x1297ca0, 0xc4204ab960)
	/go/src/github.com/fnproject/fn/api/server/error_response.go:57 +0x57f
github.com/fnproject/fn/api/server.handleErrorResponse(0xc4200e82c0, 0x1297ca0, 0xc4204ab960)
	/go/src/github.com/fnproject/fn/api/server/error_response.go:25 +0xa9
github.com/fnproject/fn/api/server.(*Server).handleFnInvokeCall(0xc42040eb40, 0xc4200e82c0)
	/go/src/github.com/fnproject/fn/api/server/runner_fninvoke.go:53 +0x1da
github.com/fnproject/fn/api/server.(*Server).(github.com/fnproject/fn/api/server.handleFnInvokeCall)-fm(0xc4200e82c0)
	/go/src/github.com/fnproject/fn/api/server/server.go:1198 +0x34
github.com/fnproject/fn/vendor/github.com/gin-gonic/gin.(*Context).Next(0xc4200e82c0)
	/go/src/github.com/fnproject/fn/vendor/github.com/gin-gonic/gin/context.go:108 +0x43
github.com/fnproject/fn/api/server.(*Server).runMiddleware(0xc42040eb40, 0xc4200e82c0, 0x0, 0x0, 0x0)
	/go/src/github.com/fnproject/fn/api/server/middleware.go:42 +0x29f
github.com/fnproject/fn/api/server.(*Server).rootMiddlewareWrapper.func1(0xc4200e82c0)
	/go/src/github.com/fnproject/fn/api/server/middleware.go:32 +0x58
github.com/fnproject/fn/vendor/github.com/gin-gonic/gin.(*Context).Next(0xc4200e82c0)
	/go/src/github.com/fnproject/fn/vendor/github.com/gin-gonic/gin/context.go:108 +0x43
github.com/fnproject/fn/api/server.panicWrap(0xc4200e82c0)
	/go/src/github.com/fnproject/fn/api/server/gin_middlewares.go:190 +0x51
github.com/fnproject/fn/vendor/github.com/gin-gonic/gin.(*Context).Next(0xc4200e82c0)
	/go/src/github.com/fnproject/fn/vendor/github.com/gin-gonic/gin/context.go:108 +0x43
github.com/fnproject/fn/api/server.panicWrap(0xc4200e82c0)
	/go/src/github.com/fnproject/fn/api/server/gin_middlewares.go:190 +0x51
github.com/fnproject/fn/vendor/github.com/gin-gonic/gin.(*Context).Next(0xc4200e82c0)
	/go/src/github.com/fnproject/fn/vendor/github.com/gin-gonic/gin/context.go:108 +0x43
github.com/fnproject/fn/api/server.apiMetricsWrap.func1.1(0xc4200e82c0)
	/go/src/github.com/fnproject/fn/api/server/gin_middlewares.go:154 +0x2f8
github.com/fnproject/fn/vendor/github.com/gin-gonic/gin.(*Context).Next(0xc4200e82c0)
	/go/src/github.com/fnproject/fn/vendor/github.com/gin-gonic/gin/context.go:108 +0x43
github.com/fnproject/fn/api/server.traceWrap(0xc4200e82c0)
	/go/src/github.com/fnproject/fn/api/server/gin_middlewares.go:97 +0x507
github.com/fnproject/fn/vendor/github.com/gin-gonic/gin.(*Context).Next(0xc4200e82c0)
	/go/src/github.com/fnproject/fn/vendor/github.com/gin-gonic/gin/context.go:108 +0x43
github.com/fnproject/fn/api/server.loggerWrap(0xc4200e82c0)
	/go/src/github.com/fnproject/fn/api/server/gin_middlewares.go:212 +0x1ea
github.com/fnproject/fn/vendor/github.com/gin-gonic/gin.(*Context).Next(0xc4200e82c0)
	/go/src/github.com/fnproject/fn/vendor/github.com/gin-gonic/gin/context.go:108 +0x43
github.com/fnproject/fn/vendor/github.com/gin-gonic/gin.(*Engine).handleHTTPRequest(0xc42040ea20, 0xc4200e82c0)
	/go/src/github.com/fnproject/fn/vendor/github.com/gin-gonic/gin/gin.go:361 +0x586
github.com/fnproject/fn/vendor/github.com/gin-gonic/gin.(*Engine).ServeHTTP(0xc42040ea20, 0x12a3940, 0xc420495630, 0xc42042ed00)
	/go/src/github.com/fnproject/fn/vendor/github.com/gin-gonic/gin/gin.go:326 +0x153
github.com/fnproject/fn/vendor/go.opencensus.io/plugin/ochttp.(*Handler).ServeHTTP(0xc4200adf40, 0x12a3940, 0xc420495630, 0xc420152c00)
	/go/src/github.com/fnproject/fn/vendor/go.opencensus.io/plugin/ochttp/server.go:86 +0x1c8
net/http.serverHandler.ServeHTTP(0xc42040ab60, 0x12a25c0, 0xc420242460, 0xc420152c00)
	/usr/local/go/src/net/http/server.go:2697 +0xbc
net/http.(*conn).serve(0xc4200d20a0, 0x12a3f00, 0xc420503240)
	/usr/local/go/src/net/http/server.go:1830 +0x651
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2798 +0x27b
"

Describe the results you expected: Wanted it to work :(

Additional information you deem important (e.g. issue happens only occasionally): Happens 100% of the time

Output of fn version (CLI command):

Client version is latest version: 0.5.65
Server version:  0.3.682

Additional environment details (OSX, Linux, flags, etc.): Windows 10, 1809

Daniel15 avatar Mar 30 '19 15:03 Daniel15

Hi, thanks for reporting. But here’s the thing, you have problems creating containers. That’s what you have in your logs.

Make sure you can do that and Fn has reliable connection with Docker.

Also, FYI, we don’t have any guarantees that Fn works on Windows. So, you may face with problems.

denismakogon avatar Mar 30 '19 15:03 denismakogon

But here’s the thing, you have problems creating containers.

The Fdk server itself is a container though, and that's running fine?

λ docker ps
CONTAINER ID        IMAGE                COMMAND             CREATED             STATUS              PORTS
                NAMES
df875f5e35fd        fnproject/fnserver   "./fnserver"        10 minutes ago      Up 10 minutes       2375/tcp, 0.0.0.0:8080->8080/tcp   fnserver

I don't have issues with other Docker containers.

FYI, we don’t have any guarantees that Fn works on Windows. So, you may face with problems.

OK. I can try to use a VM instead. I assume a local fn client can connect to a remote server?

Daniel15 avatar Mar 30 '19 16:03 Daniel15

Thanks for raising this!

FWIW what is happening here is that Fn server is creating a directory to share unix sockets between it and Fn containers - because fn is itself running in docker on the desktop it needs to share that directory with the docker host. :

e.g. (on OSX) :

  • Fn cli starts fn server with /Users/Me/.fn/iofs/ --> mounted into /iofs/ in the fnserver (this directory has to be on the host as d4m only lets you mount volumes from the host OS into a container, not from the VM that docker is running on)
  • Fn server starts a container , creates a socket dir /iofs/container1/-> /Users/Me/.fn/iofs/container1/ and it asks docker to mount /Users/Me/.fn/iofs/container1/ into /iofs on the actual function container. - fn server start waiting for container to open a listening socket in its iofs directory
  • Container starts, opens a socket, starts listening for connections on /iofs/xyz.sock
  • fn server notices this file exists (using inotify) - starts sending messages to container.

with Docker4Mac we hit some problems with what could and couldn't be shared with docker (e.g. user Home is sharable, /tmp/ on the host is not) - I wonder if we're hitting something similar here (or if windows file names are themselves causing a problem) .

I'm just trying to find a windows 10 pro env I can test on , while I'm doing that there are some environment variables that you can set on the fn server that might help out:

FN_IOFS_DOCKER_PATH - is the host path e.g. that the iofs shared dir exists on- this needs to be a directory that is shared with docker (e.g. d:\docker) - I wonder if setting this to a directory you created (e.g. "d:/path/to/iofs/ ) will work

FN_IOFS_PATH - this is the path that the above path is mounted into the fnserver container

It's a guess but I'm curious if the following would work:

mkdir d:\fndata

docker run --rm -i --name fnserver -v d:/fndata/:/iofs -e FN_IOFS_DOCKER_PATH=d:/fndata \
   -e FN_IOFS_PATH=/iofs  -v /var/run/docker.sock:/var/run/docker.sock  -p 8080:8080 \
   --entrypoint ./fnserver fnproject/fnserver

zootalures avatar Mar 30 '19 16:03 zootalures

Thanks for the detailed comment @zootalures! I tried the command you mentioned (except replacing d: with c: since I have no D: drive):

λ docker run --rm -i --name fnserver -v c:/fndata/:/iofs -e FN_IOFS_DOCKER_PATH=c:/fndata -e FN_IOFS_PATH=/iofs  -v /var/run/docker.sock:/var/run/docker.sock  -p 8080:8080 --entrypoint ./fnserver fnproject/fnserver
time="2019-03-30T16:33:01Z" level=info msg="Registering container driver 'docker'"
time="2019-03-30T16:33:01Z" level=info msg="Registering log provider 's3'"
time="2019-03-30T16:33:01Z" level=info msg="Registering data store provider 'sql'"
time="2019-03-30T16:33:01Z" level=info msg="Registering log provider 'sql'"
time="2019-03-30T16:33:01Z" level=info msg="Registering sql helper 'mysql'"
time="2019-03-30T16:33:01Z" level=info msg="Registering sql helper 'postgres'"
time="2019-03-30T16:33:01Z" level=info msg="Registering sql helper 'sqlite'"
time="2019-03-30T16:33:01Z" level=info msg="Setting log level to" fields.level=info
time="2019-03-30T16:33:01Z" level=info msg="Connecting to DB" url="sqlite3:///app/data/fn.db"
time="2019-03-30T16:33:01Z" level=info msg="datastore dialed" datastore=sqlite3 max_idle_connections=256 url="sqlite3:///app/data/fn.db"
time="2019-03-30T16:33:01Z" level=info msg="agent starting cfg={MinDockerVersion:17.10.0-ce ContainerLabelTag: DockerNetworks: DockerLoadFile: FreezeIdle:50ms HotPoll:200ms HotLauncherTimeout:1h0m0s HotPullTimeout:10m0s HotStartTimeout:5s AsyncChewPoll:1m0s DetachedHeadRoom:6m0s MaxResponseSize:0 MaxHdrResponseSize:0 MaxLogSize:1048576 MaxTotalCPU:0 MaxTotalMemory:0 MaxFsSize:0 PreForkPoolSize:0 PreForkImage:busybox PreForkCmd:tail -f /dev/null PreForkUseOnce:0 PreForkNetworks: EnableNBResourceTracker:false MaxTmpFsInodes:0 DisableReadOnlyRootFs:false DisableDebugUserLogs:false IOFSEnableTmpfs:false IOFSAgentPath:/iofs IOFSMountRoot:c:/fndata IOFSOpts: ImageCleanMaxSize:0 ImageCleanExemptTags: ImageEnableVolume:false}"
time="2019-03-30T16:33:01Z" level=info msg="no docker auths from config files found (this is fine)" error="open /root/.dockercfg: no such file or directory"
time="2019-03-30T16:33:01Z" level=info msg="available memory" cgroup_limit=9223372036854771712 head_room=268435456 total_memory=1377939456
time="2019-03-30T16:33:01Z" level=info msg="ram reservations" avail_memory=1109504000 ram_async_hw_mark=887603200
time="2019-03-30T16:33:01Z" level=info msg="available cpu" avail_cpu=2000 total_cpu=2000
time="2019-03-30T16:33:01Z" level=info msg="cpu reservations" cpu=2000 cpu_async_hw_mark=1600

        ______
       / ____/___
      / /_  / __ \
     / __/ / / / /
    /_/   /_/ /_/
        v0.3.682

time="2019-03-30T16:33:01Z" level=info msg="Fn serving on `:8080`" type=full

Then when I do fn invoke helloapp hellopy, it still mentions /tmp/iofs:

time="2019-03-30T16:33:51Z" level=error msg="Could not create container" app_id=01D77Q5XT2NG8G00GZJ0000001 cpus= error="API error (500): invalid mode: /tmp/iofs" fn_id=01D77Q6AN1NG8G00GZJ0000002 id=01D77Q6GZ3NG8G00GZJ0000004 idle_timeout=30 image="hellopy:0.0.3" memory=256 stack=CreateContainer

Is that supposed to mention /iofs instead?

If I swap the entry point from ./fnserver to env, it does indeed show the environment variable:

λ docker run --rm -i --name fnserver -v c:/fndata/:/iofs -e FN_IOFS_DOCKER_PATH=c:/fndata -e FN_IOFS_PATH=/iofs  -v /var/run/docker.sock:/var/run/docker.sock  -p 8080:8080 --entrypoint env fnproject/fnserver
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=1242416e8d7c
FN_IOFS_DOCKER_PATH=c:/fndata
FN_IOFS_PATH=/iofs
DOCKER_CHANNEL=stable
DOCKER_VERSION=17.12.0-ce
DIND_COMMIT=3b5fac462d21ca164b3778647420016315289034
HOME=/root

Daniel15 avatar Mar 30 '19 16:03 Daniel15

Hmm odd, /tmp/iofs is the default so something isn’t right here

zootalures avatar Mar 30 '19 17:03 zootalures

Hmm actually /tmp/iofs is the container mount inside the function container (So it should be trying to mount c:/fndata/xyzzy/ ->/tmp/iofs in the function) . I think we mount this with some flags and I wonder if windows docker for windows doesn’t support them , going to look in detail in a bit (and try it once this 15GB Windows image has downloaded!)

zootalures avatar Mar 30 '19 17:03 zootalures

I’m not sure what d4w does but also suspect inotify wont work, we can work around that with polling though

zootalures avatar Mar 30 '19 17:03 zootalures

Ok - confirmed a few things are going on here:

  • We need to make sure directories exist prior to mounting on windows (d4w won't create a source dir as part of a bind mount as for linux)
  • unix sockets do_not work when mounted via host volumes on d4m - there is some magic for /var/run/docker.sock which does not translate over the standard (cifs) host volume shares
  • paths needed to be treated carefully - all forward slashes etc.

For unix sockets we end up in a similar situation as we had for mac except we could create them on xyhve shared volume mounts but can’t on windows-

There is a gnarly solution which is insecure :

  • fn cli ensures that a docker volume "fniofs" exists,
  • starts fn with this mounted with some flags
  • fn starts containers with this mounted and tells them where to put their unix sockets.

the downside here is that each function can see each other functions sockets - v bad but it would work I think.

Docker doesn't support mounting paths of volumes, and unless I'm mistaken this is the only way to get a pure-blood linux dir that supports unix sockets and can be shared between containers in d4w

zootalures avatar Mar 31 '19 16:03 zootalures

Thank you so much for the investigation, @zootalures!

the downside here is that each function can see each other functions sockets - v bad but it would work I think.

I don't think anyone would actually use Docker for Windows in production, so maybe a warning on startup would suffice? (eg. "Due to Windows limitations, Fn is running in an INSECURE mode that is only for development purposes!")

Daniel15 avatar Mar 31 '19 17:03 Daniel15

yep, agreed, going to do a bit more research but that's where it'll be if I can't solve another way

zootalures avatar Mar 31 '19 17:03 zootalures

Any work around for this? Hitting the same problem on windows 10 with docker desktop as well.

wenjian80 avatar Apr 01 '20 04:04 wenjian80

I got it to work by installing fn inside WSL 2 and running them from there,

andrew24601 avatar Jul 15 '20 02:07 andrew24601

Its still not working with latest fn sdk on windows 10 pro 64 bit os.

fn version Client version is latest version: 0.5.99 Server version: 0.3.749

Tried with latest docker desktop community edition v2.3.0.4. Its giving same error as mentioned above:

time="2020-08-01T07:41:14Z" level=info msg="container wait error, sending error to client" action="server.handleFnInvokeCall)-fm" app_id=01EEMFF5WTNG8G00GZJ0000001 call_id=01EEMFFEFWNG8G00GZJ0000003 error="API error (500): invalid mode: /tmp/iofs" fn_id=01EEMFFBEKNG8G00GZJ0000002

Note: C:\Users\ABC.fn\data and C:\Users\ABC.fn\iofs are successfully configured in Resources >> File Sharing Settings of docker desktop community edition. I confirm these folder does exists at the mentioned path in filesystem.

Tried running with following cmd: docker run --rm -i --name fnserver -v C:/Users/ABC/.fn/iofs/:/iofs -e FN_IOFS_DOCKER_PATH=C:/Users/ABC/.fn/iofs -e FN_IOFS_PATH=/iofs -v /var/run/docker.sock:/var/run/docker.sock -p 8080:8080 --entrypoint ./fnserver fnproject/fnserver

I understand fn won't be used for production in windows, but it will be good if its supported for development or else kindly document that latest fn sdk does not works on windows 10 64 bit yet.

aaraodeo avatar Aug 01 '20 16:08 aaraodeo

Any solution for above ??

bhushankhaire avatar Aug 18 '20 12:08 bhushankhaire

See if you can get it running under WSL. I can run the example scripts successfully under both:

  • Windows 10 Pro 64 bit using WSL
  • Windows 10 Home 64 bit using WSL 2

In both cases I was also using file system paths on the WSL file system, not on the /mnt/c path.

andrew24601 avatar Aug 18 '20 20:08 andrew24601

Is this related 'docker bind propagation'? Does fn server require this feature??

https://docs.docker.com/storage/bind-mounts/#configure-bind-propagation

Configure bind propagation

Bind propagation defaults to rprivate for both bind mounts and volumes. It is only configurable for bind mounts, and only on Linux host machines. Bind propagation is an advanced topic and many users never need to configure it.

Bind propagation refers to whether or not mounts created within a given bind-mount can be propagated to replicas of that mount. Consider a mount point /mnt, which is also mounted on /tmp. The propagation settings control whether a mount on /tmp/a would also be available on /mnt/a. Each propagation setting has a recursive counterpoint. In the case of recursion, consider that /tmp/a is also mounted as /foo. The propagation settings control whether /mnt/a and/or /tmp/a would exist.

TheKoguryo avatar Jan 27 '23 01:01 TheKoguryo