fn
fn copied to clipboard
fnproject
Some help required running Fn on Kubernetes
Hello all. I've been enjoying trying out Fn and things seem to be working pretty well for me running a simple server with fn start and even running my own server using docker run, so I'm good to go on simple use-cases. I've come a bit unstuck however trying to get Fn running on Kubernetes.
I'm aware that this is a distributed configuration involving several different fnproject/fnserver Node Types and have read up on that topology here https://medium.com/fnproject/distributed-fn-beyond-fn-start-2ee0baf6fb77 and here https://github.com/fnproject/docs/blob/master/fn/operate/production.md
I'm aware of the Fn helm chart https://github.com/fnproject/fn-helm and I'm using that as my starting point but to try and get my head around things a little my goal was to try and get things running using a simple NodePort service type to run on my local machine.
I'm running a very simple local docker registry using the registry:2 image on localhost:5000 and my setup is working fine with OpenFaaS, which I set up as a control to try and rule out any silly issues with my basic set-up.
I'll try to give some detail of exactly where I'm at as follows:
I created a deployment fn.yaml using the helm chart as a template e.g.
git clone https://github.com/fnproject/fn-helm.git
helm dep build fn-helm/fn
helm template --name fn --namespace fn fn-helm/fn > fn.yaml
to keep things tidy I created a fn namespace on my K8s cluster
kubectl create namespace fn
Next I edited the fn.yaml to include the fn namespace and add the NodePort stuff. The relevant parts are below, apart from adding the fn namespace I haven't changed the fn-mysql, fn-redis, fn-fn-flow or fn-fn-ui parts - I'm just focussing on the core function CRUD and invocation parts ATM so the ke parts are:
---
# Source: fn/templates/fn-runner-node-service.yaml
apiVersion: v1
kind: Service
metadata:
name: fn-fn-runner
namespace: "fn"
labels:
app: fn-fn
chart: fn-0.1.0
heritage: Tiller
release: fn
spec:
type: ClusterIP
ports:
- name: grpc
port: 9191
targetPort: 9191
selector:
app: fn-fn
role: runner
---
# Source: fn/templates/fn-service.yaml
apiVersion: v1
kind: Service
metadata:
name: fn-fn
namespace: "fn"
labels:
app: fn-fn
chart: fn-0.1.0
heritage: Tiller
release: fn
spec:
# type: ClusterIP
type: NodePort
ports:
- name: api
port: 80
targetPort: 80
nodePort: 30080
- name: runner-lb
port: 90
targetPort: 90
nodePort: 30090
selector:
app: fn-fn
role: fn-service
---
---
# Source: fn/templates/fn-deployment.yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: fn-fn
namespace: "fn"
labels:
app: fn-fn
chart: fn-0.1.0
heritage: Tiller
release: fn
spec:
replicas: 1
template:
metadata:
labels:
app: fn-fn
chart: fn-0.1.0
heritage: Tiller
release: fn
role: fn-service
spec:
containers:
- name: api
image: fnproject/fnserver:latest
imagePullPolicy: Always
resources:
limits:
cpu: 300m
memory: 2Gi
requests:
cpu: 150m
memory: 512Mi
securityContext:
privileged: true
ports:
- containerPort: 80
readinessProbe:
httpGet:
path: /v2/apps
port: 80
initialDelaySeconds: 3
periodSeconds: 3
env:
- name: FN_DB_PASSWD
valueFrom:
secretKeyRef:
name: fn-mysql
key: mysql-password
- name: FN_DB_HOST
value: fn-mysql
- name: FN_MQ_HOST
value: fn-redis
- name: FN_PORT
value: "80"
- name: FN_NODE_TYPE
value: "api"
- name: FN_PUBLIC_LB_URL
# value: http://fn.lb.fn.internal:90
value: http://10.192.0.2:30090
- name: FN_DB_URL
value: "mysql://fnapp:$(FN_DB_PASSWD)@tcp($(FN_DB_HOST):3306)/fndb"
- name: FN_LOG_LEVEL
value: "INFO"
- name: FN_MQ_URL
value: "redis://$(FN_MQ_HOST):6379/"
- name: runner-lb
image: fnproject/fnserver:latest
imagePullPolicy: Always
resources:
limits:
cpu: 300m
memory: 2Gi
requests:
cpu: 150m
memory: 512Mi
securityContext:
privileged: true
ports:
- containerPort: 90
env:
- name: FN_NODE_TYPE
value: "lb"
- name: FN_GRPC_PORT
value: "9191"
- name: FN_PORT
value: "90"
- name: FN_RUNNER_API_URL
value: http://fn-fn.fn.svc.cluster.local:80
- name: FN_RUNNER_ADDRESSES
value: fn-fn-runner.fn.svc.cluster.local:9191
- name: FN_LOG_LEVEL
value: "INFO"
---
# Source: fn/templates/fn-runner-node-deployment.yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: fn-fn-runner
namespace: "fn"
labels:
app: fn-fn
chart: fn-0.1.0
heritage: Tiller
release: fn
spec:
replicas: 3
template:
metadata:
labels:
app: fn-fn
chart: fn-0.1.0
heritage: Tiller
release: fn
role: runner
spec:
containers:
- name: runner
image: fnproject/fnserver:latest
imagePullPolicy: Always
resources:
limits:
cpu: 500m
memory: 5Gi
requests:
cpu: 250m
memory: 1Gi
securityContext:
privileged: true
ports:
- containerPort: 9191
env:
- name: FN_NODE_TYPE
value: "pure-runner"
- name: FN_GRPC_PORT
value: "9191"
- name: FN_LOG_LEVEL
value: "INFO"
---
The relevent bits really are that in the fn-service I changed the type to NodePort and added nodePort: 30080 to api and nodePort: 30090 to runner-lb. The other change is that under fn-deployment I changed the FN_PUBLIC_LB_URL value from http://fn.lb.fn.internal:90 to http://10.192.0.2:30090
When I deploy this with
kubectl apply -f ./fn.yaml
I see a successful deployment in the Kubernetes dashboard. I have set up my context to point to the required API URL and registry:
fn list contexts
CURRENT NAME PROVIDER API URL REGISTRY
* default default http://10.192.0.2:30080 localhost:5000
and fn version is happy, correctly returning the server version:
fn version
Client version is latest version: 0.5.69
Server version: 0.3.687
As far as I can see it looks like my API is running fine and I seem to be able to correctly run the CRUD operations, my issue is with the fn invoke, which is clearly rather important :-)
I've tested using http://fnproject.io/tutorials/Introduction/ as follows:
fn init --runtime go --trigger http gofn
Creating function at: /gofn
Function boilerplate generated.
func.yaml created.
fn create app goapp
Successfully created app: goapp
fn --verbose deploy --app goapp
Deploying gofn to app: goapp
Bumped to version 0.0.2
Building image localhost:5000/gofn:0.0.2
FN_REGISTRY: localhost:5000
Current Context: default
Sending build context to Docker daemon 5.12kB
Step 1/10 : FROM fnproject/go:dev as build-stage
dev: Pulling from fnproject/go
cd784148e348: Pull complete
7e273b0dfc44: Pull complete
952c3806fd1a: Pull complete
daee55ea059d: Pull complete
18dd7cffb4bd: Pull complete
02ef0ff2b601: Pull complete
Digest: sha256:fad7d110064e144fe9de90c2199146d83e6f9fdd6d6129bba1e7c7b67b81f8a3
Status: Downloaded newer image for fnproject/go:dev
---> 96c8fb94a8e1
Step 2/10 : WORKDIR /function
---> Running in 10d0f4cbf78c
Removing intermediate container 10d0f4cbf78c
---> d6fe38e1aa4a
Step 3/10 : RUN go get -u github.com/golang/dep/cmd/dep
---> Running in 04d806267d73
Removing intermediate container 04d806267d73
---> a6b205de47e7
Step 4/10 : ADD . /go/src/func/
---> 48fd62a784e7
Step 5/10 : RUN cd /go/src/func/ && dep ensure
---> Running in 7e224f97c846
Removing intermediate container 7e224f97c846
---> 5c285fadec58
Step 6/10 : RUN cd /go/src/func/ && go build -o func
---> Running in 32ff032d1cdd
Removing intermediate container 32ff032d1cdd
---> 294a608d7833
Step 7/10 : FROM fnproject/go
latest: Pulling from fnproject/go
ba7f5deea89d: Pull complete
a243235fe902: Pull complete
Digest: sha256:d30b00a023590955fc450d109bf57a649f69f7403fec164912dc85ade4749dc8
Status: Downloaded newer image for fnproject/go:latest
---> d9de4156fe50
Step 8/10 : WORKDIR /function
---> Running in eac3b7cc6be0
Removing intermediate container eac3b7cc6be0
---> 31f13b455c60
Step 9/10 : COPY --from=build-stage /go/src/func/func /function/
---> 945549427c78
Step 10/10 : ENTRYPOINT ["./func"]
---> Running in 43bf9010a9bf
Removing intermediate container 43bf9010a9bf
---> 24969be1ac7a
Successfully built 24969be1ac7a
Successfully tagged localhost:5000/gofn:0.0.2
Parts: [localhost:5000 gofn:0.0.2]
Pushing localhost:5000/gofn:0.0.2 to docker registry...The push refers to repository [localhost:5000/gofn]
27e712241c73: Pushed
6009b4785a95: Pushed
db794edf2401: Pushed
0b32a561a108: Pushed
0.0.2: digest: sha256:d438a665d0d227142d5066994a67a8a48c14b8689e742a3b319080a3deb15418 size: 1157
Updating function gofn using image localhost:5000/gofn:0.0.2...
Successfully created function: gofn with localhost:5000/gofn:0.0.2
Successfully created trigger: gofn
Trigger Endpoint: http://10.192.0.2:30090/t/goapp/gofn
As far as I can see this all looks good so far and the Trigger Endpoint port is what I think I'm expecting it to be given how I set up the fn-service NodePort
and if I check my local registry by browsing to
http://localhost:5000/v2/_catalog
I can see gofn.
however.........
If I now do
fn invoke goapp gofn
The call hangs for a while, before eventually returning with:
Error invoking function. status: 500 message: internal server error
Following the instructions in https://medium.com/fnproject/distributed-fn-beyond-fn-start-2ee0baf6fb77 I tried:
fn inspect fn goapp gofn
which returns:
{
"annotations": {
"fnproject.io/fn/invokeEndpoint": "http://10.192.0.2:30090/invoke/01D7M8KVR41BT061G0A0000002"
},
"app_id": "01D7M8GH271BT061G0A0000001",
"created_at": "2019-04-04T13:29:07.588Z",
"id": "01D7M8KVR41BT061G0A0000002",
"idle_timeout": 30,
"image": "localhost:5000/gofn:0.0.2",
"memory": 128,
"name": "gofn",
"timeout": 30,
"updated_at": "2019-04-04T13:29:07.588Z"
}
again the IP/port of the invokeEndpoint are what I think I was expecting.
If I do
curl http://10.192.0.2:30090
I see
{"goto":"https://github.com/fnproject/fn","hello":"world!"}
But I think that's the welcome from the base endpoint not the function 'cause the function has "Hello World" with capitals
curl http://10.192.0.2:30090/invoke/01D7M8KVR41BT061G0A0000002
yields
{"message":"Method not allowed: GET /invoke/01D7M8KVR41BT061G0A0000002"}
and
curl http://10.192.0.2:30090/t/goapp/gofn
which is the trigger endpoint hangs for a bit and returns
{"message":"internal server error"}
I'd really appreciate any thoughts on this, I appreciate that using NodePort isn't the recommended approach but as far as I can see it *should" be possible to make it work OK and much of what I'm seeing looks like I'm expecting (apart from the invoke not working of course :-D). As I mentioned previously I got OpenFaaS working on the same set up so I think the underlying infrastructure is basically sound.
MTIA
Hi!
curl http://10.192.0.2:30090/invoke/01D7M8KVR41BT061G0A0000002
Invoke API allows only HTTP POST.
curl http://10.192.0.2:30090/t/goapp/gofn which is the trigger endpoint hangs for a bit and returns {"message":"internal server error"}
Meaning that you have problems with communication between Fn API and Fn LB nodes. You'd need to investigate what's going in LB logs, i.e., why it gives 502 back.
Anyways, the recommended approach is to use an ingress controller or LoadBalancer, but not NodePort.
Hi, but that curl was only one aspect of the issue and doesn't explain why the basic
fn invoke goapp gofn
fails. I actually only tried the curl direct against the invoke endpoint to see what happens and as I also mentioned I tried a curl against the trigger endpoint too via
curl http://10.192.0.2:30090/t/goapp/gofn
That is the endpoint mentioned in the
fn --verbose deploy --app goapp
If I follow the "Invoke with Curl" from the tutorial http://fnproject.io/tutorials/Introduction/ replacing http://localhost:8080 with my host and port e.g.
curl -H "Content-Type: application/json" http://10.192.0.2:30090/t/goapp/gofn-trigger
I see it hang and eventually return
{"message":"internal server error"}
So the fact that I used GET on the invoke API for one case above is a bit of a red herring. Indeed if I do
curl -X POST http://10.192.0.2:30090/invoke/01D7M8KVR41BT061G0A0000
002
I see
{"message":"internal server error"}
I appreciate that " the recommended approach is to use an ingress controller or LoadBalancer, but not NodePort." and I mentioned that I was aware of that in my original post, but the point is that it should be able to make it work and the point of trying it is to see how it all hangs together so I was really looking for some advice in what I might be doing wrong in terms of the config because I thought that the main piece of config I needed was to set the FN_PUBLIC_LB_URL and I think that's looking OK.
Re "Meaning that you have problems with communication between Fn API and Fn LB nodes" I pretty much realised that, but the crux of my question was actually "why", which is why I was asking. I'm not entirely sure where to look - I was hoping my original post had enough detail for some insight.
Re "You'd need to investigate what's going in LB logs, i.e., why it gives 502 back." could you give some suggestions on which might be the most useful logs to look at and where they might be located - so far a basic stand alone Fn server has "just worked", so I've got limited experience digging around the Fn logs. Looking at the fn-fn-runner-" logs under Pods in the K8s dashboard doesn't yield too much and looks kind of what I see when I stand ap a Fn server stand alone that is the Pod logs end up with
...
...
2019-04-04T13:19:08.400288639Z time="2019-04-04T13:19:08Z" level=info msg="available memory" cgroup_limit=5368709120 head_room=536870912 total_memory=11276562432
2019-04-04T13:19:08.400321904Z time="2019-04-04T13:19:08Z" level=info msg="ram reservations" avail_memory=4831838208 ram_async_hw_mark=3865470566
2019-04-04T13:19:08.400856554Z time="2019-04-04T13:19:08Z" level=info msg="available cpu" avail_cpu=500 total_cpu=4000
2019-04-04T13:19:08.40104528Z time="2019-04-04T13:19:08Z" level=info msg="cpu reservations" cpu=500 cpu_async_hw_mark=400
2019-04-04T13:19:08.4010813Z time="2019-04-04T13:19:08Z" level=warning msg="Severaly Limited CPU: cpu < 1000m (1 CPU)"
2019-04-04T13:19:08.401087193Z time="2019-04-04T13:19:08Z" level=warning msg="Running pure runner in insecure mode!"
2019-04-04T13:19:08.401157101Z time="2019-04-04T13:19:08Z" level=info msg="Pure Runner listening on :9191"
2019-04-04T13:19:08.40152587Z
2019-04-04T13:19:08.401537152Z ______
2019-04-04T13:19:08.401539982Z / ____/___
2019-04-04T13:19:08.401542221Z / /_ / __ \
2019-04-04T13:19:08.401544498Z / __/ / / / /
2019-04-04T13:19:08.401546715Z /_/ /_/ /_/
2019-04-04T13:19:08.401548988Z v0.3.687
2019-04-04T13:19:08.401551252Z
2019-04-04T13:19:08.401597867Z time="2019-04-04T13:19:08Z" level=info msg="Fn serving on `:8080`" type=pure-runner
One thing I'm wondering about though, looking at the Kubernetes config I can't see any volume mounts referring to the Docker socket so am I correct in thinking that the Fn server containers are running in Docker in Docker mode? All the fnproject/fnserver containers have privileged: true in their securityContext which also makes me wonder if DinD is being used. I ask because another thing I'm wondering about is whether my issue is down to something not seeing the container registry - would there be some log message somewhere if the fnproject/fnserver containers couldn't see the registry and weren't able to pull the function container?
Thanks.
Looking at the fn-fn-runner-" logs
That's not where you should look for, lb node is the one which accepts invocations, you'd need to look there. You need to trace what's going on in lb container logs in order to see why the node gives you internal server error.
One thing I'm wondering about though, looking at the Kubernetes config I can't see any volume mounts referring to the Docker socket so am I correct in thinking that the Fn server containers are running in Docker in Docker mode?
true
Hey @fadams - a quick look at your kube config, and I'm guessing that the runner node(s) is missing a couple of env vars it needs to run functions. fn start is a decent display of them here https://github.com/fnproject/cli/blob/master/commands/start.go#L57 - though, you shouldn't need to mount through to a host volume, just specifying FN_IOFS_DOCKER_PATH and FN_IOFS_PATH might do the trick. we should add those to the image or make some defaults, but alas, hope that helps! I can try to look into this in more detail soon
if you can paste lb and runner logs here, it could also help (or slack may be faster). it may help to run FN_LOG_LEVEL=debug while this is an issue, as well, to at least figure out how far a request might be getting. the lb will have a log entry for the internal server error that was returned to you, at least, that would be the place to look.
@rdallman issue is not related to IOFS because with ingress controller or LoadBalancer service type deployment works as expected with no problems.
I feel like that’s something related to networking (caused by NodePort).
In the logs for runner-lb I'm seeing the following after the failed invoke
2019-04-04T17:26:39.433659239Z time="2019-04-04T17:26:39Z" level=info msg="Fn serving on `:90`" type=lb
2019-04-04T17:38:37.435994403Z time="2019-04-04T17:38:37Z" level=error msg="error from API server, retrying" error="Get http://fn-fn.fn.svc.cluster.local:80/v2/fns/01D7MPM07K1BT044G0A0000002: dial tcp: lookup fn-fn.fn.svc.cluster.local: Try again"
2019-04-04T17:38:42.541672752Z time="2019-04-04T17:38:42Z" level=error msg="error from API server, retrying" error="Get http://fn-fn.fn.svc.cluster.local:80/v2/fns/01D7MPM07K1BT044G0A0000002: dial tcp: lookup fn-fn.fn.svc.cluster.local: Try again"
2019-04-04T17:38:47.659563141Z time="2019-04-04T17:38:47Z" level=error msg="error from API server, retrying" error="Get http://fn-fn.fn.svc.cluster.local:80/v2/fns/01D7MPM07K1BT044G0A0000002: dial tcp: lookup fn-fn.fn.svc.cluster.local: Try again"
2019-04-04T17:38:53.039800675Z time="2019-04-04T17:38:53Z" level=error msg="error from API server, retrying" error="Get http://fn-fn.fn.svc.cluster.local:80/v2/fns/01D7MPM07K1BT044G0A0000002: dial tcp: lookup fn-fn.fn.svc.cluster.local: Try again"
2019-04-04T17:38:58.227511648Z time="2019-04-04T17:38:58Z" level=error msg="error from API server, retrying" error="Get http://fn-fn.fn.svc.cluster.local:80/v2/fns/01D7MPM07K1BT044G0A0000002: dial tcp: lookup fn-fn.fn.svc.cluster.local: Try again"
2019-04-04T17:39:04.534988301Z time="2019-04-04T17:39:04Z" level=error msg="error from API server, retrying" error="Get http://fn-fn.fn.svc.cluster.local:80/v2/fns/01D7MPM07K1BT044G0A0000002: dial tcp: lookup fn-fn.fn.svc.cluster.local: Try again"
2019-04-04T17:39:04.535546123Z time="2019-04-04T17:39:04Z" level=error msg="internal server error" action="server.handleFnInvokeCall)-fm" error="Get http://fn-fn.fn.svc.cluster.local:80/v2/fns/01D7MPM07K1BT044G0A0000002: dial tcp: lookup fn-fn.fn.svc.cluster.local: Try again" fn_id=01D7MPM07K1BT044G0A0000002 stack="goroutine 257 [running]:\nruntime/debug.Stack(0xc420082300, 0x1299360, 0xc4204a7c80)\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0xa7\ngithub.com/fnproject/fn/api/server.HandleErrorResponse(0x12a4540, 0xc42055c6c0, 0x7fdc730be618, 0xc4200e02c0, 0x1299360, 0xc4204a7c80)\n\t/go/src/github.com/fnproject/fn/api/server/error_response.go:57 +0x57f\ngithub.com/fnproject/fn/api/server.handleErrorResponse(0xc4200e02c0, 0x1299360, 0xc4204a7c80)\n\t/go/src/github.com/fnproject/fn/api/server/error_response.go:25 +0xa9\ngithub.com/fnproject/fn/api/server.(*Server).handleFnInvokeCall(0xc4200b7d40, 0xc4200e02c0)\n\t/go/src/github.com/fnproject/fn/api/server/runner_fninvoke.go:53 +0x1da\ngithub.com/fnproject/fn/api/server.(*Server).(github.com/fnproject/fn/api/server.handleFnInvokeCall)-fm(0xc4200e02c0)\n\t/go/src/github.com/fnproject/fn/api/server/server.go:1161 +0x34\ngithub.com/fnproject/fn/vendor/github.com/gin-gonic/gin.(*Context).Next(0xc4200e02c0)\n\t/go/src/github.com/fnproject/fn/vendor/github.com/gin-gonic/gin/context.go:108 +0x43\ngithub.com/fnproject/fn/api/server.(*Server).runMiddleware(0xc4200b7d40, 0xc4200e02c0, 0x0, 0x0, 0x0)\n\t/go/src/github.com/fnproject/fn/api/server/middleware.go:42 +0x29f\ngithub.com/fnproject/fn/api/server.(*Server).rootMiddlewareWrapper.func1(0xc4200e02c0)\n\t/go/src/github.com/fnproject/fn/api/server/middleware.go:32 +0x58\ngithub.com/fnproject/fn/vendor/github.com/gin-gonic/gin.(*Context).Next(0xc4200e02c0)\n\t/go/src/github.com/fnproject/fn/vendor/github.com/gin-gonic/gin/context.go:108 +0x43\ngithub.com/fnproject/fn/api/server.panicWrap(0xc4200e02c0)\n\t/go/src/github.com/fnproject/fn/api/server/gin_middlewares.go:190 +0x51\ngithub.com/fnproject/fn/vendor/github.com/gin-gonic/gin.(*Context).Next(0xc4200e02c0)\n\t/go/src/github.com/fnproject/fn/vendor/github.com/gin-gonic/gin/context.go:108 +0x43\ngithub.com/fnproject/fn/api/server.panicWrap(0xc4200e02c0)\n\t/go/src/github.com/fnproject/fn/api/server/gin_middlewares.go:190 +0x51\ngithub.com/fnproject/fn/vendor/github.com/gin-gonic/gin.(*Context).Next(0xc4200e02c0)\n\t/go/src/github.com/fnproject/fn/vendor/github.com/gin-gonic/gin/context.go:108 +0x43\ngithub.com/fnproject/fn/api/server.apiMetricsWrap.func1.1(0xc4200e02c0)\n\t/go/src/github.com/fnproject/fn/api/server/gin_middlewares.go:154 +0x2f8\ngithub.com/fnproject/fn/vendor/github.com/gin-gonic/gin.(*Context).Next(0xc4200e02c0)\n\t/go/src/github.com/fnproject/fn/vendor/github.com/gin-gonic/gin/context.go:108 +0x43\ngithub.com/fnproject/fn/api/server.traceWrap(0xc4200e02c0)\n\t/go/src/github.com/fnproject/fn/api/server/gin_middlewares.go:97 +0x507\ngithub.com/fnproject/fn/vendor/github.com/gin-gonic/gin.(*Context).Next(0xc4200e02c0)\n\t/go/src/github.com/fnproject/fn/vendor/github.com/gin-gonic/gin/context.go:108 +0x43\ngithub.com/fnproject/fn/api/server.loggerWrap(0xc4200e02c0)\n\t/go/src/github.com/fnproject/fn/api/server/gin_middlewares.go:212 +0x1ea\ngithub.com/fnproject/fn/vendor/github.com/gin-gonic/gin.(*Context).Next(0xc4200e02c0)\n\t/go/src/github.com/fnproject/fn/vendor/github.com/gin-gonic/gin/context.go:108 +0x43\ngithub.com/fnproject/fn/vendor/github.com/gin-gonic/gin.(*Engine).handleHTTPRequest(0xc4200b7c20, 0xc4200e02c0)\n\t/go/src/github.com/fnproject/fn/vendor/github.com/gin-gonic/gin/gin.go:361 +0x586\ngithub.com/fnproject/fn/vendor/github.com/gin-gonic/gin.(*Engine).ServeHTTP(0xc4200b7c20, 0x12a3ec0, 0xc42025a050, 0xc420146100)\n\t/go/src/github.com/fnproject/fn/vendor/github.com/gin-gonic/gin/gin.go:326 +0x153\ngithub.com/fnproject/fn/vendor/go.opencensus.io/plugin/ochttp.(*Handler).ServeHTTP(0xc420204050, 0x12a3ec0, 0xc42025a050, 0xc420268000)\n\t/go/src/github.com/fnproject/fn/vendor/go.opencensus.io/plugin/ochttp/server.go:86 +0x1c8\nnet/http.serverHandler.ServeHTTP(0xc4201e6f70, 0x12a2b40, 0xc4204ac0e0, 0xc420268000)\n\t/usr/local/go/src/net/http/server.go:2697 +0xbc\nnet/http.(*conn).serve(0xc4200ca0a0, 0x12a4480, 0xc4201f08c0)\n\t/usr/local/go/src/net/http/server.go:1830 +0x651\ncreated by net/http.(*Server).Serve\n\t/usr/local/go/src/net/http/server.go:2798 +0x27b\n"
That fn-fn.fn.svc.cluster.local looks interesting.
In my fn.yaml I had to replace the FN_PUBLIC_LB_URL from value: http://fn.lb.fn.internal:90 to value: http://10.192.0.2:30090
I'm thinking then that I probably need to replace the FN_RUNNER_API_URL from value: http://fn-fn.fn.svc.cluster.local:80 to value: http://10.192.0.2:30080 - which is the API URL pointed to by the fn CLI though not sure why it's not seeing the internal address?
I've also experienced issues elsewhere where I've needed a closing period/dot after .local addressess e.g. I'm wondering to try value: fn-fn.fn.svc.cluster.local. too to see if that makes a difference
I'll try tweaking those, but please keep the thoughts coming as there might well be more than one thing causing my woes (I've still got a nagging feeling that the DinD might be having an issue seeing my slightly quick and dirty local container registry)
Thanks for all the replies so far
On the IOFS things I definitely needed them when I tried firing up fnserver directly via docker run e.g. like this:
docker run --rm -i \
--name fnserver \
-v /var/run/docker.sock:/var/run/docker.sock \
-p 8080:8080 \
-v ${PWD}/data:/app/data \
-v ${PWD}/iofs:/iofs \
-e FN_IOFS_DOCKER_PATH=${PWD}/iofs \
-e FN_IOFS_PATH=/iofs \
--entrypoint=./fnserver \
fnproject/fnserver
which information, as it happens, I got from a previous conversation with @rdallman
Which also reminds me about the reservations I had about running fnserver privileged, which seems to be the case too for the current Kubernetes config. It cerainly seems possible to run fnserver by bind mounting the parent's docker socket and it then not having to be run privileged - though it's a question orthogonal to the main thrust of this thread.
On the IOFS things I definitely needed them when I tried firing up fnserver directly via docker run
You need IOFS configuration only in case of local Fn, but the one you use on k8s.
Local Fn needs a docker link via socket mount, but in k8s we use DIND mode where you don’t need to configure IOSF because it works by default, but doesn’t work in local Fn without proper configuration.
Hello, I seem to have made some progress, bur I could still do with some insight.
I've managed to "nursemaid" my setup so that fn invoke goapp gofn runs and returns what I expect.
To do this, in addition to the changes I mentioned in my original post around modifying the fn-service yaml to add the node ports and fn-deployment yaml to change FN_PUBLIC_LB_URL to value: http://10.192.0.2:30090 for api I also had to change the runner-lb FN_RUNNER_API_URL to value: http://10.192.0.2:30080 (that's the one I wondered about a couple of replies back) and in addition to that I had to change FN_RUNNER_ADDRESSES from value: fn-fn-runner.fn.svc.cluster.local:9191 to value: fn-fn-runner.fn.svc.cluster.local.:9191 that is to say I had to add a trailing period after the .local part.
As it happens I've come across the issue of needing the trailing period elsewhere with my setup, but I'm not clear why, I've previously come across a comment that a trailing period signifies an "absolute address" but it's unclear why I need it with my K8s deployment when, I'm guessing, that you guys don't need it. As I say I've seen this elsewhere but I'm relatively inexperienced with Kubernetes and couldn't say whether it's my system being overly pedantic or other systems being overly lax in how those addresses are parsed or whether it's something else entirely - I'd definitely appreciate any insights from those more experienced with K8s quirks.
Once I got all the addressing sorted in my yaml I finally hit the issue that I was half expecting to hit, which was a problem seeing my local Docker registry. As it happens the error message relating to that was pretty self-explanatory:
fn invoke goapp gofn
Error invoking function. status: 502 message: Failed to pull image 'localhost:5000/gofn:0.0.3': Get http://localhost:5000/v2/: dial tcp 127.0.0.1:5000: getsockopt: connection refused
as a really hacky workaround I execed into the runner and proxied to my host via
docker run --name registry-proxy -d -e LISTEN=':5000' -e TALK=\"\$(/sbin/ip route|awk '/default/ { print \$3 }'):5000\" -p 5000:5000 tecnativa/tcp-proxy
obviously that'll have to change, but it did mean that I eventually got
fn invoke goapp gofn
{"message":"Hello World"}
Ultimately I'm going to be running a much nicer (and more secure) local container registry, but in the mean time do you have any thoughts on how I might be able to configure so the DinD docker can see my registry? The reason I was using localhost for the registry was because the 127.0.0.0/8 network is in insecure registries by default but clearly inside the fnproject/fnserver runner container 127.0.0.1 refers to its own localhost. I see in https://github.com/fnproject/docs/blob/master/fn/operate/private_registries.md there is an example of a private registry on localhost, which is similar to my setup and all of the CRUD and image building works fine but I can't see how that would work with DinD? Again I'd appreciate any thoughts.