ksail icon indicating copy to clipboard operation
ksail copied to clipboard
verified publisher icon devantler

Trying out the tool: got exception when 'Creating 'flux-system' namespace

Open msthtrifork opened this issue 1 year ago • 10 comments

Description

ksail up test-clusterproduces:

I tried the command several time, so the output may vary from actual repro-steps. E.g. First destroying the cluster etc.

🐳 Checking Docker is running
✔ Docker is running

🔥 Destroying cluster 'test-cluster'
INFO[0000] Deleting cluster 'test-cluster'
INFO[0000] Deleting cluster network 'k3d-test-cluster'
INFO[0000] Deleting 2 attached volumes...
INFO[0000] Removing cluster details from default kubeconfig...
INFO[0000] Removing standalone kubeconfig file (if there is one)...
INFO[0000] Successfully deleted cluster test-cluster!

🧮 Creating pull-through registries
► Creating pull-through registry 'proxy-docker.io' on port '5001' for 'https://registry-1.docker.io/'
✔ Registry 'proxy-docker.io' already exists. Skipping
► Creating pull-through registry 'proxy-registry.k8s.io' on port '5002' for 'https://registry.k8s.io/'
✔ Registry 'proxy-registry.k8s.io' already exists. Skipping
► Creating pull-through registry 'proxy-gcr.io' on port '5003' for 'https://gcr.io/'
✔ Registry 'proxy-gcr.io' already exists. Skipping
► Creating pull-through registry 'proxy-ghcr.io' on port '5004' for 'https://ghcr.io/'
✔ Registry 'proxy-ghcr.io' already exists. Skipping
► Creating pull-through registry 'proxy-quay.io' on port '5005' for 'https://quay.io/'
✔ Registry 'proxy-quay.io' already exists. Skipping
► Creating pull-through registry 'proxy-mcr.microsoft.com' on port '5006' for 'https://mcr.microsoft.com/'
✔ Registry 'proxy-mcr.microsoft.com' already exists. Skipping

🧮 Creating OCI registry
► Creating registry 'manifests' on port '5050'
✔ Registry 'manifests' already exists. Skipping

📥 Pushing manifests
► pushing artifact to localhost:5050/test-cluster:1728541475
✔ artifact successfully pushed to localhost:5050/test-cluster@sha256:f4a4622f7162f11c7bec068074d54391fdaada63f38587f81af3ab48ef9930c4
► tagging artifact
✔ artifact tagged as localhost:5050/test-cluster:latest

🚀 Provisioning cluster 'test-cluster'
INFO[0000] Using config file test-cluster-k3d-config.yaml (k3d.io/v1alpha5#simple)
WARN[0000] No node filter specified
INFO[0000] Created named volume 'k3d-test-cluster-storage'
INFO[0000] Prep: Network
INFO[0000] Created network 'k3d-test-cluster'
INFO[0000] Created image volume k3d-test-cluster-images
INFO[0000] Starting new tools node...
INFO[0000] Starting node 'k3d-test-cluster-tools'
INFO[0001] Creating node 'k3d-test-cluster-server-0'
INFO[0001] Creating LoadBalancer 'k3d-test-cluster-serverlb'
INFO[0001] Using the k3d-tools node to gather environment information
INFO[0001] Starting new tools node...
INFO[0001] Starting node 'k3d-test-cluster-tools'
INFO[0002] Starting cluster 'test-cluster'
INFO[0002] Starting servers...
INFO[0002] Starting node 'k3d-test-cluster-server-0'
INFO[0004] All agents already running.
INFO[0004] Starting helpers...
INFO[0004] Starting node 'k3d-test-cluster-serverlb'
INFO[0011] Injecting records for hostAliases (incl. host.k3d.internal) and for 3 network members into CoreDNS configmap...
INFO[0013] Cluster 'test-cluster' created successfully!
INFO[0013] You can now use it like this:
kubectl cluster-info

🌐 Creating 'flux-system' namespace
Unhandled exception: System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.IO.IOException: Unable to write data to the transport connection: Broken pipe.
 ---> System.Net.Sockets.SocketException (32): Broken pipe
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.CreateException(SocketError error, Boolean forAsyncThrow)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.SendAsyncForNetworkStream(Socket socket, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.SendAsyncForNetworkStream(ReadOnlyMemory`1 buffer, SocketFlags socketFlags, CancellationToken cancellationToken)
   at System.Net.Sockets.NetworkStream.WriteAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken)
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecuteFromThreadPool(Thread threadPoolThread)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
   at System.Threading.Thread.StartCallback()
--- End of stack trace from previous location ---

   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.AddHttp2ConnectionAsync(QueueItem queueItem)
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at k8s.Kubernetes.SendRequestRaw(String requestContent, HttpRequestMessage httpRequest, CancellationToken cancellationToken)
   at k8s.AbstractKubernetes.ICoreV1Operations_CreateNamespaceWithHttpMessagesAsync[T](V1Namespace body, String dryRun, String fieldManager, String fieldValidation, Nullable`1 pretty, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
   at k8s.AbstractKubernetes.k8s.ICoreV1Operations.CreateNamespaceWithHttpMessagesAsync(V1Namespace body, String dryRun, String fieldManager, String fieldValidation, Nullable`1 pretty, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
   at k8s.CoreV1OperationsExtensions.CreateNamespaceAsync(ICoreV1Operations operations, V1Namespace body, String dryRun, String fieldManager, String fieldValidation, Nullable`1 pretty, CancellationToken cancellationToken)
   at KSail.Provisioners.ContainerOrchestrator.KubernetesProvisioner.CreateNamespaceAsync(String context, String name)
   at KSail.Commands.Up.Handlers.KSailUpCommandHandler.HandleAsync(String clusterName, String configPath, String manifestsPath, String kustomizationsPath, Int32 timeout, Boolean noSOPS, Boolean skipLinting, CancellationToken token)
   at KSail.Commands.Up.KSailUpCommand.<.ctor>b__7_1(InvocationContext context)
   at System.CommandLine.Invocation.AnonymousCommandHandler.InvokeAsync(InvocationContext context)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<<BuildInvocationChain>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass17_0.<<UseParseErrorReporting>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass12_0.<<UseHelp>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass22_0.<<UseVersionOption>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass19_0.<<UseTypoCorrections>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseSuggestDirective>b__18_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass16_0.<<UseParseDirective>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<RegisterWithDotnetSuggest>b__5_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass8_0.<<UseExceptionHandler>b__0>d.MoveNext()

Steps to Replicate

On Macos M1 Max

  1. Step 1: ksail init test-cluster
  2. Step 2: ksail up test-cluster -sl

Fails with the exception:

Unhandled exception: System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.IO.IOException: Unable to write data to the transport connection: Broken pipe.
 ---> System.Net.Sockets.SocketException (32): Broken pipe

I used -sl has the linting just hangs for ages.

msthtrifork avatar Oct 10 '24 06:10 msthtrifork

Can I set verbosity or debug flags to get more information? - I am open to contribute a PR if necessary 😄

msthtrifork avatar Oct 10 '24 06:10 msthtrifork

Are you on Windows?

I am currently working on v2 for the end of the month. It will add proper windows support.

It should work in WSL for now. Have you tried that? :-)

devantler avatar Oct 10 '24 06:10 devantler

No, unfortunately not. I am on a Mac (m1 max, macos 14).

msthtrifork avatar Oct 10 '24 06:10 msthtrifork

Can I set verbosity or debug flags to get more information? - I am open to contribute a PR if necessary 😄

Not right now. You are welcome to contribute, but so much will change with v2, that I recommend you to wait a little more with contributions :-)

devantler avatar Oct 10 '24 06:10 devantler

No, unfortunately not. I am on a Mac (m1 max, macos 14).

Hmm did not see that. Will have a look later today. It should work 🤔

devantler avatar Oct 10 '24 06:10 devantler

Hmm, I cannot reproduce on my Mac M2. It is quite wierd because that error seems to come from the https://github.com/kubernetes-client/csharp library, which is very stable.

Can you try checking the following:

  • can you access the K3d cluster with k9s or kubectl after it fails?
  • how does the test-cluster-k3d-config.yaml file look?
  • check if your .kube/config looks correct. KSail reads the context from there, so if there ar multiple with the same name or something similar, that might be causing issues.

devantler avatar Oct 10 '24 07:10 devantler

can you access the K3d cluster with k9s or kubectl after it fails?

Can't access it, no.

how does the test-cluster-k3d-config.yaml file look?

The contents of test-cluster-k3d-config.yaml:

apiVersion: k3d.io/v1alpha5
kind: Simple
metadata:
  name: test-cluster
volumes:
  - volume: k3d-test-cluster-storage:/var/lib/rancher/k3s/storage
network: k3d-test-cluster
options:
  k3s:
    extraArgs:
      - arg: "--disable=traefik"
        nodeFilters:
          - server:*
registries:
  config: |
    mirrors:
      "docker.io":
        endpoint:
          - http://host.k3d.internal:5001
      "registry.k8s.io":
        endpoint:
          - http://host.k3d.internal:5002
      "gcr.io":
        endpoint:
          - http://host.k3d.internal:5003
      "ghcr.io":
        endpoint:
          - http://host.k3d.internal:5004
      "quay.io":
        endpoint:
          - http://host.k3d.internal:5005
      "mcr.microsoft.com":
        endpoint:
          - http://host.k3d.internal:5006

check if your .kube/config looks correct. KSail reads the context from there, so if there ar multiple with the same name or something similar, that might be causing issues.

I had someting similar to test-cluster, so I tried with "small-cluster" instead, but same result.

msthtrifork avatar Oct 10 '24 12:10 msthtrifork

Hmm, everything looks like it should, and what you are describing seems unrelated to KSail. Behind the scenes it calls the k3d binary to spin up a cluster. I do not alter the logic for that at all, and by default it will set your kubeconfig context to that new cluster. I suppose something fishy is going on with that, since it cannot connect to whatever your context is set too.

What is the output of the following commands, after running ksail up <cluster-name>:

  • kubectl config get-contexts
  • kubectl config get-clusters

devantler avatar Oct 10 '24 12:10 devantler

I think you are right about that. Its seems unrelated to ksail.

get-contexts

*         k3d-small-cluster                                                    k3d-small-cluster              admin@k3d-small-cluster
          k3d-test-cluster                                                     k3d-test-cluster               admin@k3d-test-cluster

get-clusters

k3d-test-cluster
k3d-small-cluster
  • some other clusters (for work ;-) )

Don't waste time on this. I will try a different machine (linux) to see if resolves it. I think it might be the anti-virus / security program running on the particular machine.

msthtrifork avatar Oct 10 '24 12:10 msthtrifork

Hmm could be. Definitely something weird going on here. If you resolve it, you are very welcome to share :-)

devantler avatar Oct 10 '24 13:10 devantler

It worked flawlessly on my Linux machine and it was in fact a security/firewall thing on my machine. I am closing this issue. :-D

Edit: just to be clear. Its now working on my mac m1 max!

msthtrifork avatar Oct 16 '24 09:10 msthtrifork