Server responds with 404 at startup

[erikvanoosten]

When starting a new zio-http server, e.g. something like Server.serve(routes.toHttpApp), for a short time the server responds with a 404 status to all requests, even to requests that are covered by the given routes.

To reproduce:

1. start a process that continuously tries to do an http request (see below for a script)
2. start some zio-http server
3. observe that the process from step 1:

• first prints failed to connect (server not started yet)
• then prints 404 responses (server started but routes not active yet)
• then prints 200 responses (routes became active)

The expected behavior is that all routes are served immediately.

Example script to do continuous http calls

while true; do curl --silent --show-error -i --connect-timeout 0.1 http://localhost:8080/ | head -n 1; done

[jdegoes]

/bounty $50

[algora-pbc[bot]]

~~## 💎 $50 bounty • ZIO~~

~~### Steps to solve:~~ ~~1. Start working: Comment /attempt #2381 with your implementation plan~~ ~~2. Submit work: Create a pull request including /claim #2381 in the PR body to claim the bounty~~ ~~3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts~~

~~Thank you for contributing to zio/zio-http!~~

~~Add a bounty • Share on socials~~

Attempt	Started (GMT+0)	Solution
🔴 @uzmi1	Oct 28, 2023, 2:16:58 PM	WIP
🔴 @dalalsoham	Dec 3, 2023, 10:04:23 AM	WIP
🟢 @masonedmison	May 28, 2024, 7:52:49 PM	#2872

[raphlcx]

@erikvanoosten do you mind providing additional context on your setup?

I've tried to setup a basic web server using zio-http, and i'm getting a consistent HTTP 200 immediately after the server starts.

My setup:

• CPU: arm64
• Scala 3.3.0
• sbt 1.9.3
• openjdk 20.0.1
• my test project dep: "dev.zio" %% "zio-http" % "3.0.0-RC2"

[0pg]

It seems because there are gaps between server bootstrap completion (while materializing layer) and install HttpApp (after materializing layer) how about returning internal server error response until the HttpApp is installed?? 🤔

Or shouldn't we get this as parts of spec as long as we can dynamically install httpApp?

[erikvanoosten]

@erikvanoosten do you mind providing additional context on your setup?

@0pg: I did not actually try to reproduce this myself. I noticed that when a new instance starts in our production environment (running on GCP Cloud Run), it first serves 404s. This issue was created after @vigoo confirmed my hypothesis on Discord (https://discord.com/channels/629491597070827530/819703129267372113/1139203450433962024).

The service (from http point of view) is very minimal and doesn't do anything fancy.

[0pg]

@erikvanoosten yeah as you expected it seems like the netty server is already bound and listens port before installing our httpApp (whether service is huge or not) in provided layer. because we are providing to install httpApp dynamically, I think for the same endpoint the client can be received not only success, failure and other responses but 404 response anytime

[uzmi1]

/attempt #2381

Options

• Cancel my attempt

/claim #2381

[uzmi1]

/Claim#2500

[uzmi1]

Erics solution is here- Description: During the startup of a new ZIO-HTTP server, specifically initiated with Server.serve(routes.toHttpApp), a race condition occurs. This results in the server responding with a 404 status to all requests, even those covered by the given routes. The issue is attributed to the server not being fully initialized when continuous requests commence.

Impact:

Failed connections due to the server not being started yet. Subsequent 404 responses as the server initializes but routes are not active. Finally, 200 responses are observed when routes become active. Steps to Reproduce:

Start a process continuously attempting HTTP requests (e.g., using the provided script). Start a ZIO-HTTP server using Server.serve(routes.toHttpApp). Observe the process output: Initially prints "failed to connect" (server not started yet). Then prints 404 responses (server started but routes not active yet). Finally prints 200 responses (routes became active). Expected Behaviour: All routes should be served immediately upon server startup, without a period of 404 responses.

Solution: Introduce proper synchronization mechanisms to ensure that the server is fully initialized before processing requests. The provided practical solution utilizes ZIO's Semaphore for synchronization and ZManaged.makeEffect to manage resources effectively. The Ref serves as a synchronization mechanism to coordinate between server startup and continuous requests.

code-

import zio._ import zhttp.http._ import zhttp.service._ import zio.console._

object ZioHttpServer extends App {

// Define your routes val routes: HttpApp[Any, Nothing] = HttpApp.collect { case Method.GET -> Root => Response.text("Hello, ZIO HTTP!") }

// Semaphore to signal server initialization val serverReady: Semaphore = Semaphore.make(0)

// ZIO effect to create and run the HTTP server val server: ZIO[zio.ZEnv, Throwable, Unit] = for { _ <- putStrLn("Starting ZIO HTTP server...") _ <- ZManaged.makeEffect(serverReady.withPermit.fork)(_.interrupt).use { _ => Server .start(8080, routes) .provideCustomLayer(ServerChannel.live) } } yield ()

// ZIO effect to signal that the server is ready val serverReadyEffect: ZIO[zio.ZEnv, Nothing, Unit] = serverReady.release

// Main ZIO effect to run the server and signal readiness val program: ZIO[zio.ZEnv, Throwable, Unit] = for { _ <- server.fork _ <- serverReadyEffect } yield ()

// Run the program override def run(args: List[String]): URIO[zio.ZEnv, ExitCode] = program.exitCode }

Recommendation: Consider implementing the provided solution or similar synchronization mechanisms to eliminate the race condition and ensure seamless initialization of the ZIO-HTTP server.

[uzmi1]

/bounty $50

[vigoo]

I think the solution to this should be that ServerLive lazily calls NettyDriver#start when the first install call happens instead of during its initialization

[algora-pbc[bot]]

@uzmi1: Reminder that in 7 days the bounty will become up for grabs, so please submit a pull request before then 🙏

[algora-pbc[bot]]

The bounty is up for grabs! Everyone is welcome to /attempt #2381 🙌

[dalalsoham]

/attempt #2381

Options

• Cancel my attempt

[algora-pbc[bot]]

@dalalsoham: Reminder that in 7 days the bounty will become up for grabs, so please submit a pull request before then 🙏

[algora-pbc[bot]]

The bounty is up for grabs! Everyone is welcome to /attempt #2381 🙌

[masonedmison]

/attempt #2381

Options

• Cancel my attempt

[algora-pbc[bot]]

💡 @masonedmison submitted a pull request that claims the bounty. You can visit your bounty board to reward.

[algora-pbc[bot]]

@masonedmison: You've been awarded a $50 bounty by ZIO! 👉 Complete your Algora onboarding to collect the bounty.

[algora-pbc[bot]]

🎉🎈 @masonedmison has been awarded $50! 🎈🎊

• Share on socials
• Give feedback

[masonedmison]

This can be closed now; #2872 resolves this and has been merged.