🐛 (macOS) Prefork is not working as expected

[ReneWerner87]

Fiber version v2.5.0

OS Systemversion: macOS 11.2.1 (20D74) Kernel-Version: Darwin 20.3.0

Issue description In the prefork concept, one child process is started for each cpu core and the work of the handlers is divided between these processes, so when you output the process number of the child process in the handler, different ids should appear, which is not the case here.

The goal of the ticket is to check where the problem is, because with fasthttp it works.

Code snippet

package main

import (
	"fmt"
	"log"
	"math/rand"
	"os"
	"time"

	"github.com/gofiber/fiber/v2"
)

func main() {
	app := fiber.New(fiber.Config{Prefork: true})
	app.Get("/test", func(c *fiber.Ctx) error {
		fmt.Printf("ProcessId: %v, %v, isChild: %v\t", os.Getpid(), os.Getppid(), fiber.IsChild())
		rand.Seed(time.Now().UnixNano())
		n := rand.Intn(10) // n will be between 0 and 10
		//fmt.Printf("Sleeping %d seconds...\n", n)
		time.Sleep(time.Duration(n)*time.Second)

		return c.SendStatus(200)
	})

	log.Fatal(app.Listen(":3000"))
}

Output:

$shell: go run main.go                                                                                                           

 ┌───────────────────────────────────────────────────┐  ┌───────────────────────────────────────────────────┐
 │                    Fiber v2.5.0                   │  │ Child PIDs ... 6635, 6636, 6637, 6638, 6639, 6640 │
 │               http://127.0.0.1:3000               │  │ 6641, 6642, 6643, 6644, 6645, 6646                │
 │                                                   │  └───────────────────────────────────────────────────┘
 │ Handlers ............. 2  Processes .......... 12 │
 │ Prefork ........ Enabled  PID .............. 6634 │
 └───────────────────────────────────────────────────┘

ProcessId: 6635, 6634, isChild: true
ProcessId: 6635, 6634, isChild: true
ProcessId: 6635, 6634, isChild: true
ProcessId: 6635, 6634, isChild: true
ProcessId: 6635, 6634, isChild: true
ProcessId: 6635, 6634, isChild: true
ProcessId: 6635, 6634, isChild: true
ProcessId: 6635, 6634, isChild: true

Working Fasthttp example:

package main

import (
	"fmt"
	"math/rand"
	"os"
	"time"

	"github.com/valyala/fasthttp"
	"github.com/valyala/fasthttp/prefork"
)

func main() {
	server := &fasthttp.Server{
		Handler: fastHTTPHandler,
	}
	// Wraps the server with prefork
	preforkServer := prefork.New(server)
	if err := preforkServer.ListenAndServe(":3000"); err != nil {
		panic(err)
	}

}

// request handler in fasthttp style, i.e. just plain function.
func fastHTTPHandler(ctx *fasthttp.RequestCtx) {
	fmt.Printf("ProcessId: %v, %v, isChild: %v\t", os.Getpid(), os.Getppid(), prefork.IsChild())
	rand.Seed(time.Now().UnixNano())
	n := rand.Intn(10) // n will be between 0 and 10
	//fmt.Printf("Sleeping %d seconds...\n", n)
	time.Sleep(time.Duration(n)*time.Second)
}

Output:

$shell: go run main.go

ProcessId: 6733, 6728, isChild: true
ProcessId: 6734, 6728, isChild: true
ProcessId: 6729, 6728, isChild: true
ProcessId: 6734, 6728, isChild: true
ProcessId: 6732, 6728, isChild: true
ProcessId: 6731, 6728, isChild: true
ProcessId: 6736, 6728, isChild: true
ProcessId: 6735, 6728, isChild: true
ProcessId: 6730, 6728, isChild: true

[welcome[bot]]

Thanks for opening your first issue here! 🎉 Be sure to follow the issue template! If you need help or want to chat with us, join us on Discord https://gofiber.io/discord

[ReneWerner87]

$shell: docker run --rm -p 3000:3000 -it -v ${PWD}:/usr/app/src -w /usr/app/src golang:alpine sh

The same code base works in the linux container, i.e. it should be a problem specific to the operating system.

[guaychou]

Thank you for creating my issue

[renanbastos93]

@ReneWerner87 I have macOS, do you need help? Can I investigate too?

[kiyonlin]

@ReneWerner87 I have macOS, do you need help? Can I investigate too?

Really appreciate it if you can help!

[softexpert]

I don't know if it is related, but on my AWS Linux server I get panic: listen tcp :6443: bind: address already in use when Prefork is set to true. If Prefork is set to false, it runs perfectly fine. I built the app with Fiber 2.7.1 . The exception is triggered on the line ln, err := tls.Listen("tcp", fmt.Sprintf(":%v", models.SKFRONTWS_Port), tlsCfg) The app is run with sudo for testing.

Interestingly enough, same binary will accept to run with Prefork set to true on my Linux laptop.

[ReneWerner87]

Should not be related

[softexpert]

This means opening a separate issue, right ?

[ReneWerner87]

Depends on whether you take the adapter package, then you make there an issue on

The error message says in any case, that somehow the port is already occupied, you must look at your implemtation, maybe this is not possible in the aws

[kiyonlin]

preforkServer.Reuseport = true makes fasthttp demo get the same issue.

[ReneWerner87]

pls check https://github.com/kavu/go_reuseport

[hitrop]

Is there any update on this?

[renanbastos93]

Sorry everyone, I never looked at this issue again. Are there any updates?

[sixcolors]

@ReneWerner87 this still seems to be an issue, I'll have a look this week and see if I can fix:

~/prefork » go run main.go                      sixcolors@Jason-McNeils-Mac-Pro

 ┌───────────────────────────────────────────────────┐  ┌───────────────────────────────────────────────────┐
 │                   Fiber v2.52.0                   │  │ Child PIDs ... 80628, 80629, 80630, 80631, 80632  │
 │               http://127.0.0.1:3000               │  │ 80633, 80634, 80635, 80636, 80637, 80638, 80639   │
 │       (bound on host 0.0.0.0 and port 3000)       │  │ 80640, 80641, 80642, 80643, 80644, 80645, 80646   │
 │                                                   │  │ 80647, 80648, 80649, 80650, 80651                 │
 │ Handlers ............. 2  Processes .......... 24 │  └───────────────────────────────────────────────────┘
 │ Prefork ........ Enabled  PID ............. 80627 │ 
 └───────────────────────────────────────────────────┘ 

ProcessId: 80628, 80627, isChild: true	ProcessId: 80628, 80627, isChild: true	ProcessId: 80628, 80627, isChild: true	ProcessId: 80628, 80627, isChild: true	ProcessId: 80628, 80627, isChild: true	ProcessId: 80628, 80627, isChild: true	ProcessId: 80628, 80627, isChild: true	ProcessId: 80628, 80627, isChild: true	ProcessId: 80628, 80627, isChild: true

[sixcolors]

Preliminary investigation: Fasthttp prefork.go Reuseport = false by default (except on windows, where it must be true). If Reuseport = true we get the same results in fasthttp example from description:

~/prefork » go run fast.go                      sixcolors@Jason-McNeils-Mac-Pro
ProcessId: 81649, 81648, isChild: true	ProcessId: 81649, 81648, isChild: true	ProcessId: 81649, 81648, isChild: true	ProcessId: 81649, 81648, isChild: true	ProcessId: 81649, 81648, isChild: true	ProcessId: 81649, 81648, isChild: true	ProcessId: 81649, 81648, isChild: true	ProcessId: 81649, 81648, isChild: true	ProcessId: 81649, 81648, isChild: true	ProcessId: 81649, 81648, isChild: true	ProcessId: 81649, 81648, isChild: true	ProcessId: 81649, 81648, isChild: true	ProcessId: 81649, 81648, isChild: true

This is due to the differences in how macOS and Linux handle the SO_REUSEPORT socket option. On Linux, the kernel will evenly distribute incoming connections among all the processes listening on the same port. However, on macOS, the kernel will not distribute the connections, which results in one process (in this case, the first child process) handling all the connections. See https://stackoverflow.com/questions/14388706/how-do-so-reuseaddr-and-so-reuseport-differ

Fasthttp uses a file Listener when Reuseport = false, so that's why it's working.

[sixcolors]

I am unsure about using a file listener-based approach on macOS. I thought a lightweight load balancer that uses ports greater than 1024 might be a better option. What do you think?

[ReneWerner87]

Sounds good