Perf/fastapi orjson response optimization

[Arunodoy18]

closes: https://github.com/apache/airflow/issues/59082

---

^ Add meaningful description above Read the Pull Request Guidelines for more information. In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed. In case of a new dependency, check compliance with the ASF 3rd Party License Policy. In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[Arunodoy18]

I have resolved the issue. Please assign reviewers to the following and let me know if some following changes is need to be done . Thank you.

[tirkarthi]

Thanks @Arunodoy18 for the PR. Do you have any benchmarks using load testing tools?

@pierrejeambrun We are trying out an internal change similar to the PR to see if the change gives noticeable improvement in performance in terms of serialization time as part of the overall response cycle.

[tirkarthi]

I tried airflow api-server -w 10 on my local laptop with sqlite database using hey tool for a 17kb response compressed to 2kb. Below are the numbers after a few tries so that the workers are warmed up. I saw at max 5% improvement in average time and sometimes the comparison was almost equal. I also don't see the requests per second improving as noted in the fastapi issue where it was noted that serialization using json_encoder was blocking. So the serialization cost looks negligible compared to the overall response time.

On main

hey -n 1000 -c 100 -H 'Cookie: session=7a96019f-07e1-49c6-80c7-846a22b1c8d8._SINWWBZKzkqxPaPaDengiEdir0; _token=eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxIiwianRpIjoiN2Y5N2Y2MWI3NDVlNGU4NDlmYjJmYTg2NWNkNzQ4NDgiLCJpc3MiOltdLCJhdWQiOiJhcGFjaGUtYWlyZmxvdyIsIm5iZiI6MTc2NTI5NjY5MywiZXhwIjoxNzY1MzAwMjkzLCJpYXQiOjE3NjUyOTY2OTN9._ShMZGsOJvDX4e3oUrDSDBlD1yNevDXu4kZ0fodnvCaDOT3ASOGSYu-QmVwhSoovqRi8RtpcETzuHMyO_fmmRw' 'http://localhost:8000/ui/dags?dag_runs_limit=14&limit=15&offset=0&exclude_stale=true&order_by=dag_display_name'

Summary:
  Total:	7.3286 secs
  Slowest:	1.4896 secs
  Fastest:	0.0540 secs
  Average:	0.7018 secs
  Requests/sec:	136.4515

With PR

hey -n 1000 -c 100 -H 'Cookie: session=7a96019f-07e1-49c6-80c7-846a22b1c8d8._SINWWBZKzkqxPaPaDengiEdir0; _token=eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxIiwianRpIjoiN2Y5N2Y2MWI3NDVlNGU4NDlmYjJmYTg2NWNkNzQ4NDgiLCJpc3MiOltdLCJhdWQiOiJhcGFjaGUtYWlyZmxvdyIsIm5iZiI6MTc2NTI5NjY5MywiZXhwIjoxNzY1MzAwMjkzLCJpYXQiOjE3NjUyOTY2OTN9._ShMZGsOJvDX4e3oUrDSDBlD1yNevDXu4kZ0fodnvCaDOT3ASOGSYu-QmVwhSoovqRi8RtpcETzuHMyO_fmmRw' 'http://localhost:8000/ui/dags?dag_runs_limit=14&limit=15&offset=0&exclude_stale=true&order_by=dag_display_name'

Summary:
  Total:	7.7158 secs
  Slowest:	1.5425 secs
  Fastest:	0.0665 secs
  Average:	0.7186 secs
  Requests/sec:	129.6038