http4s-example

http4s-example copied to clipboard

• Get Started with Http4s[fn:2]

#+HTML: Source Code

This guide covers getting up and running a production ready http4s example.

After reading this guide, you will know:

• How to install Http4s, create a new Http4s application, and connect your application to a database.
• The general layout of a Http4s application.
• The basic principles of FP design.
• How to define and migrate database schema with Flyway[fn:6] and Doobie[fn:7]
• How to stream data using Http4s and Doobie
• How to do A/B testing via finagle feature toggle
• How to add Observability such as logging, distributed tracing, metrics by using Zipkin[fn:8], Prometheus[fn:9]
• How to test Http4s endpoints
• How to package and deploy a Http4s application

** Prerequisites You don't need to install anything else other than just Nix[fn:10]:

• Nix ~sh <(curl -L https://nixos.org/nix/install)~ [fn:1]

#+begin_quote 📝 Nix is a Functional package manager for Linux and macOS, for windows user, WSL(Windows Subsystem for Linux)[fn:5] is required. #+end_quote

To make sure our dev environment is 100% reproducible, please lock the nix channel to a fix version: *** for macOS #+begin_example nix-channel --add https://nixos.org/channels/nixpkgs-20.09-darwin nixpkgs nix-channel --update #+end_example

*** for Linux(or Windows WSL) #+begin_example nix-channel --add https://nixos.org/channels/nixos-20.09 nixpkgs nix-channel --update #+end_example

** Creating a new Http4s Application

#+begin_example ~ nix-shell -p sbt

sbt new jcouyang/http4s.g8 #+end_example

You can either answer all those question that it prompt or press =Enter= all the way to the end.

*** Verify your environment

exit the previous temporary =nix-shell -p sbt= if you are still in #+begin_example

exit #+end_example

Enter nix-shell and start the server

#+begin_quote 📝 first time =nix-shell= may take few minutes to download dependencies defined in =shell.nix= such as =sbt=, =docker= etc #+end_quote

#+begin_example ~ nix-shell

sbt ~reStart #+end_example

#+begin_quote 📝 Let us assume that all future prefix of =>= represent for command in =nix-shell=, and =~= for =bash=. #+end_quote

You should able to see an empty list =[]= since there is nothing in database yet. #+begin_example ~ curl localhost:8080/joke #+end_example

To run test simply #+begin_example

sbt test #+end_example

Now you have a proven working environment for the service to test and run, let us see how we build it.

*** file structure

| File/Folder | Purpose | |--------------------+----------------------------------------------------------------| | .github | folder of github workflow etc. | | .scalafmt.conf | Specification of how to format Scala source code | | build.sbt | Specify build tasks and Scala library dependencies | | db | Database migrations | | docker-compose.yml | Definition of how to boot local services like zipkin, postgres | | project | sbt plugins | | shell.nix | Nix shell configuration | | src | Scala source | | target | Compiled target |

*** source structure #+begin_example ~ tree src src ├── main │   ├── resources │   │   ├── com │   │   │   └── twitter │   │   │   └── toggles │   │   │   └── configs │   │   │   └── com.your.domain.http4sexample.json │   │   └── logback.xml │   └── scala │   └── com │   └── your │   └── domain │   └── http4sexample │   ├── Config.scala │   ├── Main.scala │   ├── NatureTransfomation.scala │   ├── package.scala │   ├── resource │   │   ├── database.scala │   │   ├── http.scala │   │   ├── logger.scala │   │   ├── package.scala │   │   ├── toggle.scala │   │   └── trace.scala │   └── route │   ├── config.scala │   ├── joke.scala │   └── package.scala └── test └── scala └── com └── your └── domain └── http4sexample ├── SpecHelper.scala └── route └── JokeSpec.scala #+end_example

| File/Folder | Purpose | |------------------------------------+-------------------------------------------------------------| | com.your.domain.http4sexample.json | feature toggles | | logback.xml | log config | | Config.scala | Application Config as code | | Main.scala | The entry point of the program | | NatureTransfomation.scala | A helper for kind to kind transformation | | package.scala | index of common types and function across whole application | | resource/database.scala | Database resource, transactor, helper methods etc | | resource/http.scala | Http Client resource | | resource/package.scala | index of all resources | | resource/toggle.scala | Resource of feature toggles | | resource/trace.scala | Resource of zipkin tracing | | route/config.scala | API route of ~/config~ endpoint | | route/joke.scala | API route of ~/joke~ endpoint | | route/package.scala | Index of all APIs | | SpecHelper.scala | Common helper methods for test like database connection | | route/JokeSpec.scala | Test Specification of route ~/joke~ |

There are 3 tiers composite the application:

• =root=: such as =Main.scala= where all the side effects actually happen
• =resource=: definitions of side effects
• =route=: where the actual business is defined

** Data migration

Before we start to build the joke service, what we first is to design a database table, to store the detail of jokes.

You might ask, where is our local DB?

The Postgres DB is defined in =docker-compose.yml= for local development #+begin_src yaml db: image: postgres:10 environment: - POSTGRES_DB=joke - POSTGRES_HOST_AUTH_METHOD=trust ports: - 5432:5432 #+end_src Where =POSTGRES_DB=joke= will help creating the database and name it =joke=.

You don't need to run DB migration manually most of the time, since nix-shell hook will run it for you. #+begin_example shellHook = '' set -a source app.env set +a source ops/bin/deps-up sbt 'db/run migrate' cat ops/sbt-usage.txt set +e ''; #+end_example

Every time you enter =nix-shell=, you will see the migration log: #+begin_example nix-shell Creating network "http4s-example_default" with the default driver Creating http4s-example_zipkin_1 ... done Creating http4s-example_db_1 ... done [info] welcome to sbt 1.3.13 (Azul Systems, Inc. Java 1.8.0_202) [info] loading settings for project http4s-example-build from plugins.sbt,metals.sbt ... [info] loading project definition from /Users/jichao.ouyang/Develop/http4s-example/project [info] loading settings for project root from build.sbt ... [info] set current project to http4s-example (in build file:/Users/jichao.ouyang/Develop/http4s-example/) [info] running Main migrate Sep 14, 2020 12:14:15 PM org.flywaydb.core.internal.license.VersionPrinter printVersionOnly INFO: Flyway Community Edition 6.5.5 by Redgate Sep 14, 2020 12:14:15 PM org.flywaydb.core.internal.database.DatabaseFactory createDatabase INFO: Database: jdbc:postgresql://localhost:5432/joke (PostgreSQL 10.14) Sep 14, 2020 12:14:15 PM org.flywaydb.core.internal.command.DbValidate validate INFO: Successfully validated 1 migration (execution time 00:00.015s) Sep 14, 2020 12:14:15 PM org.flywaydb.core.internal.schemahistory.JdbcTableSchemaHistory create INFO: Creating Schema History table "public"."flyway_schema_history" ... Sep 14, 2020 12:14:15 PM org.flywaydb.core.internal.command.DbMigrate migrateGroup INFO: Current version of schema "public": << Empty Schema >> Sep 14, 2020 12:14:15 PM org.flywaydb.core.internal.command.DbMigrate doMigrateGroup INFO: Migrating schema "public" to version 1.0 - CreateJokeTable #+end_example

To manually migrate when schema changed: #+begin_example

sbt "db/run migration" #+end_example

Migration file located in =db/src/main/scala/db/migration= #+begin_example ~ tree db/src db/src └── main └── scala ├── DoobieMigration.scala ├── Main.scala └── db └── migration └── V1_0__CreateJokeTable.scala #+end_example

A migration file is actually a Scala [[https://tpolecat.github.io/doobie/][doobie]] source code. #+begin_src scala class V1_0__CreateJokeTable extends DoobieMigration { override def migrate = sql"""create table joke ( id serial not null constraint joke_pk primary key, text text not null, created timestamptz default now() not null )""".update.run } #+end_src

The prefix =V1_0__= in class name means version 1.0, detail of naming convention please refer to [[https://flywaydb.org/documentation/migrations#java-based-migrations][Flyway]]

Now we have database scheme set, next we need an API to save data into the new table.

** Save a joke =POST /joke= To be to able to save data, a database library such as [[https://tpolecat.github.io/doobie/][Doobie]] or [[https://getquill.io/][Quill]] is required.

The following example uses Quill: #+begin_src scala -n val CRUD = AppRoute { // <- (ref:route) case req @ POST -> Root / "joke" => for { has <- Kleisli.ask[IO, HasDatabase] // <- (ref:kleisli) joke <- Kleisli.liftF(req.as[Repr.Create]) // <- (ref:reqbody) id <- has.transact(run(quote { // <- (ref:quill) query[Dao.Joke] .insert(.text -> lift(joke.text)) .returningGenerated(.id) })) _ <- log.infoF(s"created joke with id $id") resp <- Created(json"""{"id": $id}""") } yield resp } #+end_src 0. [[(route)][=AppRoute=]] is simply a wrapper of Http4s' =HttpRoutes.of[IO]= but dependencies injectable.

1. [[(kleisli)][=Kleisli.ask=]] is something like =@Inject= in Java world except everything is lazy, when you =ask[IO, HasDatabase]=, it will =<-= a instance =has= of =HasDatabase= type [fn:4]
2. We also need to read the body from the req using Http4s DSL [[(reqbody)][=req.as[Repr.Create]=]] will parse the body and return a =IO[Repr.Create]=. We need to =liftF= because the =for= comprehension is type =Kleisli[IO, HasXYZ, Response[IO]]=.
3. =has= has type =HasDatabase=, which means it has database =transact= method, when =run= convert Quill's =quote= into =ConnectionIO[A]=, =transact= can execute it in one transaction.

#+begin_quote 📝 It is pretty cool that Quill will translate the DSL directly into SQL at compile time:

[[https://www.evernote.com/l/ABeuNCR1bIpMa4xqHKyccGy5mbbxVrzlj2AB/image.png]] #+end_quote

If you're not fan of Macro it is very easy to switch back to doobie DSL: #+begin_src scala -n val CRUD = AppRoute { case req @ POST -> Root / "joke" => for { has <- Kleisli.ask[IO, HasDatabase] joke <- Kleisli.liftF(req.as[Repr.Create]) id <- has.transact( sql"insert into joke (text) values ${joke.text}".update.withUniqueGeneratedKeys("id")) // <- (ref:doobie) _ <- log.infoF(s"created joke with id $id") resp <- Created(json"""{"id": $id}""") } yield resp } #+end_src

** Stream some jokes =GET /joke= Similarly you will probably figure out how to implement a =GET /joke= endpoint already.

But we has some killer feature in Http4s, we can stream the list of jokes direct from DB to response body. Which means you don't actually need to read all jokes into memory, and then return it back at one go, the data of jokes can actually flow through your Http4s server without accumulating in the memory.

#+begin_src scala -n case GET -> Root / "joke" => Kleisli .ask[IO, HasDatabase] .flatMap( db => Ok( db.transact(stream(quote { // <- (ref:stream) query[Dao.Joke] })) .map(Repr.View.from) ) ) #+end_src

[[(stream)][=stream=]] is provide by doobie, which returns =Stream[ConnectionIO, A]=, when =transact= it we will get a =Stream[IO, A]=, luckly Http4s response accept a =Stream[IO, A]= as long as we have a =EntityEncoder[IO, A]=.

** Feature Toggle =GET /joke/:id= It is too straightforward to implement a =GET /joke/:id=: #+begin_src scala case GET -> Root / "joke" / IntVar(id) => for { has <- Kleisli.ask[IO, HasDatabase] joke <- log.infoF(s"getting joke $id") *> Kleisli.liftF( IO.shift(IO.contextShift(ExecutionContext.global)) ) *> has.transact(run(quote { query[Dao.Joke].filter(_.id == lift(id)).take(1) })) resp <- joke match { case a :: Nil => Ok(a) case _ => NotFound(id) } } yield resp #+end_src

Let's add some feature to it, for instance, if there is no joke in database, how about randomly generate some dad joke? And we like 50% of users can see random joke instead of hitting =NotFound=

To prepare a feature toggle in Finagle, you have to put a file in directory =src/main/resources/com/twitter/toggles/configs/com.your.domain.http4sexample.json=. where =com.your.domain.http4sexample= is your application package.

And then put in the toggle: #+begin_src json { "toggles": [ { "id": "com.your.domain.http4sexample.useDadJoke", "description": "random generate dad joke", "fraction": 0.5 } ] } #+end_src

It is good practice to have =id= naming with proper namespace too.

=0.5= fraction means there will be 50% chance for the toggle to be on status.

How can we use this toggle in source code?[fn:3]

Inject =HasToggle= effect #+begin_src diff

• has <- Kleisli.ask[IO, HasDatabase]

• has <- Kleisli.ask[IO, HasDatabase with HasToggle] #+end_src

Switch on the toggle #+begin_src scala -n dadJoke = // <- (ref:declare) if (has.toggleOn("com.your.domain.http4sexample.useDadJoke")) log.infoF(s"cannot find joke $id") *> dadJokeApp.flatMap(NotFound(_)) else NotFound(id) resp <- joke match { case a :: Nil => Ok(a) case _ => dadJoke // <- (ref:usage) } #+end_src =dadJokeApp= is a HTTP effect which call another API, we will go through later.

Here is another advantage of FP over Imperative Programming, [[(declare)][=dadJoke=]] is lazy and referential transparent, which means I can place it anywhere, and whenever I reference it will always be the same thing. While in Imperative Programming this won't be always true, i.e. when you declare a =val printlog = println("log")= it will execute immediately where it declared. But later on when you refer to =printlog=, it is not the same thing it was defined. Since the log is already print, it won't print again.

So, simply declare a =dadJoke= won't execute =dadJokeApp= to actually send out the request. We can safely put it for later usage in [[(usage)][=pattern matching=]]

** Random dad joke =GET /random-joke= To get a random dad joke remotely, you will need a Http client that talk connected to the remote host.

Finagle Client is actually a RPC client, which means a client will bind to particular service.

Assuming we have already define a =jokeClient= in =HasClient=, a dad joke endpoint will be as simple as: #+begin_src scala val dadJokeApp = Kleisli.ask[IO, HasClient].flatMapF(_.jokeClient.expectDadJoke) #+end_src

The client can be make from =resource/package.scala= and then inject into =AppResource= #+begin_src scala js <- http.mk(cfg.jokeService) #+end_src

where =cfg.jokeService= is =uri"https://icanhazdadjoke.com"=

** Tracing Metrics and Logging Finagle already provide sophisticated tracing and metrics, zipkin tracing is by default enable, but it is sample rate is 0.1%, to verify it work, we could start the server with parameter

#+begin_example

sbt '~reStart -zipkin.initialSampleRate=1' #+end_example

Sample rate 1 means 100% of trace will report to zipkin.

#+begin_example curl localhost:8080/random-joke #+end_example *** Logging You can see the server console will print something like: #+begin_example root [7cb6f08c27a8b33c finagle/netty4-2-2] INFO c.y.d.h.r.joke - generating random joke root [7cb6f08c27a8b33c finagle/netty4-2-2] INFO c.y.d.h.r.joke - getting dad joke... #+end_example

Logs belong to the same request will print the exactly same =TRACE ID=

Logger format can be adjusted in =src/main/resources/logback.xml= #+begin_src xml [%X{trace.id} %thread] %highlight(%-5level) %cyan(%logger{15}) - %msg %n #+end_src *** Zipkin Tracing if you grab =7cb6f08c27a8b33c= and search as trace id in =localhost:9411=

https://www.evernote.com/l/ABdFXDYBcnBAFYGQ-8X_us6xcsq42kL2Vn0B/image.png

It will show the trace of the request, from the trace you can simply tell that our server took 3.321s to response, where 2.955s was spend in requesting =icanhazdadjoke.com=.

*** Prometheus Metrics If you have [[https://prometheus.io][Prometheus]] setup, scrap =localhost:9990/metrics= to get server and client metrics.

** Why Resource of resource The resource maker's type is slightly tricky because it is =Resource[IO, Resource[IO, AppResource]]=: #+begin_src scala def mk(implicit ctx: ContextShift[IO]): Resource[IO, Resource[IO, AppResource]] = for { cfg <- Resource.liftF(Config.all.load[IO]) js <- http.mk(cfg.jokeService) db <- database.transactor } yield Resource.make(IO { new AppResource { val config = cfg val jokeClient = js val database = db } }) { res => res.logEval } #+end_src

Why should we have nested Resource here?

These are actually two different kinds of resource, the first level is whole server scope, all requests through this server share the same resource.

• config
• database
• HTTP client

In another word, these resources are acquired when server start, closed when server close. And there are few resources not share across server, they are acquired when request arrived, closed when response sent:

• trace
• toggle
• logger

** Test Once we implemented all CRUD endpoints for =/joke=, testing these endpoints actually are very easy via [[https://github.com/typelevel/scalacheck][ScalaCheck]] property based testing: #+begin_src scala -n property("CRUD") { implicit val appRes = new TestAppResource // <- (ref:testResource) forAll { (requestBody: joke.Repr.Create, updateBody: joke.Repr.Create) => when(appRes.toggleMap.apply(useDadJokeToggleName)) // <- (ref:toggleOff) .thenReturn(Toggle.off(useDadJokeToggleName)) createAndDelete(requestBody) // <- (ref:createDelete) .use { id => assertEquals(query(id).flatMap(.as[joke.Repr.View]).unsafeRunSync().text, requestBody.text) update(id, updateBody) // <- (ref:update) .map( => assertEquals(query(id).flatMap(_.as[joke.Repr.View]).unsafeRunSync().text, updateBody.text)) } .unsafeRunSync() // <- (ref:execute) } } #+end_src

To test all CRUD we just need scalacheck to randomly generate arbitrary create request body and update request body.

1. New a fake resource [[(testResource)][TestAppResource]], defined in =SpecHelper.scala=
2. Don't forget to [[(toggleOff)][toggle off]] our fancy dad joke toggle
3. Make [[(createDelete)][create and delete]] a resource so our test data will always clean after assertion #+begin_src scala def createAndDelete(req: joke.Repr.Create)(implicit router: HttpApp[IO]) = Resource.makeIO, String(delete) #+end_src
4. Assert there will be a joke created
5. [[(update)][Update the joke]] and then query again to verify the data is updated
6. Don't hesitate to [[(execute)][unsafeRunSync]] the Resource, it is OK to fail fast at runtime in test.

** Package and deploy To package the server into a runable binary, simply: #+begin_example

sbt bootstrap #+end_example

To run: #+begin_example

./http4s-example #+end_example

Package it to docker to ship to heroku or k8s #+begin_example

docker build . -t http4s-example #+end_example

The same way we can package and deploy migration scripts as well

#+begin_example

sbt db/bootstrap ./http4s-example-db-migration migrate #+end_example

• Footnotes

[fn:10] https://nixos.org/

[fn:9] https://prometheus.io/

[fn:8] https://zipkin.io

[fn:7] https://tpolecat.github.io/doobie/

[fn:6] https://flywaydb.org/documentation/

[fn:5] https://docs.microsoft.com/en-us/windows/wsl/install-win10

[fn:4] Kleisli is also known as ReaderT https://blog.oyanglul.us/scala/into-the-readert-verse

[fn:3] https://github.com/jcouyang/http4s-example/blob/master/src/main/scala/com/your/domain/http4sexample/resource/toggle.scala#L9

[fn:2] follow the structure of [[https://guides.rubyonrails.org/getting_started.html][Getting Started with Rails]]

[fn:1] If you're using macOS Catalina follow https://nixos.org/manual/nix/stable/#sect-macos-installation