How to update Entity ADT inside a migration that is foreign key of another entities

[Znack]

Hello! Beam is awesome library, I'm so grateful for the elegant types you have implemented here and the powerful migration system. I was missing for a such type safe migrations like you have and when I found Beam about a week ago I dive immediately into developing the demo blog project. And now I have a problem with this demo project where I have used migrations: I've added the column to the one table and all my relations that were defined before has a primary key to outdated version of such a column. And I have no idea even how to redefine all relations in new migration file (although it is very undesirable things, because it will be extremely boilerplaty). Here is the most short example I have prepared to illustrate the problem. The full version of files you can find in the Gist.

Consider the demo blog project where we have defined UserT and PostT (notice PostT has a field _postUserId :: PrimaryKey UserT f) in migration file V0001CreateUserAndPostTables.hs.
Example of a such migration file:

type User = UserT Identity
data UserT f = User
  { _userId :: Columnar f (SqlSerial Int)
  , _userName :: Columnar f Text
  } deriving (Generic, Beamable)
instance Table UserT where
  data PrimaryKey UserT f = UserId (Columnar f (SqlSerial Int))
                        deriving (Generic, Beamable)
  primaryKey = UserId . _userId

type Post = PostT Identity
data PostT f = Post
  { _postId :: Columnar f (SqlSerial Int)
  , _postContent :: Columnar f Text
  , _postAuthor :: PrimaryKey UserT f
  } deriving (Generic, Beamable)
instance Table PostT where
  data PrimaryKey PostT f = PostId (Columnar f (SqlSerial Int))
                        deriving (Generic, Beamable)
  primaryKey = PostId . _postId

migration ::
     ()
  -> Migration PgCommandSyntax (CheckedDatabaseSettings Postgres DemoblogDb)
migration () =
  DemoblogDb <$>
  createTable
    "user"
    (User (field "user_id" serial) (field "name" (varchar (Just 255)) notNull)) <*>
  createTable
    "post"
    (Post
       (field "post_id" serial)
       (field "content" text notNull)
       (UserId (field "user_id" smallint)))

Then we have added the column _userCreatedAt for UserT in next migration V0002AddCreatedAtColumnForUser.hs and so we redefine UserT ADT like this:

module Schema.Migrations.V0002AddCreatedAtColumnForUser
  ( module Schema.Migrations.V0001CreateUserAndPostTables
  , module Schema.Migrations.V0002AddCreatedAtColumnForUser
  ) where

import qualified Schema.Migrations.V0001CreateUserAndPostTables as V0001 hiding
  ( PrimaryKey(UserId)
  )
import Schema.Migrations.V0001CreateUserAndPostTables hiding
  ( DemoblogDb(..)
  , PrimaryKey(UserId)
  , User
  , UserId
  , UserT(..)
  , migration
  ) -- to reexport the old ADT  

-- ... boilerplate with Beam imports, Data.Text etc...

data UserT f = User
  { _userId :: Columnar f (SqlSerial Int)
  , _userName :: Columnar f Text
  , _userCreatedAt :: Columnar f LocalTime
  } deriving (Generic, Beamable)
instance Table UserT where
  data PrimaryKey UserT f = UserId (Columnar f (SqlSerial Int))
                        deriving (Generic, Beamable)
  primaryKey = UserId . _userId

migration ::
     CheckedDatabaseSettings Postgres V0001.DemoblogDb
  -> Migration PgCommandSyntax (CheckedDatabaseSettings Postgres DemoblogDb)
migration oldDb = DemoblogDb <$> alterUserTable <*> preserve (V0001._post oldDb)
  where
    alterUserTable = alterTable (V0001._user oldDb) tableMigration
    tableMigration oldTable =
      User (V0001._userId oldTable) (V0001._userName oldTable) <$>
      addColumn (field "created_at" timestamptz (defaultTo_ now_) notNull)

All this stuff well-typed and works correctly, I have tested the migration (although in database there is no foreign key constraint from post table to user, this is sad although I suppose it is not yet implemented in Beam entirely). But the problem arises when I want to use PostT, for example when I try to create an entity like this:

createPost content userId =
  BeamExtensions.runInsertReturningList (_post db) $
  insertExpressions
    [Post default_ (val_ content) (UserId $ fromIntegral userId)]
--                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ here we have next error:
-- Couldn't match type ‘UserT’
--                 with ‘V0001.UserT’
-- NB: ‘V0001.UserT’ is defined at
--       V0001CreateUserAndPostTables.hs:(18,1)-(21,32)
--     ‘UserT’ is defined at
--       V0002AddCreatedAtColumnForUser.hs:(35,1)-(39,32)
-- Expected type: PrimaryKey
--                   V0001.UserT
--                   (QExpr Database.Beam.Postgres.Syntax.PgExpressionSyntax s')
--   Actual type: PrimaryKey
--                   UserT (QExpr Database.Beam.Postgres.Syntax.PgExpressionSyntax s')

So we have to pass the old version of UserT to Post constructor. Although if I will redefine the entire Post ADT in the second migration too (which is extremely undesirable), I should somehow to preserve the table data in migration, but I cannot use the old good preserve (like I did above in second migration: preserve (V0001._post oldDb)) because oldDb is also have a version of old database from V0001 migration. So I'm stuck and I will be very happy to have a some help here :)

[tathougies]

This is an interesting point, which I admittedly haven't thought of yet.

Off the top of my head, the only way around is to redefine it. Alternatively, you can parameterize Post by the user type it's supposed to use (either V0001.UserT or UserT), but I think you may run into some issues with that.

I'm marking this as a feature request, and I'll do some thinking this weekend. Thanks for the report.

[Znack]

Ok, I will have a couple of experiments with redefining dependant entities and with parameterizing the Post. Thanks for the quick reply, ideas and for including the issue to next milestone :)

[Znack]

Here is my short report about redefining the Post entity.

In the second migration file I want to add column to UserT. I have redefined the User with new field, then I have redefined the Post too. The same structure was used and only link to new UserT ADT was changed for the author field. If there were other dependant entities, I had to redefine all of them too. Now the problem arose how to preserve the post table in the migrations, because I have the new ADT for the Posts although I don't want to change anything in the database. We have quite similar function preserve although it require to having the instance of our new database. We could do it only if compose all our migrations and pass them to evaluateDatabase. I have the attempt to define the migration steps right there to illustrate the problem about cyclic dependancies, below the migration step itself:

-- V0002ExampleBlog.hs
migration ::
     CheckedDatabaseSettings Postgres V0001.DemoblogDb
  -> Migration PgCommandSyntax (CheckedDatabaseSettings Postgres DemoblogDb)
migration oldDb =
  DemoblogDb 
    <$> alterUserTable 
    <*> preserve (_post v0002CheckedDb)  -- notice we have used v0002CheckedDb here
  where
    alterUserTable = alterTable (V0001._user oldDb) tableMigration
    tableMigration oldTable =
      User
        (V0001._userId oldTable)
        (V0001._userName oldTable)
        (V0001._userCreatedAt oldTable) <$>
      addColumn (field "created_at" timestamptz (defaultTo_ now_) notNull)

v0002migrations ::
     MigrationSteps PgCommandSyntax () (CheckedDatabaseSettings Postgres DemoblogDb)
v0002migrations =
  migrationStep "Add user and author tables" V0001.migration >>>
  migrationStep "Add field created_at to user table" migration

v0002CheckedDb :: CheckedDatabaseSettings Postgres DemoblogDb
v0002CheckedDb = evaluateDatabase v0002migrations

v0002Db :: DatabaseSettings Postgres DemoblogDb
v0002Db = unCheckDatabase v0002CheckedDb

However it did not work because we have infinite recursive dependancies here: to set up migrations we have to have new version of database, but new version of database is available only if we have migrations :(

[Znack]

All my attempts of parameterization PostT were failed :( I was unable even to derive the Beamable instance, there was a lot of problems with constraint of default methods of Beamable.

@tathougies maybe you have some other ideas how this problem (I'm about the entire problem of the link to the outdated UserT entity) can be solved?

[tathougies]

What were the errors you received? Were they on latest master (not hackage or stackage, as parameterized Beamable support is new)

[Znack]

Oh, I've just managed to redefine PostT :) I had no compilation error with this method, I've had the infinite recursion there, although now I understand my stupid mistake: I tried to retrieve CheckedDatabaseSettings with evaluateDatabase although I had to do it with defaultMigratableDbSettings (and TypeApplicaitons extension):

-- V0002ExampleBlog.hs
currentDb :: CheckedDatabaseSettings Postgres DemoblogDb
currentDb = defaultMigratableDbSettings @PgCommandSyntax  -- this hurts my mind little bit
-- because I wasn't aware that I can pass types even when
-- it is not expected explicitly in the defaultMigratableDbSettings

migration ::
     CheckedDatabaseSettings Postgres V0001.DemoblogDb
  -> Migration PgCommandSyntax (CheckedDatabaseSettings Postgres DemoblogDb)
migration oldDb =
  DemoblogDb 
    <$> alterUserTable 
    <*> preserve (_post currentDb)  -- notice we have used currentDb here
  where
    alterUserTable = alterTable (V0001._user oldDb) tableMigration
    tableMigration oldTable =
      User
        (V0001._userId oldTable)
        (V0001._userName oldTable)
        (V0001._userCreatedAt oldTable) <$>
      addColumn (field "created_at" timestamptz (defaultTo_ now_) notNull)

[Znack]

@tathougies I had used only beam-migrate from the latest master, today I switched beam-core too and it helped me a lot, thank you! Although now I have a problem with generated SQL. You can see the details in separate repository: https://github.com/Znack/beam-migration-issue Please pay attention to both migration files: first and second.

I have described the problem and how to reproduce it in the README, and in case I missed something please tell me what and I will append the information. Very shortly the problem with generated INSERT command where somehow beam generated "id" field instead of "post_id" for dependant parameterized table.

[Znack]

I have tried SELECT command and it generated invalid SQL too, I have updated the demo repository.

[tathougies]

The issue is that here you call preserve on a table taken from a database schema created with defaultMigratableDbSettings. You need to get rid of currentDb and instead use preserve on the table from oldDb. Unfortunately, without linear types, I can't enforce this at the type-level via the compiler. You have to be disciplined about it.

[Znack]

Yeah, it was silly approach, I had to understand that I will loose all columns info with defaultMigratableDbSettings. Although if I want to use oldDb, I have to have its polymorphic version, so I have parameterized the whole DemoblogDb type, now I'm unable to construct foreign key relation in first migration because it requires specific version of UserT to use the constructor for PrimaryKey user f field. I could use UserId in first migration but of course it will be ill-typed :( I have refreshed the repository demonstrating the problem: https://github.com/Znack/beam-migration-issue, can you please give me a hint how can I declare user_id field in V0001 migration for the Post model?

[tathougies]

@Znack did you figure it out? Just realized I never saw your last question?

[Znack]

@tathougies Unfortunately I was not skilled enough to solve this problem, now I'm learning type extensions of GHC and already have quite solid understanding of many Beam principles, I and my colleague even try to implement generic endpoints for Servant and Beam, than we'll try to return to this issue, I hope we will be able to do something useful later when will be much more experienced