Generating Migrations file from Models definitions and changes

[sgaseretto]

Today you need to specify all the columns from a table in the migrations file and then write them all again in the respective model file. Basically you have to repeat yourself. Since there isn't yet an official documentation from SearchLight, I was wondering if there is a way to detect the changes from the model definition and generating the migrations file from it, instead of having to repeat yourself two times (also, making the process less prompt to human errors). This feature will be similar to what Alembic does to generate migrations for SQLAlchemy in python, it sees the changes in a model and generates a migration file only with the changes made to the model.

[FrankUrbach]

You are right. But the nature of SearchLight is to support different databases. If you want create things like that a foundation of what you are searching isn't that hard. But you have to change the files to get the full potential of the migrations (create an index e.g.). I started something similar in the past. But before we start to implement this I would discuss this with Adrian (@essenciary). To get it working in all databases SearchLight support today this will take a little time to implement it and I don't want waste my time if Adrian don't want to get this in the package.

[sgaseretto]

Thanks for the rapid response! In my point of view, this feature will be a very nice productivity booster for Genie.jl developers!

[sgaseretto]

@FrankUrbach this functionality got me thinking all day. Since you tried to implement something like this in the past, how do you recommend me to start it? I was thinking on building something like a new package (SearchLightMigrations) that works on top of SearchLight.Migration to try the idea of generating the migrations files from the model file, but since I'm new to Julia (also I've always used this functionality from other libraries without digging into the library to see how it was implemented) I was wondering what are some suggestions you could give me to start implementing this. Maybe it will start as a separate Package and then, once it becomes more mature (with support for all the DBs supported by SearchLight), it gets merged to SearchLight (or as an alternative while your team implements a better way to do this)

[FrankUrbach]

@sgaseretto the easiest way to do something like that is to use the reflection capabilities of Julia. For each struct you can get the field names and the types of the fields. You can make a dictionary with types you want accept for the database and as value the translation into the string you have to use for the description in the migration file. Then you use a template file where you can file in the description you have generated. But you will stuck by adding indexes or something else as the fields itself because you haven't any information inside a struct which field you want preparing as an in Index field. The rest should be possible to do. If you want start such a package let me know where it is and I can help you if you stuck somewhere.

[sgaseretto]

You read my mind! That was what I was trying today, building a dict using Model.name.names and Model.types to use them to start generating the migration files. You are right about the indexes, I was thinking on building custom structs like Fields to add if they have to be indexes or things like limit length (when they are strings) like in Django, and then when getting the model struct, see which fields need indexes, but I don't even know yet if something like that is achievable (I'm coming from OOP, so I don't know how this can be done with Julia at the moment, but I believe that probably yes). Another crucial functionality that I'll experiment with is how to know the differences between a current Table and the Model changes, to determine which columns where added and/or deleted when generating new migration files. I'll continue with some experiments and then I'll set up a repo to share it, I believe it will have to be following this instructions.

Also, I believe that at least with the basic functionalities to automatically generate the migrations files will be sufficient, then for some advance configurations, developers will have to manually edit the migrations files. Something similar happens with Alembic, when trying to autogenerate the migrations files, some things are detected by the library when trying to determine the migrations, and some others cannot be detected

[sgaseretto]

@FrankUrbach do you know if there is a correct way (if posible) to try and test SearchLight in a standalone way with the REPL or in Jupyter, the same way it can be done with Genie? I want to test some things but having to build a whole Genie Project to try somethings feels a little bit awkward sometimes.

[FrankUrbach]

@sgaseretto It isn't necessary to set up a whole Genie project. Make a project add SearchLight and the Adapter you need for your database and try what you want. It should work without Genie. E.g. see the tests of the Postgresql and the MySQL adapters. They work without one line of Genie code.

[sgaseretto]

Thanks a lot! I'll check them

[sgaseretto]

I don't know if I'm overcomplicating things, but the only idea I have is to specify in the Model definition if indexes are required or not by defining custom structs like DbId, for example:

  mutable struct StringDbType
      value::String
      limit::Int = 0 # When parsing this, 0 will mean that there is no limit specified
      index::Bool = True
  end

The drawback I see is that then the property that has this "Type" can't be manipulated as String, unless we overload all the operations that can be done to a String for this type... But probably there can be an easier way, but I'm not seeing it, what do you think @FrankUrbach ?