steve-community
Separation of source, data and object code
This is the first time that I encounter a project which requires the database to compile the source into an object.
What I would like is to have separate procedure:
-
Build: Run mvn package and obtain the target jar file without the need for a functioning database
-
Upgrade: There should be a target to upgrade/install the database schema, inside the final jar file. I would like to run
java -jar steve.jar -Dmain.properties=my_conf_file upgrade. -
Run: Once the database is set and upgraded and the config is updated, I would like just to run
java -jar steve.jar -Dmain.properties=my_conf_file
I believe this should be possible by just moving the targets around.
That would require a complete rework.
The database migration is done using maven/flyway and the sources are generated from the resulting database.
this is not a new request. previously, our standpoint was: there are some reasons why we went this way. it is a lot work to change it, but we might do it if it comes up a lot more.
see the previous discussion for our arguments: https://github.com/RWTH-i5-IDSG/steve/issues/320
indeed, nothing is set in stone, but we need to take multiple things into account (e.g. time/effort to invest, the user base we have, etc.) when changing it. your request is also overlapping with an existing open issue: https://github.com/RWTH-i5-IDSG/steve/issues/366
i want to prevent duplicates, so, feel free to express your ideas there and close this one.
Hello @goekay, I've read the other issue so I'm up to speed with that discussion, although the tones are a bit unpleasant.
So if you don't mind, I would like to start a different discussion, mostly a technical brainstorm on how to improve the situation with the least effort, as I have full respect for free software developers.
I understand the problem is that ORM code is generated w.r.t a particular database flavor/version. Is there any way to pre-generate these with the trade-off of locking the jar file onto that particular database version? After all, you have control of the schema and the migration paths.
My idea would be to always pre-generate as if we are in the current schema version, but still bundle the SQL migrations and run them unless the database is also at the current schema version.
If the above is possible, what would be the version lock? MySQL/MariaDB major/minor? It would be fine even to have an identical version as the reason I'm bringing this up is to prepare a docker container which holds both the database and the app coupled together as a monolithic app (yes, i know it's counter-trend, but I think it's excellent for quick prototyping, testing, and small scale production deployments - I will publish on both github and dockerhub next monday, it still needs a bit of polishing).
Hi, Not here to add oil to the fire, but rather weigh in on the "requesting" side of the scale. Having read the other threads, it seems that the main reason for glueing all the configuration into the final jar is to be end user friendly in the way that the final jar is usable without further configuration and the target audience "consists of mostly non tech-savvy hobbyists" rather than developers.
Without having looked at the code itself I can understand that changing the ORM mapper and the way how/when the schema is generated/controlled is probably a lot of work. Nonetheless I think it would be useful to further investigate possibilities to separate the configuration from the binary.
Or maybe at least offer a possibility to override the configuration of the database at runtime. As you seem to be willing to support the usage of containerization with offering Docker and Kubernetes configuration files, I would appreciate to explore the possibility of a more generally usable container.
I ended up in this thread when I started to create a container that I was planning to upload to Docker HUB and use it for separate installations (at work and at home). For this usecase I now need to maintain two separate containers in private repositories to not expose database credentials.
This thread doesn't ssem to be too active and I'm unsure if anyone is willing to work on such a refactoring, but I would be happy to see this sometime in the future. Currently I'm happy that this project exists and "walk the extra mile" to get it working with private containers. 😃
Cheers!
since this request started to come up more often, i am willing to take another look at it. however, for a healthy discussion i want to separate two topics:
-
removing the need of database for building the app: this is what i want to tackle. i will try to move the database migrations to runtime during start-up. the necessary, generated database models/code will be part of the delivered artifact.
-
supporting multiple database types: this more tricky (if not impossible) to realize because the migration files are tailored to mysql/mariadb. on the other hand, the "orm" we use, i.e. jooq, supports multiple databases and a migration should be possible. however, jooq is a DSL for sql and exposes the capabilities of the database in java. this means, it is not guaranteed that the feature set between mysql and, for example postgres, will be the same. there will be uncovered cases.
i think the interest is mostly in 1, and not 2. am i correct @thg2k @justsomebody42 ?
i think @justsomebody42 just wanted to be the db credentials not be compiled into the artefact but provided e.g. during start up, which is a reasonable request.
steve supports since 2016 an external properties file: https://github.com/RWTH-i5-IDSG/steve/releases/tag/steve-2.1.0 @justsomebody42 are you aware of this? or am i missing/misinterpreting something?
steve even logs the existence of this feature during start-up: https://github.com/RWTH-i5-IDSG/steve/blob/master/src/main/java/de/rwth/idsg/steve/utils/PropertiesFileLoader.java#L54-L55
Thanks for the feedback and considering to work on this! My primary need indeed is an image that could be provided publicly without security concerns regarding configuration data. So, overriding the provided data COULD be enough, if I provide a test DB to build the image, but I would think that topic 1) @goekay suggested would be the best way to approach this. If the configuration data included in the image is more or less useless, it would at least not be security relevant.
Looking at deployment scenarios for Docker and Kubernetes it would make sense to consider the possibility to configure the database (and possibly other values) via environment variables. But I assume that this would be more/a different effort than removing the need of the database during the image build step. For both Kubernetes and Docker it would be possible to provide the external properties file, so it's rather a convenience feature than a necessity ;)
i started to experiment with this in https://github.com/RWTH-i5-IDSG/steve/tree/remove_db_build
Thanks for working on this! Is this in a state where it can already be tested? Our do you maybe even consider to merge it soon?
Hi,
has there been any more discussion around this? It seems that the branch referenced above has been removed. Is this still something that's considered?
@goekay It is painful to have a working database dependency and compilation issues when you load the project. The experiment branch looks good. Any update about that idea?
... Any update about that idea?
i was and am waiting for feedback from users about the correctness and completeness.
when we import project, import project successfully but after this multiple compilation error occurred is there any solution for this issue.
@sushant9930 what is exactly the issue? You are just showing that eclipse is complaining.
I imported this project then updated maven and changed DB credentials as per README but it showed me multiple compilation errors like some methods are undefined. Could you please give a proper flow for the building code?
On Thu, Apr 27, 2023 at 11:00 PM Julien Herr @.***> wrote:
@sushant9930 https://github.com/sushant9930 what is exactly the issue? You are just showing that eclipse is complaining.
— Reply to this email directly, view it on GitHub https://github.com/steve-community/steve/issues/420#issuecomment-1526073674, or unsubscribe https://github.com/notifications/unsubscribe-auth/A22NZHTG4VZJVMNM6RMWIM3XDKULHANCNFSM4STIHGIA . You are receiving this because you were mentioned.Message ID: @.***>
proper flow to build the project is given on README step by step. you should use CLI commands as documented.
apparently, you want to work on source code and opened it in your IDE, i.e. eclipse. in this case you are an advanced user and you should know what you are doing. eclipse has problems with compilation/builds. i am not a user of eclipse. i dropped it years ago when it had problems with proper maven support.
from the screenshot i can imagine that it does not support lombok, because it cannot recognise getters that would be auto-generated by lombok. i don't know why.
it is not nice that you are hijacking an issue with your question that does not have anything with the issue.
I was confused because Eclipse shows multiple compilation errors in every file, so I didn't understand. I am a fresher, so I don't know some things. you're right it shows a compilation error in the Lombok library, JOOQ, jetty, etc. I will try again as per the README flow.
On Fri, Apr 28, 2023 at 4:04 AM Sevket Gökay @.***> wrote:
proper flow to build the project is given on README https://github.com/steve-community/steve#configuration-and-installation step by step. you should use CLI commands as documented.
apparently, you want to work on source code and opened it in your IDE, i.e. eclipse. in this case you are an advanced user and you should know what you are doing. eclipse has problems with compilation/builds. i am not a user of eclipse. i dropped it years ago when it had problems with proper maven support.
from the screenshot i can imagine that it does not support lombok, because it cannot recognise getters that would be auto-generated by lombok. i don't know why.
it is not nice that you are hijacking an issue with your question that does not have anything with the issue.
— Reply to this email directly, view it on GitHub https://github.com/steve-community/steve/issues/420#issuecomment-1526646361, or unsubscribe https://github.com/notifications/unsubscribe-auth/A22NZHRCJYO6RJM3KNLA6P3XDLX7NANCNFSM4STIHGIA . You are receiving this because you were mentioned.Message ID: @.***>
Could you please give me suggestions for another IDE which works better with Maven and this project?
On Fri, Apr 28, 2023 at 7:52 AM sushant d mane @.***> wrote:
I was confused because Eclipse shows multiple compilation errors in every file, so I didn't understand. I am a fresher, so I don't know some things. you're right it shows a compilation error in the Lombok library, JOOQ, jetty, etc. I will try again as per the README flow.
On Fri, Apr 28, 2023 at 4:04 AM Sevket Gökay @.***> wrote:
proper flow to build the project is given on README https://github.com/steve-community/steve#configuration-and-installation step by step. you should use CLI commands as documented.
apparently, you want to work on source code and opened it in your IDE, i.e. eclipse. in this case you are an advanced user and you should know what you are doing. eclipse has problems with compilation/builds. i am not a user of eclipse. i dropped it years ago when it had problems with proper maven support.
from the screenshot i can imagine that it does not support lombok, because it cannot recognise getters that would be auto-generated by lombok. i don't know why.
it is not nice that you are hijacking an issue with your question that does not have anything with the issue.
— Reply to this email directly, view it on GitHub https://github.com/steve-community/steve/issues/420#issuecomment-1526646361, or unsubscribe https://github.com/notifications/unsubscribe-auth/A22NZHRCJYO6RJM3KNLA6P3XDLX7NANCNFSM4STIHGIA . You are receiving this because you were mentioned.Message ID: @.***>