Vulcan
Vulcan copied to clipboard
VulcanJS
Beyond Meteor
This is a place to centralize questions about how we could one day move beyond the Meteor ecosystem.
Package-based Architecture
How does NPM handle package-based architectures? Should we use Lerna.js?
To clarify: in Vulcan's current architecture, 100% of the code is inside separate Meteor packages, there is no "app" code per se. This means the entirety of the codebase can be overridden on a package-by-package basis if a developer needs to do so.
Server/Client Split
In a Meteor package you can do:
api.mainModule('lib/server/main.js', 'server');
api.mainModule('lib/client/main.js', 'client');
What would be the NPM equivalent? Can import foo from 'foo' have different values on the client and server like with Meteor?
What about things like
if (Meteor.isServer) {
//…
}
Fibers
Where do we still rely on Fibers? Could we replace this by async/await code?
What about Meteor.wrapAsync? Meteor.bindEnvironment?
Meteor.startup(...)
What is the equivalent of Meteor.startup() in, say, Next.js?
Accounts
What do we use for Accounts? Auth0? Can we still preserve backwards compatibility with existing Meteor accounts?
Mongo
What do we do about Mongo.Collection? How do we run and connect to Mongo? What do we do about Collection.find()?
Hi, do u hear spectrum. I got a lot ideas from their source code, and made a simple similar next.js app for my previous project. U may take a look on their github. Rethinkdb+graphql+backpack+passportjs U may get some thoughts from their structure
It's possible to achieve Meteor-like "isomorphism" by using Webpack as the build tool, thanks to its fine-grained configurability. While "zero config" is great for application developers, configurability is vital for framework developers not wanting to reinvent some wheels, imho :)
E.g. Meteor.isServer and Meteor.isClient can be easily emulated via Webpack's DefinePlugin – I do it in my current project (pretty large codebase) that uses Webpack to bundle Meteor client-side code (with some help from meteor-client-bundler)
Perhaps the above can also be done using other bundlers like Parcel and Fusebox (if anybody has experience with them, please share), but Webpack is clearly the most popular one today – almost the de-facto build tool for new web apps today.
The whole journey of migration away from Meteor could be a long one, so strategically it might be a good idea to do it gradually and iteratively. The fine-grain control that Webpack allows would come in handy for that.
Perhaps it would be best to start with only the client-side code first, because there's less of Meteor magic there, especially since this project seems to be already using Apollo-client instead of Minimongo etc. That way, you'd also not have to worry about removing Meteor.wrapAsync and Meteor.bindEnvironment initially, as they are thankfully only relevant on the server-side. Similarly, decisions on Meteor.startup and Mongo.Collection on server could be deferred. On client side, I believe Mongo.Collection may not be getting used much, and should anyways be migrated to Apollo-client if there's any stray usage.
During iterative migration, you may need to use both Webpack and Meteor for building and bundling client-side code; achievable with meteor-client-bundler. If it was feasible to build a Webpack loader or plugin for loading Meteor packages, that would have been a better approach – something I have wondered but never had the bandwidth to pursue. Alternatively, meteor-webpack could also be tried out.
I believe Meteor is still quite superior a framework and toolchain for server-side. It would make good architectural sense to have a separate "Backend Server" (Apollo-server on Meteor) that provides Web APIs for different datasources and a separate "Presentation Server" that provides isomorphic view rendering (via a framework specifically built for that purpose, like Next.js). This may also open up the possibility of the community implementing non-React presentation layers like Nuxt.js and Angular-Universal.
While for the server-side Meteor code, the "modernization" could involve migrating towards async/await and Promise. Wherever it gets tricky to use async keyword for any function declaration (because it would start returning a Promise<Value> instead of the original Value, and hence would affect the entire chain of calls), Promise.await provides a good stopgap (note it's Meteor/Fiber specific). But migrating server-side can be considered less urgent, imo. If you're interested in alternatives to Meteor for the backend too, I'd suggest considering Nest.js – a framework that I'd definitely consider for my next greenfield Node.js project
Wow thanks for the detailed comments! That makes a lot of sense. I hadn't heard about Nest.js, I'll check it out as well.
Here's an unofficial timeline for the next couple months:
- August: release OpenCRUD version with new API.
- September-November: work on Apollo 2.0 and React Router 4.0 version (work on client side first and optionally add in Meteor SSR support later)
- December-January: depending on current state of the Meteor ecosystem, work on Next.js migration.
I'd likely start incremental migration towards Next.js + Meteor for my own app. If I make any significant discoveries, will definitely share them here
Since Both of reactxp and Nest.js are made with typescript, What about making a fullstack framework based on Reactxp as frontend and Nest.js as backend?
@SachaG before starting the move to Next.js, check out Fusion.js that was open sourced by Uber's web platform team couple of days ago! The more I think about its architecture and features, the more it looks like what I'd imagine Meteor 2.0 to be 🤩
Perhaps ideally Vulcan's features should be reusable npm packages, so that they are as framework-agnostic as possible. And choose one of the frameworks like Next.js and Fusion.js as the showcase implementation. Do you think that's a direction worth considering?
w.r.t. https://github.com/VulcanJS/Vulcan/issues/2017#issuecomment-408011847, I'm now thinking of first evaluating Fusion.js for the presentation layer of my app.
Hello, I recommend to avoid all Microsoft pseudo new technologies like ReactXP, I have been MS "expert" for 15 years with several certifications.
But since about 7 years I have seen many of the "new" MS politics, like with MS SQL Server, Windows Server, etc. going only on the paying model. And none of them are as good as .NET in the beginning or Windows 7, Windows NT, etc. So now we are leaving away from MS and using only true open source alternatives like Debian, Nodejs, Ubuntu and many others.
MS can change all their politics in the future and, like buying Github, can be against their users very easily.
Thanks a lot to all for this excellent framework.
Just to keep everybody up to date, I've now had calls with core members of both the Next and Fusion teams.
I would say both projects look great, and they both have their pros and cons. My very rough impression so far is that migrating to Next might have a little bit more upshot (more mature product, larger community) but migrating to Fusion would potentially be a lot easier due to the fact that it uses a plugin architecture very similar to Vulcan.
So anyway, I'm keeping a close eye on both for now :)
TLDR
I wish you would consider loopback.io as a backend to move to. I think we can place Vulcan's schema define everything concept on top of loopback authentication, datasource and ORM. Or at least, it is worth to learn their design if you decided to create your own.
Vulcan concept is interesting
I read your concept and it was interesting. I agree that meteor is currently the most user-friendly backend could be found in the market and we should be moving to lighter framework. I would like to share my opinion as follow
Backend suggestion
Have you considered loopback.io as a backend ? It has user management, authentication, ORM and REST api generation out of the box. With additional package available, I can get graphql online. I am waiting for their 4.x release on early 2019. It would be a big change and should already cross the half-line of their milestones by now.
This is what I currently have after playing around for a while. I integrated react-admin and fix some rest adaptor issues and vola!! admin interface ready. Please find the code here:
https://github.com/meepeek/loopback3-with-admin
My opinion about loopback
PROS
- remote datasource is killing feature. You may use another loopback server as a datasource, which makes it easy to split the services to servers but still share some or all data. You can use a model and model's methods from remote datasource like it was on the server.
- datasources also support various databases, file storage and memory are included.
- relational model supported through ORM, even if you use non-relational datasource. Impressive performance. Database specific options can be configure to each model to increase efficiency.
- REST auto generation, with additional module, graphql and graphiql get generated automatically also.
CONS
- Roles and ACL implementation are quite complex. Unnecessary complex, IMO. If you could fix that, it would be awesome.
- you may use async/await but it not fully support yet. need bluebird to wrap the framework's methods before it fully support in LB4.
- React integration / GraphQL will be officially support in LB4. currently, they were just plugin and workaround
- Isomophic feature (loopback on browser) has been beta for so long with no progress. I really wish to use that since it can do remote datasource to the server as well.
Hmm, I don't know if it makes sense to migrate to something like Loopback. I feel like at this point we'd be more or less rewriting the entire framework from scratch. And also I don't think it would help with the main reasons why we'd be migrating in the first place (better bundling/SSR)?
Imo Vulcan should provide components that depends only on apollo. So we can choose whatever ui library and platform we prefer(next, fusion, cra, gatsby ecc) And maybe implement 1 of them like bootstrap + next as default
what if we use assembler instead? in the backend? it would be very very fast!
Imo Vulcan should provide components that depends only on apollo. So we can choose whatever ui library and platform we prefer(next, fusion, cra, gatsby ecc)
That's an interesting idea but it would basically be a different project. Also running a bunch of React components on Fusion/Next/Gatsby/etc. is not the issue, the problem would be ensuring your Vulcan components can connect to your GraphQL back-end. Unless we end up in a world where there's a common standard for GraphQL schemas and APIs, you'd basically have to do 90% of what Vulcan does yourself which would defeat the point…
Unless we end up in a world where there's a common standard for GraphQL schemas and APIs, you'd basically have to do 90% of what Vulcan does yourself which would defeat the point…
@SachaG I think the current feature set and usability of Vulcan can be retained, and yet its reach be made broader by expanding on Rockson's suggestion. I had actually suggested something similar in my earlier comment:
Perhaps ideally Vulcan's features should be reusable npm packages, so that they are as framework-agnostic as possible. And choose one of the frameworks like Next.js and Fusion.js as the showcase implementation. Do you think that's a direction worth considering?
To summarize the whole idea properly, Vulcan should probably be broken into separate projects that play well together:
- A set of NPM packages that implement "end-to-end GraphQL data layer of various features" i.e server-side and/or client-side modules of those features.
- They could be tightly coupled with Apollo, at least to start with, especially on the client-side
- Reference UI implementation of these features, using React.
- That should hopefully not look much different than today's implementation.
- Encourage community to implement them with other view technologies
- Reference SSR implementation using a framework like Next.js or Fusion.js.
- Encourage community to implement them with other frameworks
- Reference GraphQL service implementation.
- Can continue to be on Meteor.js, at least initially
Easier said than done. But the changes should mostly be about removing the dependence on (and assumption of) Meteor API and packages, at least for client-side stuff. I think this should be doable one package at a time, hopefully with the community chiming in to accelerate the process.
For the MongoDB dependency, you can take a look at Feathers adapter system. Basically they kept the MongoDB syntax as a common syntax (selectors, filtering) and let people use custom adapters to connect other kind of databases. That's similar to a micro-ORM.
@gaurav- I think those are all great ideas, and if I was starting Vulcan from scratch today I might go that route. But at this point I just think the folks at Prisma or Apollo are closer to building this than I am.
Also given that making Vulcan work on a single stack is already an infinite quantity of work as far as I can tell, making it work on two or three stacks would mean two or three infinities of work. So I'll pass on that thank you very much.
@SachaG my suggestion is to maintain only one stack officially (aka "reference implementation"). But in doing so, I see an opportunity to also allow users to use their own stacks, if they choose to.
The path to that "one stack" (be it Next.js, Fusion.js or something else) could be gradual, like I proposed earlier in some detail: https://github.com/VulcanJS/Vulcan/issues/2017#issuecomment-406778893
A small part that would need a refactor is the newsletter system. It relies on a Meteor package while we should be able to do the same with Node cron (or any other npm package relevant to this) and bindEnvironment. See https://stackoverflow.com/questions/40687237/cron-jobs-in-meteor.
That article is pretty lazy, since it brushes over half of the MEAN stack (Express, Mongo, and arguably Node) and just tells you to use a third-party service as your API server.
- December-January: depending on current state of the Meteor ecosystem, work on Next.js migration.
Any updates on this front? It's now March 2019.
@jay-jlm Migrating away from Meteor is not currently on the roadmap. I think it'd just be too much work at this point and would kill any momentum Vulcan currently has. Let's see how far we can go with Meteor for now :)
I see there is a lot of activity in the Next.js version of Vulcan. Has there been a new decision made about whether Vulcan will be migrated away from Meteor?
Hi @ErikDakoda ,
We have indeed started to migrate toward Next, as a replacement to Meteor. Note that we will have to keep Meteor around for a while, we have legacy apps to maintain but even new ones being developed The server part of Vulcan will be abstracted away as NPM packages. Next week I'll improve the development environment of https://github.com/VulcanJS/vulcan-npm. It will be doable to maintain both Next and Meteor versions with a shared codebase
The process will probably be like this:
- Creating a Next Starter with a dev environment as mature as Vulcan's: https://github.com/VulcanJS/vulcan-next-starter (in progress, will be done before july)
- Creating packages to help consuming a Vulcan backend from any frontend (so be it Meteor or Next or even other frameworks, eg Gatsby) => those first 2 step will allow to consume an existing Vulcan Meteor backend from any JS app, with Next as a first-class citizen
- Creating packages to help spawning a Vulcan backend in any Express compatible architecture. When you look at it, at this point, Vulcan is already more an Express framework than a Meteor framework.
- Making the Next application "full-stack ready", by adding a way to easily spawn a DB + an apollo sever route => this will be the true replacement of Meteor by Next
What triggered this migration is the addition of API routes in Next in v9, which makes it an hybrid framework, supporting both full-stack and pure frontend usage.
Okay, thank you for the update! I agree with the decision to move most of the code to NPM packages. Then the next biggest hurdle to moving away from Meteor will be the Accounts functionality. Very interesting - I'll be keeping an eye on this.
Regarding accounts we've introduced GraphQL endpoints in the latest release, so in the backend we don't depend on Meteor. We lack the UI at this point though, and support way less features than Meteor does, that's indeed a big subject.