convex-backend icon indicating copy to clipboard operation
convex-backend copied to clipboard

πŸ’‘Feature Proposal: Allow escaped files and folders

Open wkd-kapsule opened this issue 6 months ago β€’ 13 comments

Hi,

First of all, huge thanks to anyone involved in this project. This is a real gem and a pleasure to use.

The feature I'd like to see added to Convex is a way to escape folders inside the convex folder. In order to have more control on their organization and the api generated.


Situation

Let's say I have a user table and I'd like its api to be: api.user.login api.user.checkout.{create,resume} api.user.crud.{get,create,update,drop}

The only way to get such a result would be with this structure:

convex
β”œβ”€ user
β”‚ β”œβ”€ checkout.ts     <!-- {create, resume} -->
β”‚ └─ crud.ts         <!-- {get, create, update, drop} -->
β”œβ”€ user.ts           <!-- {login} -->

This has three drawbacks (or more that I didn't think about):

  • user.ts lives out of the user folder.
  • Both names have to be kept in sync to correctly resolve the api.
  • All api.user.* endpoints have to live in user.ts. If I declare api.user.login and api.user.notRelatedToLogin, both would have to be in the same file. I can't separate them by context.

It quickly becomes a mess as the number of endpoints increases.


Proposal

Allow for a naming syntax that will escape the file or folder's name upon resolving the api. I use () to mimic Next.js routing's files convention. Used on a folder, it should only escape it, leaving the sub-folders as is.

<!-- Both generates the same api -->

convex
β”œβ”€ user
β”‚ β”œβ”€ checkout.ts      <!-- {create, resume} -->
β”‚ β”œβ”€ crud.ts          <!-- {get, create, update, drop} -->
β”‚ └─ (auth.ts)        <!-- {login} -->

convex
β”œβ”€ user
β”‚ β”œβ”€ checkout.ts      <!-- {create, resume} -->
β”‚ β”œβ”€ crud.ts          <!-- {get, create, update, drop} -->
β”‚ └─ (sensitive)    
β”‚   └─ (auth.ts)      <!-- {login} -->


<!-- Now I can add that endpoint, separated from the other -->

convex
β”œβ”€ user
β”‚ β”œβ”€ checkout.ts      <!-- {create, resume} -->
β”‚ β”œβ”€ crud.ts          <!-- {get, create, update, drop} -->
β”‚ └─ (sensitive)    
β”‚   β”œβ”€ (auth.ts)      <!-- {login} -->
β”‚   β”œβ”€ (whatever.ts)  <!-- {notRelatedToLogin} -->

Concern

The structure below will generate two api.member.checkout.*. But I guess it shouldn't be hard to catch during npx convex dev|deploy, like you do when detecting other irregular patterns.

<!-- This throws an Exception at compile|build time -->

convex
β”œβ”€ user
β”‚ β”œβ”€ checkout.ts      <!-- {create, resume} -->
β”‚ β”œβ”€ crud.ts          <!-- {get, create, update, drop} -->
β”‚ └─ (oopsie)    
β”‚   └─ checkout.ts    <!-- {create, notResume} -->


<!-- This one is fine -->

convex
β”œβ”€ user
β”‚ β”œβ”€ checkout.ts      <!-- {create, resume} -->
β”‚ β”œβ”€ crud.ts          <!-- {get, create, update, drop} -->
β”‚ └─ (not-oopsie)    
β”‚   └─ checkout.ts    <!-- {notCreate, notResume} -->

Conclusion

I really think this could further improve the already great DX around Convex.

Thanks to anyone bringing feedback or support to that proposal.

wkd-kapsule avatar May 22 '25 13:05 wkd-kapsule

I think you can use index.ts (eg user/index.ts) to get the API you're looking for. Try it?

nipunn1313 avatar May 22 '25 20:05 nipunn1313

I think you can use index.ts (eg user/index.ts) to get the API you're looking for. Try it?

I did and unfortunately it doesn't work. It generates api.user.index.*. But still, even if it worked it wouldn't give as much customization as my proposal. All api.user.* would have to live in index.ts.

wkd-kapsule avatar May 23 '25 19:05 wkd-kapsule

ah yeah - you're right. Sorry for the misleading suggestion.

Yeah - this feature idea is good. Either via an index.ts or doing something similar to next. We'll keep in in mind for going forward. @ianmacartney has been making a list of ideas for a 2.0 of Convex - and something in this space seems reasonable.

nipunn1313 avatar May 23 '25 20:05 nipunn1313

I'm glad that you like the proposal. It will be cool to see it in a future version. Maybe even before 2.0? It doesn't seem like a difficult problem to tackle or involving a breaking change.

Anyway, if and whenever you implement this, please consider my version of it instead of the one using index.ts. As mentioned, the latter would solve some cases but leave other drawbacks. If your team takes the time to add the feature, they might as well go for the fully customizable version rather than something that people will eventually complain about again.

wkd-kapsule avatar May 24 '25 16:05 wkd-kapsule

Here's the way I think about it:

Supporting index.ts would be convenient, I agree. It looks a bit ugly/verbose, and it's hard to migrate from having users.ts to /users without a breaking change.

I see folder naming as an intent to organize code differently than where you expose endpoints. This is already possible today! You can put code in a lib/checkout/ folder, then re-export them from a top-level /orders.ts. You can decide whether to export it in both locations, or only in one. Here's a repo I just made to show how to do it: https://github.com/ianmacartney/code-organization-pattern

The things I don't like about /(foo)/[bar]/[[...baz]] and other framework-specific tricks:

  1. It's not clear where a function is defined.
  2. There's nothing in typescript that will prevent you from defining the same api path in multiple files
  3. If you don't currently use Next.js, this behavior is strange and unexpected. I personally am always unsure what each magic filename trick will do. For a website UI it makes more sense, since the user sees the URL and you want it to look pretty. For an internal API, it's just a matter of functionality and making it easy for the developer. Adding /user/:id/foo is a less type-safe way to add parameters.
  4. The lack of adherence to established patterns - /index has been around since the start of the internet. Each new file naming "feature" is framework-specific. Tanstack Start has /foo/$bar, and others have their own quirks, but none of them afaik share these folder paterns.

I know this probably isn't what you want to hear, but this is my honest take. Hopefully the repo is a good example

ianmacartney avatar May 25 '25 21:05 ianmacartney

I agree that having an index.ts would actually be really convenientβ€”it would cover some edge cases where you don’t want to create a separate sub-path.

As for the rest, I’m with @ianmacartney: you should try to separate your helper functions from what I call the router folder, where I export the routes.

The idea is to use the router folder strictly to expose your API (and maybe include some controller logic), and move any shared logic or utility functions into a separate helpers folder. I find this architecture pretty clean and maintainable.

Also, it helps separate the core logic of the app from the permissions control, so you only need to manage permissions in a single entry point.

franckdsf avatar Jun 02 '25 12:06 franckdsf

Thanks for the feedback from everyone and sorry for my late response. Here are my thoughts:

  1. It's not clear where a function is defined.

Ctrl + click would still work and be the most used way to locate a function's definition. Otherwise, my guess is that if you organize your folders with escaped names in a certain way, you'll probably be able to find your way through them as it's your own organization anyway. Whether you're solo or your team decided of a consensus.


  1. There's nothing in typescript that will prevent you from defining the same api path in multiple files

Just like nothing stopped me from using redis in a mutation, getting an error at runtime telling me to use an action. As I mentionned in the proposal, my guess was that your CLI could catch duplicate api's paths during npx convex dev|deploy. Just like it already does for other wrong patterns.


  1. If you don't currently use Next.js, this behavior is strange and unexpected. I personally am always unsure what each magic filename trick will do. For a website UI it makes more sense, since the user sees the URL and you want it to look pretty. For an internal API, it's just a matter of functionality and making it easy for the developer. Adding /user/:id/foo is a less type-safe way to add parameters.
  2. The lack of adherence to established patterns - /index has been around since the start of the internet. Each new file naming "feature" is framework-specific. Tanstack Start has /foo/$bar, and others have their own quirks, but none of them afaik share these folder paterns.

My proposal needs only one magic trick that will escape the folder/file's name. If you were to pick (), no need for _ [] [...] [[...]]. I get why you're reluctant, but Convex itself already introduces new patterns that I had to learn. As how queries and mutations don't run in node, must be deterministic, can't execute js for more than one second, and so on.

This fundamentally changed the way my api was originally built with tRPC - Drizzle - Turso. I've spent a lot of time scratching my head as I was migrating, due to these Convex-specific patterns. I really don't see how introducing an escape pattern could be problematic. Especially that it would be opt-in and not something forced upon you, unlike the ones cited above.


I see folder naming as an intent to organize code differently than where you expose endpoints. This is already possible today! You can put code in a lib/checkout/ folder, then re-export them from a top-level /orders.ts. You can decide whether to export it in both locations, or only in one. Here's a repo I just made to show how to do it: https://github.com/ianmacartney/code-organization-pattern

Supporting index.ts would be convenient, I agree. It looks a bit ugly/verbose, and it's hard to migrate from having users.ts to /users without a breaking change

I know this probably isn't what you want to hear, but this is my honest take. Hopefully the repo is a good example

Unfortunately, the repo is empty. Is that the dev's version of being Rickrolled? lol.

Anyway, I have overlooked the fact that you can define your functions anywhere and then export them from the convex folder. It's probably worth a mention somewhere in your Docs. In addition with support for index.ts, I think it'd get rid of all the drawbacks my proposal aimed to fix.

Thanks again for the feedback!

wkd-kapsule avatar Jun 16 '25 08:06 wkd-kapsule

Oops, just pushed: https://github.com/ianmacartney/code-organization-pattern Thanks for the input!

On Mon, Jun 16, 2025 at 1:32β€―AM WKD @.***> wrote:

wkd-kapsule left a comment (get-convex/convex-backend#102) https://github.com/get-convex/convex-backend/issues/102#issuecomment-2975581523

Thanks for the feedback from everyone and sorry for my late response. Here are my thoughts:

  1. It's not clear where a function is defined.

Ctrl + click would still work and be the most used way to locate a function's definition. Otherwise, my guess is that if you organize your folders with escaped names in a certain way, you'll probably be able to find your way through them as it's your own organization anyway. Whether you're solo or your team decided of a consensus.

  1. There's nothing in typescript that will prevent you from defining the same api path in multiple files

Just like nothing stopped me from using redis in a mutation, getting an error at runtime telling me to use an action. As I mentionned in the proposal, my guess was that your CLI could catch duplicate api's paths during npx convex dev|deploy. Just like it already does for other wrong patterns.

  1. If you don't currently use Next.js, this behavior is strange and unexpected. I personally am always unsure what each magic filename trick will do. For a website UI it makes more sense, since the user sees the URL and you want it to look pretty. For an internal API, it's just a matter of functionality and making it easy for the developer. Adding /user/:id/foo is a less type-safe way to add parameters.
  2. The lack of adherence to established patterns - /index has been around since the start of the internet. Each new file naming "feature" is framework-specific. Tanstack Start has /foo/$bar, and others have their own quirks, but none of them afaik share these folder paterns.

My proposal needs only one magic trick that will escape the folder/file's name. If you were to pick (), no need for _ [] [...] [[...]]. I get why you're reluctant, but Convex itself already introduces new patterns that I had to learn. As how queries and mutations don't run in node, must be deterministic, can't execute js for more than one second, and so on.

This fundamentally changed the way my api was originally built with tRPC - Drizzle - Turso. I've spent a lot of time scratching my head as I was migrating, due to these Convex-specific patterns. I really don't see how introducing an escape pattern could be problematic. Especially that it would be opt-in and not something forced upon you, unlike the ones cited above.

I see folder naming as an intent to organize code differently than where you expose endpoints. This is already possible today! You can put code in a lib/checkout/ folder, then re-export them from a top-level /orders.ts. You can decide whether to export it in both locations, or only in one. Here's a repo I just made to show how to do it: https://github.com/ianmacartney/code-organization-pattern

Supporting index.ts would be convenient, I agree. It looks a bit ugly/verbose, and it's hard to migrate from having users.ts to /users without a breaking change

I know this probably isn't what you want to hear, but this is my honest take. Hopefully the repo is a good example

Unfortunately, the repo is empty. Is that the dev's version of being Rickrolled? lol.

Anyway, I have overlooked the fact that you can define your functions anywhere and then export them from the convex folder. It's probably worth a mention somewhere in your Docs. In addition with support for index.ts, I think it'd get rid of all the drawbacks my proposal aimed to fix. Thanks again for the feedback!

β€” Reply to this email directly, view it on GitHub https://github.com/get-convex/convex-backend/issues/102#issuecomment-2975581523, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACZQWZGYBP7EG2BQH2ORHL3DZ6I7AVCNFSM6AAAAAB5WBBCRSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSNZVGU4DCNJSGM . You are receiving this because you were mentioned.Message ID: @.***>

ianmacartney avatar Jun 16 '25 18:06 ianmacartney

Thanks for the update. If I'm correct, your structure creates both api.myFunctions.myMutation and api.baz.buz.myMutation which are the same function. I'm wondering, what is the use case for this? Wouldn't you want to only use the pattern of barApi?

wkd-kapsule avatar Jun 16 '25 20:06 wkd-kapsule

The former is backwards compatible to today's guidance (exporting everything), and doesn't require a separate export { myMutation } on the module you may forget. Both should work though

On Mon, Jun 16, 2025 at 1:45β€―PM WKD @.***> wrote:

wkd-kapsule left a comment (get-convex/convex-backend#102) https://github.com/get-convex/convex-backend/issues/102#issuecomment-2978077947

Thanks for the update. If I'm correct, your structure creates both api.myFunctions.myMutation and api.baz.buz.myMutation which are the same function. I'm wondering, what is the use case for this? Wouldn't you want to only use the pattern of barApi?

β€” Reply to this email directly, view it on GitHub https://github.com/get-convex/convex-backend/issues/102#issuecomment-2978077947, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACZQW6VQKUKZTHKJPHP2ZL3D4UIJAVCNFSM6AAAAAB5WBBCRSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSNZYGA3TOOJUG4 . You are receiving this because you were mentioned.Message ID: @.***>

ianmacartney avatar Jun 16 '25 21:06 ianmacartney

Right now Im declaring custom object in api layer on frontend. Would be nice to be able to configure it on convex side

convex/
β”œβ”€ models/
β”‚  β”œβ”€ user/
β”‚  β”‚  β”œβ”€ schema.ts
β”‚  β”‚  β”œβ”€ types.ts
β”‚  β”‚  β”œβ”€ index.ts // CRUD
β”‚  β”œβ”€ post/
β”‚  β”‚  β”œβ”€ index.ts // CRUD
src/
β”œβ”€ lib/
β”‚  β”œβ”€ convexApi.ts

// convexApi.ts

export const convexApi = {
  user: api.models.user.index,
  post: api.models.post.index,
};

olzhas-kalikhan avatar Oct 04 '25 14:10 olzhas-kalikhan

i support this..keeping backend modular is nightmare with convex.image

jimmyharika avatar Oct 26 '25 15:10 jimmyharika

If there was a way to bypass file-based routing and provide explicit routers (which could be nested), and you could register a whole file with something like import * as foo from "./foo.js", then I think we'd end up in a more powerful place. Where file-based routing is the convenient day-1 routing solution, but once you have a structure you want to enforce you can build yourself a structure that's concise, where the API doesn't need to match your directories. Then things like api.foo.bar could be replaced with fooRouter.bar and sections of the codebase could be more well contained. Just a thought, but would that provide what you're looking for?

On Sun, Oct 26, 2025 at 8:03β€―AM Jimmy Harika @.***> wrote:

jimmyharika left a comment (get-convex/convex-backend#102) https://github.com/get-convex/convex-backend/issues/102#issuecomment-3448611846

i support this..keeping backend modular is nightmare with convex.5213A44B-5F31-43DC-928A-B67D091DFBBA.jpeg (view on web) https://github.com/user-attachments/assets/6b7eef82-4d48-48cc-98eb-1cf2b1c08332

β€” Reply to this email directly, view it on GitHub https://github.com/get-convex/convex-backend/issues/102#issuecomment-3448611846, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACZQW7HTIQIZGODP2KGNFD3ZTPDBAVCNFSM6AAAAAB5WBBCRSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTINBYGYYTCOBUGY . You are receiving this because you were mentioned.Message ID: @.***>

ianmacartney avatar Oct 29 '25 23:10 ianmacartney