twenty
twenty copied to clipboard
twentyhq
Export as csv
Scope & Context
It would be handy to be able to export the data as csv.
Just fyi the use-case I had in mind right now is for Twenty itself:
- Build scripts that fetches contributors from Github
- Push contributor to Twenty via the newly released external API
- We will use Twenty to track engagement with contributors (e.g. send swag when we finally have some :-/), not spam. We can add a custom field like "Community champion" that is managed manually
- Export contributors as csv, then we can upload them weekly on Framer (CMS) to showcase the people who helped build this project (and add special mention for community champions)
Expected behavior
Add an export button below (icon FileExport)
It should create a csv file to download what I'm seeing now on the page. So if I'm on a specific view or I have filters set, the filters should be applied to that export. Sorting should be applied too.
Thanks a lot @mrkirthi-24! Feel free to check the strategy with us before starting implementation if you have doubts
@mrkirthi-24 This should be done on the backend side as we have a pagination, we will need to introduce a new graphql endpoint
I was convinced we'd do this on the backend side but I changed my mind. I think we should do it on the frontend.
The business use-cases for data exports range from distinct needs:
- Large data exports that need to be processed async, they are stored on S3 and sent via email, they usually contain very raw data (all columns available, closer to the database layer than the metadata layer)
- Small in-app data exports that match applied filters. For exemple the marketing team wants to export 100 contacts to send an email campaign in another software.
And there roughly are 3 ways we could implement it within the app:
- Frontend: loop over API results and build the CSV file on the client side
- Backend Light: loop over pg_graphql queries and build the CSV file on the server side, return it directly
- Backend Heavy: dispatch a job, job loops over pg_graphql queries and build the CSV file on the server side, uploads it to S3, email it to client.
The Pros/Cons are the following:
- Frontend: timeout depends on client so potentially higher than in Backend Light ; good user experience that reflects "what you see is what you get"
- Backend Light: lower timeout set by server ; average user experience as I see no progress bar
- Backend Heavy: no timeout ; poor user experience as I have to wait for an email
I think we should go with the Frontend solution in the Options menu. And probably one day we'll have to build the Backend Heavy solution within Settings. But the "Backend Light" solution feels like a bad intermediate.
I will update the issue accordingly.
New Quest!
A new Quest has been launched in @twentyhq’s repo. Merge a PR that solves this issue to loot the Quest and earn your reward.
Loot of 250 USD has been stashed in this issue to reward the solver!
🗡 Comment @quest-bot embark to check-in for this Quest and start solving the issue. Other solvers will be notified!
⚔️ When you submit a PR, comment @quest-bot loot #2183 to link your PR to this Quest.
Questions? Check out the docs.
![quest-bot[bot] avatar](https://avatars.githubusercontent.com/in/378551?v=4 "Published by a pub.dev verified publisher"){kind=small}
@FelixMalfait I have another task assigned to me currently. Would you mind if I give this a try once I complete that?
@mrkirthi-24 has embarked on their Quest 🗡
- @mrkirthi-24 has been on GitHub since 2019.
- They have merged 12 public PRs in that time.
- Their swords are blessed with
JavaScriptandTypeScriptmagic ✨ - They have contributed to this repo before.
Questions? Check out the docs.
Hi @FelixMalfait I am having problems starting frontend code. Please help.
I am using yarn setup
@mrkirthi-24 pretty hard to debug these kind of things remotely. This is the kind of issue where starting from a fresh start could help (remove node_module, re-run yarn, close and re-open VSCode, etc.)
@anoopw3bdev has embarked on their Quest 🗡
- @anoopw3bdev has been on GitHub since 2018.
- They have merged 4 public PRs in that time.
- Their swords are blessed with
JavaScriptandCSSmagic ✨ - They have contributed to this repo before.
Questions? Check out the docs.
Hi @FelixMalfait My understanding on the implementation of this feature is that, it relies completely on the frontend. However, I have some doubts in the technical inputs shared
Set a small delay between each query to avoid overloading the server (100ms ?)Set a limit: 100 queries
Can you elaborate little more on what do you mean by the above two points. (I believe that the data that have to be converted as csv is already available in frontend).
Hey @anoopw3bdev we need to handle pagination so we will only be able to fetch a limit number of records at a time (60?). If we do a for loop with 100 requests in a row I'm afraid this will overload the browser and make some people's computer freeze. It could also hit the server's API rate limit. Instead, we'd like to have a nice feedback loop with progress % and give a little time to breath to the browser and the server.
This PR hasn't been merged yet https://github.com/twentyhq/twenty/pull/3566/files but in the future it would be cool if we used the LOGGED_IN_SHORT_TTL and LOGGED_IN_LONG_TTL to determine respectively the delay between queries and the max number of queries.
Thanks @FelixMalfait After carefully considering technical challenges involved in this feature, I believe it would be best to reassign this task to someone else who may be bit more experienced with codebase to handle it efficiently. Hence reassigning myself from this issue.
@tatethurston has embarked on their Quest. 🗡
- @tatethurston has been on GitHub since 2012.
- They have merged 428 public PRs in that time.
- Their swords are blessed with
JavaScriptandShellmagic ✨ - They haven't contributed to this repo before.
Questions? Check out the docs.
@FelixMalfait I took a first pass in #4034. There's some refactoring/reorganization it could use, but I have a few broader, clarifying questions:
- Are there existing test patterns / examples you could point me to? Anything to help clarify expectations around tests. It seems like the general approach today is unit tests for utils, hooks and services, but not React components?
- What kind of presentation / formatting do you expect for the csv columns? Eg dates, links, null/empties, etc. Example
🧚 @tatethurston has submitted PR https://github.com/twentyhq/twenty/issues/4034 and is claiming the loot.
Keep up the pace, or you'll be left in the shadows.
cc @mrkirthi-24 @anoopw3bdev
Questions? Check out the docs.
Quest solved!
Congratulations! Your efforts have paid off. A PR that solves this Quest has been merged.
@tatethurston, you have been victorious in Quest #2183 🗡
💰 To claim your $250 reward follow instructions here
🧚 Thanks to all bold adventurers for attempting this quest! Sign-up to Quine.sh to access more exciting Quests like this one ⚔️
This Quest is now closed ✨
Questions? Check out the docs.