Separate Collections to different files

[jaxnoth]

Is your feature request related to a problem? Please describe. We have multiple developers working on separate APIs. Sometimes, Thunderclient does not get refreshed after a pull and the push afterwards overwrites changes made by another developer.

Describe the solution you'd like Separating out each collection to a unique file would mitigate the issue. This would also allow for better grouping if the source needs to be altered outside of the Thunderclient UI.

Describe alternatives you've considered We are taking measures to refresh the Thunderclient after each pull, but this still is subject to human error.

Implementation:

[rangav]

Hi @jaxnoth thanks for the feedback.

changes to db files reload collections automatically is in roadmap. For now you can reload by clicking the Reload Data icon manually

[surfmuggle]

I would love to have the option to use multiroot-workspaces.

Inside a multiroot-workspace each folder would contain one collection and its requests as a single or multiple files

Example

For githubs rest apis for issues the Workspace file GithubRestIssues.code-workspace inside the root folder GithubRestIssues would look like this

{
  "folders": [
    {  "path": "./Issues" },
    {  "path": "./IssueAssignees" },
    {  "path": "./IssuesComments"  }
  ]
}

Each folder under the root folder GithubRestIssues would be a collection

1. Folder Issues as a collection for all request regarding Issues
2. Folder IssueAssignees as above for Issue assignees
3. Folder IssuesComments as above for Issue comments

Screenshot of multiroot-workspace

This demo shows what i mean. Thunder-client would allow me to store the request files into folders. Each "top-level-folder" would be it's own workspace. A workspace-folder can have subfolders.

In the above example i want to issue rest-api calls against github, forgejo and others. A single thunder-client file would be stored in a workspace (top level folder) or in a directory below a workspace (subfolder of top-level-folder). Each file could have one or more api calls

[rangav]

@jaxnoth We have implemented Auto-Reload the sidebar, when the db files change in v2.0.0

See all features released https://github.com/rangav/thunder-client-support/releases/tag/v2.0.0

Do you still need splitting of each collection to separate file?

[rherzfeld]

I would like to point out that #457 might be a viable solution for the above request, as it would allow multiple files. Depending on the implementation, a collection could be considered as a .http file (with multiple requests) or a folder.

[jaxnoth]

The Auto-Reload should fix the immediate issue we were having. My team hasn't fully tested out the feature yet. It would appear that others have similar request for splitting out the single file. I look forward to seeing future developments in those areas. Thanks for working on this!

[anantanandgupta]

I came here searching for the solution for a multi root workspace. but seems that no one is having such an issue

i have a vscode workspace with multiple project folders. e.g.

{
  "folders": [
    {
      "name": "Web Site",
      "path": "../path1/web"
    },
    {
      "name": "APIs",
      "path": "../../path2/api"
    }
  ],
  "settings": {
    "thunder-client.saveToWorkspace": true,
  }
},

now I am not able to tell the thunder-client to choose a specific folder at this moment. the setting thunder-client.workspaceRelativePath always creates a folder with a value set in the setting relative to the first project in the array above. Eventually in case of the multi root workspace, i should be able to dictate where to load from and save to the thunder-tests folder.

[feature request] the name of the folder should be customizable say from thunder-tests -> api-test.

Please allow the selection of the workspace folder as the root of the workspace in place considering the folder at index 0 as the default workspace root.

[rangav]

Hi @anantanandgupta

When you have multiple workspaces the extension will choose the first one.

you can use thunder-client.workspaceRelativePath to specify a different path

example path - ../parent/folder

[anantanandgupta]

I will try the parent folder notation ../. But the issue is that ... As shown in the example, the project folder may not have same parent. They may have different paths on my drive. I may be just aggregating two different projects located at different paths all together.

[surfmuggle]

Hello @anantanandgupta regarding your comment They may have different paths on my drive i wondered if this folder and file structure would be possible with a multiroot workspace.

var multiroot-wokrspace  = {
  "folders": [
     { "path": "C:/2021/Github_http_API" }, // workspace 1  
	 { "path": "D:/2022/GitLap_API" }, // workspace 2
	 { "path": "E:/2023/Gitea_API" }, // workspace 3
	 { "path": "F:/2024/Forgejo_API" } // workspace 4
  ],
  "settings": {}
}

The absolute paths above are not working on my environment. And based on https://github.com/microsoft/vscode/pull/97441 i am uncertain if different absolute paths are supported for every feature in vscode

[anantanandgupta]

Give me 30 min and i will share how in real life it can be ... I have mac and Ubuntu. But will create a vm for windows to replicate the real life use case. By the way if you have an extension named live server, they handle this case very well by identifying the multi root workspaces. They let me choose the location from where i will be serving the content. But give me some time and i will be back with various possibilities.

[anantanandgupta]

Hi @rangav

I have prepared a short video to demonstrate the issue and a sample from other extension settings where they handle it differently, by using the project name. i have seen the other extension working even if there is no name provided in the settings of the workspace. if you need i can quickly create a video on how they handle the projects without the project name in the settings. but for now on my mac i have created a workspace, manually created 2 folders at different location on my system. so that i can show you the issue over here. the thunder client choose the first folder in the array.

here is the link for the video shared on the YouTube (duration: 10min, privately shared)

[Sam-Lin-MillersLab]

I think there are 2 different ideas here.

1. Separate collections to different files.
2. multi-root workspace support.

I don't think there is a conflict between 2 ideas. But more and more people use monorepo now and the size of collection is getting larger and larger. Obviously it will be more painful to merge if there are more team members working in a monorepo.

[joaonunogomes]

Hi @rangav :) Plus one for separating collections into different files. A possible approach I see is to have one folder (on the file system) for each collection with a specific thunderclient.json. The structure of each thunderclient.json could be the same as is today. This approach would impact (as far as I know 😄) the thunderCollection.json file.

Example:

Let's say I want to have a folder with several collections. I could create a new folder on my project named customCollectionFolder and inside that folder have N folders for N collections, each one containing only the respective thunderclient.json.

My suggestion is to add a new property (requestsFolder) to thunderCollection.json entries, that defines the path of thunderclient.json specific to that collection.

thunderCollection.json example:

[
    {
      "_id": "63e43c94-0b3f-4456-86f6-01de3f29c8e9",
      "colName": "My Test Collection 1",
      "requestsFolder": "./collections/my-test-collection-1", -> contains one file "thunderclient.json" with all requests related data
      "created": "2023-05-04T08:52:11.016Z",
      "sortNum": 65000,
      "folders": []
    }, 
    {
      "_id": "63e43c94-0b3f-4456-86f6-01de3f29c8e9",
      "colName": "My Test Collection 2",
      "requestsFolder": "./collections/customCollectionFolder/my-test-collection-2", -> contains one file "thunderclient.json" with all requests related data
      "created": "2023-05-04T08:52:11.016Z",
      "sortNum": 65000,
      "folders": []
    },
    {
      "_id": "63e43c94-0b3f-4456-86f6-01de3f29c8e9",
      "colName": "My Test Collection 3",
      "requestsFolder": "./collections/customCollectionFolder/my-test-collection-3", -> contains one file "thunderclient.json" with all requests related data
      "created": "2023-05-04T08:52:11.016Z",
      "sortNum": 65000,
      "folders": []
    },
    {
      "_id": "63e43c94-0b3f-4456-86f6-01de3f29c8e9",
      "colName": "My Test Collection 4",
      "requestsFolder": "./collections/customCollectionFolder/my-test-collection-4", -> contains one file "thunderclient.json" with all requests related data
      "created": "2023-05-04T08:52:11.016Z",
      "sortNum": 65000,
      "folders": []
    }
]

Hope this makes sense. Thanks in advance!

[rangav]

Hi @joaonunogomes thanks for the feedback, we plan implement something similar to what you suggested. Will provide more details when working on this feature

[joaonunogomes]

we plan implement something similar to what you suggested. Will provide more details when working on this feature

@rangav Awesome! Do you have an expected date for that? Also, if you need contributions fell free to ask.

[rangav]

Hi @joaonunogomes we launched subscription plans for the use of extension software for teams to support our development costs. I hope you saw this announcement.

[rangav]

ETA by the end of June

[rangav]

Hi all, Started working on this feature.

New file structure

• Each Environment and Collection will be in a separate file
• Environment file name format tc_env_{uniqueID}.json
• Collection file name format tc_col_{uniqueID}.json
• When you rename of collection or environment name, the file names won't change
• The Collection file content format will be similar to the collection exported file.

tc_env_3485d83939.json
tc_env_833h7s9334.json
tc_col_4938922938.json
tc_col_7829499jhs9.json

Let me know your feedback.

[cw1934]

@rangav While I see the intent with the unique IDs for file names, I think file names that match the collection/environment names would be better overall, especially for quick matching of files and their contents when evaluating changes in Git Client when committing and merging on an ongoing basis. Renaming the file if the Collection/Environment is renamed in the TC UI should be straightforward, and would also mirror the default behavior of items in the Visual Studio Solution Explorer & VS Code Explorer View.

[rangav]

@cw1934  In VS Code Explorer, you work with files directly, so you name them properly, like customerData.js etc. In TC you will work with Extension UI, where the collection name will be Customer's Data 2023. So the following process needs to be done:

1. The collection/env name needs to be converted to the slug with tc_col prefix tc_col_customer_data_2023.json
2. If the name is too long, then we have to shorten the file name.
3. Renaming a collection/environment name causes a change in the git file. If you don't stage it before checking in, the history may be lost.

If you guys don't see any issues with the above point 3, then I can implement the solution as mentioned in points 1,2.

[cw1934]

@rangav

1. Agreed, some sanitation of the file name would have to be done in order to make sure it would follow restrictions on the applicable OS it's running on.
2. Also agreed, trimming the file name would need to be accounted for, but hopefully a rare case that there is a collection/environment name over ~245 characters.
3. I'm fine with this, and would even go as far as saying that this would be the expected behavior for any who use Git.

Thanks very much!

[rangav]

@cw1934 how we deal with non-english collection names like Japanese, Korean etc..?

[cw1934]

@rangav I am not sure how OSs handle characters from other languages when it comes to file names. The only file name character validation I've seen only ever deals with special characters. I would think that there is a package available that could handle the conversion from any string to a OS-friendly file name.

[rangav]

Hi All, I finished the development of this feature. Will upload vsix file here for testing tomorrow.

Let me know who is available to test and give feedback?

[rangav]

Please find the vsix file to test the new database design.

• vscode-thunder-client-2.7.8.vsix.zip

Please test for the following use cases and let me know your feedback.

• Test for the long collection or env name
• Test for the same collection name
• Test rename collection or environment
• Test a large number of collections
• move folder/request to a different collection

[joaonunogomes]

Hi, Sorry if I'm late. Just noticed these notifications. I get the idea behind the decision of having files with the Ids, however, it would be great if each person/team can organize the files as they want. I see two possible approaches here:

1. For example, you could add a setting so we can set the collection file location and the environment file location.
2. Take leverage of the prefixes tc_col and tc_env and automatically locate all collection and environment files inside the project.

If I may, I would like to vote on approach 2, to enable teams to organize their files as they please. Having everything in the root file may become very unorganized for bigger projects.

Regarding your request, I was able to test the following scenarios:

• Test for the same collection name ✅
• Test rename collection or environment ✅
• Test a large number of collections ✅
  • Thunderclient automatically splitted my collections into several files, everything seems to be working so far

[rangav]

@joaonunogomes thanks for testing and providing feedback.

We already have a setting for approach 2 that you requested. Use the setting Workspace Relative Path - see below image

https://github.com/rangav/thunder-client-support#git-sync

[joaonunogomes]

With the vsix you sent we have the following:

/thunder-tests
    /tc_col_COL1.json
    /tc_col_COL2.json
    /tc_col_COL3.json
    /tc_col_COL4.json
    /tc_col_COL5.json
    /tc_env_ENV1.json
    /tc_env_ENV2.json
    /tc_env_ENV3.json
    /tc_env_ENV4.json
    /tc_env_ENV5.json

From what I could get from the existing settings (Workspace Relative Path), it seems it only enables a way to customize the location of the thunderclient files (Workspace Path). What I was suggesting above was something that could enable the following:

/thunder-tests
     /collections
        /tc_col_COL1.json
        /tc_col_COL2.json
        /tc_col_COL3.json
        /tc_col_COL4.json
        /tc_col_COL5.json
    /environments
        /tc_env_ENV1.json
        /tc_env_ENV2.json
        /tc_env_ENV3.json
        /tc_env_ENV4.json
        /tc_env_ENV5.json

or even:

/thunder-tests
     /collections
        /someFolderToOrganizeFiles
           /tc_col_COL1.json
           /tc_col_COL2.json
        /tc_col_COL3.json
        /tc_col_COL4.json
        /tc_col_COL5.json
    /environments
        /tc_env_ENV1.json
        /tc_env_ENV2.json
        /tc_env_ENV3.json
        /tc_env_ENV4.json
        /tc_env_ENV5.json

Is this possible?

[rangav]

@joaonunogomes now I understood.

The below is possible, and the other suggestion is too complicated.

/thunder-tests
     /collections
        /tc_col_COL1.json
        /tc_col_COL2.json
        /tc_col_COL3.json
        /tc_col_COL4.json
        /tc_col_COL5.json
    /environments
        /tc_env_ENV1.json
        /tc_env_ENV2.json
        /tc_env_ENV3.json
        /tc_env_ENV4.json
        /tc_env_ENV5.json

[joaonunogomes]

Yes, I get the other is too complicated and too much magical as well. Splitting it into a collections folder and a environments will also make the files more organized. If you can do that I don't see any issues :)