Rework Workspace and Settings apis

[kabentley]

Make the apis for loading Settings and WorkspaceResources easier to use. It makes it possible to store SettingsDictionaries in cloud workspaces and load them automatically when you open an iModel.

This includes breaking changes to the (beta) apis for:

• Settings Schemas
• Settings Dictionaries
• Workspaces
• CloudSqlite

It also introduces the Workspace.Editor namespace that can be used to modify Workspaces.

@johnnyd710 this will require work for the Settings Editor. It enforces that all Settings defined by a SettingSchema must all have the same prefix. That will undoubtedly break existing code in iTS, since settings will have a different name. You also need to add the concept of storing SettingsDictionaries in cloud containers, as we discussed.

Also, @johnnyd710, @wgoehrig, @MichaelSwigerAtBentley we should talk about how the settings editor will allow picking cloud-based WorkspaceDbs. We will probably need some kind of queries from the BlobContainer service.

[kabentley]

This still needs more documentation in Workspace.md and lots of examples, but the api is now ready for review.

[pmconne]

I'll be taking over this PR. I am currently familiarizing myself with it. It could help to adjust the PR description to describe what specific changes were made for what specific purposes, as many of the changes appear to be cosmetic. e.g., in what specific ways does this make settings "easier to use".

[kabentley]

ways does this make settings "easier to use".

It doesn't make "settings" easier to use, it makes "loading Settings and WorkspaceResources easier to use". The previous approach had a setting called "cloud/containers" that was an array of container properties with an "alias". Then, there was a setting called "workspace/databases" that was an array of workspaceDb properties that referenced entries in the "cloud/container" array through their alias. This eliminates both of those in favor of the new setting type "itwin/core/workspace/workspaceDbList", making the api much simpler.

I will describe the other changes separately.

[pmconne]

In a subsequent PR we need to address discrepancies with font dbs, including:

• They don't follow the persistence and retrieval mechanisms described by the WorkspaceDb interface.
• No one ever calls iModelJsNative.addFontWorkspace, so native code cannot find workspace fonts.
• We need to make sure the native list is updated when settings change in core-backend.
• FontManager maintains a static list of font workspaces, which poses at least two problems
  • Race conditions - worker threads (e.g., for tile generation) will want stable access to the list of font dbs while the main thread is trying to change the list's contents. Currently zero synchronization.
  • The static list does not account for font dbs specified in one particular iModel's workspace.
• Adding geometry that uses a font requires updating the font name<->Id mapping in the iModel, which requires an exclusive lock on the entire iModel.

The issues with the static list and reacting to Settings changes are also relevant to GCS workspace dbs.

[pmconne]

@kabentley is it assumed that each WorkspaceDb holds exactly one type of resource? If I wanted to write the logic to, say, look up all the text styles available to me, it'd be something like:

• Get the value of the setting that tells me the list of workspace dbs containing text styles.
• Walk through the list of workspace dbs.
• Assume every row in the strings table is valid stringified JSON representing a text style.
• Decode the JSON into a TextStyle.

If somebody messes up and puts a dimension style workspace db into my text style db list, or adds a dimension style into one of my text style dbs, difficult-to-diagnose errors will occur at run-time. There is currently no mechanism to prohibit either of those occurrences. I can of course validate the JSON and ignore things that aren't text styles, but that just hides the configuration problem from the user.

If someone is inspecting a .itwin-workspace file to determine what kind of resources it contains, they'll have to hope someone filled in the optional description in the manifest sufficiently, or that it's obvious from the names or contents of the data present.

If we want to permit dimension styles and text styles and whatever else to be stored in the same workspace db, we need some mechanism for distinguishing between types of resources and preventing name collisions, whether a naming scheme, a type column, etc. To iterate through text styles, I should not have to look at each string in the strings table and determine that 1) it is JSON and 2) the JSON represents a text style.

(Of course, a single workspace db is able to contain up to 3 types of resources without these problems, if each has a different persistence (file, blob, string) - 4 if you count fonts, which are stored differently. This seems awkward to me though).

If we want to enforce that a workspace db contains only resources of a single type, we again need some notion of "resource type" beyond "file, string, or blob (or font)".

People will presumably want tools to view the contents of a workspace db, e.g., to see what text styles it contains and the properties of the styles. They will also want tools to add new resources to workspaces. I don't want us (and every app developer who invents their own kinds of resources) to have to write a separate UI for viewing/editing each type of resource. It might be useful for each resource in the workspace db to specify its type, and for a schema to be provided for each resource type, so both people and programs can understand the contents of the workspace db.

[kabentley]

I meant to reply to this comment:

Presumably it is meaningless for non-arrays (including objects, though you could make a case for a higher-priority setting overriding some properties of a lower-priority one).

VS Code does that for objects, but I strongly suggest we don't allow it. It creates a big complicated, ambiguous mess for queries. I don't see it being worthwhile or necessary.

[pmconne]

VS Code does that for objects, but I strongly suggest we don't allow it. It creates a big complicated, ambiguous mess for queries. I don't see it being worthwhile or necessary.

Fine by me. I will rename cumulative to combineArrays.

[pmconne]

Some questions from @Josh-Schifter below. I am starting on rewriting Workspace.md now, with an eye toward making sure it provides answers to all of these (among others).

What is the admin workflow for managing workspaces?

• At present, an admin must create the cloud containers, create the styles (and decide versioning), and configure specific workspace versions to iModels.
• Where should all that happen?

What is the unit of versioning for workspace information?

• Should all ‘Drawing Production’ data be in one block, versioned together?
• Should users (admins) be able to create and manage the blocks and decide on their contents?

What is the admin workflow for creating new versions?

• I assume that admins will make changes over multiple sessions before publishing a new version. Is that correct?
• How can they test their changes? Can they point their session to the unpublished version?
• Should the UI for changing data and publishing versions be part of our mango app?

How are new versions consumed by iTwins / iModels?

• Who will decide when to consume a new version?
• I assume that consuming a new version will imply a changeset is created in the target iModel. Is that right?
• Where will that UI be?
• Specific code will be needed to apply the new version (ex. dimension style). How will we assure that code is present when applying?
• This is related to the earlier question about which data is versioned together.

How iModels consume data from the application default workspaces?

• They will also be versioned – by us. But what version will the iModels get?
• Will there be a way to update the version for a particular iModel?
• Is this handled the same or differently from user managed workspaces?

How will admins organize “Org level workspaces”?

• Will every style in an Org workspace by available for every iModel?
• What if they want a subset of those styles for a particular project?
• Will they need to create multiple workspacedbs for each possible subset?
• Wouldn’t that force them to either a) create a separate db for each style or b) duplicate some styles between dbs?

[mergify[bot]]

This pull request is now in conflicts. Could you fix it @kabentley? 🙏 To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

[pmconne]

@ben-polinsky @williamkbentley docs build errors are so tedious to reverse-engineer...can you tell what source code file+line this is referring to?

Invalid Link: "[manifest]($core-backend.WorkspaceDb)" in file reference/core-backend/workspace/editableworkspacedb/index.html"

[williamkbentley]

@ben-polinsky @williamkbentley docs build errors are so tedious to reverse-engineer...can you tell what source code file+line this is referring to?

Invalid Link: "[manifest]($core-backend.WorkspaceDb)" in file reference/core-backend/workspace/editableworkspacedb/index.html"

The [manifest] one does not make sense to me. I dont see that link in the EditableWorkspaceContainer docs. Maybe its trying to make it in the updateManifest method? I need to ask someone else.

[pmconne]

The [manifest] one does not make sense to me.

@kabentley had get manifest(): WorkspaceDbManifest on the WorkspaceDb interface. I switched it to readonly manifest: WorkspaceDbManifest, maybe the docs generator likes that better.

EDIT: Yeah, it's happy now.

[williamkbentley]

EDIT: Yeah, it's happy now.

I cringe when docs generation dictates code design. @ben-polinsky , maybe something to look into

[Josh-Schifter]

• I think workspace.md could be improved by expanding a bit on the opening paragraph. For example, a small block diagram showing how Resources are stored inside containers and referred to by Settings. As it is now, it is easy for the reader (of the first paragraph) to gloss over the distinction between settings and resources.

• As far as I can see, workspace.md does not attempt to address any of the questions I raised previously. Has there been an attempt to answer those somewhere else?

[pmconne]

I think workspace.md could be improved by expanding a bit on the opening paragraph. For example, a small block diagram showing how Resources are stored inside containers and referred to by Settings. As it is now, it is easy for the reader (of the first paragraph) to gloss over the distinction between settings and resources.

The article is designed to gradually introduce concepts rather than overwhelm by front-loading everything. But a diagram in the intro that shows the organization of a worksapce without delving immediately into the specifics is a good idea.

As far as I can see, workspace.md does not attempt to address any of the https://github.com/iTwin/itwinjs-core/pull/6646#issuecomment-2127684175.

I find this a little baffling. I attempted to specifically address these questions. The exceptions are those about workflows and UI for admins to manage workspaces, because those will be in a Mango app which is not yet available. I refer to it obliquely here:

Note: Creating and managing data in workspaces is a task for administrators, not end-users. Administrators will typically use a specialized application with a user interface designed for this task. For the purposes of illustration, the following examples will use the WorkspaceEditor API directly.

The other exception is this:

What is the admin workflow for creating new versions?

• I assume that admins will make changes over multiple sessions before publishing a new version. Is that correct?
• How can they test their changes? Can they point their session to the unpublished version?

We discussed this via email. Currently it's not possible to edit a new version over multiple sessions. I proposed a solution for a follow-up PR. I can't document it here because it doesn't exist yet.

Unless you literally mean you didn't find answers to any of your questions, please be more specific about which questions remain unanswered. You should definitely be able to answer the question "What is the unit of versioning for workspace information".

[kabentley]

Currently it's not possible to edit a new version over multiple sessions.

I don't understand why not. It will be up to the editor app, but a "session" may end before the write lock is released.

[pmconne]

I don't understand why not. It will be up to the editor app, but a "session" may end before the write lock is released.

Are you suggesting the admin user request the write lock, overriding the default expiration time of 1 hour with something like 1 week, and make sure they remember to release the write lock before the end of the week?

That is awkward. It doesn't allow the user to switch to a different device, or for them to release the lock so a different user can contribute their own changes to the same version. It doesn't permit sharing the WIP new version with other (potentially non-admin) users to test.

We talked about the idea that pre-release versions should be editable even after the changes are "published", to address these problems.

[pmconne]

a small block diagram showing how Resources are stored inside containers and referred to by Settings

@Josh-Schifter I am bad at diagramming. Does this come anywhere close to capturing what you want to see?

[Josh-Schifter]

I find this a little baffling.

Ok. My statement was overly broad. I'll try to narrow down.

If the answer to my questions on workflows is 'we don't know yet'... that is valid. It doesn't change the fact that those questions were never addressed.

The exceptions are those about workflows and UI for admins to manage workspaces, because those will be in a Mango app which is not yet available.

Is the intention that Bentley will develop a single app which admins will use to manage workspaces?

The answer regarding 'what is the unit of versioning', seems to be 'WorkspaceDb'. But a WorkspaceDb "can contain any number of resources of any kind"... which I guess means that the implementation doesn't care which resources are versioned together. That's nice but it doesn't answer the original question. Who will decide what goes into which workspaceDb? Will it be platform/bis, or app developers, or cad admins?

Likewise, I asked about how new versions of resources will be consumed by iModels. I get that there will be a setting which will make the new version available. I guess that setting will be modified by the admin using the as-yet-defined Mango app? That doesn't explain how text style changes will get propagated to text elements.

All my questions were "answered / not answered" in this same style. Sure, some of the low lying mechanics are more clear now, but the workflows are not. Who is responsible for defining the workflows?

[Josh-Schifter]

a small block diagram showing how Resources are stored inside containers and referred to by Settings

@Josh-Schifter I am bad at diagramming. Does this come anywhere close to capturing what you want to see?

Since the diagram is meant to go with the first paragraph, I'd like it to include all the main terms from there, and only the ones from there. As of now, the main terms from the first paragraph are Workspace, Setting, Container, and Resource. But the diagram is missing 'Resource' and includes the additional terms 'Settings Dictionary' and 'WorkspaceDb'.

[pmconne]

I'd like it to include all the main terms from there, and only the ones from there.

I don't want the diagram to lie by omitting dictionaries and workspace dbs. Perhaps the intro paragraph should briefly mention them.

[Josh-Schifter]

I don't want the diagram to lie by omitting dictionaries and workspace dbs. Perhaps the intro paragraph should briefly mention them.

Fine with me.

[pmconne]

All my questions were "answered / not answered" in this same style. Sure, some of the low lying mechanics are more clear now, but the workflows are not. Who is responsible for defining the workflows?

I see the disconnect - thanks. This PR, including the documentation, is about an API that workflows can be built on top of. The API is meant to enable various user workflows, not to impose a specific one. Applications are going to want to configure their default workspaces. Administrators will want to customize those and define their standards. Both will organize their resources in the ways that makes sense for them. Platform has a limited role: providing the APIs and a couple of broadly useful, publicly accessible workspace containers for things like GCSes and fonts.

Certainly a bunch of programmers are not going to define the end user workflows.

[pmconne]

@Josh-Schifter does this work better? Updated intro paragraph as discussed.

[Josh-Schifter]

@Josh-Schifter does this work better? Updated intro paragraph as discussed.

I like it. Thanks.

[Josh-Schifter]

Certainly a bunch of programmers are not going to define the end user workflows.

How do we know this system is suitable for purpose when we don't even know what the purpose is?

Repeating what I wrote before: Who is responsible for defining the workflows?

[pmconne]

How do we know this system is suitable for purpose when we don't even know what the purpose is?

Repeating what I wrote before: Who is responsible for defining the workflows?

Come on now. We know what the purpose is. The details of how the functionality is exposed to end users is a question for PM and UX folks from application teams. Could we have started with the workflows and designed the API based on them? Maybe, but we didn't. We didn't define the workflows of how an end user places text into a drawing before we implemented the TextAnnotation APIs either.

Discussion with PM and UX regarding end user workflows isn't going to occur here on github.