Corrupted Grafico files

[jbsarrodie]

This is a copy of an issue create by @stefjoosten on Archi repository but related to GRAFICO

Grafico is, if I'm informed correctly, meant for multi-user support with Git. In principle, this works fine. Except in some cases.... In our project, we have experienced three times that Grafico-files were corrupted. This issue documents the most recent occurence.

An Archimate-repository of Bart den Dulk (bdendulk) contains an example of a corrupted Archimate repository. Loading it in Archi by means of a Grafico-import goes well. It can then be inspected and changes as any other Archimate-repo. However, saving the (unchanged) model as an ordinary Archimate-repo yields the following error:

Analysis

1. It seems that the example contains 4 violations of a system invariant of Archi. Maybe Archi does not check all of its system invariants when importing from Grafico. I would expect a corrupted file to be rejected when it is imported.
2. In commit "Herstel actie 4 orphaned relaties" @bdendulk has fixed the problem by hand. The difference between this commit and the original gives helpful symptoms of the problems. Bart's fix is consistent with the error message shown above.

Diagnosis

1. It seems to me that Archi should be more particular in guarding its system invariants.
2. The question how the Grafico input got corrupted needs to be answered as well. For now, we can only guess. One (and the only) guess is that a commit in Git was performed before Archi was done saving the model to Grafico. There is no experimental validation of this guess yet.

[bdendulk]

This issue is related to the following discussion on the forum. I can solve the error message by looking for the relations that are shown on the diagram mentioned in the error message but are not present in the repository.

This can be tracked by clicking on a relation in the view pane (with the mentioned diagram opened) and then look into the Analysis tab in the Properties pane. Here you will see that the relation actually is not present in any diagram but is however shown on the viewed diagram. By deleting the relation in the diagram the error message disappears.

This has happened to me already 3 times and now a collegue encountered the same type of issue as well.

[bdendulk]

http://forum.archimatetool.com/index.php?topic=208.msg1062#msg1062

[jbsarrodie]

Hi,

Re Diagnosis #1: Archi is not responsible for this issue as the Grafico plugin hook into the code and doesn't use existing model loading code.

Re Diagnosis #2: I guess I know how this happened. At least this can easily be simulated if two people clone the same repo at the same commit id and both use the same elements for quite different purpose: user #1 delete them while user #2 add them in new views. Both users can save/commit/push to the central repo, merge can be done without any issues (because at the end, users haven't changed the same files) but when trying to import the whole model, elements refered to from new views no more exist. This is not checked by the plugin and you can still work on the whole model, but when trying to save to a .archimate file, more in depth checks are done and the issue is raised.

In theory this should not happen if work is well splitted between architects, and is very similar to what would happen in development if a developer deleted some part of the code (say a java method) while another developer make use of this exact code elsewhere. That's exactly why continuous integration system were created for: check overall coherence at repository level, not file level.

So what can be done?

• The immediate solution could be to explain all your colleagues that they should not delete elements in the model, and have only one architect allowed to do it when a clean-up is really needed.
• An alternative could be to commit your changes in 2 times: the first commit would contain new and changes files while the second one would contain the deletion
• In the mid term, I'm planning to add some control when loading to detect such issues and do a cleanup (ie. deleting elements that still exist in views but are no more in the model). I'll do this for sure for Archi 4 but this could be done for Archi 3.x also.
• In the longuer term, we are planing to build a real repository for Archi that would be built upon a kind of GitHub clone and add convertion to/from .archimate files. As the conversion will be done on the server, several mecanisms will be implemented to split add/change/delete commits and do some automatic cleanup with associated commit if needed.

[bdendulk]

Hi JB,

I am almost certain I can confirm diagnosis #2. This is what actually happend concerning the relations mentioned in the error message. Actually person 2 copied parts of a diagram (including the mentioned relations) and started working. In the meantime I (person 1) deleted the relations referenced to in the error message in the original diagram. The branch with the deletions were then merged with the branch being worked on by person 2.

[bdendulk]

In addition to my previous comment: sofar the issues only stems from relations not being shown in the Analysis tab while being present on the view. The error does not seem to occur with other type of elements.

[jbsarrodie]

Hi,

I've just published a new beta release that supports Archi 4 and now warns when XML files are missing. I'm working on a new version that will cleanup the mess in such cases, but I have to figure what the best solution is (delete screwed concepts or create new one with empty name, empty doc...)

Regards

JB

[mwherman2000]

How is model versioning handled? ...what do you envision versioning be implemented? ...wouldn't versioning address this issue? ...that is, views created within a particular version of a model will always be consistent with the underlying repository content?

[stefjoosten]

All of the above sounds like working on solutions, whilst not having a proper diagnosis. In that case, I can give my two-cents worth solution as a proposal as well.

It is possible that the problems are caused by conflicting keys in the internal data structure of Archi. These keys are exported to Grafico-files. Keys are always meant to keep objects within one data-space apart. However, by exporting them to Grafico and subsequently to a Git-repo, the data-spaces of multiple users are merged into combined data-spaces. This is a likely explanation for the high probability of Git-conflicts that we have experienced in practice. It is also consistent with the screenshot we see above. (I would like to see some empirical evidence to corroborate or invalidate this hypothesis).

If this analysis holds, the software must guarantee that different users cannot generate conflicting keys. This is possible by encoding the computer-user combination in the key values.

[jbsarrodie]

@stefjoosten As I said in a previous comment, this issue has been fixed on the latest version of Grafico: the plugin will check for consistency and warn if an issue happened (and provide the list of missing concepts). As it is now, this is only a warning and you have to fix it (the most probable solution being to roll-back a merge), but in the next release I'll add an option to fix it for you (but this will of course be the simpliest option: delete orphaned references).

[jbsarrodie]

@mwherman2000 Versionning is not managed by Grafico but by your underlying versioning system.

In this case the issue was very likely due to logically conlicting edits but technically valid merge. The same kind of issue exist with development and lead to successful merge but non working software. This has long been salved through continuous integration and continuous testing. So the same has to be implemented in what I call the Grafico ecosystem: there should be a continuous integration system plugged to the Git repository that would check integrity after (or even before) each an every (merge) commit. But as for the moment there is no such global solution, the plugin will check for inconsistency during loading (but that's an "after the fact" control).