Scaffold with differences

[jonifen]

The scaffold deliverable is great for getting an initial build set up on a new machine. However, it feels like it needs to go one step further to fit my use case.

I envisage that I would be able to create document types/templates etc. in my local copy of Umbraco, then scaffold the output which would result in a delivery file containing only the differences since last scaffold, that I can then commit to source control that CI/CD will then deploy onto a production environment.

I have tried doing a second scaffold (without the install step) and that runs through, but it only "installs" new document types and doesn't amend existing document types should I have made any changes to it.

Could it be that I am I going about things all wrong? Is there a suggested way to achieve what I'm trying to do?

[aaronpowell]

Short answer - I'd love to do this, it's just really hard.

I had a look at trying to do this a few years back, had a crack (the branch is still around) but ultimately wasn't able to get it working to the point I was happy to merge it in. The challenge that you have is how do you determine the current state to perform the "diff" against.

Let's say you've got 5 deliveries, and you're preparing to do the 6th, well you need to work out what has changed since the last time. But the last time only contains the doctype News Item, because that was added then, and since then you've edited News Item and you've edited Home Page. Now we need to basically build up a picture of "last known state", so we have to replay all the DocType/Data Type/Macro/Template/etc. creates and edits so we know what it looked like and compare that to the current database.

So you can see where the complexity resides, the last delivery represents the last set of chances which is a delta on all the previous ones, meaning to get last complete state requires analysis of all the previous deltas.

Now you can get around this, or at least simplify it somewhat, by going down the route that uSync does, having only a single file for each thing you're tracking, and just keep updating that one file. But as I've said in the talks I've given on automating Umbraco with Chauffeur I don't agree with the uSync model as it makes it a lot harder to understand at what point in time a change happened, I prefer that level of granularity.

Maybe there are other ways to approach it that are possibly not as foolproof but can be close to reliable like looking at the timestamp in the tracking table and finding items that have update dates since then, but that then means you have to ensure you run all your deliveries locally too.

Yeah, it's tricky 😛.

[jonifen]

😄 Thanks for the quick response.

This is similar thinking to mine... how to know if an item has been added or edited since the last Chauffeur scaffold was done.

I'll try creating a new deliverable to see if I can figure something out. But first I'll see if I can dig out your old branch first to make sure I don't go end up down the same path.

[aaronpowell]

Consider the old branch dead, it's quite diverged from master so I probably should just delete it 😝

[jonifen]

I opted to go with a tracking table that will be updated on saving an item (doc type/template etc.) and added a ChangesDeliverable to output the changes (which mostly mimics the ScaffoldDeliverable in many ways). I'm expecting to be able to run the DeliveryDeliverable as per usual, and it mostly works.

Any change to a template is carried across no problems (presumably as it's just replacing the file contents on disk). However, a document type will only update if something is added. If a field is removed, it doesn't remove it at the destination. I suspect that what is happening here is that the DeliveryDeliverable is only appending and not replacing things that are in the database?

Am I a million miles away with my thought process?

(Apologies if I'm missing a blatantly obvious point, I've only been working with Umbraco for about a fortnight, so I'm still pretty new 😄 )

Edit: I've created an example scenario in a repo here: https://github.com/jonifen/umbraco-chauffeur-changesets if this helps explain things at all?

Quick explanation of what delivery is doing what:

• Delivery '20180525_0213-Changes' introduces a second textbox to the document type and includes it in the associated template.
• Delivery '20180525_0237-Changes' then removes the second textbox from the document type and removes it from the associated template. The template updates correctly, the document type still contains the second textbox.

[aaronpowell]

@jonifen I can see the direction that you're going with that approach, but I disagree with it slightly. I'm not a fan of adding another database table if I don't have to, so I wouldn't want to have that additional tracking table that you've introduced.

Instead, I want to follow what Umbraco provides on determining updates of the stuff that it tracks. Now I originally (and probably naively) assumed you could just rely on the umbracoNode table to get an updateDate and just follow from there. But of course, there isn't an updateDate in that table is there, meaning that we can't rely on it 🤦‍♂️.

Not to be deterred through I did some digging and it looks like you can follow from the audit trail backwards (since it has a datestamp that you can work from). So I'd go with the approach of:

• Check get all the document type ID's from the umbracoNode table
• Cross-join them to the umbracoLog table
• Where-clause against that for the appropriate date range
• Use the newly found ID's in the Umbraco API to get the stuff for serialization
• Repeat for other things

Ultimately this means that I wouldn't have to have Chauffeur running with the Umbraco event system, and I wouldn't have to worry about tracking with my own database table. Also, it's easier to write tests for that 😉.

[jonifen]

Thanks for the pointers on this, I hadn't even noticed the UmbracoLog table 😊 I'll look into moving towards that idea instead.

I think either way of building the deliveries should still produce deliveries with the same expected content though, which leads me back to the delivering part of the process. It doesn't appear to be importing the document types etc. correctly if things have been removed from it prior to the package being created. Templates etc. seem OK as they just get copied in and replace the existing file, but the types that are stored in the database seem to only get added or appended to, but never removed or truncated. Any ideas around that?

[aaronpowell]

Ah sorry, missed that part of the comment.

Yes, removals are a bit... special. The export that you get from Umbraco of a doctype will only contain what the doctype looks like right now, so it doesn't know about any removed properties, that's simply not stored anywhere in the database. I addressed this as part of issue #58 by providing a way which you can remove a property from a doctype. It's a separate step which obviously makes it a bit more annoying.

Trying to automate the creation of that step though? Well, that becomes tricky. As I said originally the only truly foolproof way that you can do a "generate diff" step is that you have to replay all deliveries up until the present to know what "last known state" looks like, then compare the properties against each doctype and find the ones that were removed.

Otherwise, you have to do a bunch of stuff that tracks things while Umbraco runs, which as I said, I'm not really keen on doing.

[jonifen]

I've just realised that this issue is still open and I've not touched Umbraco in quite some time and can't remember my thought process at the time. With that in mind, I think this issue can be closed 🙂 Given the time passed, I'll close it as stale.