dotnet watch with .NET 9 fails to update markdown

[phil-scott-78]

I was trying to run with .NET 9, and it seems dotnet watch isn't calling MetadataUpdateHandler for content files changes. After updating a markdown file, I'll see the messages that a change has been detected, but the call to MetadataUpdateHandler doesn't occur. There might also be a race condition introduces for updating razor files too that I want to track down, sometimes when editing those I'll get an exception about collections being updated while enumerating them in BlazorStaticService, a different collection each time. I suspect multiple calls are coming in at once.

I found this issue in aspnetcore regarding resx files not being updated with a similar pattern, so it might be the issue to track with ASP.NET - https://github.com/dotnet/aspnetcore/issues/60835

[tesar-tech]

Would you mind to switch the target framework to dotnet 8 and test it there?

[phil-scott-78]

actually, you are right - markdown edits aren't reflected in .net 8 either. Only changes to razor (or csharp).

I went as far as creating an incredibly minimal ASP.NET app with only a single class marked MetadataUpdateHandler. Same thing with .net 8 and .net 9. . Code changes yes, content files no. Logger indicates it sees the file updated, informs no C# changes to apply and does not call the MetadataUpdateHandler.

[tesar-tech]

That's weird, because it worked for dotnet 8 for sure. There were always some troubles tho. Can you share the minimal repo? I will start from there.

[phil-scott-78]

sure, I was trying to track this down with about as basic of a ASP.NET app as possible here https://github.com/phil-scott-78/hotreloadbug

as to reproducing with BlazorStatic, just a dotnet new with the template will do it.

one thing I discovered this morning - in the 9.0.3 SDK that was released a few days ago, they included a potentially interesting addition to the MetadataUpdate code - https://github.com/dotnet/sdk/blob/release/9.0.3xx/src/BuiltInTools/HotReloadAgent/MetadataUpdateHandlerInvoker.cs

I've included the change, but I can't get it to trigger for the life of me.

public class Reload
{
    public static void ClearCache(Type[]? types)
    {
        Console.WriteLine("Clear Cache");
    }

    public static void UpdateApplication(Type[]? types)
    {
        Console.WriteLine("Update Application");
    }

    public static void UpdateContent(string assemblyName, bool isApplicationProject, string relativePath,
        byte[] contents)
    {
        Console.WriteLine("Update Content?");
    }
}

[tesar-tech]

Ok, I see and I can reproduce the issue, but in a weird way. The hotreload is probably somehow platform dependent and also IDE dependent:

using dotnet watch outside of an ide:

• dotnet 8 hot reload works, but not on md content - exactly as you reported
• dotnet 9 dotnet watch doesn't even bother to work with c# or razor files, it just crashes with errors like:

dotnet watch ❌ error CS0518: Predefined type 'System.Void' is not defined or imported
dotnet watch ❌ error CS0518: Predefined type 'System.String' is not defined or imported

using watch in Rider:

• both "work" the same way - for c# file, but output the

dotnet watch ⌚ File updated: ./Content/file.md
dotnet watch ⌚ No C# changes to apply.

for md files.

OS: linux, dotnet --list-sdks 8.0.406 9.0.200

I don't know, it seems to me they butcher it somewhere before 8.0.406, but I am not sure. When I implemented the hot reload into BlazorStatic, I still had my Windows machine - so it might be platform dependent somehow.

What about you sdks, os?

The issue you reference is spot on. Please do upvote it - so it gets some attention. Maybe we should also share the repository of yours in there.

Sorry for not having satisfactory answer, but it seems to be out of my reach rn.

The UpdateContent looks interesting. How did you even find it?

[phil-scott-78]

I'm on Windows, using Rider as the ide but I live in the command line so I'm running everything via dotnet watch

plenty of sdks to test too, but behavior is pretty consistently off

7.0.115 [C:\Program Files\dotnet\sdk]
7.0.312 [C:\Program Files\dotnet\sdk]
7.0.407 [C:\Program Files\dotnet\sdk]
8.0.101 [C:\Program Files\dotnet\sdk]
8.0.203 [C:\Program Files\dotnet\sdk]
8.0.303 [C:\Program Files\dotnet\sdk]
8.0.401 [C:\Program Files\dotnet\sdk]
9.0.100 [C:\Program Files\dotnet\sdk]
9.0.201 [C:\Program Files\dotnet\sdk]

I suspect the "best" solution for the content files might just be a file system watcher. I went down a very similar route 4 years ago and had some success with that. My watch simply set a flag telling my model that it needed to refresh itself. So, in the case of BlazorStaticContentService the equivalent would have been the Post property checking for that flag, if good return the previous run if not setting a lock, then regenerating, cache results, and return.

For humongous sites, it'll seem a bit slower for the client because the client will refresh, and the browser is gonna be hung waiting for the rebuild. But also, for humungous sites, I believe the behavior you'd see is that the browser would refresh and serve up stale content as the refresh happens in the background.

[phil-scott-78]

oh, I found the UpdateContent method in the unit tests of the sdk and got curious. Without documentation, and honestly being pretty ignorant of the sdk solution, its hard for me to figure out the intention of what it is supposed to be used for. Only place I see it being used between both the sdk and aspnetcore is in a few unit tests. Got me.

[phil-scott-78]

ok, had a few moments. I was able to add a FileSystemWatcher for the content and had everything wiring up seeing events and the such firing as expected. I felt clever until I realized the content wasn't being updated still, even though everything said it was hitting the disk again as expected.

Took me a second, but I realized the issue - the engine is reading out of the execution directory, which is gonna be \bin\Debug\net9.0 or \obj\Debug\net9.0 when using dotnet watch. When I change a file my filesystem monitor was watching Content/Blog that lives off the project root, the GetPostsPath is grabbing content off of executing assembly in the bin or obj folder. That content isn't being regenerated.

I suspect I should be to adjust everything out of the current path like the File System Watcher, but with this being explicit I figure there is a reason it is in there so I should probably ask before investing too much time.

[phil-scott-78]

kids got sucked into Bluey so I had way more free time than expected tonight. Quick first pass at it relying on just Directory.GetCurrent rather than running based off the application assembly location. https://github.com/BlazorStatic/BlazorStatic/compare/master...phil-scott-78:BlazorStatic:file-system-watch?expand=1

A nice part is that with the file system watcher going that the markdown reloading applies if you are running via dotnet run or dotnet watch which is nice. Haven't tested things like wwwroot or linked media, but so far I'm pleased. Might hammer away more this weekend if my kids are as easily distracted.

[tesar-tech]

There is quite a lot to see in your file watcher branch.

I will hop on it later today, but please consider this:

https://github.com/BlazorStatic/BlazorStatic/pull/10

There is a reason why the md files are taken from bin folder - user can process them without affecting the original files.

Do you think, there is a way how to watch for changes in Content while copying the files into bin for the processing itself?

---

dotnet run or dotnet watch which is nice.

it indeed is.

[phil-scott-78]

Ah, I knew there was a reason this was being explicitly done, I just missed that PR. Use case makes sense, but feels like relying on that has a potential to be brittle. I went to see how @MeltyObserver was using it, but I didn't come across it the few repos I checked this morning.

But. this falls into a use case I was thinking about - I'd like to hook in and edit the HTML generated so I can do syntax highlighting server side rather than relying on clientside generation.

A series of hooks like this on BlazorStaticContentOptions<TFrontMatter> might prove pretty useful rather than relying on files being in the right spot and ease the default configuration too.

    /// <summary>
    /// Gets or sets a hook to process the markdown files before they are rendered as HTML.
    /// </summary>
    public Func<string, string> PreProcessMarkdown { get; set; } = s => s;

    /// <summary>
    /// Gets or sets a hook to process the front matter and html after markdown parsing and before it is passed to Razor.
    /// </summary>
    public Func<TFrontMatter, string, (TFrontMatter, string)> PostProcessMarkdown { get; set; } = (frontMatter, html) => (frontMatter, html);

[tesar-tech]

The advantage of having the files copied to the bin directory is the freedom you have when you process the files. You can do some overall calculations, you can process files in batches, etc.

Why do you think it's brittle?

tbh - I don't like the need to specify the content to be copied inside the csproj, but that's all I can said against it.

Maybe we can implement our own files copy, and doing such copy on every file change. wdyt?

---

As an alternative you suggest PreProcessMarkdown, but that's processing the files one by one.

You can replicate the PreProcessMarkdown with

builder.Services.AddBlazorStaticService(opt => {
   opt.AddBeforeFilesGenerationAction(() => {

            //retrieve md files, do something with them
            return Task.CompletedTask;
        });

For the PostProcessMarkdown we have

   .AddBlazorStaticContentService<ProjectFrontMatter>(opt=> {
        opt.AfterContentParsedAndAddedAction = //...
    });

Maybe the action should be actually a Func to allow async methods.

---

I saw the code you prepared, I like the file watcher. And that's something we can really utilize. But the overall solutions seems complicated and not necessary better.

My main objection is against your edits of async methods. Please keep them async. You can easily solve it by having

    private readonly List<Func<Task>> _updates = []; //new
 //   private readonly List<Action> _updates = []; //old

Or maybe even better:

private readonly List<string,Func<Task>> _updates = []; //string is filename

With the filename we do actions like copying the files for example.

---

Then there are changes that doesn't fit the current topic much. Maybe you improved some stuff and that's valuable, but we should focus on file watch only - and do the improving stuff later.

What I see as a solution is

• get the update with filewatcher,
• copy the updated file to bin folder
• run BlazorStaticExtensions.UseBlazorStaticGeneratorOnHotReload();

This will have the advantage of keeping the changes at minimum - we will fous on the filewatch only. Then we can review if the UseBlazorStaticGeneratorOnHotReload is a good solution or not. If we mix it here, it's gonna get unnecessary complicated.

Thank you for your contributions, please let me know I misunderstood something or you disagree with the steps I proposed.

[phil-scott-78]

yeah, some things got away from me solving the concurrency issues with multiple changes at once. One of the reasons for the move to the synchronous code was because I was lazy loading in the Posts property, so I'd just be wrapping up the async code in a Task awaiter to run code that isn't gonna be used in a scenario where we are in a wait state anyways. But I suspect if I drop everything and try and revert back to a file system watcher that just copies into the output folder going back to the old way that wouldn't be needed. Concurrency problems would still be there, but I believe it fails in such a way that it still works just with some exceptions in the dotnet watch output.

When I said brittle that was probably a poor choice in words. It was just unexpected. User would have to know that they need to be doing read/writes to the content out of that folder and pull it by hand, and then I don't believe it is consistent as things like wwwroot or media are served from those folders. Nothing that can't be figured out. Let me think about how to best get the file system watcher moving those files into the bin folder and tackle the rest another day.

[phil-scott-78]

I got after this a bit the other night. The issue I kept running into is that with the file system watcher multiple changes can fire off at once. A File | Save All is one scenario, but also some changes get marked as a Delete and Create. Need to figure out a way to debounce or throttle all the messages at once.

Working on this I went off the rails a bit and ended up with a fork more focused on the dotnet watch scenario that I'm pretty happy with right now. But it did involve some rather significant rewrites that I don't think would make sense merging back in. I only mention this because I think this issue is a good one for the greater good for someone to tackle here, and I don't want anyone waiting on me to get it done.

[tesar-tech]

So you figured out the dotnet watch? Does it work for you?

The debounce, for example 1.5 seconds, sounds like a good strategy to me. But I am unsure how it the file watch reliable.

[phil-scott-78]

for how I want it to work it took a pretty significant overhaul, one thing led to another and its wildly different from the project now. it's proven quite reliable, but only due to some big architectural changes built around the idea it can happen. But I made those changes without the intention of merging back in here. It doesn't take into account scenarios like editing in the output folder, for example.

so I think the best course here is the debounce. It'll also require hooking into the the dotnet watch system to somehow trigger a refresh which might be a fun project.

[junovell]

Hello sorry for the late reply, i opened #10 because i wanted to have filters for my blogs and wanted those filters to run before processing the markdown files.

thinking about it now i should've just implemented a filter delegate that runs before processing the blogs, what do you think?

we can have an AddFilter<T>() extension that takes a delegate Func<string, string> and runs it before processing the files

[phil-scott-78]

I landed on the func option too. I have one that runs right after the file is read called preMarkdown and another that runs after the conversation to HTML called postHtml that gives a chance that takes and returns a (TFrontmatter, string HTML) tuple. Both include a IServiceProvider parameter so I can grab options or registered helpers too.