Statiq.Web
Statiq.Web copied to clipboard
statiqdev
Create OPML 2.0 Module
Read OPML outlines and produce documents with content and metadata.
http://dev.opml.org/spec2.html
I created Wyam.Modules.Opml.
With this module it should be trivial to:
- Consume OPML document and render them (with rendering rules).
- Produce OPML document.
Awesome - you are on a roll! I'm going to try and publish a release today with the images stuff, but there's nothing to prevent another release soon after once this is all the way ready.
https://github.com/Wyamio/Wyam/compare/develop...dodyg:develop
- Create Wyam.Modules.Opml.Tests using NUnit Framework like the rest of the projects in the solution.
- Implement IEnumerable<T> interface so you can use
foreachin going through every single outline (including children) in the document.
You are on fire!
I'm going to go ahead and package up a release this morning. I know there's still an open question on skipping existing files and some other goodies being worked on, but it's been a while and there's been so many improvements, I want to get what we've got so far out there. As we settle some of the other pending stuff I can put together another release later in the week.
Sounds good. I shall have the user manual catch up with the image module soon.
Sigh this module is slow going.
Now that we have all the outlines and its hierarchy, we have to figure out what options to give the module users.
The OPML file assumes the existence of text attribute. We can use this to feed the content property. The rest of the attributes within an outline will go to the metadata. The level of the outline will also go to the meta data as outlinelevel.
We can allow user to split the outlines at the outline level, e.g. split anything below 1 level to separate documents.
If the outline has children, then we need to decide whether to put those text inside p or treat them as a list ul.
One scenario that this module must support is to split outlines into blog posts like what Dave Winer is doing at scripting.com.
Another scenario is to leave the outline XML format intact after splitting.
There's no rush - that's the beauty of pre-release software :)
I think you've got the right idea on text going to Content and everything else going to metadata. That sounds like the perfect way to preserve as much as possible within the document convention. My only suggestion would be to TitleCase the metadata keys from the native XML convention of camelCase for attributes since that's more common in the .NET world (and is how the built-in keys are styled).
I'll be interested to see what you end up with because it could help inform some of the other XML-based modules like the RSS reader (#40).
https://github.com/dodyg/Wyam/commit/f0b7c53bd982e45db745789bcbac63a557d59da9
Add more test for outlines with attributes other than 'text'.
https://github.com/dodyg/Wyam/commit/ff12dd5d58add1ddb59487526711184395d980b9
Now you can format the outline based on its level and set up a default formatter when a level formattter is not specified.
var opmlRenderer = new OpmlTextRenderer().SetFormatter(1, (content, metadata) =>
{
return $"<h1>{content}</h1>";
}).SetFormatter(2, (content, metadata) =>
{
return $"<h2>{content}</h2>";
});
Awesome! I haven't had a chance to look too closely at this one yet, but what I've seen so far looks great. I'll try and take a closer look this weekend.
A working scenario config file
Assemblies
.LoadFile("C:\\Users\\Dody\\Documents\\GitHub\\Wyam\\Wyam.Modules.Opml\\bin\\Debug\\Wyam.Modules.Opml.dll");
===
---
Pipelines.Add("Opml",
Download().Uris("http://hosting.opml.org/dave/spec/placesLived.opml"),
OpmlReader(),
OpmlTextRenderer(),
Meta("WritePath", "outline.html"),
WriteFiles("*.html")
);
produces
Places I've lived Boston Cambridge West Newton Bay Area Mountain View Los Gatos Palo Alto Woodside New Orleans Uptown Metairie Wisconsin Madison Florida New York Jackson Heights Flushing The Bronx
That is excellent. I really like the separation of OPML parsing from OPML HTML rendering - that'll make it easier to parse and OPML file and then do something else with it. i.e., include it as part of a bigger Razor template. And I think you were also onto something above - the rendered HTML output could provide another great input source for PowerPoint generation. Great work.
What remains is a conditional based formatting when the outline is transitioned, e.g:
- Outline going downlevel (level 1 to level 2)
- Outline going uplevel (level 3 to level 2)
Assemblies
.LoadFile("C:\\Users\\Dody\\Documents\\GitHub\\Wyam\\Wyam.Modules.Opml\\bin\\Debug\\Wyam.Modules.Opml.dll");
===
---
Pipelines.Add("Opml",
Download().Uris("http://hosting.opml.org/dave/spec/placesLived.opml"),
OpmlReader(),
OpmlTextRenderer()
.SetFormatter(1, (content, meta) => $"<h1>{content}</h1>")
.SetFormatter((content, meta) => $"<p>{content}</p>"),
Razor(),
Meta("WritePath", "outline.html"),
WriteFiles("*.html")
);
@daveaglick I have a design problem with this module.
OpmlReader generates a document for every outline contained in the opml document, e.g. a document with 10 outline elements will produce 10 IDocument.
OpmlTextRender will take all those documents and produce a single document. The problem is I don't know how to accomplish that other than reusing a single document from the previous pipeline. This is definitely a wrong approach that I need to fix.
public IEnumerable<IDocument> Execute(IReadOnlyList<IDocument> inputs, IExecutionContext context)
{
var str = new StringBuilder();
foreach(var i in inputs)
{
var level = (int)i.Metadata[MetadataKeys.OutlineLevel];
if (Formatters.ContainsKey(level))
{
var render = Formatters[level];
var output = render(i.Content, i.Metadata);
str.AppendLine(output);
}
else
{
var output = DefaultFormatter(i.Content, i.Metadata);
str.AppendLine(output);
}
}
var result = str.ToString();
var inp = inputs.First();
var meta = new List<KeyValuePair<string, object>>();
var o = inp.Clone(result, meta);
return new[] { o };
}
Take a look at the various IExecutionContext.GetNewDocument() overloads: https://github.com/Wyamio/Wyam/blob/develop/Wyam.Common/Pipelines/IExecutionContext.cs I added a bunch of variants in the develop branch a little while ago to support the code analysis stuff, which had a similar problem. Hopefully one of them will do what you need. Let me know if not and we'll figure something else out.
Now we can reproduce HTML nested list properly
Assemblies
.LoadFile("C:\\Users\\Dody\\Documents\\GitHub\\Wyam\\Wyam.Modules.Opml\\bin\\Debug\\Wyam.Modules.Opml.dll");
===
---
Pipelines.Add("Opml",
Download().Uris("http://hosting.opml.org/dave/spec/placesLived.opml"),
OpmlReader(),
OpmlTextRenderer()
.SetFormatter((content, meta) => $"<li>{content}</li>")
.SetFormatter(OutlineDirection.Up, (content, meta) => "</ul>\n</li>")
.SetFormatter(OutlineDirection.Down, (content, meta) => "<li>\n<ul>")
.SetFormatter(OutlineStartOrEnd.Start, (content, meta) => "<ul>")
.SetFormatter(OutlineStartOrEnd.End, (content, meta) => "</ul><!-- end -->")
.SetWindDownText("</ul></li>"),
Razor(),
Meta("WritePath", "outline.html"),
WriteFiles("*.html")
);
produces
<ul>
<li>Places I've lived</li>
<li>
<ul>
<li>Boston</li>
<li>
<ul>
<li>Cambridge</li>
<li>West Newton</li>
</ul>
</li>
<li>Bay Area</li>
<li>
<ul>
<li>Mountain View</li>
<li>Los Gatos</li>
<li>Palo Alto</li>
<li>Woodside</li>
</ul>
</li>
<li>New Orleans</li>
<li>
<ul>
<li>Uptown</li>
<li>Metairie</li>
</ul>
</li>
<li>Wisconsin</li>
<li>
<ul>
<li>Madison</li>
</ul>
</li>
<li>Florida</li>
<li>New York</li>
<li>
<ul>
<li>Jackson Heights</li>
<li>Flushing</li>
<li>The Bronx</li>
</ul>
</li>
</ul>
</li>
</ul><!-- end -->
Now the formatting is in good shape https://github.com/dodyg/Wyam/commit/31ad57716436e0c1a21935f103029168488a1c68. A couple of more tests needed. I will make a pull request when they are done.