How to render remote markdown using content layer API

[johnmw]

📚 Subject area/topic

markdown, content layer api

📋 Suggested page

https://docs.astro.build/en/reference/configuration-reference/#experimentalcontentlayer

📋 General description or bullet points (if proposing new content)

I think that with the new content layer API, rendering remote markdown (from a server or api) is going to be a common use case.

The Astro docs currently says "Astro does not include built-in support for remote Markdown outside of experimental content collections!"

However I was unable to find any further information or examples about rendering remote (server) markdown.

So firstly, I was wondering if there already is an good example that I have missed?

If not, I have got some rough code (shown below) that renders markdown using Astros built in functions that seems to work and might help someone. But I would really like someone to review it and let me know things that can be improved.

Example Loader that renders remote markdown:

import { defineCollection, z } from 'astro:content';
import type { Loader } from "astro/loaders";

function markdownTestLoader({}):Loader {
  return {
    name: "markdown-test-loader",
    load: async ({ config, store, parseData, generateDigest, entryTypes }) => {
      
      store.clear();
      
      // Would normally load this JSON data from API. 
      // The "content" field has the markdown I want to render
      const dummyAPIData = {
        id: "home",
        title: "Test Markdown Data",
        content: "## Markdown to Render \nTesting markdown rendering.\n- List item 1\n- List item 2"
      }
      
      const id = dummyAPIData.id;
      const data = await parseData({ id, data:dummyAPIData });
      const body = dummyAPIData.content; // markdown to render next
      const digest = generateDigest(data);

      // Render markdown to HTML using built in Astro markdown render function
      let rendered;
      const markdownEntryType = entryTypes.get(".md");
      if( markdownEntryType?.getRenderFunction ) {
        const render = await markdownEntryType.getRenderFunction(config);
        rendered = await render?.({
          id,
          data,
          body,
          digest
        });
      }

      store.set({ id, data, rendered, digest });
    }
  }
}

// Define collection with loader

const markdownTestCollection = defineCollection({
  loader: markdownTestLoader({}),
  schema: z.object({
    id: z.string(),
    title: z.string(),
    content: z.string()
  })
});

export const collections = {
  'markdownTestCollection': markdownTestCollection  
};

The markdown HTML can be displayed on the page using the "render" function and built in "Content" component:

---
import { getCollection, render } from 'astro:content';

export async function getStaticPaths() {
  const pages = await getCollection("markdownTestCollection");
  return pages.map((page) => ({    
    params: { 
      slug: page.id },
    props: { 
      page 
    },
  }));
}

const { page } = Astro.props;
const { Content, headings } = await render(page);
---
<h1>{page.data.title}</h1>
<Content />

As I said, I'm sure there are things above that could be done better.

🖥️ Reproduction of code samples in StackBlitz

No response

[Fryuni]

We don't add guides and recipes for experimental APIs, only the refence and examples in the description there. So it won't be added on the docs for v4.

The feature is stabilized on v5, so we can document it there. The docs for the LoaderContext is missing the entryTypes property which enables rendering the different content types. We can add documentation for that and how to use it on the guide for writing a loader.

[sarah11918]

Thanks for this @johnmw ! Just showing off working code is already very helpful to others, and I appreciate your care in wanting to further make sure that the code is something recommended for others to copy.

Fryuni is right that any docs for this would have to go on our v5 beta docs. I will also ask @ascorbic to take a look to ensure that we end up with something that we'd recommend as a "happy path" as a general model!

If so, then I also agree this is for the Loader API page, in one of two ways:

• an example of a loader that fetches remote Markdown (the first code snippet only. I think the other code examples are pretty standard usage once you have the loader configured properly? There's nothing there that seems unique to having built that particular loader?)

or

• a link from this page to a full recipe set of instructions that includes everything above, but uses a real data fetch from a real API. This should be something that someone can follow along and have something working in their project at the end.

[ascorbic]

I agree! Adding it to the Loader API page would work best.

[trespaul]

I just want to add that there is a single sentence in the docs that — other than the "outside of experimental content collections" bit — made me go on an hours-long wild goose chase, that being

Once queried, you can render Markdown and MDX entries to HTML using the render() function property. Calling this function gives you access to rendered HTML content, including both a <Content /> component and a list of all rendered headings.

at https://5-0-0-beta.docs.astro.build/en/guides/content-collections/#rendering-body-content

The docs need to specify that the user needs to build that functionality themselves. I'd make a PR but I don't think I know enough about Astro to cover all the bases.

Also, this would be a nice built-in function to have anyway, since, like @johnmw said, rendering markdown is probably one of the common use cases.

Update: I think v5 doesn't have entryTypes, so I cobbled together the following (with reference to this):

import { defineCollection } from 'astro:content';
import type { Loader, LoaderContext } from 'astro/loaders';
import { createMarkdownProcessor } from '@astrojs/markdown-remark';
import { getData } from 'wherever';

type Collection = 'pages' | 'updates';

function myLoader(options: { collection: Collection }): Loader {
  return {
    name: 'myLoader',
    load: async (context: LoaderContext) => {
      const data = await getData();
      const processor = await createMarkdownProcessor(context.config.markdown);
      data.forEach( async item => {
        const id = item.slug; // I'm using slug as id
        const data = await context.parseData({ id, data: item });
        const digest = context.generateDigest(data);
        const rendered = await processor.render(item.content ?? '');
        context.store.set({
          id,
          data,
          digest,
          rendered: {
            html: rendered.code,
          },
        });
      });
    },
  }
};

function makeCollection(name: Collection) {
  return defineCollection({
    loader: myLoader({ collection: name }),
    // schema: z.object...
  });
}

export const collections = {
  pages: makeCollection('pages'),
  updates: makeCollection('updates'),
};

[sarah11918]

Just pinging for freshness here now that the new v5 documentation with stable Content Layer is out! If someone feels there is still an addition to make to the new documentation, please suggest so!

[dallyh]

I just want to drop this here, there is a way to fetch remote markdown (probably not officially supported) with the @astrojs/markdown-remark package with frontmatter and headings. My only worry is that it could break anytime, because the @astrojs/markdown-remark is not documented anywhere because it is probably intended only for internal use.

import { defineCollection, reference, z } from "astro:content";
import { locales } from "@i18n-config";
import { glob } from "astro/loaders";
import type { Loader, LoaderContext } from "astro/loaders";
import { createMarkdownProcessor, parseFrontmatter } from "@astrojs/markdown-remark";

const postSchema = z.object({
	translationKey: z.string(),
	title: z.string(),
	pubDate: z.coerce.date(),
	modDate: z.coerce.date().or(z.string()).optional(),
	hidden: z.boolean().optional(),
	description: z.string(),
	featured: z.boolean().default(false).optional(),
	language: z.enum(locales as [string, ...string[]], {
		errorMap: () => ({
			message: "Please select the correct locale!",
		}),
	}),
	image: z
		.object({
			src: z.string(),
			alt: z.string(),
		})
		.or(z.string())
		.optional(),
	tags: z.array(reference("tags")),
	canonicalURL: z.string().optional(),
});

function markdownTestLoader({}): Loader {
	return {
		name: "markdown-test-loader",
		load: async ({ config, store, parseData, generateDigest, logger }: LoaderContext) => {
			store.clear();

			const res = await fetch("https://raw.githubusercontent.com/dallyh/daliborhon.dev/refs/heads/main/src/content/dev/posts/cs/2023-01-01-typography-example.md");
			const text = await res.text();

			const processor = await createMarkdownProcessor(config.markdown);
			const parsedFrontmatter = parseFrontmatter(text);
			const frontmatter = parsedFrontmatter.frontmatter;
			const rendered = await processor.render(parsedFrontmatter.content ?? "");

			const id = `${frontmatter.language}/${frontmatter.translationKey}`;
			logger.info("Loaded post: " + id);
			const data = await parseData({ id, data: frontmatter });
			const digest = generateDigest(data);

			store.set({
				id,
				data,
				digest,
				rendered: {
					html: rendered.code,
					metadata: {
						headings: rendered.metadata.headings,
						frontmatter: frontmatter
					}
				},
			});
		},
	};
}

const posts = defineCollection({
	loader: markdownTestLoader({}),
	schema: postSchema,
});

[johnmw]

Just pinging for freshness here now that the new v5 documentation with stable Content Layer is out! If someone feels there is still an addition to make to the new documentation, please suggest so!

Thanks for your work on the new docs @sarah11918 .

It would be nice if there was an officially sanctioned recipe in the docs for rendering remote markdown so we know our own methods won't suddenly break (apologies if this documentation does exist and I missed it).

There are a number of working examples above.

For example with my original 'attempt', it looks like "entryTypes" is still being passed as a parameter to the 'load' function but is still undocumented. Does that mean I shouldn't be using it? Or iscreateMarkdownProcessor(config.markdown) safe to use?

Thanks!

[sarah11918]

My only worry is that it could break anytime, because the @astrojs/markdown-remark is not documented anywhere because it is probably intended only for internal use.

Yes, I believe it is intended for internal use, and although we have precedence for informing you when even internal things change, it's not guaranteed that we will and we typically don't consider keeping internal API consistent as a priority in development.

Both of the recent examples above do appear to make use of @astrojs/markdown-remark so maybe the first thing we should determine is whether people should be accessing this directly. There is a similar issue open now re: rendering remote Markdown/MDX for RSS feed content that relies instead on using the Container API and accessing those renderers rather than our internal markdown processing. I will have to check with the platform devs and see whether perhaps something more like that is the direction to take here.

I'll ask @delucis to have a look here for similarities/differences with the other issue and what he thinks about these examples!

[ascorbic]

I think it would make sense to make @astrojs/markdown-remark a public API, but we'd need to discuss it.

[gingerchew]

Just to continue showing interest in this, I think adding a LoaderContext.render(mdString: string): RenderedContent function would be the most helpful

function myLoader({ paths }): Loader {
	return {
		name: 'my-loader',
		load: async ({ store, render, generateDigest }: LoaderContext) => {
			for (const url of paths) {
				const result = await fetch(url);
				const body = await result.text();
				
				const digest = generateDigest(text);
				
				store.set({
					id: deriveIdFromPath(path),
					data: {},
					body,
					digest,
					rendered: render(body)
				})
			}
		}
	}
}

[ascorbic]

I don't think adding it to context is the right approach, because markdown is just one of the formats that it supports. It's better to explicitly choose the renderer that you need.

[sarah11918]

Just noting that this feels like maybe it's becoming less of a docs issue (we don't document what isn't publicly supported) and more of a should we expose this publicly (which admittedly is a docs issue, but then there might be implementation details and docs would depend on the explicitly public feature built, which would then get its own documentation at that time.

So, I'm wondering if this is in fact a conversation that should NOT be happening here in docs issues, but perhaps a roadmap discussion or something? @ascorbic

[sarah11918]

This does now feel like something for the roadmap repo as an idea to make this API public, so I'll close this as docs issue but really hope someone will start a discussion about making this public there!

@johnmw @ascorbic @gingerchew

[gingerchew]

I went ahead and made one real quick here would love others input because I know I'm only scratching the surface of what could be done with it.

[HawtinZeng]

Cannot package remote content into dist directory with astro build