Add functionality to return full "AST"

[phiresky]

Currently there's only the method anchorme() which returns only text and anchorme.list() which returns each link as a span with meta information.

It would be useful to have a method that returns a full list of "tokens" representing the input text that is the same as anchorme.list() except with text spliced in between.

This is needed for any use case where you want to convert the output into a different format, such as HTML.

react-anchorme implements this like this:

https://github.com/potty/react-anchorme/blob/0a4b8c1e1eaff27f645fbe6a83d32739654357e9/src/Anchorme.tsx#L14-L38

For example:

anchorme.listIncludingText("You can try google.com.")

would return

[
  { isText: true, start: 0, end: 12, string: 'You can try ' },
  {
    start: 12,
    end: 22,
    string: 'google.com',
    isURL: true,
    host: 'google.com',
    confirmedByProtocol: false,
  },
  { isText: true, start: 22, end: 23, string: '.' }
]

Here's how the implementation could look and what I'm using right now:

type FullAnchorme = ListingProps | ({ isText: true } & BaseTokenProps)
export function listIncludingText(text: string): FullAnchorme[] {
	const matches = anchorme.list(text)

	const elements: FullAnchorme[] = []
	let lastIndex = 0
	matches.forEach((match) => {
		// Push text located before matched string
		if (match.start > lastIndex) {
			elements.push({
				isText: true,
				start: lastIndex,
				end: match.start,
				string: text.substring(lastIndex, match.start),
			})
		}

		// Push Link component
		elements.push(match)

		lastIndex = match.end
	})

	// Push remaining text
	if (text.length > lastIndex) {
		elements.push({
			isText: true,
			start: lastIndex,
			end: text.length,
			string: text.substring(lastIndex),
		})
	}

	return elements
}

(only tested a bit)