Make illegal character sanitization more robust

[mmcdole]

Following issue #180, #25 and some other issues, I'd like to make character sanitization more robust.

I've previously tried to have the code do something like the following:

func sanitizeXML(xmlData string) string {
	var buffer bytes.Buffer

	for _, r := range xmlData {
		if isLegalXMLChar(r) {
			buffer.WriteRune(r)
		} else {
			// Replace illegal characters with their XML character reference.
			// You can also skip writing illegal characters by commenting the next line.
			buffer.WriteString(fmt.Sprintf("&#x%X;", r))
		}
	}

	return buffer.String()
}

func isLegalXMLChar(r rune) bool {
	return r == 0x9 || r == 0xA || r == 0xD ||
		(r >= 0x20 && r <= 0xD7FF) ||
		(r >= 0xE000 && r <= 0xFFFD) ||
		(r >= 0x10000 && r <= 0x10FFFF)
}

However, there is an old issue #21 that indicated that when I sanitized these characters, it then messed up parsing non-utf8 feeds.

If anyone has any suggestions for how to accommodate both requirements:

• Stripping illegal characters from feeds to prevent the xml parser from throwing an error
• Allowing the parsing of non-utf8 feeds

It would be much appreciated!

[mmcdole]

I'm guessing I need to handle this by first:

1. Parsing non-UTF8 feeds into UTF8 first
2. Sanitize the feed afterwards

I could do something like:

func convertToUTF8(data []byte) (string, error) {
	reader, err := charset.NewReader(bytes.NewReader(data), "")
	if err != nil {
		return "", err
	}
	utf8Data, err := ioutil.ReadAll(reader)
	if err != nil {
		return "", err
	}
	return string(utf8Data), nil
}

func sanitizeXML(xmlData []byte) (string, error) {
	utf8Data, err := convertToUTF8(xmlData)
	if err != nil {
		utf8Data = string(xmlData) // Fallback to original data if conversion fails
	}

	var buffer bytes.Buffer

	for _, r := range utf8Data {
		if isLegalXMLChar(r) {
			buffer.WriteRune(r)
		} else {
			buffer.WriteString(fmt.Sprintf("&#x%X;", r))
		}
	}

	return buffer.String(), nil
}

I could call this at the beginning of the sanitize function, but I'm not sure what I'd do if charset.NewReader failed to detect the encoding.