Remove duplicates

[dorsiflexion]

The resulting list should be deduplicated.

Example:

"mstdn.lalafell.org"
"mstdn.lalafell.org."

To do this, dots should be stripped first – trailing ones definitely, leading ones wouldn't hurt either. Second, it should actually be deduplicated because above domains would be identical.

[Minoru]

Normalizing domain names (https://github.com/Minoru/minoru-fediverse-crawler/issues/159) should resolve this, but I'd need to double-check. Thanks for opening the issue!