osg
osg copied to clipboard
Universal sitemap generator
After downloading the binary and running I'm getting the follow error. Any idea why? ``` admin@dell-lat ~/Downloads/osg $ ./osg sitemap.xml patrickmn.com panic: crypto: requested hash function is unavailable goroutine 3...
According to RFC 7232 §4.1, when a page is returned with a 304 Not Modified header, it is recommended that headers such as Content-Type should not be sent the the...
Go's reference date uses -07:00 for the timezone, so we need to use that in the time format. If we don't, the timezone in the date provided by the server...
Google webmaster gives error - "invalid tag:urlset" in generated sitemap with below command - ``` ./sitemap -c 10 -gzip-level 1 -max-crawl 0 --no-lastmod sitemap.xml https://example.com ```
While generating a sitemap. It also adds a URL like `javascript:doSomething('http://exmple.com')` which is invalid URL.
Hello. I've just tried to crawl [http://reactivex.io/documentation](http://reactivex.io/documentation) because I found out some documentation pages appear to be hidden. Unfortunately, `osg` crashed, outputting the following stacktrace with an error signal of...
hey, i crawl a site, and in the resulting xml file, all the lastmod times are like this: 0001-01-01T00:00:00+07:00 any idea, why this could happen? my idea was that the...