postlight
feat: Improve parser to accept postman-request options
Improvement Rationale
I am using mercury-parser to extract content of RSS feed articles, but my scraping script failed to extract some articles because SSL certificate of article's website was expired or invalid.
After reviewing the source code of mercury-parser, I found that it was using postman-request, which supports strictSSL flag in fetch options.
By setting strictSSL to false, my scraping script was able to successfully scrape the content.
So, I updated mercury-parser to accept fetch options of mercury-parser to make consumers able to fully utilize postman-request under the hood.
Hi @david0leong, thanks for your support on the project! Would it be possible for you to add a few tests to pertaining to the updated functionality?
Hello! Sorry for such a late reply on this but I think we're going to deny this pull request. I recognize the problem you're facing here, but exposing the internal request mechanism like this ties the library to a dependency in a way that removes some flexibility for any future changes to its internals.
My recommendation is to make this change in a fork for your scraper (which is honestly probably what you've done, since this issue was opened many years ago). If this problem comes up again, perhaps we can expose a first-party option to allow expired certificates that doesn't rely on passing options directly to the underlying request library.
Thanks for your contribution nonetheless! If you're not satisfied by this response, please feel free to open a new issue related to it and we can discuss a different option.
