CONNECTORS-1748: Add "User-Agent platform" option for crawling mobile websites

[mingchun-zhao]

Allow me to propose a new feature for crawling mobile sites which restrict access to content based on User-Agent header. Actually, Our customer's mobile website was failing to crawl because access was restricted based on whether the User-Agent request header includes the mobile info. For this reason, we added the "User-Agent platform" option to the new "Request Headers" tab on the web repository connector page so that this mobile website doesn't fail to crawl.

• The screenshot of "User-Agent platform" option within the new "Request Headers" tab

• Crawling a mobile site will be failed when using the desktop User-Agent

• Crawling a mobile site will be successful when using the mobile User-Agent

[OpenPj]

Thank you so much Mingchun for this contribution, we will work on merging this pull request after releasing ManifoldCF 2.27 due to the drastic change we are doing for the entire release process because we don't want to change connectors.

So we will apply this change for ManifoldCF 2.28.