Bots serving pages with base href with http

[nethanjan]

Hi,

I'm running an app with node express back end and react front end. For bot I'm using Rendertron to serve the pages.

node : v12.10.0 rendertron : 3.1.0 express : 4.17.1

First I have tried with using express trust proxy disabled. For each bot types page was rendered http base href. Then I have tried enabling trust proxy "app.set('trust proxy', true)". For google bots, it's serving proper pages with https base href, but for other bots like bingbot, duckduckgo it's giving a 400 error. In rendertron console log it says "response does not exist" response

HTTP/2 400 date: Thu, 11 Feb 2021 08:37:04 GMT content-type: text/plain; charset=utf-8 content-length: 0 server: nginx/1.12.2 x-powered-by: Express access-control-allow-origin: * access-control-allow-headers: Origin, X-Requested-With, Content-Type, Accept vary: Accept-Encoding x-renderer: rendertron x-frame-options: sameorigin strict-transport-security: max-age=31536000; includeSubDomains

Any suggestions to fix this?

[AVGP]

I'm not sure I understand the issue, so I'll try to walk through the issue the way I understand it. Please correct me where I'm wrong.

• You have an express-powered web server that uses rendertron middleware.
• Requests from Googlebot work fine, but other bots get a 400 response.

Now my first question is: Did you configure the rendertron-middleware to render the pages for Googlebot, because that's not the default configuration!

When you say "http base href", you mean the <base> tag's href attribute value? This is using the protocol that is passed to it from the render request, so possibly the request arrives as non-HTTPS at the express server already? The error you posted shows that there's an nginx in between the public internet and your express.js server, so maybe that's why the protocol is switched to http.

The other thing is that this shouldn't lead to an HTTP 400 response, but I'm not sure where that's coming from in your stack. Can you render the URL just fine when you make a request directly to your express server with a bot UA that is configured to use Rendertron?