Running React app with Nginx

[svyatoslavd-orca]

Hi,

I'm trying to run React app in S3 using Nginx. The default behaviour for non-existent pages should be routing to /index.html. Here is my default.conf file for Nginx. Unfortunately it doesn't work as expected and I get 404 error on any non-existent page. I also tried to use proxy_pass in location @error404 block but I get auth error from AWS. I would appreciate any help on this issue. Thanks in advance.

# We include only the variables needed for the authentication signatures that
# we plan to use.
include /etc/nginx/conf.d/gateway/v4_js_vars.conf;

error_log /dev/stderr debug;

# Extracts only the path from the requested URI. This strips out all query
# parameters and anchors in order to prevent extranous data from being sent to
# S3.
map $request_uri $uri_path {
    "~^(?P<path>.*?)(\?.*)*$"  $path;
}

map virtual $s3_host_hdr {
    virtual "${S3_BUCKET_NAME}.${S3_SERVER}";
    path    "${S3_SERVER}:${S3_SERVER_PORT}";
    default "${S3_BUCKET_NAME}.${S3_SERVER}";
}

js_var $indexIsEmpty true;
# This creates the HTTP authentication header to be sent to S3
js_set $s3auth s3gateway.s3auth;
js_set $s3SecurityToken s3gateway.s3SecurityToken;
js_set $s3uri s3gateway.s3uri;

server {
    include /etc/nginx/conf.d/gateway/server_variables.conf;
    error_log /dev/stderr debug;

    # Don't display the NGINX version number because we don't want to reveal
    # information that could be used to find an exploit.
    server_tokens off;

    # Uncomment this for a HTTP header that will let you know the cache status
    # of an object.
    # add_header X-Cache-Status $upstream_cache_status;

    # Proxy caching configuration. Customize this for your needs.
    proxy_cache s3_cache;
    proxy_cache_valid 200 302 0;
    proxy_cache_valid 404 0;
    proxy_cache_valid 403 0;
    proxy_cache_methods GET HEAD;
    # When this is enabled a HEAD request to NGINX will result in a GET
    # request upstream. Unfortunately, proxy_cache_convert_head has to be
    # disabled because there is no way for the signatures generation code to
    # get access to the metadata in the GET request that is sent upstream.
    proxy_cache_convert_head off;
    proxy_cache_revalidate on;
    proxy_cache_background_update on;
    proxy_cache_lock on;
    proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504;
    proxy_cache_key "$request_method$host$uri";

    # If you need to support proxying range request, refer to this article:
    # https://www.nginx.com/blog/smart-efficient-byte-range-caching-nginx/

    # Do not proxy the S3 SOAP API. The S3 API has a less-documented feature
    # where the object name "soap" is used for the SOAP API. We don't allow
    # access to it.
    location /soap {
        return 404;
    }

    location /health {
        return 200;
    }

    location / {
        auth_request /aws/credentials/retrieve;
        error_log /var/log/nginx/debug.log debug;

        # Redirect to the proper location based on the client request - either
        # @s3, @s3Listing or @error405.

        js_content s3gateway.redirectToS3;
    }

    location /aws/credentials/retrieve {
        internal;
        js_content s3gateway.fetchCredentials;
    }

    location @s3 {
        # We include only the headers needed for the authentication signatures that
        # we plan to use.
        include /etc/nginx/conf.d/gateway/v4_headers.conf;

        # Don't allow any headers from the client - we don't want them messing
        # with S3 at all.
        proxy_pass_request_headers off;

        # Set the Authorization header to the AWS Signatures credentials
        proxy_set_header Authorization $s3auth;
        proxy_set_header X-Amz-Security-Token $s3SecurityToken;

        # We set the host as the bucket name to inform the S3 API of the bucket
        proxy_set_header Host $s3_host_hdr;

        # Use keep alive connections in order to improve performance
        proxy_http_version 1.1;
        proxy_set_header Connection '';

        # We strip off all of the AWS specific headers from the server so that
        # there is nothing identifying the object as having originated in an
        # object store.
        js_header_filter s3gateway.editAmzHeaders;

        # Catch all errors from S3 and sanitize them so that the user can't
        # gain intelligence about the S3 bucket being proxied.
        proxy_intercept_errors on;

        # Comment out this line to receive the error messages returned by S3
        error_page 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 420 422 423 424 426 428 429 431 444 449 450 451 500 501 502 503 504 505 506 507 508 509 510 511 =404 @error404;

        proxy_pass https://storage_urls$s3uri;
    }

    location @s3Listing {
        # We include only the headers needed for the authentication signatures that
        # we plan to use.
        include /etc/nginx/conf.d/gateway/v4_headers.conf;

        # Don't allow any headers from the client - we don't want them messing
        # with S3 at all.
        proxy_pass_request_headers off;

        # Set the Authorization header to the AWS Signatures credentials
        proxy_set_header Authorization $s3auth;
        proxy_set_header X-Amz-Security-Token $s3SecurityToken;

        # We set the host as the bucket name to inform the S3 API of the bucket
        proxy_set_header Host $s3_host_hdr;

        # Use keep alive connections in order to improve performance
        proxy_http_version 1.1;
        proxy_set_header Connection '';

        # We strip off all of the AWS specific headers from the server so that
        # there is nothing identifying the object as having originated in an
        # object store.
        js_header_filter s3gateway.editAmzHeaders;

        # Apply XSL transformation to the XML returned from S3 directory listing
        # results such that we can output an HTML directory contents list.
        xslt_stylesheet /etc/nginx/include/listing.xsl;
        xslt_types application/xml;

        # We apply an output filter to the XML input received from S3 before it
        # is passed to XSLT in order to determine if the resource is not a valid
        # S3 directory. If it isn't a valid directory, we do a dirty hack to
        # corrupt the contents of the XML causing the XSLT to fail and thus
        # nginx to return a 404 to the client. If you don't care about empty
        # directory listings for invalid directories, remove this.
        js_body_filter s3gateway.filterListResponse;

        # Catch all errors from S3 and sanitize them so that the user can't
        # gain intelligence about the S3 bucket being proxied.
        proxy_intercept_errors on;

        # Comment out this line to receive the error messages returned by S3
        error_page 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 420 422 423 424 426 428 429 431 444 449 450 451 500 501 502 503 504 505 506 507 508 509 510 511 =404 @error404;

        proxy_pass https://storage_urls$s3Uri;
    }

    location @error404 {
        # return 404;
        try_files $uri /index.html;
    }

    # Provide a hint to the client on 405 errors of the acceptable request methods
    error_page 405 @error405;
    location @error405 {
        add_header Allow "GET, HEAD" always;
        return 405;
    }
}

[dekobon]

Thank you for filing an issue. Did you set PROVIDE_INDEX_PAGE=true when starting the gateway?

[svyatoslavd-orca]

Yes, unfortunately PROVIDE_INDEX_PAGE=true didn't help.

[dekobon]

Are you using a Docker image to run? If so, what is the version/hash and where was it downloaded from (Dockerhub or github)?

[svyatoslavd-orca]

1. Docker image is ghcr.io/nginxinc/nginx-s3-gateway/nginx-oss-s3-gateway:latest-20220623
2. I think that the problem is that try_files is working only for local files and I need to use proxy_pass for location @error404 config. I tried to adjust the given above config and changed location @error404 to be like this:

location @error404 {
	include /etc/nginx/conf.d/gateway/v4_headers.conf;
        proxy_pass_request_headers off;
        proxy_set_header Authorization $s3auth;
        proxy_set_header X-Amz-Security-Token $s3SecurityToken;
        proxy_set_header Host $s3_host_hdr;
        proxy_http_version 1.1;
        proxy_set_header Connection '';
        js_header_filter s3gateway.editAmzHeaders;

	set $index_html /index.html;
	proxy_pass https://storage_urls$index_html;
}

But now I get the following error from AWS when I try to access non-existent page: The request signature we calculated does not match the signature you provided. Check your key and signing method.

[dekobon]

Can you try again with the latest version: ghcr.io/nginxinc/nginx-s3-gateway/nginx-oss-s3-gateway:latest-20220815? Be sure to enable PROVIDE_INDEX_PAGE=true.

[svyatoslavd-orca]

Tried the latest version. Didn't work. Same issue.

[svyatoslavd-orca]

What can I do with this error when using the given above location @error404 block? The request signature we calculated does not match the signature you provided. Check your key and signing method.

[dekobon]

That error means that the http signature generated does not match the one generated by S3, and thus the authentication against S3 fails.

Can you provide more details about your environment? How are you invoking the gateway (what docker command are you using)? What environment variables are being used (please do not share your AWS credentials)? What is the S3 backend (is is AWS S3 or a compatible store)?

Also, please verify that you can access S3 correctly using a s3 command line utility like s3cmd or the aws cli using the same credentials that you are using with the gateway.

[svyatoslavd-orca]

Sure. I'm using the getaway as a Kubernetes Pod. It forwards requests to AWS S3 backend. I used the latest image possible. Here are my env vars taken from the Pod (I concealed some values):

ALLOW_DIRECTORY_LIST=false
NGINX_S3_PORT_80_TCP_PROTO=tcp
S3_STYLE=virtual
S3_SERVER_PROTO=https
PROXY_CACHE_VALID_OK=0
PROVIDE_INDEX_PAGE=true
NGINX_S3_INDEX_SERVICE_HOST=xxx
NGINX_S3_SERVICE_PORT_HTTP=80
PROXY_CACHE_VALID_NOTFOUND=0
S3_REGION=xxx
NGINX_S3_INDEX_PORT_80_TCP_PROTO=tcp
NGINX_S3_PORT=tcp://xxx:80
S3_SERVER=s3.xxx.amazonaws.com
S3_DEBUG=false
NGINX_S3_PORT_80_TCP_ADDR=xxx
AWS_SIGS_VERSION=4
NJS_VERSION=0.7.5
NGINX_S3_INDEX_SERVICE_PORT=80
NGINX_S3_SERVICE_HOST=xxx
NGINX_S3_PORT_80_TCP=tcp://xxx
NGINX_S3_INDEX_PORT=tcp://xxx:80
NGINX_S3_PORT_80_TCP_PORT=80
NGINX_S3_INDEX_PORT_80_TCP=tcp://xxx:80
NGINX_S3_INDEX_PORT_80_TCP_ADDR=xxx
NGINX_VERSION=1.23.0
S3_BUCKET_NAME=xxx
S3_SERVER_PORT=443
NGINX_S3_INDEX_PORT_80_TCP_PORT=80
NGINX_S3_SERVICE_PORT=80
PROXY_CACHE_VALID_FORBIDDEN=0

Default configuration worked as expected until I wanted to get additional logic for 404 pages. The logic should be the following:

1. If page exists in AWS S3, give response with this page.
2. If page doesn't exist, return /index.html which is a React single page app.

I entered the Pod and tried to get under the hood of default.conf file. I changed only one line in location @error404: Original value (as provided with the image):

location @error404 {
    return 404;
}

My version:

location @error404 {
    try_files $uri /index.html;
}

I used curl command inside of the Pod for tests.

1. Trying to curl an existing page: curl localhost/index.html -L -v => 200 OK
2. Trying to curl non-existent page: curl localhost/non-existent -L -v => 404 NotFound

Seems like try_files doesn't work for proxying the requests to S3. Then I decided to try proxy_pass directive instead of try_files in location @error404 (I used exactly the same headers and configuration as given in location @s3 except the last 2 lines):

location @error404 {
    include /etc/nginx/conf.d/gateway/v4_headers.conf;
    proxy_pass_request_headers off;
    proxy_set_header Authorization $s3auth;
    proxy_set_header X-Amz-Security-Token $s3SecurityToken;
    proxy_set_header Host $s3_host_hdr;
    proxy_http_version 1.1;
    proxy_set_header Connection '';
    js_header_filter s3gateway.editAmzHeaders;

    rewrite ^/(.*)$ /index.html break;
    proxy_pass https://storage_urls;
}

1. Trying to curl an existing page: curl localhost/index.html -L -v => 200 OK (still working good)
2. Trying to curl non-existent page: curl localhost/non-existent -L -v => SignatureDoesNotMatch (AWS auth error now)

Could it be connected with the fact that originally I request /non-existent and being redirected to /index.html then and that's why AWS responds with SignatureDoesNotMatch? If so, how to proxy requests to S3 in a proper way?

[svyatoslavd-orca]

I guess the problem is in headers.

[svyatoslavd-orca]

I've managed to fix it. The location @error404 directive should look like this for React single page app:

location @error404 {
    set $index_html /index.html;
    proxy_pass https://${S3_BUCKET_NAME}.${S3_SERVER}$index_html;
}

[svyatoslavd-orca]

Here is another issue when enabling encryption for the bucket. I get Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4. error when try to get non-existing file, i.e. when falling back to location @error404 block.

[svyatoslavd-orca]

@dekobon I would appreciate any suggestions on this issue.

[dekobon]

I have not tried to use the gateway with KMS managed keys. I see that you set AWS_SIGS_VERSION=4, so it shouldn't be using v2 keys. I'm guessing that message is misleading. Can you provide more details on how the bucket was configured?

[svyatoslavd-orca]

I ended up using Amazon S3-managed keys (SSE-S3) but not AWS Key Management Service key (SSE-KMS).

[jr2173]

@svyatoslavd-orca - is this still working for you? Trying to use nginx-s3-gateway with a react frontend application and getting the standard 404 from nginx despite the changes below:

# default.conf.template changes

location @error404 {
    set $index_html /index.html;
    proxy_pass https://${S3_BUCKET_NAME}.${S3_SERVER}$index_html;
}

[svyatoslavd-orca]

Hi @jr2173. Yes, the given above configuration works for me. But I'm using a slightly older version of this app. Did you reload Nginx after making these changes?

[jr2173]

Hi @svyatoslavd-orca - yes rebuilt container and ensured nginx was reloaded. I am actually wondering how you are able to get the proxy_pass to /index.html working without any auth headers going to s3 - does your bucket allow public read access? When I try and disable the cache I see an AccessDenied XML response proxied from s3.

I tried re-adding extra headers to the @error404 block but encountered the same SignatureDoesNotMatch error that you did. After some more digging, I suspect the issue is that s3gateway.signatureV4 / _buildSignatureV4 are building the signature header based on the original path from r.variables.uri_path, and have no idea to use /index.html.

https://github.com/nginxinc/nginx-s3-gateway/blob/31f60b08563068f740181856f2ff285bee2fb326/common/etc/nginx/include/s3gateway.js#L566-L577

If my theory is right, then I believe there will need to be some kind of flag set perhaps within the @error404 block that allows building a signature using the path /index.html to be sent to s3.

[svyatoslavd-orca]

Hi @jr2173. I checked current config and mine and found out that this line differs in previous versions of the app: https://github.com/nginxinc/nginx-s3-gateway/blob/30529460188d4ffbd2ab8d298ce04407258c6ffb/common/etc/nginx/templates/default.conf.template#L114 Actually you don't get into @error404 block when you get 404 error. Try to add 404 to the list of error codes in this line. Now it is ... 402 403 405 406 ... Try to do ... 402 403 404 405 406 ... Also I'm using IAM role to get access to S3 bucket. This is more convenient and safer than feeding credentials to the app.

[jr2173]

@svyatoslavd-orca - tried adding 404 to the list and still either get a generic 404 from nginx if PROXY_CACHE_VALID_NOTFOUND=1m or the AccessDenied XML response from S3 if PROXY_CACHE_VALID_NOTFOUND=0 ☹️

Also using IAM role running in an ECS container, but for local testing / troubleshooting I am just passing in S3_ACCESS_KEY_ID / S3_SECRET_KEY which produces the same error.

Continuing to play with the NJS module to see if any workaround can be found there... Just so odd that you were able to get it working. It'd be so nice if the maintainers considered adding this feature out of the box.

[dekobon]

Hi @jr2173, I'm the maintainer of the project. Once we can narrow down the root cause for the problem, I'm happy to work to integrate a fix/feature. As I understand it, we do not yet understand exactly why it isn't working.

Just to clear away some ambiguity, it would be good to see what environment settings and git commit hash everyone is using (with the sensitive bits redacted).

[svyatoslavd-orca]

Hi @dekobon @jr2173. I'm using latest-20220623 version of the app. The only thing that I changed was:

location @error404 {
        # return 404; <-- This was previously
        set $index_html /index.html; # <-- Added for React app
        proxy_pass https://${S3_BUCKET_NAME}.${S3_SERVER}$index_html; # <-- Added for React app
}

[jr2173]

Hi @dekobon, thank you for your comment and interest on this. I'm using a fresh clone of master at 31f60b08563068f740181856f2ff285bee2fb326. The settings I've been using to test this out are below:

ALLOW_DIRECTORY_LIST=false
AWS_SIGS_VERSION=4
S3_ACCESS_KEY_ID=**REDACTED**
S3_SECRET_KEY=**REDACTED**
S3_BUCKET_NAME=**REDACTED**
S3_REGION=us-east-1
S3_SERVER_PORT=443
S3_SERVER_PROTO=https
S3_SERVER=s3.amazonaws.com
S3_STYLE=default
S3_DEBUG=true
PROVIDE_INDEX_PAGE=true

The S3 bucket allows access to the IAM role associated with the access key / secret and a direct request of /index.html or / returns the content in the s3-bucket/index.html as expected.

Where our issue lies (and I will try to describe this the best I can), we want any request for which S3 returns a 404 to show s3-bucket/index.html rather than the default 404 page shown by nginx.

@svyatoslavd-orca was able to get this functionality by modifying the @error404 location target in default.conf:

location @error404 {
    # return 404;
    set $index_html /index.html;
    proxy_pass https://${S3_BUCKET_NAME}.${S3_SERVER}$index_html;
}

Unfortunately, I have not been able to get this solution working in my case. From what I can tell by looking at the S3_DEBUG logs and printing out the request state as it gets passed within the NJS module, the sigv4 signature is based on the original request (i.e. /non-existing-item/) which does not match the signature for a subsequent proxy_pass to /index.html.

The only solution so far I have been able to come up with is by changing the @error404 location target to something like:

location @error404 {
    rewrite ^ /index.html redirect;
}

However, this does a hard redirect to /index.html and loses the original path which will not work for our frontend react app routing feature. Hopefully this outlines the issue we're trying to solve in a way that you can understand. In the meantime, I will continue tinkering with the NJS module / conf to see if I can get this functionality working somehow.

[jr2173]

I was able to hack together something that (kind of) works, although it requires PROXY_CACHE_VALID_NOTFOUND=0 and randomly still throws SignatureDoesNotMatch errors for some reason.

Happy to keep testing this as it's critical to a project I'm working on. Hopefully the code changes provide some clarity on what we're trying to accomplish - and perhaps eventually pave the way for similar native feature in the project.

https://github.com/nginxinc/nginx-s3-gateway/compare/master...jr2173:nginx-s3-gateway:issue50

[dekobon]

After reflecting on the issue, what would help me the most would be putting together an integration test that shows the failure. Would that be possible?

[jr2173]

Hi @dekobon - I put together a pseudo test script of the desired functionality. Let me know if you were thinking of something else and I will try to adapt.

TRY_FILE_NOTFOUND feature test

This feature will retry a request to a file that is not found to a default file defined by the TRY_FILE_NOTFOUND setting. The behavior is similar to the try_files directive in nginx but considers the proxy_pass and signature headers that are included with a request when using the nginx-s3-gateway container image.

Set environment variables

$ export S3_BUCKET=testbucket

Create settings file including TRY_FILE_NOTFOUND

$ echo "ALLOW_DIRECTORY_LIST=false
AWS_SIGS_VERSION=4
S3_ACCESS_KEY_ID=***********************
S3_SECRET_KEY=***********************
S3_BUCKET_NAME=$S3_BUCKET
S3_REGION=***********************
S3_SERVER_PORT=443
S3_SERVER_PROTO=https
S3_SERVER=s3.amazonaws.com
S3_STYLE=default
S3_DEBUG=true
PROVIDE_INDEX_PAGE=true
TRY_FILE_NOTFOUND=/index.html" >> test-settings

Upload file to be used by TRY_FILE_NOTFOUND into an empty s3 bucket

$ echo "<h1>TRY_FILE_NOTFOUND test</h1>" > "index.html"
$ aws s3 cp "index.html" s3://$S3_BUCKET/index.html
upload: ./index.html to s3://testbucket/index.html

Pull the container image

$ docker pull ghcr.io/nginxinc/nginx-s3-gateway/nginx-oss-s3-gateway:latest

Run the container in another terminal

$ docker run --env-file ./test-settings --rm -it --publish 80:80 --name nginx-s3-gateway ghcr.io/nginxinc/nginx-s3-gateway/nginx-oss-s3-gateway:latest

Ensure the file is there

$ curl http://localhost/index.html
<h1>TRY_FILE_NOTFOUND test</h1>

Now try to download a file that does not exist

Since the TRY_FILE_NOTFOUND setting is used, the response should mimic a request to the TRY_FILE_NOTFOUND file itself

❌ Test Failure (Current State):

$ curl http://localhost/does-not-exist
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>

✅ Test Pass (Future State):

$ curl http://localhost/does-not-exist
<h1>TRY_FILE_NOTFOUND test</h1>

[dekobon]

Thank you for the clear documentation of the issue. I was thinking about a test being added to test_api.sh.

[jr2173]

Hi @dekobon - I took a first stab at the test. Let me know what you think:

https://github.com/jr2173/nginx-s3-gateway/commit/3c9c5acd1fe7a670e4fd2170be9d8d89ab73fb13

[dekobon]

Hi @jr2173 I tried to run the tests, but I didn't get any test failures. Is the test set up to fail when the error case happens?

[jr2173]

I fixed a small typo with an extra fi, which still should have caused a failure. But yes, the test was added:

https://github.com/jr2173/nginx-s3-gateway/blob/66da425cef53afb84242cd7d9591387f9b3d1ef4/test.sh#L320-L321

And should produce a failure:

$ ./test.sh
...
 ▲ Testing object: GET /does-not-exist/
Response code didn't match expectation. Request [GET http://localhost:8989/does-not-exist/] Expected [200] Actual [404]
curl command: /usr/bin/curl -s -o /dev/null -w '%{http_code}' 'http://localhost:8989/does-not-exist/' 
Error running tests - outputting container logs
nginx-s3-gateway_1  | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration

If you're pulling from my forked repo, make sure to checkout the 50-integration-test branch.