Signing Key Expire within a day

[qdchong]

i encountered the error <Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; Invalid credential date. Date is not the same as X-Amz-Date.</Message>... after a day.

I thought the signing key was suppose to last for a week. can anyone help? How do i set the X-Amz-Date? Or do i need to set the amz-expire field??

[CpuID]

I've got the same issue, haven't resolved it yet.

[CpuID]

@qdchong to compare our implementations:

• we generate a new signing key once a day right now ( related )
• I've found that sometimes even after a SIGHUP with a new key in the configuration with a newly generated signing key, I still get AuthorizationHeaderMalformed errors which is really weird...

[CpuID]

@qdchong can you try implement something looking like https://tenzer.dk/nginx-with-dynamic-upstreams/ in your configs and see if it yields a difference for you? I did it about 2 days ago, so far rolled over UTC twice and haven't had an issue. Not sure if this is what did the job or not, but potentially...

[qdchong]

Hmm wow i shall try it. I was looking at what other options r there. Thanks so much

[CpuID]

I ended up getting the The authorization header is malformed; Invalid credential date. Date is not the same as X-Amz-Date. again unfortunately... :( even with our approach in place. Haven't looked deeper at why yet.

[AdrianVandierAst]

Hello :)

The README of this module is wrong. Look at https://docs.aws.amazon.com/general/latest/gr/sigv4-date-handling.html The signing key IS usable only for requests signed the same date, maybe your problem is when request are created just before midnight and send just after. I lost two days on that -_-

So, even if "It is an insecure practise to let the secret key reside on your nginx server.", we are all forced to have a cronned script at midnight exactly that regenerates the nginx configuration.

In fact, the module SHOULD generates the signing key since the promise of security is not possible.

I'm a bit disapointed by the date validation in the signature v4 protocol :(

[BrutalSimplicity]

@AdrianVandierAst

I've got a proposed solution to this that uses temporary credentials and a python script that will regenerate a new signature upon expiration. It's limited by how often it gets run by cron (minutely), but you might be able to have it run continuously and check for expiration ahead of time. I've also thought about using lua for something a bit more robust that doesn't require a hot reload.

See the PR #74

[hexmarkrecords]

I wonder if we convert the python code to C++ and include it in the module. It could update the config struct in a lazy fashion. The date found in the key scope should be checked against the current date and if different update the struct with new key scope and signing key. The single requests around midnight UTC will run a little slower but subsequent requests will have normal performance.

[hexmarkrecords]

I'm currently working on a solution - my C is pretty rusty - I should really write some unit tests for it. Will post back once ive tested it in my pre-prod environment.

[hexmarkrecords]

I have produced a forked version which calculates the signing key when the date changes. It subsequently caches the signing key and key scope in the configuration. It does break the best practices of keeping the secret key off the nginx server.

I have simplified the nginx configuration to that similar to AWS2 config found in earlier versions. It still uses AWS4 for the signing method so no change there.

Not sure how you would do it otherwise as at 23:59:59:999 a request could be in flight and reach Amazon the following day at 00:00:00:000. Amazon would then deem the request erroneous by AWS as there would be a discrepancy between the header date, the signing key and key scope.

I have added some unit tests to test the new functionality.

By all means take from it what you wish. I agree with the original author intent of keeping the keys off the box, but couldn't see a way round it without over complicating the hack.

Fork here >> https://github.com/hexmarkrecords/ngx_aws_auth

If you would like to provide a better solution, poke holes, please chime in.

[hexmarkrecords]

I have tested it the following day and my assets still load:

https://hexmarkrecords.com

All dynamic assets are now served from S3 without expiry .... Yay

[leffuy]

@hexmarkrecords Trying to use your fork, as I couldn't get the original here to work at all.

Says "unknown direct aws_service" when using your example nginx.conf. Tried taking this out and get returned a 403 with "The request signature we calculated does not match the signature you provided"

using following code

# For more information on configuration, see:
#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;

    server {
        listen       80 default_server;
        listen       [::]:80 default_server;
        server_name  _;
        #root         /usr/share/nginx/html;


        aws_access_key REDACTED; # Example AKIDEXAMPLE
        aws_secret_key REDACTED; #Example L4vRLWAO92X5L3Sqk5QydUSdB0nC9+1wfqLMOKLbRp4=
        aws_region us-west-2;
        # aws_service s3;
        aws_s3_bucket my.redacted.bucket;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location / {
            aws_sign;
            # aws_endpoint "s3.us-west-2.amazonaws.com";
            #proxy_pass_request_headers off;
            proxy_pass https://my.redacted.bucket.s3.us-west-2.amazonaws.com/;
         }

    }

# Settings for a TLS enabled server.
#
#    server {
#        listen       443 ssl http2 default_server;
#        listen       [::]:443 ssl http2 default_server;
#        server_name  _;
#        root         /usr/share/nginx/html;
#
#        ssl_certificate "/etc/pki/nginx/server.crt";
#        ssl_certificate_key "/etc/pki/nginx/private/server.key";
#        ssl_session_cache shared:SSL:1m;
#        ssl_session_timeout  10m;
#        ssl_ciphers PROFILE=SYSTEM;
#        ssl_prefer_server_ciphers on;
#
#        # Load configuration files for the default server block.
#        include /etc/nginx/default.d/*.conf;
#
#        location / {
#        }
#
#        error_page 404 /404.html;
#            location = /40x.html {
#        }
#
#        error_page 500 502 503 504 /50x.html;
#            location = /50x.html {
#        }
#    }

}

[hexmarkrecords]

@hexmarkrecords Trying to use your fork, as I couldn't get the original here to work at all.

Says "unknown direct aws_service" when using your example nginx.conf. Tried taking this out and get returned a 403 with "The request signature we calculated does not match the signature you provided"

using following code

# For more information on configuration, see:
#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;

    server {
        listen       80 default_server;
        listen       [::]:80 default_server;
        server_name  _;
        #root         /usr/share/nginx/html;


        aws_access_key REDACTED; # Example AKIDEXAMPLE
        aws_secret_key REDACTED; #Example L4vRLWAO92X5L3Sqk5QydUSdB0nC9+1wfqLMOKLbRp4=
        aws_region us-west-2;
        # aws_service s3;
        aws_s3_bucket my.redacted.bucket;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location / {
            aws_sign;
            # aws_endpoint "s3.us-west-2.amazonaws.com";
            #proxy_pass_request_headers off;
            proxy_pass https://my.redacted.bucket.s3.us-west-2.amazonaws.com/;
         }

    }

# Settings for a TLS enabled server.
#
#    server {
#        listen       443 ssl http2 default_server;
#        listen       [::]:443 ssl http2 default_server;
#        server_name  _;
#        root         /usr/share/nginx/html;
#
#        ssl_certificate "/etc/pki/nginx/server.crt";
#        ssl_certificate_key "/etc/pki/nginx/private/server.key";
#        ssl_session_cache shared:SSL:1m;
#        ssl_session_timeout  10m;
#        ssl_ciphers PROFILE=SYSTEM;
#        ssl_prefer_server_ciphers on;
#
#        # Load configuration files for the default server block.
#        include /etc/nginx/default.d/*.conf;
#
#        location / {
#        }
#
#        error_page 404 /404.html;
#            location = /40x.html {
#        }
#
#        error_page 500 502 503 504 /50x.html;
#            location = /50x.html {
#        }
#    }

}

I have something like this:

http {
    server {
        listen     8001;

        aws_access_key      %AWS_ACCESS_KEY%;
        aws_secret_key      %AWS_SECRET_ACCESS_KEY%;
        aws_region          %AWS_REGION%;
        aws_s3_bucket       %BUCKET_NAME%;
        resolver 8.8.8.8;

        location / {
            access_log off;
            error_log off;

            aws_sign;
            proxy_pass http://%BUCKET_NAME%.s3.amazonaws.com;
        }
    }

And that worked for me bud. Ive just copied it from my config that is in production. Hopefully that should help you.

[hexmarkrecords]

i'm gonna look in to the aws_service param as that should be work also. It defaults to 's3' anyway.

[hexmarkrecords]

Spotted an issue with the service parameter so i will fix that - but in the mean time it defaults to 's3'. Thanks for letting me know @leffuy

[hexmarkrecords]

pushed to master and created new tag 2.2.2 - fixed missing service optional parameter

[mwangaben]

Hello @hexmarkrecords i am trying to compile this with nginx-1.19.0 with --add-module=/home/admin/building/ngx_aws_auth-2.2.2, but when i run make this is what i get

/home/admin/building/ngx_aws_auth-2.2.2/crypto_helper_openssl.c: In function ‘ngx_aws_auth__sign_hmac’: /home/admin/building/ngx_aws_auth-2.2.2/crypto_helper_openssl.c:58:14: error: storage size of ‘hmac’ isn’t known HMAC_CTX hmac; ^~~~ /home/admin/building/ngx_aws_auth-2.2.2/crypto_helper_openssl.c:59:5: error: implicit declaration of function ‘HMAC_CTX_init’ [-Werror=implicit-function-declaration] HMAC_CTX_init(&hmac); ^~~~~~~~~~~~~ /home/admin/building/ngx_aws_auth-2.2.2/crypto_helper_openssl.c:60:5: error: ‘HMAC_Init’ is deprecated [-Werror=deprecated-declarations] HMAC_Init(&hmac, key, key_length, EVP_sha256()); ^~~~~~~~~ In file included from /usr/include/openssl/e_os2.h:13:0, from /usr/include/openssl/bio.h:13, from /home/admin/building/ngx_aws_auth-2.2.2/crypto_helper_openssl.c:10: /usr/include/openssl/hmac.h:28:1: note: declared here DEPRECATEDIN_1_1_0(__owur int HMAC_Init(HMAC_CTX *ctx, const void *key, int len, ^ /home/admin/building/ngx_aws_auth-2.2.2/crypto_helper_openssl.c:63:5: error: implicit declaration of function ‘HMAC_CTX_cleanup’ [-Werror=implicit-function-declaration] HMAC_CTX_cleanup(&hmac); ^~~~~~~~~~~~~~~~ /home/admin/building/ngx_aws_auth-2.2.2/crypto_helper_openssl.c:58:14: error: unused variable ‘hmac’ [-Werror=unused-variable] HMAC_CTX hmac; ^~~~ cc1: all warnings being treated as errors objs/Makefile:1317: recipe for target 'objs/addon/ngx_aws_auth-2.2.2/crypto_helper_openssl.o' failed make[1]: *** [objs/addon/ngx_aws_auth-2.2.2/crypto_helper_openssl.o] Error 1

Kindly help

[hexmarkrecords]

Hello @hexmarkrecords i am trying to compile this with nginx-1.19.0 with --add-module=/home/admin/building/ngx_aws_auth-2.2.2, but when i run make this is what i get

/home/admin/building/ngx_aws_auth-2.2.2/crypto_helper_openssl.c: In function ‘ngx_aws_auth__sign_hmac’: /home/admin/building/ngx_aws_auth-2.2.2/crypto_helper_openssl.c:58:14: error: storage size of ‘hmac’ isn’t known HMAC_CTX hmac; ^~~~ /home/admin/building/ngx_aws_auth-2.2.2/crypto_helper_openssl.c:59:5: error: implicit declaration of function ‘HMAC_CTX_init’ [-Werror=implicit-function-declaration] HMAC_CTX_init(&hmac); ^~~~~~~~~~~~~ /home/admin/building/ngx_aws_auth-2.2.2/crypto_helper_openssl.c:60:5: error: ‘HMAC_Init’ is deprecated [-Werror=deprecated-declarations] HMAC_Init(&hmac, key, key_length, EVP_sha256()); ^~~~~~~~~ In file included from /usr/include/openssl/e_os2.h:13:0, from /usr/include/openssl/bio.h:13, from /home/admin/building/ngx_aws_auth-2.2.2/crypto_helper_openssl.c:10: /usr/include/openssl/hmac.h:28:1: note: declared here DEPRECATEDIN_1_1_0(__owur int HMAC_Init(HMAC_CTX *ctx, const void *key, int len, ^ /home/admin/building/ngx_aws_auth-2.2.2/crypto_helper_openssl.c:63:5: error: implicit declaration of function ‘HMAC_CTX_cleanup’ [-Werror=implicit-function-declaration] HMAC_CTX_cleanup(&hmac); ^~~~~~~~~~~~~~~~ /home/admin/building/ngx_aws_auth-2.2.2/crypto_helper_openssl.c:58:14: error: unused variable ‘hmac’ [-Werror=unused-variable] HMAC_CTX hmac; ^~~~ cc1: all warnings being treated as errors objs/Makefile:1317: recipe for target 'objs/addon/ngx_aws_auth-2.2.2/crypto_helper_openssl.o' failed make[1]: *** [objs/addon/ngx_aws_auth-2.2.2/crypto_helper_openssl.o] Error 1

Kindly help

I can take a look over the weekend.

[hexmarkrecords]

Hi @mwangaben I have now updated the branch (2.3.0 and master) to build a dynamic module exporting it to /private/tmp on the host machine when using docker.

It now works with Open SSL 1.1.1 and therefore compatible with Nginx 1.19.1.

Hope it helps you out.

[hexmarkrecords]

I wonder if I should try to turn this into a proper pull request for the main repos instead of on a fork?