Dashboard denies access when Ant Media is behind an Apache reverse proxy

[arnowelzel]

Short description

I installed Ant Media Community Edition on Ubuntu 20.04 which was upgraded to 22.04. The dashbord and live streaming work fine using port 5080.

However on the same machine there is also Apache hosting a number websites and I need the dashboard an HLS streams via HTTPS on port 443. So I've also setup a reverse proxy for that as following for a virtual host which is used to access Ant Media:

RewriteEngine on
RewriteCond %{HTTP:Upgrade} =websocket [NC]
RewriteRule /(.*) ws://localhost:5080/$1 [P,L]
RewriteCond %{HTTP:Upgrade} !=websocket [NC]
RewriteRule /(.*) http://localhost:5080/$1 [P,L]
ProxyPass / http://localhost:5080/
ProxyPassReverse / http://localhost:5080/

This worked fine in the past, but after one of the last updates I could not log in to the dashboard any longer as it refuses the access with HTTP 403. However HLS streams still worked fine, so I did not notice that right away and I can't say with which version exactly that started.

A workaround wich seems to do the trick is to add ProxyAddHeaders off. Then the dashboard does no longer respond with HTP 403. But since I did not find any documentation about this issue I am not sure if this is the intended behaviour or a bug in the context that live streams still worked even without that workaround. Also this lowers security as Ant Media now sees all incoming request as they are coming from 127.0.0.1 and not the real address of the client.

Environment

• Operating system and version: Ubuntu 22.04
• Java version: openjdk 11.0.16 2022-07-19
• Ant Media Server version: Ant Media Community Edition 2.4.3 20220418_2215
• Browser name and version: Vivaldi 5.4.2753.37 (Stable channel) (64-Bit)

Steps to reproduce

1. Install Ant Media server on Ubuntu 22.04
2. Configure a virtual host in Apache 2.4 which has TLS enabled with the following configuration for the reverse proxy:

RewriteEngine on
RewriteCond %{HTTP:Upgrade} =websocket [NC]
RewriteRule /(.*) ws://localhost:5080/$1 [P,L]
RewriteCond %{HTTP:Upgrade} !=websocket [NC]
RewriteRule /(.*) http://localhost:5080/$1 [P,L]
ProxyPass / http://localhost:5080/
ProxyPassReverse / http://localhost:5080/

3. Try to access the dashboard by calling https://hostname.example

Expected behavior

The login form of the dashboard shows up.

Actual behavior

There is an error message by Tomcat telling that access is denied.

Logs

IP addresses in the logs are anonymized for privacy reasons.

Tying to login to the dashboard - fails:

2003:ec:d72a:xxxx:xxxx:xxxx:xxxx:xxxx - - [20/Aug/2022:13:44:28 +0200] "GET /rest/v2/first-login-status HTTP/1.1" 403 654
2003:ec:d72a:xxxx:xxxx:xxxx:xxxx:xxxx - - [20/Aug/2022:13:44:30 +0200] "GET / HTTP/1.1" 403 654
2003:ec:d72a:xxxx:xxxx:xxxx:xxxx:xxxx - - [20/Aug/2022:13:44:30 +0200] "GET /favicon.ico HTTP/1.1" 403 654

Live stream embedded on the website - works:

2003:ec:d72a:xxxx:xxxx:xxxx:xxxx:xxxx - - [20/Aug/2022:13:47:30 +0200] "HEAD /LiveApp/streams/520064157572272301918079_adaptive.m3u8?token=undefined&subscriberId=undefined&subscriberCode=undefined HTTP/1.1" 404 -
2003:ec:d72a:xxxx:xxxx:xxxx:xxxx:xxxx - - [20/Aug/2022:13:47:30 +0200] "HEAD /LiveApp/streams/520064157572272301918079.m3u8?token=undefined&subscriberId=undefined&subscriberCode=undefined HTTP/1.1" 200 -
2003:ec:d72a:xxxx:xxxx:xxxx:xxxx:xxxx - - [20/Aug/2022:13:47:30 +0200] "GET /LiveApp/previews/520064157572272301918079.png HTTP/1.1" 404 804
2003:ec:d72a:xxxx:xxxx:xxxx:xxxx:xxxx - - [20/Aug/2022:13:47:30 +0200] "GET /LiveApp/streams/520064157572272301918079.m3u8?token=undefined&subscriberId=undefined&subscriberCode=undefined HTTP/1.1" 200 326
2003:ec:d72a:xxxx:xxxx:xxxx:xxxx:xxxx - - [20/Aug/2022:13:47:30 +0200] "GET /LiveApp/streams/5200641575722723019180791005.ts?token=undefined&subscriberId=undefined&subscriberCode=undefined HTTP/1.1" 200 775876
2003:ec:d72a:xxxx:xxxx:xxxx:xxxx:xxxx - - [20/Aug/2022:13:47:31 +0200] "GET /LiveApp/streams/5200641575722723019180791006.ts?token=undefined&subscriberId=undefined&subscriberCode=undefined HTTP/1.1" 200 774748
2003:ec:d72a:xxxx:xxxx:xxxx:xxxx:xxxx - - [20/Aug/2022:13:47:31 +0200] "GET /LiveApp/streams/5200641575722723019180791007.ts?token=undefined&subscriberId=undefined&subscriberCode=undefined HTTP/1.1" 200 771740
2003:ec:d72a:xxxx:xxxx:xxxx:xxxx:xxxx - - [20/Aug/2022:13:47:31 +0200] "GET /LiveApp/streams/5200641575722723019180791008.ts?token=undefined&subscriberId=undefined&subscriberCode=undefined HTTP/1.1" 200 771176

Tying to login to the dashboard after applying the workaround ProxyAddHeaders off- works, but the log now records 127.0.0.1 as IP address of the client and no longer the real IP address:

127.0.0.1 - - [20/Aug/2022:13:51:30 +0200] "GET / HTTP/1.1" 200 1809
127.0.0.1 - - [20/Aug/2022:13:51:30 +0200] "GET /rest/v2/first-login-status HTTP/1.1" 200 58

[mekya]

Hi @arnowelzel ,

Thank you for very clear and detailed bug report. We'll increase its priority to resolve it ASAP. I think it's just a configuration parameter issue.

I've seen your blog post as well -> https://arnowelzel.de/en/ant-media-server-with-apache-reverse-proxy

You're doing great.

Thank you

[muratugureminoglu]

Hi @arnowelzel

Thank you for your detailed reproduce scenario. In the same way, I created it on my side, but I did not encounter any problems. Finally, could you share the output of the following command?

cat /usr/local/antmedia/conf/red5.properties |grep -i "server.allowed_dashboard_CIDR"

[arnowelzel]

Hi @arnowelzel

Thank you for your detailed reproduce scenario. In the same way, I created it on my side, but I did not encounter any problems. Finally, could you share the output of the following command?

cat /usr/local/antmedia/conf/red5.properties |grep -i "server.allowed_dashboard_CIDR"

The output is:

server.allowed_dashboard_CIDR=0.0.0.0/0

Edit: just to be sure - it works for me, but only when also adding ProxyAddHeaders off in the proxy setup. Without this, access to the dashboard via the proxy is not allowed, only directly on port 5080.

[muratugureminoglu]

Hi @arnowelzel

Could the server's old IP address be IPv4? Has an update been made about this?

I think your problem is related to IPv6. Let me test this on my side. You will not face any problem if you can assign an IPv4 to the server as a workaround solution.

[arnowelzel]

@muratugureminoglu

No, the server never had IPv4 only, it is been connected with IPv4 and IPv6 dual stack for the last 5 years now and older versions of Ant Media worked fine with that even behind the reverse proxy.

The confusing thing is, that accessing it directly via port 5080 always works - also with IPv6. Only the requests to the dashboard coming from the proxy won't be accepted without using ProxyAddHeaders off.

[muratugureminoglu]

Yes, It will work if you disable ProxyHeaders because it will forward all requests as a local address (127.0.0.1) and we need to confirm this on our side.

Edit : Can you disable IPv6 temporarily and then try it again? @arnowelzel

[arnowelzel]

@muratugureminoglu sorry, disabling IPv6 is not an option on that machine as this would disrupt network connectivity. But I'll try to find a client which is only connected using IPv4 to confirm if this is the issue.

[arnowelzel]

@muratugureminoglu I can confirm, that IPv6 is the problem here. When connecting with a client using IPv4 only, the reverse proxy works fine also without ProxyAddHeaders off. But as soon as I try it using IPv6, requests end up in HTTP 403.

[muratugureminoglu]

@arnowelzel

I have just created an issue about this issue[1]. We will solve it as soon as possible.

[1] https://github.com/ant-media/Ant-Media-Server/issues/4371

[arnowelzel]

As it turned out, Ant Media seems not to support IPv6 at all (at least not the community version I am using at the moment).

Without a proxy a client with dual stack will connect using IPv4. But when disabling IPv4 on the client or using a FQDN which only has an AAAA record, connections to port 5080 are not possible at all.

With the proxy IPv6 requests will get forwarded via IPv4 to localhost - but as long as the proxy adds the original source address in the headers, Ant Media will try to interpret that and refuses the connection if it is not an IPv4 address. By adding ProxyAddHeaders off the request will not cotain any information at all about the real source but only look like regular requests from 127.0.0.1.