Voyz
Client portal returning Access Denied page after 2 months of using IBeam
Hi,
I'd like to say thank you first for making this extremely useful tool, it's become irreplaceable in the last few months. I am not sure if what I'm reporting is a bug, but I've been trying to understand the problem for the past 3 days with no luck. I hope you have time to take a look and offer some advice.
Describe the bug
I've deployed the IBeam Docker image on a Kubernetes pod in my cluster and I have been using it normally in this capacity to place orders for around 2 months. However, this pod was restarted around 3 days ago, and it has not been able to log in to Interactive Brokers since then because the login process keeps resulting in a timeout.
The logs are saying that the timeout is caused by "http://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on" returning a page saying Access Denied and the response code 403 Forbidden instead of loading the login form. I've tried to run the Docker container locally with the same credentials and conf.yaml to see if it's account/configuration related but it returned the login form without problems. I've also tried to ping "http://api.ibkr.com/sso/Login?forwardTo=22&RL=1&ip2loc=on" directly using curl which worked in the local Docker container but returned the Access Denied page when I did it from the pod.
Environment
IBeam version: Happening on both latest (based on 0.5.1?) and 0.5.2-rc3 Docker image or standalone: Docker image used inside a Kubernetes pod Python version (standalone users only): / OS: /
Additional context
I've had a look at the Troubleshooting section but the IPs in my conf.yaml already covered that problem (and I added 10.* to make it work in the internal network).
I changed the IBEAM_GATEWAY_BASE_URL to point to http://localhost:5000/ and I am using a custom conf.yaml to disable listen ssl since I am using Kubernetes to handle HTTPS. The logs do say "Custom conf.yaml found and will be used by the Gateway" and it is correctly targeting the URL I specified in IBEAM_GATEWAY_BASE_URL when trying to authenticate so the configuration is being loaded properly. I've also tried it without the custom configuration anyway and that did not fix it.
Here are the configuration and log files. Added as .txt since that's allowed by GitHub. conf.txt env.txt logs.txt
I've had a look at other issues related to this. There was an issue that mentioned setting {number between 1 and 8}.api.ibkr.com as the proxy remote host would help but I've tried with 1 and 6 and it didn't work. My conf.yaml is still using 1.api.ibkr.com since that was my last attempt. I also found someone who had a similar problem but they posted in this issue after it was closed.
Suggest a Fix
I think Interactive Brokers might've blacklisted my Kubernetes node's IP, but I've not seen their documentation mention this can happen at any point. I have seen rate limits in their documentation but none of them are related to the authentication endpoint. If it's an undocumented rate limit on the authentication endpoint, then I guess a backoff mechanism would help. But I have no idea if this is the issue.
Thank you for maintaining this tool, it has really come in handy for testing the Interactive Brokers API.
Hey @Androoideka thanks for trying out IBeam and for your kind words 😊 Also thank you for getting into detail regarding your issue and passing over the logs and configs 👍
Looking at the log file I'm seeing some unusual behaviour:
2024-04-17 14:55:56,510|I| Gateway running but not serving yet. Consider increasing IBEAM_GATEWAY_STARTUP timeout. Error: <urlopen error [Errno 111] Connection refused>
2024-04-17 14:57:41,658|W| Cannot determine the version of IBKR website, assuming version 1
These could indicate that something is not going well with the Gateway. Maybe not enough CPU/RAM? You could try to ssh into your k8 node and do some diagnostics to see if the Gateway is started correctly. The best way would be to connect to a display from which you could open a browser and access the login page manually to verify if it loads correctly, though I'm not sure how possible this would be on k8.
I think Interactive Brokers might've blacklisted my Kubernetes node's IP
It's best to talk to IBKR support directly about this. They do mention some rate limiting and banning in their docs, so possibly this is their way of indicating it. I wish I could be of more help here, but I don't know what could be causing this error - other than that your IP is not listed in the conf.yaml, which is always worth double checking.
I'm going to close this issue due to inactivity. Thanks for your contribution and please feel free to request a reopen if you'd like to continue the discussion 👍