Rotating peers to mitigate API rate limits?

[nandubatchu]

@sammchardy is there a way this can be achieved in the existing code? Like query the other node in case of API rate limit breach?

[sammchardy]

Hi @nandubatchu this does sound like a good idea. Can you give me some more information about your requirements here.

[sammchardy]

Hi @nandubatchu I've implemented this in v0.1.18. Check the documentation for it here.

It rotates through each available peer node to increase your API rate limit.

Let me know if you have any feedback.

[nandubatchu]

@sammchardy does this work both for HttpRpcClient and HttpApiClient? Also how reliably can we use this in broadcasting transactions? Will this cause any data inconsistency issues or double spends? Do you maintain the list of peers internally for both testnet and mainnet environments?

[sammchardy]

Hi @nandubatchu. This works for the RPC client only as the HTTP client only has one endpoint.

The list of peers is fetched for the environment when you initialise the client, so it's dynamic. You can also refresh the peers with the initialise_peers function.

It won't send a transaction multiple times unless you tell it to. It just chooses a new peer to send the request to each time.

[nandubatchu]

@sammchardy

aiohttp.client_exceptions.ClientConnectorSSLError: Cannot connect to host testnet-dex.binance.org:443 ssl:None [[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)]

[nandubatchu]

@sammchardy everytime I initialize the client, it starts over from the 0th index of the peer-list? If I am initializing the client on every invocation, it always will end up to pick the same peer node?

[nandubatchu]

@sammchardy also can the HttpRpcClient and HttpApiClient can be used with our own lightnode/fullnode instances? Where we can remove the API rate limits for our usecases?

[nandubatchu]

@sammchardy

aiohttp.client_exceptions.ClientConnectorSSLError: Cannot connect to host testnet-dex.binance.org:443 ssl:None [[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)]

possibly related: https://github.com/ccxt/ccxt/issues/927

https://github.com/ccxt/ccxt/commit/caca5c46db445c73307efda3ab4ec2af13788b2b

[sammchardy]

Hi @nandubatchu if you want to use your own instances you can construct your own BinanceEnvironment instance for the HttpApiClient, or pass your listen address for HttpRpcClient.

my_env = BinanceEnvironment(api_url='<api_url'>, wss_url='<wss_url>', hrp='<bnb|tbnb>')
http_client = HttpApiClient(env=my_env)

Where the hrp is bnb of production or tbnb for testnet .

For HttpRpcClient simply use

rpc_client = HttpRpcClient(listen_addr='<my_node_url>')

[sammchardy]

I have added shuffling of the endpoints on initialisation in v0.1.19 so you will connect to a different order each time.

Regarding the SSL error, can you give more details about your OS, python version, virtualenv. Does it happen when using the production environment, or just the testnet?

[nandubatchu]

@sammchardy I am using MacOS and python 3.6.5 with virtualenv. It is happening both on production and testnet Environments.

[nandubatchu]

@sammchardy adding sslcontext = ssl.create_default_context(cafile=certifi.where()) inside _create_direct_connection in connector.py solved the issue. But there are other errors:

ERROR:asyncio:Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x10eb09b38>
ERROR:asyncio:Unclosed connector
connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x10eb268e8>, 71951.87460465)]']
connector: <aiohttp.connector.TCPConnector object at 0x10eb09b70>

[nandubatchu]

@sammchardy are you working on the fix? I would like to use it in production environment.

[sammchardy]

Did you try any of the suggestions in the issue you linked?

How about updating python to 3.7 using brew install python?