Can't ingest bitfinex data

[chairy11]

Dear Catalyst Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

• Operating System: macOS Sierra 10.12.4
• Python Version: Python 3.6.4 :: Anaconda, Inc.
• How did you install Catalyst: conda
• Python packages: ccxt 1.12.131

Now that you know a little about me, let me tell you about the issue I am having:

Description of Issue

• Expect： get data before backtest by running catalyst ingest-exchange -x bitfinex -i btc_usd

• What happened instead? raise error

Trying to ingest exchange bundle bitfinex...
[2018-07-01 12:45:48.908644] WARNING: CCXT: unable to fetch markets bitfinex: bitfinex GET https://api.bitfinex.com/v1/symbols_details HTTPSConnectionPool(host='api.bitfinex.com', port=443): Max retries exceeded with url: /v1/symbols_details (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x114e242e8>, 'Connection to api.bitfinex.com timed out. (connect timeout=10)'))
Traceback (most recent call last):
  File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/util/connection.py", line 83, in create_connection
    raise err
  File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request
    self._validate_conn(conn)
  File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
    conn.connect()
  File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/connection.py", line 284, in connect
    conn = self._new_conn()
  File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/connection.py", line 146, in _new_conn
    (self.host, self.timeout))
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.VerifiedHTTPSConnection object at 0x114e242e8>, 'Connection to api.bitfinex.com timed out. (connect timeout=10)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.bitfinex.com', port=443): Max retries exceeded with url: /v1/symbols_details (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x114e242e8>, 'Connection to api.bitfinex.com timed out. (connect timeout=10)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/ccxt/base/exchange.py", line 356, in fetch
    proxies=self.proxies
  File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/requests/adapters.py", line 496, in send
    raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='api.bitfinex.com', port=443): Max retries exceeded with url: /v1/symbols_details (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x114e242e8>, 'Connection to api.bitfinex.com timed out. (connect timeout=10)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/ccxt/ccxt_exchange.py", line 108, in init
    markets_symbols = self.api.load_markets()
  File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/ccxt/base/exchange.py", line 870, in load_markets
    markets = self.fetch_markets()
  File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/ccxt/bitfinex.py", line 297, in fetch_markets
    markets = self.publicGetSymbolsDetails()
  File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/ccxt/base/exchange.py", line 305, in request
    return self.fetch2(path, api, method, params, headers, body)
  File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/ccxt/base/exchange.py", line 302, in fetch2
    return self.fetch(request['url'], request['method'], request['headers'], request['body'])
  File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/ccxt/base/exchange.py", line 366, in fetch
    self.raise_error(RequestTimeout, method, url, e)
  File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/ccxt/base/exchange.py", line 287, in raise_error
    raise exception_type(output)
ccxt.base.errors.RequestTimeout: bitfinex GET https://api.bitfinex.com/v1/symbols_details HTTPSConnectionPool(host='api.bitfinex.com', port=443): Max retries exceeded with url: /v1/symbols_details (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x114e242e8>, 'Connection to api.bitfinex.com timed out. (connect timeout=10)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/chairy11/anaconda3/envs/catalyst/bin/catalyst", line 11, in <module>
    sys.exit(main())
  File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/__main__.py", line 609, in ingest_exchange
    csv=csv
  File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange_bundle.py", line 811, in ingest
    self.exchange = get_exchange(self.exchange_name)
  File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/utils/factory.py", line 43, in get_exchange
    exchange.init()
  File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/ccxt/ccxt_exchange.py", line 125, in init
    raise ExchangeRequestError(error=e)
catalyst.exchange.exchange_errors.ExchangeRequestError: Request failed: bitfinex GET https://api.bitfinex.com/v1/symbols_details HTTPSConnectionPool(host='api.bitfinex.com', port=443): Max retries exceeded with url: /v1/symbols_details (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x114e242e8>, 'Connection to api.bitfinex.com timed out. (connect timeout=10)'))

Here is how you can reproduce this issue on your machine:

Reproduction Steps

1. vi ~/.catalyst/data/exchanges/bitfinex/auth.json and set the bitfinex API key
2. run catalyst ingest-exchange -x bitfinex -i btc_usd ...

What steps have you taken to resolve this already?

1. I try to open the url https://api.bitfinex.com/v1/symbols_details in my brower and get the right data.
2. I try to run catalyst ingest-exchange -x bittrex -i btc_usdt, and it seems ok.

Did I do anything wrong?

Sincerely, chairy11

[lenak25]

Thanks for opening a new issue. Although the ingested historical data is downloaded from catalyst servers, as part of catalyst initialization, the supported markets are still fetched directly from the exchange (using the CCXT library), which seems to fail in your case. You are following the instructions correctly, from the stack trace, it seems you are experiencing some connectivity issues with Bitfinex.

[chairy11]

@lenak25 I meet the same iusse on binance with command catalyst ingest-exchange -x binance -i btc_usdt . I have no idea how to solve this problem.
Could I get the history data with another way?

[lenak25]

Hi @chairy11 , sorry for the delay in the response. Are you still experiencing these issues?