HTTP Task failure: Unable to connect to Postgres whilst looking up IP Blacklist on GCP using private networking

[ijonas]

Description HTTP Task failure: Unable to connect to Postgres whilst looking up IP Blacklist on GCP using private networking

When executing a HTTP Task as part of a standard "Get > Uint256" job, the task fails complaining about a "disallowed IP".

The full error in the CL Node GUI is: connections to local resources are disabled by default, if you are sure this is safe, you can enable on a per-task basis by setting allowUnrestrictedNetworkAccess="true" in the pipeline task spec, e.g. fetch [type="http" method=GET url="$(decode_cbor.url)" allowUnrestrictedNetworkAccess="true"]: error making http request: Get "https://min-api.cryptocompare.com/data/pricemultifull?fsyms=ETH&tsyms=USD": disallowed IP 40.115.22.134. Connections to local/private and multicast networks are disabled by default for security reasons: disallowed IP

The job and included HTTP task used to run successfully under smartcontract/chainlink:1.4.1. It now fails under smartcontract/chainlink:1.7.1 and 1.8.0.

The bug is actually caused by a failure of the node to lookup the list of blacklisted IP addresses. The node cannot establish a DB connection to retrieve the blacklist and in failing to do so blacklists any IP address by default. The blacklist feature was introduced in 1.5.0.

The establishment of a DB connection to fetch the blacklist seems to be different than in other parts of the node software. The node has no issues booting up and loading its configuration including lists of jobs and tasks, and the GUI is operating as expected.

Further investigation shows that the code to fetch the blacklist performs a check on the database's IP address by calling net.LookupIP(dbURL.String()) See https://github.com/smartcontractkit/chainlink/blob/v1.5.0/core/utils/http/http_allowed_ips.go#L73 The full error log is attached below. This net.LookupIP() call only exists in this part of the node software and isn't used by any parts of the node software during the establishment of a DB connection.

I think this check is failing because we're running Postgres via GCP's hosted CloudSQL using private IP networking. The node software is running under Kubernetes and can establish DB connection using the private IP address of the CloudSQL instance. The node software is also able to resolve the private IP address of the CloudSQL instance using the Kubernetes DNS service. However, the net.LookupIP() call is failing to resolve the private IP address of the CloudSQL instance. We've also tested that we can connect from inside the K8S cluster to the Postgres DB using Google's cloud_sql_proxy tool using the private IP address of the CloudSQL instance. However we cannot ping the private IP address of the CloudSQL instance from the node container.

Basic Information

• Network: Rinkeby (although issue isn't EVM network specific)
• Operating System: Container-Optimized OS from Google
• Kubernetes Version: 1.22.10-gke.600
• Commit: [log INFO line when starting node]
• Hosting Provider: GCP
• Startup Command: Kubernetes pod runnning smartcontract/chainlink:1.8.0-root ["local", "n", "-p", "/chainlink/.password", "-a", "/chainlink/.api"]

Environment Variables ADVISORY_LOCK_CHECK_INTERVAL 1s ADVISORY_LOCK_ID 1027321974924625846 ALLOW_ORIGINS * BLOCK_BACKFILL_DEPTH 10 BLOCK_HISTORY_ESTIMATOR_BLOCK_DELAY 0 BLOCK_HISTORY_ESTIMATOR_BLOCK_HISTORY_SIZE 0 BLOCK_HISTORY_ESTIMATOR_TRANSACTION_PERCENTILE 0 BRIDGE_RESPONSE_URL CHAINLINK_DEV false CHAINLINK_PORT 6688 CHAINLINK_TLS_HOST CHAINLINK_TLS_PORT 0 CHAINLINK_TLS_REDIRECT false CHAIN_TYPE DATABASE_BACKUP_FREQUENCY 1h0m0s DATABASE_BACKUP_MODE none DATABASE_BACKUP_ON_VERSION_UPGRADE true DATABASE_LOCKING_MODE lease DEFAULT_HTTP_LIMIT 32768 DEFAULT_HTTP_TIMEOUT 15s ETH_CHAIN_ID 4 ETH_HTTP_URL ETH_SECONDARY_URLS [https://rinkeby.infura.io/v3/xxx] ETH_URL wss://rinkeby.infura.io/ws/v3/xxx EVM_RPC_ENABLED true EXPLORER_URL FEATURE_EXTERNAL_INITIATORS false FEATURE_OFFCHAIN_REPORTING false FLAGS_CONTRACT_ADDRESS FM_DEFAULT_TRANSACTION_QUEUE_DEPTH 1 GAS_ESTIMATOR_MODE INSECURE_FAST_SCRYPT false JOB_PIPELINE_REAPER_INTERVAL 1h0m0s JOB_PIPELINE_REAPER_THRESHOLD 24h0m0s JSON_CONSOLE true KEEPER_BASE_FEE_BUFFER_PERCENT 20 KEEPER_CHECK_UPKEEP_GAS_PRICE_FEATURE_ENABLED false KEEPER_DEFAULT_TRANSACTION_QUEUE_DEPTH 1 KEEPER_GAS_PRICE_BUFFER_PERCENT 20 KEEPER_GAS_TIP_CAP_BUFFER_PERCENT 20 KEEPER_MAXIMUM_GRACE_PERIOD 100 KEEPER_REGISTRY_CHECK_GAS_OVERHEAD 200000 KEEPER_REGISTRY_PERFORM_GAS_OVERHEAD 150000 KEEPER_REGISTRY_SYNC_INTERVAL 30m0s KEEPER_REGISTRY_SYNC_UPKEEP_QUEUE_SIZE 10 KEEPER_TURN_FLAG_ENABLED false KEEPER_TURN_LOOK_BACK 1000 LEASE_LOCK_DURATION 10s LEASE_LOCK_REFRESH_INTERVAL 1s LINK_CONTRACT_ADDRESS LOG_FILE_DIR /chainlink LOG_FILE_MAX_AGE 0 LOG_FILE_MAX_BACKUPS 1 LOG_FILE_MAX_SIZE 1.02gb LOG_LEVEL debug LOG_SQL false OCR_DEFAULT_TRANSACTION_QUEUE_DEPTH 1 OCR_TRACE_LOGGING false P2PV2_ANNOUNCE_ADDRESSES [] P2PV2_BOOTSTRAPPERS [] P2PV2_DELTA_DIAL 15s P2PV2_DELTA_RECONCILE 1m0s P2PV2_LISTEN_ADDRESSES [] P2P_BOOTSTRAP_CHECK_INTERVAL 20s P2P_BOOTSTRAP_PEERS [] P2P_DHT_LOOKUP_INTERVAL 10 P2P_INCOMING_MESSAGE_BUFFER_SIZE 10 P2P_LISTEN_IP 0.0.0.0 P2P_LISTEN_PORT P2P_NETWORKING_STACK V1 P2P_NEW_STREAM_TIMEOUT 10s P2P_OUTGOING_MESSAGE_BUFFER_SIZE 10 P2P_PEER_ID REAPER_EXPIRATION 240h0m0s ROOT /chainlink SECURE_COOKIES false SESSION_TIMEOUT 15m0s SHUTDOWN_GRACE_PERIOD 5s TELEMETRY_INGRESS_LOGGING false TELEMETRY_INGRESS_SERVER_PUB_KEY TELEMETRY_INGRESS_URL TRIGGER_FALLBACK_DB_POLL_INTERVAL 30s

Node Logs: {"caller":"http/http_allowed_ips.go:62", "err":"failed to lookup IP for DB URL: lookup postgresql://chainlink:[email protected]:5432/chainlink-rinkeby?application_name=Chainlink+1.8.0+%7CDefault: no such host", "errVerbose":"lookup postgresql://chainlink:[email protected]:5432/chainlink-rinkeby?application_name=Chainlink+1.8.0+%7CDefault: no such host failed to lookup IP for DB URL github.com/smartcontractkit/chainlink/core/utils/http.isBlacklistedIP /chainlink/core/utils/http/http_allowed_ips.go:76 github.com/smartcontractkit/chainlink/core/utils/http.isRestrictedIP /chainlink/core/utils/http/http_allowed_ips.go:60 github.com/smartcontractkit/chainlink/core/utils/http.makeRestrictedDialContext.func1 /chainlink/core/utils/http/http_allowed_ips.go:105 net/http.(*Transport).dial /usr/local/go/src/net/http/transport.go:1169 net/http.(*Transport).dialConn /usr/local/go/src/net/http/transport.go:1607 net/http.(*Transport).dialConnFor /usr/local/go/src/net/http/transport.go:1449 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1571", "ip":"40.115.22.134", "level":"error", "logger":"1.8.0@bb6e95a", "msg":"Failed to check IP blacklist status, this IP will be blocked", "sentryEventID":null, "stacktrace":"github.com/smartcontractkit/chainlink/core/utils/http.isRestrictedIP /chainlink/core/utils/http/http_allowed_ips.go:62 github.com/smartcontractkit/chainlink/core/utils/http.makeRestrictedDialContext.func1 /chainlink/core/utils/http/http_allowed_ips.go:105 net/http.(*Transport).dial /usr/local/go/src/net/http/transport.go:1169 net/http.(*Transport).dialConn /usr/local/go/src/net/http/transport.go:1607 net/http.(*Transport).dialConnFor /usr/local/go/src/net/http/transport.go:1449", "ts":"2022-09-03T04:35:58.438Z"}

Steps to Reproduce Deploy a chainlink node to GKE using the smartcontract/chainlink:1.8.0-root image. Configure the node to use a CloudSQL instance as the database. Attempt to connect to the CloudSQL instance using the private IP address.

Deploy a standard "Get > Uint256" job to the node. Observe that the job fails to run the httpGet task.

Additional Information The workaround is to allowUnrestrictedNetworkAccess="true" on the httpGet task so that the blacklist is bypassed. This is not ideal as it raises security concerns.

[rgottleber]

Thank you for raising this issue. We will take a look.