Enable company proxy/global agent support for the REST connector

[rolandwang19]

I'm running Budibase on Centos7 behind a proxy. I've set all the global_agent proxy variables, and I'm able to access budibase, and the templates are loading, so I believe it's working as they weren't loading and throwing an error before the variables were set. But now I'm trying to create a REST API Query, and get a ENETUNREACH error. Not sure if all the variables are setup properly, or if I'm missing a configuration somewhere specifically for this.

version: "3"

# optional ports are specified throughout for more advanced use cases.

services:
  app-service:
    restart: unless-stopped
    image: budibase.docker.scarf.sh/budibase/apps
    container_name: bbapps
    environment:
      GLOBAL_AGENT_HTTP_PROXY: http://10.121.80.116:8080
      GLOBAL_AGENT_HTTPS_PROXY: http://10.121.80.116:8080
      GLOBAL_AGENT_NO_PROXY: couchdb-service,app-service,worker-service,minio-service,redis-service,localhost, watchtower-service
      SELF_HOSTED: 1
      COUCH_DB_URL: http://${COUCH_DB_USER}:${COUCH_DB_PASSWORD}@couchdb-service:5984
      WORKER_URL: http://worker-service:4003
      MINIO_URL: http://minio-service:9000
      MINIO_ACCESS_KEY: ${MINIO_ACCESS_KEY}
      MINIO_SECRET_KEY: ${MINIO_SECRET_KEY}
      INTERNAL_API_KEY: ${INTERNAL_API_KEY}
      BUDIBASE_ENVIRONMENT: ${BUDIBASE_ENVIRONMENT}
      PORT: 4002
      JWT_SECRET: ${JWT_SECRET}
      LOG_LEVEL: info
      SENTRY_DSN: https://[email protected]/5338131
      ENABLE_ANALYTICS: "true"
      REDIS_URL: redis-service:6379
      REDIS_PASSWORD: ${REDIS_PASSWORD}
    depends_on:
      - worker-service
      - redis-service

  worker-service:
    restart: unless-stopped
    image: budibase.docker.scarf.sh/budibase/worker
    container_name: bbworker
    environment:
      GLOBAL_AGENT_HTTP_PROXY: http://10.121.80.116:8080
      GLOBAL_AGENT_HTTPS_PROXY: http://10.121.80.116:8080
      GLOBAL_AGENT_NO_PROXY: couchdb-service,app-service,worker-service,minio-service,redis-service,localhost, watchtower-service
      SELF_HOSTED: 1
      PORT: 4003
      CLUSTER_PORT: ${MAIN_PORT}
      JWT_SECRET: ${JWT_SECRET}
      MINIO_ACCESS_KEY: ${MINIO_ACCESS_KEY}
      MINIO_SECRET_KEY: ${MINIO_SECRET_KEY}
      MINIO_URL: http://minio-service:9000
      APPS_URL: http://app-service:4002
      COUCH_DB_USERNAME: ${COUCH_DB_USER}
      COUCH_DB_PASSWORD: ${COUCH_DB_PASSWORD}
      COUCH_DB_URL: http://${COUCH_DB_USER}:${COUCH_DB_PASSWORD}@couchdb-service:5984
      SENTRY_DSN: https://[email protected]/5338131
      INTERNAL_API_KEY: ${INTERNAL_API_KEY}
      REDIS_URL: redis-service:6379
      REDIS_PASSWORD: ${REDIS_PASSWORD}
    depends_on:
      - redis-service
      - minio-service
      - couch-init

  minio-service:
    restart: unless-stopped
    image: minio/minio
    volumes:
      - minio_data:/data
    environment:
      MINIO_ACCESS_KEY: ${MINIO_ACCESS_KEY}
      MINIO_SECRET_KEY: ${MINIO_SECRET_KEY}
      MINIO_BROWSER: "off"
    command: server /data
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  proxy-service:
    restart: unless-stopped
    ports:
      - "${MAIN_PORT}:10000"
    container_name: bbproxy
    image: budibase/proxy
    depends_on:
      - minio-service
      - worker-service
      - app-service
      - couchdb-service

  couchdb-service:
    restart: unless-stopped
    image: ibmcom/couchdb3
    environment:
      - COUCHDB_PASSWORD=${COUCH_DB_PASSWORD}
      - COUCHDB_USER=${COUCH_DB_USER}
    volumes:
      - couchdb3_data:/opt/couchdb/data

  couch-init:
    image: curlimages/curl
    environment:
      PUT_CALL: "curl -u ${COUCH_DB_USER}:${COUCH_DB_PASSWORD} -X PUT couchdb-service:5984"
    depends_on:
      - couchdb-service
    command: ["sh","-c","sleep 10 && $${PUT_CALL}/_users && $${PUT_CALL}/_replicator; fg;"]

  redis-service:
    restart: unless-stopped
    image: redis
    command: redis-server --requirepass ${REDIS_PASSWORD}
    volumes:
      - redis_data:/data

  watchtower-service:
    restart: always
    image: containrrr/watchtower
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    command: --debug --http-api-update bbapps bbworker bbproxy
    environment:
      - WATCHTOWER_HTTP_API=true
      - WATCHTOWER_HTTP_API_TOKEN=budibase
      - WATCHTOWER_CLEANUP=true
    labels:
      - "com.centurylinklabs.watchtower.enable=false"

volumes:
  couchdb3_data:
    driver: local
  minio_data:
    driver: local
  redis_data:
    driver: local 

[rolandwang19]

@shogunpurple Any idea why rest API queries don't seem to be using the proxy? The templates are loading, where before they weren't before adding the global-agent environment variables. I'm also able to curl the proxy from within the container.

[rolandwang19]

Any update on this? Seems like with many companies more and more worried about security, that this would keep many more advanced tech organizations from leveraging the full functionality of Budibase. @shogunpurple

[rolandwang19]

Any update on this @shogunpurple

[shogunpurple]

@rolandwang19 after checking, the REST API agent does not use the proxy because it's a separately instantiated object - we need to add global agent support to the REST connector for this to work. Will update the title of this issue to reflect

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity.

[lanedsmu]

I believe we're experiencing this bug (ref discussion). Any update on this issue?

[lanedsmu]

This also raises a question: is it possible, via logs or otherwise, to see which traffic is routed through the proxy?

[bengalaviz]

Maybe this would make it easier to implement using proxy with node-fetch?

https://www.zenrows.com/blog/node-fetch-proxy

[nitrinity]

Hi, we have the same problem. Is there a new status or a known workaround? Somehow we have to move forward. Thank you, kind regards!

[lanedsmu]

Hi, we have the same problem. Is there a new status or a known workaround? Somehow we have to move forward. Thank you, kind regards!

@nitrinity, FWIW, our "solution" was simply to open our datacenter firewall from the Budibase server to the remote API endpoint. This allowed the traffic outbound without requiring a proxy.