OpenMetadata icon indicating copy to clipboard operation
OpenMetadata copied to clipboard

Published 20 hours ago • verified publisher icon open-metadata

Prepare OM infra visuals

Open pmbrull opened this issue 3 years ago • 3 comments
trafficstars

Prepare diagrams for the different types of deployments so users can easily understand what is the proposed architecture in each case:

  • [ ] bare metal - @darth-coder00
  • [ ] docker - @ayush-shah
  • [x] k8s - @akash-jain-10

pmbrull avatar May 17 '22 08:05 pmbrull

@akash-jain-10 I am assigning you as I cannot find the other members in the assignee list. Feel free to pass the load! Thanks 🙏

pmbrull avatar May 17 '22 08:05 pmbrull

Sure! Thanks @pmbrull !

akash-jain-10 avatar May 17 '22 12:05 akash-jain-10

Draft image for k8s Architecture from Helm Deployments -

om-k8s

akash-jain-10 avatar May 26 '22 12:05 akash-jain-10

@akash-jain-10 @Vj-L any updates?

pmbrull avatar Dec 23 '22 08:12 pmbrull

@shahsank3t @akash-jain-10 need updates on this ticket

harshach avatar Jan 08 '23 04:01 harshach

Draft Architecture for Bare Metal : Baremetal Visual Diagram

Vj-L avatar Jan 11 '23 18:01 Vj-L

Architecture Diagram for Docker Deployment: Baremetal Visual Diagram-Page-2(1)

Vj-L avatar Jan 13 '23 12:01 Vj-L

Architecture Diagram for Docker Deployment:

Baremetal Visual Diagram-Page-2(2)

Detailed Description:

Port Mapping / Port Forwarding

For OpenMetadata-Server

We are shipping the OpenMetadata server and UI at 8585, and the ingestion container (Airflow) at 8080. You can change the port number's according to your requirement. As an example, You could update the ports to serve OpenMetadata Server and UI at port 80

To achieve this

You just have to update the ports mapping of the openmetadata-server in the docker-compose.yml file under openmetadata-server docker service section. ports:

  • "80:8585"

Once the port is updated if there are any containers running remove them first using docker compose down command and then recreate the containers once again by below command docker compose up --build -d

For Ingestion-Server

We are shipping the OpenMetadata server and UI at 8585, and the ingestion container (Airflow) at 8080. You can change the port number's according to your requirement. As an example, You could update the ports to serve Ingestion Server and UI at port 80

To achieve this

You just have to update the ports mapping of the openmetadata-server in the docker-compose.yml file under ingestion-server docker service section. ports:

  • "80:8080"

Also update the Airflow environment variable in openmetadata-server section

AIRFLOW_HOST: '<AIRFLOW_HOST:-<AIRFLOW_HOST:80>'

Once the port is updated if there are any containers running remove them first using docker compose down command and then recreate the containers once again by below command

docker compose up --build -d

PROD Deployment of OpenMetadata Using Docker

If you are planning on going to PROD, we recommend to validate below points:

MySQL and OpenSearch (ElasticSearch) are available. OpenMetadata-Server require the minimum configuration of 2vCPU and 6Memory (GiB) OpenMetadata-Ingestion require the minimum configuration of 2vCPU and 8Memory (GiB) We also recommend to bind Docker Volumes for data persistence. Learn how to do so here. Steps for Deploying Ingestion Download the docker-compose.yml file from the release page here. Update the environment variables below for OpenMetadata-Ingestion Docker Compose backed systems to connect with Database.

MySQL Environment Variables for ingestion service

DB_HOST: '<DB_HOST_NAME>'
DB_PORT: '<DB_PORT>'
AIRFLOW_DB: '<AIRFLOW_DATABASE>'
AIRFLOW_DB_SCHEME: '<AIRFLOW_DB_SCHEME>'
DB_USER: '<AIRFLOW_DB_USER>'
DB_PASSWORD: '<AIRFLOW_DB_PASSWORD>'

Once the environment variables values with the RDS are updated then provide this environment variable file as part of docker compose command.

docker compose --env-file ./config/.env.prod up -d openmetadata_ingestion Steps for Deploying OpenMetadata-Server Download the docker-compose.yml file from the release page here. Update the environment variables below for OpenMetadata-Ingestion Docker Compose backed systems to connect with Database and ElasticSearch and Ingestion.

MySQL Environment Variables

DB_DRIVER_CLASS='com.mysql.cj.jdbc.Driver'
DB_SCHEME='mysql'
DB_USE_SSL='true'
DB_USER_PASSWORD='<OPENMETADATA_DB_USER_PASSWORD>'
DB_SCHEME='mysql'
DB_HOST='<DB_HOST>'
DB_USER='<OPENMETADATA__USER_NAME>'
OM_DATABASE='<OPENMETADATA_DATABASE_NAME>'
DB_PORT='<DB_PORT>'

ElasticSearch Environment Variables

ELASTICSEARCH_SOCKET_TIMEOUT_SECS='60'
ELASTICSEARCH_USER='<ELASTICSEARCH_USERNAME>'
ELASTICSEARCH_CONNECTION_TIMEOUT_SECS='5'
ELASTICSEARCH_PORT='443'
ELASTICSEARCH_SCHEME='https'
ELASTICSEARCH_BATCH_SIZE='10'
ELASTICSEARCH_HOST='<ELASTICSEARCH_HOST_URL>'
ELASTICSEARCH_PASSWORD='<ELASTICSEARCH_PASSWORD>'

Ingestion or Airflow Environment Variables

AIRFLOW_HOST: '<AIRFLOW_HOST_URL>'
SERVER_HOST_API_URL: '<OPENMETADATA_HOST_URL_WITH_SCHEME/api>'

Once the environment variables values with the RDS are updated then provide this environment variable file as part of docker compose command.

docker compose --env-file ./config/.env.prod up -d openmetadata_server Run OpenMetadata with AWS Services If you are running OpenMetadata in AWS, it is recommended to use Amazon RDS and Amazon OpenSearch Service.

We support

Amazon RDS (MySQL) engine version upto 8.0.29 Amazon OpenSearch (ElasticSearch) engine version upto 7.10 or Amazon OpenSearch engine version upto 1.3 Amazon RDS (PostgreSQL) engine version upto 14.2-R1 For Production Systems, we recommend Amazon RDS to be in Multiple Availibility Zones. For Amazon OpenSearch (or ElasticSearch) Service, we recommend Multiple Availibility Zones with minimum 3 Master Nodes.

Once you have the RDS and OpenSearch Services Setup, you can update the environment variables below for OpenMetadata Docker Compose backed systems to connect with Database and ElasticSearch.

MySQL Environment Variables

DB_DRIVER_CLASS='com.mysql.cj.jdbc.Driver'
DB_SCHEME='mysql'
DB_USE_SSL='true'
DB_USER_PASSWORD='<DATABASE_USER_PASSWORD>'
DB_SCHEME='mysql'
DB_HOST='<DATABASE_HOST_NAME>'
DB_USER='<DATABASE_USER_NAME>'
OM_DATABASE='<DATABASE_NAME>'
DB_PORT='<DATABASE_PORT>'
# ElasticSearch Environment Variables
ELASTICSEARCH_SOCKET_TIMEOUT_SECS='60'
ELASTICSEARCH_USER='<ELASTICSEARCH_USERNAME>'
ELASTICSEARCH_CONNECTION_TIMEOUT_SECS='5'
ELASTICSEARCH_PORT='443'
ELASTICSEARCH_SCHEME='https'
ELASTICSEARCH_BATCH_SIZE='10'
ELASTICSEARCH_HOST='<ELASTICSEARCH_HOST_URL>'
ELASTICSEARCH_PASSWORD='<ELASTICSEARCH_PASSWORD>'

Replace the environment variables values with the RDS and OpenSearch Service ones and then provide this environment variable file as part of docker compose command.

docker compose --env-file ./config/.env.prod up -d openmetadata_server

Vj-L avatar Jan 30 '23 05:01 Vj-L

@open-metadata/devops can we move this to the docs please instead of having it here?

pmbrull avatar Jun 28 '23 15:06 pmbrull

Hi @pmbrull ,

Please review this beautified docker architecture and let me know if any other corrections are required or not. Thanks in advance.

docker_om_architecture-3

  • Will add below overview directly into the document page:
High-level overview: 

* Deploying with MySQL 3306 /PostgreSQL 5432 : Download docker-compose.yml / docker-compose-postgres.yml from the link: https://github.com/open-metadata/OpenMetadata/releases
/OpenMetadata/releases.
* We are shipping the Elasticsearch service and Ul at 9200.
* We are shipping the OpenMetadata server and Ul at 8585.
* We are shipping the ingestion container (Airflow) at 8080.
* You can change the port number's according to your requirement.

dhruvinmaniar123 avatar Jun 29 '23 08:06 dhruvinmaniar123