Do anyone tried "cse install --config config.yaml" without encrypted config.yaml for CSE 2.6.1??

[ChandraRatra]

[ChandraRatra]

Do anyone have installed CSE 2.6.1 behind proxy. Need to know the URL's required to allow through proxy for CSE 2.6.1 installation, Template and Cluster deployment.

[rocknes]

Hi,

CSE 2.6.1 can be installed with an unencrypted config fie, you need to provide the flag -s to the command cse install.

I am not sure about your other ask regarding the proxy. Can you please elaborate?

Regards Aritra Sen

[ChandraRatra]

Thanks, I will try -s during cse install and will confirm you. I don't have direct internet connectivity to my CSE server, need to use proxy to provide internet connection. Also through proxy only allowed url's can be access. So need the list of URL's or repositories required to be access through internet during CSE installation, template deployment and Cluster creation.

[rocknes]

I have replied to Varun, will try to compile the list by early next week,

[ChandraRatra]

Hello, This is information regarding using proxy configuration incase direct internet connectivity not available to cse server. edit proxy file on cse server and added proxy details as per your environment. This will provide internet connectivity to cse server File: proxy File location /etc/sysconfig/proxy vi /etc/sysconfig/proxy

Enable a generation of the proxy settings to the profile.

This setting allows to turn the proxy on and off while

preserving the particular proxy setup.

PROXY_ENABLED="yes"

Some programs (e.g. wget) support proxies, if set in

the environment.

Example: HTTP_PROXY="http://proxy.provider.de:3128/"

HTTP_PROXY="http://PROXY_IP_ADDRESS"

Example: HTTPS_PROXY="https://proxy.provider.de:3128/"

HTTPS_PROXY="http://PROXY_IP_ADDRESS"

Example: FTP_PROXY="http://proxy.provider.de:3128/"

FTP_PROXY="http://PROXY_IP_ADDRESS"

Example: GOPHER_PROXY="http://proxy.provider.de:3128/"

GOPHER_PROXY="http://PROXY_IP_ADDRESS"

Example: SOCKS_PROXY="socks://proxy.example.com:8080"

SOCKS_PROXY="http://PROXY_IP_ADDRESS"

Example: SOCKS5_SERVER="office-proxy.example.com:8881"

SOCKS5_SERVER="http://PROXY_IP_ADDRESS"

Example: NO_PROXY="www.me.de, do.main, localhost"

NO_PROXY="localhost, 127.0.0.1, VCD_IP_ADDRESS, INTERNAL_IP_ADDRESS

If Proxy in your environment only allow whitelist URL's, add below list of URL's in proxy whitelist. List no need

https://raw.githubusercontent.com/vmware/container-service-extension-templates/master/template.yaml https://raw.githubusercontent.com/vmware/container-service-extension-templates/master/scripts/ubuntu-16.04_k8-1.17_weave-2.6.0_rev1/* https://raw.githubusercontent.com/vmware/container-service-extension-templates/master/scripts/ubuntu-16.04_k8-1.16_weave-2.6.0_rev1/* https://raw.githubusercontent.com/vmware/container-service-extension-templates/master/scripts/ubuntu-16.04_k8-1.15_weave-2.5.2 _rev3/* https://raw.githubusercontent.com/vmware/container-service-extension-templates/master/scripts/photon-v2_k8-1.14_weave-2.5.2 _rev2/* https://cloud-images.ubuntu.com/releases/xenial/release-20180418/ubuntu-16.04-server-cloudimg-amd64.ova http://dl.bintray.com/vmware/photon/2.0/GA/ova/photon-custom-hw11-2.0-304b817.ova http://archive.ubuntu.com/ubuntu/dists/* http://archive.ubuntu.com/ubuntu/dists/xenial/* http://archive.ubuntu.com/ubuntu/dists/xenial-updates/* http://security.ubuntu.com/ubuntu/dists/xenial-security/* http://archive.ubuntu.com/ubuntu/dists/xenial-backports/* https://download.docker.com/linux/ubuntu/dists/xenial/* https://download.docker.com/linux/ubuntu/dists/xenial/stable/* https://packages.cloud.google.com/apt/dists/kubernetes-xenial/* http://dl.bintray.com http://git.io http://gcr.io https://kubernetes.io https://docker.io http://cloud.weave.works/* https://cloud.weave.works/k8s/* https://cloud.weave.works/k8s/net?k8s-version=1.17.2&v=2.6.0

Now we need to provide proxy configuration for template deployment Once you run cse install command, ~/.cse_scripts/ folders will be created. Under this folder there will be a folder for each kubernetes template (do refer to the “cse-install_date_time.log” during installation for any additional URL) /photon-v2_k8-1.14_weave-2.5.2_rev2 /ubuntu-16.04_k8-1.15_weave-2.5.2_rev3 /ubuntu-16.04_k8-1.16_weave-2.6.0_rev1 /ubuntu-16.04_k8-1.17_weave-2.6.0_rev1

each folder have cust.sh, mstr.sh, node.sh file. Update these 3 files as per below information For Ubuntu template, update cust.sh #!/usr/bin/env bash set -e

disable ipv6 to avoid possible connection errors

echo 'net.ipv6.conf.all.disable_ipv6 = 1' >> /etc/sysctl.conf echo 'net.ipv6.conf.default.disable_ipv6 = 1' >> /etc/sysctl.conf echo 'net.ipv6.conf.lo.disable_ipv6 = 1' >> /etc/sysctl.conf sudo sysctl -p echo 'nameserver 10.41.67.1' >> /etc/resolvconf/resolv.conf.d/tail resolvconf -u export FTP_PROXY="http://PROXY_IP_ADDRES/" export HTTP_PROXY="http://PROXY_IP_ADDRES/" export HTTPS_PROXY="http://PROXY_IP_ADDRES/" echo 'Acquire::http::proxy "http://PROXY_IP_ADDRES";' >> /etc/apt/apt.conf echo 'Acquire::https::proxy "http://PROXY_IP_ADDRES";' >> /etc/apt/apt.conf echo 'Acquire::ftp::proxy "http://PROXY_IP_ADDRES";' >> /etc/apt/apt.conf echo 'https_proxy = http://PROXY_IP_ADDRES/' >> /etc/wgetrc echo 'http_proxy = http://PROXY_IP_ADDRES/' >> /etc/wgetrc echo 'ftp_proxy = http://PROXY_IP_ADDRES/' >> /etc/wgetrc echo 'use_proxy = on' >> /etc/wgetrc echo 'proxy = http://PROXY_IP_ADDRES' >> ~/.curlrc systemctl restart networking.service while [ systemctl is-active networking != 'active' ]; do echo 'waiting for network'; sleep 5; done

growpart /dev/sda 1 || : resize2fs /dev/sda1 || :

redundancy: https://github.com/vmware/container-service-extension/issues/432

systemctl restart networking.service while [ systemctl is-active networking != 'active' ]; do echo 'waiting for network'; sleep 5; done

echo 'installing kubernetes' export DEBIAN_FRONTEND=noninteractive apt-get -q update -o Acquire::Retries=3 -o Acquire::http::No-Cache=True -o Acquire::http::Timeout=30 -o Acquire::https::No-Cache=True -o Acquire::https::Timeout=30 -o Acquire::ftp::Timeout=30 apt-get -q install -y apt-transport-https ca-certificates curl software-properties-common curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - cat <<EOF > /etc/apt/sources.list.d/kubernetes.list deb http://apt.kubernetes.io/ kubernetes-xenial main EOF add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" apt-get -q update -o Acquire::Retries=3 -o Acquire::http::No-Cache=True -o Acquire::http::Timeout=30 -o Acquire::https::No-Cache=True -o Acquire::https::Timeout=30 -o Acquire::ftp::Timeout=30 apt-get -q install -y docker-ce=5:19.03.5~3-0~ubuntu-xenial apt-get -q install -y kubelet=1.17.2-00 kubeadm=1.17.2-00 kubectl=1.17.2-00 kubernetes-cni=0.7.5-00 systemctl restart docker while [ systemctl is-active docker != 'active' ]; do echo 'waiting for docker'; sleep 5; done

echo 'installing required software for NFS' apt-get -q install -y nfs-common nfs-kernel-server systemctl stop nfs-kernel-server.service systemctl disable nfs-kernel-server.service

prevent updates to software that CSE depends on

apt-mark hold open-vm-tools apt-mark hold docker-ce apt-mark hold kubelet apt-mark hold kubeadm apt-mark hold kubectl apt-mark hold kubernetes-cni apt-mark hold nfs-common apt-mark hold nfs-kernel-server

echo 'upgrading the system' apt-get -q update -o Acquire::Retries=3 -o Acquire::http::No-Cache=True -o Acquire::http::Timeout=30 -o Acquire::https::No-Cache=True -o Acquire::https::Timeout=30 -o Acquire::ftp::Timeout=30 apt-get -y -q -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" dist-upgrade

/etc/machine-id must be empty so that new machine-id gets assigned on boot (in our case boot is vApp deployment)

https://jaylacroix.com/fixing-ubuntu-18-04-virtual-machines-that-fight-over-the-same-ip-address/

truncate -s 0 /etc/machine-id rm /var/lib/dbus/machine-id || : ln -fs /etc/machine-id /var/lib/dbus/machine-id || : # dbus/machine-id is symlink pointing to /etc/machine-id

sync sync echo 'customization completed'

For Ubuntu template, update mstr.sh

#!/usr/bin/env bash set -e export FTP_PROXY="http://PROXY_IP_ADDRES/" export HTTP_PROXY="http://PROXY_IP_ADDRES/" export HTTPS_PROXY="http://PROXY_IP_ADDRES/" export NO_PROXY="127.0.0.1,INTERNAL_IP_ADDRESS" mkdir /etc/systemd/system/docker.service.d echo '[Service]' >> /etc/systemd/system/docker.service.d/http-proxy.conf echo 'Environment="HTTP_PROXY=http://PROXY_IP_ADDRES"' >> /etc/systemd/system/docker.service.d/http-proxy.conf echo '[Service]' >> /etc/systemd/system/docker.service.d/https-proxy.conf echo 'Environment="HTTPS_PROXY=http://PROXY_IP_ADDRES"' >> /etc/systemd/system/docker.service.d/https-proxy.conf systemctl daemon-reload systemctl restart docker

while [ systemctl is-active docker != 'active' ]; do echo 'waiting for docker'; sleep 5; done kubeadm init --kubernetes-version=v1.17.2 > /root/kubeadm-init.out mkdir -p /root/.kube cp -f /etc/kubernetes/admin.conf /root/.kube/config chown $(id -u):$(id -g) /root/.kube/config

export kubever=$(kubectl version --client | base64 | tr -d '\n') wget --no-verbose -O /root/weave.yml "https://cloud.weave.works/k8s/net?k8s-version=$kubever&v=2.6.0" kubectl apply -f /root/weave.yml systemctl restart kubelet while [ systemctl is-active kubelet != 'active' ]; do echo 'waiting for kubelet'; sleep 5; done

For Ubuntu template, update node.sh #!/usr/bin/env bash set -e

mkdir /etc/systemd/system/docker.service.d echo '[Service]' >> /etc/systemd/system/docker.service.d/http-proxy.conf echo 'Environment="HTTP_PROXY=http://PROXY_IP_ADDRES"' >> /etc/systemd/system/docker.service.d/http-proxy.conf echo '[Service]' >> /etc/systemd/system/docker.service.d/https-proxy.conf echo 'Environment="HTTPS_PROXY=http://PROXY_IP_ADDRES"' >> /etc/systemd/system/docker.service.d/https-proxy.conf

systemctl daemon-reload systemctl restart docker

while [ systemctl is-active docker != 'active' ]; do echo 'waiting for docker'; sleep 5; done kubeadm join --token {token} {ip}:6443 --discovery-token-unsafe-skip-ca-verification

For Photon template, update cust.sh #!/usr/bin/env bash

set -e

cat << EOF > /etc/systemd/system/iptables-ports.service [Unit] After=iptables.service Requires=iptables.service [Service] Type=oneshot ExecStartPre=/usr/sbin/iptables -P INPUT ACCEPT ExecStartPre=/usr/sbin/iptables -P OUTPUT ACCEPT ExecStart=/usr/sbin/iptables -P FORWARD ACCEPT TimeoutSec=0 RemainAfterExit=yes [Install] WantedBy=iptables.service EOF export FTP_PROXY="http://PROXY_IP_ADDRES/" export HTTP_PROXY="http://PROXY_IP_ADDRES/" export HTTPS_PROXY="http://PROXY_IP_ADDRES/" echo 'https_proxy = http://PROXY_IP_ADDRES/' >> /etc/wgetrc echo 'http_proxy = http://PROXY_IP_ADDRES/' >> /etc/wgetrc echo 'ftp_proxy = http://PROXY_IP_ADDRES/' >> /etc/wgetrc echo 'use_proxy = on' >> /etc/wgetrc echo 'proxy = http://PROXY_IP_ADDRES' >> ~/.curlrc

chmod 0644 /etc/systemd/system/iptables-ports.service systemctl enable iptables-ports.service systemctl start iptables-ports.service

update repo info (needed for docker update)

tdnf makecache -q tdnf update tdnf -y

echo 'installing kubernetes' tdnf install -yq wget kubernetes-1.14.6-3.ph2 kubernetes-kubeadm-1.14.6-3.ph2

echo 'installing docker' tdnf install -yq docker-18.06.2-6.ph2 systemctl enable docker systemctl start docker while [ systemctl is-active docker != 'active' ]; do echo 'waiting for docker'; sleep 5; done

echo 'installing NFS software' tdnf -y install nfs-utils systemctl stop nfs-server.service systemctl disable nfs-server.service

echo 'upgrading security packages' tdnf update tdnf -y

this update needs to be the last step due to required reboot after kernel update (https://bbs.archlinux.org/viewtopic.php?id=203966)

tdnf should be improved to handle dependent package exclusion better. refer to jira PHO-548

tdnf update --security --exclude "open-vm-tools,xerces-c,procps-ng,docker" -y

/etc/machine-id must be empty so that new machine-id gets assigned on boot (in our case boot is vApp deployment)

echo -n > /etc/machine-id sync sync echo 'customization completed'

For Photon template, update mstr.sh #!/usr/bin/env bash set -e export FTP_PROXY="http://PROXY_IP_ADDRES/" export HTTP_PROXY="http://PROXY_IP_ADDRES/" export HTTPS_PROXY="http://PROXY_IP_ADDRES/" export NO_PROXY="127.0.0.1,$(echo INTERNAL_IP_ADDRESS.{1..255} | sed 's/ /,/g'),INTERNAL_IP_ADDRESS,localhost"

Setup daemon.

cat > /etc/docker/daemon.json <<EOF { "exec-opts": ["native.cgroupdriver=systemd"], "log-driver": "json-file", "log-opts": { "max-size": "100m" }, "storage-driver": "overlay2" } EOF

mkdir /etc/systemd/system/docker.service.d echo '[Service]' >> /etc/systemd/system/docker.service.d/http-proxy.conf echo 'Environment="HTTP_PROXY=http://INTERNAL_IP_ADDRESS"' >> /etc/systemd/system/docker.service.d/http-proxy.conf echo '[Service]' >> /etc/systemd/system/docker.service.d/https-proxy.conf echo 'Environment="HTTPS_PROXY=http://INTERNAL_IP_ADDRESS"' >> /etc/systemd/system/docker.service.d/https-proxy.conf

systemctl daemon-reload systemctl restart docker

while [ systemctl is-active docker != 'active' ]; do echo 'waiting for docker'; sleep 5; done kubeadm init --kubernetes-version=v1.14.6 > /root/kubeadm-init.out mkdir -p /root/.kube cp -f /etc/kubernetes/admin.conf /root/.kube/config chown $(id -u):$(id -g) /root/.kube/config echo 'installing weave' export kubever=$(kubectl version --client | base64 | tr -d '\n') wget --no-verbose -O /root/weave.yml "https://cloud.weave.works/k8s/net?k8s-version=$kubever&v=2.5.2" kubectl apply -f /root/weave.yml systemctl restart kubelet while [ systemctl is-active kubelet != 'active' ]; do echo 'waiting for kubelet'; sleep 5; done

For Photon template, update node.sh #!/usr/bin/env bash set -e

mkdir /etc/systemd/system/docker.service.d echo '[Service]' >> /etc/systemd/system/docker.service.d/http-proxy.conf echo 'Environment="HTTP_PROXY=http://INTERNAL_IP_ADDRESS"' >> /etc/systemd/system/docker.service.d/http-proxy.conf echo '[Service]' >> /etc/systemd/system/docker.service.d/https-proxy.conf echo 'Environment="HTTPS_PROXY=http://INTERNAL_IP_ADDRESS"' >> /etc/systemd/system/docker.service.d/https-proxy.conf

systemctl daemon-reload systemctl restart docker

while [ systemctl is-active docker != 'active' ]; do echo 'waiting for docker'; sleep 5; done kubeadm join --token {token} {ip}:6443 --discovery-token-unsafe-skip-ca-verification