container-service-extension
container-service-extension copied to clipboard
Do anyone tried "cse install --config config.yaml" without encrypted config.yaml for CSE 2.6.1??
Do anyone have installed CSE 2.6.1 behind proxy. Need to know the URL's required to allow through proxy for CSE 2.6.1 installation, Template and Cluster deployment.
Hi,
CSE 2.6.1 can be installed with an unencrypted config fie, you need to provide the flag -s
to the command cse install
.
I am not sure about your other ask regarding the proxy. Can you please elaborate?
Regards Aritra Sen
Thanks, I will try -s during cse install and will confirm you. I don't have direct internet connectivity to my CSE server, need to use proxy to provide internet connection. Also through proxy only allowed url's can be access. So need the list of URL's or repositories required to be access through internet during CSE installation, template deployment and Cluster creation.
I have replied to Varun, will try to compile the list by early next week,
Hello, This is information regarding using proxy configuration incase direct internet connectivity not available to cse server. edit proxy file on cse server and added proxy details as per your environment. This will provide internet connectivity to cse server File: proxy File location /etc/sysconfig/proxy vi /etc/sysconfig/proxy
Enable a generation of the proxy settings to the profile.
This setting allows to turn the proxy on and off while
preserving the particular proxy setup.
PROXY_ENABLED="yes"
Some programs (e.g. wget) support proxies, if set in
the environment.
Example: HTTP_PROXY="http://proxy.provider.de:3128/"
HTTP_PROXY="http://PROXY_IP_ADDRESS"
Example: HTTPS_PROXY="https://proxy.provider.de:3128/"
HTTPS_PROXY="http://PROXY_IP_ADDRESS"
Example: FTP_PROXY="http://proxy.provider.de:3128/"
FTP_PROXY="http://PROXY_IP_ADDRESS"
Example: GOPHER_PROXY="http://proxy.provider.de:3128/"
GOPHER_PROXY="http://PROXY_IP_ADDRESS"
Example: SOCKS_PROXY="socks://proxy.example.com:8080"
SOCKS_PROXY="http://PROXY_IP_ADDRESS"
Example: SOCKS5_SERVER="office-proxy.example.com:8881"
SOCKS5_SERVER="http://PROXY_IP_ADDRESS"
Example: NO_PROXY="www.me.de, do.main, localhost"
NO_PROXY="localhost, 127.0.0.1, VCD_IP_ADDRESS, INTERNAL_IP_ADDRESS
If Proxy in your environment only allow whitelist URL's, add below list of URL's in proxy whitelist. List no need
https://raw.githubusercontent.com/vmware/container-service-extension-templates/master/template.yaml https://raw.githubusercontent.com/vmware/container-service-extension-templates/master/scripts/ubuntu-16.04_k8-1.17_weave-2.6.0_rev1/* https://raw.githubusercontent.com/vmware/container-service-extension-templates/master/scripts/ubuntu-16.04_k8-1.16_weave-2.6.0_rev1/* https://raw.githubusercontent.com/vmware/container-service-extension-templates/master/scripts/ubuntu-16.04_k8-1.15_weave-2.5.2 _rev3/* https://raw.githubusercontent.com/vmware/container-service-extension-templates/master/scripts/photon-v2_k8-1.14_weave-2.5.2 _rev2/* https://cloud-images.ubuntu.com/releases/xenial/release-20180418/ubuntu-16.04-server-cloudimg-amd64.ova http://dl.bintray.com/vmware/photon/2.0/GA/ova/photon-custom-hw11-2.0-304b817.ova http://archive.ubuntu.com/ubuntu/dists/* http://archive.ubuntu.com/ubuntu/dists/xenial/* http://archive.ubuntu.com/ubuntu/dists/xenial-updates/* http://security.ubuntu.com/ubuntu/dists/xenial-security/* http://archive.ubuntu.com/ubuntu/dists/xenial-backports/* https://download.docker.com/linux/ubuntu/dists/xenial/* https://download.docker.com/linux/ubuntu/dists/xenial/stable/* https://packages.cloud.google.com/apt/dists/kubernetes-xenial/* http://dl.bintray.com http://git.io http://gcr.io https://kubernetes.io https://docker.io http://cloud.weave.works/* https://cloud.weave.works/k8s/* https://cloud.weave.works/k8s/net?k8s-version=1.17.2&v=2.6.0
Now we need to provide proxy configuration for template deployment Once you run cse install command, ~/.cse_scripts/ folders will be created. Under this folder there will be a folder for each kubernetes template (do refer to the “cse-install_date_time.log” during installation for any additional URL) /photon-v2_k8-1.14_weave-2.5.2_rev2 /ubuntu-16.04_k8-1.15_weave-2.5.2_rev3 /ubuntu-16.04_k8-1.16_weave-2.6.0_rev1 /ubuntu-16.04_k8-1.17_weave-2.6.0_rev1
each folder have cust.sh, mstr.sh, node.sh file. Update these 3 files as per below information For Ubuntu template, update cust.sh #!/usr/bin/env bash set -e
disable ipv6 to avoid possible connection errors
echo 'net.ipv6.conf.all.disable_ipv6 = 1' >> /etc/sysctl.conf
echo 'net.ipv6.conf.default.disable_ipv6 = 1' >> /etc/sysctl.conf
echo 'net.ipv6.conf.lo.disable_ipv6 = 1' >> /etc/sysctl.conf
sudo sysctl -p
echo 'nameserver 10.41.67.1' >> /etc/resolvconf/resolv.conf.d/tail
resolvconf -u
export FTP_PROXY="http://PROXY_IP_ADDRES/"
export HTTP_PROXY="http://PROXY_IP_ADDRES/"
export HTTPS_PROXY="http://PROXY_IP_ADDRES/"
echo 'Acquire::http::proxy "http://PROXY_IP_ADDRES";' >> /etc/apt/apt.conf
echo 'Acquire::https::proxy "http://PROXY_IP_ADDRES";' >> /etc/apt/apt.conf
echo 'Acquire::ftp::proxy "http://PROXY_IP_ADDRES";' >> /etc/apt/apt.conf
echo 'https_proxy = http://PROXY_IP_ADDRES/' >> /etc/wgetrc
echo 'http_proxy = http://PROXY_IP_ADDRES/' >> /etc/wgetrc
echo 'ftp_proxy = http://PROXY_IP_ADDRES/' >> /etc/wgetrc
echo 'use_proxy = on' >> /etc/wgetrc
echo 'proxy = http://PROXY_IP_ADDRES' >> ~/.curlrc
systemctl restart networking.service
while [ systemctl is-active networking
!= 'active' ]; do echo 'waiting for network'; sleep 5; done
growpart /dev/sda 1 || : resize2fs /dev/sda1 || :
redundancy: https://github.com/vmware/container-service-extension/issues/432
systemctl restart networking.service
while [ systemctl is-active networking
!= 'active' ]; do echo 'waiting for network'; sleep 5; done
echo 'installing kubernetes'
export DEBIAN_FRONTEND=noninteractive
apt-get -q update -o Acquire::Retries=3 -o Acquire::http::No-Cache=True -o Acquire::http::Timeout=30 -o Acquire::https::No-Cache=True -o Acquire::https::Timeout=30 -o Acquire::ftp::Timeout=30
apt-get -q install -y apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
apt-get -q update -o Acquire::Retries=3 -o Acquire::http::No-Cache=True -o Acquire::http::Timeout=30 -o Acquire::https::No-Cache=True -o Acquire::https::Timeout=30 -o Acquire::ftp::Timeout=30
apt-get -q install -y docker-ce=5:19.03.5~3-0~ubuntu-xenial
apt-get -q install -y kubelet=1.17.2-00 kubeadm=1.17.2-00 kubectl=1.17.2-00 kubernetes-cni=0.7.5-00
systemctl restart docker
while [ systemctl is-active docker
!= 'active' ]; do echo 'waiting for docker'; sleep 5; done
echo 'installing required software for NFS' apt-get -q install -y nfs-common nfs-kernel-server systemctl stop nfs-kernel-server.service systemctl disable nfs-kernel-server.service
prevent updates to software that CSE depends on
apt-mark hold open-vm-tools apt-mark hold docker-ce apt-mark hold kubelet apt-mark hold kubeadm apt-mark hold kubectl apt-mark hold kubernetes-cni apt-mark hold nfs-common apt-mark hold nfs-kernel-server
echo 'upgrading the system' apt-get -q update -o Acquire::Retries=3 -o Acquire::http::No-Cache=True -o Acquire::http::Timeout=30 -o Acquire::https::No-Cache=True -o Acquire::https::Timeout=30 -o Acquire::ftp::Timeout=30 apt-get -y -q -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" dist-upgrade
/etc/machine-id must be empty so that new machine-id gets assigned on boot (in our case boot is vApp deployment)
https://jaylacroix.com/fixing-ubuntu-18-04-virtual-machines-that-fight-over-the-same-ip-address/
truncate -s 0 /etc/machine-id rm /var/lib/dbus/machine-id || : ln -fs /etc/machine-id /var/lib/dbus/machine-id || : # dbus/machine-id is symlink pointing to /etc/machine-id
sync sync echo 'customization completed'
For Ubuntu template, update mstr.sh
#!/usr/bin/env bash set -e export FTP_PROXY="http://PROXY_IP_ADDRES/" export HTTP_PROXY="http://PROXY_IP_ADDRES/" export HTTPS_PROXY="http://PROXY_IP_ADDRES/" export NO_PROXY="127.0.0.1,INTERNAL_IP_ADDRESS" mkdir /etc/systemd/system/docker.service.d echo '[Service]' >> /etc/systemd/system/docker.service.d/http-proxy.conf echo 'Environment="HTTP_PROXY=http://PROXY_IP_ADDRES"' >> /etc/systemd/system/docker.service.d/http-proxy.conf echo '[Service]' >> /etc/systemd/system/docker.service.d/https-proxy.conf echo 'Environment="HTTPS_PROXY=http://PROXY_IP_ADDRES"' >> /etc/systemd/system/docker.service.d/https-proxy.conf systemctl daemon-reload systemctl restart docker
while [ systemctl is-active docker
!= 'active' ]; do echo 'waiting for docker'; sleep 5; done
kubeadm init --kubernetes-version=v1.17.2 > /root/kubeadm-init.out
mkdir -p /root/.kube
cp -f /etc/kubernetes/admin.conf /root/.kube/config
chown $(id -u):$(id -g) /root/.kube/config
export kubever=$(kubectl version --client | base64 | tr -d '\n')
wget --no-verbose -O /root/weave.yml "https://cloud.weave.works/k8s/net?k8s-version=$kubever&v=2.6.0"
kubectl apply -f /root/weave.yml
systemctl restart kubelet
while [ systemctl is-active kubelet
!= 'active' ]; do echo 'waiting for kubelet'; sleep 5; done
For Ubuntu template, update node.sh #!/usr/bin/env bash set -e
mkdir /etc/systemd/system/docker.service.d echo '[Service]' >> /etc/systemd/system/docker.service.d/http-proxy.conf echo 'Environment="HTTP_PROXY=http://PROXY_IP_ADDRES"' >> /etc/systemd/system/docker.service.d/http-proxy.conf echo '[Service]' >> /etc/systemd/system/docker.service.d/https-proxy.conf echo 'Environment="HTTPS_PROXY=http://PROXY_IP_ADDRES"' >> /etc/systemd/system/docker.service.d/https-proxy.conf
systemctl daemon-reload systemctl restart docker
while [ systemctl is-active docker
!= 'active' ]; do echo 'waiting for docker'; sleep 5; done
kubeadm join --token {token} {ip}:6443 --discovery-token-unsafe-skip-ca-verification
For Photon template, update cust.sh #!/usr/bin/env bash
set -e
cat << EOF > /etc/systemd/system/iptables-ports.service [Unit] After=iptables.service Requires=iptables.service [Service] Type=oneshot ExecStartPre=/usr/sbin/iptables -P INPUT ACCEPT ExecStartPre=/usr/sbin/iptables -P OUTPUT ACCEPT ExecStart=/usr/sbin/iptables -P FORWARD ACCEPT TimeoutSec=0 RemainAfterExit=yes [Install] WantedBy=iptables.service EOF export FTP_PROXY="http://PROXY_IP_ADDRES/" export HTTP_PROXY="http://PROXY_IP_ADDRES/" export HTTPS_PROXY="http://PROXY_IP_ADDRES/" echo 'https_proxy = http://PROXY_IP_ADDRES/' >> /etc/wgetrc echo 'http_proxy = http://PROXY_IP_ADDRES/' >> /etc/wgetrc echo 'ftp_proxy = http://PROXY_IP_ADDRES/' >> /etc/wgetrc echo 'use_proxy = on' >> /etc/wgetrc echo 'proxy = http://PROXY_IP_ADDRES' >> ~/.curlrc
chmod 0644 /etc/systemd/system/iptables-ports.service systemctl enable iptables-ports.service systemctl start iptables-ports.service
update repo info (needed for docker update)
tdnf makecache -q tdnf update tdnf -y
echo 'installing kubernetes' tdnf install -yq wget kubernetes-1.14.6-3.ph2 kubernetes-kubeadm-1.14.6-3.ph2
echo 'installing docker'
tdnf install -yq docker-18.06.2-6.ph2
systemctl enable docker
systemctl start docker
while [ systemctl is-active docker
!= 'active' ]; do echo 'waiting for docker'; sleep 5; done
echo 'installing NFS software' tdnf -y install nfs-utils systemctl stop nfs-server.service systemctl disable nfs-server.service
echo 'upgrading security packages' tdnf update tdnf -y
this update needs to be the last step due to required reboot after kernel update (https://bbs.archlinux.org/viewtopic.php?id=203966)
tdnf should be improved to handle dependent package exclusion better. refer to jira PHO-548
tdnf update --security --exclude "open-vm-tools,xerces-c,procps-ng,docker" -y
/etc/machine-id must be empty so that new machine-id gets assigned on boot (in our case boot is vApp deployment)
echo -n > /etc/machine-id sync sync echo 'customization completed'
For Photon template, update mstr.sh #!/usr/bin/env bash set -e export FTP_PROXY="http://PROXY_IP_ADDRES/" export HTTP_PROXY="http://PROXY_IP_ADDRES/" export HTTPS_PROXY="http://PROXY_IP_ADDRES/" export NO_PROXY="127.0.0.1,$(echo INTERNAL_IP_ADDRESS.{1..255} | sed 's/ /,/g'),INTERNAL_IP_ADDRESS,localhost"
Setup daemon.
cat > /etc/docker/daemon.json <<EOF { "exec-opts": ["native.cgroupdriver=systemd"], "log-driver": "json-file", "log-opts": { "max-size": "100m" }, "storage-driver": "overlay2" } EOF
mkdir /etc/systemd/system/docker.service.d echo '[Service]' >> /etc/systemd/system/docker.service.d/http-proxy.conf echo 'Environment="HTTP_PROXY=http://INTERNAL_IP_ADDRESS"' >> /etc/systemd/system/docker.service.d/http-proxy.conf echo '[Service]' >> /etc/systemd/system/docker.service.d/https-proxy.conf echo 'Environment="HTTPS_PROXY=http://INTERNAL_IP_ADDRESS"' >> /etc/systemd/system/docker.service.d/https-proxy.conf
systemctl daemon-reload systemctl restart docker
while [ systemctl is-active docker
!= 'active' ]; do echo 'waiting for docker'; sleep 5; done
kubeadm init --kubernetes-version=v1.14.6 > /root/kubeadm-init.out
mkdir -p /root/.kube
cp -f /etc/kubernetes/admin.conf /root/.kube/config
chown $(id -u):$(id -g) /root/.kube/config
echo 'installing weave'
export kubever=$(kubectl version --client | base64 | tr -d '\n')
wget --no-verbose -O /root/weave.yml "https://cloud.weave.works/k8s/net?k8s-version=$kubever&v=2.5.2"
kubectl apply -f /root/weave.yml
systemctl restart kubelet
while [ systemctl is-active kubelet
!= 'active' ]; do echo 'waiting for kubelet'; sleep 5; done
For Photon template, update node.sh #!/usr/bin/env bash set -e
mkdir /etc/systemd/system/docker.service.d echo '[Service]' >> /etc/systemd/system/docker.service.d/http-proxy.conf echo 'Environment="HTTP_PROXY=http://INTERNAL_IP_ADDRESS"' >> /etc/systemd/system/docker.service.d/http-proxy.conf echo '[Service]' >> /etc/systemd/system/docker.service.d/https-proxy.conf echo 'Environment="HTTPS_PROXY=http://INTERNAL_IP_ADDRESS"' >> /etc/systemd/system/docker.service.d/https-proxy.conf
systemctl daemon-reload systemctl restart docker
while [ systemctl is-active docker
!= 'active' ]; do echo 'waiting for docker'; sleep 5; done
kubeadm join --token {token} {ip}:6443 --discovery-token-unsafe-skip-ca-verification