Installation Issue List

[Starmys]

1. Add / Remove nodes

   • 5267: Add manual for adding node to existing PAI cluster
     • track potential issues for clusters with multiple master/etcd nodes
   • 5239: Two tasks failed when removing nodes
     • in remove-node.yaml, set gather_facts: yes
   • 5167: Add / Remove Nodes with layout.yaml
     • use ./paictl.py node add -n a b & ./paictl.py node remove -n a b

2. Installation Enhancement

   • 5231: Issues of installation from scatch
     • add AAD, pylon and marketplace related services in quick_start script;
     • hived-computing-device-envs in service-configurations config file has changed from arrays to string
   • 4802: Install openpai on existing k8s cluster
   • 5152: Enhancement in quick-start installation and upgrade 3rd components
   • 4680: Kubespray quick start enhancement

3. 5100: Installation script refinement

   • dev-box can be inside master node
   • P3 all in one deployment: single node cluster support
     • allow master node to be worker at the same time;
   • uninstallation doc :
     • doc for version < 1.0
     • doc for version >= 1.0
   • ns 'pai-storage' already exists: if quick-start-service.sh fails, the ns may have already been created.

[Starmys]

Issue when exchange worker and master (delete and re-deploy): etcd config conflicts

[Starmys]

Plan

1. docker dev box

   1. add into dockerfile

      # basic tools
      apt-get install software-properties-common
      apt-get update
      apt-get install sudo
      # python3.6
      sudo add-apt-repository ppa:deadsnakes/ppa
      sudo apt update
      sudo apt install python3.6
      sudo rm /usr/bin/python3
      sudo ln -s /usr/bin/python3.6 /usr/bin/python3
      sudo pip3 install setuptools
      # ansible, etc.
      The bottom half of pai/contrib/kubespray/script/environment.sh
      

   2. docker run ... -v ~/pai:/root/pai -v ~/pai-deploy:/root/pai-deploy -v ~/.ssh:/root/.ssh

2. integrate the steps of add / remove nodes into ./paictl scale

   1. before ./paictl scale: modify layout.yml and services-configuration.yaml manually
   2. preparation
      1. check if layout.yaml conflicts with services-configuration.yaml
      2. compare the input layout.yml with /cluster_configuration/layout.yml to fetch the nodelist to add / remove
      3. ./paictl config push -p /udpated-config
      4. modify kubespray
         1. remove-node.yaml: gather_facts: yes
         2. roles/remove-node/post-remove/tasks/main.yml: remove run_once: true
   3. add nodes

      # check docker daemon config
      requirement.sh --limit ...
      # raise a notice to change docker daemon config and reload docker daemon manually if necessary
      # add node to k8s cluster
      cd /root/pai-deploy/kubespray
      ansible-playbook -i inventory/pai/hosts.yml scale.yml --become --become-user=root -e "@inventory/gcrv100/openpai.yml" --limit=nodelist
      # update config and restart service
      cd /root/pai/
      ./paictl service stop -n cluster-configuration rest-server hivedscheduler job-exporter
      ./paictl service start -n cluster-configuration rest-server hivedscheduler job-exporter
      

   4. remove nodes

      # remove node from k8s cluster
      cd /root/pai-deploy/kubespray
      ansible-playbook -i inventory/pai/hosts.yml remove-node.yml --become --become-user=root -e "@inventory/gcrv100/openpai.yml" --limit=nodelist
      # update config and restart service
      cd /root/pai/
      ./paictl service stop -n cluster-configuration rest-server hivedscheduler job-exporter
      ./paictl service start -n cluster-configuration rest-server hivedscheduler job-exporter
      

[fanyangCS]

related https://github.com/microsoft/pai/issues/2558

[fanyangCS]

related https://github.com/microsoft/pai/issues/4521