pxf icon indicating copy to clipboard operation
pxf copied to clipboard

Fix pxf home file structure in script

Open yimingli-vmware opened this issue 1 year ago • 0 comments

Now if we fly a pipeline for the multi-cluster branch, the pipeline will go red because the file structure under PXF_HOME has changed. See more details in PR 1089.

  1. This PR updates the related file paths in the scripts:
  • Updated examples in README
  • Renamed TEMPLATES_DIR=$(PXF_HOME)/templates to SERVER_TEMPLATES_DIR="$(PXF_HOME)/templates/servers"
  • Created CONF_TEMPLATES_DIR="${PXF_HOME}/templates/conf" for copying conf templates from PXF_HOME to PXF_BASE
  1. This PR also provides a workaround for the pipeline failure when using PXF_HOME as the PXF_BASE.

In the current configure_pxf_server function, if user uses the default value for PXF_BASE which is PXF_HOME, the script will skip running pxf prepare, and continues to copy pxf-site.xml from ${PXF_HOME}/templates/servers/pxf-site.xml to ${PXF_HOME}/servers/pxf-site.xml. The prepare script will populate conf and server directories under PXF_BASE. Since now the servers folder is nested inside templates in the PXF_HOME and the pxf prepare is skipped, the pipeline will fail.

The workaround is manually setting a PXF_BASE for the tasks which were using PXF_HOME as the default value. We need to revert them after we fix the issue:

  • delete line 77 in test-cli-tpl.yml
  • delete line 84 and line 98 in test-file-tpl.yml
  • delete line 195 and line 216 in test-multinode-tpl.yml
  • revert the change in line 82 and line 84 in test-tpl.yml
  • delete line 41 in test-upgrade-extension-tpl.yml
  1. Now the pipeline is almost green except one task test-pxf-gp6-ext-hdp2-upgrade-extension-centos7. Since it uses pxf 6.6.0 package which has the old file structure, the pipeline fails for No such file or directory.

yimingli-vmware avatar Apr 11 '24 18:04 yimingli-vmware