conflicting dependencies in KFP pip requirements

[thesuperzapper]

As of Elyra 3.8.1, KFP pipelines will only work with Runtime images that use pip versions before 20.3 (the version which added the new resolver).

If using a newer version of pip you will get this error from the pod when trying to execute a Kubeflow pipeline generated by Elyra:

INFO: pip is looking at multiple versions of ipykernel to determine which version is compatible with other requirements. This could take a while.
ERROR: Cannot install nbclient==0.4.1 and nbconvert==6.4.5 because these package versions have conflicting dependencies.
The conflict is caused by:
    The user requested nbclient==0.4.1
    nbconvert 6.4.5 depends on nbclient<0.6.0 and >=0.5.0
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict
ERROR: ResolutionImpossible: for help visit 
https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

While I believe commit https://github.com/elyra-ai/elyra/commit/e29d2eb3858721e1bc824c0f86d3dd142cbe328c will fix this in the next release, I think we should do the following:

1. Release a patch version 3.8.2 (otherwise KFP support is effectively broken in 3.8)
2. Update our unit tests to use a Runtime image with a newer version of pip (so we can detect this kind of issue in the future)

[ptitzler]

@thesuperzapper can you please confirm the Python version that's installed in the problematic runtime image(s)?

We are planning to release version 3.9 later this week. If we manage to stick to the timeline I'm not sure if it's worthwhile the effort to retrofit the fix, considering that users would benefit more from upgrading to that version which includes new features. @akchinSTC?

[akchinSTC]

@thesuperzapper - 3.9.0 has been released which should include changes to fix this issue.

[ptitzler]

Update our unit tests to use a Runtime image with a newer version of pip (so we can detect this kind of issue in the future)

We've added a new Makefile task that users can use to verify that a particular custom runtime image meets the Elyra runtime image requirements for generic nodes. These way certain problems can be surfaced before a user attempts to use the newly created custom image in a pipeline.