wg-build-test-release
wg-build-test-release copied to clipboard
openedx
MySQL 8 Support
Data Working Group: 2022-01-27 Meeting & MySQL Upgrade Path Proposal
openedx/platform-roadmap#119 - It looks like MySQL 8 will be supported for Nutmeg.
According to the most recent update regarding the update, MySQL 8.0 is still a goal for Nutmeg. It seems Nutmeg will be compatible with Mysql 5.7, so it's up to us to decide if we proceed with the update to 8.0.
Hi @jfavellar90!! according to this PR seems that Nutmeg compatibility with MySQL 5.7 and MySQL 8.0 was merged. Do you think that we need something else? Thank you so much
Hi mi amigo @jfavellar90. Do we have any update here?. Please let me know if I can help you with this!!!
Has the mysql support added unicode support in a way that will fix the issue with emojis in titles as mentioned here and here and here, so that people don't need to do anything special to support it out of the box?
Hello @regisb: please let me know if I'm wrong, but is the palm release test instance going to use Mysql 8? Or are there other issues to be resolved first?
I would like Palm (Tutor 16.0.0) to run on MySQL 8.0, so yes I expect that the test instance is going to use that version. I have not yet attempted to upgrade but I was told that Open edX supports MySQL 8.0, so I don't expect any major issue. For the record, edX.org still runs 5.7, but only because upgrading would cause too much downtime.
@regisb how can I upgrade (a copy of) my server to tutor 16.0.0 and Palm so I can test that it's working?
- Work in a temporary project root:
export TUTOR_ROOT=/path/to/some/tmp/file - Install tutor (Olive) from pypi:
pip install tutor - Launch an Olive platform:
tutor local launch - Check that the platform works
- Install tutor v16 (Palm):
pip install git+https://github.com/overhangio/tutor@palm#egg=tutor - Upgrade:
tutor local launch - Check that it works
Thank you @regisb. I got through the first tutor local launch, but when I went to log in, it seemed it didn't take my password anymore. And when I re-set it via the CLI and logged in, I saw that all my classes were gone. So I guess it's just working from a completely default environment?
If I just don't set the tmp TUTOR_ROOT would the pip and launch commands instead do an upgrade in place? Because that's more what I want to test works (and of course I want to test what's working vs. breaking on my existing classes.)
(FWIW after I wrote that I then tried "unset -f TUTOR_ROOT", then "pip install git+https://github.com/overhangio/tutor@palm#egg=tutor", which gave
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
tutor-xqueue 15.0.1 requires tutor<16.0.0,>=15.0.0, but you have tutor 16.0.0 which is incompatible.
tutor-webui 15.0.0 requires tutor<16.0.0,>=15.0.0, but you have tutor 16.0.0 which is incompatible.
tutor-richie 14.0.0 requires tutor<15.0.0,>=14.0.0, but you have tutor 16.0.0 which is incompatible.
tutor-notes 15.0.4 requires tutor<16.0.0,>=15.0.0, but you have tutor 16.0.0 which is incompatible.
tutor-minio 15.1.0 requires tutor<16.0.0,>=15.0.0, but you have tutor 16.0.0 which is incompatible.
tutor-mfe 15.0.6 requires tutor<16.0.0,>=15.0.0, but you have tutor 16.0.0 which is incompatible.
tutor-license 15.0.0 requires tutor<16.0.0,>=15.0.0, but you have tutor 16.0.0 which is incompatible.
tutor-indigo 15.0.0 requires tutor<16.0.0,>=15.0.0, but you have tutor 16.0.0 which is incompatible.
tutor-forum 15.1.1 requires tutor<16.0.0,>=15.0.0, but you have tutor 16.0.0 which is incompatible.
tutor-ecommerce 15.0.2 requires tutor<16.0.0,>=15.0.0, but you have tutor 16.0.0 which is incompatible.
tutor-discovery 15.0.0 requires tutor<16.0.0,>=15.0.0, but you have tutor 16.0.0 which is incompatible.
tutor-android 15.0.0 requires tutor<16.0.0,>=15.0.0, but you have tutor 16.0.0 which is incompatible.
I tried to update the plugins, but got an error:
tutor plugins update
Error applying action: func=<function discover_package.<locals>.load at 0x7feca5dc2830> contexts=['plugins']'
⚠️ Failed to enable plugin 'mfe': (tutor 16.0.0 (/home/ubuntu/.local/lib/python3.10/site-packages), Requirement.parse('tutor<16.0.0,>=15.0.0'))
⚠️ The current environment stored at /home/ubuntu/tmp_tutor/env is not up-to-date: it is at v15.3.6 while the 'tutor' binary is at v16.0.0. You should upgrade the environment by running:
tutor config save
Fetching index https://overhang.io/tutor/main/palm/plugins.yml...
Failed to update index. Request error: HTTP Error 404: Not Found
Plugin index local cache: /home/ubuntu/tmp_tutor/env/plugins/index/cache.yml
and then I tried
tutor local launch
which gave
Error applying action: func=<function discover_package.<locals>.load at 0x7f08dfdbbac0> contexts=['plugins']'
⚠️ Failed to enable plugin 'mfe': (tutor 16.0.0 (/home/ubuntu/.local/lib/python3.10/site-packages), Requirement.parse('tutor<16.0.0,>=15.0.0'))
===============================================
Upgrading from an older release
===============================================
You are about to upgrade your Open edX platform from Olive to Palm
It is strongly recommended to make a backup before upgrading. To do so, run:
tutor local stop # or 'tutor dev stop' in development
sudo rsync -avr "$(tutor config printroot)"/ /tmp/tutor-backup/
In case of problem, to restore your backup you will then have to run: sudo rsync -avr /tmp/tutor-backup/ "$(tutor config printroot)"/
Are you sure you want to continue? [Y/n]
⚠️ This command only performs a partial upgrade of your Open edX platform. To perform a full upgrade, you should run `tutor local launch` (or `tutor dev launch` in development).
⚠️ The current environment stored at /home/ubuntu/tmp_tutor/env is not up-to-date: it is at v15.3.6 while the 'tutor' binary is at v16.0.0. You should upgrade the environment by running:
tutor config save
docker compose -f /home/ubuntu/tmp_tutor/env/local/docker-compose.yml -f /home/ubuntu/tmp_tutor/env/dev/docker-compose.yml --project-name tutor_dev stop
unknown shorthand flag: 'f' in -f
See 'docker --help'.
<cut>
Error: Command failed with status 125: docker compose -f /home/ubuntu/tmp_tutor/env/local/docker-compose.yml -f /home/ubuntu/tmp_tutor/env/dev/docker-compose.yml --project-name tutor_dev stop
But the fact that it is still referencing /home/ubuntu/tmp_tutor, which is what I set the TUTOR_ROOT to before, makes me think the unsetting of the variable was not enough. But if it isn't, then I think this is still meaning there's an error in the upgrade process because it doesn't like the docker -f flag?
If I just don't set the tmp TUTOR_ROOT would the pip and launch commands instead do an upgrade in place? Because that's more what I want to test works (and of course I want to test what's working vs. breaking on my existing classes.)
Yes, this is correct.
But the fact that it is still referencing /home/ubuntu/tmp_tutor, which is what I set the TUTOR_ROOT to before, makes me think the unsetting of the variable was not enough.
Right. I suspect that you did not properly unset the variable. Indeed, unset -f is only for un-setting functions:
$ export POUAC=1
$ unset -f POUAC
$ env | grep POUAC
POUAC=1
$ unset POUAC
$ env | grep POUAC
# nothing, variable was properly unset
You should use unset TUTOR_ROOT (or open a new shell).
EDIT: I followed my own instructions above and they work.
I reverted my VM to a known-good working Olive w/ tutor, version 15.3.5, with TUTOR_ROOT unset (since I broke something elsewhere during testing), I did pip install git+https://github.com/overhangio/tutor@palm#egg=tutor (which seemed to work), and then did tutor local launch which failed with the same docker command.
Are you sure you don't have a typo and it shouldn't be "docker-compose" instead of "docker compose"? Or is that a thing supported by newer versions? (because on the Open edX forums too I noticed you used "docker compose" without the dash.)
Full output below:
ubuntu@ip-172-31-23-17:~$ tutor --version
tutor, version 15.3.5
ubuntu@ip-172-31-23-17:~$ pip install git+https://github.com/overhangio/tutor@palm#egg=tutor
Defaulting to user installation because normal site-packages is not writeable
Collecting tutor
Cloning https://github.com/overhangio/tutor (to revision palm) to /tmp/pip-install-bcsmdm88/tutor_d8252dde6d7448af86c392f82f0ea6bf
Running command git clone --filter=blob:none --quiet https://github.com/overhangio/tutor /tmp/pip-install-bcsmdm88/tutor_d8252dde6d7448af86c392f82f0ea6bf
Running command git checkout -b palm --track origin/palm
Switched to a new branch 'palm'
Branch 'palm' set up to track remote branch 'palm' from 'origin'.
Resolved https://github.com/overhangio/tutor to commit 288f42b5133a44e5b73a17e50aeb5e653e39b28a
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: kubernetes in ./.local/lib/python3.10/site-packages (from tutor) (25.3.0)
Requirement already satisfied: appdirs in ./.local/lib/python3.10/site-packages (from tutor) (1.4.4)
Requirement already satisfied: mypy in ./.local/lib/python3.10/site-packages (from tutor) (0.991)
Requirement already satisfied: typing-extensions>=4.4.0 in ./.local/lib/python3.10/site-packages (from tutor) (4.4.0)
Requirement already satisfied: pyyaml>=6.0 in ./.local/lib/python3.10/site-packages (from tutor) (6.0)
Requirement already satisfied: jinja2>=2.10 in /usr/lib/python3/dist-packages (from tutor) (3.0.3)
Requirement already satisfied: click>=8.0 in /usr/lib/python3/dist-packages (from tutor) (8.0.3)
Requirement already satisfied: pycryptodome in ./.local/lib/python3.10/site-packages (from tutor) (3.16.0)
Requirement already satisfied: urllib3>=1.24.2 in /usr/lib/python3/dist-packages (from kubernetes->tutor) (1.26.5)
Requirement already satisfied: certifi>=14.05.14 in /usr/lib/python3/dist-packages (from kubernetes->tutor) (2020.6.20)
Requirement already satisfied: setuptools>=21.0.0 in /usr/lib/python3/dist-packages (from kubernetes->tutor) (59.6.0)
Requirement already satisfied: requests-oauthlib in ./.local/lib/python3.10/site-packages (from kubernetes->tutor) (1.3.1)
Requirement already satisfied: python-dateutil>=2.5.3 in ./.local/lib/python3.10/site-packages (from kubernetes->tutor) (2.8.2)
Requirement already satisfied: google-auth>=1.0.1 in ./.local/lib/python3.10/site-packages (from kubernetes->tutor) (2.15.0)
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from kubernetes->tutor) (2.25.1)
Requirement already satisfied: websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 in /usr/lib/python3/dist-packages (from kubernetes->tutor) (1.2.3)
Requirement already satisfied: six>=1.9.0 in /usr/lib/python3/dist-packages (from kubernetes->tutor) (1.16.0)
Requirement already satisfied: mypy-extensions>=0.4.3 in ./.local/lib/python3.10/site-packages (from mypy->tutor) (0.4.3)
Requirement already satisfied: tomli>=1.1.0 in ./.local/lib/python3.10/site-packages (from mypy->tutor) (2.0.1)
Requirement already satisfied: rsa<5,>=3.1.4 in ./.local/lib/python3.10/site-packages (from google-auth>=1.0.1->kubernetes->tutor) (4.9)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/lib/python3/dist-packages (from google-auth>=1.0.1->kubernetes->tutor) (0.2.1)
Requirement already satisfied: cachetools<6.0,>=2.0.0 in ./.local/lib/python3.10/site-packages (from google-auth>=1.0.1->kubernetes->tutor) (5.2.0)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/lib/python3/dist-packages (from requests-oauthlib->kubernetes->tutor) (3.2.0)
Requirement already satisfied: pyasn1>=0.1.3 in /usr/lib/python3/dist-packages (from rsa<5,>=3.1.4->google-auth>=1.0.1->kubernetes->tutor) (0.4.8)
Building wheels for collected packages: tutor
Building wheel for tutor (pyproject.toml) ... done
Created wheel for tutor: filename=tutor-16.0.0-py3-none-any.whl size=137431 sha256=b5b8cfcb3cb6e02e4892cef0ee5eb5fee312ad17cfe2cf59611ad3bdfc60e6e7
Stored in directory: /tmp/pip-ephem-wheel-cache-439v7em4/wheels/8d/b0/bf/9f0bbd1e3848e89fb264a5d0141ca7be27af4f30bd92f943a5
Successfully built tutor
Installing collected packages: tutor
Attempting uninstall: tutor
Found existing installation: tutor 15.3.5
Uninstalling tutor-15.3.5:
Successfully uninstalled tutor-15.3.5
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
tutor-xqueue 15.0.1 requires tutor<16.0.0,>=15.0.0, but you have tutor 16.0.0 which is incompatible.
tutor-webui 15.0.0 requires tutor<16.0.0,>=15.0.0, but you have tutor 16.0.0 which is incompatible.
tutor-richie 13.0.2 requires tutor<14.0.0,>=13.0.0, but you have tutor 16.0.0 which is incompatible.
tutor-notes 15.0.4 requires tutor<16.0.0,>=15.0.0, but you have tutor 16.0.0 which is incompatible.
tutor-minio 15.1.0 requires tutor<16.0.0,>=15.0.0, but you have tutor 16.0.0 which is incompatible.
tutor-mfe 15.0.6 requires tutor<16.0.0,>=15.0.0, but you have tutor 16.0.0 which is incompatible.
tutor-license 15.0.0 requires tutor<16.0.0,>=15.0.0, but you have tutor 16.0.0 which is incompatible.
tutor-forum 15.1.1 requires tutor<16.0.0,>=15.0.0, but you have tutor 16.0.0 which is incompatible.
tutor-ecommerce 15.0.2 requires tutor<16.0.0,>=15.0.0, but you have tutor 16.0.0 which is incompatible.
tutor-discovery 15.0.0 requires tutor<16.0.0,>=15.0.0, but you have tutor 16.0.0 which is incompatible.
tutor-android 15.0.0 requires tutor<16.0.0,>=15.0.0, but you have tutor 16.0.0 which is incompatible.
tutor-indigo 15.0.0 requires tutor<16.0.0,>=15.0.0, but you have tutor 16.0.0 which is incompatible.
Successfully installed tutor-16.0.0
ubuntu@ip-172-31-23-17:~$ tutor plugins list
Error applying action: func=<function discover_package.<locals>.load at 0x7f0cf9db5900> contexts=['plugins']'
⚠️ Failed to enable plugin 'forum': (tutor 16.0.0 (/home/ubuntu/.local/lib/python3.10/site-packages), Requirement.parse('tutor<16.0.0,>=15.0.0'))
Error applying action: func=<function discover_package.<locals>.load at 0x7f0cf9db69e0> contexts=['plugins']'
⚠️ Failed to enable plugin 'indigo_mine': (tutor 16.0.0 (/home/ubuntu/.local/lib/python3.10/site-packages), Requirement.parse('tutor<16.0.0,>=15.0.0'))
Error applying action: func=<function discover_package.<locals>.load at 0x7f0cf9db6830> contexts=['plugins']'
⚠️ Failed to enable plugin 'license': (tutor 16.0.0 (/home/ubuntu/.local/lib/python3.10/site-packages), Requirement.parse('tutor<16.0.0,>=15.0.0'))
Error applying action: func=<function discover_package.<locals>.load at 0x7f0cf9db5c60> contexts=['plugins']'
⚠️ Failed to enable plugin 'mfe': (tutor 16.0.0 (/home/ubuntu/.local/lib/python3.10/site-packages), Requirement.parse('tutor<16.0.0,>=15.0.0'))
NAME STATUS VERSION
Bulk_Email_Plugin installed 0.1.0
YTAPI_Plugin installed 0.1.0
android installed 15.0.0
custom_reg_form_plugin installed 0.1.0
disable_course_discovery installed 0.1.0
disable_markdown_safemode installed 0.1.0
discovery installed 15.0.0
ecommerce installed 15.0.2
fix_ACE_notifications installed 0.1.0
fix_email_retry_plugin installed 1.0.0
forum installed 15.1.1
googleanalytics installed 0.1.0
hide_optional_research_fields installed 0.1.0
indigo_mine installed 15.0.0
license installed 15.0.0
mfe installed 15.0.6
minio installed 15.1.0
notes installed 15.0.4
richie installed 13.0.2
set_default_enrollment installed 0.1.0
webui installed 15.0.0
xqueue installed 15.0.1
ubuntu@ip-172-31-23-17:~$ tutor local launch
Error applying action: func=<function discover_package.<locals>.load at 0x7ff78b7a5900> contexts=['plugins']'
⚠️ Failed to enable plugin 'forum': (tutor 16.0.0 (/home/ubuntu/.local/lib/python3.10/site-packages), Requirement.parse('tutor<16.0.0,>=15.0.0'))
Error applying action: func=<function discover_package.<locals>.load at 0x7ff78b7a69e0> contexts=['plugins']'
⚠️ Failed to enable plugin 'indigo_mine': (tutor 16.0.0 (/home/ubuntu/.local/lib/python3.10/site-packages), Requirement.parse('tutor<16.0.0,>=15.0.0'))
Error applying action: func=<function discover_package.<locals>.load at 0x7ff78b7a6830> contexts=['plugins']'
⚠️ Failed to enable plugin 'license': (tutor 16.0.0 (/home/ubuntu/.local/lib/python3.10/site-packages), Requirement.parse('tutor<16.0.0,>=15.0.0'))
Error applying action: func=<function discover_package.<locals>.load at 0x7ff78b7a5c60> contexts=['plugins']'
⚠️ Failed to enable plugin 'mfe': (tutor 16.0.0 (/home/ubuntu/.local/lib/python3.10/site-packages), Requirement.parse('tutor<16.0.0,>=15.0.0'))
===============================================
Upgrading from an older release
===============================================
You are about to upgrade your Open edX platform from Olive to Palm
It is strongly recommended to make a backup before upgrading. To do so, run:
tutor local stop # or 'tutor dev stop' in development
sudo rsync -avr "$(tutor config printroot)"/ /tmp/tutor-backup/
In case of problem, to restore your backup you will then have to run: sudo rsync -avr /tmp/tutor-backup/ "$(tutor config printroot)"/
Are you sure you want to continue? [Y/n]
⚠️ This command only performs a partial upgrade of your Open edX platform. To perform a full upgrade, you should run `tutor local launch` (or `tutor dev launch` in development).
⚠️ The current environment stored at /home/ubuntu/.local/share/tutor/env is not up-to-date: it is at v15.3.5 while the 'tutor' binary is at v16.0.0. You should upgrade the environment by running:
tutor config save
docker compose -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.yml -f /home/ubuntu/.local/share/tutor/env/dev/docker-compose.yml --project-name tutor_dev stop
unknown shorthand flag: 'f' in -f
See 'docker --help'.
Usage: docker [OPTIONS] COMMAND
A self-sufficient runtime for containers
Options:
--config string Location of client config files (default "/home/ubuntu/.docker")
-c, --context string Name of the context to use to connect to the daemon (overrides DOCKER_HOST env var and default context set with "docker context use")
-D, --debug Enable debug mode
-H, --host list Daemon socket(s) to connect to
-l, --log-level string Set the logging level ("debug"|"info"|"warn"|"error"|"fatal") (default "info")
--tls Use TLS; implied by --tlsverify
--tlscacert string Trust certs signed only by this CA (default "/home/ubuntu/.docker/ca.pem")
--tlscert string Path to TLS certificate file (default "/home/ubuntu/.docker/cert.pem")
--tlskey string Path to TLS key file (default "/home/ubuntu/.docker/key.pem")
--tlsverify Use TLS and verify the remote
-v, --version Print version information and quit
Management Commands:
builder Manage builds
config Manage Docker configs
container Manage containers
context Manage contexts
image Manage images
manifest Manage Docker image manifests and manifest lists
network Manage networks
node Manage Swarm nodes
plugin Manage plugins
secret Manage Docker secrets
service Manage services
stack Manage Docker stacks
swarm Manage Swarm
system Manage Docker
trust Manage trust on Docker images
volume Manage volumes
Commands:
attach Attach local standard input, output, and error streams to a running container
build Build an image from a Dockerfile
commit Create a new image from a container's changes
cp Copy files/folders between a container and the local filesystem
create Create a new container
diff Inspect changes to files or directories on a container's filesystem
events Get real time events from the server
exec Run a command in a running container
export Export a container's filesystem as a tar archive
history Show the history of an image
images List images
import Import the contents from a tarball to create a filesystem image
info Display system-wide information
inspect Return low-level information on Docker objects
kill Kill one or more running containers
load Load an image from a tar archive or STDIN
login Log in to a Docker registry
logout Log out from a Docker registry
logs Fetch the logs of a container
pause Pause all processes within one or more containers
port List port mappings or a specific mapping for the container
ps List containers
pull Pull an image or a repository from a registry
push Push an image or a repository to a registry
rename Rename a container
restart Restart one or more containers
rm Remove one or more containers
rmi Remove one or more images
run Run a command in a new container
save Save one or more images to a tar archive (streamed to STDOUT by default)
search Search the Docker Hub for images
start Start one or more stopped containers
stats Display a live stream of container(s) resource usage statistics
stop Stop one or more running containers
tag Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE
top Display the running processes of a container
unpause Unpause all processes within one or more containers
update Update configuration of one or more containers
version Show the Docker version information
wait Block until one or more containers stop, then print their exit codes
Run 'docker COMMAND --help' for more information on a command.
To get more help with docker, check out our guides at https://docs.docker.com/go/guides/
Error applying action: func=<function _stop_on_local_start at 0x7ff78b762a70> contexts=[]'
Error: Command failed with status 125: docker compose -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.yml -f /home/ubuntu/.local/share/tutor/env/dev/docker-compose.yml --project-name tutor_dev stop
docker-compose is unsupported starting from June 2023 (that's today!) and replaced by the Compose plugin: https://docs.docker.com/compose/install/
I suggest you upgrade your installation of Docker and Compose.