mq-container
mq-container copied to clipboard
ibm-messaging
Apple M1 or ARM64 support
Wanted to know if there was any intentions to make a arm64 compatible container available particularly for M1 Mac users who want to spin up an MQ instance?
At the company I work at, they've started to ship out M1 Macs to new starters and none of them have been able to successfully start/create a queue manager using the in-built amd64 emulation provided by Docker (using qemu). Previously we've been using Intel Macs which are natively running amd64 processor so there's no issues there.
Error 71 creating queue manager: AMQ8101S: IBM MQ error (893) has occurred.
See the below which is reporting the same issue: https://github.com/docker/for-mac/issues/5961
No immediate plans to do this. I think this is request would be better off submitted as a formal idea for MQ: https://ibm.biz/mqideas
There is some precedent for using MQ on ARM for the Raspberry Pi, but that won't directly translate into a container image.
I've faced the same problem with M1 Mac. Our company is heavily relied on the IBM ecosystem. Therefore, we have to find other alternative MQ to run in the development environment.
I've also faced the same problem. The only way to get around this is currently to run a virtual machine with an intel docker image or changing our mac book which is not acceptable. If IBM could just provide a pre built tar for the mq for developers I could build my own image.
Same here. We also started to ship M1 Macs to our new employees. Even in a long run, Apple will only provide M1s and at some point the Intel based Macs will be the minority. So the need for an ARM based image will increase over time.
Question, I have the same issue where IBM MQ does not work in docker, but I can't get past the java virtual machine error that this issue:
does not seem to be getting. Any ideas why?
Log where error occurs:
2021-12-11T11:04:38.901Z Creating queue manager QM1
2021-12-11T11:04:38.901Z Starting web server
2021-12-11T11:04:40.174Z Error 22 starting web server:
Error: Port Library failed to initialize: -1
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
We have this issue too. Please concider a M1 version. (we moved also from intel)
This issue is really interesting for building up a local dev environment.
Kind Regards :-)
Here's a link to the formal idea (RFE) for MQ software to support ARM processors: https://integration-development.ideas.ibm.com/ideas/MESNS-I-369
Adding votes on that idea is the most effective way to signal your support for this feature, which needs to be added to the base MQ software before we can then use it containers.
For those of you wanting to run MQ on M1, I've written a blogpost here with instructions on how to do it with Podman. It will be slower than native due to using x86_64 emulation but it's better than nothing
I need a solution that works with Testcontainers. The emulation for AMD on M2 works without issues for all images i've come across so I don't think its the emulation that breaks but the IBM MQ believes "there are multiple instances running" and shuts down. Can this mechanism be turned off somehow?
CPU architecture: amd64 Linux kernel version: 5.15.49-linuxkit Base image: Red Hat Enterprise Linux 8.5 (Ootpa) Running as user ID 1001 with primary group 0, and supplementary groups 0 Capabilities (bounding set): chown,dac_override,fowner,fsetid,kill,setgid,setuid,setpcap,net_bind_service,net_raw,sys_chroot,mknod,audit_write,setfcap seccomp enforcing mode: filtering Process security attributes: none No volume detected. Persistent messages may be lost You cannot run more than one instance of this program
Sooooooooooo ... is there any intention to support ARM any time soon? I've tried building the image for ARM myself but seems rather complicated. Isn't there somebody out there that did IBMs job for them already? Any known ARM image versions?
BTW the problem also appears when using the S390x architecture:
File docker-compose.yml:
services: ibmMQ: image: ibmcom/mq:9.2.4.0-r1 platform: linux/s390x ports: - "1414:1414" - "9443:9443" environment: MQ_QMGR_NAME: "QM" LICENSE: "accept" MQ_ENABLE_METRICS : "false" MQ_DEV: "true" MQ_ADMIN_NAME: "admin" MQ_ADMIN_PASSWORD: "incentage" volumes: - ./docker-ibm-mq-config.mqsc:/etc/mqm/20-config.mqsc - ./ibmcom-data:/mnt/mqm
Result:
2023-02-28 11:17:03 2023-02-28T10:17:03.421Z CPU architecture: s390x 2023-02-28 11:17:03 2023-02-28T10:17:03.425Z Linux kernel version: 5.15.49-linuxkit 2023-02-28 11:17:03 2023-02-28T10:17:03.426Z Base image: Red Hat Enterprise Linux 8.5 (Ootpa) 2023-02-28 11:17:03 2023-02-28T10:17:03.427Z Running as user ID 1001 with primary group 0, and supplementary groups 0 2023-02-28 11:17:03 2023-02-28T10:17:03.428Z Capabilities (bounding set): chown,dac_override,fowner,fsetid,kill,setgid,setuid,setpcap,net_bind_service,net_raw,sys_chroot,mknod,audit_write,setfcap 2023-02-28 11:17:03 2023-02-28T10:17:03.429Z seccomp enforcing mode: filtering 2023-02-28 11:17:03 2023-02-28T10:17:03.429Z Process security attributes: none 2023-02-28 11:17:03 2023-02-28T10:17:03.429Z Detected 'fuse.grpcfuse' volume mounted to /mnt/mqm 2023-02-28 11:17:03 2023-02-28T10:17:03.732Z You cannot run more than one instance of this program
Multiple instances issue is due to the following function in process.go compiled as part of the container image: https://github.com/ibm-messaging/mq-container/blob/53790e17c14ef39af21d48327eeb535238dcca43/cmd/runmqserver/process.go#L48
It does a simple count of the number of times that runmqdevserver appears in the ps -ef --format cmd command output. However, when the image is run under emulation on M1/M2 Mac machines, the ps listing looks like this:
/usr/bin/qemu-x86_x64 /usr/local/bin/runmqdevserver runmqdevserver
Subsequently it thinks its detected more than 1 instance of runmqdevserver. However, even if you fix this issue (ie. by reading the output line by line) you will still run into this error:
Error 71 creating queue manager: AMQ8101S: IBM MQ error (893) has occurred.
It seems there is some issue creating the queue manager and can be minimally reproduced as per below:
docker run -it --platform linux/amd64 --entrypoint /bin/sh ibmcom/mq:latest
crtmqm -c ABC QM1
The FDC files can be obtained accordingly using docker cp for inspection.
+-----------------------------------------------------------------------------+
| |
| IBM MQ First Failure Symptom Report |
| ========================================= |
| |
| Date/Time :- Fri April 14 2023 05:25:45 UTC |
| UTC Time :- 1681449945.779745 |
| UTC Time Offset :- 0 (UTC) |
| UTC Date/Time :- 2023-04-14 05:25:45.779745 (1681449945779745) |
| Host Name :- d7cee84f65bc |
| Operating System :- Linux 5.15.49-linuxkit |
| OS Details :- Red Hat Enterprise Linux 8.5 (Ootpa) |
| PIDS :- 5724H7251 |
| LVLS :- 9.2.4.0 |
| Product Long Name :- IBM MQ for Linux (x86-64 platform) |
| Vendor :- IBM |
| O/S Registered :- 0 (libmqmcs_r.so) Unzipped |
| Data Path :- /mnt/mqm/data |
| Installation Path :- /opt/mqm |
| Installation Name :- Installation1 (126) |
| License Type :- Developer |
| Probe Id :- HL000001 |
| Application Name :- MQM |
| Component :- hlgStartLog |
| Source filename :- /build/slot1/p920_P/src/lib/logger/amqhlgc0.c |
| Line Number :- 656 |
| Build Date :- Nov 5 2021 |
| Build Level :- p924-L211105.DE |
| Build Type :- IKAP - (Production) |
| Effective UserID :- 1001 (assumed to be mqm) |
| Real UserID :- 1001 (assumed to be mqm) |
| Program Name :- crtmqm |
| Arguments :- -c ABC QM1 |
| Addressing mode :- 64-bit |
| LANG :- en_US.UTF-8 |
| Process :- 67 |
| Process(Thread) :- 67 |
| Thread :- 1 Unknown |
| QueueManager :- QM1 |
| SubpoolName :- QM1 |
| Perf Counter :- 517275518734 |
| UserApp :- FALSE |
| ConnId(2) QM :- 2 |
| Last HQC :- 2.0.0-705408 |
| Last HSHMEMB :- 2.4.0-4478656 |
| Last ObjectName :- |
| Major Errorcode :- xecF_E_UNEXPECTED_RC |
| Minor Errorcode :- hrcE_MQLO_SEMA |
| Probe Type :- MSGAMQ6118 |
| Probe Severity :- 1 |
| Probe Description :- AMQ6118S: An internal IBM MQ error has occurred |
| (40006118) |
| FDCSequenceNumber :- 0 |
| Arith1 :- 545286149 (0x20806805) |
| |
+-----------------------------------------------------------------------------+
@master131 how did you get to the AMQ8101S: IBM MQ error (893) has occurred. error? Were you using an M1 mac and emulating the processes? I'm not seeing MQ Advanced for Developers released for ARM/M1 chips.
https://github.com/ibm-messaging/mq-container/blob/master/docs/building.md#building-a-developer-image
I've captured some error logs from running my patched container.
AMQ119.0.FDC.txt MQERR01.LOG AMQERR01.json.txt
Inside I am seeing some strange logs, but am unfamiliar if they're causing issues.
Q: Should this first file descriptor point to /dev/null? That seems wrong.
File Descriptors
lr-x------ 1 1001 0 64 2023-05-04 20:38:25.143804 /proc/119/fd/0 -> crw-rw-rw- 1 1001 0 64 2023-05-04 20:37:42.463449 /dev/null
l-wx------ 1 1001 0 64 2023-05-04 20:38:25.143804 /proc/119/fd/1 -> pipe:[40812]
l-wx------ 1 1001 0 64 2023-05-04 20:38:25.143804 /proc/119/fd/2 -> pipe:[40812]
lrwx------ 1 1001 0 64 2023-05-04 20:38:25.143804 /proc/119/fd/3 -> -rw-rw-r-- 1 1001 0 64 2023-05-04 20:38:09.404449 /mnt/mqm/data/sockets/@SYSTEM/qmgrlocl/dd34396f3287/Installation1/Trace.shm
lrwx------ 1 1001 0 64 2023-05-04 20:38:25.143804 /proc/119/fd/4 -> -rw-rw-r-- 1 1001 0 64 2023-05-04 20:38:19.847804 /mnt/mqm/data/qmgrs/QM1/master
lrwx------ 1 1001 0 64 2023-05-04 20:38:25.143804 /proc/119/fd/5 -> -rw-rw---- 1 1001 0 64 2023-05-04 20:38:19.822804 /mnt/mqm/data/sockets/QM1/qmgrlocl/dd34396f3287/Enqueue.lck
lr-x------ 1 1001 0 64 2023-05-04 20:38:25.143804 /proc/119/fd/6 -> -rw-rw-r-- 1 1001 0 64 2023-05-04 20:38:19.848804 /mnt/mqm/data/qmgrs/QM1/active
l-wx------ 1 1001 0 64 2023-05-04 20:38:25.143804 /proc/119/fd/7 -> -rw-rw---- 1 1001 0 64 2023-05-04 20:38:25.144804 /mnt/mqm/data/errors/AMQ119.0.FDC
lr-x------ 1 1001 0 64 2023-05-04 20:38:25.143804 /proc/119/fd/8 -> dr-x------ 1 1001 0 64 2023-05-04 20:38:25.143804 /proc/119/fd
| Major Errorcode :- xecF_E_UNEXPECTED_RC |
| Minor Errorcode :- hrcE_UNEXPECTED_ERROR |
| Probe Type :- MSGAMQ6118 |
| Probe Severity :- 1 |
| Probe Description :- AMQ6118S: An internal IBM MQ error has occurred |
| (40006118) |
In https://github.com/ibm-messaging/mq-container/issues/327#issuecomment-499473879 I see that /mnt/mqm (and other directories) have mqm/mqm permissions, but the M1 mac has 1001/root for the user/group. Would those permissions be cause crtmqm to exit with a 71 code?
@master131 how did you get to the
AMQ8101S: IBM MQ error (893) has occurred.error? Were you using an M1 mac and emulating the processes? I'm not seeing MQ Advanced for Developers released for ARM/M1 chips.https://github.com/ibm-messaging/mq-container/blob/master/docs/building.md#building-a-developer-image
Was just running the x86_64 version via docker under emulation on M1 Mac use command earlier (with a fix to the runmqserver).
In #327 (comment) I see that
/mnt/mqm(and other directories) have mqm/mqm permissions, but the M1 mac has 1001/root for the user/group. Would those permissions be cause crtmqm to exit with a 71 code?
The changelog states:
Can now run as a random user, instead of the "mqm" user, which has now been removed. This adds compatability for the Red Hat OpenShift restricted SCC. The default image UID is 1001.
So its working for you with some latest version? Could you please be a little more specific how we can run IBM MQ on M1? I guess you followed this https://github.com/ibm-messaging/mq-container/blob/master/docs/building.md#building-a-developer-image but wich branch/version?
Thanks to everyone working on this!
So its working for you with some latest version?
No, it's not - not unless you resort to workarounds that have already been documented online such as using podman or UTM:
https://medium.com/@thejackyhuang/mq-dev-on-m1-with-utm-8b3c90403ef8 https://community.ibm.com/community/user/integration/blogs/piers-walter/2022/07/28/running-x86-64-containers-such-as-mq-on-podman-on?CommunityKey=183ec850-4947-49c8-9a2e-8e7c7fc46c64
Ah ... sorry just woke up and got exited ^^ Still thanks for working on it :) I don't understand how the x86 emulation can lead to this kind of problems but using a virtual machine somehow works as expected. Something very strange going on. Also nice catch with that ps output issue.
I'm building an x86 image on my intel mac and running it emulated on my M1 mac. I've got https://github.com/ibm-messaging/mq-container/pull/524 to get past the first error, but am seeing AMQ8101S: IBM MQ error (893) has occurred. still.
Is anyone more familiar with MQ errors to know how to work around that? I have full logs posted in https://github.com/ibm-messaging/mq-container/issues/476#issuecomment-1535448737
The building.md documentation in this repo, has been updated to reflect that it's now possible to build your own container for both amd64 and arm64 (Apple Silicon) architectures.
https://github.com/ibm-messaging/mq-container/blob/master/docs/building.md#building-a-developer-image
Works perfectly out-of-the-box. 👏
% docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
29c832bb6c81 ibm-mqadvanced-server-dev:9.3.3.0-arm64 "runmqdevserver" 3 minutes ago Up About a minute 1414/tcp, 9157/tcp, 9443/tcp
hello @master131 @pascaldekloe, how did you managed to run it on an m1 or m2 chip? could you give me a hand? thank you
@pascaldekloe yes, also found this instructions: https://community.ibm.com/community/user/integration/blogs/richard-coppen/2023/06/30/ibm-mq-9330-container-image-now-available-for-appl thanks for your help, it worked for me.