Starting Docker container pulsar-manager-0.3.0 with internal PostgreSQL does not work

[malaskowski]

Running Pulsar Manager with the default Pulsar Helm Chart settings ends with not working manager instance.

Steps to reproduce

Run the container using the default:

docker pull apachepulsar/pulsar-manager:v0.3.0
docker run -it \
   -p 9527:9527 -p 7750:7750 \
   -e REDIRECT_HOST="http://127.0.0.1" \
   -e REDIRECT_PORT="9527" \
   -e DRIVER_CLASS_NAME="org.postgresql.Driver" \
   -e URL="jdbc:postgresql://127.0.0.1:5432/pulsar_manager" \
   -e USERNAME="pulsar" \
   -e PASSWORD="pulsar" \
   -e LOG_LEVEL="DEBUG" \
   apachepulsar/pulsar-manager:v0.3.0

Result

First, the user is asked for the information by the adduser command from the https://github.com/apache/pulsar-manager/blob/v0.3.0/docker/entrypoint.sh#L27 (looks like fix for this is pretty easy: adduser --disabled-password --gecos "" --ingroup pulsar pulsar). However, later running the /pulsar-manager/startup.sh fails with:

/pulsar-manager/startup.sh: 21: initdb: not found
/pulsar-manager/startup.sh: 22: pg_ctl: not found
createdb: error: could not connect to database template1: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
psql: error: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?

Then in the /pulsar-manager/pulsar-manager.log you can see the connection error (as the DB didn't start):

2023-01-13 10:49:51.463 ERROR 156 --- [pool-2-thread-1] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Exception during pool initialization.

org.postgresql.util.PSQLException: Connection to 127.0.0.1:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.

Finally, the backend never starts properly.

Expected result

The Pulsar Manager creates an internal DB instance and runs properly.

Additional details

I'm running the command above on M1 Apple Silicon processor and Docker Desktop for Mac v4.15.0.

[splio-rbechon]

Hello @malaskowski

It seems that there is an issue in startup script. Database can't be created because binary path is not in the environment variable PATH. Another related issue has been created here You can find a commit that will provide you a way to fix this here

Unfortunately, your container will fail each time during init so you will have to fix this manually until the patch is deployed

Have a nice day Robin

[doug-ba]

@splio-rbechon , the fix was committed in August of 2022. Is there any chance of getting a new container image release? The last release is effectively broken.

[doug-ba]

@tuteng , sorry to ping you directly, but it looks like you push the container images. Is there any chance of getting a new version of the pulsar-manager image with the fix for this pushed?

[tuteng]

@urfreespace Can you help take a look at it? thanks

[bpereto]

@urfreespace @splio-rbechon @nodece can someone of you release (or poke the maintainer) to release a new release with commit https://github.com/apache/pulsar-manager/commit/7c5fd1028c6c1a5a85dec1f1609951a2df64944a included?

thank you!

[Zand3r24]

Just wanted to say I'm also hitting this, and waiting for a new image to be built with the fix.

[tuteng]

streamnative/pulsar-manager:v0.3.1-rc2

Since the official release may take some time, I tried to build an image manually, can you try it? @bpereto @Zand3r24 @doug-ba @malaskowski @splio-rbechon

[tisonkun]

@tuteng Out of curiosity, are you working on v0.3.1 release based on current master branch?

[Aaron-ML]

Hitting this as well, looks like a lot of this stuff is on hold or what?

[malaskowski]

@tuteng I was still prompted on docker run:

Enter the new value, or press ENTER for the default
	Full Name []:
	Room Number []:
	Work Phone []:
	Home Phone []:
	Other []:
Is the information correct? [Y/n] Y

But manager started correctly and works, thank you.

[bpereto]

^

streamnative/pulsar-manager:v0.3.1-rc2

Since the official release may take some time, I tried to build an image manually, can you try it? @bpereto @Zand3r24 @doug-ba @malaskowski @splio-rbechon

keeps crashing:

2023-05-10 15:28:19,418 INFO success: pulsar-manager-backend entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-05-10 15:28:19,418 INFO exited: pulsar-manager-backend (exit status 1; not expected)
2023-05-10 15:28:20,422 INFO spawned: 'pulsar-manager-backend' with pid 1145

[bpereto]

@tisonkun I will test v0.4.0-candidate-2 if you will release a docker image.

[enzo-dechaene]

still have the back-end crashing in the 4.0.0

[bpereto]

I builded the docker image with FROM openjdk:17-slim-bullseye and needed an extra java opts: JAVA_OPTS: "--add-opens java.base/java.lang=ALL-UNNAMED"

with this setup it works fine.

https://github.com/apache/pulsar-manager/issues/509#issuecomment-1544740033

or use an external database ( ex. bitnami postgresql helm chart) for a production deployment, as I have.

[kocaki]

Hi,

I am deploying Pulsar in Kubernetes (Azure) and am facing the same issue. I deploy using Helm. Is there a way to solve this for my deployment? If yes, how?

Thanks in advance!

[malaskowski]

@kocaki Try to run pulsar manager with the following configData:

configData:
      SPRING_CONFIGURATION_FILE: "/pulsar-manager/pulsar-manager/application.properties"
      # Workaround for https://github.com/apache/pulsar-manager/issues/505
      URL: ""

This works with apachepulsar/pulsar-manager:v0.3.0 for me.

[kocaki]

@malaskowski thanks for your answer! It looks like pulsar manager does more now because now I get " username or password is wrong". I try to login with pulsar and pulsar but it does not work.

When I check logs I see:
`Starting PostGreSQL Server
Adding group `pulsar' (GID 1000) ...
Done.
Adding user `pulsar' ...
Adding new user `pulsar' (1000) with group `pulsar' ...
Creating home directory `/home/pulsar' ...
Copying files from `/etc/skel' ...
Changing the user information for pulsar
Enter the new value, or press ENTER for the default
Use of uninitialized value $answer in chop at /usr/sbin/adduser line 582.
Use of uninitialized value $answer in pattern match (m//) at /usr/sbin/adduser line 583.
/pulsar-manager/startup.sh: 21: initdb: not found
/pulsar-manager/startup.sh: 22: pg_ctl: not found
createdb: error: could not connect to database template1: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
psql: error: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
	Full Name []: 	Room Number []: 	Work Phone []: 	Home Phone []: 	Other []: Is the information correct? [Y/n] Starting Pulsar Manager Front end
Starting Pulsar Manager Back end
Start Pulsar Manager by specifying a configuration file.
2023-06-08 07:04:43,465 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
2023-06-08 07:04:43,466 INFO supervisord started with pid 64
2023-06-08 07:04:44,469 INFO spawned: 'pulsar-manager-backend' with pid 66
2023-06-08 07:04:45,562 INFO success: pulsar-manager-backend entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

So there is still an issue somewhere. I also tried deploying 0.4.0. In the logs of that release I do see that the db is created but for some reason also that one does not work. Logs of 0.4.0:

Starting PostGreSQL Server
Adding group `pulsar' (GID 1000) ...
Done.
Adding user `pulsar' ...
Adding new user `pulsar' (1000) with group `pulsar' ...
Creating home directory `/home/pulsar' ...
Copying files from `/etc/skel' ...
Changing the user information for pulsar
Enter the new value, or press ENTER for the default
Use of uninitialized value $answer in chop at /usr/sbin/adduser line 582.
Use of uninitialized value $answer in pattern match (m//) at /usr/sbin/adduser line 583.
pg_ctl: directory "/data/postgresql" does not exist
	Full Name []: 	Room Number []: 	Work Phone []: 	Home Phone []: 	Other []: Is the information correct? [Y/n] The files belonging to this database system will be owned by user "pulsar".
This user must also own the server process.

The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".

Data page checksums are disabled.

creating directory /data/postgresql ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    /usr/lib/postgresql/13/bin/pg_ctl -D /data/postgresql -l logfile start

waiting for server to start....2023-06-07 14:44:08.940 UTC [64] LOG:  starting PostgreSQL 13.11 (Debian 13.11-0+deb11u1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2023-06-07 14:44:08.940 UTC [64] LOG:  listening on IPv6 address "::1", port 5432
2023-06-07 14:44:08.940 UTC [64] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2023-06-07 14:44:08.949 UTC [64] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-06-07 14:44:08.956 UTC [65] LOG:  database system was shut down at 2023-06-07 14:44:08 UTC
2023-06-07 14:44:08.962 UTC [64] LOG:  database system is ready to accept connections
 done
server started
ALTER ROLE
CREATE DATABASE
GRANT
You are now connected to database "pulsar_manager" as user "pulsar".
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
Starting Pulsar Manager Front end
Starting Pulsar Manager Back end
Start servie no enable JWT.
2023-06-07 14:44:09,764 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
2023-06-07 14:44:09,766 INFO supervisord started with pid 92
2023-06-07 14:44:10,769 INFO spawned: 'pulsar-manager-backend' with pid 93
2023-06-07 14:44:11,843 INFO success: pulsar-manager-backend entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-06-07 14:44:11,843 INFO exited: pulsar-manager-backend (exit status 1; not expected)
2023-06-07 14:44:12,847 INFO spawned: 'pulsar-manager-backend' with pid 118
2023-06-07 14:44:13,936 INFO success: pulsar-manager-backend entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-06-07 14:44:13,936 INFO exited: pulsar-manager-backend (exit status 1; not expected)
2023-06-07 14:44:14,939 INFO spawned: 'pulsar-manager-backend' with pid 143
2023-06-07 14:44:16,006 INFO success: pulsar-manager-backend entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-06-07 14:44:16,006 INFO exited: pulsar-manager-backend (exit status 1; not expected)
2023-06-07 14:44:17,009 INFO spawned: 'pulsar-manager-backend' with pid 168
2023-06-07 14:44:18,072 INFO success: pulsar-manager-backend entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-06-07 14:44:18,072 INFO exited: pulsar-manager-backend (exit status 1; not expected)
2023-06-07 14:44:19,075 INFO spawned: 'pulsar-manager-backend' with pid 193
2023-06-07 14:44:20,140 INFO success: pulsar-manager-backend entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-06-07 14:44:20,140 INFO exited: pulsar-manager-backend (exit status 1; not expected)

Do you have any suggestions?

[kocaki]

@malaskowski

Pulsar Manager config:

pulsar_manager:
  component: pulsar-manager
  replicaCount: 1
  # True includes annotation for statefulset that contains hash of corresponding configmap, which will cause pods to restart on configmap change
  restartPodsOnConfigMapChange: false
  # nodeSelector:
  # cloud.google.com/gke-nodepool: default-pool
  annotations: {}
  tolerations: []
  gracePeriod: 30
  resources:
    requests:
      memory: 250Mi
      cpu: 0.1
  configData:
    SPRING_CONFIGURATION_FILE: "/pulsar-manager/pulsar-manager/application.properties"
    REDIRECT_HOST: "http://127.0.0.1"
    REDIRECT_PORT: "9527"
    DRIVER_CLASS_NAME: org.postgresql.Driver
    URL: jdbc:postgresql://127.0.0.1:5432/pulsar_manager
    LOG_LEVEL: DEBUG
    ## If you enabled authentication support
    ## JWT_TOKEN: <token>
    ## SECRET_KEY: data:base64,<secret key>
  ## Pulsar manager service
  ## templates/pulsar-manager-service.yaml
  ##
  service:
    type: LoadBalancer
    port: 9527
    targetPort: 9527
    annotations: {}
  ## Pulsar manager ingress
  ## templates/pulsar-manager-ingress.yaml
  ##
  ingress:
    enabled: false
    annotations: {}
    tls:
      enabled: false

      ## Optional. Leave it blank if your Ingress Controller can provide a default certificate.
      secretName: ""

    hostname: ""
    path: "/"

  ## If set use existing secret with specified name to set pulsar admin credentials.
  existingSecretName:
  admin:
    user: pulsar
    password: pulsar

# These are jobs where job ttl configuration is used
# pulsar-helm-chart/charts/pulsar/templates/pulsar-cluster-initialize.yaml
# pulsar-helm-chart/charts/pulsar/templates/bookkeeper-cluster-initialize.yaml
job:
  ttl:
    enabled: false
    secondsAfterFinished: 3600

[splio-rbechon]

Hello @kocaki

Do you have this in your helm template file that deployed pulsar manager ?

          - name: PULSAR_MANAGER_OPTS
            value: "$(PULSAR_MANAGER_OPTS)

On my side, this variable was the root cause of Supervisord issue ... I just removed it to fix my issue

[kocaki]

Hi @splio-rbechon,

It is existing in the pulsar-manager-deployment.yaml but looks different:

          - name: PULSAR_MANAGER_OPTS
            value: "$(PULSAR_MANAGER_OPTS) -Dlog4j2.formatMsgNoLookups=true"

Tried a new deployment without it but it is still stuck on the same. Logs:

Starting PostGreSQL Server
Adding group `pulsar' (GID 1000) ...
Done.
Adding user `pulsar' ...
Adding new user `pulsar' (1000) with group `pulsar' ...
Creating home directory `/home/pulsar' ...
Copying files from `/etc/skel' ...
Changing the user information for pulsar
Enter the new value, or press ENTER for the default
Use of uninitialized value $answer in chop at /usr/sbin/adduser line 582.
Use of uninitialized value $answer in pattern match (m//) at /usr/sbin/adduser line 583.
/pulsar-manager/startup.sh: 21: initdb: not found
/pulsar-manager/startup.sh: 22: pg_ctl: not found
createdb: error: could not connect to database template1: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
psql: error: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
	Full Name []: 	Room Number []: 	Work Phone []: 	Home Phone []: 	Other []: Is the information correct? [Y/n] Starting Pulsar Manager Front end
Starting Pulsar Manager Back end
Start Pulsar Manager by specifying a configuration file.
2023-06-08 08:59:30,389 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
2023-06-08 08:59:30,390 INFO supervisord started with pid 64
2023-06-08 08:59:31,393 INFO spawned: 'pulsar-manager-backend' with pid 65
2023-06-08 08:59:32,538 INFO success: pulsar-manager-backend entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

So with the suggestion of @malaskowski the backend startup problem has been solved but now there is still a db issue.

[enzo-dechaene]

It finally started yesterday with pulsar 0.4.0 for me, can you try adding this conf :

    PASSWORD: pulsar
    USERNAME: pulsar
    REDIRECT_HOST: your_ingress_url
    REDIRECT_PORT: '80'
    URL: jdbc:postgresql://127.0.0.1:5432/pulsar_manager

[kocaki]

Hi @enzo-dechaene,

Unfortunately also failing with this setup. Logs show me that the pular_manager_backend keeps restarting. Can you share your pulsar manager config so I can compare?

[malaskowski]

@kocaki The default pulsar/pulsar credentials does not work in my project's Pulsar setup.

I followed the instructions for creating user via the backend API:

MANAGER_POD=`kubectl get pods --namespace pulsar --selector component=pulsar-manager --no-headers -o custom-columns=":metadata.name"`
CSRF_TOKEN=$(kubectl exec --namespace pulsar -it $MANAGER_POD -- curl http://localhost:7750/pulsar-manager/csrf-token)
kubectl exec --namespace pulsar -it $MANAGER_POD -- curl \
-H 'X-XSRF-TOKEN: $CSRF_TOKEN' \
-H 'Cookie: XSRF-TOKEN=$CSRF_TOKEN;' \
-H "Content-Type: application/json" \
-X PUT http://localhost:7750/pulsar-manager/users/superuser \
-d '{"name": "admin", "password": "mypassword", "description": "console admin", "email": "[email protected]"}'

This is also fairly easy to automate.

[kocaki]

Hi @malaskowski,

Thanks I will try it. With 0.4.0 I see that the db is being created in the pod but unfortunately the pulsar_manager_backend keeps restarting.

Are you running the db embedded or using an external db and version 0.3.0 or 0.4.0? Because the embedded db will be destroyed once you redeploy the pod right or will it migrate the db to the new pod?

[enzo-dechaene]

Hello, @kocaki,

I'm using db postgresql embedded with pulsar manager 0.4.0.

For the deployment of Pulsar Manager, I only refer to the configmap configuration in the env parameter:

    spec:
      containers:
        - env:
            - name: PULSAR_CLUSTER
              value: pulsar
          envFrom:
            - configMapRef:
                name: pulsar-pulsar-manager
          image: apachepulsar/pulsar-manager:v0.4.0
          imagePullPolicy: Always
          name: pulsar-pulsar-manager
          ports:
            - containerPort: 9527
              name: 9527tcp
              protocol: TCP
          resources:
            requests:
              cpu: 100m
              memory: 250Mi
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /data
              name: pulsar-manager-data
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      terminationGracePeriodSeconds: 30
      volumes:
        - name: pulsar-manager-data
          emptyDir: {}

The config map

  DRIVER_CLASS_NAME: org.postgresql.Driver
  JWT_TOKEN: my_pulsar_admin_token
  LOG_LEVEL: DEBUG
  PASSWORD: pulsar
  USERNAME: pulsar
  REDIRECT_HOST: your_ingress_url
  REDIRECT_PORT: '80'
  SECRET_KEY: my_secret_key
  SPRING_CONFIGURATION_FILE: /pulsar-manager/pulsar-manager/application.properties
  URL: jdbc:postgresql://127.0.0.1:5432/pulsar_manager

Notice that USERNAME and PASSWORD is not the credential for access the interface. It's the username of database.

You can check all parameter on this link

If you want to add user pulsar, you must connect to the pulsar manager pod and run this script

apt update && apt install curl

CSRF_TOKEN=$(curl http://localhost:9527/pulsar-manager/csrf-token)
curl -H "X-XSRF-TOKEN: $CSRF_TOKEN" -H "Cookie: XSRF-TOKEN=$CSRF_TOKEN;" -H 'Content-Type: application/json' -X PUT http://localhost:7750/pulsar-manager/users/superuser -d '{"name": "pulsar", "password": "pulsar", "description": "Administrateur", "email": "[email protected]"}'

[kocaki]

Hi @enzo-dechaene,

Thanks for the info and help. The problem is that the pulsar-manager-backend keeps restarting and we cannot find out why.

The config looks as following:

  configData:
    REDIRECT_HOST: "http://127.0.0.1"
    REDIRECT_PORT: "80"
    DRIVER_CLASS_NAME: org.postgresql.Driver
    URL: jdbc:postgresql://127.0.0.1:5432/pulsar_manager
    LOG_LEVEL: DEBUG
    SPRING_CONFIGURATION_FILE: "/pulsar-manager/pulsar-manager/application.properties"
    #Workaround for https://github.com/apache/pulsar-manager/issues/505
    PASSWORD: pulsar
    USERNAME: pulsar
    LOG_LEVEL: DEBUG

I attached the log of the pulsar manager where you can see that db is being created successfully with 0.4.0 but then it the backend keeps restarting so even if I should create a user I am unable to login.

Our configuration is in Azure AKS. When I check the resources I see there is an internal and external ip assigned and necessary ports are set open automatically. Besides Pulsar Manager also the Proxy pod is restarting with the error it cannot start http server on port 80. No idea if it is related to each other.

If I install version 0.1.0, I can login on the webinterface but receive a lot of errors (500 internal server error). And with 3.0 there is this known issue regarding the db which this topic is about.

Tried also an external PostgreSql db but looks like it is not being used. DB stays empty and no errors in log.

pulsar_manager_1.txt

Update:

I just tried the command to create a user with v0.3.0 (with fix suggested by @malaskowski) and despite the errors about the db in the log, I am now able to login in Pulsar Manager. I will test functionality and troubleshoot the proxy issue but that is finally some progress... thanks both!

[jesperbagge]

Just wanted to chip in with what I came out with after a bit of a rabbit chase and how I got things to not repeatledy restart.

On version 0.4.0 (but some of the errrors are probably there in 0.3.0 as well) I found out that entrypoint.sh checks if the ENV variable SPRING_CONFIGURATION_FILE is configured. If it is, it tries to start supervisord with a reference to this config. However, the docker/supervisord-configuration-file.conf is configured to look for the config file in another ENV variable. Namely ENV_SPRING_CONFIGURATION_FILE :(

So both those ENV:s has to point to the same config file.

Fixing that, the backend still restarts since the helm chart for the pulsar manager deployment creates an ENV named PULSAR_MANAGER_OPTS that references itself. Also found by @kocaki here.

So that ENV also has to be fixed.

My final version of the values.yaml file looks like this:

pulsar_manager:
  configData:
    ENV_SPRING_CONFIGURATION_FILE: "/pulsar-manager/pulsar-manager/application.properties"
    SPRING_CONFIGURATION_FILE: "/pulsar-manager/pulsar-manager/application.properties"
    PULSAR_MANAGER_OPTS: " -Dlog4j2.formatMsgNoLookups=true"

With those three ENV:s configured I managed to get the backend into a running state but still had to manually create a user as described by @malaskowski here.

[htbmw]

Also stumbled upon the issue with the postgres not starting up in 0.3.0 as well as the restarting backend issue. Updated my values to specify 0.4.0 and that solved the postgres issue.

Then it took a bit further reading here and after I had changed the PULSAR_MANAGER_OPTS to remove the self-referenced value, the backend was working.

Then I also discovered here that the pulsar:pulsar credentials is not for the frontend and that you have to generate that yourself using the API.

Finally I can use pulsar-manager after two late nights :-)

[jak78]

Hi! I reproduced the problem and @jesperbagge suggestion works also fine for me. I just submitted a PR to fix the boot issue.

[enzo-dechaene]

I think the chart deployment on the pulsar/pulsar repository needs to be updated as well ?