postgresql-embedded How to run this on a buildserver through jenkins (postmaster.pid does not exist)?

Locally, everything works fine, but I need to be able to run this on a buildserver with jenkins. The server is not owned by us, and we do not have access to it, except through jenkins.

The EmbeddedPostgres is created with target/database as dataDir. This works locally.

For jenkins though, I get:

2018-12-28 13:39:32.634  WARN 28546 --- [           main] r.y.q.embed.postgresql.PostgresProcess   : Possibly failed to run initdb: The files belonging to this database system will be owned by user "jenkins".

This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".

The default database encoding has accordingly been set to "UTF8".

The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/jenkins/workspace/BACKEND - Build and archive [...]/target/database ... ok

creating subdirectories ... ok

selecting default max_connections ... 100

selecting default shared_buffers ... 128MB

selecting dynamic shared memory implementation ... posix

creating configuration files ... ok

 running bootstrap script ... ok

 2018-12-28 13:39:52.625 ERROR 28546 --- [           main] r.y.q.embed.postgresql.PostgresProcess   : Failed to read PID file (File '/var/lib/jenkins/workspace/BACKEND - Build and archive DEVELOPMENT-[...]/target/database/postmaster.pid' does not exist)

java.io.FileNotFoundException: File '/var/lib/jenkins/workspace/BACKEND - Build and archive [...]/target/database/postmaster.pid' does not exist

Any ideas how to get this to work?

Dec 28 '18 12:12 ThomasStubbe

Can you show your configuration? How exactly do you set folder?

Dec 31 '18 16:12 AleksanderBrzozowski

Getting this problem as well

Feb 10 '19 22:02 JasonTsao

Similar issue under Jenkins in Windows environment.

Initialization is made using following code (it runs with no problem from console using same build script):

embeddedPostgres = new EmbeddedPostgres (Version.Main.V11);
    // starting Postgres, using cached runtime config to optimize performance subsequent test execution.
    url = embeddedPostgres
      .start (EmbeddedPostgres.cachedRuntimeConfig ("Y:/temp/EmbeddedDatabaseTestRule"), "localhost",
        35441, "test",
        "postgres","test",
        Arrays.asList ("-E", "utf8"), Arrays
          .asList ("-c", "log_directory=Y:/temp/EmbeddedDatabaseTestRule/logs", "-c", "log_connections=on", "-c",
            "log_disconnections=on", "-c", "log_statement=all", "-c", "logging_collector=on", "-c", "log_lock_waits=on",
            "-c", "log_duration=on", "-c", "log_line_prefix=%m [%v]", "-c", "log_rotation_size=102400", "-c",
            "maintenance_work_mem=1GB", "-c",
            "max_parallel_maintenance_workers=" + Runtime.getRuntime ().availableProcessors ()));

Log output

2019-02-26 16:30:09 INFO  [main] (EmbeddedDatabaseConfiguration:EmbeddedDatabaseConfiguration.java:44) - Database configuration: com.jdisc.inventory.common.configuration.DatabaseConfiguration@654c1a54[databaseHost=localhost,databaseUser=postgres,databasePassword=test,databaseEncryptionKey=<null>,databasePort=35441,databaseROPassword=test,databaseROUser=postgresro] 
2019-02-26 16:30:19 INFO  [main] (Executable:Executable.java:109) - start AbstractPostgresConfig{storage=Storage{dbDir=C:\Windows\TEMP\postgresql-embed-bc883ad8-a899-45ec-83bd-3b47b80342f8\db-content-535fce1d-30ac-4865-97fe-fa7a4b6102c5, dbName='test', isTmpDir=true}, network=Net{host='localhost', port=35441}, timeout=Timeout{startupTimeout=15000}, credentials=Credentials{username='postgres', password='test'}, args=[], additionalInitDbParams=[-E, utf8]}
2019-02-26 16:30:21 WARN  [main] (PostgresProcess:PostgresProcess.java:122) - Possibly failed to run initdb:
The files belonging to this database system will be owned by user "SYSTEM".

This user must also own the server process.



The database cluster will be initialized with locale "English_United States.1252".

The default text search configuration will be set to "english".



Data page checksums are disabled.



fixing permissions on existing directory C:/Windows/TEMP/postgresql-embed-bc883ad8-a899-45ec-83bd-3b47b80342f8/db-content-535fce1d-30ac-4865-97fe-fa7a4b6102c5 ... ok

creating subdirectories ... ok

selecting default max_connections ... 100

selecting default shared_buffers ... 128MB

selecting dynamic shared memory implementation ... windows

creating configuration files ... ok

running bootstrap script ... ok


2019-02-26 16:30:48 ERROR [main] (PostgresProcess:PostgresProcess.java:258) - Failed to read PID file (File 'C:\Windows\TEMP\postgresql-embed-bc883ad8-a899-45ec-83bd-3b47b80342f8\db-content-535fce1d-30ac-4865-97fe-fa7a4b6102c5\postmaster.pid' does not exist)
java.io.FileNotFoundException: File 'C:\Windows\TEMP\postgresql-embed-bc883ad8-a899-45ec-83bd-3b47b80342f8\db-content-535fce1d-30ac-4865-97fe-fa7a4b6102c5\postmaster.pid' does not exist
	at org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:292)
	at org.apache.commons.io.FileUtils.readLines(FileUtils.java:1835)
	at org.apache.commons.io.FileUtils.readLines(FileUtils.java:1869)
	at ru.yandex.qatools.embed.postgresql.PostgresProcess.onAfterProcessStart(PostgresProcess.java:256)
	at de.flapdoodle.embed.process.runtime.AbstractProcess.<init>(AbstractProcess.java:116)
	at ru.yandex.qatools.embed.postgresql.AbstractPGProcess.<init>(AbstractPGProcess.java:19)
	at ru.yandex.qatools.embed.postgresql.PostgresProcess.<init>(PostgresProcess.java:68)
	at ru.yandex.qatools.embed.postgresql.PostgresExecutable.start(PostgresExecutable.java:25)
	at ru.yandex.qatools.embed.postgresql.PostgresExecutable.start(PostgresExecutable.java:13)
	at de.flapdoodle.embed.process.runtime.Executable.start(Executable.java:108)
	at ru.yandex.qatools.embed.postgresql.EmbeddedPostgres.start(EmbeddedPostgres.java:194)
	at com.jdisc.EmbeddedDatabaseRule.before(EmbeddedDatabaseRule.java:50)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:46)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:534)
	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1196)
	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1041)
2019-02-26 16:30:51 INFO  [main] (Executable:Executable.java:109) - start AbstractPostgresConfig{storage=Storage{dbDir=C:\Windows\TEMP\postgresql-embed-bc883ad8-a899-45ec-83bd-3b47b80342f8\db-content-535fce1d-30ac-4865-97fe-fa7a4b6102c5, dbName='test', isTmpDir=true}, network=Net{host='localhost', port=35441}, timeout=Timeout{startupTimeout=15000}, credentials=Credentials{username='postgres', password='test'}, args=[test], additionalInitDbParams=[]}
2019-02-26 16:30:55 WARN  [main] (PostgresProcess:PostgresProcess.java:280) - Could not create database first time (0 of 3 trials)
2019-02-26 16:30:57 INFO  [main] (Executable:Executable.java:109) - start AbstractPostgresConfig{storage=Storage{dbDir=C:\Windows\TEMP\postgresql-embed-bc883ad8-a899-45ec-83bd-3b47b80342f8\db-content-535fce1d-30ac-4865-97fe-fa7a4b6102c5, dbName='test', isTmpDir=true}, network=Net{host='localhost', port=35441}, timeout=Timeout{startupTimeout=15000}, credentials=Credentials{username='postgres', password='test'}, args=[test], additionalInitDbParams=[]}
2019-02-26 16:31:01 WARN  [main] (PostgresProcess:PostgresProcess.java:280) - Could not create database first time (1 of 3 trials)
2019-02-26 16:31:04 INFO  [main] (Executable:Executable.java:109) - start AbstractPostgresConfig{storage=Storage{dbDir=C:\Windows\TEMP\postgresql-embed-bc883ad8-a899-45ec-83bd-3b47b80342f8\db-content-535fce1d-30ac-4865-97fe-fa7a4b6102c5, dbName='test', isTmpDir=true}, network=Net{host='localhost', port=35441}, timeout=Timeout{startupTimeout=15000}, credentials=Credentials{username='postgres', password='test'}, args=[test], additionalInitDbParams=[]}
2019-02-26 16:31:08 WARN  [main] (PostgresProcess:PostgresProcess.java:280) - Could not create database first time (2 of 3 trials)
2019-02-26 16:31:10 INFO  [main] (Executable:Executable.java:109) - start AbstractPostgresConfig{storage=Storage{dbDir=C:\Windows\TEMP\postgresql-embed-bc883ad8-a899-45ec-83bd-3b47b80342f8\db-content-535fce1d-30ac-4865-97fe-fa7a4b6102c5, dbName='test', isTmpDir=true}, network=Net{host='localhost', port=35441}, timeout=Timeout{startupTimeout=15000}, credentials=Credentials{username='postgres', password='test'}, args=[test], additionalInitDbParams=[]}
2019-02-26 16:31:15 WARN  [main] (PostgresProcess:PostgresProcess.java:280) - Could not create database first time (3 of 3 trials)
2019-02-26 16:31:15 INFO  [main] (Executable:Executable.java:109) - start AbstractPostgresConfig{storage=Storage{dbDir=C:\Windows\TEMP\postgresql-embed-bc883ad8-a899-45ec-83bd-3b47b80342f8\db-content-535fce1d-30ac-4865-97fe-fa7a4b6102c5, dbName='test', isTmpDir=true}, network=Net{host='localhost', port=35441}, timeout=Timeout{startupTimeout=15000}, credentials=Credentials{username='postgres', password='test'}, args=[], additionalInitDbParams=[-E, utf8]}

It could be an issue of privileged execution, but there is a lack of diagnostics information to confirm.

Feb 26 '19 14:02 gtsarenkov

After several days of debugging it was really privileged execution issue. To solve it I used pg_ctl to start PostgeSQL under Jenkins in Windows node. The example of using PgCtl can be found in ru.yandex.qatools.embed.postgresql.TestPostgresWithPgCtl.

Please note there several issues with this approach:

There was exception inside de.flapdoodle.embed.process-2.0.2.jar when trying to determine PID. I've to add jna.jar and jna-platform.jar to classpath (the project was started with ant script, so dependency handling was not available). Since JDK11 is wrongly treated as "legacy" JVM. (Only JDK 9 will work without jna.jar and jna-platform.jar).
additional parameters for postgres are ignored (only port and host are propagated).
sometimes stop of the postgres left postgres.exe processes running. So, before calling PostgresProcess.stop () method I have to call Command.PgCtl with config.withArgs ("stop");:

      final Command cmd = Command.PgCtl;
      final FixedPath cachedDir = new FixedPath ("Y:/temp/EmbeddedDatabaseTestRule");
      final PostgresStarter<PostgresExecutable, PostgresProcess> runtime = PostgresStarter.getInstance (
        new RuntimeConfigBuilder ().defaults (cmd).artifactStore (
          new PostgresArtifactStoreBuilder ().defaults (cmd).tempDir (cachedDir).download (
            new PostgresDownloadConfigBuilder ().defaultsForCommand (cmd)
              .packageResolver (new PackagePaths (cmd, cachedDir)).build ())).build ());
      final PostgresConfig config;
      config = new PostgresConfig (<databaseEngineVersion>,
          new AbstractPostgresConfig.Net ("localhost",
            <remembered port>),
          new AbstractPostgresConfig.Storage ("test"),
          new AbstractPostgresConfig.Timeout (),
          new AbstractPostgresConfig.Credentials ("postgres",
            "test"), cmd);
        config.withArgs ("stop");
        final PostgresExecutable exec = runtime.prepare (config);
        exec.start ();

There is a delay in shutdown. So, you cannot use fixed port for postgress. Instead I applied SocketUtil.findFreePort (). So, next batch of tests could run without waiting for previous Postgres instance to shutdown.

Mar 04 '19 09:03 gtsarenkov