embedded-postgres issue in Apple Silicon (M1)

[Vivek-M-08]

The project i am working on is using the following dependency:

<dependency>
    <groupId>io.zonky.test</groupId>
    <artifactId>embedded-postgres</artifactId>
    <version>1.2.6</version>
    <scope>test</scope>
</dependency>

But while executing the test cases i am getting the following errors
Error: Missing postgres binaries

Exception encountered when invoking run on a nested suite - Missing postgres binaries
java.lang.IllegalStateException: Missing postgres binaries

Attempted Solution: Dependency Upgrade

To resolve the missing binaries issue, an attempt was made to upgrade the embedded-postgres dependency to the latest version:

<dependency>
    <groupId>io.zonky.test</groupId>
    <artifactId>embedded-postgres</artifactId>
    <version>2.0.4</version>
    <scope>test</scope>
</dependency>

New Error: Process failed during initdb

After upgrading the dependency, a new error is encountered:

Exception encountered when invoking run on a nested suite - Process [/var/folders/f7/kt5z7zpj61v5b757h_wxcyh80000gn/T/embedded-pg/PG-95baeb96ab6751ea71c17b280cad95e9/bin/initdb, -A, trust, -U, postgres, -D, /var/folders/f7/kt5z7zpj61v5b757h_wxcyh80000gn/T/epg6492089817289658487, -E, UTF-8] failed
java.lang.IllegalStateException: Process [/var/folders/f7/kt5z7zpj61v5b757h_wxcyh80000gn/T/embedded-pg/PG-95baeb96ab6751ea71c17b280cad95e9/bin/initdb, -A, trust, -U, postgres, -D, /var/folders/f7/kt5z7zpj61v5b757h_wxcyh80000gn/T/epg6492089817289658487, -E, UTF-8] failed

Please help to resolve this issue.

[tomix26]

Try to add this dependency: https://mvnrepository.com/artifact/io.zonky.test.postgres/embedded-postgres-binaries-darwin-arm64v8

[Vivek-M-08]

@tomix26 The issue still persist even after adding the above mentioned dependency.

Exception encountered when invoking run on a nested suite - Process [/var/folders/f7/kt5z7zpj61v5b757h_wxcyh80000gn/T/embedded-pg/PG-16f255bae154432086b16236cfb59f2e/bin/initdb, -A, trust, -U, postgres, -D, /var/folders/f7/kt5z7zpj61v5b757h_wxcyh80000gn/T/epg16314842684659684511, -E, UTF-8] failed
java.lang.IllegalStateException: Process [/var/folders/f7/kt5z7zpj61v5b757h_wxcyh80000gn/T/embedded-pg/PG-16f255bae154432086b16236cfb59f2e/bin/initdb, -A, trust, -U, postgres, -D, /var/folders/f7/kt5z7zpj61v5b757h_wxcyh80000gn/T/epg16314842684659684511, -E, UTF-8] failed
	at io.zonky.test.db.postgres.embedded.EmbeddedPostgres.system(EmbeddedPostgres.java:633)
	at io.zonky.test.db.postgres.embedded.EmbeddedPostgres.initdb(EmbeddedPostgres.java:250)
	at io.zonky.test.db.postgres.embedded.EmbeddedPostgres.<init>(EmbeddedPostgres.java:157)
	at io.zonky.test.db.postgres.embedded.EmbeddedPostgres$Builder.start(EmbeddedPostgres.java:584)
	at org.ekstep.analytics.util.EmbeddedPostgresql$.start(EmbeddedPostgresql.scala:15)
	at org.ekstep.analytics.exhaust.TestOnDemandDruidExhaustJob.beforeAll(TestOnDemandDruidExhaustJob.scala:49)
	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
	at org.ekstep.analytics.util.BaseSpec.run(BaseSpec.scala:8)
	at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:45)
	at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13(Runner.scala:1346)
	at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13$adapted(Runner.scala:1340)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1340)
	at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24(Runner.scala:1031)
	at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24$adapted(Runner.scala:1010)
	at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1506)
	at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1010)
	at org.scalatest.tools.Runner$.run(Runner.scala:850)
	at org.scalatest.tools.Runner.run(Runner.scala)
	at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2or3(ScalaTestRunner.java:43)
	at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:26)

[Vivek-M-08]

@tomix26 I can see an error in this file.

[tomix26]

It seems to me like an issues with a postgres driver. What version of the driver are you using?

Nevertheless, first I would recommend to try the previous working version of embedded postgres 1.2.6 together with the embedded-postgres-binaries-darwin-arm64v8 dependency. After that step you can try to upgrade to a newer version of embedded postgres. There is a changelog here: https://github.com/zonkyio/embedded-postgres/releases/tag/v2.0.0

If you encounter any other problems, please send me the full log, not images, thank you.

[Vivek-M-08]

Hi @tomix26

I have adding both the dependency as suggested.

<dependency>
	<groupId>io.zonky.test</groupId>
	<artifactId>embedded-postgres</artifactId>
	<version>2.0.4</version>
	<scope>test</scope>
</dependency>
<dependency>
	<groupId>io.zonky.test.postgres</groupId>
	<artifactId>embedded-postgres-binaries-darwin-arm64v8</artifactId>
	<version>15.3.0</version>
</dependency>

However, I continue to see the same error; please view the following error log.

Exception encountered when invoking run on a nested suite - Process [/var/folders/f7/kt5z7zpj61v5b757h_wxcyh80000gn/T/embedded-pg/PG-16f255bae154432086b16236cfb59f2e/bin/initdb, -A, trust, -U, postgres, -D, /var/folders/f7/kt5z7zpj61v5b757h_wxcyh80000gn/T/epg2326912685824662426, -E, UTF-8] failed
java.lang.IllegalStateException: Process [/var/folders/f7/kt5z7zpj61v5b757h_wxcyh80000gn/T/embedded-pg/PG-16f255bae154432086b16236cfb59f2e/bin/initdb, -A, trust, -U, postgres, -D, /var/folders/f7/kt5z7zpj61v5b757h_wxcyh80000gn/T/epg2326912685824662426, -E, UTF-8] failed
	at io.zonky.test.db.postgres.embedded.EmbeddedPostgres.system(EmbeddedPostgres.java:633)
	at io.zonky.test.db.postgres.embedded.EmbeddedPostgres.initdb(EmbeddedPostgres.java:250)
	at io.zonky.test.db.postgres.embedded.EmbeddedPostgres.<init>(EmbeddedPostgres.java:157)
	at io.zonky.test.db.postgres.embedded.EmbeddedPostgres$Builder.start(EmbeddedPostgres.java:584)
	at org.ekstep.analytics.util.EmbeddedPostgresql$.start(EmbeddedPostgresql.scala:15)
	at org.ekstep.analytics.exhaust.TestOnDemandDruidExhaustJob.beforeAll(TestOnDemandDruidExhaustJob.scala:49)
	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
	at org.ekstep.analytics.util.BaseSpec.run(BaseSpec.scala:8)
	at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:45)
	at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13(Runner.scala:1346)
	at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13$adapted(Runner.scala:1340)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1340)
	at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24(Runner.scala:1031)
	at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24$adapted(Runner.scala:1010)
	at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1506)
	at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1010)
	at org.scalatest.tools.Runner$.run(Runner.scala:850)
	at org.scalatest.tools.Runner.run(Runner.scala)
	at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2or3(ScalaTestRunner.java:43)
	at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:26)

By visiting this link, #78, I too attempted to comprehend the problem. appending the command line output for a couple commands for the directory on my local that is causing the problem.

user@PWA177 ~ % cd /private/var/folders/f7/kt5z7zpj61v5b757h_wxcyh80000gn/T/embedded-pg/PG-16f255bae154432086b16236cfb59f2e/bin
user@PWA177 bin % ls
initdb		pg_ctl		postgres
user@PWA177 bin % ./postgres
postgres does not know where to find the server configuration file.
You must specify the --config-file or -D invocation option or set the PGDATA environment variable.
user@PWA177 bin % file postgres
postgres: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64]
postgres (for architecture x86_64):	Mach-O 64-bit executable x86_64
postgres (for architecture arm64):	Mach-O 64-bit executable arm64
user@PWA177 bin % uname -a
Darwin PWA177.local 22.5.0 Darwin Kernel Version 22.5.0: Thu Jun  8 22:22:19 PDT 2023; root:xnu-8796.121.3~7/RELEASE_ARM64_T8103 arm64
user@PWA177 bin % ./initdb -V
initdb (PostgreSQL) 15.3
user@PWA177 bin % ./pg_ctl -V
pg_ctl (PostgreSQL) 15.3

[tomix26]

Have you tried embedded-postgres 1.2.6 ?

[Vivek-M-08]

Yes i have tried embedded-postgres 1.2.6 with embedded-postgres-binaries-darwin-arm64v8 dependency and got the same error as mentioned above.

But when i try embedded-postgres 1.2.6 dependency alone. I get the following error

Exception encountered when invoking run on a nested suite - Missing postgres binaries
java.lang.IllegalStateException: Missing postgres binaries

[tomix26]

Ok, great, good to know.

Try to run the binaries manually with the following command in the /private/var/folders/f7/kt5z7zpj61v5b757h_wxcyh80000gn/T/embedded-pg/PG-16f255bae154432086b16236cfb59f2e/bin directory:
pg_ctl -D ./data start

The output should give you more information about what is wrong and why the database is not starting.

[Vivek-M-08]

I tried to execute the above mentioned command in the specified directory, it says that command not found.

user@PWA177 ~ % cd /private/var/folders/f7/kt5z7zpj61v5b757h_wxcyh80000gn/T/embedded-pg/PG-16f255bae154432086b16236cfb59f2e/bin

user@PWA177 bin % ls
initdb		pg_ctl		postgres

user@PWA177 bin % pg_ctl -D ./data start
zsh: command not found: pg_ctl

As all the three files[initdb, pg_ctl, postgres] inside this directory are Unix Executable Files. I tried with different commands.

user@PWA177 bin %  ./pg_ctl -V       
pg_ctl (PostgreSQL) 15.3

user@PWA177 bin %  ./pg_ctl start    
pg_ctl: no database directory specified and environment variable PGDATA unset
Try "pg_ctl --help" for more information.

user@PWA177 bin % ./pg_ctl -D ./data start
pg_ctl: directory "data" does not exist

[tomix26]

What about mkdir ./data && ./pg_ctl -D ./data start ?

[Vivek-M-08]

Should i add any config related to database cluster in the data directory

user@PWA177 bin % mkdir ./data && ./pg_ctl -D ./data start
pg_ctl: directory "data" is not a database cluster directory

user@PWA177 bin % ls                                      
data		initdb		pg_ctl		postgres

[tomix26]

Yes, you're right. It's been a while since I last tried it. The following command should work:
mkdir ./data && ./bin/initdb -D ./data && ./bin/pg_ctl -D ./data start

[Vivek-M-08]

Both commands were successful, and I attached the console logs as well. what can be the next steps?

user@PWA177 bin % ./initdb -D ./data 
The files belonging to this database system will be owned by user "user".
This user must also own the server process.

The database cluster will be initialized with this locale configuration:
  provider:    libc
  LC_COLLATE:  C
  LC_CTYPE:    UTF-8
  LC_MESSAGES: C
  LC_MONETARY: C
  LC_NUMERIC:  C
  LC_TIME:     C
The default database encoding has accordingly been set to "UTF8".
initdb: could not find suitable text search configuration for locale "UTF-8"
The default text search configuration will be set to "simple".

Data page checksums are disabled.

fixing permissions on existing directory data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Asia/Kolkata
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    pg_ctl -D ./data -l logfile start

user@PWA177 bin % ./pg_ctl -D ./data start 
waiting for server to start....2023-07-12 22:37:21.341 IST [11590] LOG:  starting PostgreSQL 15.3 on x86_64-apple-darwin20.6.0, compiled by Apple clang version 12.0.0 (clang-1200.0.32.29), 64-bit
2023-07-12 22:37:21.342 IST [11590] LOG:  listening on IPv6 address "::1", port 5432
2023-07-12 22:37:21.342 IST [11590] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2023-07-12 22:37:21.343 IST [11590] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2023-07-12 22:37:21.345 IST [11593] LOG:  database system was shut down at 2023-07-12 22:35:28 IST
2023-07-12 22:37:21.347 IST [11590] LOG:  database system is ready to accept connections
 done
server started

[tomix26]

Ok, so the database is starting fine. Now run your tests again and send me the full log. If there are any potentially sensitive information in the log you can send it to my private email: tomix26@gmail.com