pgloader
pgloader copied to clipboard
dimitri
FATAL error: :UTF-8 stream decoding error on #<SB-SYS:FD-STREAM
In Ubuntu 18.04.02 Desktop I'm trying to load a sqlite database into postgres-11:
(base) marco@marco-U36SG:~/turnserver-4.5.0.8/examples/var/db$ pgloader --debug --verbose
/home/marco/turnserver-4.5.0.8/examples/scripts/longtermsecuredb/turndb ./turndb
postgresql:///new_pg_appdb
sb-impl::*default-external-format* :UTF-8
tmpdir: #P"/tmp/pgloader/"
KABOOM!
FATAL error: :UTF-8 stream decoding error on #<SB-SYS:FD-STREAM for "file /home/marco
/turnserver-4.5.0.8/examples/var/db/turndb" {10056ED2D3}>: the octet sequence #(213 0) cannot
be decoded.
Date/time: 2020-01-14-09:29An unhandled error condition has been signalled: :UTF-8 stream
decoding error on #<SB-SYS:FD-STREAM for "file /home/marco/turnserver-4.5.0.8/example
/var/db/turndb" {10056ED2D3}>: the octet sequence #(213 0) cannot be decoded.
Backtrace for: #<SB-THREAD:THREAD "main thread" RUNNING {10006305B3}>
0: (TRIVIAL-BACKTRACE:PRINT-BACKTRACE-TO-STREAM #<SB-IMPL::STRING-OUTPUT-
STREAM {10056F4253}>)
1: (TRIVIAL-BACKTRACE:PRINT-BACKTRACE #<SB-INT:STREAM-DECODING-ERROR
{10056F40B3}> :OUTPUT NIL :IF-EXISTS :APPEND :VERBOSE NIL)
2: ((FLET "H0" :IN PGLOADER::MAIN) #<SB-INT:STREAM-DECODING-ERROR
{10056F40B3}>)
3: (SB-KERNEL::%SIGNAL #<SB-INT:STREAM-DECODING-ERROR {10056F40B3}>)
4: (ERROR #<SB-INT:STREAM-DECODING-ERROR {10056F40B3}>)
5: (SB-KERNEL:WITH-SIMPLE-CONDITION-RESTARTS ERROR NIL SB-INT:STREAM-
DECODING-ERROR :EXTERNAL-FORMAT :UTF-8 :STREAM #<SB-SYS:FD-STREAM for "file /
home/marco/turnserver-4.5.0.8/examples/var/db/turndb" {10056ED2D3}> :OCTETS #(213 0))
6: (SB-IMPL::STREAM-DECODING-ERROR-AND-HANDLE #<SB-SYS:FD-STREAM for "file
/home/marco/turnserver-4.5.0.8/examples/var/db/turndb" {10056ED2D3}> 2)
7: (SB-IMPL::FD-STREAM-READ-N-CHARACTERS/UTF-8 #<SB-SYS:FD-STREAM for "file
/home/marco/turnserver-4.5.0.8/examples/var/db/turndb" {10056ED2D3}> #<(SIMPLE-ARRAY
CHARACTER (512)) SQLite format 3@ ... {10056EF4BF}> 4 508 NIL)
8: (SB-INT:FAST-READ-CHAR-REFILL #<SB-SYS:FD-STREAM for "file /home/marco
/turnserver-4.5.0.8/examples/var/db/turndb" {10056ED2D3}> NIL)
9: (SB-IMPL::ANSI-STREAM-READ-STRING-FROM-FRC-BUFFER #<(SIMPLE-ARRAY
CHARACTER (4096)) SQLite format 3@ ... {10056F000F}> #<SB-SYS:FD-STREAM for
"file /home/marco/turnserver-4.5.0.8/examples/var/db/turndb" {10056ED2D3}> 0 NIL)
10: (SB-IMPL:ANSI-STREAM-READ-SEQUENCE #<(SIMPLE-ARRAY CHARACTER (4096))
SQLite format 3@ ... {10056F000F}> #<SB-SYS:FD-STREAM for "file /home/marco
/turnserver-4.5.0.8/examples/var/db/turndb" {10056ED2D3}> 0 NIL)
11: (READ-SEQUENCE #<(SIMPLE-ARRAY CHARACTER (4096)) SQLite format 3@ ...
{10056F000F}> #<SB-SYS:FD-STREAM for "file /home/marco/turnserver-4.5.0.8/examples
/var/db/turndb" {10056ED2D3}> :START 0 :END NIL)
12: (ALEXANDRIA.0.DEV:READ-STREAM-CONTENT-INTO-STRING #<SB-SYS:FD-STREAM
for "file /home/marco/turnserver-4.5.0.8/examples/var/db/turndb" {10056ED2D3}> :BUFFER-SIZE
4096)
13: (ALEXANDRIA.0.DEV:READ-FILE-INTO-STRING #P"/home/marco/turnserver-4.5.0.8
/examples/var/db/turndb" :BUFFER-SIZE 4096 :EXTERNAL-FORMAT NIL)
14: (PGLOADER.PARSER:PARSE-COMMANDS-FROM-FILE #P"/home/marco/turnserver-
4.5.0.8/examples/var/db/turndb")
15: (PGLOADER:RUN-COMMANDS #P"/home/marco/turnserver-4.5.0.8/examples/var/db
/turndb" :START-LOGGER NIL :FLUSH-SUMMARY T :SUMMARY NIL :LOG-FILENAME NIL
:LOG-MIN-MESSAGES NIL :CLIENT-MIN-MESSAGES NIL)
16: (PGLOADER::PROCESS-COMMAND-FILE ("/home/marco/turnserver-4.5.0.8/examples
/scripts/longtermsecuredb/turndb" "./turndb" "postgresql:///new_pg_appdb") :FLUSH-SUMMARY
T)
17: (PGLOADER.MONITOR::CALL-WITH-MONITOR #<CLOSURE (LAMBDA NIL :IN
PGLOADER::MAIN) {10055070CB}>)
18: (PGLOADER::MAIN ("pgloader" "--debug" "--verbose" "/home/marco/turnserver-4.5.0.8
/examples/scripts/longtermsecuredb/turndb" "./turndb" "postgresql:///new_pg_appdb"))
19: ((LAMBDA NIL :IN "/build/pgloader-Mx2SY9/pgloader-3.5.2/dumper-2SKVI5f7.lisp"))
20: ((FLET SB-UNIX::BODY :IN SAVE-LISP-AND-DIE))
21: ((FLET "WITHOUT-INTERRUPTS-BODY-36" :IN SAVE-LISP-AND-DIE))
22: ((LABELS SB-IMPL::RESTART-LISP :IN SAVE-LISP-AND-DIE))
2020-01-14T08:29:16.010000Z NOTICE Starting pgloader, log system is ready.
2020-01-14T08:29:16.018000Z INFO Starting monitor
2020-01-14T08:29:16.022000Z INFO Stopping monitor
What I am doing here?
:UTF-8 stream decoding error on
#<SB-SYS:FD-STREAM for "file /home/marco/turnserver-4.5.0.8/examples/var/db/turndb"
{10056ED2D3}>:
the octet sequence #(213 0) cannot be decoded.
The strange thing is that doing this:
(base) marco@marco-U36SG:~/turnserver-4.5.0.8/examples/scripts/longtermsecure$ psql -h
127.0.0.1 -U postgres
Password for user postgres:
psql (11.6 (Ubuntu 11.6-1.pgdg18.04+1))
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression:
off)
Type "help" for help.
postgres=# create database turndb owner turn;
CREATE DATABASE
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
turndb | turn | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
(4 rows)
(base) marco@marco-U36SG:~/turnserver-4.5.0.8/examples/var/db$ ls -lah
total 32K
drwxr-xr-x 2 marco marco 4,0K gen 13 18:14 .
drwxr-xr-x 3 marco marco 4,0K gen 13 15:39 ..
-rw-r--r-- 1 marco marco 22K nov 24 2018 turndb
(base) marco@marco-U36SG:~/turnserver-4.5.0.8/examples/var/db$ pgloader --debug --verbose
./turndb postgres:///turndb
sb-impl::*default-external-format* :UTF-8
tmpdir: #P"/tmp/pgloader/"
2020-01-14T09:19:14.017000Z NOTICE Starting pgloader, log system is ready.
2020-01-14T09:19:14.028000Z INFO Starting monitor
2020-01-14T09:19:14.031000Z INFO Stopping monitor
seems working fine.
But the database remains empty, without tables:
(base) marco@marco-U36SG:~/turnserver-4.5.0.8/examples/var/db$ psql -h 127.0.0.1 -U postgres
Password for user postgres:
psql (11.6 (Ubuntu 11.6-1.pgdg18.04+1))
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression:
off)
Type "help" for help.
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
turndb | turn | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
(4 rows)
postgres=# \c turndb
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression:
off) You are now connected to database "turndb" as user "postgres". turndb=# \dt Did not find any relations. turndb=#
How to solve the problem? Marco
Is it possible for you to post the file /home/marco/turnserver-4.5.0.8/examples/var/db/turndb somewhere?
From the backtrace, I see that SBCL seems to fail at reading UTF-8, which means that it attempts to read a non-UTF-8 file as UTF-8.
I hit similar errors trying to ingest from a vanilla shell script running on a MacOS zsh terminal.
I'm hoping to avoid the issue by running the ingest from an Ubuntu host.
