pgloader
pgloader copied to clipboard
dimitri
KABOOM! FATAL error: There is no thread support in this instance.
Sorry, the bug report below probably does not meet the minimal requirements. However, /usr/bin/pgloade is a binary and the error message even with debug just does not give me a clue. Can be that I should file a bug report to the Raspbian or Debian maintainers.
- [ ] pgloader --version
pgloader version "3.6.1"
compiled with SBCL 1.4.15.debian```
<fill pgloader version here>
- [ ] did you test a fresh compile from the source tree?
No
- [ ] did you search for other similar issues?
Tried, no results.
- [ ] how can I reproduce the bug?
pgloader mysql://homeassistant:onohLAH4feHeJS6j@localhost/homeassistant?charset=utf8mb4 postgres://@/hass
KABOOM!
FATAL error: There is no thread support in this instance.
An unhandled error condition has been signalled:
There is no thread support in this instance.
What I am doing here?
There is no thread support in this instance.
With debug:
homeassistant@raspberrypi:~/.homeassistant $ pgloader -d mysql://homeassistant:onohLAH4feHeJS6j@localhost/homeassistant?charset=utf8mb4 postgres://@/hass
mkdir -p #P"/tmp/pgloader/"
pgloader version 3.6.1
sb-impl::*default-external-format* :UTF-8
tmpdir: #P"/tmp/pgloader/"
KABOOM!
FATAL error: There is no thread support in this instance.
Date/time: 2020-03-15-00:33An unhandled error condition has been signalled:
There is no thread support in this instance.
Backtrace for: #<SB-THREAD:THREAD "main thread" RUNNING {504E7001}>
0: (TRIVIAL-BACKTRACE:PRINT-BACKTRACE-TO-STREAM #<SB-IMPL::STRING-OUTPUT-STREAM {5489E7E9}>)
1: (TRIVIAL-BACKTRACE:PRINT-BACKTRACE #<BORDEAUX-THREADS::BORDEAUX-MP-CONDITION {5489E6C1}> :OUTPUT NIL :IF-EXISTS :APPEND :VERBOSE NIL)
2: ((FLET "H0" :IN PGLOADER::MAIN) #<BORDEAUX-THREADS::BORDEAUX-MP-CONDITION {5489E6C1}>)
3: (SB-KERNEL::%SIGNAL #<BORDEAUX-THREADS::BORDEAUX-MP-CONDITION {5489E6C1}>)
4: (ERROR #<BORDEAUX-THREADS::BORDEAUX-MP-CONDITION {5489E6C1}>)
5: (BORDEAUX-THREADS::%MAKE-THREAD #<unused argument> #<unused argument>)
6: (LPARALLEL.KERNEL::MAKE-WORKER-THREAD #<LPARALLEL.KERNEL:KERNEL :NAME "lparallel" :WORKER-COUNT 1 :USE-CALLER NIL :ALIVE T :SPIN-COUNT 2000 {5489E4F9}> #S(LPARALLEL.KERNEL::WORKER :HANDSHAKE/FROM-WORKER #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL #S(LPARALLEL.RAW-QUEUE:RAW-QUEUE :HEAD NIL :TAIL NIL) :LOCK (NIL) :CVAR NIL) :HANDSHAKE/TO-WORKER #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL #S(LPARALLEL.RAW-QUEUE:RAW-QUEUE :HEAD NIL :TAIL NIL) :LOCK (NIL) :CVAR NIL) :EXIT-NOTIFICATION #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL #S(LPARALLEL.RAW-QUEUE:RAW-QUEUE :HEAD NIL :TAIL NIL) :LOCK (NIL) :CVAR NIL) :THREAD NIL :RUNNING-CATEGORY NIL :INDEX 0 :TASKS #S(LPARALLEL.SPIN-QUEUE:SPIN-QUEUE :HEAD #1=(LPARALLEL.SPIN-QUEUE::DUMMY) :TAIL #1#)) "lparallel" ((PGLOADER.PARAMS:*LOG-FILENAME* . #P"/tmp/pgloader/pgloader.log") (PGLOADER.PARAMS:*LOG-MIN-MESSAGES* . :DEBUG) (PGLOADER.PARAMS:*CLIENT-MIN-MESSAGES* . :DEBUG) (PGLOADER.MONITOR:*MONITORING-QUEUE*) (*ERROR-OUTPUT* . #<SYNONYM-STREAM :SYMBOL SB-SYS:*STDERR* {4F07D8B1}>) (PGLOADER.PARAMS:*ROOT-DIR* . #P"/tmp/pgloader/") (*STANDARD-OUTPUT* . #<SYNONYM-STREAM :SYMBOL SB-SYS:*STDOUT* {4F203C29}>) (PGLOADER.PARAMS:*SUMMARY-PATHNAME*) (PGLOADER.MONITOR::*SECTIONS* QUOTE #S(PGLOADER.STATE::STATE :PRELOAD #S(PGLOADER.STATE::PGSTATE :TABLES #<HASH-TABLE :TEST EQUAL :COUNT 0 {5489DDF9}> :TABNAMES NIL :READ 0 :ROWS 0 :ERRS 0 :SECS 0.0 :RS 0.0 :WS 0.0 :BYTES 0) :DATA #S(PGLOADER.STATE::PGSTATE :TABLES #<HASH-TABLE :TEST EQUAL :COUNT 0 {5489E061}> :TABNAMES NIL :READ 0 :ROWS 0 :ERRS 0 :SECS 0.0 :RS 0.0 :WS 0.0 :BYTES 0) :POSTLOAD #S(PGLOADER.STATE::PGSTATE :TABLES #<HASH-TABLE :TEST EQUAL :COUNT 0 {5489E2C9}> :TABNAMES NIL :READ 0 :ROWS 0 :ERRS 0 :SECS 0.0 :RS 0.0 :WS 0.0 :BYTES 0) :BYTES 0 :SECS 0.0)) (LPARALLEL.KERNEL:*KERNEL* . #<LPARALLEL.KERNEL:KERNEL :NAME "lparallel" :WORKER-COUNT 1 :USE-CALLER NIL :ALIVE T :SPIN-COUNT 2000 {5489E4F9}>)))
7: (LPARALLEL.KERNEL::MAKE-WORKER #<LPARALLEL.KERNEL:KERNEL :NAME "lparallel" :WORKER-COUNT 1 :USE-CALLER NIL :ALIVE T :SPIN-COUNT 2000 {5489E4F9}> 0 #S(LPARALLEL.SPIN-QUEUE:SPIN-QUEUE :HEAD #1=(LPARALLEL.SPIN-QUEUE::DUMMY) :TAIL #1#))
8: (LPARALLEL.KERNEL::%FILL-WORKERS #(0) #<LPARALLEL.KERNEL:KERNEL :NAME "lparallel" :WORKER-COUNT 1 :USE-CALLER NIL :ALIVE T :SPIN-COUNT 2000 {5489E4F9}>)
9: (LPARALLEL.KERNEL::FILL-WORKERS #(0) #<LPARALLEL.KERNEL:KERNEL :NAME "lparallel" :WORKER-COUNT 1 :USE-CALLER NIL :ALIVE T :SPIN-COUNT 2000 {5489E4F9}>)
10: (LPARALLEL.KERNEL:MAKE-KERNEL 1 :NAME "lparallel" :BINDINGS ((PGLOADER.PARAMS:*LOG-FILENAME* . #P"/tmp/pgloader/pgloader.log") (PGLOADER.PARAMS:*LOG-MIN-MESSAGES* . :DEBUG) (PGLOADER.PARAMS:*CLIENT-MIN-MESSAGES* . :DEBUG) (PGLOADER.MONITOR:*MONITORING-QUEUE*) (*ERROR-OUTPUT* . #<SYNONYM-STREAM :SYMBOL SB-SYS:*STDERR* {4F07D8B1}>) (PGLOADER.PARAMS:*ROOT-DIR* . #P"/tmp/pgloader/") (*STANDARD-OUTPUT* . #<SYNONYM-STREAM :SYMBOL SB-SYS:*STDOUT* {4F203C29}>) (PGLOADER.PARAMS:*SUMMARY-PATHNAME*) (PGLOADER.MONITOR::*SECTIONS* QUOTE #S(PGLOADER.STATE::STATE :PRELOAD #S(PGLOADER.STATE::PGSTATE :TABLES #<HASH-TABLE :TEST EQUAL :COUNT 0 {5489DDF9}> :TABNAMES NIL :READ 0 :ROWS 0 :ERRS 0 :SECS 0.0 :RS 0.0 :WS 0.0 :BYTES 0) :DATA #S(PGLOADER.STATE::PGSTATE :TABLES #<HASH-TABLE :TEST EQUAL :COUNT 0 {5489E061}> :TABNAMES NIL :READ 0 :ROWS 0 :ERRS 0 :SECS 0.0 :RS 0.0 :WS 0.0 :BYTES 0) :POSTLOAD #S(PGLOADER.STATE::PGSTATE :TABLES #<HASH-TABLE :TEST EQUAL :COUNT 0 {5489E2C9}> :TABNAMES NIL :READ 0 :ROWS 0 :ERRS 0 :SECS 0.0 :RS 0.0 :WS 0.0 :BYTES 0) :BYTES 0 :SECS 0.0))) :CONTEXT NIL :SPIN-COUNT NIL :USE-CALLER NIL)
11: (PGLOADER.MONITOR:START-MONITOR :START-LOGGER T :QUEUE NIL :LOG-FILENAME NIL :LOG-MIN-MESSAGES NIL :CLIENT-MIN-MESSAGES NIL)
12: (PGLOADER.MONITOR::CALL-WITH-MONITOR #<CLOSURE (LAMBDA NIL :IN PGLOADER::MAIN) {5489E36D}>)
13: (PGLOADER::MAIN ("pgloader" "-d" "mysql://homeassistant:onohLAH4feHeJS6j@localhost/homeassistant?charset=utf8mb4" "postgres://@/hass"))
14: ((LAMBDA NIL :IN "/build/pgloader-YEgT9v/pgloader-3.6.1/dumper-2SKVI5f7.lisp"))
15: ((FLET SB-UNIX::BODY :IN SAVE-LISP-AND-DIE))
16: ((FLET "WITHOUT-INTERRUPTS-BODY-14" :IN SAVE-LISP-AND-DIE))
17: ((LABELS SB-IMPL::RESTART-LISP :IN SAVE-LISP-AND-DIE))
What I am doing here?
There is no thread support in this instance
Thread support is provided by the lisp implementation, here SBCL 1.4.15. Which platform are you using, where SBCL does not have threads? Consider using a newer version of SBCL with thread support for your platform (see http://www.sbcl.org) or using Clozure-CL maybe?
I have same problem,
pi@raspberrypi:~/TAOP/data/cdstore$ sudo apt show sbcl
Package: sbcl
Version: 2:1.4.16-2
Priority: optional
Section: lisp
Maintainer: Debian Common Lisp Team <[email protected]>
Installed-Size: 38.1 MB
Provides: lisp-compiler, sbcl-fasl-loader-78
Depends: libc6 (>= 2.28), zlib1g (>= 1:1.1.4)
Recommends: binfmt-support
Suggests: sbcl-doc, sbcl-source, slime
Breaks: cl-asdf (<< 2:3.1.5)
Homepage: http://www.sbcl.org
Download-Size: 7,798 kB
APT-Sources: http://raspbian.raspberrypi.org/raspbian buster/main armhf Packages
pi@raspberrypi:~/TAOP/data/cdstore$ pgloader --version
pgloader version "3.6.1"
compiled with SBCL 1.4.15.debian
pi@raspberrypi:~/TAOP/data/cdstore$
Installed pgloader using apt .
1.4.16 is very old and will likely not run modern pgloader. Can you try with a newer SBCL version?
Is your SBCL built with threading support? Run sbcl and type *features*; check if :THREAD-SUPPORT is present within the output.
After looking into this issue, I've stumbled across this source that says that SBCL does not support threading on 32-bit ARM architectures. As far as I can tell from the trace-back above this issue (so far) has only occurred on Raspberry Pis (including mine), which happen to have a 32-bit ARM architecture. Does this mean that pgloader cannot be used on 32-bit ARM architectures? Also some more information I discovered while trying to get this to work is when using apt-get install sbcl, the latest stable version available to the Pi is 2:1.4.16-2 (which can be seen in @eldemcan's traceback). However, it appears that regardless of the version of SBCL, when using pgloader --version it always displays compiled with SBCL 1.4.15.debian, even after reinstalling pgloader . I have no idea why that's happening.
I am not well-versed enough in SPCL or Linux to develop a work-around or to even determine whether or not one is even possible. Would greatly appreciate if someone could tell me whether trying to get pgloader to work on a 32-bit architecture is work the effort or if it is just a lost cause.
Does this mean that pgloader cannot be used on 32-bit ARM architectures?
AFAIK CCL supports threads on arm32.
AFAIK CCL supports threads on arm32.
So what would be the next step in trying to get this to work. As I mentioned above, I am not very well versed with Linux or SPCL. Tbh, I don't even know what AFAIK CCL is.
CCL is Clozure Common Lisp, another implementation of CL.
I think the steps would look like this: grab a copy from https://github.com/Clozure/ccl/releases; make the CCL available in the path symlinked to ccl; edit https://github.com/dimitri/pgloader/blob/master/Makefile#L6 to say ccl instead of sbcl; build pgloader.
CCL is Clozure Common Lisp, another implementation of CL.
I think the steps would look like this: grab a copy from https://github.com/Clozure/ccl/releases; make the CCL available in the path symlinked to
ccl; edit https://github.com/dimitri/pgloader/blob/master/Makefile#L6 to saycclinstead ofsbcl; build pgloader.
Alright I will try this and report back my findings.
While trying to build I got this message.
Loading "zs3"
> Error: Error while trying to load definition for system ironclad from
> pathname
> /home/pi/pgloader/build/quicklisp/dists/quicklisp/software/ironclad-v0.50/ironclad.asd:
>
> :IF-FEATURE is an invalid initarg to INITIALIZE-INSTANCE for #<STANDARD-CLASS MODULE>.
> Valid initargs: (:NAME :VERSION :DESCRIPTION :LONG-DESCRIPTION
> :IN-ORDER-TO :DO-FIRST :PARENT :PATHNAME
> :AROUND-COMPILE :ENCODING :PROPERTIES :COMPONENTS
> :IF-COMPONENT-DEP-FAILS :DEFAULT-COMPONENT-CLASS).
> While executing: (:INTERNAL ASDF::LOAD-SYSDEF), in process listener(1).
> Type :GO to continue, :POP to abort, :R for a list of available restarts.
> If continued: Skip evaluation of (ql:quickload "pgloader")
> Type :? for other options.
1 >
#862 might be a related issue, but I am not sure.
Which CCL version are you using? (Try running ccl and posting the greeting banner printed by the compiler.)
I ran ccl and here is the output ⬇
Clozure Common Lisp Version 1.12 (v1.12) LinuxARM32
For more information about CCL, please see http://ccl.clozure.com.
CCL is free software. It is distributed under the terms of the Apache
Licence, Version 2.0.
I am pretty sure this is the latest version.
Yes, CCL 1.12 is the latest. Strange.
Once you get this error message while building, can you issue (asdf:asdf-version)?
Alright while building I got the same error as above. Here's the output from (asdf:asdf-version) ⬇
"2.26"
Wow! 2.26 is ancient - no doubt that pgloader doesn't build.
We need to download the recent ASDF from https://common-lisp.net/project/asdf/archives/asdf.lisp and load it before pgloader is attempted to be built.
At https://github.com/dimitri/pgloader/blob/master/Makefile#L120-L125 there is a passage:
$(BUILDAPP_CCL): $(QLDIR)/setup.lisp
mkdir -p $(BUILDDIR)/bin
$(CL) $(CL_OPTS) --load $(QLDIR)/setup.lisp \
--eval '(ql:quickload "buildapp")' \
--eval '(buildapp:build-buildapp "$@")' \
--eval '(quit)'
Let's try modifying that to:
$(BUILDAPP_CCL): $(QLDIR)/setup.lisp
mkdir -p $(BUILDDIR)/bin
$(CL) $(CL_OPTS) \
--load /home/your/path/to/asdf.lisp \
--load $(QLDIR)/setup.lisp \
--eval '(ql:quickload "buildapp")' \
--eval '(buildapp:build-buildapp "$@")' \
--eval '(quit)'
Where /home/your/path/to/asdf.lisp points to the ASDF release you have downloaded.
@dimitri: it seems that pgloader has an implicit dependency on ASDF3 - this should be specified somewhere in the code, just like split-sequence does:
#.(unless (or #+asdf3.1 (version<= "3.1" (asdf-version)))
(error "You need ASDF >= 3.1 to load this system correctly."))
Hmmm... something weird happened. I did as you instructed above, however, the same error appeared. And even more puzzling is when I ran (asdf:asdf-version) it still printed "2.26", even though I installed the new version and added it to the Makefile.
Here's my Makefile for reference:
# pgloader build tool
APP_NAME = pgloader
VERSION = 3.6.2
# use either sbcl or ccl
CL = armcl
# default to 4096 MB of RAM size in the image
DYNSIZE = 4096
LISP_SRC = $(wildcard src/*lisp) \
$(wildcard src/monkey/*lisp) \
$(wildcard src/utils/*lisp) \
$(wildcard src/load/*lisp) \
$(wildcard src/parsers/*lisp) \
$(wildcard src/pg-copy/*lisp) \
$(wildcard src/pgsql/*lisp) \
$(wildcard src/sources/*lisp) \
pgloader.asd
BUILDDIR = build
LIBS = $(BUILDDIR)/libs.stamp
QLDIR = $(BUILDDIR)/quicklisp
MANIFEST = $(BUILDDIR)/manifest.ql
LATEST = $(BUILDDIR)/pgloader-latest.tgz
BUNDLEDIST = 2020-02-18
BUNDLENAME = pgloader-bundle-$(VERSION)
BUNDLEDIR = $(BUILDDIR)/bundle/$(BUNDLENAME)
BUNDLE = $(BUILDDIR)/$(BUNDLENAME).tgz
BUNDLETESTD= $(BUILDDIR)/bundle/test
ifeq ($(OS),Windows_NT)
EXE = .exe
COMPRESS_CORE = no
DYNSIZE = 1024 # support for windows 32 bits
else
EXE =
endif
PGLOADER = $(BUILDDIR)/bin/$(APP_NAME)$(EXE)
BUILDAPP_CCL = $(BUILDDIR)/bin/buildapp.ccl$(EXE)
BUILDAPP_SBCL = $(BUILDDIR)/bin/buildapp.sbcl$(EXE)
ifeq ($(CL),sbcl)
BUILDAPP = $(BUILDAPP_SBCL)
BUILDAPP_OPTS = --require sb-posix \
--require sb-bsd-sockets \
--require sb-rotate-byte
CL_OPTS = --noinform --no-sysinit --no-userinit
else
BUILDAPP = $(BUILDAPP_CCL)
CL_OPTS = --no-init
endif
ifeq ($(CL),sbcl)
COMPRESS_CORE ?= $(shell $(CL) --noinform \
--quit \
--eval '(when (member :sb-core-compression cl:*features*) (write-string "yes"))')
endif
# note: on Windows_NT, we never core-compress; see above.
ifeq ($(COMPRESS_CORE),yes)
COMPRESS_CORE_OPT = --compress-core
endif
DEBUILD_ROOT = /tmp/pgloader
all: $(PGLOADER)
clean:
rm -rf $(LIBS) $(QLDIR) $(MANIFEST) $(BUILDAPP) $(PGLOADER) docs/_build
$(QLDIR)/local-projects/qmynd:
git clone --depth 1 https://github.com/qitab/qmynd.git $@
$(QLDIR)/local-projects/cl-ixf:
git clone --depth 1 https://github.com/dimitri/cl-ixf.git $@
$(QLDIR)/local-projects/cl-db3:
git clone --depth 1 https://github.com/dimitri/cl-db3.git $@
$(QLDIR)/local-projects/cl-csv:
git clone --depth 1 https://github.com/AccelerationNet/cl-csv.git $@
$(QLDIR)/setup.lisp:
mkdir -p $(BUILDDIR)
curl -o $(BUILDDIR)/quicklisp.lisp http://beta.quicklisp.org/quicklisp.lisp
$(CL) $(CL_OPTS) --load $(BUILDDIR)/quicklisp.lisp \
--load src/getenv.lisp \
--eval '(quicklisp-quickstart:install :path "$(BUILDDIR)/quicklisp" :proxy (getenv "http_proxy"))' \
--eval '(quit)'
quicklisp: $(QLDIR)/setup.lisp ;
clones: $(QLDIR)/local-projects/cl-ixf \
$(QLDIR)/local-projects/cl-db3 \
$(QLDIR)/local-projects/cl-csv \
$(QLDIR)/local-projects/qmynd ;
$(LIBS): $(QLDIR)/setup.lisp
$(CL) $(CL_OPTS) --load $(QLDIR)/setup.lisp \
--eval '(push :pgloader-image *features*)' \
--eval '(setf *print-circle* t *print-pretty* t)' \
--eval '(push "$(PWD)/" ql:*local-project-directories*)' \
--eval '(ql:quickload "pgloader")' \
--eval '(quit)'
touch $@
libs: $(LIBS) ;
$(MANIFEST): $(LIBS)
$(CL) $(CL_OPTS) --load $(QLDIR)/setup.lisp \
--eval '(ql:write-asdf-manifest-file "$(MANIFEST)")' \
--eval '(quit)'
manifest: $(MANIFEST) ;
$(BUILDAPP_CCL): $(QLDIR)/setup.lisp
mkdir -p $(BUILDDIR)/bin
$(CL) $(CL_OPTS) \
--load /home/pi/pgloader/asdf.lisp \
--load $(QLDIR)/setup.lisp \
--eval '(ql:quickload "buildapp")' \
--eval '(buildapp:build-buildapp "$@")' \
--eval '(quit)'
$(BUILDAPP_SBCL): $(QLDIR)/setup.lisp
mkdir -p $(BUILDDIR)/bin
$(CL) $(CL_OPTS) --load $(QLDIR)/setup.lisp \
--eval '(ql:quickload "buildapp")' \
--eval '(buildapp:build-buildapp "$@")' \
--eval '(quit)'
buildapp: $(BUILDAPP) ;
$(PGLOADER): $(MANIFEST) $(BUILDAPP) $(LISP_SRC)
mkdir -p $(BUILDDIR)/bin
$(BUILDAPP) --logfile /tmp/build.log \
$(BUILDAPP_OPTS) \
--sbcl $(CL) \
--asdf-path . \
--asdf-tree $(QLDIR)/local-projects \
--manifest-file $(MANIFEST) \
--asdf-tree $(QLDIR)/dists \
--asdf-path . \
--load-system cffi \
--load-system cl+ssl \
--load-system mssql \
--load src/hooks.lisp \
--load-system $(APP_NAME) \
--entry pgloader:main \
--dynamic-space-size $(DYNSIZE) \
$(COMPRESS_CORE_OPT) \
--output [email protected]
# that's ugly, but necessary when building on Windows :(
mv [email protected] $@
pgloader: $(PGLOADER) ;
pgloader-standalone:
$(BUILDAPP) $(BUILDAPP_OPTS) \
--sbcl $(CL) \
--load-system $(APP_NAME) \
--load src/hooks.lisp \
--entry pgloader:main \
--dynamic-space-size $(DYNSIZE) \
$(COMPRESS_CORE_OPT) \
--output $(PGLOADER)
test: $(PGLOADER)
$(MAKE) PGLOADER=$(realpath $(PGLOADER)) CL=$(CL) -C test regress
save: ./src/save.lisp $(LISP_SRC)
$(CL) $(CL_OPTS) --load ./src/save.lisp
check-saved:
$(MAKE) PGLOADER=$(realpath $(PGLOADER)) CL=$(CL) -C test regress
clean-bundle:
rm -rf $(BUNDLEDIR)
rm -rf $(BUNDLETESTD)/$(BUNDLENAME)/*
$(BUNDLETESTD):
mkdir -p $@
$(BUNDLEDIR):
mkdir -p $@
$(CL) $(CL_OPTS) --load $(QLDIR)/setup.lisp \
--eval '(defvar *bundle-dir* "$@")' \
--eval '(defvar *pwd* "$(PWD)/")' \
--eval '(defvar *ql-dist* "$(BUNDLEDIST)")' \
--load bundle/ql.lisp
$(BUNDLEDIR)/version.sexp: $(BUNDLEDIR)
echo "\"$(VERSION)\"" > $@
$(BUNDLE): $(BUNDLEDIR) $(BUNDLEDIR)/version.sexp
cp bundle/README.md $(BUNDLEDIR)
cp bundle/save.lisp $(BUNDLEDIR)
sed -e s/%VERSION%/$(VERSION)/ < bundle/Makefile > $(BUNDLEDIR)/Makefile
git archive --format=tar --prefix=pgloader-$(VERSION)/ master \
| tar -C $(BUNDLEDIR)/local-projects/ -xf -
make QLDIR=$(BUNDLEDIR) clones
tar -C build/bundle \
--exclude bin \
--exclude test/sqlite \
-czf $@ $(BUNDLENAME)
bundle: clean-bundle $(BUNDLE) $(BUNDLETESTD)
tar -C $(BUNDLETESTD) -xf $(BUNDLE)
make -C $(BUNDLETESTD)/$(BUNDLENAME)
$(BUNDLETESTD)/$(BUNDLENAME)/bin/pgloader --version
test-bundle:
$(MAKE) -C $(BUNDLEDIR) test
deb:
# intended for use on a debian system
mkdir -p $(DEBUILD_ROOT) && rm -rf $(DEBUILD_ROOT)/*
rsync -Ca --exclude 'build' \
--exclude '.vagrant' \
./ $(DEBUILD_ROOT)/
cd $(DEBUILD_ROOT) && make -f debian/rules orig
cd $(DEBUILD_ROOT) && debuild -us -uc -sa
cp -a /tmp/pgloader_* /tmp/cl-pgloader* build/
rpm:
# intended for use on a CentOS or other RPM based system
mkdir -p $(DEBUILD_ROOT) && rm -rf $(DEBUILD_ROOT)
rsync -Ca --exclude=build/* ./ $(DEBUILD_ROOT)/
cd /tmp && tar czf $(HOME)/rpmbuild/SOURCES/pgloader-$(VERSION).tar.gz pgloader
cd $(DEBUILD_ROOT) && rpmbuild -ba pgloader.spec
cp -a $(HOME)/rpmbuild/SRPMS/*rpm build
cp -a $(HOME)/rpmbuild/RPMS/x86_64/*rpm build
pkg:
# intended for use on a MacOSX system
mkdir -p $(DEBUILD_ROOT) && rm -rf $(DEBUILD_ROOT)/*
mkdir -p $(DEBUILD_ROOT)/usr/local/bin/
mkdir -p $(DEBUILD_ROOT)/usr/local/share/man/man1/
cp ./pgloader.1 $(DEBUILD_ROOT)/usr/local/share/man/man1/
cp ./build/bin/pgloader $(DEBUILD_ROOT)/usr/local/bin/
pkgbuild --identifier org.tapoueh.pgloader \
--root $(DEBUILD_ROOT) \
--version $(VERSION) \
./build/pgloader-$(VERSION).pkg
latest:
git archive --format=tar --prefix=pgloader-$(VERSION)/ v$(VERSION) \
| gzip -9 > $(LATEST)
check: test ;
.PHONY: test pgloader-standalone docs bundle
I might have applied this to just one step but there seem to be more in the makefile.
Basically - look where Quicklisp, which is --load $(QLDIR)/setup.lisp, is loaded - and before each such occurrence, try to load the new ASDF in the same style in which we've edited the above makefile step.
Alright we made progress, but we are not quite there yet. It appears that the fix you suggested worked to fix the error above, however, there is a new error. Also this time around when I ran (asdf:asdf-version) it outputed "3.3.4" instead of "2.26". I think we are close! 🙃
[package usocket]..............
Read error between positions 66 and 163 in /home/pi/pgloader/build/quicklisp/dists/quicklisp/software/usocket-0.8.3/backend/openmcl.lisp.
> Error: Foreign function not found: ARM-LINUX::|gethostname|
> While executing: CCL::LOAD-EXTERNAL-FUNCTION, in process listener(1).
> Type :GO to continue, :POP to abort, :R for a list of available restarts.
> If continued: Skip evaluation of (ql:quickload "pgloader")
> Type :? for other options.
1 >
You can see the Makefile file I used here.
Yes, ASDF 3.3.4 is what we want to have - that's good.
Now we have a different compilation issue - usocket fails to build on CCL for arm32 since the gethostname function was not found, and I do not know why that is the case.
Please file an issue on https://github.com/Clozure/ccl and link me there.
Alright so I was able to solve the issue I mentioned above. See this issue for reference.
However, now I run into another (which I believe) is directly related to pgloader. Here is the new error ⬇
;; loading system "cffi"
;; loading system "cl+ssl"
;; loading system "mssql"
;; loading file #P"/home/pi/pgloader/src/hooks.lisp"
;; loading system "pgloader"
Fatal SIMPLE-ERROR:
Compilation failed: In POSTGRES-ARRAY-STRING-TO-ARRAY: Undeclared free variable WOL=CTC/WOL in /home/pi/pgloader/build/quicklisp/dists/quicklisp/software/postmodern-20200715-git/postmodern/util.lisp
Fatal error in "buildapp" : Error #<SILENT-EXIT-ERROR #x14640BFE>
make: *** [Makefile:146: build/bin/pgloader] Error 255
I tried rebuilding from scratch as well and I got a similar error ⬇
;; loading system "cffi"
;; loading system "cl+ssl"
;; loading system "mssql"
;; loading file #P"/home/pi/pgloader-test/src/hooks.lisp"
;; loading system "pgloader"
Fatal error in "buildapp" : Error #<SILENT-EXIT-ERROR #x1466CE56>
make: *** [Makefile:146: build/bin/pgloader] Error 255
Congratulations - you have found a Postmodern error! :open_mouth:
See the ticket for a workaround in Postmodern code.
https://github.com/marijnh/Postmodern/issues/247 is now fixed. Please try using the newest version from master and let's continue trying to get this to work!
I think pgloader will, by default, pick up the Quicklisp version, which is regenerated monthly. @dimitri - do you have any pointers?
Alright I built from scratch (by deleting the old folder and using git clone again). Looks like pgloader didn't pick up the latest version. 😐 Here's the same error as before ⬇
;; loading system "cffi"
;; loading system "cl+ssl"
;; loading system "mssql"
;; loading file #P"/home/pi/pgloader/src/hooks.lisp"
;; loading system "pgloader"
Fatal error in "buildapp" : Error #<SILENT-EXIT-ERROR #x1463EACE>
make: *** [Makefile:146: build/bin/pgloader] Error 255
[Edit] Here's a longer version of the error ⬇
; Loading "buildapp"
[package buildapp]...........
;; loading system "buildapp"
mkdir -p build/bin
build/bin/buildapp.ccl --logfile /tmp/build.log \
\
--sbcl ccl \
--asdf-path . \
--asdf-tree build/quicklisp/local-projects \
--manifest-file build/manifest.ql \
--asdf-tree build/quicklisp/dists \
--asdf-path . \
--load-system cffi \
--load-system cl+ssl \
--load-system mssql \
--load src/hooks.lisp \
--load-system pgloader \
--entry pgloader:main \
--dynamic-space-size 4096 \
\
--output build/bin/pgloader.tmp
;; loading system "cffi"
;; loading system "cl+ssl"
;; loading system "mssql"
;; loading file #P"/home/pi/pgloader/src/hooks.lisp"
;; loading system "pgloader"
Fatal error in "buildapp" : Error #<SILENT-EXIT-ERROR #x1463EACE>
make: *** [Makefile:146: build/bin/pgloader] Error 255
The current Makefile organisation and buildapp use can make it hard to debug building pgloader in many cases. That's why there's also the https://github.com/dimitri/pgloader/blob/master/src/save.lisp option, and the make CL=ccl save target. Please try that one, and as all it does is the following, you could also try the following instead:
ccl64 --no-init --load ./src/save.lisp
That way @phoe might have more clues and will be in a better position to help debug the building of pgloader on your architecture.
Error 1
Alright I tried rebuilding with ccl --no-init --load ./src/save.lisp and got this error ⬇
272,874 bytes in 0.03 seconds (8075.11KB/sec)
> Error: Unexpected HTTP status for #<URL "http://beta.quicklisp.org/archive/qmynd/2019-07-10/qmynd-20190710-git.tgz">: 206
> While executing: (:INTERNAL QL-HTTP:HTTP-FETCH), in process listener(1).
> Type :GO to continue, :POP to abort, :R for a list of available restarts.
> If continued: Skip loading "./src/save.lisp"
> Type :? for other options.
1 >
I am using ccl instead of ccl64 because the computer in question is 32bit ARM and I don't have the ccl64 command. Also when I ran (asdf:asdf-version) it outputted "3.3.3" instead of the normal "3.3.4".
Error 2
I tried rebuilding again, just to make the error was reproducible, and the error happened at a different time then the previous error. The errors seem similar in nature though, both of them have Error: Unexpected HTTP status for #<URL ... > in them. Also btw, (asdf:asdf-version) is still "3.3.3".
To load "pgloader":
Load 1 ASDF system:
pgloader
; Loading "pgloader"
To load "mssql":
Load 4 ASDF systems:
asdf cffi iterate parse-number
Install 2 Quicklisp releases:
cl-mssql garbage-pools
> Error: Unexpected HTTP status for #<URL "http://beta.quicklisp.org/archive/garbage-pools/2013-07-20/garbage-pools-20130720-git.tgz">: 206
> While executing: (:INTERNAL QL-HTTP:HTTP-FETCH), in process listener(1).
> Type :GO to continue, :POP to abort, :R for a list of available restarts.
> If continued: Skip loading "./src/save.lisp"
> Type :? for other options.
1 >
Misc
I tried rebuilding again just out of curiosity to see what would happen, and I got the same result as Error 2. Once again, (asdf:asdf-version) is still "3.3.3".
The 206 status is generated from some intermediate process - the Quicklisp software does not request ranges. Do you perhaps have a HTTP proxy in your environment?
Alright, it turns out (I think) that the "HTTP proxy" I had was nginx. Once I disabled it, the error above went away. But now I get another error and this one is short and sweet. All it says is Segmentation fault and that's it.
[Edit]
I also tried rebuilding with sudo and still got the same simple error Segmentation fault.
Ouch. This is the worst. I hate debugging non-memory-safe code.
Could you post the full console output?