kons-9
kons-9 copied to clipboard
kaveh808
SBCL Port
When loading I get an error: Don't know how to REQUIRE COCOA. [Condition of type SB-INT:EXTENSION-FAILURE]
The relevant file is ccl64.lisp, the contents of which can be selectively ignored by prepending each expression with #-ccl
All defconstant forms seem to error with the function they're calling having not been defined. I am guessing this means SBCL defines constants before evaluating the rest of the form, so we probably need to wrap the functions they rely on in a eval-when.
This affects at least the files kernel/color and kernel/point
The following files use the #_ ffi call in CCL which won't work in SBCL and will need to be addressed: kernel/opengl.lisp plugins/he-mesh.lisp plugins/l-system.lisp plugins/particle.lisp ui.lisp
For the SBCL port, I moved (almost?) all the CCL/MacOS specific code to opengl.lisp and minimal-ui.lisp.
To help with your work, in kons-9.asd comment in minmal-ui and comment out ui. Also for now simply comment out any plugin files which are making OpenGL calls directly.
@kaveh808 Would you be OK with adding a couple dependencies to help ease the porting to other platforms/lisp implementations? Looking at the opengl.lisp file, it looks like the static-vectors system would provide a portable implementation of make-heap-ivector. The cl-opengl system would provide a portable openGL FFI interface. cl-opengl would also pull in cffi and alexandria.
EDIT: It looks like static-vectors wouldn't be needed. CFFI can take care of allocating the heap vectors. (with-foreign-array etc)
@jolby Yes, by all means. I expected that as we went cross-platform we would need to add some dependencies. The ones you mention seem reasonable.
I decided to do a little exploring to see how far I can get with the program by taking out the CCL/macOS dependent code.
In kons-9.asd I commented out the requirement (:file "ccl64").
In package.lisp I commented out line 2 (:use #:common-lisp #:ccl)
sbcl was then able to load utils.lisp and I got the error
; in: DEFUN PRINT-CLASS-HIERARCHY ; ((KONS-9::SUBCLASSES (KONS-9::CLASS-DIRECT-SUBCLASSES KONS-9::CLASS))) ; caught COMMON-LISP:ERROR: ; illegal function call
and a cascade of cryptic errors following it, all with KONS-9:: prefixing functions and such:
; in: DEFUN FLATTEN-LIST
; ((KONS-9::NULL KONS-9::L) KONS-9::NIL)
; caught COMMON-LISP:ERROR:
; illegal function call`
;
; ((KONS-9::ATOM KONS-9::L) (KONS-9::LIST KONS-9::L))
; caught COMMON-LISP:ERROR:
; illegal function call
etc.
The specific line it complained about is line 8 from utils.lisp: (let ((subclasses (class-direct-subclasses class)))
This is defining a class based upon my understanding of CLOS. Was this class somehow tied in with CCL or the objective-c interface with macOS? If so, we'll need to take that in to account when porting to sbcl.
I think those meta-class calls are not part of the MOP standard, and are CCL specific. They are not being used elsewhere in the code, so you can just comment them out.
It's not defining a class. It is getting a list of subclasses of a given class. I only used it to print out the class hierarchy.
Right: ccl::class-direct-superclasses is CCL-specific.
The MOP isn't part of the Common Lisp standard; it's just an informal de facto standard. sbcl has the sb-mop package, which provides its interpretation of the MOP, including sb-mop:class-direct-superclasses. If you want to be portable, you can quickload :closer-mop and use the symbols exported from it (including closer-mop:class-direct-superclasses).
In general, you're going to run into a lot of CCL-specific code in Kaveh's project. One thing you could do, Kaveh, to make their lives easier, would be to remove the :use #:ccl from the definition of the kons-9 package. That would then force you to use package qualifiers everywhere you reference ccl-specific symbols, which would then make it much easier for folks working on the port to find those calls.
(Of course, it's a good deal of drudgery fixing up all those symbol references, but somebody has to do it, if a port is going to happen.)
Another thing to look for is symbols that start with "#/" or "#$" or "#_". These character sequences prefix names autogenerated by ffigen for the CCL Objective-C interface (if you have CCL installed, you can find the generated interfaces in ccl/
The meanings of the prefixes are:
- #/ An Objective-C method name
- #$ A global variable or constant
- #_ A C function name
You'll notice there are a lot of them. A port to sbcl needs to replace all of them with suitable ffi constructs from CFFI or SB-ALIEN and friends.
Done and pushed. Besides the MOP call (which can be commented out), the only ccl-specific calls were to make-heap-ivector and dispose-heap-ivector in opengl.lisp.
Pretty much all the calls starting with "#/" and the like are isolated in opengl.lisp and minimal-ui.lisp. I commented out the files in kons-9.asd which made direct ccl calls -- l-system.lisp and particle.lisp
I attempted to run it with these new changes and am getting a package lock error:
debugger invoked on a SB-EXT:SYMBOL-PACKAGE-LOCKED-ERROR in thread
#<THREAD "main thread" RUNNING {1004F08113}>:
Lock on package COMMON-LISP violated when defining DEBUG as a function while
in package KONS-9.
SB-EXT:PACKAGE-LOCK-VIOLATION #<PACKAGE "COMMON-LISP"> :SYMBOL DEBUG :FORMAT-CONTROL "defining ~S as a function" :FORMAT-ARGUMENTS (DEBUG))
Unfortunately this doesn't give me much to work with, and being this is an ecosystem I'm not familiar with, I'm not able to track down the source of the error.
cl-user> (symbol-package 'debug) #<package "COMMON-LISP"> cl-user> (describe 'debug) common-lisp:debug [symbol]
debug names an optimization policy quality: (undocumented)
cl-user> (defun debug ()) ; Debugger entered on #<symbol-package-locked-error "proclaiming ~S as a function" {1003C5BAC3}>
Apparently the symbol "cl:debug" cannot be defined as a function.
On Sun, Aug 14, 2022 at 1:07 PM kai wuffy @.***> wrote:
I attempted to run it with these new changes and am running in to a package lock error when attempting to run it:
debugger invoked on a SB-EXT:SYMBOL-PACKAGE-LOCKED-ERROR in thread #<THREAD "main thread" RUNNING {1004F08113}>: Lock on package COMMON-LISP violated when defining DEBUG as a function while in package KONS-9. SB-EXT:PACKAGE-LOCK-VIOLATION #<PACKAGE "COMMON-LISP"> :SYMBOL DEBUG :FORMAT-CONTROL "defining ~S as a function" :FORMAT-ARGUMENTS (DEBUG))
Unfortunately this doesn't give me much to work with, so I can't track down the source of the error.
— Reply to this email directly, view it on GitHub https://github.com/kaveh808/kons-9/issues/18#issuecomment-1214424661, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABGMMIAE7MSO6MHRZR35SDVZEYXRANCNFSM56MKHHEA . You are receiving this because you are subscribed to this thread.Message ID: @.***>
Fixed and pushed file utils.lisp -- renamed debug to my-debug.
On Sun, Aug 14, 2022 at 8:21 AM Andrew Wolven @.***> wrote:
cl-user> (symbol-package 'debug) #<package "COMMON-LISP"> cl-user> (describe 'debug) common-lisp:debug [symbol]
debug names an optimization policy quality: (undocumented)
cl-user> (defun debug ()) ; Debugger entered on #<symbol-package-locked-error "proclaiming ~S as a function" {1003C5BAC3}>
Apparently the symbol "cl:debug" cannot be defined as a function.
On Sun, Aug 14, 2022 at 1:07 PM kai wuffy @.***> wrote:
I attempted to run it with these new changes and am running in to a package lock error when attempting to run it:
debugger invoked on a SB-EXT:SYMBOL-PACKAGE-LOCKED-ERROR in thread #<THREAD "main thread" RUNNING {1004F08113}>: Lock on package COMMON-LISP violated when defining DEBUG as a function while in package KONS-9. SB-EXT:PACKAGE-LOCK-VIOLATION #<PACKAGE "COMMON-LISP"> :SYMBOL DEBUG :FORMAT-CONTROL "defining ~S as a function" :FORMAT-ARGUMENTS (DEBUG))
Unfortunately this doesn't give me much to work with, so I can't track down the source of the error.
— Reply to this email directly, view it on GitHub https://github.com/kaveh808/kons-9/issues/18#issuecomment-1214424661, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AABGMMIAE7MSO6MHRZR35SDVZEYXRANCNFSM56MKHHEA
. You are receiving this because you are subscribed to this thread.Message ID: @.***>
— Reply to this email directly, view it on GitHub https://github.com/kaveh808/kons-9/issues/18#issuecomment-1214426418, or unsubscribe https://github.com/notifications/unsubscribe-auth/AYYMUOODBPAE5YWQG7CET5DVZE2IVANCNFSM56MKHHEA . You are receiving this because you were mentioned.Message ID: @.***>
I had a go at translating the ccl/openGL FFI calls to use cl-opengl/CFFI. I just used a little emacs utility to do it in the buffer. I also added :cffi and :cl-opengl as dependencies in the .asd file, and commented out the loading of the other files in order to isolate the opengl/CFFI changes. The new opengl interface is in kernel/portable-opengl.lisp. The elisp utility is in tools/ .I am at least able to load this on linux/sbcl now, but no further testing. This work can be viewed in this branch:
https://github.com/jolby/kons-9/tree/use-cl-opengl
If this is useful I can try to clean it up for a pull request or maybe Kaveh and others will just want to give it a spin and make their own changes.
Cool. Thanks for doing this.
I grabbed the code and had a go at replacing opengl.lisp with portable-opengl.lisp but ran into some problems.
I have narrowed it down. Just adding:
:depends-on (#:cffi #:cl-opengl #:cl-glu)
to the kons-9.asd file (and without using portable-opengl.lisp) gives me:
;Loading #P"ccl:cocoa-ide;fasls;xinspector.dx64fsl.newest"........
Error: problems loading bundle: can't determine class name While executing: GUI::INIT-COCOA-IDE, in process Initial(0). ;;; ;;; #<PROCESS Initial(0) [Reset] #x3020000B139D> requires access to Shared Terminal Input ;;; Type (:y 0) to yield control to this thread. ;;;
and then (run) complains:
The value NIL is not of the expected type MACPTR. [Condition of type TYPE-ERROR]
Restarts: 0: [RETRY] Retry SLIME REPL evaluation request. 1: [*ABORT] Return to SLIME's top level. 2: [ABORT-BREAK] Reset this thread 3: [ABORT] Kill this thread
Backtrace: 0: ((:OBJC-DISPATCH NEXTSTEP-FUNCTIONS:|performSelectorOnMainThread:withObject:waitUntilDone:|) #<Unknown Arguments>) 1: (CCL::CALL-IN-EVENT-PROCESS #<Compiled-function (:INTERNAL RUN) (Non-Global) #x302003C3498F>) 2: (RUN) 3: (CCL::CALL-CHECK-REGS RUN) 4: (CCL::CHEAP-EVAL (RUN)) --more--
Do I need to quickload the dependencies?
Hi @kaveh808. It wouldn't hurt to try quickloading those deps, just to be sure they load for you on your system without errors. If they do load and you're getting this error, then maybe there are stale fasls from your previous loading of the kons-9 system that are interfering with loading this stripped down version. If I start heavily editing a previously loaded system, I'll sometimes nuke my fasl cache dir ($HOME/.cache/common-lisp on my debian/linux system), and restart/reload my emacs/sly session. Not very elegant but it's sometimes nice to know you're starting from a clean slate.
Thanks @jolby.
Did the quickloads, but same error.
Cleared out the fasl files in my cache, but same error.
If there is no easy solution, maybe no need to worry about this. This combination (CCL and cffi and cl-opengl) is not one we're targeting. It needs to work with SBCL down the road when we have a Mac port of SBCL/Cocoa.
I've made some more progress on the sbcl/linux port. I've added the start of a cl-glfw3 backend and I'm at least able to get a window up with the grid lines and axes showing! @kaveh808 maybe you can see if you can run it on your mac system using sbcl? Same branch as before:
https://github.com/jolby/kons-9/tree/use-cl-opengl
Most excellent. Pinging in @mikelevins who has been working on the SBCL/MacOS port. He will probably send out an update soon, but there are some thorny issues getting SBCL to work properly with Cocoa.
But this is great. Maybe comment in the kernel files and see if they load, and if so, try running the first few tests in demo.lisp?
Well done.
Hi @kaveh808 - actually all the kernel files are loaded, along with several of the plugin files. I am able to try running the first few tests in the demo file, but for most of them I don't see any changes. I am able to run the transforms and hierarchy demo and it produces several smaller axes in addition to the central main one. When I run the rotate examples they all rotate. I'm not seeing any other shapes though, just the tri-colored axes.
Hmm, odd. If you see the axes and they rotate, that means the draw methods are being called. If you don't see any geometry, maybe check if the display flags have the correct values?
(defparameter *display-filled?* t)
(defparameter *display-wireframe?* t)
(defparameter *display-points?* t)
@kaveh808 Cool-- That did the trick! I must have toggled those settings off when I was playing around hitting keys to see if glfw3 was capturing them and passing the key events on. It looks like I can run most of the demos. I'll keep playing around and see how it goes.
I checked out jolby's code on an Ubuntu 22.04 system and loaded it in sbcl 2.1.11. I tried out about the first third of the code in demo.lisp and it all seems to work as advertised.
It doesn't work with cl-glfw3 and sbcl 2.2.6 on macOS. Loading jolby's code works fine, and if I load it into a terminal session I can pose the glfw window. It renders the reference grid and the colored axes as expected. However, any attempt to eval a form that would cause the GL scene to re-render--for example, running any of the demo.lisp code--causes GL event handling to hang. The window still responds to move and activate events, but that's the WindowServer. Moving the mouse over the content area causes it to change to the spinning beachball that is macOS' indication that a process is hung, and a control-click on the sbcl terminal's Dock icon confirms the same thing, offering a message that the "Application is not Responding".
This result is consistent with my prvous experiments with cl-glfw and sbcl under macOS, and, as in those previous experiments, trying it from a SLIME session instead of a terminal means that the glfw window never appears at all.
Kaveh's original CCL code relies on the CCL GUI infrastructure, which uses a couple of ApplicationServices APIs to change the process to process type 1 (kProcessTransformToForegroundApplication), and tell the WindowServer it's the front process. I know the APIs in question and I've tracked down the system library that provides them. I'll see if I can use the same technique CCL does to solve the problem on macOS.
I'm not certain that it will be enough. The CCL code does a little setup before calling these APIs. I read over the code a couple times and I don't see any obvoious gotchas, but I may have missed something.
Anyhow, I'll see what I can do and report back. If I get good results, I'll report them in Issue 16.
Like I said before, with slime+sbcl+macos you can interrupt the initial thread and run the glfw code from that interrupt function and it will work just fine. The main problem is that slime and sly stopped allowing access to the initial thread in the inferior lisp buffer, which it used to do a while ago. I don't know why they removed that functionality, but the workaround is simple.
On Wed, Aug 17, 2022 at 4:09 PM mikel evins @.***> wrote:
I checked out jolby's code on an Ubuntu 22.04 system and loaded it in sbcl 2.1.11. I tried out about the first third of the code in demo.lisp and it all seems to work as advertised.
It doesn't work with cl-glfw3 and sbcl 2.2.6 on macOS. Loading jolby's code works fine, and if I load it into a terminal session I can pose the glfw window. It renders the reference grid and the colored axes as expected. However, any attempt to eval a form that would cause the GL scene to re-render--for example, running any of the demo.lisp code--causes GL event handling to hang. The window still responds to move and activate events, but that's the WindowServer. Moving the mouse over the content area causes it to change to the spinning beachball that is macOS' indication that a process is hung, and a control-click on the sbcl terminal's Dock icon confirms the same thing, offering a message that the "Application is not Responding".
This result is consistent with my prvous experiments with cl-glfw and sbcl under macOS, and, as in those previous experiments, trying it from a SLIME session instead of a terminal means that the glfw window never appears at all.
Kaveh's original CCL code relies on the CCL GUI infrastructure, which uses a couple of ApplicationServices APIs to change the process to process type 1 (kProcessTransformToForegroundApplication), and tell the WindowServer it's the front process. I know the APIs in question and I've tracked down the system library that provides them. I'll see if I can use the same technique CCL does to solve the problem on macOS.
I'm not certain that it will be enough. The CCL code does a little setup before calling these APIs. I read over the code a couple times and I don't see any obvoious gotchas, but I may have missed something.
Anyhow, I'll see what I can do and report back. If I get good results, I'll report them in Issue 16.
— Reply to this email directly, view it on GitHub https://github.com/kaveh808/kons-9/issues/18#issuecomment-1218492413, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABGMMN7IJHJISGPR3SCAU3VZVIG5ANCNFSM56MKHHEA . You are receiving this because you commented.Message ID: @.***>
PS, I have glfw support in cl-vulkan, but what's causing your hangs are lack of proper event handling. In vktk, I have code which shows how to set up glfw event handling if you wanna take a look.
On Wed, Aug 17, 2022 at 6:39 PM Andrew Wolven @.***> wrote:
Like I said before, with slime+sbcl+macos you can interrupt the initial thread and run the glfw code from that interrupt function and it will work just fine. The main problem is that slime and sly stopped allowing access to the initial thread in the inferior lisp buffer, which it used to do a while ago. I don't know why they removed that functionality, but the workaround is simple.
On Wed, Aug 17, 2022 at 4:09 PM mikel evins @.***> wrote:
I checked out jolby's code on an Ubuntu 22.04 system and loaded it in sbcl 2.1.11. I tried out about the first third of the code in demo.lisp and it all seems to work as advertised.
It doesn't work with cl-glfw3 and sbcl 2.2.6 on macOS. Loading jolby's code works fine, and if I load it into a terminal session I can pose the glfw window. It renders the reference grid and the colored axes as expected. However, any attempt to eval a form that would cause the GL scene to re-render--for example, running any of the demo.lisp code--causes GL event handling to hang. The window still responds to move and activate events, but that's the WindowServer. Moving the mouse over the content area causes it to change to the spinning beachball that is macOS' indication that a process is hung, and a control-click on the sbcl terminal's Dock icon confirms the same thing, offering a message that the "Application is not Responding".
This result is consistent with my prvous experiments with cl-glfw and sbcl under macOS, and, as in those previous experiments, trying it from a SLIME session instead of a terminal means that the glfw window never appears at all.
Kaveh's original CCL code relies on the CCL GUI infrastructure, which uses a couple of ApplicationServices APIs to change the process to process type 1 (kProcessTransformToForegroundApplication), and tell the WindowServer it's the front process. I know the APIs in question and I've tracked down the system library that provides them. I'll see if I can use the same technique CCL does to solve the problem on macOS.
I'm not certain that it will be enough. The CCL code does a little setup before calling these APIs. I read over the code a couple times and I don't see any obvoious gotchas, but I may have missed something.
Anyhow, I'll see what I can do and report back. If I get good results, I'll report them in Issue 16.
— Reply to this email directly, view it on GitHub https://github.com/kaveh808/kons-9/issues/18#issuecomment-1218492413, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABGMMN7IJHJISGPR3SCAU3VZVIG5ANCNFSM56MKHHEA . You are receiving this because you commented.Message ID: @.***>
I'll take a look. If you happen to have a particular piece of code in mind, feel free to guide me.
Okay, I've got some better results to report.
Here's a screenshot of my slightly hacked version of jolby's hacked version of Kaveh's demo.lisp running on sbcl 2.2.6 on macOS with a SLIME session connected to it:
https://i.imgur.com/J5nXlc4.png
The window responds to events as expected. C-x C-e in the demo.lisp buffer on the right causes the polyhedra to render as expected. I can close the glfw window, then reopen it with (show-window scene) and it reappears, still responsive (though the grayscales in the shapes are changed), and C-x C-e in demo.lisp still yields updates as expected.
There are some caveats that I'll get to below, but this is certainly forward progress.
I compared jolby's code with examples/main.lisp in awolven's VkTk repo. The names of the FFI definitions differ, but the event handling looks the same.
I did notice, though that jolby's code doesn't appear to use sb-int:with-float-traps-masked, and I know that is held to be necessary for use of glfw on macos, so I copied minimal-glfw-ui.lisp and wrapped sb-int:with-float-traps-masked around the body expression of the handler-bind in show-window.
I also added (ql:quickload :swank) to my version of the source file, and added a call to create a swank server, for reasons I'll explain below.
After that, and following the steps below, I got the results described above.
[From a terminal window:]
- rlwrap sbcl
- (ql:quickload :kons-9)
- (load "minimal-glfw3-ui-mle.lisp") ; this is my hacked version
- (in-package :kons-9)
- (show-window scene)
[In Emacs:] 6. M-s slime-connect [In the Emacs demo.lisp buffer:] 7. C-x C-e each form in demo.lisp, resulting in rendering of the corresponding shapes.
Now, as I said, there are some caveats. For one, if I don't follow the above sequence of steps exactly, I see various kinds of breakage. Changing the order variously yields failure of the glfw window to appear, or hangs in the terminal session, or bogus stack frames.
It's necessary to start sbcl from a terminal; starting it from emacs with M-x slime doesn't work, as awolven suggests. You get a hung slime-connect with a lot of CPU fan activity until you kill the process.
Besides adding the floating-point trap mask to jolby's code, I also added
(eval-when (:compile-toplevel :load-toplevel :execute) (ql:quickload :swank))
to the toplevel of minimal-glfw-ui.lisp before the definition of show-window, then added
(swank:create-server :port 4005)
to show-window before the handler-bind around the event loop. That way, when I create the glfw window, I also create a swank server that I can connect to with SLIME (but it's a temporary hack that should be redone properly).
So, it's a bunch of adhockery, and it's fragile as anything, but it puts a working glfw window on the macOS screen with a SLIME session talking to it.
I will next move to a separate fork, and begin examining what I can do to clean this up. I'll report futher progress in Issue #16.
Excellent!
(though the grayscales in the shapes are changed)
That happens for me as well when I reopen a window. I suspect it's some OpenGL state I'm not initializing.
But really nice work.
I got my mac out. I'm running macOS Monterey 12.5.1, emacs 25.3.50.1, sbcl 2.1.7, slime 2.20, and I can open a glfw window and render graphics like this:
(sb-thread:interrupt-thread (sb-thread:main-thread) #'(lambda () (sb-int:with-float-traps-masked (:invalid :inexact :overflow :underflow :divide-by-zero) (main (make-instance 'demo-application :width 640 :height 480)))))