Shadowed symbols in Xerox Common Lisp do not behave as expected

[eblanton]

I am not completely convinced that this is a "bug," per se, it may be a usage problem related to the residential environment and files package.

When creating a function in a package defined in an XCL file that shadows a Lisp function, after saving and reloading the XCL file the shadowed symbol is no longer a shadow, but redefines the external symbol. For example, if I create a package PROBLEM, then, from that package, shadow and define PROBLEM:FIRST, after saving, logging out, and then reloading the package PROBLEM the newly-defined function will have become LISP:FIRST.

To Reproduce Steps to reproduce the behavior:

1. Open a clean environment
2. From an Interlisp exec, run

   (XCL:DEFINE-FILES-ENVIRONMENT "PROBLEM" :PACKAGE (DEFPACKAGE "PROBLEM" (:USE "LISP" "XCL")) :READTABLE "XCL")
   

3. From an XCL exec, run

   (in-package 'problem)
   (shadow 'first)
   (defun first (l) (cl:first (cl:rest l)))
   

4. From an Interlisp exec, run

   (MAKEFILES)
   

   Select the file PROBLEM for the file info "PROBLEM" and the function PROBLEM:FIRST. Note that at this point, the function is PROBLEM:FIRST and the exec knows that!
5. From the Interlisp exec, run (LOGOUT T)
6. Restart the VM
7. From the default XCL exec, run

   (load 'problem)
   (first '(a b c))
   

Note the stack overflow calling CL:FIRST from CL:FIRST, because the fact that the redefined first was in PROBLEM was somehow lost.

Context (please complete the following information):

• OS: Linux
• OS Version: Debian 12
• Host arch: x86_64
• Maiko version: 68c74e4e
• IL:MAKESYSDATE: 31-May-2024 18:48:14

[masinter]

We clearly need to enhance the environment to do better with packages, and in particular the operations around changing packages like calls to cl:shadow.
(XCL:DEFINE-FILES-ENVIRONMENT "PROBLEM" :PACKAGE (DEFPACKAGE "PROBLEM" (:USE "LISP" "XCL")) :READTABLE "XCL") followed by (cl:inpackage :problem) and (cl:shadow cl:first)

creates a files-environment that doesn't match the DEFPACKAGE within. Right?

[eblanton]

I don't actually know how to tell that for sure; I am attaching the generated file (as .txt because otherwise GitHub can't handle it) to this post. It appears to have the appropriate package statements in various places, but I don't know what it should look like. PROBLEM.txt

I should also mention that adding a (P (DEFPACKAGE ...)) to the file with IL:DC does not change any of the relevant behavior as far as I can tell, so I left it out above.

[tfeb]

This is probably a dumb comment as my knowledge of the files system has rotted almost entirely, but is there a way of associating an arbitrary form to be evaluated with a file? Because surely the issue here is that you've said (shadow ...) to an exec and that has not been stashed in the file to be evaluated when it's reloaded.

[eblanton]

I could put a (cl:shadow) form in the (P ...) form, I assume, but the point is that at runtime it knows this detail, and it's not saving it. As I said, maybe this is a misunderstanding of how things should work on my part, but it feels to me (as a user) like this should be handled by automatically when the file is updated. The (shadow) form is indeed run interactively, but the idea that the resulting defun is shadowed is part of the image.

[eblanton]

(I just checked, putting (CL:SHADOW 'PROBLEM:FIRST) in the (P ...) form does fix this problem. This is a reasonable workaround for me to continue with https://github.com/orgs/Interlisp/discussions/1747.

[tfeb]

I'm not sure how Interlisp thinks of this, but in CL terms the function is not shadowed: the symbol which names it is, and such shadowing would affect any other uses of that name. I personally think that capturing operations on names (symbols) should be distinct from capturing function definitions &c.

[eblanton]

OK, I was wrong. The above workaround does not work if the package is defined and exports the shadowed symbol. That is, this crashes:

(P (DEFPACKAGE "PROBLEM"
      (:USE "LISP" "XCL")
      (:EXPORT PROBLEM::FIRST))
   (CL:SHADOW 'CL::FIRST))

But this works as intended, except that PROBLEM:FIRST is inaccessible from outside the package:

(P (CL:SHADOW 'CL:FIRST))

[nbriggs]

I think this should be done as (and you can do it in a XCL exec)

(DEFINE-FILE-ENVIRONMENT "PROBLEM" :PACKAGE
     (DEFPACKAGE "PROBLEM" (:USE "LISP" "XCL") (:SHADOW :FIRST))
  :READTABLE "XCL")

(IN-PACKAGE "PROBLEM")
(DEFUN FIRST (L) (CL:FIRST (CL:REST L)))

and then in an Interlisp exec (to avoid typing all the IL:s)

(MAKEFILES)

and then you should be able to logout/restart fresh, and (LOAD "PROBLEM") and have no problem with doing (in either exec)

    (PROBLEM::FIRST '(A B C))

or in an XCL exec (to avoid messing up your Interlisp exec:

    (in-package "PROBLEM")
    (first '(A B C))

There is a bug - though: The CLHS says that both the :shadow argument to defpackage, and the argument to cl:shadow can be a (list of) string designator which is a character, a symbol, or a string. Medley's cl:shadow ONLY takes a (list of) symbols.

This means you have to be careful in your defpackage - you don't want to use a symbol in the package you're defining as part of the definition - even if it's a local name - because if it ends up exported in the environment and the defpackage/define-file-environment gets rewritten there's a chance it will get written as an external symbol and then fail on reloading. That's why I wrote the example using a keyword :FIRST - I would have preferred to write "FIRST".

[tfeb]

I am not sure whether the aim is to be compatible with CL or not. If the aim is to be compatible with CL then there are some things which need to be adjusted.

Everything below talks about what CL says should happen (unless I have misread things, but I am reasonably sure I have not).

First of all, the form

(DEFPACKAGE "PROBLEM"
  (:USE "LISP" "XCL")
  (:EXPORT PROBLEM::FIRST))

is not legal. It's not legal because it's trying to export a symbol whose name is "FIRST" from a package called PROBLEM, which may not exist when the form is read. That means that in general this form is a read-time error. The form should instead be

(DEFPACKAGE "PROBLEM"
  (:USE "LISP" "XCL")
  (:EXPORT "FIRST"))

or

(DEFPACKAGE "PROBLEM"
  (:USE "LISP" "XCL")
  (:EXPORT #:FIRST))

Now after one of these corrected forms is evaluated the PROBLEM package:

• uses LISP and XCL
• exports a symbol named "FIRST".

So the first question is: what symbol is exported? Well, assuming that at least one of XCL and LISP exports a symbol named "FIRST", that is the symbol which is exported. In particular no new symbol is created in PROBLEM. If neither symbol exports such a symbol then a new symbol will be created and exported. If both packages export a symbol with that name but they are not the same this is an error.

I will assume that it is the case that at least one one of the packages exports that symbol and if they both export it it's the same symbol. I will also assume that this symbol is the same symbol as CL:FIRST.

Given that a possibly surprising side-effect is that CL:FIRST is now present in PROBLEM: it's not just accessible, it's present, because it is exported. This is crucial to understand what happens next.

So, next is

(CL:SHADOW 'CL:FIRST)

What this does is dependent on what *PACKAGE* is at the time it is evaluated. I will assume that it is PROBLEM, but to be safe you should probably say (CL:SHADOW 'CL:FIRST "PROBLEM").

If the package is indeed PROBLEM then the result is almost certainly not what you think.

SHADOW:

• looks for a symbol with the given name(s) in the package
  • if it is present (not just by inheritance) it then adds that symbol to the shadowing symbols list of the package if it is not already there
  • if it is not present it then interns a new symbol with this name in the package and adds it to the shadowing symbols list of the package

Well, CL:FIRST is present in PROBLEM, so this will now add CL:FIRST to PROBLEM's shadowing symbols list.

And now we're done: the end result is that

• there is a package called PROBLEM which exports CL:FIRST, so PROBLEM:FIRST and CL:FIRST are the same symbol
• CL:FIRST is also on PROBLEM's shadowing symbols list, so if you, for instance, defined another package FOO which exported FOO:FIRST which is not the same symbol as CL:FIRST, then using that package in PROBLEM will not result in an error and will mean that PROBLEM:FIRST is still CL:FIRST.

I'm fairly willing to bet that not much of this is what you wanted to happen. It also looks to me like Medley is doing the right thing here.

By far the best way to make sure what you probably want to happen actually happens is to do it all in the defpackage form:

(defpackage "PROBLEM"
  (:use "CL")
  (:shadow #:first)
  (:export #:first))

for instance will create a new symbol named "FIRST" in PROBLEM, add it to the shadowing symbols list of the package and then export it. So now (eq 'problem:first 'cl:first) is nil.

Dealing with the kind of obscure problems above is what defpackage is for.

If you really want to do things by steam, the right approach is something like

(defpackage "PROBLEM"
  (:use "CL"))

(shadow "FIRST" "PROBLEM")
(export (find-symbol "FIRST" "PROBLEM") "PROBLEM")

Finally I realise that the underlying motivation here is to follow examples in a book. I have not checked the book, but it is at least possible that it's simply wrong.

[eblanton]

I am not sure whether the aim is to be compatible with CL or not. If the aim is to be compatible with CL then there are some things which need to be adjusted.

In the case of this issue, that's not the concern I have (although it may be a concern for Medley). However, it is my concern to try to understand where Medley is departing from CL, or CLtL2 (which I think it is closer to?).

First of all, the form

(DEFPACKAGE "PROBLEM"
  (:USE "LISP" "XCL")
  (:EXPORT PROBLEM::FIRST))

is not legal. It's not legal because it's trying to export a symbol whose name is "FIRST" from a package called PROBLEM, which may not exist when the form is read. That means that in general this form is a read-time error.

This is, however, what Medley seems to expect. In particular, it requires the double-colon at definition time when editing the filecoms, but after it has resolved everything it saves it with or without a package specifier at all according to some rules I have not fully grokked yet.

The form should instead be

(DEFPACKAGE "PROBLEM"
  (:USE "LISP" "XCL")
  (:EXPORT "FIRST"))

Medley expects that what follows export are symbols. Neither "FIRST" nor #:FIRST will load.

So, next is

(CL:SHADOW 'CL:FIRST)

What this does is dependent on what *PACKAGE* is at the time it is evaluated. I will assume that it is PROBLEM, but to be safe you should probably say (CL:SHADOW 'CL:FIRST "PROBLEM").

I should clarify that I have tried various shadowing statements in various locations and packages; what I am attempting to emulate is Common Lisp:

(in-package 'problem)
(shadow 'first)

However, some formulations that work in the on-line terminal do not load after passing through (FILES?) and (MAKEFILES). Perhaps this is part of the issue, and the formulation that I should be using is not loading, while the forumulations that load do not do the right thing.

Please note the several different approaches in comments above; the very first approach was exactly what you are suggesting, and I have walked through some alternates in the intervening time, in order to find a file that both loads and has the desired effect (which is a symbol interned in PROBLEM that shadows the FIRST in LISP).

Finally I realise that the underlying motivation here is to follow examples in a book. I have not checked the book, but it is at least possible that it's simply wrong.

The book is fine (at least as far as this is concerned) and works correctly in, e.g., SBCL.

[pamoroso]

Medley Common Lisp is probably closer to CLtL1. There is code to bring it closer to CLtL2 but it has not been merged into mainline as it needs review and a lot of work.

[tfeb]

If you load a file in SBCL (or any conforming CL) containing the following text

(in-package "CL-USER")

(defpackage "PROBLEM"
  (:use "CL")
  (:export #:first))                    ;or "FIRST"

(in-package "PROBLEM")
(shadow 'first)

You will absolutely not construct a symbol problem:first which is distinct from cl:first.

Further if I run the version of Medley at online.interlisp.org with a Common Lisp exec (initial package XCL-USER) and type

(defpackage "PROBLEM"
  (:use "XCL" "LISP")
  (:export "FIRST"))

This will export LISP:FIRST from the PROBLEM package, as it should. Shadowing (as above) works as it should (ie does effectively nothing here).

Perhaps after being saved, this does something different. I have no idea, and I think that's the limit of what I can do to try to help

[masinter]

I still think there will probably be some use cases where you want to READ an s-experssion which presumes a different and incompatible package arrangement than what is currently installed in the residential environment

Examples include: symbols that were previously exported or shadowed or changes to the user environment.
Ther are several conexts where what is typed is not interpreted as a case-sensitive symbol in a package but rather is treated as if it had bee entered as a package-independent keyword:

• Exec commands -- both package and case independent

• LOOP macro okeywords FOR, AS, IS, COLLECT, IN are coerced to keyword

• Masterscope sintax EDIT WHERE ANY CALLS IS BY ...

• lexically scoped variables! (These arennt really package independent, but if you change the package of a file from FOO to BAZ, you can also within scope change X (which prints as FOO::X) to X (which prints as X).

• Exec commands the line is a package-indepenent command rather than a package-aware symbol. Two approaches: *nuse some (speciial variable controlled version of PRINT that causes more package info unambiguous?

• use some way of deferring the package interpretation of what READ returns to resolve package issues.

• (Loops uses #,expression to mean 'interpret expression at LOAD time of compiled file rather than READ time' Can this be used for deferring package decisions.

[nbriggs]

Let’s fix the cl:shadow argument type problem before we take on the bigger issue.

[eblanton]

Perhaps after being saved, this does something different. I have no idea, and I think that's the limit of what I can do to try to help

I appreciate your help. In this case, I think we're looking at a difference between CLtL and ANSI CL; CLtL uses symbols in some places that ANSI CL recommends or requires strings or keywords, and apparently the shadowing semantics are somewhat different.

In Medley (and according to CLtL2 and CL:AIP), this does what I want:

(defpackage "PROBLEM" (:USE "LISP" "XCL"))
(in-package "PROBLEM")
(shadow 'first)
(defun first ...)

I start getting problems when I a) save and reload, or b) export symbols. I've tried many ways to express this, but had no luck.