medley
medley copied to clipboard
Interlisp
Errors loading LOOPS sources
Loading LOOPS into a newly built sysout, where it started with a reasonable DEFAULTFILETYPES list, but something appears to have smashed the list in a way that causes OPENSTREAM to choke when it tries to CL:ASSOC an entry on the DEFAULTFILETYPES alist because there's a non-string/non-atom object as a key. No idea what did this.
save the LISP and FULL and "mid" sysouts -- loading POSTSCRIPTSTREAM adds (PS . TEXT) there; let's track down the smash if we can. I have a commit that reworks the initialization
I've saved all of "tmp" after the loadup. The value of DEFAULTFILETYPELIST is OK at the beginning, OK after loading LOADLOOPS (the source), but when I execute (LOADLOOPS NIL 'SOURCE) it loads a couple of files and then dies as above with the smashed atom on the a-list.
If this is coming from deep inside the UFS implementation, that begs the question (apart from this particular failure): what is the point of trying to guess the type of a file? Does it matter in any way?
On Mar 10, 2022, at 12:44 PM, Nick Briggs @.***> wrote:
Current loadups produce this result instead... https://user-images.githubusercontent.com/8138841/157751014-4a861159-5e9a-446f-adb4-a55c55e3367a.png — Reply to this email directly, view it on GitHub https://github.com/Interlisp/medley/issues/733#issuecomment-1064486282, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQSTUJMTYR4JPOMAZBWJ4BLU7JNJ7ANCNFSM5QFFGKUA. You are receiving this because you are subscribed to this thread.
I don't believe this is coming from the UFS implementation. I don't think the other problem was caused by the UFS implementation either -- just the symptoms showed up there. Also, I'm not sure that I agree with the DEFAULTFILETYPELIST that a PDF file is TEXT (they almost always contain binary data). It's unclear to me, either, why it needs to know whether a file is TEXT or BINARY...
@nbriggs Also, I'm not sure that I agree with the DEFAULTFILETYPELIST that a PDF file is TEXT (they almost always contain binary data). It's unclear to me, either, why it needs to know whether a file is TEXT or BINARY...
I may have this out of context but PDF files should always be treated as binary and on GIT, there should be a .gitattributes that establishes that.
The problem stems from at least the suppression of CR/LF and LF translation when text files are moved between different platforms and/or git/GitHub (a problem that FTP clients also had to deal with). Also, there should not be DIFFs between binaries and no prospect of Unicode mishaps by mistreatment of files as platform-specific text.
If this is relevant in the context of this discussion :).
the DEFAULTFILETYPELIST variable was an innocent bystander. I don't think it's used for much
grep -lri defaultfiletypelist
shows
docs/Sun Users Guide/5-MEDLEY-FILES.TEDIT
docs/Sun Users Guide/INDEX.TEDIT
docs/html-sunguide/Sun-Users-Guide_files/content.htm
loadups/full.sysout
loadups/lisp.venuesysout
loadups/starter.sysout
loadups/full.venuesysout
loadups/whereis.hash
loadups/lisp.sysout
tmp/full.sysout
tmp/init.sysout
tmp/init-mid.sysout
tmp/init.dlinit
tmp/whereis.hash
tmp/lisp.sysout
library/POSTSCRIPTSTREAM
library/POSTSCRIPTSTREAM.LCOM
seems it's contained to UFS. EDITCALLERS(DEFAULTFILETYPELIST UFS) makes it looks like it has some kind of effect on EOL handling. This is all before MIME so the expectations about file "type" based on file extension might be a leftover. The recent PR changing DEFAULTFILETYPELIST initiation was to get rid of code inside LOADUP-FULL that mucked with it.
DEFAULTFILETYPELIST appears only to be used internal to UFS. We inherited the text/binary distinction from the fact that ftp in those days did EOL conversion, but in hindsite it was a mistake to import that into our file handling. Because whether or not we want to treat a file as binary or not depends on the Lisp function that we are using to operate on it—in reading with a character function, all EOL varaitions should map to our internal EOL, in writing it is only when we are applying a character printing function that we would worry about the EOL convention on the file, whether or not the file is text or not (it could have text regions and binary regions).
It may be helpful to tell git that a given file is binary, so it knows not to bother to do a meaningless compare. I don’t know about that.
In GITFNS, I look at inside the file to see whether it is a lisp sourcefile, a compiled file, a tedit file, to choose a useful comparison function. If unrecognized, I then look at PRINTFILETYPES, which typically does look at extensions, and I tend to avoid comparing on files that it thinks are binary. Sort of an attempt to behave like git in that situation.
But maybe that is a fool's errand. Unlike git, the GITFNS comparison is menu driven—you have to select the files that you want to compare. If you happen to select a binary file you won’t get a meaningful result, but otherwise, no harm no foul.
On Mar 10, 2022, at 7:26 PM, Larry Masinter @.***> wrote:
the DEFAULTFILETYPELIST variable was an innocent bystander. I don't think it's used for much
grep -lri defaultfiletypelist shows
docs/Sun Users Guide/5-MEDLEY-FILES.TEDIT docs/Sun Users Guide/INDEX.TEDIT docs/html-sunguide/Sun-Users-Guide_files/content.htm loadups/full.sysout loadups/lisp.venuesysout loadups/starter.sysout loadups/full.venuesysout loadups/whereis.hash loadups/lisp.sysout tmp/full.sysout tmp/init.sysout tmp/init-mid.sysout tmp/init.dlinit tmp/whereis.hash tmp/lisp.sysout library/POSTSCRIPTSTREAM library/POSTSCRIPTSTREAM.LCOM seems it's contained to UFS. EDITCALLERS(DEFAULTFILETYPELIST UFS) makes it looks like it has some kind of effect on EOL handling. This is all before MIME so the expectations about file "type" based on file extension might be a leftover. The recent PR changing DEFAULTFILETYPELIST initiation was to get rid of code inside LOADUP-FULL that mucked with it.
— Reply to this email directly, view it on GitHub https://github.com/Interlisp/medley/issues/733#issuecomment-1064736105, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQSTUJJGR7CRIN5CSIENSELU7K4MZANCNFSM5QFFGKUA. You are receiving this because you commented.
The Git software and settings for text/binary decisions is completely separate from the determination in Lisp.
https://docs.github.com/en/get-started/getting-started-with-git/configuring-git-to-handle-line-endings
separate issue---the per-repo .gitattributes can't be used because lisp source files don't have a filename extension. But without that the heuristic (for deciding text/binary) doesn't work with some TEDIT files with long ascii initial sections.
I think we may be getting burned by the LOOPSSPEEDUPS implementation of Make-Not-Reference-Counted
Interesting. I wonder what their strategy is, whether it is better than what we did in the LFG and finite state implementations. There we added BLCONS (backlink CONS—reference count only the CDR) and XCONS (Xpointers for both) datatypes surrounded by a substantial suite of the natural functions, i.s.oprs etc. to operate on them.
On Mar 11, 2022, at 12:41 PM, Nick Briggs @.***> wrote:
I think we may be getting burned by the LOOPSSPEEDUPS implementation of Make-Not-Reference-Counted
— Reply to this email directly, view it on GitHub https://github.com/Interlisp/medley/issues/733#issuecomment-1065506188, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQSTUJIXJDKGDHOHJGJ6L4LU7OVXDANCNFSM5QFFGKUA. You are receiving this because you commented.
This LOOPS code came from the envos 2.0 release. Before the "new" CDR-coding change. So if loops is mucking about with things at that level, the behavior is sure to be wrong. It's unlikely to be just a matter of recompiling or running interpreted.
In our BACKLINK module we use XPOINTER and FULLXPOINTER datatype field declarations. Would those be vulnerable to this system-level change? Seems to work.
It looks as though it's a non-reference-counted block of 100 POINTERs, they're not doing anything with list cells. Looking closer, though, I'm suspecting this: LLDATATYPE:
(PUTPROPS \GETDTD MACRO
((typeNum)
(ADDBASE \DTDSpaceBase (ITIMES typeNum 18))))
)
and LOOPSSPEEDUP:
(PUTPROPS \GETDTD MACRO ((typeNum)
(ADDBASE \DTDSpaceBase (LLSH typeNum 4))))
)
Something about the allocation changed to make it * 18 instead of * 16 to get the datatype.
Fixing that improved things a lot. I suppose that the definition is there at all so that the loops code can be compiled without needing to load EXPORTS.ALL to get the \GETDTD macro definition...?
Now I'm stuck on an apparently missing definition at load time for function $AV which appears to be a reader for ActiveValues.
It didn't define $AV when it loaded LOOPSACTIVEVALUE -- I think it tried to LOADCOMP it. Plain LOADing it got me the definition, but now it's failing on attempting to \PUTBASEBYTE a value of 2047 (in LOOPSWINDOW; --
(from URAID:)
13dc8 : 0x 50 0x300e *local* IL:PTR {IL:\PTRHUNK6}0x50300e
13dca : 0x e 0x 1 *local* IL:DISP 1
13dcc : 0x e 0x 7ff *local* CL:BYTE 2047
from
(METH Window ChoiceMenu (itemCV)
(* Create a menu which allows subitems to be displayed. Cache it in the menus field of the
browser under the property name itemCV)
(category (Window)))
I think what I'm seeing is that this implementation of \RPLPTR:
(\RPLPTR
[LAMBDA (OBJ OFFSET VAL) (* lmm " 3-NOV-81 12:10")^F^A
(UNINTERRUPTABLY
(\ADDREF VAL)
(\DELREF (\GETBASEPTR (SETQ OBJ (ADDBASE OBJ OFFSET))
0))
(\PUTBASEBYTE OBJ 1 (\HILOC VAL)) (* ; "\PUTBASEPTR smashes the high byte")
(\PUTBASE OBJ 1 (\LOLOC VAL))
VAL)])
is failing when it is asked to deposit the pointer for a (happens to be a LISTP) VAL whose \HILOC is 2047.
It's for the old memory allocation that had 24 bit addresses not 28. It was never tested interpreted.
Compiled calls to \RPLPTR turn to opcodes ADDBASE and \RPLPTR.N 0:
OK. After some messing about to deal with another issue:
If one file "A" LOADCOMPs another "B" then FILESLOAD later considers that "B" is loaded even if it was doing a (FILESLOAD (SOURCE) B) -- which means that the functions defined in "B" aren't present when they should be.
In a 32 MB vmem I can load LOOPS sources -- now I don't know what to do with it. I guess I should recompile all the source and then see if the compiled files will load into a larger (256 MB) image.
I've recompiled all the LOOPS sources, except for a couple of patch files, which I still need to investigate. The DFASLs will load into a 256 MB image now. Still need to be able to do the compilation in a 32 MB image where the LOOPS source can be loaded because of the calls to \RPLPTR. Still some rough edges around its directory setup.
But now that the \RPLPTR problem is fixed, most of this seems like a wild goose chase...
there are other problems with loading Loops sources with the # readmacro.
There's a comment in LOOPSPRINTCOMS about how it used to call SETALLSYNTAX to set "all" the system readtables to make #, and other things use the Loops interpretation. You don't want to set the syntax of # in any of the common readtables. But there are lots of places that use (CONCAT LoopsReadMacro ",") in order to print things in a way that they're supposed to read in again. Common Lisp and interlisp use # for something else.
Change LoopsReadMacroChar to some other value? Only change it in OLD-INTERLISP-FILE and make sure that's set in their MAKEFILE-ENVIORNMENT?
Only run Loops with OLD-INTERNET-T?
This would for sure mess up the examples in the manual.
the "problem" with the readtable was a red herring (or a wild goose?).
@nbriggs I tried this again with a LISP.SYSOUT with no Greet and the files in the 'before' branch and had the same result of trying to read in an @AV expression. That was confusing me, I thought I had somehow made a mistake and introduced a problem.
It is reasonable to believe that Envos/Venue rarely if ever tried loading and running Loops interpreted because it would have been too slow.
I really want to get Truckin going... when making players it generates players whose name is NIL -- object names seem to be stored in a hash table.
I don't believe that people who worked on the code at PARC didn't load all the source -- they wouldn't run much of it interpreted, but the option there to (LOADLOOPS NIL 'SOURCE) isn't just there to cause trouble.
A simple experiment: I did a BREAK(DOFILESLOAD), loaded LOADLOOPS, and called (LOADLOOPS NIL 'SOURCE). I let the DOFILESLOAD through the (null) two first calls to DOFILESLOAD, and the third one to pick up the main call with LOOPSFILES as the FILES argument. After that, all of the LOADCOMPs and other loads I just did a RETURN NIL on them except the call when it's doing a LOADCOMP of LLDATATYPE and LLGC which I just turned into loading SYSEDIT.
With that, all of Loops could load with no problem. So I think it's just overconstrained. Is there any reason to want to load some of loops and have the dependencies expressed this way?
I'm less sure how to turn off the LOADCOMP/LOAD problem in the sources but leave it so that CLEANUP will do the right thing.
Topic: "LOADING LOOPS SOURCE" Start Time: Aug 18, 2022 05:18 AM Pacific Time (US and Canada)
Meeting Recording: https://us06web.zoom.us/rec/share/b4HW7llBb6IYjYt9lYW6DFHvw9WO0zJ6UFe90ft-euDt3gmgv55sKKtcXowlB1vC.Wn7ImRMli26SMus1
Access Passcode: R6SNqh+G
LOADLOOPS.TXT is a Dribble of which interior DOFILESLOAD should load. You should also load EXPORTS.ALL (or load SYSEDIT) before loading LOOPSSPEEDUP.
I think I will close this because the main problem of loading enough loops in source form is solved; will continue with other loops puzzles instead.
I still believe it is unexpected/wrong behavior that if you (FILESLOAD (LOADCOMP) X) and then do (FILESLOAD (SOURCE) X) that it does NOT load the complete source because (some part of it) is already loaded.
for now I just added a DONTEVAL@LOAD around/before the FILES (LOADCP) of other LOOPSFILES. I also changed the LOADCOP/FILES of LLGC and LLDATATPES to just a DONTEVAL@LOAD (SOURCE) SYSEDIT since I was worried. with https://github.com/Interlisp/loops/pull/14 it's possible to cd loops/system
(FILESLOAD LOADLOOPS)
(LOADLOOPS NIL 'SOURCES)
(FOR X IN LOOPSFILES DO (CL:COMPILE-X))
(FOR X IN LOOPFILES DO (LOAD (CONCAT X ".DFASL")
ETC
With Interlisp/loops#14 you can load it into a LISP.SYSOUT, but loading it into a FULL.SYSOUT breaks with this problem:
