Skip to content

Rmk88 split screen updates and color, eliminate reliance on STREAMHINT xpointer #2119

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 70 commits into
base: master
Choose a base branch
from
Open

Rmk88 split screen updates and color, eliminate reliance on STREAMHINT xpointer #2119

wants to merge 70 commits into from

Conversation

rmkaplan
Copy link
Contributor

As usual, a lot of files changed. Internal cleanup was mostly aimed at eliminating all internal dependencies on the STREAMHINT xpointer from the TEXTOBJ to the TEXTSTREAM. The xpointer is still there, against the possibility of client code still getting the TEXTOBJ object. But that code must separately hold on to the stream, else it will get collected.

This presents a menu-based interface for splitting and unsplitting windows, without reserving the right-edge slice of the window for that purpose (and the mouse tracking and cursor switching to a little square box). The old behavior is still there, controlled by the variable TEDIT.OP.WIDTH (was 12, near 0).

This supports color (although untested) as a character-looks property COLOR, BLACK by default.

See TEDIT-RELEASENOTES.

@rmkaplan
Copy link
Contributor Author

I changed the filel loadup order to avoid the "undefined" messages in fuller.database

Also, another internal change worth mentioning is to the THISLINE vector: Instead of in effect runcoding the character-looks changes across a line, there is now a separate field for the character looks of each character. So THISLINE now supports random access, making it easier to do backward adjustments (eventually for kerning). In a next version, I think the running X position should be stored as a separte field so that the displayer doesn't compute it on the fly from the width.

MattHeffron
MattHeffron reviewed Apr 24, 2025
View reviewed changes
library/tedit/TEDIT-FNKEYS
@@ -88,13 +90,13 @@
(* ;; "")

(if (LISTP CODE)
then (CONS (CHARNAME (CAR CODE)
then (CONS (CHARCODE.ENCODE (CAR CODE)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're not passing the NONCHARIDENTITY to the recursive calls.

MattHeffron
MattHeffron reviewed Apr 24, 2025
View reviewed changes
library/tedit/TEDIT-FNKEYS Outdated
@@ -108,21 +110,23 @@
else (LET ((CHARSET (LRSH CODE 8))
(CHAR (LOGAND CODE 255))
(ASCIICODE (LOGAND CODE 127))
CSETNAME CHARNAME ASCIINAME)
CSETNAME CHARCODE.ENCODE ASCIINAME)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renaming the variable here seems like a chance for confusion.

@MattHeffron MattHeffron mentioned this pull request Apr 24, 2025
Closed
@rmkaplan
Copy link
Contributor Author

Updated CHARCODE.ENCODE to pass NONCHARIDENTITY in recursive calls, also reverted internal variable name to CHARNAME (previous replace was over-eager)

@pamoroso
Copy link
Contributor

On Linux Mint 22.1 Cinnamon I tested these changes up to a8f2fcc.

Window splitting works well but I still notice the mouse pointer turns to a square when hovering close to the right edge of the window. I don't mind this and it causes no issues, it's just an observation.

I get the error TEDIT.OP.WIDTH is an unbound variable. when evaluating TEDIT.OP.WIDTH.

@rmkaplan
Copy link
Contributor Author

It's \TEDIT.OP.WIDTH (backslash), sorry for the mistake.

I set it to 0 to turn it off by default, I think now I have to set it to -1 to avoid it still popping up.

@pamoroso
Copy link
Contributor

pamoroso commented Apr 24, 2025 •
edited
Loading

After experimenting a bit with window splitting as mentioned earlier I executed the Quit title bar command of the TEdit window I was experimenting with. I experimented with the file TEDIT-RELEASENOTES.TEDIT open in TEdit divided into two panes but at the end of the session I unsplit the lower one. Yet after Quit TEdit quit in the state shown in this screenshot:

tedit-split-window

@pamoroso
Copy link
Contributor

I confirm \TEDIT.OP.WIDTH works.

A clarification on my latest post: to quit TEdit I used the Quit command, not Exit.

@rmkaplan
Copy link
Contributor Author

rmkaplan commented Apr 24, 2025 via email

@pamoroso
Copy link
Contributor

I did something similar in the test session and unsplit the lower pane prior to Quitting. In the session I selected some text and copied some to the clipboard. I took no further actions or executed other commands.

By the way, other than the missing title bar the leftover window was an ordinary TEdit window with the document still intact and scrollable.

@pamoroso
Copy link
Contributor

I updated to 1ceb1a4 and tested again by opening TEDIT-RELEASENOTES.TEDIT in TEdit.

I split and unsplit the window several times into up to 4 panes. At some point, when unsplitting the panes one by one to return to one pane, I got a break window with this error:

INTERLISP-ERROR

In ERROR:
ARG NOT TEXTOBJ
NIL

The TEdit window disappeared leaving this corrupted screen:

tedit-split-error

The context of the error:

MOUSE/11(debug)BTV
   MESS1 "ARG NOT TEXTOBJ"
   MESS2 NIL
   NOBREAK NIL
ERROR
\LISPERROR
\DTEST.UFN
   PANE {WINDOW}#122,40000
\TEDIT.CURSORMOVEDFN
   FNLST \TEDIT.CURSORMOVEDFN
   WINDOW {WINDOW}#122,40000
   CHECKFORDON'TFLG NIL
DOUSERFNS
   SI::*DUMMY-FOR-CATCH* T
   SI::*CATCH-RETURN-FROM* (&)
WINDOW.MOUSE.HANDLERA0032
   \MHCOM \TEDIT.CURSORMOVEDFN
   \MHPROCESS NIL
   \MHWINDOW {WINDOW}#122,40000
   \MOUSEBUSY T
   SI::NLSETQ-VALUE NIL
   *PROCEED-CASES* (&)
   SI::*NLSETQFLAG* NIL
WINDOW.MOUSE.HANDLER
   \OLDTTY #<Process OLDMOUSE#5/121,127306>
   \MOUSEBUSY NIL
\MOUSE.PROCESS
   *FORM* (\MOUSE.PROCESS)
   *ARGVAL* NIL
   *TAIL* NIL
   *FN* \MOUSE.PROCESS
\EVALFORM
   %#FORM# (\MOUSE.PROCESS)
   *CURRENT-PROCESS* #<Process MOUSE/121,127204>
   HELPFLAG BREAK!
   \CURRENTDISPLAYLINE 0
   \#DISPLAYLINES 40
   \LINEBUF.OFD #<IO Stream on T/174,72500>
   *READTABLE* #<ReadTable INTERLISP/174,71714>
   \PRIMTERMTABLE {TERMTABLEP}#174,64700
   \PRIMTERMSA {CHARTABLE}#167,74404
   TtyDisplayStream 
#<Output Display Stream/173,157000>
   SI::*RESETFORMS* NIL
   \INTERRUPTABLE T
   \TTYWINDOW NIL
   READBUF NIL
   \TERM.OFD #<Output Stream on T/170,120300>
   *STANDARD-OUTPUT* #<Output Stream on T/170,120300>
   *STANDARD-INPUT* #<IO Stream on T/174,72500>
\MAKE.PROCESS0
T

Medley head is mk88--Split-screen-updates-and-color:

commit 1ceb1a42c23cd05d61451f5234b76d4675b44027
Author: rmkaplan <[email protected]>
Date:   Thu Apr 24 10:05:37 2025 -0700

    Make the split region really go away by default

Maiko head is master:

commit 4994982a90565ae18d540eb402047a9d6ef5713d
Author: Matt Heffron <[email protected]>
Date:   Tue Feb 11 20:03:01 2025 -0800

    Adds the non-keypad arrow keys to be detected.
    With additional Medley calls to KEYACTION (below), they should work in TTYIN (i.e., XCL and Interlisp EXECs).
    I didn't try TEDIT.
    They are unrecognized in SEDIT. (In any case, their use there probably is pretty restricted by the nature of SEDIT.)
    
    (KEYACTION 'KEYPAD8 '(("Meta,^" 56 NOLOCKSHIFT) . IGNORE))

@pamoroso
Copy link
Contributor

I forgot to mention that I run Medley under VNC.

@rmkaplan
Now with an explicit test for -1
a43a80c 
Maybe that will really turn off the splitting region.
@pamoroso
Copy link
Contributor

I've been testing the past few commits up to a43a80c and in all of them experienced issues and errors when Ctrl-selecting text for deletion, for example Ctrl-middle-click to delete a word.

Sometimes the text is highlighted in reverse but nothing happens when releasing Ctrl and the mouse button. In such cases I Ctrl-middle-click the same text again and get an error. For example, in TEdit I wrote the text:

one

two

three

four

five

Then I Ctrl-middle-clicked each word and blank line until five was only highlighted in reverse with no deletion. Ctrl-middle-clicking again yielded this break window with the error:

TYPE-MISMATCH

In CL::%%NOT-NUMBER-ERROR:
{CL100/22470:Gacha10} is not a number

tedit-delete

The context:

MOUSE/5(debug)BTV
CL::%%NOT-NUMBER-ERROR
\SLOWPLUS2
   LINE {L82/9640: *5-5* FED}
   X 15
   NEWSEL {SEL: unset 101/27238}
   SELOPERATION DELETE
   TSTREAM #<IO Text Stream/130,5500>
   BUTTON MIDDLE
   WORDSELFLG T
   TEXTOBJ {TEXTOBJ}#122,32600
   CHARSLOT NIL
   CLOOKS NIL
   CHNO 5
   X0 NIL
   XLIM 16
   DCH NIL
   SELCHAR NIL
   PASTRIGHT T
   THISLINE {THISLINE}#122,31764
   RIGHTHALF NIL
   CS {}#110,170512
   CHAR NIL
   CHARW {CL100/22470:Gacha10}
   CHARCL NIL
\TEDIT.SCAN.LINE
   X 30
   Y 366
   NEWSEL {SEL: unset 101/27238}
   TSTREAM #<IO Text Stream/130,5500>
   SELOPERATION DELETE
   PANE {WINDOW}#122,26664
   BUTTON MIDDLE
   CURSEL {SEL: 1-3-R � 101/27204}
   REGIONTYPE NIL
   TEXTOBJ {TEXTOBJ}#122,32600
   LINE {L82/9640: *5-5* FED}
   PARAFIRSTCHNO NIL
   PARALASTCHNO NIL
   SELFN NIL
\TEDIT.XYTOSEL
   SI::*CLEANUP-FORMS* SI::RESETUNWIND
   TTYPROC #<Process TEdit/174,24000>
   TSTREAM #<IO Text Stream/130,5500>
   X 30
   Y 366
   DS #<Output Display Stream/130,5700>
   OLDX 30
   OLDY 366
   PREG (0 0 482 380)
   TEXTOBJ {TEXTOBJ}#122,32600
   CURSEL {SEL: 1-3-R � 101/27204}
   NEWSEL {SEL: unset 101/27238}
   CUROPERATION DELETE
   NEWOPERATION DELETE
   PENDINGDEL NIL
   READONLY NIL
SI::*UNWIND-PROTECT*
   PANE {WINDOW}#122,26664
   LISPXHIST NIL
   SI::*RESETFORMS* ((& {CURSOR}#157,141334))
   RESETSTATE NIL
MODERN-ORIG-\TEDIT.BUTTONEVENTFN
   WINDOW {WINDOW}#122,26664
   ORIGFUNCTION MODERN-ORIG-\TEDIT.BUTTONEVENTFN
   ANYWHERE NIL
   TITLEPROPORTION NIL
   CORNERREGION (756 358 490 400)
   TOPMARGIN 12
   CORNER NIL
   ATTACHEDREGION NIL
MODERNWINDOW.BUTTONEVENTFN
   WINDOW {WINDOW}#122,26664
   ORIGFUNCTION MODERN-ORIG-\TEDIT.BUTTONEVENTFN
   ANYWHERE NIL
   TITLEPROPORTION NIL
   CORNERREGION (756 358 490 400)
   TOPMARGIN "* Tedit Window"
MODERNWINDOW.BUTTONEVENTFN
   W {WINDOW}#122,26664
   STREAM NIL
\TEDIT.BUTTONEVENTFN
   W {WINDOW}#122,26664
   STREAM NIL
\TEDIT.BUTTONEVENTFN
   SI::*DUMMY-FOR-CATCH* T
   SI::*CATCH-RETURN-FROM* (&)
\PROTECTED.APPLYA0001
   FN \TEDIT.BUTTONEVENTFN
   WINDOW {WINDOW}#122,26664
   SI::NLSETQ-VALUE NIL
   *PROCEED-CASES* (&)
   SI::*NLSETQFLAG* NIL
\PROTECTED.APPLY
   \MHCOM \TEDIT.BUTTONEVENTFN
   \MHPROCESS #<Process TEdit/174,24000>
   \MHWINDOW {WINDOW}#122,26664
   \MOUSEBUSY T
WINDOW.MOUSE.HANDLER
   \OLDTTY #<Process TEdit/174,24000>
   \MOUSEBUSY NIL
\MOUSE.PROCESS
   *FORM* (\MOUSE.PROCESS)
   *ARGVAL* NIL
   *TAIL* NIL
   *FN* \MOUSE.PROCESS
\EVALFORM
   %#FORM# (\MOUSE.PROCESS)
   *CURRENT-PROCESS* #<Process OLDMOUSE/174,24410>
   HELPFLAG BREAK!
   \CURRENTDISPLAYLINE 0
   \#DISPLAYLINES 40
   \LINEBUF.OFD #<IO Stream on T/174,72500>
   *READTABLE* #<ReadTable INTERLISP/174,71714>
   \PRIMTERMTABLE {TERMTABLEP}#174,64740
   \PRIMTERMSA {CHARTABLE}#174,65000
   TtyDisplayStream 
#<Output Display Stream/146,12300>
   SI::*RESETFORMS* NIL
   \INTERRUPTABLE T
   \TTYWINDOW NIL
   READBUF NIL
   \TERM.OFD #<Output Stream on T/170,116300>
   *STANDARD-OUTPUT* #<Output Stream on T/170,116300>
   *STANDARD-INPUT* #<IO Stream on T/174,72500>
\MAKE.PROCESS0
T

Aside from that I no longer get crashes when splitting and unsplitting. In addition the mouse pointer no longer turns into a square when hovering close to the right edge of the window.

@rmkaplan
Copy link
Contributor Author

I pushed some a few changes having to do with the menu bar behavior for tabs. I also fixed up some mentions of XCCS on the way to MCCS.

Note that there is currently a pdf hardcopy problem with this PR: The default color is BLACK, and that is one of the ways of specifying colors according to the documentation (man color). But postscript (\DSPCOLOR.PSC) only accepts a number between 0 and 1, which is not a legitimate way of specifying colors. So it fails.

Immediate fix is to MOVD(NILL \DSPCOLOR.PSC). But probably the argument checking should be installed in the generic DSPCOLOR, bringing in some of the predicates from LLCOLOR. And postscript eventually should be changed to approximate more of what can be specified.

@rmkaplan
Copy link
Contributor Author

@pamoroso I have not been able to replicate the faillure with your one-two... example.

The closest I have come is to type a single line "one" with no EOL into an empty window, then ctrl-middle-select. The screen doesn't get updated (the letters o n e are still there), and they disappear with redisplay or with typing in. So I think I missed a case where the line should be updated, but it doesn't cause an error.

Is there any more specific recipe to get the break?

@MattHeffron
Copy link
Contributor

@rmkaplan See my PR #2127 for my attempt to support correct color specifications.

@rmkaplan
Copy link
Contributor Author

@pamoroso I believe I have fixed the screen update for an edit (delete) to the top line. I still don't get a break.

@rmkaplan
TEDIT-STRESS: STRESS-INSERT now doesn't display
ba8d5af 
STRESS-INSERTWINDOW has the previous code
@nbriggs
Copy link
Contributor

nbriggs commented Jun 4, 2025

I tried running with the current maiko master lde/ldex - started a fresh full.sysout, I have very little in my init, just loaded background-yield. In the XCL exec, (IL:LOAD "TEDIT-STRESS.LCOM") then CONN, then CONN projects/medley/lispusers, then (IL:STRESSDISPLAY). It loaded DOC-OBJECTS.LCOM then DATEFORMAT-EDITOR.LCOM, and broke under IL:GDATE. Processing BITMAP-GALLERY.TEDIT.

If I open an Interlisp Exec window and do it from there -- no problem. If I interrupt it and go back to the XCL exec and try it again, it fails in the same way. Something is sensitive to where it's started from that probably shouldn't be!

image

@rmkaplan
Copy link
Contributor Author

rmkaplan commented Jun 4, 2025 via email

@rmkaplan
Copy link
Contributor Author

rmkaplan commented Jun 4, 2025 via email

@rmkaplan
Copy link
Contributor Author

rmkaplan commented Jun 4, 2025 via email

@nbriggs
Copy link
Contributor

nbriggs commented Jun 4, 2025

I loaded PSEUDOHOSTS, set up MEDLEY to point to {dsk}<Users>briggs>Projects>medley>, edited DIRECTORIES to use {medley}<lispusers> and so on, then loaded (latest) TEDIT-STRESS.LCOM, connected to {medley}<lispusers> and started STRESSINSERT -- it has run 488 cycles so far with no error. If it's PSEUDOHOSTS that's causing it, it's more complicated than one simple use.

@rmkaplan
TEDIT-STRESS: a little cleanup
1dae7c3 
STRESSINSERT/STRESSDELETE don't' bring up blank window, added STRESSDELETEWINDOW
@rmkaplan
Copy link
Contributor Author

rmkaplan commented Jun 4, 2025

One other thing: I'm also running with ARRAYBLOCKCHECKING set to T, in addition to all the explicit calls to \PARSEARRAYSPACE.

@nbriggs
Copy link
Contributor

nbriggs commented Jun 4, 2025

OK - ARRAYBLOCKCHECKING set to T and it fails for me - but in the Lisp implementation of \PARSEARRAYSPACE, not the Maiko array space checking in gcfinal.c -- I wonder if the code matches.

@rmkaplan
Copy link
Contributor Author

rmkaplan commented Jun 4, 2025

It was suggested in the meeting today that I try it with PSEUDOHOSTS loaded but without the default setting up of the LI pseudohost. That didn't make a difference.

It was also suggested that I put out the code if anyone wants to test on other platforms. So:

Checkout rmk88 #2119 and build a full loadup. Start it up (with or without greeting), and define/load the following function:

(DEFINEQ (CRASH? (LAMBDA (ARRAYBLOCK PSEUDOHOST)
(SETQ ARRAYBLOCKCHECKING ARRAYBLOCK)
(CL:WHEN PSEUDOHOST
(LOAD 'PSEUDOHOSTS.LCOM))
(LOAD 'TEDIT-STRESS.LCOM)
(CNDIR (MEDLEYDIR "lispusers"))
(STRESSINSERT))))

Then execute CRASH? with or without the argument parameters. Defaults to setting ARRAYBLOCKCHECKING to NIL and not loading PSEUDOHOSTS. ARRAYBLOCK T may now be the bad case (fails for Nick too).

@rmkaplan
Copy link
Contributor Author

rmkaplan commented Jun 5, 2025 via email

@nbriggs
Copy link
Contributor

nbriggs commented Jun 5, 2025

I think the code is the same. I don't think that anything that happens during the Lisp array space check could cause the Maiko code to [de]allocate and modify the array space the Lisp code is looking at.

@rmkaplan
Copy link
Contributor Author

rmkaplan commented Jun 5, 2025 via email

@nbriggs
Copy link
Contributor

nbriggs commented Jun 6, 2025

I don't know if something is really wrong. I've got one here with mismatched header/trailer lengths, it's a block on the free-list. It looks wrong -- the header says the length is 98, the trailer says 228.

(arrayblock)  (arlen = 98, inuse = 0, gctype = 0, password = 5461, fwd = 4682046, bkwd = 4681358)

Near the beginning of this free block is what looks like the trailer of a another block that appears to be of length 228, and then the header of a block of length 90. I guess that's possible if blocks were merged.

It might be easier to reproduce if what it did wasn't totally random. Perhaps print the result of (RANDSET T)
once before it starts calling (RAND ...) so that we can go back and reset the random state the next time and in theory get exactly the same sequence of numbers for the inserts.

I think I need to rewrite the C code that dumps array blocks so that I can direcly read the forward/backward pointers (right now it labels the locations with the native address, but the forward/backward pointers in the data are Lisp pointers (relative to the Lisp world memory base).

@rmkaplan
Copy link
Contributor Author

rmkaplan commented Jun 6, 2025 via email

@rmkaplan
Copy link
Contributor Author

rmkaplan commented Jun 6, 2025

On each run the RAND state is saved (by SAVERANDSTATE) in a file RANDSTATE on the connected directory. To do another run with the previous randstate, set USELASTRANDSTATE to T.

@nbriggs
Copy link
Contributor

nbriggs commented Jun 6, 2025

With some more helpful output in the maiko "printarrayblock()" code -- what I'm seeing looks like this

free block@45bc72 header length 447 cells =>next block@45bff0, trailer@45bfee says length 447 cells
free block@45bf00 header length 120 cells =>next block@45bff0, prev block@45bc72
          trailer@45bfee says length 447 cells
used block@45bff0 header length 4052 cell =>next block@45df98, prev block@45bc72

so there's a free block at the end of another free block such that they share a trailer whose length matches the outer free block, and I'm not sure which block has a pointer to the interior free block - but someone did because it got checked and the missing/mismatched trailer caused the failure.

This says to me that there's a bug somewhere in the array block splitting/merging code in maiko. Sigh.

@rmkaplan
Copy link
Contributor Author

rmkaplan commented Jun 6, 2025 via email

@nbriggs
Copy link
Contributor

nbriggs commented Jun 7, 2025

The stack looked like this, this last time:

*Error* URAID Called:
Enter the URaid
"ARRAYBLOCK Header and Trailer length don't match"

< l u
  0 :    0x13fd6 : IL:RAID
  1 :    0x13fc0 : IL:\MP.ERROR
  2 :    0x13f9a : IL:\CHECKARRAYBLOCK
  3 :    0x13f76 : IL:\CHECKARRAYBLOCK
  4 :    0x13f5a : IL:\PARSEARRAYSPACE1
  5 :    0x13f44 : IL:\PARSEARRAYSPACE
  6 :    0x13f2e : IL:CHECKARRAYS
  7 :    0x13eec : IL:STRESSDISPLAY
  8 :    0x13ec6 : IL:\EVALFORM
  9 :    0x13ea8 : IL:FAULTEVAL
 10 :    0x13e8a : IL:\EVALFORM
 11 :    0x13e76 : IL:EVAL
 12 :    0x13e52 : IL:EVAL-INPUT
 13 :    0x13de0 : IL:DO-EVENT
 14 :    0x13daa : XCL:EXECA0001A0002
 15 :    0x13d72 : XCL:EXECA0001
 16 :    0x13ccc : IL:\DO.PROGV
 17 :    0x13c4e : XCL:EXEC
 18 :    0x13c22 : IL:\EVALFORM
 19 :    0x13c08 : CL:PROGN
 20 :    0x13be8 : IL:\EVALFORM
 21 :    0x13b9a : IL:\MAKE.PROCESS0
 22 :    0x11802 : CL:T

< f 1
IVAR -------
  13fb6 : 0x   e  0x  1b  IL:CODE  27
  13fb8 : 0x  7f  0x9db8  CL:STRING  "ARRAYBLOCK Header and Trailer length don't match"
  13fba : 0x  45  0xbf00  IL:ARG1  {unknown}0x45bf00
  13fbc : 0x   0  0x  4c  IL:ARG2  CL:T

## STACK BF at 0x13fbe ##
[cnt=0 ]
ivar : 0x3fb6
>> Bf's ivar says 0x13fb6 vs. IVar says 0x13fce
Fname is IL:\MP.ERROR
## STACK FX at 0x13fc0 ##
[cnt = 0 ]
 #alink           0x3fa4 
 fnhead   0x2e53c4 
 nextblock        0x3fce 
 pc               0x5d 
 nametbl  0x16bf00 
 #blink           0x0 
 #clink           0x0 
  13fca : 0x   0  0x   0  
  13fcc : 0x   0  0x  4c

@masinter
Copy link
Member

masinter commented Jun 7, 2025

  1. I still would like to know if this is a new problem (introduced after 2020) or an old one (there's a test that succeeds in an old one -- it fails in Venue Sysout.
  2. if the problem shows up when bad data is found on the free list, the thing we have to find is an operation where the free list is being modified in abnormal ways. Someone made the suggestion, which I think was a good idea, which was to fill the block with some well known bit pattern (unlikely to be written naturally), and then and then monitor every access to arrayblock data to check for the bit pattern.

@nbriggs
Copy link
Contributor

nbriggs commented Jun 7, 2025

I'm looking at it, and writing code (in C) to help me look at the structure of array space. It's not done yet, but I'm using it on an original Venue lisp.sysout.

@nbriggs
Copy link
Contributor

nbriggs commented Jun 9, 2025

Tried with CHECKARRAY set NIL and ARRAYBLOCKCHECKING set T -- it took a lot longer (39 rounds of STRESSDISPLAY, then 222 rounds of STRESSINSERT) - to get a header/trailer length mismatch, and it's (again) coming from the Lisp side \CHECKARRAYBLOCK. The backtrace shows:

< l u
  0 :    0x14334 : IL:RAID
  1 :    0x1431e : IL:\MP.ERROR
  2 :    0x142f8 : IL:\CHECKARRAYBLOCK
  3 :    0x142d4 : IL:\CHECKARRAYBLOCK
  4 :    0x142aa : IL:\ALLOCBLOCK.OLD
  5 :    0x14286 : IL:\MAIKO.ALLOCBLOCK
  6 :    0x14262 : IL:BITMAPCREATE
  7 :    0x14238 : IL:\READBINARYBITMAP
  8 :    0x14220 : IL:BMOBJ.GETFN3
  9 :    0x14200 : IL:READIMAGEOBJA0001
 10 :    0x141c8 : IL:READIMAGEOBJ
 11 :    0x141a0 : IL:\TEDIT.GET.OBJECT
 12 :    0x14154 : IL:\TEDIT.GET.PIECES3
 13 :    0x13fde : IL:\TEDIT.GET.FORMATTED.FILE
 14 :    0x13fa2 : SI:*UNWIND-PROTECT*
 15 :    0x13f86 : IL:\TEDIT.OPENTEXTSTREAM.PIECES
 16 :    0x13f50 : SI:*UNWIND-PROTECT*
 17 :    0x13f34 : IL:OPENTEXTSTREAM
 18 :    0x13eee : IL:STRESSINSERT

\MAIKO.ALLOCBLOCK is the implementation that's being run for the \ALLOCBLOCK call at the end of BITMAPCREATE.

A URaid "B" block dump shows (around the header)

     0x1109914d0 (0x  47aa68)        0
     0x1109914d4 (0x  47aa6a)        0
     0x1109914d8 (0x  47aa6c)        0
     0x1109914dc (0x  47aa6e) aaa802f6 <- array header
     0x1109914e0 (0x  47aa70) aaa90083
     0x1109914e4 (0x  47aa72)    60006
     0x1109914e8 (0x  47aa74)    60006
     0x1109914ec (0x  47aa76)    60006

and around the trailer

     0x110992090 (0x  47b048) 406403b2
     0x110992094 (0x  47b04a) 13640103
     0x110992098 (0x  47b04c) 9c640101
     0x11099209c (0x  47b04e) 6700001b
     0x1109920a0 (0x  47b050) cff092b1
     0x1109920a4 (0x  47b052)   fe0900
     0x1109920a8 (0x  47b054)   1525b2
     0x1109920ac (0x  47b056) 2a406700
     0x1109920b0 (0x  47b058)   1bcf0a <- array trailer
     0x1109920b4 (0x  47b05a)     10d3
     0x1109920b8 (0x  47b05c) b50382b1
     0x1109920bc (0x  47b05e)   e64168
     0x1109920c0 (0x  47b060) f0b40382
     0x1109920c4 (0x  47b062) b100dd40
     0x1109920c8 (0x  47b064) 6f005ebb
     0x1109920cc (0x  47b066) e80a0000

Clearly this item tagged as array trailer is not actually an array trailer - it's missing the 0xaaa8 bits (top 13 bits: #b1010101010101xxx) so the "header/trailer length mismatch" is an artifact of the length test coming after the trailer password test and it only recording the last error it detects.

I'm suspicious about the two cells both with array block passwords at the beginning of the block that faulted, and the second word - 0xaaa90083 if interpreted as an array block header leads to a matching array block trailer and all the blocks after that it seem like well-formed blocks.

@nbriggs
Copy link
Contributor

nbriggs commented Jun 12, 2025

Some progress -- fixed up "makefreearrayblock()" in Maiko to set the freed memory to 0xBAADF00D, and then ran the STRESSDISPLAY code, with all the checking on. Died in CHECKARRAYS with tag BEFOREDISPLAY, where the block it was checking on looked like:

< B 0x4703aa
This array block: 0x4703aa.  Previous: 0x452390.  Next: 0x48e3c4.
        password: 0x1755     gctype: 0x2   in use: 1
  Header  Length: 61453 cells.

 Trailer  Length: 0 cells.

     0x109733704 (0x  470382) baadf00d
     0x109733708 (0x  470384) baadf00d
     0x10973370c (0x  470386) baadf00d
     0x109733710 (0x  470388) baadf00d
     0x109733714 (0x  47038a) baadf00d
     0x109733718 (0x  47038c) baadf00d
     0x10973371c (0x  47038e) baadf00d
     0x109733720 (0x  470390) baadf00d
     0x109733724 (0x  470392) baadf00d
     0x109733728 (0x  470394) baadf00d
     0x10973372c (0x  470396) baadf00d
     0x109733730 (0x  470398) baadf00d
     0x109733734 (0x  47039a) baadf00d
     0x109733738 (0x  47039c) baadf00d
     0x10973373c (0x  47039e) baadf00d
     0x109733740 (0x  4703a0) baadf00d
     0x109733744 (0x  4703a2) baadf00d
     0x109733748 (0x  4703a4) baadf00d
     0x10973374c (0x  4703a6) baadf00d
     0x109733750 (0x  4703a8) baadf00d
     0x109733754 (0x  4703aa) baadf00d <- array header
     0x109733758 (0x  4703ac) baadf00d
     0x10973375c (0x  4703ae) baadf00d
     0x109733760 (0x  4703b0) baadf00d
     0x109733764 (0x  4703b2) baadf00d
     0x109733768 (0x  4703b4) baadf00d
     0x10973376c (0x  4703b6) baadf00d
     0x109733770 (0x  4703b8) baadf00d
     0x109733774 (0x  4703ba) baadf00d
     0x109733778 (0x  4703bc) baadf00d
     0x10973377c (0x  4703be) baadf00d
     0x109733780 (0x  4703c0) baadf00d
     0x109733784 (0x  4703c2) baadf00d
     0x109733788 (0x  4703c4) baadf00d
     0x10973378c (0x  4703c6) baadf00d
     0x109733790 (0x  4703c8) baadf00d
     0x109733794 (0x  4703ca) baadf00d
     0x109733798 (0x  4703cc) baadf00d
     0x10973379c (0x  4703ce) baadf00d
     0x1097337a0 (0x  4703d0) baadf00d
. . .
     0x10976f734 (0x  48e39a)        0
     0x10976f738 (0x  48e39c)        0
     0x10976f73c (0x  48e39e)        0
     0x10976f740 (0x  48e3a0)        0
     0x10976f744 (0x  48e3a2)        0
     0x10976f748 (0x  48e3a4)        0
     0x10976f74c (0x  48e3a6)        0
     0x10976f750 (0x  48e3a8)        0
     0x10976f754 (0x  48e3aa)        0
     0x10976f758 (0x  48e3ac)        0
     0x10976f75c (0x  48e3ae)        0
     0x10976f760 (0x  48e3b0)        0
     0x10976f764 (0x  48e3b2)        0
     0x10976f768 (0x  48e3b4)        0
     0x10976f76c (0x  48e3b6)        0
     0x10976f770 (0x  48e3b8)        0
     0x10976f774 (0x  48e3ba)        0
     0x10976f778 (0x  48e3bc)        0
     0x10976f77c (0x  48e3be)        0
     0x10976f780 (0x  48e3c0)        0
     0x10976f784 (0x  48e3c2)        0 <- array trailer
     0x10976f788 (0x  48e3c4)        0
     0x10976f78c (0x  48e3c6)        0
     0x10976f790 (0x  48e3c8)        0

so somebody had a pointer to block 0x4703aa, but this was smashed by being merged into another block.

The history looks like (cblock: check array block; other self-explanatory. reclaim ptr is the data address, so 2 lisp words after the actual block)

cblock@0x46fc26
reclaim: ptr 0x46fc28
makefreearrayblock@0x46fc26
cblock@0x46fc26
cblock@0x46fc26
cblock@0x4703aa
arrayblockmerger 0x46fc26 (962 cells) with following 0x4703aa (497 cells)  -- there's our block getting smashed.
makefreearrayblock@0x46fc26
cblock@0x46fc26
cblock@0x472078
reclaim: ptr 0x47207a
makefreearrayblock@0x472078
cblock@0x471fe0
arrayblockmerger 0x471fe0 (76 cells) with following 0x472078 (516 cells)
makefreearrayblock@0x471fe0
cblock@0x471fe0
cblock@0x471fe0
cblock@0x472480
cblock@0x472a76
reclaim: ptr 0x472a78
makefreearrayblock@0x472a76
cblock@0x47299a
arrayblockmerger 0x47299a (110 cells) with following 0x472a76 (529 cells)
makefreearrayblock@0x47299a
cblock@0x47299a
cblock@0x47299a
cblock@0x472e98

@rmkaplan
Copy link
Contributor Author

rmkaplan commented Jun 12, 2025 via email

@nbriggs
Copy link
Contributor

nbriggs commented Jun 12, 2025

I'll be trying it with STRESSINSERT. Got a lot on my schedule today so it may be late or tomorrow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@MattHeffron MattHeffron MattHeffron left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@rmkaplan @pamoroso @MattHeffron @nbriggs @masinter