[PATCH 5.5.660] Not OK: perl v5.5.660 on powerpc-machten 4.1.4 (UNINSTALLED)

[p5pRT]

Migrated from rt.perl.org#2223 (status was 'resolved')

Searchable as RT2223$

[p5pRT]

From domo@computer.org

The following patch applies to otherwise-untweaked 5.5.660 on top of
the patch in my mail of 25th February "Re​: [PATCH 5.5.660] squash
malloc.c compile warnings". It does three things​:

1. Fixes a bug in malloc.c which prevented a second realloc of the
  same buffer from working when -DPLAIN_MALLOC and -DNO_FANCY_MALLOC
  are defined. (And possibly at other times too. But perl's malloc
  doesn't work on MachTen unless they're defined, so I can't tell.)

2. Removes MachTen-specific code from malloc.c, replacing it with
  a hints file definition if necessary (which it isn't, on recent
  MachTens).

3. Updates the MachTen hints file, to reflect point 2, and to enable
  System V IPC, which finally works in MachTen 4.1.4.

With these changes, MachTen passes all tests except pragma/warning
#235. (See Not OK report of 23rd February and Gurusamy Sarathy's
reply of the 25th.)

Inline Patch

--- perl5.5.660/malloc.c-fixed-compile-warn	Mon Feb 28 21:41:14 2000
+++ perl5.5.660/malloc.c	Mon Feb 28 21:46:37 2000
@@ -1743,7 +1743,8 @@ Perl_realloc(void *mp, size_t nbytes)
 		nmalloc[bucket]--;
 		nmalloc[pow * BUCKETS_PER_POW2]++;
 #endif 	    
-		*(cp - M_OVERHEAD) = pow * BUCKETS_PER_POW2; /* Fill index. */
+		((union overhead *)(cp - M_OVERHEAD))->ov_index
+		  = pow * BUCKETS_PER_POW2; /* Fill index. */
 		MALLOC_UNLOCK;
 		goto inplace_label;
 	    } else {
@@ -1991,17 +1992,6 @@ Perl_dump_mstats(pTHX_ char *s)
 
 #   if defined(__MACHTEN_PPC__) || defined(NeXT) || defined(__NeXT__) || defined(PURIFY)
 #      define PERL_SBRK_VIA_MALLOC
-/*
- * MachTen's malloc() returns a buffer aligned on a two-byte boundary.
- * While this is adequate, it may slow down access to longer data
- * types by forcing multiple memory accesses.  It also causes
- * complaints when RCHECK is in force.  So we allocate six bytes
- * more than we need to, and return an address rounded up to an
- * eight-byte boundary.
- *
- * 980701 Dominic Dunlop <domo@computer.org>
- */
-#      define SYSTEM_ALLOC_ALIGNMENT 2
 #   endif
 
 #   ifdef PERL_SBRK_VIA_MALLOC
--- perl5.5.660/hints/machten.sh-as-delivered	Sat Sep  4 21:01:49 1999
+++ perl5.5.660/hints/machten.sh	Thu Feb 24 21:41:38 2000
@@ -1,18 +1,23 @@
+#! /bin/bash
 # machten.sh
-# This is for MachTen 4.0.3.  It might work on other versions and variants too.
+# This is for MachTen 4.1.4.  It might work on other versions and variants
+# too.  If it doesn't, tell me, and I'll try to fix it -- domo@computer.org
 #
 # Users of earlier MachTen versions might need a fixed tr from ftp.tenon.com.
 # This should be described in the MachTen release notes.
 #
 # MachTen 2.x has its own hint file.
 #
-# This file has been put together by Andy Dougherty
+# The original version of this file was put together by Andy Dougherty
 # <doughera@lafcol.lafayette.edu> based on comments from lots of
 # folks, especially 
 # 	Mark Pease <peasem@primenet.com>
 #	Martijn Koster <m.koster@webcrawler.com>
 #	Richard Yeh <rcyeh@cco.caltech.edu>
 #
+# Remove dynamic loading libraries from search; enable SysV IPC with
+# MachTen 4.1.4 and above; define SYSTEM_ALIGN_BYTES for old MT versions
+#                      -- Dominic Dunlop <domo@computer.org> 000224
 # Disable shadow password file access: MT 4.1.1 has necessary library
 # functions, but not header file (or documentation)
 #                      -- Dominic Dunlop <domo@computer.org> 990804
@@ -71,8 +76,16 @@
 # by -DPLAIN_MALLOC and -DNO_FANCY_MALLOC.
 usemymalloc=${usemymalloc:-y}
 
+# Older versions of MachTen malloc() data on a two-byte boundary, which
+# works, but slows down operations on long, float and double data.
+# Perl's malloc() can compensate if SYSTEM_ALLOC_ALIGNMENT is suitably
+# defined.
+if expr "$osvers" \< "4.1" >/dev/null
+then
+system_alloc_alignment=" -DSYSTEM_ALLOC_ALIGNMENT=2"
+fi
 # Do not wrap the following long line
-malloc_cflags='ccflags="$ccflags -DPLAIN_MALLOC -DNO_FANCY_MALLOC -DUSE_PERL_SBRK"'
+malloc_cflags='ccflags="$ccflags -DPLAIN_MALLOC -DNO_FANCY_MALLOC -DUSE_PERL_SBRK$system_alloc_alignment"'
 
 # When MachTen does a fork(), it immediately copies the whole of
 # the parent process' data space for the child.  This can be
@@ -153,19 +166,22 @@
 # friends.  Use setjmp and friends instead.
 expr "$osvers" \< "4.0.3" > /dev/null && d_sigsetjmp='undef'
 
-# System V IPC support in MachTen 4.1 is incomplete (missing msg function
+# System V IPC before MachTen 4.1.4 is incomplete (missing msg function
 # prototypes, no ftok()), buggy (semctl(.., ..,  IPC_STATUS, ..) hangs
-# system), and undocumented.  Claim it's not there until things improve.
+# system), and undocumented.  Claim it's not there at all before 4.1.4.
+if expr "$osvers" \< "4.1.4" >/dev/null
+then
 d_msg=${d_msg:-undef}
 d_sem=${d_sem:-undef}
 d_shm=${d_shm:-undef}
+fi
 
 # Get rid of some extra libs which it takes Configure a tediously
-# long time never to find on MachTen
+# long time never to find on MachTen, or which break perl
 set `echo X "$libswanted "|sed -e 's/ net / /' -e 's/ socket / /' \
     -e 's/ inet / /' -e 's/ nsl / /' -e 's/ nm / /' -e 's/ malloc / /' \
     -e 's/ ld / /' -e 's/ sun / /' -e 's/ posix / /' \
-    -e 's/ cposix / /' -e 's/ crypt / /' \
+    -e 's/ cposix / /' -e 's/ crypt / /' -e 's/ dl / /' -e 's/ dld / /' \
     -e 's/ ucb / /' -e 's/ bsd / /' -e 's/ BSD / /' -e 's/ PW / /'`
 shift
 libswanted="$*"
@@ -213,6 +229,7 @@
         Propagating recommended variable nmopts
         Propagating recommended variable malloc_cflags...
         Propagating recommended variable reg_infty
+        Propagating recommended variable system_alloc_alignment
 Read the File::Find documentation for more information about dont_use_nlink
 
 Your perl will be built with a stack size of ${stack_size}k and a regular

Perl Info

Site configuration information for perl v5.5.660:

Configured by domo at Thu Feb 24 17:07:31 WET 2000.

Summary of my perl5 (revision 5.0 version 5 subversion 660) configuration:
  Platform:
    osname=machten, osvers=4.1.4, archname=powerpc-machten
    uname='machten ppp100 5 0.5 powerpc '
    config_args='-Doptimize=-g -de'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=undef use5005threads=undef useithreads=undef
    usesocks=undef useperlio=undef d_sfio=undef
    use64bits=undef uselargefiles=define usemultiplicity=undef
  Compiler:
    cc='cc', optimize='-g', gccversion=2.8.1
    cppflags='-DNOTDEF_MACHTEN -DREG_INFTY=2047 -DDEBUGGING -I/usr/local/include'
    ccflags ='-DNOTDEF_MACHTEN -DREG_INFTY=2047 -DDEBUGGING -I/usr/local/include'
    stdchar='char', d_stdstdio=undef, usevfork=true
    intsize=4, longsize=4, ptrsize=4, doublesize=8
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=8
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=4
    alignbytes=8, usemymalloc=y, prototype=define
  Linker and Libraries:
    ld='ld', ldflags =' -Xlstack=1048576 -L/usr/local/lib'
    libpth=/usr/local/lib /usr/lib
    libs=-lndbm -lgdbm -ldb -lm -lc
    libc=/usr/lib/libc.a, so=so, useshrplib=false, libperl=libperl.a
  Dynamic Linking:
    dlsrc=dl_none.xs, dlext=none, d_dlsymun=undef, ccdlflags=''
    cccdlflags='', lddlflags=''

Locally applied patches:
    


@INC for perl v5.5.660:
    lib
    /usr/local/lib/perl5/5.5.660/powerpc-machten
    /usr/local/lib/perl5/5.5.660
    /usr/local/lib/perl5/site_perl/5.00557/powerpc-machten
    /usr/local/lib/perl5/site_perl/5.00557
    /usr/local/lib/perl5/site_perl/5.00556/powerpc-machten
    /usr/local/lib/perl5/site_perl/5.00556
    /usr/local/lib/perl5/site_perl
    .


Environment for perl v5.5.660:
    HOME=/home/domo
    LANG (unset)
    LANGUAGE (unset)
    LD_LIBRARY_PATH=/usr/lib
    LOGDIR (unset)
    PATH=/sbin:/usr/sbin:/home/domo/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11/bin:/usr/libexec
    PERL_BADLANG (unset)
    SHELL=/bin/sh
-- 
Dominic Dunlop

[p5pRT]

From @gsar

On Mon, 28 Feb 2000 22​:44​:02 +0100, Dominic Dunlop wrote​:

+++ perl5.5.660/malloc.c Mon Feb 28 21​:46​:37 2000
@​@​ -1743,7 +1743,8 @​@​ Perl_realloc(void *mp, size_t nbytes)
nmalloc[bucket]--;
nmalloc[pow * BUCKETS_PER_POW2]++;
#endif
- *(cp - M_OVERHEAD) = pow * BUCKETS_PER_POW2; /* Fill index. */
+ ((union overhead *)(cp - M_OVERHEAD))->ov_index
+ = pow * BUCKETS_PER_POW2; /* Fill index. */
MALLOC_UNLOCK;
goto inplace_label;
} else {

This appears to result in a coredump on HP-UX. I've applied the rest of your
patch.

Sarathy
gsar@​ActiveState.com

[p5pRT]

From domo@village.uunet.lu

At 09​:02 -0800 2000-02-29, Gurusamy Sarathy wrote​:

This appears to result in a coredump on HP-UX. I've applied the rest of your
patch.

Well, as far as MachTen is concerned, it's the key to the whole
thing​: it cures a core-dump with a hair trigger. What's to be done
(bearing in mind, of course, that there are rather more HP than
MachTen hosts out there)?

Ilya, your take on this, please. AFAICT, the logic was busted​: the
location in the malloc overhead union being updated prior to the
patch was the magic number byte; the correct byte to update is the
index, one further on. In fact, I first wrote the change as

+ *(cp - M_OVERHEAD+1) = pow * BUCKETS_PER_POW2; /*
Fill index. */

but decided to rewrite it with a cast and member reference the
better to show what is going on (as is generally done elsewhere in
malloc.c).

Again, AFAICT, the problem was exposed for MachTen by this change
from Ilya's "[PATCH 5.005_61] Malloc fixes and docs" of 24 Sep 1999
(http​://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/1999-09/msg01172.html)​:

@​@​ -1576,18 +1678,22 @​@​ Perl_realloc(void *mp, size_t nbytes)
newarena = (1 << pow) + POW2_OPTIMIZE_SURPLUS(pow *
BUCKETS_PER_POW2);
require = newarena - onb - M_OVERHEAD;

- if (getpages_adjacent(require)) {
+ MALLOC_LOCK;
+ if (cp - M_OVERHEAD == last_op /* We *still* are the last chunk */
+ && getpages_adjacent(require)) {
#ifdef DEBUGGING_MSTATS
nmalloc[bucket]--;
nmalloc[pow * BUCKETS_PER_POW2]++;
#endif
*(cp - M_OVERHEAD) = pow * BUCKETS_PER_POW2; /* Fill index. */
[end of patch chunk omitted]

Before the extra clause was added to the if (), the code it enclosed
was never exercised on MachTen. FWIW, turning RCHECK on also results
in the code not being reached on MachTen​: realloc is always done "the
hard way".

The test program to tickle the problem on a DEBUGGING, NO_RCHECK perl
for MachTen is​:

./miniperl -e '$^D |= 128; $x .= "x" x 2044 for (1..3)'

which causes two reallocs for $x. Before applying my fix, the second
crapped out saying "Bad realloc" because it found an index value
where it expected to find a magic number. (Why perl subsequently
crashed, I didn't investigate.) (Before you ask, no it's not an
optimizer problem.)
--
Dominic Dunlop

[p5pRT]

From ilya@math.ohio-state.edu

On Tue, Feb 29, 2000 at 08​:40​:47PM +0100, Dominic Dunlop wrote​:

This appears to result in a coredump on HP-UX. I've applied the rest of your
patch.

Well, as far as MachTen is concerned, it's the key to the whole
thing​: it cures a core-dump with a hair trigger. What's to be done
(bearing in mind, of course, that there are rather more HP than
MachTen hosts out there)?

Ilya, your take on this, please. AFAICT, the logic was busted​: the
location in the malloc overhead union being updated prior to the
patch was the magic number byte; the correct byte to update is the
index, one further on. In fact, I first wrote the change as

+ *(cp - M_OVERHEAD+1) = pow * BUCKETS_PER_POW2; /*
Fill index. */

Aha, I know why this is happening. See layouts near line 564​: index
is always in the leading byte. This contradict the layout of ovu
(which is used without PACK_MALLOC only, and any system with
sane/fixable sbrk() uses PACK_MALLOC).

Please try exchanging ovu_index and ovu_magic *without* the above patch.

Ilya

[p5pRT]

From domo@computer.org

At 14​:56 -0500 2000-02-29, Ilya Zakharevich wrote​:

Please try exchanging ovu_index and ovu_magic *without* the above patch.

Yup, that's it. Thank you. Here's the patch​:

Inline Patch

--- perl5.5.660/malloc.c~       Tue Feb 29 21:20:34 2000
+++ perl5.5.660/malloc.c        Tue Feb 29 21:24:09 2000
@@ -435,8 +435,8 @@ union       overhead {
         double  strut;                  /* alignment problems */
  #endif
         struct {
-               u_char  ovu_magic;      /* magic number */
                 u_char  ovu_index;      /* bucket # */
+               u_char  ovu_magic;      /* magic number */
  #ifdef RCHECK
                 u_short ovu_size;       /* actual block size */
                 u_int   ovu_rmagic;     /* range magic number */

End of patch

Let's hope that does not give systems with real sbrk() indigestion.

MachTen still fails pragma/warning #235 as before. Next on my list.
--
Dominic Dunlop