CDE/cde/programs/dtksh/README-DEVELOPER

/* $XConsortium: README-DEVELOPER /main/3 1996/07/15 14:12:13 drk $ */

(NOTE: As of CDE 2.3.1 and later, this entire document is obsolete and only kept
 for historical purposes, the updating process has been streamlined to a simple
 two step process:
 1. cd ksh93; git pull
 2. Merge upstream init.c with ours, keeping all of our changes.
 and thats all there is to it.)

IMPORTANT!!!  PLEASE READ ON!!
------------------------------

The intention of this file is to outline the procedure to use, anytime
a new update of the ksh-93 source is received.  When you receive the
new source package, you will first need to do the following:

   0) mkdir ksh93-base
   1) copy the packing into this directory
   2) uudecode the package
   3) uncompress the package
   4) un-tar the package into ksh93-base (you may need to use
      cpio, if that is what Korn used to create his package).

Once the source ball has been un-tar'ed, go down into the hierarchy,
until you see a directory called 'ship' ; don't go down into the 'ship'
directory, simply stop at its parent.  Dave Korn uses his own mechanism
for creating the actual ksh-93 sources.  It's not like you can simply
look at what is in the 'ship' directory.  Instead, you need to run his
'shipin' tool, which takes his base source ball, and applies all of his
patches, thus generating the source tree.  To start this process, which
will result in the latest source files being generated, issue the following
commands:

   <First, log onto a 10.0 build machine, such as hpcvusp>
   /bin/ksh
   export SHELL=/bin/ksh
   nohup /bin/ksh ship/shipin -u -L&
   tail -f nohup.out

When this completes, you will notice that the current directory now
contains a series of other subdirectories; the only ones we really care
about now are the 'man', 'ship' and the 'src' subdirectories.  
The trick now is to determine which of the files in 
these subdirectories have changed, so that you can update the real 
dtksh/ksh93 source tree.  You will want to copy into the dtksh/ksh93/ship
directory the newest versions of the following files:

   shipiffe.sh
   shipout
   shipin
   shipproto.c
   shipop.c

Go into the 'shipin' file, and locate the several places where it attempts
to mail out information (I look for the word 'mail', until I find what
looks like the correct places).  Remove this code, since we don't want
a ton of mail messages happening each time we build.

Within the 'ship' directory, you will notice a series of subdirectories;
each one of these corresponds to one of the pieces making up the ksh93
client (i.e. there is one for each library: libast, libcmd, libvdelta, etc).
Within each of these subdirectories is another subdirectory, whose name
represents the date that source depot was created (i.e. 950619).  You will
need to recreate these dated directories, along with their contents, within
your dtksh tree (under dtksh/ksh93/ship).  Within each dated directory,
copy over (and check in) the items, copyright, UNCRATED and owner files.
Check in a dummy 'base' file also.  Doing these steps will allow the ksh
build to proceed, without it ever again attempting to uncrate new ksh93
sources.  The 'items' file is particularly important, since it indicates
which other libraries the indicated piece is dependent upon.

====================

In /clone/fred/DtkshTools I have a collection of scripts which make it easier 
to update a dtksh tree with new ksh sources.  The steps for determining which
files have changed are as follows:

1) First get a list of all of the files in the new source tree.  I do this 
   with the following steps:

       cd ksh93-base     (This is the tree you just untar'ed)
       find src -print > srcList

   This gives you a complete list of all of the source files.

2) Use this list to determine which source files are new or have changed:

      /clone/fred/DtkshTools/CheckKshSrc dtksh ksh93-base srcList > diffList

   The output generated by the above script will tell you which files have
   changed, which have not changed, and which have been added.  

3) You now need to take the diffList, and modify it so that it only
   contains the name of new or changed files within the ksh93 source
   tree:

      /clone/fred/DtkshTools/CreateTrimList diffList > trimList

4) Now, we will delete the files in the dtksh tree which we are going
   to replace; we do this before we copy over the new versions:

      /clone/fred/DtkshTools/TrimOldSrcs /clone/fred/dtksh trimList

5) We now need to check out (Rcs -l) the files we are going to replace:

      /clone/fred/DtkshTools/CheckOutSrcs /clone/fred/dtksh trimList

   If at any time you want to get a complete list of checked out files
   in your dtksh tree, issue the following commands:

      find /clone/fred/dtksh -type d > dirList
      /clone/fred/DtkshTools/RunRcslocks dirList

6) We are now ready to copy the new source files out of the ksh93-base
   directory, and into the dtksh directory:

      cd /clone/fred
      /clone/fred/DtkshTools/CopyNewSrcs ksh93-base dtksh trimList

====================

You're not done yet, however.  We have found it necessary to modify
several of the stock ksh sources, Makefiles and Mamfiles, to accommodate
problems we have encountered, or to enable options not turned on by
default, such as multibyte support.  You will need to make sure that
these changes get rolled back in, before checking in the new source.
The following source files need to have changes added:

   src/cmd/ksh93/Makefile  (*** THIS STEP IS OBSOLETE ***)
   ----------------------

      Change: MULTIBYTE ==
      To:     MULTIBYTE == 1

   src/cmd/ksh93/Mamfile  (*** THIS STEP IS OBSOLETE ***)
   ---------------------

      Change: setv MULTIBYTE -UMULTIBYTE
      To:     setv MULTIBYTE -DMULTIBYTE

   src/cmd/ksh93/OPTIONS  (*** THIS STEP IS OBSOLETE ***)
   ---------------------

      Change: MULTIBYTE=0
      To:     MULTIBYTE=1

***

The following two files must change, to prevent the ksh source from linking
in its own malloc package.  This creates problems on the IBM platform, due
to the way their libraries are built.  As it turns out, the IBM Xt library
already has all of its malloc/free/etc calls resolved to their standard
libc malloc functions.  If ksh defines its own malloc package, then dtksh
ends of effectively linked simultaneously to two different malloc packages.
This rapidly leads to a core dump, due to cross contamination of the 
memory pools (i.e. A value obtained by XtMalloc() passed to free() ). As
it turns out, this also creates problems on the HP platform, as was
recently discovered (Note that it appears that the problem which was
seen on the HP platform was due to a defect in the ksh93 memory allocation
code; this defect had not yet been fixed in the ksh93-d release.  Until
the time that this is fixed, we must continue to be sure to not link
in ksh93's malloc package).

   src/lib/libast/Makefile
   -----------------------

      Remove:  calloc.c malloc.c from the list of sources

   src/lib/libast/Mamfile
   ----------------------
      Remove:  calloc.o malloc.o from the archive line; DO NOT REMOVE
               ANY OF THE OTHER REFERENCES TO malloc OR calloc, AS
               THIS WILL CAUSE THE BUILD TO BREAK.

***

Another file which needs special handling is builtins.c, which is in
the base dtksh directory.  This file was first created by taking the
file 'dtksh/ksh93/src/cmd/ksh93/data/builtins.c', and adding into its
list of builtin commands, the list we added for X, Xt, Xm and Dt.  The
best way to update this file is to first goto 'dtksh/ksh93/src/cmd/ksh93/data'
and do an Rcsdiff, to find out what changes Dave Korn made.  You will
then make those same changes to the dtksh/builtins.c file.

   dtksh/builtins.c
   ----------------

      Fold in any changes made to dtksh/ksh93/src/cmd/ksh93/data/builtins.c

***

Another file which needs special handling is aliases.c, which is in
the base dtksh directory.  This file was first created by taking the
file 'dtksh/ksh93/src/cmd/ksh93/data/aliases.c', and adding into its
list of commands aliases, the list we added for X, Xt, Xm and Dt.  The
best way to update this file is to first goto 'dtksh/ksh93/src/cmd/ksh93/data'
and do an Rcsdiff, to find out what changes Dave Korn made.  You will
then make those same changes to the dtksh/aliases.c file.

   dtksh/aliases.c
   ---------------

      Fold in any changes made to dtksh/ksh93/src/cmd/ksh93/data/aliases.c


HP-SPECIFIC Ksh-93 SOURCE MODIFICATIONS:
========================================

The dtksh code requires a handful of changes to be made to the ksh93
source itself.  Most of these changes relate to either localization
customizations, additional multibyte support, or fixes for environment
variable mirroring.  For each of the files listed below, two versions
have been checked into the RCS vaults: the original (unaltered) version,
and the version with our customizations.  This will make it easy to
determine the changes we have made, so that the next time we receive
new code from Korn, it is a relatively simple task to again merge in the
changes.  Note that all of the changes relating to multibyte support
have been passed onto David Korn, and that hopefully they will show up 
in the official source sometime in the future (after the ksh93-d release).

Modification 1:
---------------

The ksh (and dtksh) executables use a special client (suid_exec) when
trying to execute an suid script; this special client makes sure that
you really have permission to run the suid script.  The client, as
supplied by David Korn, has hardcoded assumptions about where it lives
(/etc) and where the valid (and secure) places are where shells can
live.  We need to change this so that it knows the client lives in
the cde tree, and that in addition to the normal 'safe' shell locations,
/usr/dt/bin is also a safe location.  To do this, the following changes
must be made, each time new ksh source is received:

   dtksh/ksh93/src/cmd/ksh93/data/msg.c
   ------------------------------------

      Change:

         e_sysprofile = "/etc/profile";
         e_suidprofile = "/etc/suid_profile";
         e_suidexec = "/etc/suid_exec";

      To:

         e_sysprofile = PROFILEPATH;
         e_suidprofile = SUIDPROFILEPATH;
         e_suidexec = SUIDEXECPATH;


   dtksh/ksh93/src/cmd/ksh93/sh/suid_exec.c
   ----------------------------------------

      Change:

         #define THISPROG "/etc/suid_exec"

      To:

         #define THISPROG SUIDEXECPATH

      Also, search for the block of code like the following (around line 256):

         if (shell == 0 || !endsh(shell) || (
                  !in_dir("/bin",shell) &&
                  !in_dir("/usr/bin",shell) &&

      And change it as follows:

         if (shell == 0 || !endsh(shell) || (
                  !in_dir(CDE_INSTALLATION_TOP"/bin",shell) &&
                  !in_dir("/bin",shell) &&
                  !in_dir("/usr/bin",shell) &&

Modification 2:
---------------

During CDE development, it was discovered that some shared library
implementations (i.e. at least IBM) bound a shared library to all
of its dependencies at the time that the library was build.  This
created problems for libraries which made getenv() or putenv()
calls, because we expected those calls to be resolved against the
getenv() and putenv() calls provided in the ksh93 libraries, not
against the standard libc versions.  This was a problem because
ksh keeps track of its own environment, and does not pass the
information on through to the standard libc environment.  The result
was that certain key pieces of information (i.e LANG) might be set
by a shell script, but when calls where made to shared library functions,
they would not see this setting.  The fix is to modify the ksh93 code
to *also* pass along any environment settings onto the libc functions;
we call this 'environment variable mirroring'.  We won't describe the
exact changes here; you can find them by diff'ing the last two revisions
for the following files:

   src/cmd/ksh93/include/nval.h  (New field added to Namval structure)
   src/cmd/ksh93/sh/name.c       (Mirroring code implemented here)

Modification 3:
---------------

There are a handful of multibyte and localization changes which we
must make.  Some of them relate to adding message catalog support
to ksh, while others relate to making changes so that Japanese SJIS character
will work.  The files which must be modified are:

   src/cmd/ksh93/sh/init.c      (Message cat and file desc locking/unlocking)
   src/cmd/ksh93/sh/lex.c       (Japanese SJIS processing fixes)
   src/cmd/ksh93/sh/macro.c     (Multibyte fixes)
   src/cmd/ksh93/bltins/print.c (SJIS multibyte fix to echo/print command)
   src/cmd/ksh93/sh/userinit.c  (Add call to setlocale() in sh_userinit())

Modification 4:
---------------

There appears to be a bug in the ksh93 build process, which shows up
on the HP platform.  David Korn was unable to find the cause, so we
need to modify one of his build scripts:

   ksh93/src/cmd/ksh93/features/options.sh
   ---------------------------------------

      Towards the top of the script, there is the following set of
      statements:

         : get the option settings from the options file
         . $OPTIONS

      You need to modify this so that it is like the following:

         : get the option settings from the options file
         if [ -z "$OPTIONS" ] ; then
           OPTIONS="OPTIONS"
         fi
         . $OPTIONS


=======================

Each time a new release is received from Dave Korn, we also need to copy
his latest ksh93 man page from src/cmd/ksh93/sh.1 into 
/x/cdesrc/doc/cde1/man/man1/ksh93.  You will need to make one change to 
the man page, so that it prints out the ksh information, instead of sh 
information.  The change is one approximately line 7.  

   Change:
   -------

      .nr Z 0 \" set to 1 when command name is ksh rather than sh

   To:
   ---

      .nr Z 1 \" set to 1 when command name is ksh rather than sh

===================

One last note.  There is a shell script called 'MakeClean' in the root
directory of the dtksh tree.  This script cleans out files in the ksh93
portion of the tree, which will force a complete build to occur.  Since
the ksh93 portion does not use Imake (it uses Mam or something bazarre
like that), we needed to jury rig something which would act like our
normal 'make clean' .  If you discover any other ksh93 files which need
to be removed during a 'make clean' operation, simply add another line
to the MakeClean script.  A good indication that a file needs to be
added is if you make a clone from one of the source trees, do a 'make clean',
start a 'make', and somewhere during the building of the ksh93 sources,
you see an error message telling you that you did not have permission to
overwrite something within your clone tree.  The ksh93 build process builds
a lot of files on the fly, and moves things around to other directories, so
these file typically need to be cleared out before doing a complete make.