May 23, 2010 A Guide to Porting SPP Code to 64-bit IRAF ========================================== ============================================================================== Revision History: May 23, 2010 Initial Release =============================================================================== Contents: --------- 1) Introduction 1.1) A Discussion of 64-Bit Port Strategies 1.2) Other Major Changes 2) Code Structures to Review 2.1) Use of Memr in Structures 2.2) Size Calculations Based on Datatype Sizes 2.2.1) SZ_INT / SZ_INT32 Usage 2.2.2) SZ_STRUCT /SZ_STRUCT32 Usage 2.2.3) SZ_PIXEL / TY_PIXEL Usage 2.2.4) The sizeof() Function 2.3) MII (Machine Independent Integer) Interface Usage 2.4) The read() / write() Functions 2.5) The ipak32() / iupk32() Functions 2.5.1) The imul32() Function 2.6) External Binary Formats 3) Available Tools 4) Testing the Code ============================================================================== 1) Introduction This document is intended to serve as a guide for SPP programmers and external package developers wishing to port their code to 64-bit platforms under the IRAF v2.15 release. The 64-bit port of IRAF was done in a way intended to minimize the number of changes required, however there is no implicit guarantee that just because code compiles under v2.15 that it will run correctly on a 64-bit system. All developers are encouraged to review their code for the recommended changes to ensure compatability with 64-bit platforms. The code changes required for 64-bit IRAF fall into several catagories: - The relative size of datatypes within SPP data structures has changed and so code assumptions made about the size of integer and pointer types in structures to be reviewed, - The interaction of SPP programs with external binary formats that use a 32-bit integer size must correct code doing I/O based on the size of integer data, - Interactions by C code with VOS or IRAF Kernel procedures must be reviewed to ensure proper use of the XINT data type, - In all applications, the method of declaring SPP data structures, in particular the use of Memr, has been changed and all code must be reviewed for its use then changed if necessary. In the first case we realize that on 64-bit systems the size of an INT datatype has doubled, this less-obviously implies 1) that STRUCT datatypes have similarly increased in size to accomodate the larger INT, and 2) then that the relative size of REAL and DOUBLE wrt to the size of a STRUCT element has thus also been changed. Details of these relative size changes are discussed below. In the second case, maintaining compatability with external binary formats also means the use of INT in reading or writing those data need to be be reviewed for potential changes. For instance, innocent-looking SPP code such as call write (fd, buf, (nelem * SZ_INT)) will write a different number of bytes to a file depending on whether it is run on a 32- or 64-bit system. For this reason, new macro values such as SZ_INT32 have been introduced to specify the size of a 32-bit integer regardless of the platform being used, allowing applications to enforce the concept of a 32-bit integer where needed. Note also in this example that since 'buf' is naturally declared as an SPP 'int' array, simply replacing this code with SZ_INT32 would then only write half the number of actual bytes when run on a 64-bit system in applications that might actually want to write a native 64-bit integer. To address this, new procedures for packing and unpacking integers between 32- and 64-bit values have also been introduced and should be used where needed. VOS interfaces such as IMIO and PLIO will handle these changes automatically, this caution refers to e.g. legacy binary TABLES files, graphics formats such as GIF, some network protocols or code that handles binary data outside of the normal VOS routines. For SPP code, we control the generated intermediate languages (i.e. Fortran and C) to define the integer data size as needed. C code such as the LIBC interface or CL/ECL however still assume an 'int' datatype is 32-bits (the GCC compiler enforces this). Fotunately, the C code in the core system makes use of an XINT datatype to abstract the C integer from the corresponding SPP datatype, however care must be taken in the interface between the C and SPP codes to ensure the proper type is passed and returned between procedures. Interfaces such as CVOS, C bindings to IMFORT or any application making use of VOS or Kernel procedures must be reviewed to ensure proper use of the int/XINT boundary. Lastly, past SPP coding standard required that use of certain data types within structures use a macro to align the pointer indexing, e.g. For double-precision For character strings define FOO Memd[P2D($1+10)] define STR1 Memc[P2C($1+10)] define BAR Memd[P2D($1+12)] define STR2 Memc[P2C($1+10+SZ_FNAME)] Note that for DOUBLE types it is not only the use of P2D() that's required, but also that extra room be allowed in the structure index to accomodate the value. For CHAR data we require only the P2C() macro plus whatever length of string is required. In the new system, we now require use of a new P2R() macro around all Memr[] elements to address the difference between the size of REAL and STRUCT on 64-bit systems. These macros are defined differently for each platform and may be no-ops on a particular system, however they ensure proper pointer alignment in a machine-independent manner, and since this definition is done at the system level, applications are free to use the macro to ensure compatability without worrying about the runtime platform. 1.1) A Discussion of 64-Bit Port Strategies There are two viable approaches to porting applications or systems to 64-bits(*): the LP64 model where only long and pointer type become 64-bit, and the ILP64 model where integers are also promoted to 64-bits. In both models 'short' remain 16-bit and floating-point/double are unchanged. The LP64 model is generally recommended for new software and what's supported (indeed, enforced) by most unix systems and compilers. For legacy systems, a choice must be made whether to modify the code to use the LP64 model faithfully, or to retain any implicit relation between data types and use the ILP64 model if possible. The LP64 model is the only model supported by GCC compilers and for some projects may be the only path forward, however it is highly resource- intensive since it requires that every line of code be examined to ensure a separation between use of int and pointer, and to promote int to long where the possibility exists this value could exceed 32-bits. Each change to the code offers the opportunity for a new bug to be introduced, and the large number of changes required means multiple interacting bugs could be impossible to debug. A system like IRAF has millions of lines of code, the majority of which are in external packages lacking the manpower resources to do a major code overhaul and the testing required to certify it. An ILP64 upgrade path is an attractive option because we can preserve implicit assumptions made about the interchangeability of int and pointer uses at the expense of some extra memory (which represents a small waste by modern standards). Becuase SPP is a pre-processed language we also have complete control over the target compiled language and can thus work around restrictions imposed by compiler supporting only the LP64 model. However, aside from wasted memory we must also deal with assumptions about the relative size of datatypes within the code and how these interact with elements outside the system (binary files, network protocols, etc). By promoting INT to the size of POINTER on 64-bit we change the relationship between sizeof(INT) and other datatypes, forcing us to examine our data structures for possible conflicts. Such a change only actually affects the REAL datatype in structures and so we limit the impact to a single class of change, i.e. the use of REAL in SPP structures, instead of the many potential interactions between INT and POINTER. Likewise, the number of applications or interfaces dealing with external entities is also relatively small and can be addressed in the interface or in a very limited number of applications. Although external packages are free to implement anything possible in the core system, we felt that by porting the core interfaces using the above model would do much of the work required to port external code and limit the changes required to external code to a few targeted areas that could be easily addressed by outside developers. Unlike past platform ports of IRAF, the move to 64-bit computing cannot be achieved without some level of change to existing code. We've tried to minimize the changes required and indeed nearly all of the external packages currently in use have been easily upgraded and are ready for use (pending approval or re-release by the developers). Additionally, these changes preserve backwards compatability with legacy data files and provide a single code-base for all supported IRAF platforms. (*) There is additionally an LLP64 model that promotes only pointers to 64-bits and adds a 'long-long' 64-bit integer type, however this is used only on Microsoft 64-bit systems and was not considered for this project. 1.2) Other Major V2.15 Changes o New MEMIO Interface The MEMIO interface was necessarily changed to accomodate the 64-bit upgrade, specifically we added extra structure to a memory object to aid in debugging (e.g. sentinal values before/after a segment, a reporting mechanism etc). This adds a slight bit of overhead to each heap memory allocation but we consider this to be minor given the capabilities of today's systems. Environment variables can also now be used to control the behavior of this interface to allow for improved debugging or to aide in code development. These variables may be set in either the CL or unix environments and include: MEMIO_CLEAR Zero all memory allocations, i.e. treat all malloc() calls as though they were calloc(). Use of realloc() is unchainged. MEMIO_COLLECT Perform garbage collection when a task exits. Can be used to curtail problems with memory leaks in long-running script tasks. MEMIO_DEBUG Print debugging info about the interface. MEMIO_REPORT Report memory usage statistics when a task completes. This can be used to verify that a task is using a reasonable amount of memory for the machine. MEMIO_WATCH Check a segment's lower/upper sentinal values for corruption when a pointer is freed. Can be use to detect over/underflow closer to the point at which it actually occurs in the code. o Single 'linux' Architecture for 32-bit Systems This is part of the overall architecture changes implemented in v2.15 systems. The original logic for separate 'suse', 'redhat' and 'linux' architectures was due to differences between these platforms in the GCC/Glibc systems used which were inherently incompatibile. These differences have since become irrelevant and impose a support burden to maintain and so a common 'linux' architecture is now preferred. Because external packages may lag in the changes needed to switch architectures, we try to support legacy references to e.g. 'redhat' as an alias to the new 'linux' structure and hope that the previous deprecation of 'suse' has now been removed entirely. Since 64-bit support requires a new architecture name there will be no confusion between the 32- and 64-bit versions of the system. o Mac OSX Architecture Changes In previous IRAF releases the 'macintel' archicture was used for all OSX versions running on Intel hardware, and 'macosx' was reserved for PowerPC (PPC) CPUs. In order to avoid yet another IRAF architecture name to distinguish 64 and 32-bit Intel systems, and because OSX systems are moving generally towards Intel CPUs with 64-bit capabilities anyway, that future IRAF systems should favor 64-bit architectures as users migrate to new hardware. What this means is that starting with v2.15 the IRAF 'macintel' architecture will refer exclusively to 64-bit Intel hardware. In order to retain compatibility with older systems, the 'macosx' IRAF architecture will encompass 32-bit binaries for both PPC and Intel CPUs in a Universal binary format. This is a departure from past platform architecture naming conventions where a new unique name is used, however future MacOSX systems will all be Intel-based and hardware will be 64-bit capable so shifting the focus to 'macintel' being the primary platform and 'macosx' supporting older systems makes the most sense. (In hindsight we would have preferred to be able to use 'macosx' as it is more descriptive). o 32-bit Mac Universal Binaries The 'macosx' architecture now includes both PPC and Intel binaries in the system libraries (and those produced for external packages) and executables. These binaries are compatible for OSX 10.4 and later systems and the system will automatically use the appropriate binary for the hardware (e.g. on an OSX 10.4 PowerPC machine the PPC binary for the 'macosx' arch will execute the PPC binary in the iraf executable file). While earlier IRAF versions would allow that PPC 'macosx' binaries could be run on Intel systems using the Rosetta sytem, this isn't currently allowed with v2.15 as an Intel system would automatically use the Intel binary. Thus, to test PPC binaries you will need a machine with a PPC CPU. This is seen as something that would affect a minimal number of users. o Compile-Time VOS/Kernel Prototype Checking All library code in the IRAF 'Virtual Operating Interface' (VOS) and IRAF Kernal (i.e. the $iraf/unix//os procedures) now have function prototypes that are checked during compilation of any SPP code. o Intra-Package Prototype Checking This is a feature that will be added in the near future and will provide the same level of prototype checking as with the core system libraries. The idea is that a 'libpkg.a' will produce a local prototype file that ensures all calls within the package meet the prototype calls. As with VOS/Kernel checking, this will validate the code within a package to trap any potential errors. o All C Library Code Converted to "Clean" ANSI-C As part of the 64-bit port, all C library code in the VOS and kernel was converted from the old-style "K&R" style to true ANSI C with full function prototypes. Additionally, the code was cleaned up to remove any warnings generated by the compiler, i.e. compilation with "-Wall" under GCC now compiles cleanly to further ensure the integrity of the code. o Cross-Compilation of Architectures Compilation is now defined by the user environment rather than the machine used to do the build, as before. What this means is that (within reason) it is now possible to cross-compile for a different IRAF architecture on machines which support such compilation. For instance, on a 64-bit Mac OSX Snow Leopard system (where the native IRAF architecture is 'macintel') one can now compile code for the 32-bit 'macosx' architecture (either PPC or Intel) by simply setting the 'IRAFARCH' environment varable as 'macosx' and reconfiguring the system/package approriately. The same is true for Linux 64-bit systems and 'linux64' and 'linux' architectures, meaning one doesn't need access to machines to produce binaries. Compilation flags are set auomatically based on the IRAFARCH variable in order to produce the desired binaries. o User-Selectable Architectures As with compilation, the IRAFARCH variable can be used to determine which binaries to run. For example, on a 64-bit Linux system (where the default arch would be 'linux64') the commands % setenv IRAFARCH linux % cl would allow a user to run the 32-bit 'linux' binaries and thus switch easily between systems to compare results and validate the science results of a reduction/analysis task. The same is true for Mac (Intel) systems where the 64-bit macintel and 32-bit macosx architectures could be selected by the user. o SVG Graphics Device for Better Web Presentation of Plots SVG graphics are an XML format supported by many modern browsers that permit scalable graphics for web presentation. This means that plots produced by IRAF may be used in web pages without loss of resolution or aliasing effects seen when using e.g. GIF images of a plot. A new 'g-svg' graphcap device was added to make use of this new driver. It produces a file called 'sgiXXXX.svg' in the current working directory (where 'XXXX' is a process id) and may be used as e.g. cl> contour dev$pix dev=g-svg or in cursor mode with a ":.snap g-svg". Either instance should be followed with a 'gflush' or ':.gflush' to flush to output to disk. SVG files can be embedded into HTML documents with the tag, the tag, or the