iraf.net

We have been seeing slow performance with IRAF 2.12.2a on Solaris 9/10 systems using the default Solaris binaries. Has anyone else noticed this? Does anyone know if recompiling core IRAF on a Solaris 10 system significantly improves performance on these machines?Thanks,
Bryan

Slow compared to what, expectations or earlier runs on the same machine? I was able to find two roughly identical machines here running Solaris 7 and Solaris 9 and for the most part a benchmark script ran in about the same time. There was the expected difference when using NFS-mounted disks and some unexplained slowness on CPU-intensive tasks that happened only occassionally. However, the fact the script ran in the same time even once backs up my first thought that recompiling the system wouldn't really help. You'll need to first rule out things like NFS performance, excessive system paging, full /tmp areas, runaway httpd on the machine, etc. It would be simpler to isolate this to a real problem if you used a single compiled task in your tests rather than a script that may have its own issues. We don't use Solaris 9/10 here on most machines, others may have more relevant real-world experience. Hopt this helps.-Mike

We ran the IRAF benchmark on several machines. IRAF r/w images is very slow on Solaris from at least 8 onwards.The benchmark on Solaris 10 takes 8 seconds , and 9.4 seconds on Red Hat using the native file format, _but_ if one uses FITS format instead then the results change dramatically:
18.2 seconds Red Hat Entreprise 4
237.4 seconds Solaris 10 x86We used different filesystems for the test and it made no difference : /tmp (UFS) RAMdisc NFS. RHE used ext3 instead of UFS, of course.
The problem is with the FITS format. If ones runs in the native format then there is no problem. The test systems were the same (dual boot Solaris 10 and RHE 4).
Sun Fire z20 with 4 x Dual Core AMD Opteron(tm) Processor 270 with 4Gb RAM.Any one got any ideas?

The original thread didn't mention this was x86 Solaris. Regardless, I was able to find a box locally running Solaris 10 (x86) and my standard benchmark script ran in equal times using either imh or fits as the format. If you'd like to run the same test my script is at ftp://iraf.noao.edu/pub/fitz/bench.clAre you using the 64-bit version of the OS (I can't duplicate that here though)? Does your test add a lot of header keywords causing the FITS image to be rewritten frequently? If the above script runs in about the same time you'll need to profile your test a bit better to isolate the problem.To answer the original question: Recompiling the system probably won't produce any great improvement (certainly not an order of magnitude!). If this is a 64-bit OS then it isn't even possible at the moment.-Mike

Fitz,The same problem happens on both SPARC and X86. I just happened to have the x86 results closer to hand. I'll get back to you with the results of bench.cl.64 bit is running on both platforms, but iraf would use the 32bit part of it. - Output from isainfo -v below.* SunOS sbfsr2 5.10 Generic_118844-28 i86pc i386 i86pc
64-bit amd64 applications
pause sse2 sse fxsr amd_3dnowx amd_3dnow amd_mmx mmx cmov amd_sysc cx8
tsc fpu
32-bit i386 applications
pause sse2 sse fxsr amd_3dnowx amd_3dnow amd_mmx mmx cmov amd_sysc cx8
tsc fpu
* SunOS sbfsr1 5.10 Generic_118822-23 sun4u sparc SUNW,Sun-Fire-V210
64-bit sparcv9 applications
vis2 vis
32-bit sparc applications
vis2 vis v8plus div32 mulBryan, I emailed you to tell you where I've put the bench.cl on each machine.s.

Folks, On Solaris 10 x86 v20z I got Bench started at 23:12:28
Bench ended at 23:12:42

Total execution time = 14.0 seconds
Total time Make 5 imgs Proc 3 imgs Combine 3 imgs Median 1 img
14.0 3.0 2.0 1.0 8.0
im>

Well, I think that the test used by fitz it isn't fare enough. All the computation is always than in geis format. I'm been usgin the modified version by MJF on Dec 2003, what allow to use either format, finding than the results can be as much worst than 34 secs (geis) versus 292 secs (fits), running on 10 images, testted on 64 and 32 bits kernel, for Solaris 10 x86.
The same test has been repeated on different architectures and Solaris version, always showing the same behavior.
Jorge

Just to clarify: 'geis' files are the old ST format (e.g. ".hhh" files), I think you mean 'oif' files (meaning ".imh" extensions).
I never intended my benchmark to be indicative of yours, but the point is that the runtimes are nearly identical regaardless of whether the imtype is imh or fits (only a part of the script is i/o intensive, but the results are the same). This implies there's nothing inherently slow in the system, but something about your script is behaving differently (frequent header keyword updates is one explanation)Can anyone post a trivial example script that demonstrates this behavior?-Mike

Mike,
could you try http://iraf.noao.edu/iraf/ftp/pub/bench.cl, using imtype = "imh" and then imtype = "fits".
On my sytem, running Solaris 10 32-bit i386 kernel modules (results are equivalent for 64), I got the following:cl> show imtype
imh
cl> bench 10

Bench started at 17:42:26

=====> Making images...
=====> Normalizing flat...
=====> Subtracting zero from 10 images...
=====> Dividing flat into 10 images...
=====> Combining 10 images...
=====> Median filtering 2 images...

=====> Deleting all images...

Bench started at 17:42:26
Bench ended at 17:42:59

Total execution time = 33.0 seconds
Total time Make 12 imgs Proc 10 imgs Combine 10 imgs Median 1 img
33.0 7.0 4.0 5.0 17.0
cl> set imtype = "fits"
cl> bench 10

Bench started at 17:45:14

=====> Making images...
=====> Normalizing flat...
=====> Subtracting zero from 10 images...
=====> Dividing flat into 10 images...
=====> Combining 10 images...
=====> Median filtering 2 images...

=====> Deleting all images...

Bench started at 17:45:14
Bench ended at 17:50:05

Total execution time = 291.0 seconds
Total time Make 12 imgs Proc 10 imgs Combine 10 imgs Median 1 img
291.0 246.0 14.0 10.0 18.0Jorge.

Jorge,
My apologies: The script I was using had explicit .imh extensions on the image names and so the imtype wasn't being used, hence identical results. I fixed this and got the following:[code:1:29648b6020]
cl> reset imtype = imh
cl> reset imdir = HDR$
cl> show imtype
imh
cl> bench
:
:
Total execution time = 47.0 seconds
Total time Make 5 imgs Proc 3 imgs Combine 3 imgs Median 1 img
47.0 13.0 8.0 4.0 22.0cl> reset imtype = fits
cl> flpr 0
cl> bench
:
:
Total execution time = 151.0 seconds
Total time Make 5 imgs Proc 3 imgs Combine 3 imgs Median 1 img
151.0 105.0 17.0 5.0 24.0
[/code:1:29648b6020]Clearly something's going on but the difference isn't quite as dramatic as yours. I'll look a little deeper to see if I can figure out what is happening.-Mike

I now officially recant my suggestion that relinking won't help.The problem reported was confirmed on Sol10 boxes of both the x86 and sparc flavors. However, on my test systems there were no compilers installed and so profiling was limited to what I could derive from truss/vmstat/iostat. There was no conclusive evidence, but there we hints that this is a memory/paging issue of some kind (involving shared system libs). The disk i/o for FITS was slightly more than for imh but no more than expected, >90% of the excess time was "someplace else".Anyway, on a Solaris 5.6 (sparc) box I simply relinked the stock V2.12.2a system and moved those bins to the Solaris 10 box (w/ no compiler). Running the relinked binaries not only brought the FITS times in-line but also improved the imh performance as well. For this experiment I simply relinked with a "-z" flag so as not to have to rebuild the shared library.On x86 there no iraf shared library and I have no comparable box here (the original Solaris 7 x86 is now dead) but suspect relinking against the native host libs might show similar improvement. I'm out of time to track this any further but please try relinking the systems to see if the problem is fixed. To do this you would: cl> cd iraf$
cl> mkpkg ssun # or 'sunos' for x86
cl> mkpkg -z update # relink system
cl> cd noao$ # repeat for NOAO pkg
cl> mkpkg -p noao ssun
cl> mkpkg -z -p noao update...and the like for all external packages. However with just the above you should be able to rerun the bench.cl script to check the times. In any case, please let me know what happens.-Mike

Thanks Mike, what you have suggest it works, the execution time has been reduced almost 3 times. Having very similar run time for both images format.On Solaris 10 (64-bit sparcv9 kernel modules) my bench version take:
a)on fits
Total execution time = 126.0 seconds
Total time Make 12 imgs Proc 10 imgs Combine 10 imgs Median 1 img
126.0 44.0 26.0 14.0 40.0
b)on imh
Total execution time = 119.0 seconds
Total time Make 12 imgs Proc 10 imgs Combine 10 imgs Median 1 img
119.0 50.0 11.0 12.0 46.0Now I need to work on the Solaris x86 version.
Thanks again,
Jorge