iraf.net

I ran into something interesting and unexpected yesterday. I was running a script I wrote that loops through guider images (100x100 pixels each) and pulls some info from the headers with hselect, then uses tstat to get the means and sigmas (i.e. nothing particularly strenuous). With the data, the iraf install, and my scripts on an nfs disk things ran abysmally slow. It was 27 times faster on my laptop! I tried copying the data to a local scratch disk and it was still slow. I did a "mkiraf" also on the local scratch disk and logged into cl from there and it was very fast. Some other testing suggests it is the location of the uparm directory on the nfs disk that seemed to be causing the extreme slowdown. The script was fast even with the data and the iraf install on the nfs disk but a local uparm directory. Does this make sense? I believe the nfs disk is mounted as vers=3 in /etc/fstab. The machine I was running things on is new, running CentOS5. Dave (LBT Observatory)

NFS is known to have performance problems in some cases, I'm surprised at a factor of 27 though. Uparm directories are updated when a task completes so each task in the script will be hitting a file. One way to avoid this is to use the 'cache' command at the start of the script to get an in-memory copy of the param file to be used instead. For example,[code:1:740092ef2c]
procedure foo ()begin
cache hselect, tstat, imstat, ..... ....rest of the script
end
[/code:1:740092ef2c]It's also worth investigating the NFS slowness at the host level, e.g. is vers=3 correct for the server, are there any system timeouts, etc. A network routing problem could also explain poor performance.-Mike

I tried both 'cache ("hselect", "tstat")' and 'cache hselect, tstat' immediately after the "begin" line and both give me an "illegal variable declatation" message. I needed to put the cache declaration after all of the internal task variable declarations for the script to run.If I just type "cache" at the cl prompt, I get a list of currently cached tasks, and images.imutil (for hselect) and tables.ttools (for tstat) are already listed. Hselect and tstat are not explicitly listed. Neither seemed to speed up the running of the script using my home$uparm directory. I checked again, and if I use a local uparm (uparm set to "./uparm" in my login.cl) and the directory is available in my working directory then the script runs very fast. Everything else (iraf, data, scripts) are on the nfs disk.Still sound like an nfs issue? In the /etc/fstab on this machine, the nfs disk is mounted like this:nfs:/foo /foo nfs bg,rw,hard,intr,vers=3,tcp,time0=600,rsize=32768,wsize=32768 0 0 The only nfs disk I have (personally) set up before just lists "defaults" in the options field on the machine mounting the nfs partition, but I do not use this machine for running iraf. I suppose I can get our IT guys to mount this disk to a separate mount point using "defaults" in the options field and see if that helps. I'll post how that goes.

[quote:26a8401a74]
If I just type "cache" at the cl prompt, I get a list of currently cached tasks, and images.imutil (for hselect) and tables.ttools (for tstat) are already listed. Hselect and tstat are not explicitly listed. [/quote:26a8401a74]If the cache command worked as expected you should explicityly see 'imutil.hselect' in the 'cache' command listing.[quote:26a8401a74]Still sound like an nfs issue? [/quote:26a8401a74]I honestly can't say for sure what it is, but I *can* say that IRAF doesn't know whether you're on an NFS disk or not. From what you say, certainly there's something happening in the uparm directory seems to be triggering the problem, but without seeing the script I don't know whether this is an underlying host problem or some abuse of the uparm directory by the script (some tasks use uparm as a scratch area).I did run a simple test script where I call hselect 10,000 times and it didn't seem to matter whether I was using an nfs-mounted uparm or not. Note that TSTAT does explicitly update ~6 parameters to it parameter file with each call, one simple test would be to comment out the tstat and see if the uparm location is still a problem (and whether a 'cache tstat' then helps). Lastly, because tstat it updating its params, "proper" script style suggest that you cache the task anyway to avoid problems with multiple instances running in the background.-Mike
[/quote]

Hi Mike, The slow step is in the loop that hselects on the image headers. This is done to build the input file for the tstat task that is only called three times (for the three columns in the file). I'd be happy to post the entire script, but this is the for loop where hselect is called and seems to be the slow step: for ( i=N1; i<=N2; i+=1 ) {
printf (" working on image #%d \r",i)
### Make image name, check if it exists
giname = path2data//prefx//"000000"+i//".fits"
if (!access(giname)) { goto next_giname } ### Write filename to outlis (for later imcombine):
print (giname, >> outlis) ### Hselect data from headers
xc = 0.0 ; yc = 0.0 ; xfw = 0.0 ; yfw = 0.0
hselect (giname, "CENTPIXX,CENTPIXY,FWHM_X,FWHM_Y", yes) | \
scan (xc,yc,xfw,yfw) ### Rescale header data as needed
FWHM = sqrt(xfw*xfw + yfw*yfw)/root2 ### Write out data to file
printf (" %.3f %.3f %.3f \n", xc, yc, FWHM, >> loopout) next_giname:
} # End of for loop I'll be back at the telescope in a couple weeks and can do more testing then.

Try refactoring the loop so that you create an @list of the desired image names, call hselect only once on that @list (hence the uparm would be updated just once), and then loop over the output list to get the values needed for the FWHM calculation. If this really is the culprit, I would expect a "cache hselelect" in the script to also fix it.-Mike