iraf.net

Hi,I'm in the process of reducing several large data sets (total size is about 130 gigabytes broken into ~20 GB chunks) of camera stability tests.Since the data are fairly uniform, most of this work can be done non-interactively. In general, my computer can handle this sort of work, but it could be better. I'm looking for some tricks to make it go faster. I am running OS 10.6, but have at my disposal other computers running leopard (intel and ppc) that I can press into slavery as well (after I transfer the files over- hoo boy).RAM. The two RAM-intensive tasks I'm running are daophot (non-interactive, one .coo file for the time series) and quadred's ccdproc overscan correction. In daophot's parameters I have set cache to no, and in ccdproc 0 is the number of megabytes to cache into system memory. When I open up activity monitor, however, I see that every time that IRAF opens an image, it gets stored into inactive memory. Inactive memory is supposed to be reused by the computer when free ram is gone, but my mac will just start swapping and paging out, probably because it sees my task all at the same thing. My kludgy workaround is to periodically execute the developer tools' "purge" command at an extra xterm prompt, say every 10 minutes or so when I've eaten up 6 of my 8 gigs of RAM. Purge freezes the system for about 5-10 seconds, but I get the memory back. Preventing the time lost from slow swaps and the eaten hard disk is important to me. I guess my question is whether there's a better way to deal with this problem without having to babysit my bulimic computer. A script? Or a setting I don't know about?CPU- At most my computer is using 60% of the CPU on reduction, and when reduction begins (I think IRAF has to read in the list first) it only uses 5% of the CPU- is there a way to allocate more of the cpu to what I'm doing. What should I be conscious of when running my computer to the max?HD- My dual-amp, overscan-corrected images are twice the size of their originals. Is there a way to process these images so they are not double in size?Other programs? I've disabled spotlight's mdworker (nothing like adding 8000 new files to make your indexer go nuts). Also, are there any scripts I should run to clean up/defrag after? The advice that says this isn't necessary also assumes you aren't creating 100s of GB worth of stuff and then moving it elsewhere later. Thanks!

What you describe is, for the most part, the way normal systems behave. There's no way that IRAF is allocating 6GB of space, however your OS will always try to use as much memory as it can by creating a 'buffer cache' of files it might need again. If you process images sequentially then each image may only be used once (but is put in the cache anyway), but if you use e.g. a reference image repeatedly then the system isn't always hitting to disk to re-read it.You SHOULD however be using the daophot cache param and the ccdproc cache - this keeps the data in active program memory meaning that it won't be swapped out. However keep in mind that if you're using v2.14 or earlier then you've only got 2GB of addressable space no matter how much physical RAM is installed.Not all tasks are CPU-intensive, there's no way to "force" the system to use 100% of the CPU for arbitrary tasks. As for the image size, that's what happens when your 16-bit integer raw data become 32-bit floating-point....the CHPIXTYPE task can be used to change the pixel type back to shorts (with scaling) but keep in mind that image needs to then be "unpacked" back to real by other tasks that use it. For long-term storage, you can use the 'fpack' task in CFITSIO or in the FITSUTIL external package to compress the FITS files (and usually recover more than a factor of 2).