iraf.net

So, I'm now running IRAF on a lovely quad-core Mac Pro. Since IRAF by and large is single threaded, I'm looking for ways to speed things up and take advantage of all my cores. For example wheN I reduce spectra I typically generate a "doit" file that is something like this:[code:1:a7e5df788d]myspectask file1 output1
myspectask file2 output2
...
myspectask file24 output24[/code:1:a7e5df788d]So, my first instinct was to split this into 3-4 chunks, and then run IRAF 3-4 times and let each instance of IRAF churn away at a file, ala
[code:1:a7e5df788d]ecl> cl < doit_1[/code:1:a7e5df788d]and then in another[code:1:a7e5df788d]ecl> cl < doit_2[/code:1:a7e5df788d]However, I notice that IRAF can handle tasks in the background with the ampersand .. much like UNIX. Two questions then... one, I notice that I lose the interactive prompt when I do & because text streams onto the screen from apall and refspec and such... I should redirect this somehow so I can still see the cl prompt, no? Just with the > presumably? So maybe...[code:1:a7e5df788d]cl> myspectask file1 output1 & > log1
[/code:1:a7e5df788d]In any event, the real question is how should I set up these background tasks in a big doit script. I'd only want to run 3 or 4 at a time... so I do I distribute the ampersands so that I'm not running 20 apextracts at once? My instinct is to try...[code:1:a7e5df788d]myspectask file1 output1 & > log1
myspectask file2 output2 & > log2
myspectask file3 output3 > log3
myspectask file4 output4 & > log4...[/code:1:a7e5df788d]So in my mind the fourth line won't start until the 3rd line (a non-background task) finishes... and presumably the first two will finish around the same time.. so if every 3rd is non-background, this should work, no?Any suggestions appreciated!

---

The re-born Mac IRAF web site: http://macsingularity.org

Hi Marcos,A detailed answer would depend on knowning specifically what is in your 'myspectask', which I presume is a script of some kind, but maybe some comments will help. First, even though IRAF tasks aren't multi-threaded (any change would have to be made individually for each application), the system itself is inherently multi-process. Aside from the CL interpreter, there is the x_system.e process locked in the process cache, plus whatever other binaries get started depending on which tasks you use. A process will remain running in the cache until you 'flpr' or it is swapped out by another task. The size of this cache is determined by the cl.szprcache parameter, min size 2, max size 10 and default size 4. So even a modest script might be swapping tasks through the cache so part of what your system is doing is restarting the binaries and context switching. You can reset the size fo the cache to minimize this, but also keep in mind that not all tasks are CPU-intensive so it will be rare to have the CPU pegged all the time anyway.Background tasks work like they do in unix, basically a CL is spawned to execute the command as a separate process (or group of processes per the above). There is a limit of 20 active jobs in the background. I think the problem you may be having though is that the '&' belongs at the end of the line, any file redirection belongs before the ampersand. Note however that background jobs will share the same uparm directory so be careful of using tasks that update their parameters (e.g. IMGETS) conflicting with each other. Also note that executing via indirection means that all cl input will be redirected, i.e. a prompt will be answered by the next line of the redirected file and not the CL prompt (is that what you meant by seeing the prompt again?).So, you can split the input lists and start several background tasks, but that doesn't mean you'll completely occupy the machine (unless you crank up a bunch of these things simultaneously). Hope this helps.Cheers,
-Mike

Well, the primary task within "myspec" that takes a while is apall on echelle data with clean=yes. A "doit" script with three &s and then one non-& seemed to work (check out Macsingularity for a screenshot of four x_apextract.e's running at the same time) but then after cranking out all four cores and doing four extractions at once... the CL stopped processing the doit script and didn't move on to the next four would-be simultaneous tasks....

---

The re-born Mac IRAF web site: http://macsingularity.org

Ok, so doing myspectask whatever whoever > log1 &three times with ampersands and once without in a doit file and then running the doit file with the normal cl < doit sort of process seemed to work just fine. I'd get 4 (occasionally 5 for a brief second) apextract's going and the whole process went quite quickly.The only trick now is automatically putting in the ampersand's while skipping every fourth line....myspectask (actually called myspectraextract) incidentally is a cl task/script that simply calls apall, refspectra, ecreidentify so as to extract, name, and dispersion correct my spectra. I suppose it's my own version of something like doecslit.....

---

The re-born Mac IRAF web site: http://macsingularity.org

HMm, well this worked once and now isn't working... doing the 'myspectask' with some ampersands worked nicely in one directory. It blew through four and then another four and then the last two... always doing about 4 at a time...but, I tried the same method in a diferent directory and it does four... then starts up one and gives me a "no more background job slots" error and stops.I thought it could do up to 20? It seems to be balking at maybe 5 possibly 6... as I think 3 are still going as it tries to crank up another 2-3.Well, putting a "sleep 3" in the doit file to let the cl catch up such that only four "myspectask" scripts are running at once seems to have solved the "no more background job slots" problem more or less. Granted the "myspectask" script itself is a series of other task but they all sort of run in sequence so I figured I was well under the 20 jobs limit...

---

The re-born Mac IRAF web site: http://macsingularity.org