| massey |
09/12/2014 04:59AM (Read 1069 times)
|
|
|
Status: offline
Registered: 02/10/2006
Posts: 162
|
I'm running a script to reduce some data from Las Campanas, and I realized that it was gobbling up all of my machine's memory (32Gb) in a matter of a a few minutes. (This is on one of the new Mac Pro's running Mavericks; I see similar behavior on my laptop, although there there's only 8Gb to eat up.) At the start of the scrip tthere is a loop that does a few hedits, and as that runs I can see memory getting used at the rate of about 1 Gb every second or two. I attach a copy of the script, although I don't think it's doing anything unusual. Furthermore, once the script ends the memory remains in use according to "activity monitor." Is there any way I can prevent this?
PHP Formatted Code
procedure e2v (dum )
string dum
string *blist
begin
string s1 ,s2 ,s3 ,t ,b
int i
#First add IRAF compatible IMAGETYP values using EXPTYPE
if (access ("listoffiles"))
delete ("listoffiles")
files ("*c1.fits,*c2.fits,*c3.fits,*c4.fits",>"listoffiles")
blist ="listoffiles"
while(fscan (blist ,s1 )!=EOF ) {
imget (s1 ,"EXPTYPE")
s2 =imget .value
if(s2 =="Bias")
hedit (s1 ,"IMAGETYP","BIAS",add +,ver -,show +)
if(s2 =="Flat")
hedit (s1 ,"IMAGETYP","DOME FLAT",add +,ver -,show +)
if(s2 =="Object")
hedit (s1 ,"IMAGETYP","OBJECT",add +,ver -,show +)
imget (s1 ,"FILTER")
s2 =imget .value
hedit (s1 ,"FILTERS",s2 ,add +,ver -,show +)
}
t ="[3:2050,2:2056]"
b ="[2052:2175,2:2056]"
hedit ("*.fits","DATASEC",del +,ver -,show +,up +)
hedit ("*c1.fits,*c2.fits,*c3.fits,*c4.fits","TRIMSEC",t ,up +,ver -,show +)
hedit ("*c1.fits,*c2.fits,*c3.fits,*c4.fits","BIASSEC",b ,up +,ver -,show +)
# Overscan and trim:
#
ccdproc ("*c1.fits,*c2.fits,*c3.fits,*c4.fits",over +,trim+,zerocor -,flatcor -,biassec ='image',trimsec ='image',zero ='',flat ='',illum ='',interact -)
if (access ("listofiles"))
delete ("listofiles")
#
files ("ccd*%c1.fits%%",>"listofiles")
blist ="listofiles"
while (fscan (blist ,s1 ) !=EOF ) {
#First check to see if we've already Mosaic'd this one...
s2 =s1 //"f.fits"
#c1:
if(!access (s2 )){
s2 ="c1.fits[*,*]"
s3 =s1 //s2
if(access ("tempc1.fits"))
imdelete ("tempc1.fits")
e2vlincor ('c1',s3 ,'tempc1.fits')
#c2:
s2 ="c2.fits[-*,*]"
s3 =s1 //s2
if(access ("tempc2.fits"))
imdelete ("tempc2.fits")
e2vlincor ('c2',s3 ,'tempc2.fits')
#c3:
s2 ="c3.fits[-*,-*]"
s3 =s1 //s2
if(access ("tempc3.fits"))
imdelete ("tempc3.fits")
e2vlincor ('c3',s3 ,'tempc3.fits')
#c4:
s2 ="c4.fits[*,-*]"
s3 =s1 //s2
if(access ("tempc4.fits"))
imdelete ("tempc4.fits")
e2vlincor ('c4',s3 ,'tempc4.fits')
# Now create final image
s2 =s1 //"f.fits"
imexp ("I*0.00+J*0.0",s2 ,dims ="4096,4110")
#c1-->1:2048,1:2055
s3 =s2 //"[1:2048,1:2055]"
imcopy ("tempc1.fits",s3 )
#c2-->2049:4096,1:2055
s3 =s2 //"[2049:4096,1:2055]"
imcopy ("tempc2.fits",s3 )
#c3-->2049:4096,2056:4110
s3 =s2 //"[2049:4096,2056:4110]"
imcopy ("tempc3.fits",s3 )
#c4-->1:2048,2056:4110
s3 =s2 //"[1:2048,2056:4110]"
imcopy ("tempc4.fits",s3 )
# Now move over the header
hfix ("tempc2.fits",command ="copy $fname /tmp/tempc2.hdr")
hfix (s2 ,command ="copy /tmp/tempc2.hdr $fname")
#Transpose so that N is up and E is the left.
s3 =s2 //"[-*,*]"
imtranspose (s3 ,s2 )
}
}
# hedit("*.fits","IMAGETYP",'(@"EXPTYPE")',add+,up+,ver-,show+)
hedit ("*.fits","UT",'(@"UT-TIME")',add +,ver -,show +,up +)
hedit ("*.fits","GAIN",'(@"EGAIN")',add +,ver -,show +,up +)
hedit ("*.fits","RDNOISE",'(@"ENOISE")',add +,ver -,show +,up +)
# hedit("*.fits","FILTERS",'(@"FILTER")',add+,ver-,show+,up+)
hedit ("*.fits","EPOCH",'(@"EQUINOX")',ver -,show +,up +)
hedit ("*.fits","CCDSEC",del +,ver -,show +,up +)
asthedit ('*.fits','/users/massey/dropbox/SwopeII/cmd')
observatory ("set","lco")
setjd ("*.fits")
# If the calibration frames exist, don't create new ones.
if(!access ("Zero.fits"))
zerocombine ("*f.fits",output ="Zero.fits")
ccdproc ("*f.fits",over -,trim -,zerocor +,flatcor -,zero ='Zero.fits',flat ='',illum ='')
end
|
| |
|
|
| massey |
|
09/12/2014 12:59PM
|
|
|
Status: offline
Registered: 02/10/2006
Posts: 162
|
I've made some sample frames where you can get them if you want to see this in action.
http://www.lowell.edu/users/massey/sample.tar.bz2
Thanks!
|
| |
|
|
| fitz |
|
09/12/2014 04:46PM
|
|
|
Status: offline
Registered: 09/30/2005
Posts: 4040
|
There may not be a real problem: If what you're looking at on the Activity Monitor is the total 'Memory Used' at the bottom of the GUI then this may simply be the system caching each of the images you're processing, that space will be swapped out as new files come up in the list automatically. OTOH, if what you're seeing is that the e.g. x_images.e binary is growing to 32GB then there's a memory leak somewhere we'll need to fix. Putting a 'flpr' at the bottom of your loop should also help clean out growing (iraf) process sizes.
|
| |
|
|
| massey |
|
09/12/2014 06:55PM
|
|
|
Status: offline
Registered: 02/10/2006
Posts: 162
|
Mike, I think it's real. The reason I started looking at this is that after it runs for a while it gets very, VERY slow. When I did hat I could see that the memory was pinned in activity monitor. The OSX command "purge" cures the problem, at least for a while.
I also ran the OSX command "leaks" with the script running, and I got a zillion memory leaks listed. Trying this on other processes and apps seem to give a clean bill of health. So, I do think there's a problem here.
I'll try adding a flpr or two and see if that helps. Thanks!
---phil
|
| |
|
|
| fitz |
|
09/12/2014 08:22PM
|
|
|
Status: offline
Registered: 09/30/2005
Posts: 4040
|
The 'purge' command would clear the disk caching I mentioned, but 'leaks' requires a specific process. Did you run this on the CL or x_images.e? BTW, what is the E3VLINCOR task and where did it come from? Does the Activity Monitor specifically say which process is growing?
|
| |
|
|
| |
|
New Topic
Post Reply
memory usage
Quote
