Welcome to iraf.net Friday, March 29 2024 @ 10:54 AM GMT


 Forum Index > Help Desk > Applications New Topic Post Reply
 imcombine crash with datacubes and ofsets
   
callen
 09/18/2006 11:07PM (Read 7752 times)  
+----
Newbie

Status: offline


Registered: 02/01/2006
Posts: 11
We have a user using NIFS generated data cubes that reports this:[quote:36f33522c0]The IRAF routine imcombine will combine NIFS data cubes that are not
shifted in Y (the second spatial dimension); but hangs with any shift
in that dimension; e.g. offset file for two input cubes.
0 0 0
0 1 0
fails when
0 0 0
0 0 0
works.
But
0 0 0
1 0 0
works OK.[/quote:36f33522c0]They found this in gemcombine.cl, but also reproduced the problem in imcombine... gemcombine is just an interface to imcombine.Here are the parameters they had set in gemcombine, as reported to us: [quote:36f33522c0] Options chosen were as follows:
PACKAGE = gemtools
TASK = gemcombine

input = @lii Input MEF images
output = titcub8feb.fits Output MEF image
(title = Titan 8feb06 cube) Title for output SCI plane
(combine= average) Combination operation
(reject = none) Rejection algorithm
(offsets= off) Input image offsets
(masktyp= none) Mask type
(maskval= 0.) Mask value
(scale = none) Image scaling
(zero = none) Image zeropoint offset
(weight = none) Image weights
(statsec= [*,*]) Statistics section
(expname= EXPTIME) Exposure time header keyword
(lthresh= INDEF) Lower threshold
(hthresh= INDEF) Upper threshold
(nlow = 1) minmax: Number of low pixels to reject
(nhigh = 1) minmax: Number of high pixels to reject
More
(nkeep = 1) Minimum to keep or maximum to reject
(mclip = yes) Use median in sigma clipping algorithms?
(lsigma = 3.) Lower sigma clipping factor
(hsigma = 3.) Upper sigma clipping factor
(key_ron= RDNOISE) Keyword for readout noise in e-
(key_gai= GAIN) Keyword for gain in electrons/ADU
(ron = 0.) Readout noise rms in electrons
(gain = 1.) Gain in e-/ADU
(snoise = 0.0) ccdclip: Sensitivity noise (electrons)
(sigscal= 0.1) Tolerance for sigma clipping scaling correction
(pclip = -0.5) pclip: Percentile clipping parameter
(grow = 0.) Radius (pixels) for neighbor rejection
(bpmfile= ) Name of bad pixel mask file or image.
(nrejfil= ) Name of rejected pixel count image.
(sci_ext= SCI) Name(s) or number(s) of science extension
(var_ext= VAR) Name(s) or number(s) of variance extension
(dq_ext = DQ) Name(s) or number(s) of data quality extension
More
(fl_vard= no) Make variance and data quality planes?
(logfile= gemcombine2.log) Log file
(fl_dqpr= no) Propagate all DQ values?
(verbose= yes) Verbose output?
(status = 0) Exit status (0=good)
(flist = ) Internal use only
(mode = ql)[/quote:36f33522c0]I note they don't have the offset file selected there, but clearly I assume that's just due to how they cut and pasted the parameter values. Below is an additional note when they found this also the case for imcombine (note again, the parameters above were for the gemcombine call, that's all I have at the moment to go from)[quote:36f33522c0] 07/06/06 01:52:38 PM rblum
One line summary:
More information on #1913
Details:
The problem reported for #1913 with gemcombine also occurs for imcombine.
Since gemcombine uses imcombine, the problem therefore lies with the latter.
The problem appears to be with only the second dimension index of data cubes.

The problem disappears for extracted 2-D image slices of the data cubes;
e.g., the offset file
0 1 0
2 0 0
allows imcombine to run to completion for a 2-D slice of the data cube. Yet
this same offset file used with the data cube hangs imcombine up, sometimes
also yielding this error message and requiring that the process be killed:

PANIC in `/scisoft/iraf/iraf/bin.macosx/x_images.e': segmentation violation

Otherwise, cntl-C exits the hung process.

The offset files
0 0 0
2 0 8
or
0 1 0
2 1 8
cause no problems with imcombine for data cubes (the 2 and 8 are arbitrary).

I have not seen this problem with the other two dimensions.
Only the Y axis (second index) of only the data cubes produces these problems
(I have only investigated NIFS data cubes generated from reduced obj-sky NIFS nod pairs by gemcube using the Gemini IRAF software.)[/quote:36f33522c0]We also advise the users to report this error to NOAO, sorry if it has been and this is a duplicate, I looked around for it and didn't find it. I suppose you might need data, but I thought I would run it by you right away on the information we have, maybe you can see this problem with local test data, or have seen this problem before? Thanks.
Craig

 
Profile Email
 Quote
fitz
 09/18/2006 11:07PM  
AAAAA
Admin

Status: offline


Registered: 09/30/2005
Posts: 4040
Craig,Frank will have a look when he gets a chance, but I did notice you have a 2-D 'statsec' parameter for a 3-D image, does changing it make any difference (e.g. to a null string or a 3-D section)? I seem to remember this was looked at earlier and no imcombine error could be found, can you get just the IMCOMBINE task from the cmdline to reliably fail or is it always only from gemcombine? Does the fact it's a SCISOFT distribution (what version?) make any difference?-Mike

 
Profile Email
 Quote
callen
 09/18/2006 11:07PM  
+----
Newbie

Status: offline


Registered: 02/01/2006
Posts: 11
For the record, I have not reproduced this myself...The user has advised they do get it to crash with imcombine directly, without using gemcombine at all.Why do you say it's a SCISOFT distribution, I'm not sure about that, perhaps there is something indicating that that I had not noticed. My understanding is this was a regular IRAF installation using the Gemini IRAF reduction package, initially, and then reproduced directly with imcombine, skipping the Gemini Package.thanks.

 
Profile Email
 Quote
fitz
 09/18/2006 11:07PM  
AAAAA
Admin

Status: offline


Registered: 09/30/2005
Posts: 4040
[quote:7fa0466541]Why do you say it's a SCISOFT distribution, I'm not sure about that, perhaps there is something indicating that that I had not noticed. .....[/quote:7fa0466541]The "PANIC in /scisoft/iraf/iraf....." was the giveaway. They normally just repackage the stock IRAF distribution but it's an all-in-one distro of the external packages as well and I can't say for sure the right combination of iraf version plus
tables/stsdas/fitsutil is what is being used, depends on the scisoft version. Verify the versions with the user, narrowing down the problem to a single IMCOMBINE call on a given FITS file is easier to debug than w/ the many lines of GEMCOMBINE interface in front of it.-Mike[/quote]

 
Profile Email
 Quote
callen
 09/18/2006 11:07PM  
+----
Newbie

Status: offline


Registered: 02/01/2006
Posts: 11
ah, I knew you must have seen something like that in there somewhere... I've asked the user for a specific failing imcombine call.I also recommended playing with the statsec parameter as you suggested.

 
Profile Email
 Quote
callen
 09/18/2006 11:07PM  
+----
Newbie

Status: offline


Registered: 02/01/2006
Posts: 11
I got some time to try to reproduce this myself working directly with imcombine.I used a datacube I have as the output of our testgemcube regression test. I combined it with itself using imcombine. It worked fine with no offsets, and it worked fine with offsets that are all 0s0 0 0
0 0 0but hangs with offsets 0 0 0
0 1 0there is no statsec parameter set... here is the epar screen from imcombine as it was run.[quote:491244fc01]
PACKAGE = immatch
TASK = imcombineinput = datacube,datacube List of images to combine
output = dc List of output images
(headers= ) List of header files (optional)
(bpmasks= ) List of bad pixel masks (optional)
(rejmask= ) List of rejection masks (optional)
(nrejmas= ) List of number rejected masks (optional)
(expmask= ) List of exposure masks (optional)
(sigmas = ) List of sigma images (optional)
(logfile= STDOUT) Log file(combine= average) Type of combine operation
(reject = none) Type of rejection
(project= no) Project highest dimension of input images?
(outtype= real) Output image pixel datatype
(outlimi= ) Output limits (x1 x2 y1 y2 ...)
(offsets= offsets) Input image offsets
(masktyp= none) Mask type
(maskval= 0.) Mask value
(blank = 0.) Value if there are no pixels(scale = none) Image scaling
(zero = none) Image zero point offset
(weight = none) Image weights
(statsec= ) Image section for computing statistics
(expname= ) Image header exposure time keyword(lthresh= INDEF) Lower threshold
(hthresh= INDEF) Upper threshold
(nlow = 1) minmax: Number of low pixels to reject
(nhigh = 1) minmax: Number of high pixels to reject
(nkeep = 1) Minimum to keep (pos) or maximum to reject (neg)
(mclip = yes) Use median in sigma clipping algorithms?
(lsigma = 3.) Lower sigma clipping factor
(hsigma = 3.) Upper sigma clipping factor
(rdnoise= 0.) ccdclip: CCD readout noise (electrons)
(gain = 1.) ccdclip: CCD gain (electrons/DN)
(snoise = 0.) ccdclip: Sensitivity noise (fraction)
(sigscal= 0.1) Tolerance for sigma clipping scaling corrections
(pclip = -0.5) pclip: Percentile clipping parameter
(grow = 0.) Radius (pixels) for neighbor rejection
(mode = ql)[/quote:491244fc01]

 
Profile Email
 Quote
fitz
 09/18/2006 11:07PM  
AAAAA
Admin

Status: offline


Registered: 09/30/2005
Posts: 4040
Craig,Could you send me (or post a url) the test image you used. I built a datacube with artdata and don't see a problem. I was using the latest V2.13 on a Mac system, let me know if you did something else. If the process is truly hung the you should be able to attach GDB and get a stack trace, that would help as well.-Mike

 
Profile Email
 Quote
callen
 09/18/2006 11:07PM  
+----
Newbie

Status: offline


Registered: 02/01/2006
Posts: 11
I have not had time to look into this problem the last couple weeks, I'll get you the datacube and offsets file I used, hopefully this week, otherwise, after ADASS.thanks.

 
Profile Email
 Quote
fitz
 09/18/2006 11:07PM  
AAAAA
Admin

Status: offline


Registered: 09/30/2005
Posts: 4040
Hi Craig,I can't quickly find it in the forum mail but I'll append an exchange we had while this was in our part of the Gemini Helpdesk. It includes a test script that also shows no problem with imcombine and may help you track down what is different.Cheers,
-Mike[quote:6edbc53dd2]
From: Frank Valdes <valdes@noao.edu>
Date: July 18, 2006 3:33:45 PM MST
To: rblum@noao.edu
Cc: Tracy Beck <tbeck@gemini.edu>
Subject: Re: [Fwd: Gemini HelpDesk ticket 000001913 - WORK DONE.]Hi,I was away on vacation and then didn't get time to respond. My first
contribution to this is the
following script which demonstrates vanilla 3D cubes with no header
info does not fail with imcombine. So the next step is if one of you
can demonstrate the error with a variant of this or can give me some
header listings that would allow me to create artificial data cubes
that more closely match the problem.Yours,
Frank=== testit.cl ===
# This requires artdata be loaded.# Make two test data cubes.
if (imaccess("test1")==NO)
mkpattern ("test1", output="", pattern="constant",
option="replace", v1=1.,
v2=1., size=1, title="", pixtype="real", ndim=3, ncols=10,
nlines=10,
n3=10, n4=1, n5=1, n6=1, n7=1, header="")
;
if (imaccess("test2")==NO)
mkpattern ("test2", output="", pattern="constant",
option="replace", v1=1.,
v2=1., size=1, title="", pixtype="real", ndim=3, ncols=10,
nlines=10,
n3=10, n4=1, n5=1, n6=1, n7=1, header="")
;# Create a simple offsets file with Y offsets.
if (access("offsets.dat")==NO) {
printf ("0 1 0\n", >> "offsets.dat")
printf ("0 2 0\n", >> "offsets.dat")
}
;# Combine with sum and offsets.
if (imaccess("comb"))
imdel ("comb", verify-)
;
imcombine ("test1,test2", "comb", headers="", bpmasks="", rejmasks="",
nrejmasks="", expmasks="", sigmas="", logfile="STDOUT",
combine="sum",
reject="none", project=no, outtype="real", outlimits="",
offsets="offsets.dat", masktype="none", maskvalue="0", blank=0.,
scale="none", zero="none", weight="none", statsec="", expname="",
lthreshold=INDEF, hthreshold=INDEF, nlow=1, nhigh=1, nkeep=1,
mclip=yes, lsigma=3., hsigma=3., rdnoise="0.", gain="1.",
snoise="0.",
sigscale=0.1, pclip=-0.5, grow=0.)listpix comb[*,5,5],comb[5,*,5],comb[5,5,*]
====cl> cl <testit.clJul 18 15:32: IMCOMBINE
combine = sum, scale = none, zero = none, weight = none
blank = 0.
Images Offsets
test1 0 0 0
test2 0 1 0 Output image = comb, ncombine = 2
1. 2.
2. 2.
3. 2.
4. 2.
5. 2.
6. 2.
7. 2.
8. 2.
9. 2.
10. 2.
1. 1.
2. 2.
3. 2.
4. 2.
5. 2.
6. 2.
7. 2.
8. 2.
9. 2.
10. 2.
11. 1.
1. 2.
2. 2.
3. 2.
4. 2.
5. 2.
6. 2.
7. 2.
8. 2.
9. 2.
10. 2.On Jul 7, 2006, at 3:03 PM, Robert Blum wrote:> Hi Frank,
>
> Can you guys look at this? I escalated this to tier 2, but it just
> dies there and it seems like a NOAO IRAF problem anyway.
>
> The upshot of the whole thing is a fatal error when combining NIFS
> data cubes with imcombine. The user found out that changing the
> axes so that the spectrum was in the 2nd axis worked, but it fails
> if the second axis is a spatial dimension. The user is combining
> multiple cubes with small offsets in position.
>
> Thanks,
>
> -Bob
>
> From: remedy@gemini.edu
> Date: July 7, 2006 2:46:28 PM MST
> To: rblum@noao.edu
> Subject: Gemini HelpDesk ticket 000001913 - WORK DONE.
>
>
> Gemini HelpDesk ticket 000001913 has been updated with WORK DONE by
> rblum.
>
> Track this request at
> http://helpdesk.gemini.edu/cgi-bin/helpdesk/arsmodlogin.cgi?
> id=000001913
>
> Request ID : 000001913
> Sub Category : NIFS
> One line summary : gemcombine does not combine data cubes shifted in Y
> Details : The IRAF routine gemcombine will combine NIFS
> data cubes that are not
> shifted in Y (the second spatial dimension); but hangs with any
> shift
> in that dimension; e.g. offset file for two input cubes:
> 0 0 0
> 0 1 0
> fails when
> 0 0 0
> 0 0 0
> works.
> But
> 0 0 0
> 1 0 0
> works OK.
> Options chosen were as follows:
> PACKAGE = gemtools
> TASK = gemcombine
>
> input = @lii Input MEF images
> output = titcub8feb.fits Output MEF image
> (title = Titan 8feb06 cube) Title for output SCI plane
> (combine= average) Combination operation
> (reject = none) Rejection algorithm
> (offsets= off) Input image offsets
> (masktyp= none) Mask type
> (maskval= 0.) Mask value
> (scale = none) Image scaling
> (zero = none) Image zeropoint offset
> (weight = none) Image weights
> (statsec= [*,*]) Statistics section
> (expname= EXPTIME) Exposure time header keyword
> (lthresh= INDEF) Lower threshold
> (hthresh= INDEF) Upper threshold
> (nlow = 1) minmax: Number of low pixels to
> reject
> (nhigh = 1) minmax: Number of high pixels to
> reject
> More
> (nkeep = 1) Minimum to keep or maximum to
> reject
> (mclip = yes) Use median in sigma clipping
> algorithms?
> (lsigma = 3.) Lower sigma clipping factor
> (hsigma = 3.) Upper sigma clipping factor
> (key_ron= RDNOISE) Keyword for readout noise in e-
> (key_gai= GAIN) Keyword for gain in electrons/ADU
> (ron = 0.) Readout noise rms in electrons
> (gain = 1.) Gain in e-/ADU
> (snoise = 0.0) ccdclip: Sensitivity noise
> (electrons)
> (sigscal= 0.1) Tolerance for sigma clipping
> scaling correction
> (pclip = -0.5) pclip: Percentile clipping
> parameter
> (grow = 0.) Radius (pixels) for neighbor
> rejection
> (bpmfile= ) Name of bad pixel mask file or
> image.
> (nrejfil= ) Name of rejected pixel count image.
> (sci_ext= SCI) Name(s) or number(s) of science
> extension
> (var_ext= VAR) Name(s) or number(s) of variance
> extension
> (dq_ext = DQ) Name(s) or number(s) of data
> quality extension
> More
> (fl_vard= no) Make variance and data quality
> planes?
> (logfile= gemcombine2.log) Log file
> (fl_dqpr= no) Propagate all DQ values?
> (verbose= yes) Verbose output?
> (status = 0) Exit status (0=good)
> (flist = ) Internal use only
> (mode = ql)
> Work Done :
> 07/06/06 10:54:14 AM rblum
> Hi Laurence,
>
> I do not know why the combining fails. I am going to bump
> this up. it would help if you are prepared to send example cubes so
> the IRAF experts can check your images.
>
> -Bob
> 07/06/06 01:52:38 PM rblum
> One line summary:
> More information on #1913
> Details:
> The problem reported for #1913 with gemcombine also occurs
> for imcombine.
> Since gemcombine uses imcombine, the problem therefore lies
> with the latter.
> The problem appears to be with only the second dimension
> index of data cubes.
>
> The problem disappears for extracted 2-D image slices of the
> data cubes;
> e.g., the offset file
> 0 1 0
> 2 0 0
> allows imcombine to run to completion for a 2-D slice of the
> data cube. Yet
> this same offset file used with the data cube hangs imcombine
> up, sometimes
> also yielding this error message and requiring that the
> process be killed:
>
> PANIC in `/scisoft/iraf/iraf/bin.macosx/x_images.e':
> segmentation violation
>
> Otherwise, cntl-C exits the hung process.
>
> The offset files
> 0 0 0
> 2 0 8
> or
> 0 1 0
> 2 1 8
> cause no problems with imcombine for data cubes (the 2 and 8
> are arbitrary).
>
> I have not seen this problem with the other two dimensions.
> Only the Y axis (second index) of only the data cubes
> produces these problems
> (I have only investigated NIFS data cubes generated from
> reduced obj-sky NIFS nod pairs by gemcube using the Gemini IRAF
> software.)
>
> Laurence
> 07/07/06 11:45:45 AM rblum
> I found a workaround for the fact that imcombine does not
> allow shifts
> in its second dimension in the case of data cube input files.
>
> The workaround is to use im3dtran with z-axis flip to rotate
> the data cubes so
> that the wavelength axis, which requires no shift, aligns
> with the old y-axis.
>
> im3dtran catfbrgnN20060208S0163.fits[sci][*,*,-*] c163 1 3 2
>
> Then imcombine works with an offset file to align the output
> image cubes above.
>
> But the bug in imcombine's second index must still be fixed
> for other applications.
>
> Laurence
>
> 07/07/2006 10:26:48 tbeck
> Hi Laurence,
>
> I've been following the discussion on this problem. It's
> really good that you found a work around for this. If you im3dtran
> the cube, do the combine, then im3dtran the cube back then this
> will effectively do what you wanted in the first place, right??
>
> I'm guessing that imcombine probably has not been used
> extensively with offsets for 3-D files. If the problem you see
> with the y-offset not being handled correctly truly is a bug in
> imcombine, then I very strongly urge you to let the main IRAF group
> know about this.
>
> Of course, if you had found a bug with our gemcombine script
> then we would have worked to fix it. But imcombine is one of the
> backbone NOAO IRAF tasks and we can't fix the bugs in the general
> IRAF package. We can only fix bugs in our Gemini IRAF package.
>
> So, please do let NOAO IRAF group know of this problem in
> imcombine. It would be great if they can fix this on their side so
> we can get the combining of NIFS datacubes working with shift files.
>
> Thanks very much for working on this and keeping us updated
> on your progress,
>
[/quote:6edbc53dd2]

 
Profile Email
 Quote
   
Content generated in: 0.27 seconds
New Topic Post Reply

Normal Topic Normal Topic
Sticky Topic Sticky Topic
Locked Topic Locked Topic
New Post New Post
Sticky Topic W/ New Post Sticky Topic W/ New Post
Locked Topic W/ New Post Locked Topic W/ New Post
View Anonymous Posts 
Anonymous users can post 
Filtered HTML Allowed 
Censored Content 
dog allergies remedies cialis 20 mg chilblain remedies


Privacy Policy
Terms of Use

User Functions

Login