Welcome to iraf.net Friday, May 03 2024 @ 10:24 PM GMT


 Forum Index > Help Desk > Applications New Topic Post Reply
 combining eso wfi data still shows "lines" after a
   
wierdling
 03/21/2009 05:58AM (Read 5986 times)  
+----
Newbie

Status: offline


Registered: 03/20/2009
Posts: 6
Hello all,I just recently found that I can download data from the ESO WFI telescope, but on some of the data, after processing, there are still "streaks" (in the final stacked image).Please forgive me if I use the wrong terms, I don't have any formal training (everything I know about IRAF I have taught myself).I would like to know if anyone has any ideas on what I am doing wrong (I am just looking to make "pretty pictures" from the data, not scientific stuff).Here is the final image (with settings showing the "lines"): http://wierdling.longboys.net/iraf/irafImage.jpgI am running IRAF on a CENTOS os. Here are the steps that I do (it is for some data for M 83 by the way):
PHP Formatted Code

esowfi> esohdr W*.fits
esowfi> esohdr W*.fits
esowfi> epar ccdproc -- see #1
esowfi> epar zerocombine -- see #2
esowfi> zerocombine W*.fits
esowfi> epar flatcombine -- see #3
esowfi> flatcombine W*.fits -- take output and make a list called flatfiles
esowfi> epar ccdproc -- see #4
esowfi> ccdproc W*.fits
esowfi> msczero -- update zero reference for each image in the list
esowfi> msczero -- create list of coords
esowfi> epar msccmatch -- see #5
esowfi> msccmatch
esowfi> epar mscimage -- see #6
esowfi> mscimage
esowfi> epar mscimatch -- see #7
esowfi> mscimatch -- see #8 for output
esowfi> mscstack

#1 -- epar for ccdproc

                                   I R A F  
                    Image Reduction and Analysis Facility
PACKAGE = mscred
   TASK = ccdproc

images  =               W*fits  List of Mosaic CCD images to process
(output =                     ) List of output processed images
(bpmasks=                     ) List of output bad pixel masks
(ccdtype=               object) CCD image type to process
(noproc =                   no) List processing steps only?

(xtalkco=                   no) Apply crosstalk correction?
(fixpix =                  yes) Apply bad pixel mask correction?
(oversca=                  yes) Apply overscan strip correction?
(trim   =                  yes) Trim the image?
(zerocor=                   no) Apply zero level correction?
(darkcor=                   no) Apply dark count correction?
(flatcor=                   no) Apply flat field correction?
(sflatco=                   no) Apply sky flat field correction?
(split  =                   no) Use split images during processing?
(merge  =                   no) Merge amplifiers from same CCD?

(xtalkfi=                     ) Crosstalk file
(fixfile=             @badPics) List of bad pixel masks
(saturat=                INDEF) Saturated pixel threshold
(sgrow  =                    0) Saturated pixel grow radius
(bleed  =                INDEF) Bleed pixel threshold
(btrail =                   20) Bleed trail minimum length
(bgrow  =                    0) Bleed pixel grow radius
(biassec=             !biassec) Overscan strip image section
(trimsec=             !trimsec) Trim data section
(zero   =                     ) List of zero level calibration images
(dark   =                     ) List of dark count calibration images
(flat   =                     ) List of flat field images
(sflat  =                     ) List of secondary flat field images
(minrepl=                   1.) Minimum flat field value

(interac=                   no) Fit overscan interactively?
(functio=             legendre) Fitting function
(order  =                    4) Number of polynomial terms or spline pieces
(sample =                    *) Sample points to fit
(naverag=                    1) Number of sample points to combine
(niterat=                    1) Number of rejection iterations
(low_rej=                   3.) Low sigma rejection factor
(high_re=                   3.) High sigma rejection factor
(grow   =                   0.) Rejection growing radius
(mode   =                   ql)


#2 -- epar for zerocombine
                                  I R A F  
                    Image Reduction and Analysis Facility
PACKAGE = mscred
   TASK = zerocombine

input   =              W*.fits  List of zero level images to combine
(output =                 Zero) Output zero level name
(combine=              average) Type of combine operation
(reject =            avsigclip) Type of rejection
(ccdtype=                 zero) CCD image type to combine
(process=                  yes) Process images before combining?
(delete =                   no) Delete input images after combining?
(scale  =                 none) Image scaling
(statsec=                     ) Image section for computing statistics
(nlow   =                    0) minmax: Number of low pixels to reject
(nhigh  =                    1) minmax: Number of high pixels to reject
(nkeep  =                    1) Minimum to keep (pos) or maximum to reject (neg)
(mclip  =                  yes) Use median in sigma clipping algorithms?
(lsigma =                   3.) Lower sigma clipping factor
(hsigma =                   3.) Upper sigma clipping factor
(rdnoise=                   0.) ccdclip: CCD readout noise (electrons)
(gain   =                   1.) ccdclip: CCD gain (electrons/DN)
(snoise =                   0.) ccdclip: Sensitivity noise (fraction)
(pclip  =                 -0.5) pclip: Percentile clipping parameter
(blank  =                   0.) Value if there are no pixels
(mode   =                   ql)


#3 -- epar for flatcombine
                                  I R A F  
                    Image Reduction and Analysis Facility
PACKAGE = mscred
   TASK = flatcombine

input   =              W*.fits  List of flat field images to combine
(output =                    F) Output flat field root name
(combine=              average) Type of combine operation
(reject =            avsigclip) Type of rejection
(ccdtype=                 flat) CCD image type to combine
(process=                  yes) Process images before combining?
(subsets=                  yes) Combine images by subset parameter?
(delete =                   no) Delete input images after combining?
(scale  =                 mode) Image scaling
(statsec=                     ) Image section for computing statistics
(nlow   =                    1) minmax: Number of low pixels to reject
(nhigh  =                    1) minmax: Number of high pixels to reject
(nkeep  =                    1) Minimum to keep (pos) or maximum to reject (neg)
(mclip  =                  yes) Use median in sigma clipping algorithms?
(lsigma =                   3.) Lower sigma clipping factor
(hsigma =                   3.) Upper sigma clipping factor
(rdnoise=                   0.) ccdclip: CCD readout noise (electrons)
(gain   =                   1.) ccdclip: CCD gain (electrons/DN)
(snoise =                   0.) ccdclip: Sensitivity noise (fraction)
(pclip  =                 -0.5) pclip: Percentile clipping parameter
(blank  =                   1.) Value if there are no pixels
(mode   =                   ql)



#4 -- epar for ccdproc
                                  I R A F  
                    Image Reduction and Analysis Facility
PACKAGE = mscred
   TASK = ccdproc

images  =               W*fits  List of Mosaic CCD images to process
(output =                     ) List of output processed images
(bpmasks=                     ) List of output bad pixel masks
(ccdtype=               object) CCD image type to process
(noproc =                   no) List processing steps only?

(xtalkco=                   no) Apply crosstalk correction?
(fixpix =                  yes) Apply bad pixel mask correction?
(oversca=                  yes) Apply overscan strip correction?
(trim   =                  yes) Trim the image?
(zerocor=                  yes) Apply zero level correction?
(darkcor=                   no) Apply dark count correction?
(flatcor=                  yes) Apply flat field correction?
(sflatco=                   no) Apply sky flat field correction?
(split  =                   no) Use split images during processing?
(merge  =                   no) Merge amplifiers from same CCD?

(xtalkfi=                     ) Crosstalk file
(fixfile=             @badPics) List of bad pixel masks
(saturat=                INDEF) Saturated pixel threshold
(sgrow  =                    0) Saturated pixel grow radius
(bleed  =                INDEF) Bleed pixel threshold
(btrail =                   20) Bleed trail minimum length
(bgrow  =                    0) Bleed pixel grow radius
(biassec=             !biassec) Overscan strip image section
(trimsec=             !trimsec) Trim data section
(zero   =                 Zero) List of zero level calibration images
(dark   =                     ) List of dark count calibration images
(flat   =           @flatfiles) List of flat field images
(sflat  =                     ) List of secondary flat field images
(minrepl=                   1.) Minimum flat field value

(interac=                   no) Fit overscan interactively?
(functio=             legendre) Fitting function
(order  =                    4) Number of polynomial terms or spline pieces
(sample =                    *) Sample points to fit
(naverag=                    1) Number of sample points to combine
(niterat=                    1) Number of rejection iterations
(low_rej=                   3.) Low sigma rejection factor
(high_re=                   3.) High sigma rejection factor
(grow   =                   0.) Rejection growing radius

#5 -- epar for msccmatch
                                  I R A F  
                    Image Reduction and Analysis Facility
PACKAGE = mscred
   TASK = msccmatch

input   =                @rcIn  List of input mosaic exposures
coords  =             rccoords  Coordinate file (ra/dec)
(outcoor=                     ) List of updated coordinate files
(usebpm =                  yes) Use bad pixel masks?
(verbose=                   no) Verbose?

                                # Coarse Search
(nsearch=                   10) Maximum number of positions to use in search
(search =                   3.) Translation search radius (arcsec)
(rsearch=                   5.) Rotation search radius (deg)

                                # Fine Centroiding
(cbox   =                   11) Centering box (pixels)
(maxshif=                   5.) Maximum centering shift to accept (arcsec)
(csig   =                  0.1) Maximum centering uncertainty to accept (arcsec)
(cfrac  =                  0.5) Minimum fraction of accepted centers
(listcoo=                  yes) List centered coordinates in verbose mode?

                                # WCS Fitting
(nfit   =                    4) Min for fit (>0) or max not found (<=0)
(rms    =                   2.) Maximum fit RMS to accept (arcsec)
(fitgeom=              general) Fitting geometry
(reject =                   3.) Fitting rejection limit (sigma)
(update =                  yes) Update coordinate systems?
(interac=                   no) Interactive?
(fit    =                  yes) Interactive fitting?
(graphic=             stdgraph) Graphics device
(cursor =                     ) Graphics cursor

accept  =                  yes  Accept solution?
(mode   =                   ql)

#6 -- epar for mscimage
                                  I R A F  
                    Image Reduction and Analysis Facility
PACKAGE = mscred
   TASK = mscimage

input   =                @rcIn  List of input mosaic exposures
output  =               @rcOut  List of output images
(format =                image) Output format (image|mef)
(pixmask=                  yes) Create pixel mask?
(verbose=           )_.verbose) Verbose output?

                                # Output WCS parameters
(wcssour=                image) Output WCS source (image|parameters)
(referen=                     ) Reference image
(ra     =                INDEF) RA of tangent point (hours)
(dec    =                INDEF) DEC of tangent point (degrees)
(scale  =                INDEF) Scale (arcsec/pixel)
(rotatio=                INDEF) Rotation of DEC from N to E (degrees)

                                # Resampling parmeters
(blank  =                   0.) Blank value
(interpo=               linear) Interpolant for data
(minterp=               linear) Interpolant for mask
(boundar=              reflect) Boundary extension
(constan=                   0.) Constant boundary extension value
(fluxcon=                   no) Preserve flux per unit area?
(ntrim  =                    8) Edge trim in each extension
(nxblock=                 2048) X dimension of working block size in pixels
(nyblock=                 1024) Y dimension of working block size in pixels

                                # Geometric mapping parameters
(interac=                   no) Fit mapping interactively?
(nx     =                   10) Number of x grid points
(ny     =                   20) Number of y grid points
(fitgeom=              general) Fitting geometry
(xxorder=                    4) Order of x fit in x
(xyorder=                    4) Order of x fit in y
(xxterms=                 half) X fit cross terms type
(yxorder=                    4) Order of y fit in x
(yyorder=                    4) Order of y fit in y
(yxterms=                 half) Y fit cross terms type


(fd_in  =                     )
(fd_ext =                     )
(fd_coor=                     )
(mode   =                   ql)

#7 -- epar for mscimatch
                                  I R A F  
                    Image Reduction and Analysis Facility
PACKAGE = mscred
   TASK = mscimatch

input   =               @rcOut  List of images
coords  =             rccoords  File of coordinates
(bpm    =               @rcbpm) List of bad pixel masks
(measure=                     ) Measurment file
(scale  =                  yes) Determine scale?
(zero   =                  yes) Determine zero offset?
(box1   =                   21) Inner box size for statistics
(box2   =                   51) Outer box size for statistics
(lower  =                   1.) Lower limit for good data
(upper  =                INDEF) Upper limit for good data
(niterat=                    3) Number of sigma clipping iterations
(sigma  =                   3.) Sigma clipping factor
(interac=                   no) Interactive?
(verbose=                  yes) Verbose?
accept  =                  yes  Accept scaling and update images?
(mode   =                   ql)

#8 -- output for mscimatch
MSCIMATCH:
  Reading region coordinates from rccoords
    48 coordinates read
  Measuring regions in rc1.fits ...
  Using bad pixel mask rc1_bpm.pl ...
    48 good regions measured
  Measuring regions in rc2.fits ...
  Using bad pixel mask rc2_bpm.pl ...
    40 good regions measured
  Measuring regions in rc3.fits ...
  Using bad pixel mask rc3_bpm.pl ...
    47 good regions measured
  Measuring regions in rc4.fits ...
  Using bad pixel mask rc4_bpm.pl ...
    45 good regions measured
  Measuring regions in rc5.fits ...
  Using bad pixel mask rc5_bpm.pl ...
    40 good regions measured
  Determining scale factors ...
            rc1.fits: 1.0000 (0.0000)     0.00 (0.00)
            rc2.fits: 1.1855 (0.0802)   336.62 (858.67)
            rc3.fits: 1.0588 (0.0380)   108.96 (246.84)
            rc4.fits: 0.9974 (0.0454)   171.42 (232.07)
            rc5.fits: 1.0047 (0.0445)  -192.14 (776.87)
Accept scaling and update images? (yes):


#9  -- epar mscstack
                                  I R A F  
                    Image Reduction and Analysis Facility
PACKAGE = mscred
   TASK = mscstack

input   =               @rcOut  List of images to combine
output  =         m83_rc5.fits  Output image
(headers=                     ) List of header files (optional)
(bpmasks=                     ) List of bad pixel masks (optional)
(rejmask=                     ) List of rejection masks (optional)
(nrejmas=                     ) List of number rejected masks (optional)
(expmask=                     ) List of exposure masks (optional)
(sigmas =                     ) List of sigma images (optional)

(combine=               median) Type of combine operation (median|average)
(reject =            avsigclip) Type of rejection
(masktyp=            goodvalue) Mask type
(maskval=                   0.) Mask value
(blank  =                   0.) Value if there are no pixels

(scale  =            !mscscale) Image scaling
(zero   =             !msczero) Image zero point offset
(weight =                 none) Image weights
(statsec=                     ) Image section for computing statistics

(lthresh=                   1.) Lower threshold
(hthresh=                INDEF) Upper threshold
(nlow   =                    1) minmax: Number of low pixels to reject
(nhigh  =                    1) minmax: Number of high pixels to reject
(nkeep  =                    1) Minimum to keep (pos) or maximum to reject (neg)
(mclip  =                  yes) Use median in sigma clipping algorithms?
(lsigma =                   3.) Lower sigma clipping factor
(hsigma =                   3.) Upper sigma clipping factor
(rdnoise=                   0.) ccdclip: CCD readout noise (electrons)
(gain   =                   1.) ccdclip: CCD gain (electrons/DN)
(snoise =                   0.) ccdclip: Sensitivity noise (fraction)
(sigscal=                  0.1) Tolerance for sigma clipping scaling corrections
(pclip  =                 -0.5) pclip: Percentile clipping parameter
(grow   =                   0.) Radius (pixels) for neighbor rejection
(mode   =                   ql)
 

 
Profile Email Website
 Quote
valdes
 03/21/2009 05:58AM  
+++++
Active Member

Status: offline


Registered: 11/11/2005
Posts: 728
Hello,You have done quite well in understanding the processing order and tasks. Everything you sent looks reasonable and, to first order, the fact that you get a stack with things registered is a good thing.Clearly the stripes result from the mscstack step when it deals with the gaps in the mosaic. I suspect that the gaps are not being recognized which is the responsibility of the bad pixel masks. In mscstack you seem to have the right parameters for the masks. Could you post the output of mscstack? You should display a mask to make sure it makes sense. When you display a mask it may be useful to specifying the display thresholds since any automatic algorithm wil see mostly zero.The only other comment I would make, which I don't think would make much of a difference in your final image, is to use "combine=average" instead of a median and to not use any rejection. The values you are using are useful for rejecting things like cosmic rays but in a large field "pretty picture" these would not be visible or important and an average gives a better "deeper" result.So let me look at the output of mscstack. Another useful output would be the results of mimstat (masked statistics). Give the list going into imcombine, a list of the matching masks for "imasks", and use a small amount of clipping "nclip=3".Yours,
Frank Valdes

 
Profile Email
 Quote
wierdling
 03/21/2009 05:58AM  
+----
Newbie

Status: offline


Registered: 03/20/2009
Posts: 6
Thank you Frank,I will re-run my data tonight and post the requested output.Sincerely,James

 
Profile Email Website
 Quote
wierdling
 03/21/2009 05:58AM  
+----
Newbie

Status: offline


Registered: 03/20/2009
Posts: 6
Here is the requested output.From mscstack:
[code:1:66f975145f]
Mar 27 22:44: IMCOMBINE
combine = average, scale = , zero = , weight =
reject = minmax, nlow = 1, nhigh = 1
lthreshold = 1.
blank = 0.
masktype = goodval, maskval = 0
Images Exp Offsets Maskfile
rc1.fits 299.9 268 198 rc1_bpm
rc2.fits 299.9 133 301 rc2_bpm
rc3.fits 299.9 402 400 rc3_bpm
rc4.fits 299.9 537 97 rc4_bpm
rc5.fits 299.9 0 0 rc5_bpm Output image = m83_rc9, ncombine = 40
[/code:1:66f975145f]And from mimstat:[code:1:66f975145f]
esowfi> mimstat
List of input images (@rcOut): @rcOut
# IMAGE NPIX MEAN STDDEV MIN MAX
rc1.fits 62856249 807.5 34.43 635.9 996.1
rc2.fits 63174698 991.9 33.44 849.6 1146.
rc3.fits 62996935 761.4 34.85 587.8 952.1
rc4.fits 62818543 624. 34.3 448.3 817.6
rc5.fits 62734268 531.1 35.64 346.4 734.9[/code:1:66f975145f]I tried running mscstack with various settings:MASKTYPE set to badvalues, which gave me a black image with some white spots,
masktype set to "!BPM", which didn't do anything new, (as expected)I also converted one of the .pl files (bad pixel masks) to a fits image and viewed it in DS9. With the low value set to 0, and the high value set to 1, it showed the chip gaps in white with the "image" areas in black.Setting combine to "average" did reduce the chip gap artifacts a bit, but they are still there.Thank you,James

 
Profile Email Website
 Quote
valdes
 03/21/2009 05:58AM  
+++++
Active Member

Status: offline


Registered: 11/11/2005
Posts: 728
Hi James,I see a month has slipped away from me already.Thank you for the reply. The output you sent doesn't show anything wrong to me. So I don't have anything more to contribute. It remains a challenge to make a composite image without the kinds of streak artifacts you are working to minimize. So all I can really offer is that you seem to be doing the right things and that getting rid of the streaks requires working on getting the masking right and the relative scalings correct.Yours,
Frank

 
Profile Email
 Quote
Anonymous:
 03/21/2009 05:58AM  



Hi,Maybe my experience can help James.
I had run CCDPROC with fixpix = yes. The fixfile in this case was in terms of the location of bad rows and columns and not as a mask. In my case the streaks in the 2 D images disappear after I ran CCDPROC. This thing was not explained in Massey's guide on Iraf reduction but in the guide for echelle data reduction.Cheers,Siti

 
 Quote
wierdling
 03/21/2009 05:58AM  
+----
Newbie

Status: offline


Registered: 03/20/2009
Posts: 6
I am really new to using IRAF for reduction of data that didn't come from the Hubble, so my mistake was probably a newbie one. After playing with other data, I have found that even if some datasets (plates, exposures, what do you call them?) have the same exposure time, they can still be "brighter" or "dimmer" (which is probably the exact wrong way to describe them). When combining images that are basically not the same brightness, it gives the lines. Now all I need to do is figure out a way to "level" the images so they all have the same brightness. Thanks for the help though Smile This is a darned good forum.

 
Profile Email Website
 Quote
Anonymous:
 03/21/2009 05:58AM  



[quote:1ce0de79ee]
I have found that even if some datasets (plates, exposures, what do you call them?) have the same exposure time, they can still be "brighter" or "dimmer" (which is probably the exact wrong way to describe them)[/quote:1ce0de79ee]Strange because if the detector is a CCD, the intensity should be linear with respect to exposure time.[quote:1ce0de79ee]
Now all I need to do is figure out a way to "level" the images so they all have the same brightness.[/quote:1ce0de79ee]A simple way to do it would be to apply IMSTAT on the images to get the average pixel reading then IMARITH them to add/subtract/multiply etc. Maybe there are more elegant ways to do it, I don't know.If you just want to make pretty pictures, wouldn't it be easier to just edit those lines? See
https://iraf.net/irafdocs/clean/Siti

 
 Quote
   
Content generated in: 0.20 seconds
New Topic Post Reply

Normal Topic Normal Topic
Sticky Topic Sticky Topic
Locked Topic Locked Topic
New Post New Post
Sticky Topic W/ New Post Sticky Topic W/ New Post
Locked Topic W/ New Post Locked Topic W/ New Post
View Anonymous Posts 
Anonymous users can post 
Filtered HTML Allowed 
Censored Content 
dog allergies remedies cialis 20 mg chilblain remedies


Privacy Policy
Terms of Use

User Functions

Login