Welcome to iraf.net Friday, March 29 2024 @ 04:45 AM GMT


 Forum Index > Help Desk > General IRAF New Topic Post Reply
 More DECam image questions and issues
   
andrej
 04/25/2015 02:23AM (Read 4746 times)  
+++--
Chatty

Status: offline


Registered: 08/21/2009
Posts: 57
Greetings.
I am having issues with imcombine on some DECam data. I am discovering that there is some slight blur/misalignment when using imcombine for multiple dither points of a single frame. I am using combine=median and offset=wcs.


*EXAMPLE WITH DECAM FRAME N3*

I have linked you to an example of one of the CCDs (frame N3). Each file was generated from an original stack of five subs, already median-combined and trimmed. Each file is one of the 10-point dithers ("D01" to "D10" appear in the file names correspondingly).

I have also included one of the imcombined FITS files in the folder:

https://drive.google.com/folderview?id=0B_eLF8j7l3I-fkFZdXFPMWdzSHJoZEJGQ1o2Y2ZtVU9RdDNJSWNac3JsMDJibm5jTTdhcVE&usp=sharing

I noticed that when we combine the 10 dither points using imcombine, the alignment is slightly off. See the combined fits file "MicroscopiumMedianFN3Combined.fits" in the drive link. I don’t understand why the images produced by imcombine are slightly offset when the 10-points of the image are merged together.


*EXAMPLE WITH DECAM FRAME N1*

Even more bizarre, is when we choose a different frame (say, frame N1) and combine the 10 dither points using this frame, imcombine doesn't work right, and the resulting image has echoing. See the combined fits file "MicroscopiumCombinedN1.fits" in the drive link.


I don't understand why imcombine is producing these kinds of blurry images even though we have set offset=wcs. The offsets for the stack of 10 theoretically ought to be the same since each point of the dither is of the same target for all CCDs simultaneously.

I mean, we could make a manual offset adjustment to the DECam data, but for each CCD and dither point that just seems like a lot of manual work.

What's going on?

 
Profile Email
 Quote
valdes
 04/27/2015 10:20PM  
+++++
Active Member

Status: offline


Registered: 11/11/2005
Posts: 728
Hello,

My first question is what data product type are you using? This would be the PROCTYPE keyword or, for more recent data, the name of the downloaded file would have something lik ooi or opi. The point here is that one type, called InstCal, has a calibrated world coordinate system but has not been resampled to a distortion free grid. The other, called Resampled, should allow good registration and combining with IMCOMBINE provided you use offsets=world.

Yours,
Frank Valdes

 
Profile Email
 Quote
andrej
 04/29/2015 03:33PM  
+++--
Chatty

Status: offline


Registered: 08/21/2009
Posts: 57
They are all InstCal and the name of the file has "ooi"

So, should I just throw out all the InstCal files and start again with Resampled? Is there anything different about the image quality (or any sort of manual calibration that I need to do) that is in InstCal that isn't in Resampled?

 
Profile Email
 Quote
valdes
 04/29/2015 03:46PM  
+++++
Active Member

Status: offline


Registered: 11/11/2005
Posts: 728
Yes that is right, the simplest way to combine is to use the rsampled versions with imcombine using the parameter offsets="world". The resampled versions exactly align except for an integer offset. Note this only applies when the exposures have the same tangent point: the keywords CRVAL1 and CRVAL2. This will be the case for pipeline reduced data if the exposures largely overlap.

The only difference between InstCal and Resampled is the remapping of the pixels to a standard orientation and grid with distortions removed. In some cases the resampled versions will also have a local (within a CCD) sky structure subtracted but this onlly happens in certain types of datasets. The remapping is done with a high quality sinc interpolation. Some people do science on the InstCal to avoid any concerns about interpolation, but combining pixel data can only be done with a remapping step. So scientifically the two types are essentially the same.

Frank

 
Profile Email
 Quote
andrej
 04/30/2015 10:06PM  
+++--
Chatty

Status: offline


Registered: 08/21/2009
Posts: 57
I appreciate the clarification and with that I plan to retrieve the Resampled data as appropriate.

 
Profile Email
 Quote
andrej
 05/11/2015 04:37PM  
+++--
Chatty

Status: offline


Registered: 08/21/2009
Posts: 57
So I am trying to download the Resampled versions of the DECam image files, instead of the InstCal files. I have made absolutely sure that I am retrieving files that correspond to my observations and with the correct time stamps.

But I am encountering two weird issues:

[1] While the "InstCal" fits.fz files ("ooi"), which contain the full DECam field, are about 315 MB each, the corresponding "Resampled" fits.fz files ("opi") are only 4.7 MB each and contain only the frame [S7]. Why do the resampled data files not contain the full DECam field?

[2] Also, I am having trouble with using the Download Manager on a workstation. The NOAO Science Archive Download Manager, the one that uses Java, will download these 4.7 MB resampled image files fine, but it won't download the 315 MB files. The manager just hangs there forever when it gets to these files and will not download them. I don't understand why it is having these problems since I used to be able to use this same program to download the 315 MB files last year.

What's going on? How do I get the full "Resampled" DECam field images so that I can do imcombine?

 
Profile Email
 Quote
valdes
 05/12/2015 04:21PM  
+++++
Active Member

Status: offline


Registered: 11/11/2005
Posts: 728
Hi,

There was a bug in the pipeline for a period of time where the resampled files contained only one extension as you found. The immediate solution is to reprocess. Please tell me the calendar nights you are interested in. I'm sorry for the problems.

As for the second problem you should report it to sdmhelp@noao.edu or Mark Dickinson at med@noao.edu.

Yours,
Frank

 
Profile Email
 Quote
andrej
 05/14/2015 02:41PM  
+++--
Chatty

Status: offline


Registered: 08/21/2009
Posts: 57
Hi, the nights are:

July 2, 2014 and July 3, 2014

 
Profile Email
 Quote
valdes
 05/18/2015 11:04PM  
+++++
Active Member

Status: offline


Registered: 11/11/2005
Posts: 728
Hi,

The data have been reprocessed. When you go to the NOAO portal you can put the string "v2" in the filename section of the query form to select only this latest processing.

Because the fields in your program contain very large sources the global or local sky subtraction version may not be to your liking, though photometry should be unaffected since nearby sky estimation is usually part of the process.

Yours,
Frank Valdes

 
Profile Email
 Quote
andrej
 05/19/2015 11:19PM  
+++--
Chatty

Status: offline


Registered: 08/21/2009
Posts: 57
Alright. When I put

v2

into the filename section of the query form, it keeps telling me

"Result Set Empty"

I tried both with and without wildcards and it gave the same result. I tried both the Original Filename and the Archive Filename parameters, but that didn't make it work. I made sure to put in my program number and unchecked everything else. So, I'm not sure what's going on. Though I do see the new files with the "v2" in the file name when I query all of the files for my program, so I can weed out the necessary files manually if need be.

 
Profile Email
 Quote
constant
 05/29/2015 03:22AM  
+----
Newbie

Status: offline


Registered: 05/26/2015
Posts: 9
Hi,
I am combining some DECam data these days. I found a software that called SWARP may do these things easily. I also want to use iraf to combine them. I have some trouble in getting the shift in different images.Because I have a lot of images, I don't want to use imexamine to get the shift of each image like this website:http://www.astro.washington.edu/courses/astro480/co-add_unm.html. Could you please help me, if possible.

 
Profile Email
 Quote
fitz
 05/29/2015 03:49AM  
AAAAA
Admin

Status: offline


Registered: 09/30/2005
Posts: 4040

If you're using the pipeline reduced data you can probably just use the offset="wcs" option to IMCOMBINE to use the WCS to derive the image offsets automatically, e.g.

cl\$this->_split2($m[0]) imcombine *.fits new.fits offset="wcs"


 
Profile Email
 Quote
constant
 05/31/2015 04:52AM  
+----
Newbie

Status: offline


Registered: 05/26/2015
Posts: 9
Thank you. Big Grin

 
Profile Email
 Quote
andrej
 06/01/2015 10:06PM  
+++--
Chatty

Status: offline


Registered: 08/21/2009
Posts: 57
That is what I am doing as well. However, I am trying to create a master mosaic of 600 FITS files (I took each of the original 60 CCD images, cleaned up the data one frame at a time, and combined it with 10 dither points)... the data are all in a 3 sq. degree region of the sky.

So I just want to use imcombine to my 600 FITS files, each 30 MB in size, so that I have a master mosaic, so that I can use Source Extractor to do a weak lensing analysis of the whole wide field.

But when I try to run my routine on my files, using

imcombine input=M* output=MCombined.fits offset=wcs combine=median

I receive the following error message:

"ERROR: Out of space in image header (IMCMB498)"

How do I get around this?

 
Profile Email
 Quote
fitz
 06/01/2015 10:12PM  
AAAAA
Admin

Status: offline


Registered: 09/30/2005
Posts: 4040
The size of an output image header is controlled by the 'min_lenuserarea' environment variable, this is set as a system default in the hlib$zzsetenv.def file or optionally in your login.cl. The default value is 64000 chars but can be increased as needed before running the task. For example,

cl\$this->_split2($m[0]) reset min_lenuserarea = 256000
cl\$this->_split2($m[0]) flpr 0 # zero, no oh, to flush all processes
cl\$this->_split2($m[0]) imcombine *.fits .......

You can uncomment the setting and change the value in your login.cl as well.

 
Profile Email
 Quote
andrej
 06/03/2015 03:12PM  
+++--
Chatty

Status: offline


Registered: 08/21/2009
Posts: 57
This worked. I again appreciate the help.

 
Profile Email
 Quote
andrej
 07/16/2015 10:19PM  
+++--
Chatty

Status: offline


Registered: 08/21/2009
Posts: 57
So, I have an update.

I am able to create a master mosaic, consisting of 10 dither points in a line, and 5 subs within each dither.

With the Resampled data, there is still a bright section caused by frame S7. In each sub, half of the image (sections 1024x4096) is about 70 points brighter in value than the rest of the image.
I realize that I could do imarith and manually adjust the brightness but could you point me to a routine that can correct this?

Also, the master mosaic has sections of lighter and darker due to the dithers:
https://drive.google.com/file/d/0B_eLF8j7l3I-aV9JVXpnSU9YNEU/view?usp=sharing

I first tried imcombine with scale=none. Then when I tried imcombine with scale=mode, that made the image a lot more uniform... but it still had these narrow straight segments of lines that look like trim lines:
https://drive.google.com/file/d/0B_eLF8j7l3I-bGdvdkNManJFcWM/view?usp=sharing


[1] How can I get a more uniform mosaic?

[2] Has anyone published a general pipeline for how to go about generally stacking the DECam Resampled data and then preparing the analysis for Source Extractor?

 
Profile Email
 Quote
constant
 07/18/2015 11:16AM  
+----
Newbie

Status: offline


Registered: 05/26/2015
Posts: 9
Hi,
I found some sentence about combining DECam images in NOAO Data Handbook.

http://archive.noao.edu/doc/NOAO_DHB/NOAO_Data_Handbookv1.1.pdf

Image Stacking
The CP(Community Pipeline) combines exposures that overlap significantly (exposures are considered overlapping when they have the same tangent-point after remapping). Exposure patterns that map fields larger in radius than a DECam FoV will generally produce separate co-adds rather than one very large one. The overlapping exposures are separated into multiple stacks by exposure time: very short (t 300s). If a group has more than 50 exposures it is divided in the smallest number of subgroups which all have less than 50 exposures (not for scientific reasons, but rather to make the data handling manageable). Of the exposures that are identified for a stack, some are exclud- ed if they have the potential to degrade the image quality of the stack. The following metrics are exam- ined to detect and remove such outliers:
\$this-\$this->_split2($m[0])_normalize_entities2($m[0]) Relative flux scaling (e.g., low magnitude zero-points due to clouds)
\$this-\$this->_split2($m[0])_normalize_entities2($m[0]) Seeing
\$this-\$this->_split2($m[0])_normalize_entities2($m[0]) Sky brightness
Note that fewer than half of the full-FPA exposures are allowed to be rejected; otherwise all exposures are used.
When stacking, the pipeline scales the input images to a common magnitude zero-point and sky transpar- ency. The co-adds are the weighted sum of the contributing exposures, excluding pixels marked in the data quality mask. There are two stacks produced for each set of exposures that satisfy the criteria above. One has no background adjustments beyond those applied to the non-remapped data as described previ- ously. The other subtracts an additional background from each CCD. This is a higher spatial frequency filtering, which can produce a better matching of overlapping CCDs but can also subtract light from sources subtending 10% or more of a CCD. Because it is difficult for the pipeline to decide which is ap- propriate for a data set, the two versions are produced from which the investigator can choose.

But I can not find the Pipeline. Maybe it can help you. If you find the Pipeline, please tell me.

I am also combining DECam images this days.I have some question. The PROCTYPE is resampled. Do you have some data which PRODTYPE is wtmap or dqmask? How do you deal with those data?

 
Profile Email
 Quote
constant
 07/18/2015 11:27AM  
+----
Newbie

Status: offline


Registered: 05/26/2015
Posts: 9
And how do you deal with background? I used to subtract the average of the image.

 
Profile Email
 Quote
valdes
 07/24/2015 09:47PM  
+++++
Active Member

Status: offline


Registered: 11/11/2005
Posts: 728
This subject about restacking starting with the projected images (the opi ones) is something that is coming up with multiple people. Since we do say that this is what people should do we really need to help people do it. So I need to write a guide for how to stack the projected images from DECam. I now understand the term pipeline means "how to do the steps". I will find time to write a guide. Note that this will involve using IRAF imcombine. So those who want to use other software would need to interpret the steps appropriately.

Frank

 
Profile Email
 Quote
rdzudzar
 01/14/2018 10:58PM  
+----
Newbie

Status: offline


Registered: 02/13/2017
Posts: 3
Hi, I have a question about DES DECam mozaicking.
I have for example: c4d_140123_032052_oow_g_d2.fits.fz file, it's InstCal, however, I can't find Resampled. I used FUNPACK and then IRAFs imcombine and this fails to make a combination of all frames within that file. I'm interested to obtain the large field of view for 2 pointings, can someone help with this?

 
Profile Email
 Quote
fitz
 01/15/2018 03:36PM  
AAAAA
Admin

Status: offline


Registered: 09/30/2005
Posts: 4040
A single DECam file like that contains only a single exposure, the extensions are the individual CCDs that make up the mosaic. If you have multiple exposures that cover the same field and wish to combine them, the task to use is probably the MSCSTACK task from the NFEXTERN external package. In order to properly stack the images you may need to do some other processing first, see for example

https://iraf.net/forum/viewtopic.php?showtopic=138945

 
Profile Email
 Quote
   
Content generated in: 0.57 seconds
New Topic Post Reply

Normal Topic Normal Topic
Sticky Topic Sticky Topic
Locked Topic Locked Topic
New Post New Post
Sticky Topic W/ New Post Sticky Topic W/ New Post
Locked Topic W/ New Post Locked Topic W/ New Post
View Anonymous Posts 
Anonymous users can post 
Filtered HTML Allowed 
Censored Content 
dog allergies remedies cialis 20 mg chilblain remedies


Privacy Policy
Terms of Use

User Functions

Login