A number of system enhancements are underway as part of integrating IRAF with the Virtual Observatory, in particular is the ability to seamlessly use data tables returned by VO data services as input to IRAF tasks. This has a number of implications I'll write about later, but one early pay-off of this work is an enhanced image template interface in the core system that is now in testing internally. If you aren't sure what an "image template" is, think of the "*.fits" or @files that let you pass lists of images to IRAF tasks for processing.
In the enhanced templates, things like MEF files (and soon, tables) can be expanded automatically using the @-file operator to let any task that takes an image list process the extensions automagically. Additionally, selection of images based on image header keywrods can also be done dynamically using expressions in the template strings. For example, consider the following templates:
|@file*||expand all files beginning w/ 'file'|
|@file//".fits"||append ".fits" to contents of 'file'|
|@mef.fits||expand all (image) extensions of an MEF file|
|@mef.fits[SCI]||select SCI extensions from MEF file|
|@mef.fits[SCI,2][noinherit]||select v2 SCI extns, add kernel param|
|@mef.fits[1-16x2]||select range of extensions from MEF file|
|@mef.fits[+1-8]||create a list of extensions for an MEF|
|*.fits[1:100,1:100]||append section to all FITS images|
|@@file[1:100,1:100]||append section to expanded MEFs in 'file'|
|*.fits[filter?='V']||select images w/ FILTER keyword containing 'V'|
|@*.fits[gain==3.0]||select image extns where GAIN keyword is 3.0|
|*.fits[filter?='V';gain==2.5]||select using multiple OR's expressions|
These templates could be used e.g. to run IMSTAT on all the extensions of a FITS file with a single command, as in
|cl> imstat @mef.fits||
cl> if (imext.nimages > 0)
>>> imstat @list.dat
cl> del list.dat
where before it would have been necessary to expand the extensions explicitly using a second task into a standard @file. Notice how the other templates provide even more refined selection of the extensions, either within a single MEF or across multiple images.
These changes are all fully backward compatible but provide new syntax to give users a powerful and compact way to dynamically build image lists for use by all tasks. These new features will be especially useful for script developers tired of managing temp files of image lists, or those who just need to quickly examine MEF files or a directory of images.
Leave a message or post a comment if you're interested in getting an early peek at these new features. [Note that a small source patch to your system and a relink will be required].
Enhanced Image List Template Package April 15, 2011 The enhanced image list package provides new capabilities for handling image lists, but remains backwards compatible with tasks currently using the IMT interface. The enhancements allow for expansion of MEF files into lists of extensions using the @-file operator, as well as selection of images within more general lists by means of modifiers (e.g. a simple expression such as the extname/extver or explicit extension number, or more complex boolean expressions to allow selection by header keyword). In addition, tables may now take the @-file operator to use a column containing image references as an input list. ======================================================================== TODO: - Describe syntax for use with tables and selection by row values - Describe remote image specification caching mechanism ======================================================================== Template Strings ---------------- The FNT template package supports the following forms of pattern strings: alpha, *.x, data* // .pix, [a-m]*, @list_file, nite%1%2%.1024.imh i.e. simple filenames, wildcard expansion in filenames, concatenation of filenames, @files, substitution in filenames, or a comma-delimited list of the above. The image template package (IMT) extends these patterns to allow image names followed by a cluster index or image section in  brackets. These patterns remain unchanged in the new version of the package to allow backward compatability with existing applications. Lists of these types represent *explicit* collections of images, i.e. a collection based on the image name (wildcards) or as a result of processing by some task (e.g. expansion of an MEF file to create an input @-file of expanded extension specifications). The enhanced version of the IMT package further abstracts the concept of image collections to include data objects such as MEF files or tables containing a list of image references that *implicitly* defines the list (e.g. the expanded MEF extension specification or the complete column of image references). Further, we allow this list (which might be quite broad) to be refined using modifiers or selectors on the list and thus dynamically create the list without requiring the use (and management) of intermediate files. For example, @file* expand all files beginning w/ 'file' @file//".fits" append ".fits" to contents of 'file' @mef.fits expand all (image) extensions of an MEF file @mef.fits[SCI] select SCI extensions from MEF file @mef.fits[SCI,2][noinherit] select v2 SCI extns, add kernel param @mef.fits[1-16x2] select range of extensions from MEF file @mef.fits[+1-8] create a list of extensions for an MEF *.fits[1:100,1:100] append section to all FITS images @@file[1:100,1:100] append section to expanded MEFs in 'file' *.fits[filter?='V'] select images w/ FILTER keyword containing 'V' @*.fits[gainfile [extname] [expr;...] [ikparams] [section] [@@ | @] file [extname,extver][expr;...] [ikparams] [section] [@@ | @] file [index_range] [expr;...] [ikparams] [section] -------- selectors ------- ------ modifiers ----- The file specification may be the name of a file, and image, or a table. The behavior of the @-file and @@-file operators will depend on the type of file but the @-file usage remains backward compatible when used with text files. The use of a modifier/selection on an MEF file will automatically trigger expansion of the extensions in the image and so an '@' operator is not strictly required, however only those extensions matching the selection expression will be present in the final image list. Note that the use of index ranges and extname/extver selectors are mutually exclusive, selector expressions may be added to either. @-file Operations ----------------- The @-file operator is unchanged from previous versions when used with text files of image names. Modifier/selector expressions however can now be applied to the contents of the @-file to select from the list only those images that match the selector expression, or to augment the name in the list with an additional image syntax such as a section or kernel parameters. @@-file Operations ------------------ The @@-file operator is new syntax meant to allow the contents of an @-file to be expanded automaticaly, e.g. as if there were an @-file of @-file names. Primarily this can be used to create a list of MEF image names in which an @-file would return the names of the MEF, while the @@-file syntax could be used to expand each MEF into individual extension specifications. Extension Indices ----------------- The [index_range] modifier may be used to specify an explicit set of extensions to be used. Index ranges are specified as a comma-delimited list of strings specifying individual range segments as described in the RANGES help page. The use of a '+' operator before an index range indicates the range list should be expanded without checking that the extension exists in the MEF itself. Otherwise, only those extensions present in the MEF will be included in the list. Selection Expressions --------------------- Selection expressions may be used to restrict a template list to only those images that match some boolean expression, e.g. to provide for selection based on a header keyword value. Expressions follow the same guidelines as in the HSELECT task 'expr' parameter (see the help page for details). Multiple expressions may be specified if they are separated by a semicolon however they are evaluated as a single expression of OR'd values rather than as individual expressions. This is significant when considering that expressions may contain keywords not present in all images being checked, for instance *.fits[filter?='V';gain0.5&&gain foobar.fits foo.fits // bar ==> foobar.fits foo // @file1 ==> foosif1.fits,foomef1.fits @file1 // bar ==> sif1foo.fits,mef1foo.fits