Welcome to iraf.net Wednesday, November 22 2017 @ 11:48 PM GMT
New VO Command-Line Tools -- Available for Testing
- Tuesday, November 13 2007 @ 09:23 PM GMT
- Contributed by: fitz
- Views: 2,109
A new set of tools for easily accessing Virtual Observatory data from the command-line (and thus scripting environments) is now available for testing. These tasks permit searching for VO resources and the query/access of image and catalog data from more that 15,000 sources. A key design point is the use of astronomer-friendly terms in specifying resources, general search capability (e.g. all X-ray image services), and a variety of output formats (default is CSV for easy manipulation by other tools).
The VO-CLI Project Home Page contains additional information, runnable examples, download instructions, as well as a guided tour of the software. These tools were developed as part of a larger VOClient project designed to provide a multi-language interface to VO data and services. They will be used in the upcoming IRAF NVO package but as host tools can be used from any environment. Examples of what you can do include: keyword searches to find relevant VO resources, process queries for lists of positions, easily find all HST data for a given object, or simply resolve and object name to its coordinates.
Since this is a testing release, feedback in the forums or comments is especially appreciated. Detailed examples of how to use the main data query/access task are shown in the main body of the story.
The following examples come from the VODATA documentation showing the many things that can be done with the task. Similar examples are available for the other tasks.
1) Query the GSC 2.2 catalog for stars a) within the 0.1 degree default search size around NGC 1234: b) around all positions con- tained in file 'pos.txt': c) for the list of objects given on the command line: d) query a list of services for a list of positions: e) print a count of results that would be returned from 3 services for each position in a file: % vodata gsc2.2 ngc1234 (a) % vodata gsc2.2 pos.txt (b) % vodata gsc2.2 m31,m51,m93 (c) % vodata svcs.txt pos.txt (d) % vodata hst,chandra,gsc2.2 pos.txt (e) 2) Query all (142) image services having data of the subdwarf galaxy IC 10, print a count of the results only: % vodata -c -t image any IC10 Note that we use the reserved word 'any' for the service name and constrain by image type. The task will automatically query the registry to create the list of services to be queried. 3) Print a count of X-ray catalog data around Abell2712: % vodata -c -t catalog -b x-ray any abell2712 In this case we constrain both the service type as well as the spectral coverage published for the resource in the registry. We use the reserved flag to print a count without saving results. The object name is resolved to coordinates internally. (Note: this example may take a while to run). 4) Print the column metadata returned by the RC3 catalog service: % vodata -meta rc3 The output will print the result using the default VERBOSE level, adding the -v will set the query parameter VERBOSE=2, adding -vv will set VERBOSE=3 (to print all available columns). When access- ing data the same -v flags will be required to retrieve columns at that VERBOSE level. 5) Use the Registry to query for resources using the search terms "cooling flow". Upon examining the output the user notices a Vizier paper titled "Cooling Flows in 207 clusters of Galaxies" that looks interesting. Use the vodata task to download all tables associated with this paper, save tables in the default CSV format: % voregistry cooling flow % vodata -O white97 -all J/MNRAS/292/419/ All 7 tables will be written to the current directory to files hav- ing a root name 'white97' (chosen based on the author and publica- tion date). 6) Find a suitable XMM image service, get a (brief) count of the XMM images available for 3c273, and if there aren't too many, download the images and save the extracted access URLs: % voregistry -rv -t image xmm ShortName ResourceType Title ------------------------------------------------------.... XMM-Newton SIAP/ARCHIVE XMM-Newton Archive .... % vodata -cq xmm-newton 3c273 xmm-newton 27 I XMM-Newton Archive .... % vodata -get xmm-newton 3c273 .... will query and download 27 images. 7) Query for the images available from 2MASS at a given position, extract the positions and service URLs to separate files: % vodata -e -O 2mass -t image 2mass 12:34:56.7 -23:12:45.2 The query produces files with the root name '2mass', and exten- sions of ".csv" (the main response), ".pos" (the extracted pos- itions), and ".urls" (the access references). The user inspects the files and notices that the references return both FITS and HTML files, but she only wants the FITS image date and uses vodata to download only those: % grep fits 2mass_I_001_15998.urls > images.txt % vodata images.txt or % grep fits 2mass_I_001_15998.urls | vodata -i - In both cases we pass URLs to the task which bypasses the query and directly access the images. However, in the first case we process the entire list and are able to take advantage of the -md option to increase the number of simultaneous downloads. In the second case, the -i flag causes the task to interpret each line of the input stream as a separate command and so the data are always downloaded one at a time (which is the default download behavior anyway). 8) Use vodata as a test client for a locally-installed SIAP service: % vodata -t image -s http://localhost/siap.pl 180.0 0.0 In this case we force the ServiceURL using the '-s' flag, but since we can't do a Registry query to discover what type of service this is, we must use the '-t' flag to indicate it is an image service. 9) Create a local table containing the Abell catalog. Begin with a Registry query to find likely services using the voregistry task, print a verbose description of each resource and page the results with less: % voregistry -vv -t catalog abell | less The verbose results indicate a number of services with differing requirements for what is included. We decide to use the service from HEASARC since it contains southern hemisphere data and con- straints we are interested in. Try an all-sky query to retrieve the entire catalog, use the service identifier to be specific about where we want to go: % vodata -e ivo://nasa.heasarc/abell 0.0 0.0 180.0 We use the '-e' flag to extract the positions to a separate file with a ".pos" extension so that we can use these in later queries. However, the .pos file additionally contains the ID from the origi- nal catalog in column 1. Strip this column so we're left with only RA and DEC and query for Chandra observations at each position: % cut -c6- *.pos | vodata ivo://nasa.heasarc/chanmaster -p - Here we used the unix 'cut' utility to remove the first column and pipe the resulting positions to the task, using the '-p -' option to indicate positions should be read from stding, and the IVO iden- tifier of the Chandra observation master log we also discovered from the Registry. 10) Interactively query for a count of Chandra observations of Messier objects: % vodata -cq chandra -i - m31 chandra 335 I Chandra X-Ray Observatory Data Archive : : : : : Note that by using the '-i' flag we execute each query as if we'd put the object name on the command line. Using the '-o' flag would instead read all of the objects from the stdin but then execute the queries in parallel following an EOF to terminate the input. 11) Use the STILTS task 'tpipe' to select rows from a VOTable of QSOs (made with an earlier query) where the redshift is > 0.2. Output only the positions and pipe this to vodata to generate a new query to see whether HST has observed any of these objects: % stilts tpipe ifmt=votable qso_survey.vot cmd='select "Z > 0.2"' cmd='keepcols "RA DEC"' | vodata -p - hstpaec Note that we use the '-p -' flag to tell the task to take it's list of positions from the piped input. The positions are used to query the HST Planned and Archived Exposure Catalog (hstpaec) 12) The user has a task called 'wcsinfo' that takes a list of images and outputs a text table containing the plate center and size in degrees. Use this task as part of a query for 2MASS point sources contained in each image: % wcsinfo *.fits | vodata 2mass-psc -i - Here we specify the desired service (2mass-psc) on the commandline as usual, and allow the remainder of the args (i.e. the position and search size) to be read from the stdin. This allows for vari- able search sizes but processes the positions serially. If the sizes are all the same (say 25 arcmin), multiple queries can be done simultaneously using just a position file, e.g. % wcsinfo -pos_only *.fits > centers.txt % vodata -rm 25 2mass-psc centers.txt 13) Query a large list of objects against a number of ASCA resources. Because the ASCA catalogs are served by the same machine, use the '-i' option and a command file to process only each resource sequentially while still parallelizing the object lists: % cat cmds.txt ASCA survey.tbl ASCA GIS survey.tbl ASCA GPS survey.tbl : : % vodata -i cmds.txt Note that we've needed to escape the space in the resource name in some cases. To avoid this, use of the IVO identifier for each resource is usually preferred. 14) Query the VO for GALEX data of M51. Because the ShortName GALEX is not unique, we must either specify the IVO identifier of a specific service to query, or if we're interested in results from all sup- ported data services with galex in the name: % vodata -all galex M51 The results come from the Cone and SIAP services both called GALEX, as well as an additional SIAP service called 'GALEX_Atlas'. Note that the service names are case insensitive in either case. 15) Process a list of hundreds of positions against the GSC2.2 catalog: % vodata gsc2.2 positions.txt 16) Process a list of hundreds of positions against the GSC2.2 catalog, assume that the input table has a 5 line header of text and three columns (id,ra,dec). Note that the cols option requires the optional id be specified last: % vodata --cols=2,3,1 --hskip=5 gsc2.2 positions.txt