A new set of tools for easily accessing Virtual Observatory data from the command-line (and thus scripting environments) is now available for testing. These tasks permit searching for VO resources and the query/access of image and catalog data from more that 15,000 sources. A key design point is the use of astronomer-friendly terms in specifying resources, general search capability (e.g. all X-ray image services), and a variety of output formats (default is CSV for easy manipulation by other tools).
The VO-CLI Project Home Page contains additional information, runnable examples, download instructions, as well as a guided tour of the software. These tools were developed as part of a larger VOClient project designed to provide a multi-language interface to VO data and services. They will be used in the upcoming IRAF NVO package but as host tools can be used from any environment. Examples of what you can do include: keyword searches to find relevant VO resources, process queries for lists of positions, easily find all HST data for a given object, or simply resolve and object name to its coordinates.
Since this is a testing release, feedback in the forums or comments is especially appreciated. Detailed examples of how to use the main data query/access task are shown in the main body of the story.
The following examples come from the VODATA documentation showing the many things that can be done with the task. Similar examples are available for the other tasks.
1) Query the GSC 2.2 catalog for stars a) within the 0.1 degree
default search size around NGC 1234: b) around all positions con-
tained in file 'pos.txt': c) for the list of objects given on the
command line: d) query a list of services for a list of positions:
e) print a count of results that would be returned from 3 services
for each position in a file:
% vodata gsc2.2 ngc1234 (a)
% vodata gsc2.2 pos.txt (b)
% vodata gsc2.2 m31,m51,m93 (c)
% vodata svcs.txt pos.txt (d)
% vodata hst,chandra,gsc2.2 pos.txt (e)
2) Query all (142) image services having data of the subdwarf galaxy
IC 10, print a count of the results only:
% vodata -c -t image any IC10
Note that we use the reserved word 'any' for the service name and
constrain by image type. The task will automatically query the
registry to create the list of services to be queried.
3) Print a count of X-ray catalog data around Abell2712:
% vodata -c -t catalog -b x-ray any abell2712
In this case we constrain both the service type as well as the
spectral coverage published for the resource in the registry. We
use the reserved flag to print a count without saving results. The
object name is resolved to coordinates internally. (Note: this
example may take a while to run).
4) Print the column metadata returned by the RC3 catalog service:
% vodata -meta rc3
The output will print the result using the default VERBOSE level,
adding the -v will set the query parameter VERBOSE=2, adding -vv
will set VERBOSE=3 (to print all available columns). When access-
ing data the same -v flags will be required to retrieve columns at
that VERBOSE level.
5) Use the Registry to query for resources using the search terms
"cooling flow". Upon examining the output the user notices a
Vizier paper titled "Cooling Flows in 207 clusters of Galaxies"
that looks interesting. Use the vodata task to download all tables
associated with this paper, save tables in the default CSV format:
% voregistry cooling flow
% vodata -O white97 -all J/MNRAS/292/419/
All 7 tables will be written to the current directory to files hav-
ing a root name 'white97' (chosen based on the author and publica-
tion date).
6) Find a suitable XMM image service, get a (brief) count of the XMM
images available for 3c273, and if there aren't too many, download
the images and save the extracted access URLs:
% voregistry -rv -t image xmm
ShortName ResourceType Title
------------------------------------------------------....
XMM-Newton SIAP/ARCHIVE XMM-Newton Archive ....
% vodata -cq xmm-newton 3c273
xmm-newton 27 I XMM-Newton Archive ....
% vodata -get xmm-newton 3c273
.... will query and download 27 images.
7) Query for the images available from 2MASS at a given position,
extract the positions and service URLs to separate files:
% vodata -e -O 2mass -t image 2mass 12:34:56.7 -23:12:45.2
The query produces files with the root name '2mass', and exten-
sions of ".csv" (the main response), ".pos" (the extracted pos-
itions), and ".urls" (the access references). The user inspects
the files and notices that the references return both FITS and HTML
files, but she only wants the FITS image date and uses vodata to
download only those:
% grep fits 2mass_I_001_15998.urls > images.txt
% vodata images.txt
or
% grep fits 2mass_I_001_15998.urls | vodata -i -
In both cases we pass URLs to the task which bypasses the query and
directly access the images. However, in the first case we process
the entire list and are able to take advantage of the -md option to
increase the number of simultaneous downloads. In the second case,
the -i flag causes the task to interpret each line of the input
stream as a separate command and so the data are always downloaded
one at a time (which is the default download behavior anyway).
8) Use vodata as a test client for a locally-installed SIAP service:
% vodata -t image -s http://localhost/siap.pl 180.0 0.0
In this case we force the ServiceURL using the '-s' flag, but since
we can't do a Registry query to discover what type of service this
is, we must use the '-t' flag to indicate it is an image service.
9) Create a local table containing the Abell catalog. Begin with a
Registry query to find likely services using the voregistry task,
print a verbose description of each resource and page the results
with less:
% voregistry -vv -t catalog abell | less
The verbose results indicate a number of services with differing
requirements for what is included. We decide to use the service
from HEASARC since it contains southern hemisphere data and con-
straints we are interested in. Try an all-sky query to retrieve
the entire catalog, use the service identifier to be specific about
where we want to go:
% vodata -e ivo://nasa.heasarc/abell 0.0 0.0 180.0
We use the '-e' flag to extract the positions to a separate file
with a ".pos" extension so that we can use these in later queries.
However, the .pos file additionally contains the ID from the origi-
nal catalog in column 1. Strip this column so we're left with only
RA and DEC and query for Chandra observations at each position:
% cut -c6- *.pos | vodata ivo://nasa.heasarc/chanmaster -p -
Here we used the unix 'cut' utility to remove the first column and
pipe the resulting positions to the task, using the '-p -' option
to indicate positions should be read from stding, and the IVO iden-
tifier of the Chandra observation master log we also discovered
from the Registry.
10) Interactively query for a count of Chandra observations of Messier
objects:
% vodata -cq chandra -i -
m31
chandra 335 I Chandra X-Ray Observatory Data Archive
: : : : :
Note that by using the '-i' flag we execute each query as if we'd
put the object name on the command line. Using the '-o' flag would
instead read all of the objects from the stdin but then execute the
queries in parallel following an EOF to terminate the input.
11) Use the STILTS task 'tpipe' to select rows from a VOTable of QSOs
(made with an earlier query) where the redshift is > 0.2. Output
only the positions and pipe this to vodata to generate a new query
to see whether HST has observed any of these objects:
% stilts tpipe ifmt=votable qso_survey.vot
cmd='select "Z > 0.2"'
cmd='keepcols "RA DEC"' | vodata -p - hstpaec
Note that we use the '-p -' flag to tell the task to take it's list
of positions from the piped input. The positions are used to query
the HST Planned and Archived Exposure Catalog (hstpaec)
12) The user has a task called 'wcsinfo' that takes a list of images
and outputs a text table containing the plate center and size in
degrees. Use this task as part of a query for 2MASS point sources
contained in each image:
% wcsinfo *.fits | vodata 2mass-psc -i -
Here we specify the desired service (2mass-psc) on the commandline
as usual, and allow the remainder of the args (i.e. the position
and search size) to be read from the stdin. This allows for vari-
able search sizes but processes the positions serially. If the
sizes are all the same (say 25 arcmin), multiple queries can be
done simultaneously using just a position file, e.g.
% wcsinfo -pos_only *.fits > centers.txt
% vodata -rm 25 2mass-psc centers.txt
13) Query a large list of objects against a number of ASCA resources.
Because the ASCA catalogs are served by the same machine, use the
'-i' option and a command file to process only each resource
sequentially while still parallelizing the object lists:
% cat cmds.txt
ASCA survey.tbl
ASCA GIS survey.tbl
ASCA GPS survey.tbl
: :
% vodata -i cmds.txt
Note that we've needed to escape the space in the resource name in some
cases. To avoid this, use of the IVO identifier for each resource is
usually preferred.
14) Query the VO for GALEX data of M51. Because the ShortName GALEX is
not unique, we must either specify the IVO identifier of a specific
service to query, or if we're interested in results from all sup-
ported data services with galex in the name:
% vodata -all galex M51
The results come from the Cone and SIAP services both called GALEX,
as well as an additional SIAP service called 'GALEX_Atlas'. Note
that the service names are case insensitive in either case.
15) Process a list of hundreds of positions against the GSC2.2 catalog:
% vodata gsc2.2 positions.txt
16) Process a list of hundreds of positions against the GSC2.2 catalog,
assume that the input table has a 5 line header of text and three
columns (id,ra,dec). Note that the cols option requires the
optional id be specified last:
% vodata --cols=2,3,1 --hskip=5 gsc2.2 positions.txt