iraf.net

The original IRAF Development Roadmap was posted back on September 30, 2013. It's now mid-2015 and approaching the 2016 change of development priority. Is it possible to get an update on the roadmap? For instance, which features were chosen for developement and which were not? What's their status. etc.

Curious,
Jason

Hi Jason,

A simple question with a complex answer. So, first about the FY16 changes coming to NOAO: These are still happening and have led to a significant staff reduction due to a smaller base budget, however there are a number of other projects coming to Kitt Peak as well as a greater emphasis on data management within NOAO. One such effort is called the Data Lab (datalab.noao.edu) which can optionally use IRAF for some functionality but I mention it because it is the thing keeping me from spending any significant time on IRAF specifically.

So, for the Roadmap: A poll has been up on the homepage for a while and the top answers are a python interface and spectroscopy tools. The latter we did actually work on in the form of the SPTABLE package to operate on spectra stored as tables (we'd appreciate some more feedback on it!) and may do more still as part of supporting NOAO spectrographs (discussions in progress at the moment). The python interface, I believe, would have the greatest impact on users and the greatest long-term benefit, however it is a significant (in terms of other things I need to do) chunk of time to get done. I hope, but cannot promise, that some progress might be made as part of developing things like the Data Lab, especially since even pyraf is only minimally supported now (see their FAQ) and efforts like astropy have a ways to go to replace core IRAF functionality.

So, for the future: We've been tasked with preparing an internal white paper by the end of the year that lays out a roadmap for a "future of data processing" software strategy that will be more like an implementation plan than the current roadmap document. This will have to include an inventory of critical IRAF systems within NOAO (e.g. pipelines) that can't easily be replaced, as well as a realistic plan for the long-term maintenance of IRAF and options for transitioning to newer platforms for new development projects. What is less clear is the criticality of IRAF in the broader community and how/when they might migrate to something else (note that expressing your opinions to various user's committees and funding agencies could go a long way in this regard).

I'm happy to run this site for as long as it serves a purpose, but my time is getting squeezed more and more every day. We're overdue for another patch release (last was Oct 2013) and mundane projects like creating a GitHub repo fall to the side half-completed because of other priorities. That said, the situation should be much clearer by the end of this calendar year with the paper I mentioned, we're considering a poll of some sort to help inform the content but people should feel free to reply here if they have specific comments to make as well.

Hope this answers your question.

Cheers,
-Mike

Thanks for the reply, Mike. I saw that users were most voting for a python interface and wondered about the feasibility of that and whether it's a bit feature-heavy for software nearing some sort of end-of-life. My thinking, still hoping that the community will more actively take up the reins, is that the last non-maintenance development should be on polishing the CL itself as it exists today. It's by far the hardest part of the code to understand and change successfully. Future contributors who invest the time to learn and improve the CL will be rare. In particular I'd like to see the grammar fixed, more code comments, and some high-level documentation about the structure of the core system. I know there are some documents out there about this but they seem out-dated and inadequate.

If the CL itself is the kernel, then the various packages are "user space". By comparison, developing those programs is easy. So while it's true that the spectroscopy tools currently leave something to be desired, I don't know if it makes sense to add new features at all this late in the game. The observatories and astronomers should be doing that.

The last things to do in my opinion should be on turning over development to an open model. For instance, creating a GitHub repo is a great idea and trying to get IRAF installable via the various package managers, with deb files being the most important, would also make IRAF more accessible. Plus, it would be good too to have a general reflection essay on IRAF itself at some point that discusses its design successes and mistakes and what a successor to IRAF might look like if it were designed from scratch today.

Just my 2 cents,

Thanks for your update,

Cheers,
Jason

The official 30th anniversary of IRAF is coming up next year (marked from the first public release) and there are discussions within NOAO about whether/how to mark the occasion, an essay such as you describe seems fitting. I think a big part of the success and longevity is summed up in every paper that says "data were reduced using standard IRAF tools", something the community loses with everyone writing their own, slightly different, version of CCDPROC based on versions of NumPy/AstroPy/whatever.

In writing the original Roadmap document the thinking went roughly as follows: The majority of tasks written by users (for personal use as well as in external packages) are CL scripts, even within NOAO our pipelines and packages like MSCRED are largely scripts. As you point out, there are simple grammar changes that could be made and convenience features like image operators and subroutines in scripts would do a lotto make the language more useful. Similarly, the community is clearly moving to python for scripting (just as they were clearly moving to C++, then Java, then Tcl for scripting before that, but python seems to have legs). The python interface (essentially making IRAF a callable service) was meant to preserve the science tasks in IRAF (re-implementation leads to new bugs/behaviors and science results) but separate them from the scripting environment and let the user's handle things like GUIs as they pleased. It would also work in conjuction with CL features like image operators to allow in-memory NumPy arrays to be passed to tasks and preserve other python idioms like passing in list objects instead of @-files. So this was the grand plan to preserve the "using standard IRAF tools" statements and allow development options in something other than the core CL/SPP languages that are no longer cool. That does also put the onus on astronomer/observatories to develop new tools, but what is the timescale for that to happen?

I agree some high-level documentation is needed, but that takes time as well. Plenty of tutorials exist on the web (some incorrect) and should be gathered as links on this site for one-stop shopping, that been on my list for a while. Setting up the complete GitHub repo is just a grind but I'm not very sanguine about it suddenly generating a lot of contributions. I can literally fit a download/install/start command for IRAF in a Tweet and can barely keep up with releases as is, so package managers (deb, rpm and dmg) I'm happy to leave to others who's focus is a single distribution.

At the moment the white paper I mentioned is seen as an internal document, but we'll try to make it a public document that incorporates comments such as yours.

Cheers,
-Mike