iraf.net

Hi All,

I'm having some difficulties with using the sinc function to fit the cross-correlation function in Fxcor. I wrote a script in pyraf to process a bunch of spectra and produce radial velocities with Fxcor. I have no issues if I use the other fitting routines (mainly use the gaussian function), however, when using sinc, my script will process a few spectra and then get hung up while my cpu usage stays at 100%. This forces me to use crtl + c in the terminal to quit the process. When I run my script again, it will run fine for a while (including being able to process spectra it got hung up on) but then it will stop once again. I can't figure out why it's doing this. Also when fxcor gets hung up, zero byte files are created in the directory where I'm outputting files.

Here is fxcor command I'm running in pyraf:
iraf.rv()
iraf.rv.fxcor (object_fits, template_fits, apertures="*", cursor="",
continuum=continuum, filter=filter_spec, rebin=rebin, pixcorr="no", osample=osample_reg,
rsample=osample_reg, apodize=apodize, function=function, width=width, height=height, peak=peak,
minwidth=minwidth, maxwidth=maxwidth, weights=weights, background=background, window=300.,
wincenter="INDEF", output=output_name, verbose="long", imupdate="no",
graphics="stdgraph", interactive=interact, autowrite="yes", autodraw="yes",
ccftype="image", observatory="aao", continpars=continpars, filtpars=filtpars, keywpars=keywpars)

Any help would be appreciated. I'm using a Mac OS X 10.7.5, Pyraf 2.0, IRAF 2.16, and Python 2.7.1.

Cheers,
Brett

There's not really enough information to say what's going on, e.g. is it the x_rv.e processing using the CPU or pyraf, is the 'width' parameter appropriate for your CCF peak shape, if you ran the same object spectrum through multiple times would it also fail in a reproducible way, does using the CL instead of pyraf to process the list make a difference, what files exactly are zero length, etc.

Once the failure occurs you can use the unix 'top' command to see what process is using the CPU, if this is x_rv.e then I would attach a debugger to get a stack trace, e.g. "%gdb /iraf/iraf/noao/bin.macintel/x_rv.e " where is the process ID from the 'top' command, then type 'where' to get the stack trace and post that here. OTOH if it is python/pyraf that is pegged it may be related to the graphics in some way.

Likewise, if you play with the width parameter does it change the behavior? Is the value set as the total number of points to use around the peak, or as an extent on each side of the peak? Is the peak undersampled or severely blended in any way? Is there apparent ringing in the fits that do succeed?

If you're able to reproduce the problem reliably could you upload the data to the anonftp at ftp://iraf.noao.edu/pub and post the task parameters you used (e.g. "cl\$this->_split2($m[0]) dpar fxcor") so I can try to reproduce it here. Note also I'm assuming you're using the standard release from NOAO, if this is some other distribution please let me know.

Hi fitz,

Thanks for your response. It is the x_rv.e process that is using up the CPU (or one core of the cpu). The width parameter is appropriate for the CCF peak shape. I did a test to see if it would crash processing the same spectra (but with slightly different names) over and over in a script. After cross-correlating 2-3 spectra it would fall over even though the spectra it's processing is exactly the same. I don't have a CL script so I would have to write one to process the spectra and see if that makes a difference. The files that are zero length are the output graphics from the ccf .gki and the ccf and radial velocity information (.log and .txt files).

I did a sample of the x_rv.e process (is this equivalent to running a debugger to get a stack trace?). Please see below.

Changing the width parameters does sometimes change it's behavior but it still always freezes (sometimes it runs for longer). Oddly enough, restarting my computer does allow the script to run longer before freezing. The peak is not under or over sampled and is not blended. Also no ringing in the ccf fits.

I will upload the data as soon as I can.

Cheers,
Brett


/Users/brett/iraf
/usr/local/scisoft/packages/iraf/iraf/noao/bin.macintel/x_rv.e
/usr/lib/dyld
/private/var/db/dyld/dyld_shared_cache_x86_64
-\$this->_split2($m[0])0x178aa3ec
-\$this->_split2($m[0])0x178a97d0
/dev/ttys000
/Volumes/RAIDBARTOK/Cyclops/processing_Cyclops_data/130827/Reduced/RVoutput_HD206395_sinc/unfiltered/037/27aug20037_fib12_ord05.txt
/Volumes/RAIDBARTOK/Cyclops/processing_Cyclops_data/130827/Reduced/RVoutput_HD206395_sinc/unfiltered/037/27aug20037_fib12_ord05.log
/Volumes/RAIDBARTOK/Cyclops/processing_Cyclops_data/130827/Reduced/RVoutput_HD206395_sinc/unfiltered/037/27aug20037_fib12_ord05.gki
/dev/ttys000


Sampling process 33059 for 3 seconds with 1 millisecond of run time between samples
Sampling completed, processing symbols...
Analysis of sampling x_rv.e (pid 33059) every 1 millisecond
Process: x_rv.e [33059]
Path: /usr/local/scisoft/packages/iraf/iraf/noao/bin.macintel/x_rv.e
Load Address: 0x100000000
Identifier: x_rv.e
Version: ??? (???)
Code Type: X86-64 (Native)
Parent Process: Python [32928]

Date/Time: 2013-11-28 17:00:25.671 +1100
OS Version: Mac OS X 10.7.5 (11G63)
Report Version: 7

Call graph:
2450 Thread_828676 DispatchQueue_1: com.apple.main-thread (serial)
2450 start (in x_rv.e) + 52 [0x100001084]
2450 main (in x_rv.e) + 1434 [0x100105adc]
2450 irafmn_ (in x_rv.e) + 2593 [0x100242641]
2450 sysruk_ (in x_rv.e) + 941 [0x100001439]
2450 tfxcor_ (in x_rv.e) + 1195 [0x100085d28]
2450 rvcurr_ (in x_rv.e) + 8215 [0x100051ea3]
2450 rvbatr_ (in x_rv.e) + 2697 [0x10004d965]
2450 rvyfit_ (in x_rv.e) + 1561 [0x10005df85]
2450 rvfit_ (in x_rv.e) + 2793 [0x10005bc31]
2374 rvsinc_ (in x_rv.e) + 2346 [0x100074453]
+ 1317 sincip_ (in x_rv.e) + 950,962,... [0x100074b24,0x100074b30,...]
+ 862 sincip_ (in x_rv.e) + 1023 [0x100074b6d]
+ ! 749 zzepro_ (in x_rv.e) + 78 [0x10027d53f]
+ ! : 749 feclearexcept (in libmathCommon.A.dylib) + 43,17,... [0x7fff84a244eb,0x7fff84a244d1,...]
+ ! 75 zzepro_ (in x_rv.e) + 23 [0x10027d508]
+ ! : 75 fegetexceptflag (in libSystem.B.dylib) + 15,19,... [0x7fff87d9029f,0x7fff87d902a3,...]
+ ! 28 zzepro_ (in x_rv.e) + 0,27,... [0x10027d4f1,0x10027d50c,...]
+ ! 6 DYLD-STUB$$feclearexcept (in x_rv.e) + 0 [0x100422164]
+ ! 4 DYLD-STUB$$fegetexceptflag (in x_rv.e) + 0 [0x10042216a]
+ 153 sincip_ (in x_rv.e) + 606 [0x1000749cc]
+ ! 153 sin (in libSystem.B.dylib) + 570,594,... [0x7fff87d959fa,0x7fff87d95a12,...]
+ 40 sincip_ (in x_rv.e) + 34 [0x100074790]
+ ! 30 i_nint (in x_rv.e) + 75,0,... [0x10042201d,0x100421fd2,...]
+ ! 7 i_nint (in x_rv.e) + 62 [0x100422010]
+ ! : 7 floor (in libSystem.B.dylib) + 0,52,... [0x7fff87d9b990,0x7fff87d9b9c4,...]
+ ! 3 DYLD-STUB$$floor (in x_rv.e) + 0 [0x10042219a]
+ 2 DYLD-STUB$$sin (in x_rv.e) + 0 [0x1004224b8]
76 rvsinc_ (in x_rv.e) + 2299,2237,... [0x100074424,0x1000743e6,...]

Total number in stack (recursive counted multiple, when \$this->_split2($m[0])=5):

Sort by top of stack, same collapsed (when \$this->_split2($m[0])= 5):
sincip_ (in x_rv.e) 1317
feclearexcept (in libmathCommon.A.dylib) 749
sin (in libSystem.B.dylib) 153
rvsinc_ (in x_rv.e) 76
fegetexceptflag (in libSystem.B.dylib) 75
i_nint (in x_rv.e) 30
zzepro_ (in x_rv.e) 28
floor (in libSystem.B.dylib) 7
DYLD-STUB$$feclearexcept (in x_rv.e) 6

Binary Images:
0x100000000 - 0x1004acfe7 +x_rv.e (??? - ???) /usr/local/scisoft/packages/iraf/iraf/noao/bin.macintel/x_rv.e
0x7fff67f76000 - 0x7fff67faabaf dyld (195.6 - ???) /usr/lib/dyld
0x7fff84705000 - 0x7fff8470afff libcompiler_rt.dylib (6.0.0 - compatibility 1.0.0) /usr/lib/system/libcompiler_rt.dylib
0x7fff84801000 - 0x7fff84806ff7 libsystem_network.dylib (??? - ???) /usr/lib/system/libsystem_network.dylib
0x7fff84a23000 - 0x7fff84a27fff libmathCommon.A.dylib (2026.0.0 - compatibility 1.0.0) /usr/lib/system/libmathCommon.A.dylib
0x7fff85bb9000 - 0x7fff85bfbff7 libcommonCrypto.dylib (55010.0.0 - compatibility 1.0.0) /usr/lib/system/libcommonCrypto.dylib
0x7fff869fb000 - 0x7fff869fcff7 libremovefile.dylib (21.1.0 - compatibility 1.0.0) /usr/lib/system/libremovefile.dylib
0x7fff871a0000 - 0x7fff871a2fff libquarantine.dylib (36.7.0 - compatibility 1.0.0) /usr/lib/system/libquarantine.dylib
0x7fff87d88000 - 0x7fff87db5fe7 libSystem.B.dylib (159.1.0 - compatibility 1.0.0) /usr/lib/libSystem.B.dylib
0x7fff88fb3000 - 0x7fff89090fef libsystem_c.dylib (763.13.0 - compatibility 1.0.0) /usr/lib/system/libsystem_c.dylib
0x7fff8a26d000 - 0x7fff8a28afff libxpc.dylib (77.19.0 - compatibility 1.0.0) /usr/lib/system/libxpc.dylib
0x7fff8a7d0000 - 0x7fff8a7d7fff libcopyfile.dylib (85.1.0 - compatibility 1.0.0) /usr/lib/system/libcopyfile.dylib
0x7fff8aee5000 - 0x7fff8aeedfff libsystem_dnssd.dylib (??? - ???) /usr/lib/system/libsystem_dnssd.dylib
0x7fff8afbd000 - 0x7fff8afddfff libsystem_kernel.dylib (1699.32.7 - compatibility 1.0.0) /usr/lib/system/libsystem_kernel.dylib
0x7fff8b8aa000 - 0x7fff8b8abfff libunc.dylib (24.0.0 - compatibility 1.0.0) /usr/lib/system/libunc.dylib
0x7fff8bf24000 - 0x7fff8bf25fff libdnsinfo.dylib (395.11.0 - compatibility 1.0.0) /usr/lib/system/libdnsinfo.dylib
0x7fff8c0db000 - 0x7fff8c0dcff7 libsystem_sandbox.dylib (??? - ???) /usr/lib/system/libsystem_sandbox.dylib
0x7fff8caed000 - 0x7fff8caf7ff7 liblaunch.dylib (392.39.0 - compatibility 1.0.0) /usr/lib/system/liblaunch.dylib
0x7fff8cb7a000 - 0x7fff8cbb5fff libsystem_info.dylib (??? - ???) /usr/lib/system/libsystem_info.dylib
0x7fff8cd9c000 - 0x7fff8cda0fff libdyld.dylib (195.5.0 - compatibility 1.0.0) /usr/lib/system/libdyld.dylib
0x7fff8e7e0000 - 0x7fff8e7e9ff7 libsystem_notify.dylib (80.1.0 - compatibility 1.0.0) /usr/lib/system/libsystem_notify.dylib
0x7fff8f012000 - 0x7fff8f018ff7 libunwind.dylib (30.0.0 - compatibility 1.0.0) /usr/lib/system/libunwind.dylib
0x7fff8f3f1000 - 0x7fff8f3f7fff libmacho.dylib (800.0.0 - compatibility 1.0.0) /usr/lib/system/libmacho.dylib
0x7fff90225000 - 0x7fff90233fff libdispatch.dylib (187.10.0 - compatibility 1.0.0) /usr/lib/system/libdispatch.dylib
0x7fff905a9000 - 0x7fff905aaff7 libsystem_blocks.dylib (53.0.0 - compatibility 1.0.0) /usr/lib/system/libsystem_blocks.dylib
0x7fff907bc000 - 0x7fff907bcfff libkeymgr.dylib (23.0.0 - compatibility 1.0.0) /usr/lib/system/libkeymgr.dylib
0x7fff90dfe000 - 0x7fff90e03fff libcache.dylib (47.0.0 - compatibility 1.0.0) /usr/lib/system/libcache.dylib
Sample analysis of process 33059 written to file /dev/stdout

Did you have a chance to upload the images and parameters? The function identified by your stack trace doesn't do much interesting stuff, it basically loops over some number of shifts. However, the report that the CPU begins to run away would indicate a memory problem that is overwriting the element storing that number of shifts. So far I haven't been able to reproduce the problem but I don't know whether that means it is data-related or because you're using a SciSoft version and I'm using the latest v2.16.1 where it might already have been fixed (unintentionally).

Hi Fitz,

I have been really busy the last couple of weeks, sorry about the delay. I just uploaded the images and parameters through the upload a file user function (I tried ftp in but it's saying my password is incorrect). It does sound like it might be a memory issue. Is there a way to update the IRAF version provided through SciSoft? Let me know if you have any questions regarding the spectra and parameters I uploaded. I'm curious to see if the sinc functions works without any difficulties in iraf 2.16.1.

Thanks

I don't see the files on ftp://iraf.noao.edu/pub. To upload all you should need to do is

% ftp iraf.noao.edu
..... login as 'anonymous', give your email as the passwd
ftp\$this->_split2($m[0]) cd pub
ftp\$this->_split2($m[0]) binary
ftp\$this->_split2($m[0]) mput obj.fits temp.fits params

If you install v2.16.1 it should supercede other definitions of $iraf by editing your .cshrc/.bashrc file to do the iraf setup as the last step. In some cases you could also just unpack the v2.16.1 tarball in the SciSoft iraf root directory, however this may not work for older scisoft versions.

I just uploaded the file onto the ftp server. Thanks.

Hi fitz,

I haven't heard back from you since my last post. I have decided to get rid of Scisoft pyraf/iraf and install python, iraf, and pyraf separately through the official channels. I am now running the latest iraf version 2.16.1, pyraf version 2.1.6, and Python v2.7.6. Unfortunately, in pyraf, the bug with fxcor sinc function freezing up still occurs. I wrote a cl script to process the spectra in batch mode but I can't seem to get it to run. The error I get is "ERROR: Attempt to access undefined local variable `object_listL'." despite all local and global variables being defined. The script does work in pyraf using the task command "task $cross_correlate=/Users/brett/iraf/cross_correlate_new.cl" but again freezes after cross-correlating 5-10 spectra.

To assist you in tracking down this bug, I have uploaded both the object and template spectra, my cl script (which strangely only runs in pyraf even though there is only iraf cl code, i.e. no python code), and a parameter file that you can copy and paste the input parameters at the given prompts from my script. For the script to run properly, the fxcor continuum, filter, and keyword parameters must be set (please see png files for the appropriate parameters).

When the cross_correlate_new prompts for,
name of object inlist to be processed: inlist_130827_035.txt (don't include directory info)

name of template inlist to be processed: inlist_130827_073.txt (don't include directory info)

Directory containing object spectra to be processed: /Volumes/RAIDBARTOK/Cyclops/processing_Cyclops_data/130827/Reduced_best_10cut/reducedfits/035/ (change to where the spectra is kept in observation folder 035)

Directory containing template spectra to be processed: /Volumes/RAIDBARTOK/Cyclops/processing_Cyclops_data/130827/Reduced_best_10cut/reducedfits/073/ (change to where the synthetic spectra is kept in observation folder 073)

Directory where output velocities will be placed: /Volumes/RAIDBARTOK/Cyclops/processing_Cyclops_data/130827/Reduced_best_10cut/RVoutput_synspec_sinctest_nofilt_apod5/unfiltered/ (create a new folder in the spectra folder with output and change the text to this location)

Cyclops version (1, 2): 2

Cyclops type (n, o): n

CCF fit function (parabola, gaussian, lorentzian, center1d, sinc): sinc (all fit function work except for sinc. Need to figure out why sinc doesn't work)


I have uploaded the following files to the public server: 035.zip, 073.zip, continuum_parameters.png, cross_correlate_new_par.txt, cross_correlate_new.cl, filtering_parameters.png, and keyword_parameters.png.

If you could please get back to me asap about fixing this bug I would appreciate it. I tried looking into the fxcor code and fitting code fxcor calls but I see nothing obvious I can fix. I reckon there is some sort of infinite loop going on. As stated earlier I haven't been able to test in the standard iraf due to none of my cl scripts working in iraf (all of them work in pyraf) so I don't know if this is a problem with fxcor or a problem with pyraf.

Cheers,
Brett

Sorry for the slow reply and thanks for the script and extra data. I'm able to reproduce the problem and am looking into it, but there's nothing obvious at the moment. I'll followup with a workaround or a new binary as soon as I've tracked it down.

Thank you for looking into the problem.

Cheers,
Brett

Hi Brett,

I'm still looking at the algorithm but I believe the problem boils down to a simple uninitialized array used for the interpolation. I've put a patched binary for you at

ftp://iraf.noao.edu/pub/fitz/x_rv.e.MACINTEL

Just download in binary mode, reset the execute permissions and install as $iraf/noao/bin.macintel/x_rv.e

I do note that there is no FWHM value calculated because the 'maxwidth' parameter doesn't generally extend far enough around the peak (as weak as it is) to reach the half-power point, you might consider increasing this value. If you're just interested in getting the center position then the 'center1d' function is also very accurate for this purpose. Anyway, hope this fixes it, let me know if the problems continue.

Cheers,
-Mike

Hi Mike,

Thanks for continuing to work on this problem. Unfortunately my mac osx environment is 32bit (even though I can utilize over 3gb of ram) thus I'm using iraf macosx and not macintel (even though I have an intel processor Darwin Kernel Version 11.4.2: Thu Aug 23 16:26:45 PDT 2012; rootnu-1699.32.7~1/RELEASE_I386 i386) so I can't utilize this file. I downloaded the file anyways and placed it into my iraf/iraf/noao/bin.macosx folder. I then changed the permission (-rwxr-xr-x 1 root _lpoperator 7881840 19 Feb 14:05 x_rv.e) and subsequently loaded up pyraf. When I ran my script, I got the following system error message:

Process: Python [71784]
Path: /opt/local/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
Identifier: Python
Version: ??? (???)
Code Type: X86-64 (Native)
Parent Process: Python [71688]

Date/Time: 2014-02-19 14:20:41.831 +1100
OS Version: Mac OS X 10.7.5 (11G63)
Report Version: 9

Interval Since Last Report: 847885 sec
Crashes Since Last Report: 5
Per-App Crashes Since Last Report: 5
Anonymous UUID: 358305BA-EC74-42DB-81D8-CD17A60A33A7

Crashed Thread: Unknown

Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_PROTECTION_FAILURE at 0x00007fff5fc01028

Backtrace not available

Unknown thread crashed with X86 Thread State (64-bit):
rax: 0x0000000000000055 rbx: 0x0000000000000000 rcx: 0x0000000000000000 rdx: 0x0000000000000000
rdi: 0x0000000000000000 rsi: 0x0000000000000000 rbp: 0x0000000000000000 rsp: 0x0000000000000000
r8: 0x0000000000000000 r9: 0x0000000000000000 r10: 0x0000000000000000 r11: 0x0000000000000000
r12: 0x0000000000000000 r13: 0x0000000000000000 r14: 0x0000000000000000 r15: 0x0000000000000000
rip: 0x00007fff5fc01028 rfl: 0x0000000000010203 cr2: 0x00007fff5fc01028
Logical CPU: 2

Binary images description not available


External Modification Summary:
Calls made by other processes targeting this process:
task_for_pid: 0
thread_create: 0
thread_set_state: 0
Calls made by this process:
task_for_pid: 0
thread_create: 0
thread_set_state: 0
Calls made by all processes on this machine:
task_for_pid: 232820
thread_create: 2
thread_set_state: 0

Model: MacPro1,1, BootROM MP11.005C.B08, 4 processors, Dual-Core Intel Xeon, 2.66 GHz, 16 GB, SMC 1.7f10
Graphics: NVIDIA GeForce 7300 GT, NVIDIA GeForce 7300 GT, PCIe, 256 MB
Memory Module: DIMM Riser A/DIMM 1, 2 GB, DDR2 FB-DIMM, 667 MHz, 0x8551, 0x373254323536393230484641335342202020
Memory Module: DIMM Riser A/DIMM 2, 2 GB, DDR2 FB-DIMM, 667 MHz, 0x8551, 0x373254323536393230484641335342202020
Memory Module: DIMM Riser B/DIMM 1, 2 GB, DDR2 FB-DIMM, 667 MHz, 0x8551, 0x373254323536393230484641335342202020
Memory Module: DIMM Riser B/DIMM 2, 2 GB, DDR2 FB-DIMM, 667 MHz, 0x8551, 0x373254323536343230484641335342202020
Memory Module: DIMM Riser A/DIMM 3, 2 GB, DDR2 FB-DIMM, 667 MHz, 0x8551, 0x373254323536393230484641335342202020
Memory Module: DIMM Riser A/DIMM 4, 2 GB, DDR2 FB-DIMM, 667 MHz, 0x8551, 0x373254323536393230484641335342202020
Memory Module: DIMM Riser B/DIMM 3, 2 GB, DDR2 FB-DIMM, 667 MHz, 0x8551, 0x373254323536343230484641335342202020
Memory Module: DIMM Riser B/DIMM 4, 2 GB, DDR2 FB-DIMM, 667 MHz, 0x8551, 0x373254323536393230484641335342202020
Bluetooth: Version 4.0.8f17, 2 service, 11 devices, 1 incoming serial ports
Network Service: Ethernet 1, Ethernet, en0
PCI Card: NVIDIA GeForce 7300 GT, sppci_displaycontroller, Slot-1
Serial ATA Device: M4-CT256M4SSD1, 256.06 GB
Serial ATA Device: ST32000641AS, 2 TB
Serial ATA Device: ST32000641AS, 2 TB
Serial ATA Device: ST2000DM001-9YN164, 2 TB
Parallel ATA Device: OPTIARC DVD RW AD-7170A
USB Device: Microsoft® LifeCam HD-6000 for Notebooks, 0x045e (Microsoft Corporation), 0x076f, 0xfd500000 / 5
USB Device: Keyboard Hub, apple_vendor_id, 0x1006, 0xfd400000 / 4
USB Device: USB Receiver, 0x046d (Logitech Inc.), 0xc52b, 0xfd410000 / 8
USB Device: Apple Keyboard, apple_vendor_id, 0x0220, 0xfd420000 / 7
USB Device: LaCie RuggedXL, 0x059f (LaCie), 0x1024, 0xfd300000 / 3
USB Device: hub_device, 0x058f (Alcor Micro, Corp.), 0x6254, 0xfd200000 / 2
USB Device: Mass Storage Device, 0x058f (Alcor Micro, Corp.), 0x6364, 0xfd220000 / 6
USB Device: Bluetooth USB Host Controller, apple_vendor_id, 0x8206, 0x5d200000 / 2
FireWire Device: built-in_hub, 800mbit_speed

I've put a 32-bit version of the patched binary at

ftp://iraf.noao.edu/pub/fitz/x_rv.e.MACOSX

Let me know if that still fails to run.

It's now running, however, it does stop after processing around 20 files (like it was before though it does seem to be running longer than it was before). It seems to be freezing once it reaches fib11_ord06 for some reason (I tried different data sets and it always stops at fib11_ord06 or fib12_ord06 if I tell it to skip this fider). As far as no FWHM, it seems to be working for me (see below). Strange that the array used for the interpolation is not being uninitialized sometimes.

Cheers,
Brett

Description of Fit to CCF Peak and Cross-Correlation
NOAO/IRAF V2.16.1 brett@--------------------------------- Wed 15:23:33 19-Feb-2014

Obj = `/Volumes/RAIDBARTOK/Cyclops/processing_Cyclops_data/130820/Reduced_best_10cut/reducedfits/018/20aug20018_fib16_ord05.fits[1]'star = `HD157347'
Temp = `/Volumes/RAIDBARTOK/Cyclops/processing_Cyclops_data/130827/Reduced_best_10cut/reducedfits/073/27aug20073_fib16_ord05.fits[1]'star = `synth_Teff6500_logg4'
Deltav = 1.753 Km/sec Tempvel = 0.000 Km/sec

Fit Parameters:
Function = `sinc' Width = INDEF
Height = 0. Minwidth = 3.
Peak = no Maxwidth = 15.
Weights = YES Background = 0.
Wincenter = INDEF Window = 300

Number of points fit = 15

Mean Residual = 0.0000000
Sigma of Residuals = 0.0000000
Maximum of cross-correlation is in bin = -6.
Variance of cross-correlation = 0.007911804
HJD of observation = 2456525.83679 MJD = 56525.33478
Object sample used in correlation = `A6351.00-6450.00'
Template sample used in correlation = `A6351.00-6450.00'
Tonry&Davis R value = 75.73434

Velocity Results:
Shift of peak = -5.7905 pixels
Correlation height = 0.847
FWHM of peak = 11.48171 Km/sec (=6.550013 pixels)

Velocity computed from shift = -10.1502 Km/sec
Observed velocity = -27.8696 Km/sec
Heliocentric velocity = -53.4290 +/- 0.118 Km/sec

Comments:

Hi Mike,
Have you made any progress on this problem? Sorry to continue troubling you with it.
Cheers,
Brett

Sorry, never saw the followup posting but will look again. I do remember however that (using the macintel binary at least) I was able to process your entire test data set with no hangups, I'll try again with the macosx binary just in case something didn't get updated during the recompile.

Just to follow up .... I was indeed able to reproduce the stall using the macosx binary, but not exactly where you did. The macintel binary does run to completion, I'm looking at it now...

I'm not sure what might have happened with the last binary build, but please get the macosx binary again. I was able to use it to run your dataset to completion with no stalls.

Hi Mike,

Sorry for the delay. Just finished another observing run. The new rv.e binary does indeed run without freezing, however, it doesn't seem to be computing a shift, CCF height, a proper VHelio, CCF width, or Tonry & Davis R number. Not sure why this is happening. I upload one of the output velocities (20aug20018_fib10_ord14.log) and eps plot (20aug20018_fib10_ord14.eps). The other function ccf fits work fine with the same parameters.

In addition to Fxcor problems, I have noticed (for quite some time) a bug in sgikern. sgikern converts useless .gki files to .eps files that can actually be viewed outside of IRAF. For some reason IRAF does not wait for sgikern to finish writing out .eps before letting IRAF (or Pyraf) proceed with executing more commands. This bug sometimes causes zero or incomplete files to be written (I have even seen some incomplete eps files that could still be opened with only half the plot visible). I have found a workaround for this problem (see code below) by first checking that a .eps file exist on the directory after running sgikern and then checking that the file is at least 140KB in size. If the file hasn't been written yet, I tell pyraf to sleep for a split second in a do loop until the file appears. If the file size is smaller than 140KB, I tell pyraf to go to sleep and loop until either the file is larger than 140KB or 50 loop iterations have been completed. Some of this would be avoided if sgikern would let you specify the path and filename instead of creating an eps file with a random name (sgi######.eps) in the iraf directory which I then have to search for and move to the directory I want and rename it to something that's useful. The problem isn't as severe when writing to solid state drives (probably due to the drive writing the file quick enough that subsequent commands don't interrupt the write process). It would be good to fix this bug.

The other fxcor limitation that frustrates me is that if I want to output the cross-correlation function as well as the txt, log, and gki files from fxcor, I have to run fxcor twice with the second iteration in interactive mode. This causes my script to run very slow with an annoying plot screen opening and closing (uses heaps of cpu and memory, 10gb of ram or more when cross-correlating 1000's of spectra). There should be an option added to fxcor to allow for the ccf to be outputted with all the other files without freaking rerunning fxcor!

Sorry for my IRAF rant.

Cheers,
Brett




iraf.plot.sgikern (gki_output_name, device='eps', generic='no', debug='no', verbose='no', gkiunit='no')

while len(glob.glob(iraf_dir + 'sgi' + '*.eps'))

Part of my post got cut off for some reason. Here's the rest of it.

iraf.plot.sgikern (gki_output_name, device='eps', generic='no', debug='no', verbose='no', gkiunit='no')

while len(glob.glob(iraf_dir + 'sgi' + '*.eps'))

#iraf.plot.sgikern (gki_output_name, device='eps', generic='no', debug='no', verbose='no', gkiunit='no')

#while len(glob.glob(iraf_dir + 'sgi' + '*.eps'))

Hi Mike,

Are you looking into getting these bugs fixed.

Cheers,
Brett

I agree the reported height always seems to be zero and this is a bug, however some of the other values are legitimately INDEF. For example, with a 'maxwidth' value of only 15 in some of the CCFs these 15 points are all above the half-power point in the peak and so no width is computed (and hence no R value). I don't know what you mean by a "proper" VHelio. Blindly increasing the maxwidth to something like 50 did turn up a floating-point error I'll need to investigate further.

Is there a particular reason you are using the sinc function for the fit?




The .gki file written is the device-independent graphics metacode. The SGIKERN task runs a graphics kernel to convert this to a "simple graphics interface" metacode and the another SGI translator is invoked to convert this to some device-specific output like EPS. This last stage is controlled by the dev$graphcap entry for the device, in this case



As you can see the 'sgi2ueps' translator does the work in producing the sgi$$.eps (where '$$' is a process id) file in the current working directory. If these files are appearing in the /iraf/iraf directory and that's not where you're running the script then this is some sort of pyraf issue in not properly setting the working directory. If you want the file to have a determinant/fixed name then you can simply edit the graphcap entry to replace the sgi$$.eps with something else.

I was not able to reproduce any problem using the standard CL to convert the files (which are actually more like 330K in size), e.g.



produces a directory full of sgiXXX.eps files that are all complete. In most cases a truncated plot can be attributed to the metacode exceeding the graphics buffer size as can happen with complex plots, the solution then is to increase the value of your 'cmbuflen' environment variable to something larger (although the default 128000 iraf-chars is big enough for these plots).

If you are seeing truncated plots then it may be that pyraf defines a much smaller size by default, it may also be related to the fact that pyraf has its own graphics system and this is a bug in that system. The SGI translator is invoked as an escaped (from the iraf process) command once the argument values are replaced in the command string. There is no explicit flush of the output however since this is an independent command the i/o should be flushed when it exits. I can't comment on whether the real bug is in whether pyraf is properly waiting for the process to exit, or whether the pyraf graphics system is truncating the metacode when you run FXCOR in the first place, but you might want to rant at help@stsci.edu to find out.

Since the EPS files are more than twice the limit you're setting you could try increasing that as well, but a more reliable way to tell when the EPS file is complete is to check that the last line of the file is "grestore showpage", which is the last thing the SGI translator will write. I'll post back when I've sorted out the issues with the sinc fitting.

Hi Mike,



It's understandable for some of the values to be INDEF when the CCF fit does not go past the half-power point, however, always having them as INDEF for cases when the CCF fit does cross the half-power point doesn't make any sense. The Vhelio that is being calculated is just the heliocentric correction and not the velocity shift + heliocentric correction as it should be. Generally I aim to fit approximately 2/3rd of the CCF peak. Fitting more than this introduces noise in the RVs. Fitting less than 1/2 the peak decreases RV precision.


I would like to test different fits to see how well they do for computing RVs. It's a feature of fxcor that should work. A broken feature is useless, it should be fixed or removed. I have used some of the other fits but some of them don't produce RV uncertainties (such as center1d) which is a problem for my RV weighting script. In order for me to properly weight the ~250 fider RVs to produce a single weighted RV for each observation with a precision of better than 50m/s I need the RV, RV error, peak height, peak width, and R number.




This won't work as it would involve exiting iraf, then having my script modify the dev$graphcap, then log back into iraf, CCF a spectra, exit iraf, modify the dev$graphcap again, ... for each fider I CCF (~250 fider per observation X 10 observation in a night means I would need to do this 2500). This is not practical at all. One thing I could do is change the iraf directory after each cross_correlation and then just rename the file instead of dumping the .eps files in the iraf directory and having to search for them and move/rename the files.



This must be a pyraf bug not waiting on iraf to fully write out eps files. The cmbuflen has been set to the largest value possible. The truncated plots are the result of incomplete write out and for some reason there is just enough information to open the files. Some of the files can't be opened even with some data written (similar to corrupted files) and other can't be opened because they are zero byte files.




I will send STSCI an e-mail regarding the bug.



My workaround seems to be working and no eps files are corrupt, truncated, or zero byte.

Cheers,
Brett