**2. Science Operations and Support**
**2.1 Production Correlation**
**2.1.1 Sessions and Their Experiments**
Session 1/2007 had a total of 23 user experiments correlated at JIVE.
Seven used Gbps recording, and there were nine spectral line experiments
(8 of which requiring multiple correlator passes) spread between 22GHz
water and 6.7GHz methanol masers. There was a welcome lack of station
problems: there was only one completely missed station-experiment due to
equipment-related casualties, and only three completely missed due to
weather (there were additional partial experiments missed by at least one
station due to weather). We noticed 180-degree phase jumps between some
scans at L-band in Onsala, and the station friend was able to prevent this
for occurring further.
Session 2/2007 had 17 user experiments correlated at JIVE, ten of which
were spectral line (7 multiple correlator pass experiments) covering
L-band OH and 6.7GHz methanol masers. In the course of correlating
this sessions' experiments, we set a longest uninterrupted (good) sub-job
record at 10h14m30s. Also, as the unattended operation of the correlator
became more robust, we had our first over-100 correlator-hour week.
Session 3/2007 had only 6 user experiments: the Chinese stations were
unavailable because of Chang'E obligations, and many PIs preferred to
defer their observations until they returned to the array. Four of
the experiments were globals, including another on in which the EVN
stations recorded at Gbps and GBT at 512 Mbps (using 1-bit sampling
to keep the number and BW of the subbands the same throughout the global
array). In spite of the brevity of the session, it had our first 7mm
experiment and our first experiment to correlate with 1/8s integrations.
There was also a multiple-record-setting experiment: the most
correlator passes (17), correlator hours (153), total hours required
to produce (483), and total size of resulting FITS-file data (1028.7 GB).
The many passes arose from two correlation phase-centers, each of
which required each of the 8 subbands to be correlated separately to
reduce bandwidth smearing for a wider field of view. The EVN and VLBA
components of the array wound up observing different versions of the
schedule, with incommensurate scan-switching schemes. During correlation,
the EVN stations were put into the VLBA schedule manually, with their
scan start/stop times adjusted to correlate only when they were actually
on the same source. It proved more efficient to correlate such a
schedule scan-by-scan, which led to the larger, tape-like processing factor
over 3. A seventeenth scan was added with only the EVN stations, correlating
using the schedule they observed, so that these stations could be
calibrated amonst themselves better. In the concluding C-band portion
of this session, Torun developed phase jumps in all channels on the
timescale of minutes; this persisted through the next session, but
was repaired before April 2008 e-VLBI runs.
There were three global Gbps target of opportunity observations on SN2007gr
and SN2008d between November 2007 and February 2008 (participating VLBA
stations using 1-bit sampling at 512 Mbps). As an example of the turn-around,
the first on SN2008d was observed 6-7 February, the last disk-pack arrived at JIVE
on 14 February, and the resulting FITS files were placed on the EVN Archive
on 17 February. Session 1/2008 itself was seriously affected by winter
weather. Three experiments were abandoned outright after consultation with
the PIs, and six others missed Effelsberg and/or Jodrell Bank. Azimuth drive
problems forced Torun to miss the 5cm portion of the session. In all,
18 user experiments remained, including 6 Gbps, 6 spectral line (OH, methanol),
and 3 globals.
Session 2/2008 had 21 users experiment, including 7 Gbps, 9 spectral
line (K-band water, L-band OH, methanol, 6030/5 excited OH, and high-z
water observed at 6124 MHz), and six globals coming to JIVE. There were
several firsts in this session: the first participation of the new Yebes 40m
antenna (at K-band), the first participation of EVLA antennas in some
5cm experiments, the first time more than one addition MERLIN outstation
was included in the Cambridge recording (3 outstations, in a single-pol
spectral-line experiment), and our first test at 23.5 GHz consistent
with the new VLBA K-band continuum range and also able to look for ammonia.
One of the globals set a new record for the highest number of stations
correlated in a single experiment at 21. Prior to the session, Cambridge
resolved a long-standing data-throttling problem at 1 Gbps recording.
Session 3/2008 had 15 user experiments, all but two at 512 Mbps or 1 Gbps;
two of the Gbps were globals in which NRAO stations recorded with 1-bit
sampling for 512 Mbps. There were two spectral-line experiments (water, OH),
and two other globals. The Yebes 40m antenna continued to participate at
K-band, and also joined in at X- and S/X-band. The session also saw the
first participation of the three Russian QUASAR stations (Svetloe,
Zelenchukskaya, and Badary) in a multi-frequency project at S/X-, C-, and
L-bands. One of the globals just missed re-setting the most-stations
correlated record: it scheduled 23 stations, but three could not observe.
Hartebeesthoek suffered a serious polar-mount bearing casualty, and missed
the entire session (nor is it clear exactly when it might return).
e-VLBI made huge strides in 2007-8; by the end of the period it had
become a reliable standard operating mode for the EVN. In 2007 there
were 7 user experiments comprising 82 network hours, and in 2008 the
corresponding numbers were 16 experiments and 115 network hours. Five
of these 23 experiments were target-of-opportunity observaitons.
All e-VLBI observations in 2008 used 512 Mbps, and an experiment in April
set a new record for longest sub-job of 12h48m21s. Such long continuous
runs have since become routine. A key development was the ability to
take care of individual-station problems by taking the affected station
out of the correlation temporarily without stopping the job for the
whole array. In disk-based correlation, such an occurrence would trigger
stopping the job and restarting before the problem, but that is a luxury
not afforded to real-time e-VLBI. As an example of the fast turn-around
afforded by e-VLBI, there was an observation on 30 September 2008 that
was correlated, distributed, and analyzed in time to have its results
included in an EVN proposal submitted by the 1 October 2008 deadline.
Tables [##1 and ##2] summarize projects observed, correlated, distributed,
and released in 2007 and 2008. They list the number of experiments as well
as the network hours and correlator hours for both user and test/NME
experiments. Here, correlator hours are the network hours multiplied by any
multiple correlation passes required (e.g., because of continuum/line,
separate correlation by subband/pol to maximize spectral resolution, etc.).
RECORDS:
User Experiments Test & Network Monitoring
N Ntwk_hr Corr_hr N Ntwk_hr Corr_hr
Observed 53 545 910 47 231 231
Correlated 54 529 718 51 262 286
Distributed 54 529 718 54 265 290
Released 55 533 742 54 257 286
Table [##1]: Summary of projects observed, correlated, distributed, and
released in 2007.
User Experiments Test & Network Monitoring
N Ntwk_hr Corr_hr N Ntwk_hr Corr_hr
Observed 72 689 961 48 200 208
Correlated 70 668 1110 47 194 202
Distributed 64 600 1012 45 189 197
Released 60 556 958 48 199 207
Table [##2]: Summary of projects observed, correlated, distributed, and
released in 2008.
Figure [##1] presents the work division among various correlator tasks
(production, clock-searching, network/correlator tests) as a number of hours
per week, over the past three years (2007-8 highlighted). A six-week running
average is shown. Troughs in the production hours correspond to periods when
the correlator queue ran out of possible experiments prior to the start of
the next EVN session, during which time we can focus more on correlator
testing (e-VLBI, recirculation, fine-tuning the disk-servo'ing algorithm,
etc). Bursts of time spent on network tests (cyan) in and immediately
following sessions are also apparent.
[figure 1 ar0708_f1.png goes here]
Figure [##1]: Work division among various correlator tasks, in hours per week.
Figure [##2] presents various measures of correlator efficiency.
The red line plots the completed correlator hours per during time actively
devoted to production correlation (the red line in figure [##1]). The green
line shows completed correlator hours over the total operating time of the
correlator -- the red and green lines diverge more in periods when production
takes up a smaller fraction of the total time available. The blue line shows
completed network hours over total operating time -- the green and blue lines
diverge because some experiments require multiple passes. A twelve-week
running average is shown to smooth out spurious peaks caused by periods
with no remaining production correlation.
[figure 2 ar0708_f2.png goes here]
Figure [##2]: Various measures of correlator efficiency.
Figure [##3] presents the size of the correlator queue at different stages
in the processing cycle, showing a snapshot of the status at the end of
each week. The red line plots the number of correlator hours that remain
to be correlated. The blue line plots the number of correlator hours whose
data remain to be distributed to the PI. The green line plots the number
of correlator hours associated with recording media that have yet to be
released back to the pool (in practice, release occurs prior to the
following session, leading to a blocky pattern for the green line).
The weeks of correlation for the 17-pass GC029 can be seen around the
end of 2007 and beginning of 2008; it took until just before session
3/2008 to completely empty the correlation queue again (red line to 0).
[figure 3 ar0708_f3.png goes here]
Figure [##3]: Size of various correlator queues, measured in correlator hours.
Figure [##4] shows the number of user experiments and the number network
and correlator hours correlated over the past six years, with the hours for
user experiments (diamonds) and the combination of user experiments and
NME/test observations (squares). Including both user experiments and
NME/test observations yields an essentially monotonicly increasing output
from the correlator. Over the past two years, the number of user
experiments has grown strongly thanks to the new e-VLBI observations. Also
plotted at the bottom is the number of JIVE support scientists in the
Science Operations and Support Group, using the scale on the right-hand
ordinate.
[figure 4 nsup.png goes here]
Figure [##4]: Amount of correlator and network hours plus the number of user
experiments correlated in each year, together with the number of JIVE support
scientists in the Science Operations and Support Group.
**__[ I have a concept for a variation of fig.4, with cumulative experience of support
scientist plotted rather than simple number -- still need to massage the personnel
chronological data & plotting algorithms to make this ]__**
**2.1.2 Logistics and Infrastructure**
The disk-shipping requirements are derived from the recording capacity
needed by a session (from the EVN scheduler) and the supply on-hand at the
stations (from the TOG chairman). Now that the VLBA has also shifted
to Mk5 recording, the bookkeeping of disk-flux accounting has become
more complicated. There are two sets of rules to follow:
a) the EVN policy that stations should buy two sessions' worth of disks,
hence the disk flux should balance over the same 2-session interval.
Following distribution to the stations for session 3/2008, we
had "overdistributed" a net cumulative 191.05 TB of disk-pack capacity.
b) the VLBA's need for sub-session turn-around, which essentially requires
pre-positioning the difference between what NRAO stations will
observe in globals to be correlated at JIVE and what EVN stations
will observe in globals to be correlated in Socorro. Following the
shipments in both directions for session 3/2008, we had "overdistributed"
a net cumulative 14.33 TB of disk-pack capacity.
In an effort to accelerate recycling packs somewhat faster than the
"2nd-following session" rule, we began in session 1/2008 to try consciously
to correlate higher data-rate recording experiments first, to maximize
the amount of releasable packs per given correlator hour early enough
to recycle some packs in time for the next session. This optimization is
complicated by fact that many experiments can reside on the same pack,
but it has contributed to the large net "overdistribution" with respect
to the original guidelines.
The data processor has 16 Mark 5A units, all housed inside temperature-
controlled cabinets. Some stations now record exclusively with Mark 5B
units, which we can correlate through the Mark 5A units using capabilities
of the 5A+ firmware. Work progresses on developing native Mark 5B playback,
for which purpose some of the Mark 5A units can be converted to Mark 5B
via the insertion of a new I/O card and some cable re-connections.
Because Mark 5B units could not play back Mark 5A recordings, we would
need to retain enough Mark 5A units to handle NRAO recordings until they
move to Mark 5C (which could be played back via a compatibility mode in
Mark 5B, but not Mark 5A). Maintaining the proper mix of playback units
in light of the phasing of these changes is reminiscent of that required
in the initial tape-to-disk transformation.
We continue to encounter the occasional individual bad disk (or two)
in an incoming pack. We maintain a small bench stock of disks of
various sizes so that we could replace a bad disk locally if that
is the most appropriate course of action (in light of warranty status,
urgency of recycling, etc.), and then we would get a new disk from the
pack's "owner" to replenish our bench stock. All but the highest data-rate
recordings generally play back well with a bad individual disk disconnected.
Individual boards in some station units continue to show some symptoms of
their advancing age. There are enough DMM's that seem prone to causing
byte slips that we actively manage their locations in the SUs to minimize
impact on the correlation. There are no other types of boards at this
stage, but many have very few or no spares. Native Mark 5B playback would
of course by-pass the station units. Currently Yebes 40m and Westerbork
array record exclusively Mark 5B, but Westerbork single-dish may record
via their (borrowed from us) Mark 5A unit to allow simultaneous use of
the rest of the array as WSRT. With a couple Mark-5B-only stations,
then the stock of spare SU boards would increase. There were a couple
instances in 2007-8 in which specific correlator boards exhibited problems,
but these were repaired by ASTRON (S. Zwier). Such problems have to now
always been strongly localized, and as long as we have experiments to
correlate that do not require the full correlator, we can mask out
segments having bad boards.
At the very end of 2008, the (ASTRON) cooling machine received a thorough
maintenance cycle, and the cooling fluid was recharged (it had been operating
on about 1/4 capacity). There were two instances of power glitches in
December 2008, 16 days apart, significant enough to trip everything off-line
that wasn't under UPS protection. Fortunately, none of this happened
during e-VLBI runs. There hadn't been any such power interruptions for
several years, so two so close together was quite unusual.
**2.1.3 Astronomical Features**
**__[ not sure where discussion of the evolution of e-VLBI experiment policy (proposals moving from 2-wk before to
the normal deadlines; the 3 "types" of experiments) should go. In the EVN report, it would seem to be in
the PC section. In a sense, it's not really a JIVE issue per se, but it does affect astronomical
possibilities with the EVN ]__**
We began applying a better post-correlation fraction bit-shift
correction to the phase across the band for each (baseline) visibility
for experiments starting in session 1/2007. Figure [##5] shows an
example of vector-averaged amplitude as
a function of time on some baselines from the 5cm NME N06M2.
[figure 5 -- composed of n06m2.bef.gif & n06m2.aft.gif: they
should be placed side-by-side, with "bef" to the left of "aft"]
Figure [##5]: Effect of the new post-correlation fractional bit-shift
correction on vector-averaged amplitudes as a function of time.
An experiment from session 3/2007 (EP062) was the first to use 1/8s
integrations in production correlation. The correlator can maintain
this output, as long as only 4 (of 8) boards per crate are used in
the correlation (maintaining the current maximum read-out rate of 6MB/s).
This specific experiment therefore required two correlator passes,
each with half (8 of 16) subbands. Subsequent post-correlation processing
led to an additional feature in the Measurement Set creation program
to accommodate both passes in a single measurement set (thus not requiring
the PI to accomplish this in AIPS via VBGLU), and to new glish programs
to ensure that there were no "orphan" subbands (data in only one of the
two passes for a given integration epoch) and to re-sort the single
measurement set such that resulting FITS files would not require any
special pre-processing when being read in to AIPS.
Recirculation is a means of time-sharing the correlator resources such that
data recorded at sampling rates below 32 Msample/s can achieve higher
spectral resolution -- essentially using "idling" correlator chip capacity
to process additional lags. The maximum number of frequency points per
interferometer remains at 2048 (limited by the requirement that a single
interferometer must be processed on a single correlator board), but many
spectral-line experiments now have their spectral resolution limited below
this by the number of stations or polarizations they use. These
many-station/low-BW observations can benefit from recirculation. An
example: without recirculation, an 8-station, 1 subband, 2 polarization
observation can obtain 1024 points in one correlator pass, but it would
not be possible to achieve this with cross-pols or with a ninth station.
As long as the subband bandwidth is no more than 8MHz (for cross-pols)
or 4MHz (for nine or more stations) or 2MHz (for both cross-pols and nine
or more stations), recirculation would allow single-pass correlation.
All values of recirculation up to 8 have been tested in terms of the
resulting amplitudes and phases, with only a few items remaining to
test in more detail.
Typically, only the Jb-Cm baseline has been common to both the EVN and
MERLIN correlations in combined EVN+MERLIN observations. This can
sometimes lead to difficulty in tying the two data-sets together. Because
of the 16MHz bandwidth limit in the MERLIN out-station micro-wave link, the
Cm recording has "unused" subbands in recording modes at 256 Mbps and
above. For dual-pol observations above this rate, an additional
out-station can be recorded onto the unused subbands; single-pol
observations can incorporate three additional out-stations. For such
"high" data-rate observations, there would be no additional scheduling
burden on the PI; all necessary steps can be taken by the VLBI friends at
Jodrell Bank. Observations at or below 128 Mbps would require a separate
schedule for Cm to accommodate additional outstations. Each recorded
MERLIN station would correlate as a separate station at JIVE, thus the
additional out-stations may affect the correlator loading (if the number of
stations increases to nine or more, four times fewer frequency points would
be available in a single correlator pass than with the original VLBI array.
e-VLBI has a natural application here, as the signals for the separate
out-stations can be separated on their transit to JIVE; for disk-based
observations we would need to make copies of the Cm disk-pack prior to
correlation, requiring additional disk-pack availability. The additional
intra-MERLIN baselines included in the EVN correlation would increase the
robustness of the tie between the EVN and MERLIN u-v data sets. Implementation
of this scheme in light of the forthcoming roll-out of e-MERLIN remains to be
investigated (i.e., the 128 Mbps limit on the micro-wave link will disappear,
removing the straightforward existence of "unused" subbands).
**2.2 EVN Support**
**__[ If pictures of various pages of the Archive as desired, we could start to edit a viewgraph from my
EVN symposium ppt ]__**
The automatic-ftp feature added to the field system in 2006 is used in all
network monitoring experiments (or a separate fringe-test experiment, when
an NME is scheduled well outside working hours). This sends a specified
portion of a scan directly to the software correlator computer at JIVE.
At the beginning of this biennial period, we used a version of the NICT
software correlator for these fringe tests. In the middle of 2007, we
shifted to the software correlator being developed under FABRIC/SCARIe.
This provided us with more control over the development of new modes
(VLBA-format data, cross-pols, etc.). Correlation results go to a web
page available to all the stations within a couple hours, and Skype chat
sessions during the NME provide the station friends with even more
immediate initial feedback. The presentation of the results on the web
page has also improved considerably, now showing baseline amp and phase
across the band as well as autocorrelations, and each plot is accessible
by moving the cursor over color-coded baseline/subband/polarization cell
(figure [##6]).
[figure 6 ftpFT.gif goes here]
Figure [##6]: An example of the web-page for an ftp fringe-test scan from
the NME N08L3. The amplitude-frequency plot is for Ef-Wb SB1 RR.
These ftp fringe tests continue to be very successful in identifying
telescope problems and thus have helped to "save" user experiments by
providing feedback quickly enough for the telescope staff to effect repairs,
especially as we see more new stations begin to participate in EVN
observations. An example comes as recently as the very last user experiment
of 2008 -- we could not find fringes to Yebes 40m in the ftp fringe-test
from the K-band NME N08K4; the station was able to trace this to an IF
local oscillator, which they replaced in time for the following EB037E
the next morning, in which Yebes fringes were fine.
The EVN pipeline now runs under ParselTongue (a Python interface to
classic AIPS). The new pipeline is considerably easier to use, more
robust and has much greater scope for future development due to the
improved coding environment. The pipeline scripts are available from the
ParselTongue wiki (RadioNet) and should provide a good basis for other
(semi-)automated VLBI reduction efforts. We continue to process all
experiments, including NMEs, via the pipeline, with results being posted
to the EVN web pages. The pipeline provides stations with feedback on
their general performance and in particular on their gain corrections, and
identifies stations/frequency bands with particular problems. Timely
delivery of ANTAB amplitude calibration results from the telescopes seems
to be improving, but remains an issue in e-VLBI experiments due to the
shorter time-scales involved.
In the course of pipelining, the support scientist also calculates the EVN
Reliability Indicator (ERI) for each experiment. This ERI conceptually is
the ratio of good to expected visibilities in the distributed FITS files.
Two statistics are calculated: one including all losses (including
weather), and another that discounts weather and and other natural causes
over which the EVN has no control (denoted ERI*). Figure [##7] shows the
evolution of ERI* over time, with each experiment plotted separately. The
improvement brought by the shift to disk-based recording (median ERI* not
below 84% since session 2/2004) and the even more marked improvement in
e-VLBI reliability since its inception at the end of 2006 are apparent.
[figure 7 eriAR0708.png or eri4AR0708.png goes here]
Figure [##7]: Plot of the EVN Reliability Indicator for pipelined user
experiments up through session 3/2008. Red squares denote e-VLBI experiments.
[the first plot shows 2001-2008; the second 2005-2008 (easier to see temporal
detail, but less sense of sweeping trends)]
A considerable amount of time between October 2007 and July 2008 went
into working with Westerbork on operational tests of their new digital
TADUmax back end. Thirteen separate test observations were made in
that time to check out various modes typically used in VLBI, and
to iron out the details of the bit-encodings and sampler statistics.
The advantages of TADUmax include full coverage of 128MHz total bandwidths
in Gbps observations (the previous system could get only 7/8 of the
coverage) and much more rectangular bandpass shapes. The digital filters
do add a channel-bandwidth dependent clock offset; log2vex can now
account for this in creating the correlation-controlling vex files.
**2.3 PI Support**
**__[ if we want various pictures of the proposal tool interfaces, we could extract them from
a viewgraph from my EVN symposium ppt ]__**
The EVN Archive at JIVE provides web access to the station feedback,
standard plots, pipeline results, and FITS files for experiments correlated
at JIVE. Very few PIs request distribution of their FITS files on physical
media (DAT or DVD) anymore. There were 3404 FITS files downloaded in
2007-8, from people in 32 different countries (including 21 non-EVN
countries, and 12 of those being outside the EU and associated states (in
an FP6 sense). Public access to the FITS files themselves and derived
source-specific pipeline results is governed by the EVN Archive Policy --
the complete raw FITS files and pipeline results for sources identified by
the PI as "private" have a one-year proprietary period, starting from
distribution of the last experiment resulting from a proposal. PIs can
access proprietary data via a password they arrange with JIVE. PIs receive
a one-month warning prior to the expiration of their proprietary period.
We have increased the storage available on the archive machine from 4.5 TB
to about 16 TB. The total size of the FITS files in the archive at the end
of 2008 was about 5.85 TB (a 2.77 TB gain in the two-year period); figure
[##8] shows the growth of the EVN archive size over time.
[figure 8 archvAR0708.pdf goes here]
Figure [##8]: Growth in the size of FITS files in the EVN archive.
Experiments archived in this biennial period are plotted in red.
The science operations and support group continues to contact all PIs once
the block schedule is made public to ensure they know how to obtain help
with their scheduling, and to check over schedules posted to VLBEER prior
to stations downloading them. In previous years, there had been a
handful of instances in which a station observed using a superseded version
of an experiment's schedule. New
safety features have been incorporated into the pre-observation system that
should help avoid such incidents. Indeed, with the glaring
exception of GC029, these have not recurred. In GC029, the EVN stations
all observed an older version of the schedule while the VLBA stations
observed the current one -- and the two versions had decidedly different
scan timing patterns. We had to adjust the
correlation-controling vexfile manually to include EVN stations only those portions of
scans in which they overlapped
with the proper version of the schedule. Efforts to confirm that different
parts of global arrays have the same version of the schedule redoubled
thereafter. Policy discussions about the dangers inherent in EVN and VLBA
stations accessing the schedules from independent locations also got
underway. The pre-observation communication also provides the opportunity
to inform eligible PIs about the benefits of the RadioNet trans-national
access programme, if applicable.
The RadioNet-driven NorthStar web-based proposal tool became the sole means
to submit EVN or Global VLBI proposals starting from the 1 February 2007
proposal deadline. Later, e-VLBI proposals joined the fold. Currently,
only target-of-opportunity proposals are handled outside of Northstar.
Feedback is solicited from proposers after each deadline, and their
insights are reviewed to continue to improve the user-friendliness of the
proposal tool.
JIVE hosted 16 data-reduction visits in 2007 and 10 in 2006. In addition,
through the period of the report there were seven post-graduate students
who were co-supervised by members of JIVE staff, and who visited
frequently. The visitors room has five dual-processor PCs running linux,
one windows-based PC, and a small cluster of four interconnected top-end
workstations, to accommodate processing of very large wide-field data sets
(whose monitors occupy an additional two work places.
[[2007-2008:2007-2008|back to index]]