JIVE Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== NEXPReS Second Quarterly Report ======

Download this report as a {{:nexpres:nexpres_quarterly_2010_december.pdf|PDF}}.
===== Introduction / Project Summary =====

==== Summary ====

This is the report for the end of the second qurter of the project (months 4-6).



==== Overall Deliverables Status and Timeline ====

[[nexpres:nexpres-deliv-table-1|Link to the Deliverables Table]]

Deliverables due during the second reporting quarter:

^ D#     ^ Month Due  ^
| D5.02 Metrics, selection criteria and user feedback                           |  4  |
| D5.03 Definition of specifications monitoring and remote control              |  4  |
| D7.01 Workflow Manager requirements from telescope operators (OSO)            |  4  |
| D7.02 Workflow Manager requirements from correlator operators (JIVE)          |  4  |
| D4.05 Visual Guide/Elements                                                   |  6  |
| D8.01 Interface design document of storage element API (AALTO)                |  6  |

The following deliverables were submitted to and their receipt acknowledged by the Commission during this quarter:

^ Deliverable Number and Description                                ^ confirmed on ((this is the date on which the Commission sent us email to confirm receipt of the listed deliverables)) ^
| D4.8 Project “Fact Sheet”                                         |  29 Nov 2010  |
| D1.1 Appoint Programme Manager                                    |   8 Dec 2010  |
| D1.2 Appoint Project Assistant                                    |   8 Dec 2010  |
| D1.3 Form Board                                                   |   8 Dec 2010  |
| D1.4 Create Project contact lists                                 |   8 Dec 2010  |
| D1.6 Form Management Team                                         |   8 Dec 2010  |
| D1.7 Convene Advisory Board                                       |   8 Dec 2010  |
| D1.8 Convene Management Team                                      |   8 Dec 2010  |
| D4.1 Communications Manager Appoint                               |   8 Dec 2010  |
| D4.2 Online Resources                                             |   8 Dec 2010  |
| D4.4 Display                                                      |   8 Dec 2010  |
| D4.5 Visual Guide/Elements                                        |   8 Dec 2010  |
| D5.1 Mixed e-VLBI / recorded-VLBI at stations                     |   8 Dec 2010  |
| D5.3 Definitions of specifications monitoring and remote control  |   8 Dec 2010  |

===== WP 1 - Management =====
==== Activities of Note ==== 

=== Not So Monthly Update ===

NEXPReS and project officer Siren have exchanged emails and decided to move away from monthly updates to a quarterly format.  From this month, we will produce a report every three months.  The quarter in which the period (annual) review is held, no update will be provided as the materials should (had better) be in the annual report.  I expect that this announcement will be received with minor celebration among the other work package leaders.


=== FP7 2011.1.2 Call ===

Astron is leading a call and has been in contact with JIVE to participate.  Astron has also asked for input on the construction of the proposal as it is felt that the NEXPReS was well executed.  


=== Internal Reminder ===

This is also an internal reminder indicating that we will try to be better about cc'ing the NEXPReS project's email address provided by the EC for all official communications.  To this end, we have set up an internal mailing list called NEXPReS-deliverables that will be used for official communications.  The mailing list members are our project officer, his support secretary, the project manager and the NEXPReS Secretary email accounts.  Only the project manager and the project secretary account will be able to send to the list.

=== Deliverables Table in the Wiki ===

Note that the deliverables table has been edited slightly to remove some redundant information and to make it easier to read.  It continues to be updated regularly as deliverables are completed.

=== Mailing Lists (Generally) ===

The mailing lists for the project are being used and have seen the expected bursts of traffic at the start of the project along with some periods of silence.  



==== Deliverables Status Update ==== 

WP1 has one outstanding deliverable, the Consortium Agreement, which should be confirmed by the board early in 2011.  
 

==== WP hosted meetings (as appropriate) ==== 

No Meetings were hosted by NA1 this quarter.

==== Publications, Presentations and Talks ==== 

The Project Manager attended the 8th e-Infrastructure Concertation event in Geneva, Switzerland.  The meeting was held 4-5 November 2010 at the CERN facility.  NEXPReS was given a chance to present a quick overview of the project along with other projects from the funded cohort.  Additional information about the meeting is available from the meeting website: [[http://www.e-sciencetalk.org/e-concertation/|http://www.e-sciencetalk.org/e-concertation/]].  Note, the agenda is made available via the CERN website.  A {{:nexpres:8th_e-conceratation_timetable.pdf|PDF copy}} is uploaded here in case the CERN site removes information about old meetings.  

At the meeting, the Project Manager gave an informal video interview with the Grid Talk Project.  The interview is available online on their website: [[http://gridtalk-project.blogspot.com/|http://gridtalk-project.blogspot.com/]].
   
   

===== WP 2 - EVN-NREN =====
==== Activities of Note ==== 

The EVN-NREN work package is scheduled to have a meeting in the new year.  Discussions are currently underway with several other interested groups (non-NEXPReS organizations) to have the EVN-NREN meeting immediately before/after their meeting.  The EVN-NREN meeting is seen as a good way to draw both researchers and networking professionals together.  We should know by the end of January if we will partner with another group for the meeting and be able to start advertising information about the location.


==== Deliverables Status Update ==== 

No deliverables were due this quarter.

==== WP hosted meetings (as appropriate) ==== 

The workpackage hosted no meetings this quarter.

==== Publications, Presentations and Talks ==== 

There were no presentations on behalf of the EVN-NREN this quarter.

===== WP 3 - eVSAG =====
==== Activities of Note ==== 

Contacts with the EVN PC Chairperson (Tiziana Venturi) to coordinate a possible face-to-face eVSAG meeting to be organized at the time of an EVN PC meeting.

Collected information on EVN vs. e-EVN proposals submitted in the last years.

Collected information from users regarding their experiences and expectations on e-VLBI.


==== Deliverables Status Update ==== 

No deliverables were due this quarter.

==== WP hosted meetings (as appropriate) ==== 

The workpackage hosted no meetings this quarter.

==== Publications, Presentations and Talks ==== 

There were no presentations on behalf of the eVSAG this quarter.

===== WP 4 - Communications =====
==== Activities of Note ==== 
The WP4 leader collaborated with professional artists in months 2-5 to create a visual representation of the project that could be incorporated into all outreach materials, giving them a unified look and feel. This illustration was used in the large display (D4.4) and will also be used for slide templates, print materials and an updated header for the project web site. 

The display made its debut at the 9th International e-VLBI Workshop in Perth, Australia, in month 4. It was therefore unavailable for the ASTRON/JIVE Open Day, and NEXPReS was represented instead with an A0 poster version.

Other WP4 activities this quarter included preparation for the ASTRON/JIVE Open Day (24/10/2010) and work on the first set of print materials (D4.6), due in the next quarter.

==== Deliverables Status Update ==== 
D4.5 Visual Guide/Elements was completed in month 5 (due month 6). Graphic resources including project logos, visual illustrations, colors and fonts have been posted to [[nexpres:communications:visualguide|WP4 section of the wiki]]. An email about availability and use of these resources was also sent to all project partners in month 5.

==== WP hosted meetings (as appropriate) ==== 
No WP4 meetings are necessary.

==== Publications, Presentations and Talks ==== 
WP4 leader Kristine Yun presented outreach activities of NEXPReS (plus EXPReS and JIVE) at a RadioNet Outreach Workshop in month 5. The idea was to share experiences and activities among all European radio astronomy institutes and to find ways to share resources. NEXPReS will likely be able to contribute to and benefit from shared work on brochures and videos, in particular.

===== WP 5 - Cloud Correlation =====
==== Activities of Note ==== 

At TUM, good progress was made on task 3, with updates, fixes and essential add-ons of existing code: 

  * Help pages forward
  * Remote system call handling
  * Update of MathPlot for monitoring plots
  * BugFixes of internal modules like timecalc
  * Fixes of compiler warnings and static code analysis, tests on different 32-bi/64-bit operating system platforms
  * First attempts to use the communication generator under Windows
  * Tests for an installation of a NASA Field System PC on Ubuntu Linux

Measuring functionality was added:

  * System information, network-utility (traffic load etc.)
  * Cyclic redundancy check (CRC32) for internal functionality cross-check
  * Software quality code test coverage, automatic test suite 

A Web server will be installed end of December/mid January. Several tele-conferences with colleagues at MIT Haystack Observatory and NASA Washington were held. Preparations for the standardization for System Monitoring within the MCI (Monitoring and Control Infrastructure) Collaboration Group are ongoing. 

After an initial slow start, work at JIVE picked up considerably. One staff member is now working full-time on WP5, with several others contributing to the project part-time. A new staff position was advertised, but unfortunately no suitable candidates were found so far. A new advertisement for scientific software engineering positions in WP5 and WP7 will go out in January 2011. 

Functionality was added to the locally developed Mark5 control code jive5ab. VDIF-Mark5B on-the-fly conversion software was developed and awaits testing. A first step was made towards a generic correlator user interface, which will be capable of controlling both hardware and software correlators and which will be essential for automating operations. 4Gbps-capable Mark5C recording systems were ordered from the Conduant Corporation and were delivered in November 2010. Several of these have been integrated into the correlator system and tests are ongoing.



==== Deliverables Status Update ==== 

The draft of D5.3, //Definitions of specifications monitoring and remote control//, was revised and the final version was submitted in October 2010. D5.2, //Metrics, selection criteria and user feedback//, was completed on December 21 2010.

==== WP hosted meetings (as appropriate) ==== 

None

==== Publications, Presentations and Talks ==== 

Presentations:

  * {{:project:nexpres_wp7:nexpres_wp5:arpad_evlbi_perth.ppt|EXPReS, NEXPReS, UniBoard: Developments in European VLBI}}  A. Szomoru, 9th International e-VLBI Workshop: e-VLBI and the Path to the Square Kilometre Array, Perth, Western Australia, 19 November, 2010

  *  SysMon, a robust and flexible remote monitoring system for VLBI and more, Ettl M., Neidhardt A., Mühlbauer M., Riedere M., Kronschnabel G., Himwich, E., Beaudoin C., Plötz C., Dassing R., 9th International e-VLBI Workshop: e-VLBI and the Path to the Square Kilometre Array, Perth, Western Australia, 19 November, 2010 

  * {{:project:nexpres_wp7:nexpres_wp5:arpad_shanghai10.ppt|EXPReS, NEXPReS, UniBoard: Developments in European VLBI}}  A. Szomoru, Working visit to Shanghai Astronomical Observatory, Shanghai, China, 25 November, 2010

  *  Network Communications, Schreiber, U., Neidhardt A. et al., GGOS Bureau for Networks and Communications at the AGU, San Francisco, USA, 15 December, 2010

  *  Monitoring and Control Infrastructure, Beaudoin C. et al., GGOS Bureau for Networks and Communications at the AGU, San Francisco, USA, 15 December, 2010

Posters:

  *  First proof of concept of remote attendance for future observation strategies between Wettzell (Germany) and Concepcion (Chile), Ettl M., Neidhardt A., Mühlbauer M., Plötz C., Hase H.; Sobarzo S., Herrera C., Oñate E., Zaror P., Pedreros F., Zapato O., 9th International e-VLBI Workshop: e-VLBI and the Path to the Square Kilometre Array, Perth, Western Australia, 18 October, 2010






===== WP 6 - High Bandwidth on Demand =====
==== Activities of Note ==== 

Several participants in WP6 came together for a face-to-face meeting just prior to the official NEXPReS kick-off meeting in Manchester.

Work on the architecture document (milestone MS601) is progressing well and should be finished shortly. It is somewhat delayed due to the late start of the project itself, and the fact that most participants could only start the hiring process once project was started. There have been three WP6 telecoms to coordinate the work on MS601: the minutes of all three are linked below.

[[nexpres:nexpres_wp6:telecon_2010-08-27|Minutes of WP6 Kickoff Telecon on 2010-08-27]]

[[nexpres:nexpres_wp6:telecon_2010-11-02|Minutes of WP6 MS601 telecon 2010-11-02]]

[[nexpres:nexpres_wp6:telecon_2010-11_30|Minutes of WP6 MS601 telecon 2010-11-30]]

Work has also started on designing and configuring the first international BoD path, both as a test and as a first step towards deliverable D6.1.

==== Deliverables Status Update ==== 

(none in this quarter)

==== WP hosted meetings (as appropriate) ==== 

See above for the telecons held in WP6 and their minutes

==== Publications, Presentations and Talks ==== 

The activity leader (Paul Boven, JIVE) gave several presentations:

{{:nexpres:nexpres_wp6:7thtf-storage.pdf|High speed continuous recording and playback for VLBI}}, presentation for the Terena 7th TF-Storage meeting, Poznan, Poland, 2010-09-09 (11MB)

{{:nexpres:nexpres_wp6:snrd2010.pdf|NEXPReS: Dynamische Lichtpaden voor Radio Astronomie}}, presentation for the SURFnet Relatiedagen 2010, Noordwijkerhout, NL, 2010-12-13 (10MB, in Dutch)

{{:nexpres:nexpres_wp6:9th-eVLBI-WS.pdf|Dynamic Networking for NEXPReS}}, presentation at the 9th e-VLBI Workshop, Perth, Australia, 2010-10-20 (7MB)

===== WP 7 - Computing in a Shared Infrastructure =====
==== Activities of Note ==== 

=== JIVE ===

Mark Kettenis attended the 9th International e-VLBI workshop in Perth,
Australia, and gave a presentation on WP7.

Since several key people to doing e-VLBI with software correlators
were present, a lot of useful exchange of ideas happened.  These
include discussions about the implementation of e-VLBI functionality
in the DiFX software correlator with Chris Phillips (CSIRO).  Also the
VDIF/VTP taskforce met, and made some good progress in defining the
VLBI Transport Protocol (VTP).  There is still some discussion on reply
packets.  A proposal for the information within those packets has been
made.

Progress is also being made to incorporate VDIF support into the
international VEX standard.  Within the VEX2 committee, of which Mark
Kettenis is a member, consensus has been reached about most aspects,
and an updated standard is expected to be finalized within the next
few months.  This will help us with scheduling experiments for
distributed correlation.

After the workshop Mark also visited CURTIN for two days, and fleshed
out some details of the collaboration.  Cormac Reynolds, who is head
of the VLBI group at CURTIN, will be our primary contact there.

Demonstration of the SFXC software correlator at the Dutch Research
exhibit at SC10, The International Conference for High Performance
Computing Networking Storage and Analysis in New Orleans, United
States.  During this demonstration, pre-recorded data of a pulsar
observation sitting on diskpacks at JIVE was streamed into a cluster
in Amsterdam where it was correlated.  The result of the correlation,
a pulse profile for the pulsar, was displayed at the exhibit in New
Orleans and updated while the correlation was running.  This shows
that, with the right setup, streaming data into a software correlator
is feasible and that it i possible to provide real-time feedback on the
correlation.  Some discussions with people form SARA, investigating
the potential to make use of the Dutch National Grid Infrastructure
(BIG Grid) in NEXPReS.

Unfortunately deliverable D7.2 wasn't finished before all this travel
started, and very little time was found afterwards to finish the
document.  There is very little work that still needs to be done, so
the report is expected to be finished early January.

Support for correlating multiple phase centers was added to SFXC and
is currently being tested.  This feature allows the scientists to
observe a larger part of the sky and generate several smaller datasets
instead of a single big dataset that is difficult to handle.  This
improves the survey capabilities of our instrument and infrastructure,
which will help us achieve some of the science goals mentioned in the
NEXPReS proposal.

Support for reading data directly from Mark5 diskpacks was added to
SFXC.  This allows us the bypass the fuseMk5A filesystem.  This will
greatly help deliverable D7.4 as it will simplify operations at the
observatories since it eliminates an action that was done manually.

Also, the first EVN user experiments were correlated using SFXC.  As a
result a lot of fixes to the codebase were made that improve the
stability.  This is crucial for running automated e-VLBI correlations.

Some effort was put into optimizing the corner turning done by the
channel extractor in the software correlator.  Hand-written assembly
language implementations that use SSE instructions were written for
the most common setups that are in use.  Compared to the (already
optimized) C versions of the code these are twice as fast.  The same
code can be used to convert the legacy Mark4, VLBA and Mark5B data
formats into VDIF such that separate data streams can be created that
are needed to distribute the correlation over several clusters.

=== OSO ===

At OSO work continued on gathering telescope operator requirements for the work
flow manager.  These have now been compiled into a report which was distributed
to relevant partners at the beginning of November, satisfying deliverable D7.1.
Investigations into semi-automated telescope schedule creation have begun; existing
tools used for working with schedules have been examined to get a feel for what is
required as well as locate possibilities for re-using or taking inspiration from
existing code. Programming languages appropriate to this task have also been looked at.

=== PSNC ===

To provide better performance of Workflow Framework a proper workflow description language has to be chosen. In previous version of EXPReS a self-developed solution basing on JSON was harnessed, however the effort to maintain this solution is increasing - especially when adding functionalities not foreseen at the beginning of implementation. Another aspect is to support standards or de facto standards. Regarding those arguments we decided to use an already existing workflow language. Since there are many solutions available a wide research was made in this area.
The first step was to narrow the area of interest. We have identified couple of groups of workflow languages and then found the most popular or of wider use among those groups. The groups are: BPEL family (BPEL itself and other BPEL-related languages), WebServices family (workflows basing on WS idea, using WS mechanisms or protocols), Grid/Scientific family(languages developed for Grid environments or to support scientist of different domains).

The first step was to narrow the area of interest. We have identified couple of groups of workflow languages and then found the most popular or of wider use among those groups. The groups are: BPEL family (BPEL itself and other BPEL-related languages), WebServices family (workflows basing on WS idea, using WS mechanisms or protocols), Grid/Scientific family(languages developed for Grid environments or to support scientist of different domains).  See {{nexpres:quarter_report_psnc_4_2010.pdf|PDF}} for details.

Criteria for selecting a workflow language were formulated and each language was evaluated.  Again see {{nexpres:quarter_report_psnc_4_2010.pdf|PDF}} for details.  This allowed us to choose MoML as a new language for KIWI Workflow Platform. The Kiwi Workflow Platform will be used within NEXPReS to support telescope and correlator operators. The main highlights of MoML are its extensibility, action-centric design, de-facto standard in many scientific domains and support provided by groups of interest gathered around projects such as Kepler. Another point of great significance is the availability of both – MoML engine and graphical libraries as well as some work in field of web applications interfaces.

We have also started an initial work on MoML language, as well as MoML engine which we would like to use in the NEXPReS.

=== VENT ===

At Ventspils, Normunds Jekabsons has started experimenting with the SFXC correlator on their cluster.

==== Deliverables Status Update ==== 

D7.1 was completed on November 10 2010.

D7.2 is still in progress.  It is almost completely done but still needs some small touch-ups the appendix on invoking the SFXC software correlator needs to be integrated.  This is expected to be completed in early January 2011.
==== WP hosted meetings ==== 

A telecon was held on Telecon October 12, 2010.  The minutes can be found on the project WiKi.

==== Publications, Presentations and Talks ==== 

{{:nexpres:kettenis_evlbi2010.pdf|Automated distribute correlation on a shared computing infrastructure}}, M.M. Kettenis, 9th International e-VLBI Workshop: e-VLBI and the Path to the Square Kilometre Array, Perth, Australia, 18 October 2010

Smart Infrastructure for VLBI Software Correlation (poster and demonstration), M.L. Cristea, R.J. Strijkers, R.J. Meijer, V. Korkhov, A. Belloum, M.M. Kettenis, K. Keimpema, C. de Laat, R.J. Meijer, New Orleans, US, 15-18 November 2010	  

===== WP 8 - Provisioning High-Bandwidth, High-Capacity Networked Storage on Demand ===== 

==== Activities of Note ==== 

The activity has continued as planned. WP8 has looked further into the subjects designated in the presentation of September board meeting.

It has been diffiuclt to hire people to some of the tasks requiring simultaneous expertise in several fields (Linux kernel, advanced communications, hardware and FPGAs, networking security etc.).  We finally managed to hire Esa Turtiainen beginning of December 2010 but we are still looking for at least one additional resource.  WP8 has also been able to extract in-kind contributions (not paid by NEXPReS) from Metsähovi Linux experts Timo Lindfors and Juha Aatrokoski in disk and 10GbE Linux performance related issues.

WP8 is on-track in getting explanations and solutions to most of the open questions of EXPReS 20-disk 4Gbps performance issues, CPU consumption, eliminating
expensive memcpy()s, and further understanding how to deal with close-to-10Gbps UDP networking efficiently.  Additionally, ideas how to do simultaneous reads and writes to the same set of disks and how to make best use of large RAM buffers are evolving.

==== Deliverables Status Update ==== 

Deliverable 
{{:nexpres:2010-12-21_wp8-d8.1.pdf|D8.1}}
was released mid-December (due month 6).

==== WP hosted meetings (as appropriate) ==== 

None.

==== Publications, Presentations and Talks ==== 
 
There is a plan for WP8 leader Ari Mujunen to give a colloqium talk at ASTRON/JIVE in February 2011 on the subject of evolving storage in astronomy computing.