User Tools

Site Tools


nexpres:nexpres_wp7

Deliverables

D# Title Link Date
D7.1 Workflow Manager requirements from telescope operators (OSO) DOC 2010/11/10
D7.2 Workflow Manager requirements from correlator operators (JIVE) PDF 2011/01/13

Documents

Institute Title Link
OSO Workflow Manager requirements from telescope operators DOC
PSNC Workflow Manager framework evaluation PDF
JIVE Workflow Manager requirements from correlator operators PDF
OSO NEXPReS WP7 Task 2 Design document DOC

Meetings & Telecons

Telecon August 18, 2010

14:00-15:00 CEST (12:00-13:00 UTC)

Agenda

  1. Project status
  2. Reporting
  3. Distribution of funds
  4. Upcoming deliverables & milestones
  5. Agenda for Manchester meeting
  6. Future telecons
  7. Any other business

Minutes

Present:

Mark Kettenis (MK), Aard Keimpema, Arpad Szomoru, Charles Yun (TCY) Simon Casey, Michael Lindqvist Dominik Stoklosa, Tomasz Rajtar, Katarzyna Bylec Ivars Shmeld

Agenda

 1. Project status

Not much to report. MK says the work on the software correlator at JIVE is ongoing; a preliminary implementation of VDIF support is already integrated.

 2. Reporting

There will be monthly reports to the EU. MK will collect input for the monthly status report and make it available to the project manager. TCY says the status reports will live on the WiKi; a template for the report for August is already there.

 3. Distribution of funds

*Action*

Please indicate the expected spending (man-months funded from the project and hardware) for the first year of the project to the Work Package leader, such that the Project Manager can make a proposal for the distribution of the first round of EC funding for the board meeting.

 4. Upcoming deliverables & milestones

First deliverables and milestones are the requirement documents to be written by JIVE and OSO. The documents will explain how operations at the correlator and telescope will work, and what elements of the process can be automated. We should probably aim for something like 10 pages each. Simon Casey will be working on the document from OSO.

PSNC will start work on workflow manager. Dominik and Ivars will have contact on how VENT can collaborate best on the computing side of things.

 5. Agenda for Manchester meeting

The plan is to start with short 10-20 minute presentations from JIVE, OSO and PSNC about the plans/ideas on what they're working on. Some discussion about the technology to be used (WebServices, tools, languages) seems desirable to prevent interoperability issues. DS likes to have a discussion/brainstorm about how to set up the distributed correlation. MK indicates that there should be enough time for that and will put it on the agenda.

*Action*

MK, DS and ML to create short presentations for Manchester meeting.

 6. Future telecons

Mark proposes telecons every month/two months, which seems reasonable to all participants. The telecon facilities used by JIVE are available for this purpose. Dominik suggests to look at EVO, which is used regularly for videoconferencing at PSNC. Arpad mentiones that his experiences with EVO aren't great. Next telecon will probably be scheduled somewhere in October.

 7. Any other business

Meeting September 20, 2010, Manchester

09:00-12:00 BST (08:00-11:00 UTC)

Profs meeting room 3.110

Jodrell Bank Centre for Astrophysics
The University of Manchester
Alan Turing Building,
Manchester,
M13 9PL,
United Kingdom
Phone: +44 (0)1477 571321
Fax +44 (0)161 275 4247

(same building as the board meeting, different room)

Agenda

start lead description slides
09:00 MK Introduction
09:10 MK Activities at JIVE: SFXC, correlator viewpoint PDF
09:30 DS Activities at PSNC: Workflow manager & broker PPT
09:50 ML Activities at OSO: telescope viewpoint, science PPT
10:15 Coffee
10:30 Discussion about global/distributed correlation
11:15 Discussion about technical issues: what tools/languages should we use to avoid interoperability issues, etc. etc.
12:00 Lunch

Minutes

Present:

Mark Kettenis (MK), Dominik Stoklosa (DS), Michael Lindqvist (ML), Ivars Smelds (IS), John Conway (JC, partly)

After the presentations, discussion starts on the workflow manager. MK suggests that that it needs to provide at least three “perspectives”:

  • PI (scientist)
    • submit schedule (VEX file)
    • correlation parameters
    • get notifications
  • Telescope Operator
    • accept schedule
    • get monitoring feedback
  • Correlator Operator
    • assign resources
    • get monitoring feedback

DS says that the workflow manager for NEXPReS will be web-based instead of a JAVA (web)application.

DS also says that instead of full-blown SOAP PSNC is looking at a much lighter weight JSON-based RPC to communicate with VLBI components. That should make interoperability much less of a problem.

Authorization and Authentication is discussed briefly. There clearly is a need to do this “properly” for NEXPReS. Some sort of design document may be desireable (but this is not a deliverable).

After some discussion MK proposes that the starting point of a workflow should be a schedule (provided as a VEX file) submitted by the PI or perhaps the correlator operator.

JC raises the issue of observations of transients triggered by, for example, LOFAR. He suggests that such an observation should probably use a standard setup/mode (say 6cm, 1Gbps) with a standard set of telescopes. Only the sources and times in the schedule would be different. MK and ML agree that this would be a sensible approach.

DS starts the discussion about global correlation by stating that streaming data directly into cluster nodes may not be feasable. MK says that one of the lessons learned from EXPReS was that things slow down quite drastically if uncorrelated data ever hits disk. DS says that PSNC now has some high-performance storage that is directly accessable by cluster nodes which may be used as a staging area. MK notes that other sites probably won't have such high-performance storage available.

Telecon October 12, 2010

14:00-15:00 CEST (12:00-13:00 UTC)

Agenda

  1. Project status
  2. Upcoming deliverables & milestones
  3. Telescope Feedback
    • Can we use the work done at Wetzell (TUM)?
    • See roadmap: DOC
  4. Global Correlation
    • Stream directly into cluster or do we need a staging area? Or both?
    • Direct: PDF
    • Staging: PDF
  5. Any other business

Minutes

Present: Present Mark Kettenis (MK), Aard Keimpema, Dominik Stoklosa (DS), Kasia Bylec, Tomasz Rajtar, Alexander Neidhardt (AN), Martin Ettl (ME), Simon Casey (SC), Ivars Shmeld (IS), Normunds Jekabsons (NJ), …

1. Project Status

MK mentions that our EC officer has indicated that from now on reporting will be done per quarter.

3. Telescope Feedback

AN starts with a short introdcution about the work on stations monitoring & remote control that will be done by TUM, MPG & JIVE in WP5.

MK starts that full remote telescope control is probably not needed in WP7. AN replies that full remote control isn't the real goal of their work and that various levels of monitoring and remote interaction with telescope control will be implemented.

DS asks whether the user interface is web-based or a standalone application. AN replies that they will provide a wxWidgets based GUI that is a standalone application, but that there is a clear seperation between layers that allows access to the functionality through RPC calls.

DS asks when the work that will be done in WP5, Task3 is ready. AN replies that a basic system is already working and running, and will be made available before the end of the year. Integration in the NASA field system will hopefully happen in may 2011.

MK asks if user/password is the only scheme by which Authorization and Authentication is going to be implemented. This might pose a problem for services that run as part of the WP7 correlation platforms. Embedding passwords in applications is not very secure. DS agrees with this. AN acknowledges this and agrees that some further thinking is probably needed. Grid-like certificates and ssh-based schemes are discussed shortly. DS mentions that it is possible to have so called Robot Certificates for services in addition to personal certificates. NJ suggests the use of OpenVPN to create a secure network. AN replies that this is probably not necessary since they use

SC says that a 2nd field system PC exists in Onsala for dBBC tests. That machine could possibly be used for testing.

MK thinks that if Onsola is able to participate in testing, tapping into the system devloped at Wetzell for telescope monitoring defenitely looks like a very good option.

4. Global Correlation

MK starts the discussion of the possible setups as illustrated by the diagrams. Both TCP-based datatransfer (as implemented in SFXC) and UDP-base (VDIF/VTP) datatransfer are discussed. MK mentioned that especially the UDP-based transfer may be problematic if data is to be streamed directly to cluster nodes. DS confirms this, and indicates that using intermediate storage, directly accessable from the cluster nodes) (as illustrated by the second diagram) may be necessary. Such storage, providing 6 GB/s of access bandwidth is being installed at PSNC.

MK indicates that other cluster may not have storage that is suitable for this. In particular the cluster of JIVE doesn't have this. But at JIVE direct streaming will be an option. DS suggests that it is probably necessary to support both setup. MK agrees. MK points out that it will be necessary to run some data transfer software on the storage nodes that face the outside world. DS thinks that this will be possible, but has to verify this.

DS suggests that it may be a good idea if changing the destination of individual subbands while the correlation is running would be a good idea. MK agrees and will check if such a capability is available in VDIF implementations that are currently being designed.

Ventspils will prepare an overview of the computational infrastructure they can provide. They will also try to install the software correlator and test it. They will visit PSNC when workflow manager design is a bit more solid and decide on division of labour at that point, potentially in the 2nd half of November.

Telecon March 29, 2011

14:00-15:00 CEST (12:00-13:00 UTC)

Agenda

  1. Project status
  2. Upcoming deliverables & milestones
  3. Poznan Meeting (Apr 19-20)
  4. Aveiro Meeting (May 24-26)
  5. Design document for task 2: DOC
  6. Any other business

Minutes

Present:

Mark Kettenis (MK), Aard Keimpema, Simon Casey Dominik Stoklosa (DS), … Ivars Shmeld, Normunds Jekabson

  1. Project Status
  2. Upcoming deliverables & milestones

Next deliverable is a demonstration of e-VLBI with the SFXC correlator before the end of the year. Given that the e-VLBI capabiliy is under active development and the first preliminary tests of the functionality have been successful, we should be on schedule here. DS mentions that for these tests we can use the correlator platform at JIVE for this test since it doesn't need to

Next milestone is an evaluation of work done in other WPs to see if things are mature enough for integration. MK attended some telecons for WP5 and WP8 an there may some work there that looks promising. This is especially true for the work done on telescope monitoring (and remote control) done at Wetzel in WP5. The meeting in Aveiro, Portugal in May should provide a good opportunity to do the evaluation since many people from other workpackages will be there.

  3. Poznan Meeting (Apr 19-20)
  4. Aveiro Meeting (May 24-26)
  5. Design document for task 2

DS asks whether the “triggered observations” mode is something

  6. Any other business

MK asks everbody to send their input (one or two paragraphs) describing the activity at their institute for the quarterly report.

Meeting April 19-20, 2011

Presentations

Institute Title Link
JIVE NEXPReS PDF
PSNC Introduction to Fabric PPT
PSNC New WFMS PPT
PSNC NEXPReS Use Cases PPT
PSNC External Systems PPT
PSNC Distributed Correlation-FABRIC PPT
PSNC NEXPReS Liferay PPT
nexpres/nexpres_wp7.txt · Last modified: 2011/05/06 09:48 by 127.0.0.1