JIVE Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Deliverables ======

^ D#      ^ Title   ^ Link ^ Date ^
| D7.1    | Workflow Manager requirements from telescope operators (OSO) | {{:nexpres:WP7_D7.1_OSO_2010-11-10.doc|DOC}} | 2010/11/10 |
| D7.2    | Workflow Manager requirements from correlator operators (JIVE) | {{:nexpres:WP7_D7.2_JIVE_2011-01-13.pdf|PDF}} | 2011/01/13 |

====== Documents =====

^ Institute ^ Title   ^ Link ^
| OSO | Workflow Manager requirements from telescope operators | {{:nexpres:WP7_D7.1_OSO_2010-11-10.doc|DOC}} |
| PSNC | Workflow Manager framework evaluation | {{:nexpres:quarter_report_psnc_4_2010.pdf|PDF}} |
| JIVE | Workflow Manager requirements from correlator operators | {{:nexpres:WP7_D7.2_JIVE_2011-01-13.pdf|PDF}} |
| OSO | NEXPReS WP7 Task 2 Design document | {{:nexpres:wp7_task_2_design_document_v1.1.doc|DOC}} |

====== Meetings & Telecons ======

===== Telecon August 18, 2010 =====

14:00-15:00 CEST (12:00-13:00 UTC)

==== Agenda ====

  - Project status
  - Reporting
  - Distribution of funds
  - Upcoming deliverables & milestones
  - Agenda for Manchester meeting
  - Future telecons
  - Any other business

==== Minutes ====


Present:

Mark Kettenis (MK), Aard Keimpema, Arpad Szomoru, Charles Yun (TCY)
Simon Casey, Michael Lindqvist
Dominik Stoklosa, Tomasz Rajtar, Katarzyna Bylec
Ivars Shmeld


Agenda

   1. Project status

Not much to report.  MK says the work on the software correlator at
JIVE is ongoing; a preliminary implementation of VDIF support is
already integrated.

   2. Reporting

There will be monthly reports to the EU.  MK will collect input for
the monthly status report and make it available to the project
manager.  TCY says the status reports will live on the WiKi; a
template for the report for August is already there.


   3. Distribution of funds

*Action*

Please indicate the expected spending (man-months funded from the
project and hardware) for the first year of the project to the Work
Package leader, such that the Project Manager can make a proposal for
the distribution of the first round of EC funding for the board
meeting.

   4. Upcoming deliverables & milestones

First deliverables and milestones are the requirement documents to be
written by JIVE and OSO.  The documents will explain how operations at
the correlator and telescope will work, and what elements of the
process can be automated.  We should probably aim for something like
10 pages each.  Simon Casey will be working on the document from OSO.

PSNC will start work on workflow manager.  Dominik and Ivars will have
contact on how VENT can collaborate best on the computing side of
things.

   5. Agenda for Manchester meeting

The plan is to start with short 10-20 minute presentations from JIVE,
OSO and PSNC about the plans/ideas on what they're working on.  Some
discussion about the technology to be used (WebServices, tools,
languages) seems desirable to prevent interoperability issues.  DS
likes to have a discussion/brainstorm about how to set up the
distributed correlation.  MK indicates that there should be enough
time for that and will put it on the agenda.

*Action*

MK, DS and ML to create short presentations for Manchester meeting.

   6. Future telecons

Mark proposes telecons every month/two months, which seems reasonable
to all participants.  The telecon facilities used by JIVE are
available for this purpose.  Dominik suggests to look at EVO, which is
used regularly for videoconferencing at PSNC.  Arpad mentiones that
his experiences with EVO aren't great.  Next telecon will probably be
scheduled somewhere in October.

   7. Any other business


===== Meeting September 20, 2010, Manchester =====

09:00-12:00 BST (08:00-11:00 UTC)

Profs meeting room 3.110

Jodrell Bank Centre for Astrophysics\\
The University of Manchester\\
Alan Turing Building,\\
Manchester,\\
M13 9PL,\\
United Kingdom\\
Phone: +44 (0)1477 571321\\
Fax +44 (0)161 275 4247

(same building as the board meeting, different room)

==== Agenda ====

^ start ^ lead ^ description ^ slides ^
| 09:00 | MK | Introduction | |
| 09:10 | MK | Activities at JIVE: SFXC, correlator viewpoint | {{:nexpres:wp7_kickoff_kettenis.pdf|PDF}} |
| 09:30 | DS | Activities at PSNC: Workflow manager & broker | {{:nexpres:WP7-Fabric_results.ppt|PPT}} |
| 09:50 | ML | Activities at OSO: telescope viewpoint, science | {{:nexpres:OSO_telescope_requirements.ppt|PPT}} |
| 10:15 |    | Coffee | |
| 10:30 |    | Discussion about global/distributed correlation | |
| 11:15 |    | Discussion about technical issues: what tools/languages should we use to avoid interoperability issues, etc. etc. | |
| 12:00 |    | Lunch | |

==== Minutes ====

Present:

Mark Kettenis (MK), Dominik Stoklosa (DS), Michael Lindqvist (ML),
Ivars Smelds (IS), John Conway (JC, partly)

After the presentations, discussion starts on the workflow manager.
MK suggests that that it needs to provide at least three
"perspectives":

  * PI (scientist)
    * submit schedule (VEX file)
    * correlation parameters
    * get notifications

  * Telescope Operator
    * accept schedule
    * get monitoring feedback

  * Correlator Operator
    * assign resources
    * get monitoring feedback

DS says that the workflow manager for NEXPReS will be web-based
instead of a JAVA (web)application.

DS also says that instead of full-blown SOAP PSNC is looking at a much
lighter weight JSON-based RPC to communicate with VLBI components.
That should make interoperability much less of a problem.

Authorization and Authentication is discussed briefly.  There clearly
is a need to do this "properly" for NEXPReS.  Some sort of design
document may be desireable (but this is not a deliverable).

After some discussion MK proposes that the starting point of a
workflow should be a schedule (provided as a VEX file) submitted by
the PI or perhaps the correlator operator.

JC raises the issue of observations of transients triggered by, for
example, LOFAR.  He suggests that such an observation should probably
use a standard setup/mode (say 6cm, 1Gbps) with a standard set of
telescopes.  Only the sources and times in the schedule would be
different.  MK and ML agree that this would be a sensible approach.

DS starts the discussion about global correlation by stating that
streaming data directly into cluster nodes may not be feasable.  MK
says that one of the lessons learned from EXPReS was that things slow
down quite drastically if uncorrelated data ever hits disk.  DS says
that PSNC now has some high-performance storage that is directly
accessable by cluster nodes which may be used as a staging area.  MK
notes that other sites probably won't have such high-performance
storage available.

===== Telecon October 12, 2010 =====

14:00-15:00 CEST (12:00-13:00 UTC) 

==== Agenda ====

  - Project status
  - Upcoming deliverables & milestones
  - Telescope Feedback
    * Can we use the work done at Wetzell (TUM)?
    * See roadmap: {{nexpres: 20101004RoadmapWP5Task3.doc|DOC}}
  - Global Correlation
    * Stream directly into cluster or do we need a staging area?  Or both?
    * Direct: {{nexpres:global-direct.pdf|PDF}}
    * Staging: {{nexpres:global-storage.pdf|PDF}}
  - Any other business

==== Minutes ====

Present:
Present Mark Kettenis (MK), Aard Keimpema,
Dominik Stoklosa (DS), Kasia Bylec, Tomasz Rajtar,
Alexander Neidhardt (AN), Martin Ettl (ME),
Simon Casey (SC),
Ivars Shmeld (IS), Normunds Jekabsons (NJ), ...


  1. Project Status

MK mentions that our EC officer has indicated that from now on
reporting will be done per quarter.

  3. Telescope Feedback

AN starts with a short introdcution about the work on stations
monitoring & remote control that will be done by TUM, MPG & JIVE in
WP5.

MK starts that full remote telescope control is probably not needed in
WP7.  AN replies that full remote control isn't the real goal of their
work and that various levels of monitoring and remote interaction with
telescope control will be implemented.

DS asks whether the user interface is web-based or a standalone
application.  AN replies that they will provide a wxWidgets based GUI
that is a standalone application, but that there is a clear seperation
between layers that allows access to the functionality through RPC
calls.

DS asks when the work that will be done in WP5, Task3 is ready.  AN
replies that a basic system is already working and running, and will
be made available before the end of the year.  Integration in the NASA
field system will hopefully happen in may 2011.

MK asks if user/password is the only scheme by which Authorization and
Authentication is going to be implemented.  This might pose a problem
for services that run as part of the WP7 correlation platforms.
Embedding passwords in applications is not very secure.  DS agrees
with this.  AN acknowledges this and agrees that some further thinking
is probably needed.  Grid-like certificates and ssh-based schemes are
discussed shortly.  DS mentions that it is possible to have so called
Robot Certificates for services in addition to personal certificates.
NJ suggests the use of OpenVPN to create a secure network.  AN replies
that this is probably not necessary since they use

SC says that a 2nd field system PC exists in Onsala for dBBC tests.
That machine could possibly be used for testing.

MK thinks that if Onsola is able to participate in testing, tapping
into the system devloped at Wetzell for telescope monitoring
defenitely looks like a very good option.

  4. Global Correlation

MK starts the discussion of the possible setups as illustrated by the
diagrams.  Both TCP-based datatransfer (as implemented in SFXC) and
UDP-base (VDIF/VTP) datatransfer are discussed.  MK mentioned that
especially the UDP-based transfer may be problematic if data is to be
streamed directly to cluster nodes.  DS confirms this, and indicates
that using intermediate storage, directly accessable from the cluster
nodes) (as illustrated by the second diagram) may be necessary.  Such
storage, providing 6 GB/s of access bandwidth is being installed at
PSNC.

MK indicates that other cluster may not have storage that is suitable
for this.  In particular the cluster of JIVE doesn't have this.  But
at JIVE direct streaming will be an option.  DS suggests that it is
probably necessary to support both setup.  MK agrees.
MK points out that it will be necessary to run some data transfer
software on the storage nodes that face the outside world.  DS thinks
that this will be possible, but has to verify this.

DS suggests that it may be a good idea if changing the destination of
individual subbands while the correlation is running would be a good
idea.  MK agrees and will check if such a capability is available in
VDIF implementations that are currently being designed.

Ventspils will prepare an overview of the computational infrastructure
they can provide.  They will also try to install the software
correlator and test it.  They will visit PSNC when workflow manager
design is a bit more solid and decide on division of labour at that
point, potentially in the 2nd half of November.


===== Telecon March 29, 2011 =====

14:00-15:00 CEST (12:00-13:00 UTC)


==== Agenda ====

  - Project status
  - Upcoming deliverables & milestones
  - Poznan Meeting (Apr 19-20)
  - Aveiro Meeting (May 24-26)
  - Design document for task 2: {{:nexpres:wp7_task_2_design_document_v1.1.doc|DOC}}
  - Any other business

==== Minutes ====


Present:

Mark Kettenis (MK), Aard Keimpema, Simon Casey
Dominik Stoklosa (DS), ...
Ivars Shmeld, Normunds Jekabson

    1. Project Status

    2. Upcoming deliverables & milestones

Next deliverable is a demonstration of e-VLBI with the SFXC correlator before the end of the year.  Given that the e-VLBI capabiliy is under active development and the first preliminary tests of the functionality have been successful, we should be on schedule here.  DS mentions that for these tests we can use the correlator platform at JIVE for this test since it doesn't need to 

Next milestone is an evaluation of work done in other WPs to see if things are mature enough for integration.  MK attended some telecons for WP5 and WP8 an there may some work there that looks promising.  This is especially true for the work done on telescope monitoring (and remote control) done at Wetzel in WP5.  The meeting in Aveiro, Portugal in May should provide a good opportunity to do the evaluation since many people from other workpackages will be there.

    3. Poznan Meeting (Apr 19-20)


    4. Aveiro Meeting (May 24-26)

    5. Design document for task 2

DS asks whether the "triggered observations" mode is something 

    6. Any other business

MK asks everbody to send their input (one or two paragraphs) describing the activity at their institute for the quarterly report.

===== Meeting April 19-20, 2011 =====

==== Presentations ====

^ Institute ^ Title   ^ Link ^
| JIVE | NEXPReS | {{:nexpres:poznan2011-kettenis.pdf|PDF}} |
| PSNC | Introduction to Fabric | {{:nexpres:1_Introduction_to_Fabric.pptx|PPT}} |
| PSNC | New WFMS | {{:nexpres:2_new_wfms.pptx|PPT}} |
| PSNC | NEXPReS Use Cases | {{:nexpres:3_nexpres_use_cases.pptx|PPT}} |
| PSNC | External Systems | {{:nexpres:4_external_systems.pptx|PPT}} |
| PSNC | Distributed Correlation-FABRIC | {{:nexpres:5_Distributed_correlation-Fabric.pptx|PPT}} |
| PSNC | NEXPReS Liferay | {{:nexpres:6_nexpres_liferay .pptx|PPT}} |