# Guide for FTP tests during NMEs Author: Benito Marcote Date: 3 May 2019 Version: 4.4 ## Before the NME start ### Make vaxfile In case Bob has not made it yet: Log in to **ccs**: ``` ssh -X jops@ccs cd /ccs/expr/ mkdir # if necessary cd ``` If the experiment has not been introduced into the database yet: ```bash cd /ccs/var/log2vex ``` Update the `MASTER_PROJECTS.LIS` file including the experiment name and the date of the observation. The necessary directories (e.g. the ones in `logexp_date` should be automatically populated every hour). If the *.vax* does not exist, the create it: ``` log2vex ``` Then follow the next steps: - Select Experiment. - EOP (Earth orientation parameters). - TapeLog: prefer e-vlbi (for NMEs). - Sched: from Skdfile. - Clock. - Show Clock graph: do all stations have a clock value? Is the fit OK? - Compile Vexfile (press BITSTREAMS) - Print/Save: default name. And you need to rename the file: ``` mv .vexfile .vax ``` ### Check vaxfile You should check at the end of the *.vax* file the clock section. If some values are missed you should ask the station friend about it during the Skype session and update the clocks. ## Step-by-step guide for the SFXC Log in to **head.sfxc** (where the SFXC resides). There is a copy in **tail.sfxc** in case *head* is not available. ``` ssh -X jops@head.sfxc cd ~/sfxc/ftp// # e.g. 2017_may mkdir cd scp -p jops@ccs:/ccs/expr/EXP/.vax . ``` Now you can run *ftptest.py* to create the necessary folder structure for the NME: ``` ftptest.py -h ..................... usage: ftptest.py [-h] [-a ANT] [-e SETANT] [-r REFANT] [-c CHANNELS] [-f FFT] [-s SCANS] [-d] [-g] vex Generate shit for an ftp fringe test. positional arguments: vex vex-file name optional arguments: -h, --help show this help message and exit -a ANT, --ant ANT antenna 2-letter codes -e SETANT, --setant SETANT Reference setup antenna 2-letter code -r REFANT, --refant REFANT Reference antenna 2-letter code -c CHANNELS, --channels CHANNELS channels to use, counting from 1 -f FFT, --fft FFT number of fft points to use -s SCANS, --scans SCANS ftp-scans to generate stuff for -d, --dummy make a dummy list of file locations -g, --gabri create folders according to Gabriele's taste ..................... ``` If you run it as ``` ftptest.py -d -g .vax ``` it creates all the directories and dummy files for all the ftp-tests that are present in the vaxfile. If you don't use the *-d* option the script looks for real data on *sfxc-h1*. Remember to modify the *.scan##.ctrl* file according to the data available for SFXC. Or run *ftptest.py* without the *-d* option again just before running SFXC. In case you decide to prepare everything manually: ``` cd ~/sfxc/ftp/// mkdir ctrl mkdir html mkdir output ``` and inside each directory create a directory for each scan that will be part of the FTP-tests (check the *.vax* file). Copy the *.vax* file into *ctrl/scan##/* and copy a ctrl file from a previous NME and edit it according to the FTP-test. ## Preparing the website for the NME Go to **dop288** to prepare the *ftp.html* file by adding the new NME. ``` ssh -X jops@dop288 cd /www/evlbi.org/tog/ftp_fringes/ vim ftp.html # or your favourite text editor ``` Add the path for the experiment, save it, and close it. ``` mkdir cd cp ..//index.html . vim index.html # Modify it accordingly to the NME, save it mkdir scan## # Create a directory for each scan ``` Otherwise you can also run *ftptest_report.py* to make a draft of *index.html*. ## Monitor the arriving data All participating stations send data to JIVE. ``` ssh jops@fringetest.jive.nl cd /ftp/ftpdata/ ls -ltr **00##* # where ## is the number of the scan watch -n 2 ls -ltr **00##* # To run auto. the previous command every 2 s ``` check here the right name of the data files and which has arrived. You can also use some features of *ftptest.py* right before running SFXC to include only the files that have arrived. ## Start chat session Open *Mattermost* and create a new channel for this NME with its name as channel name. Announce it in the general channel and invite all the interested people. Provide the times of the FTP tests in the chat window. Below there is a list of people (*Mattermost* names) for each station (**NOTE:** this list is not complete and some users are not registered here): - **Ef**: Effelsberg Friend (*@ef.operator*) Uwe Bach (*@ubach*) - **Wb**: Richard Blaauw (*@wb.richard_blaauw*) Henk Mulder (*wb.mulder*) Paul Vandijk (*@wb.paul_vandijk*) Jurjen Sluman (*@wb.sluman*) - **Jb**: Jodrell Bank VLBI Room (*@jbo.vlbi*) Eskil Varenius (*@jbo.eskil*) Alastair Gunn (*@alastair.gunn*) Leslie Parry (*@jbo.leslie_parry*) - **On**: Jun Yang (*@jun-yang*) - **Mc**: VLBI Operator (*@mc.operator*) Giuseppe Maccaferri (*@mc.beppe*) Andrea Maccaferri (*@a.maccaferri*) - **Nt**: Noto Friend (*@vlbifriend, @noto_vlbi*) Pietro Cassaro (*@pietro_casaro*) - **Tr**: Torun Telescope Operator (*@torun_operator*) - **Sh/T6**: VLBI Friend (*@t6_sh*) Bo Xia (*@t6_bo*) - **KVAZARs**: VLBI Friend (*@agm*) - **Hh**: HartRAO Friend (*@hh.vlbif*) - **Ys**: VLBI Friend (*@ys.oper*) Javier González García (*@ys.javi*) Pablo de Vicente (*@pablo.devicente*) - **Ur**: ??? - **Sr**: Control Room SRT (*@srt-vlbi-oper*) Carlo Migoni (*@carlo.migoni*) Gabriele Surcis (*@surcis_srt*) - **Ar**: (*@ao.evn*) - **Km**: (*@kunming40*) - **Ir**: Vladislavs Bezrukovs (*@ir.vladislavsbezrukovs*) Artis Aberfelds (*@artis.aberfelds*) Janis Steinbergs (*@steinbergs*) Arturs (*@ir.arturs*) Marcis Bleiders (*@ir.marcisb*) - **Mh**: Juha Kallunki (*@kallunki*) - **KVN**: AOC KVN (*@kvn*) - **Santa Maria**: (*@raegsmar*) - **Ghana**: (*@jchibueze*) ## Correlation with SFXC Once the NME starts and you have data from the stations you can run SFXC in *head.sfxc*: ``` cd ~/sfxc/ftp///ctrl/scan## run_sfxc.py .vax .scan##.ctrl ``` It generates all the html that are in `~/sfxc/ftp///html/scan##/` You can look at the *index.html* file by running Firefox. If everything looks OK you can copy all the files to **dop288**: ``` scp -r jops@head.sfxc:/home/jops/sfxc/ftp///html/scan##/\* /www/evlbi.org/tog/ftp_fringes//scan## ``` ## After the NME You should send an email to *EVNtech@jb.man.ac.uk* (with copy to Bob, *jops@jive.eu*) with the subject: *EVNtech: N17C1 FTP Fringe Test Results*. In the body of the email you have to write something like: ```text Version: 1 Summary: Ftp data arrived from: Ef, Wb, Jb, On, Mc, Nt, Tr, Ys, Sh, T6, Td, Yd, Md, Sd. Selected results can be found here: http://www.evlbi.org/tog/ftp_fringes/N14M1 Legend: Y = Performed as expected N = Did not perform as expected ? = No feedback available from station Blank = Not applicable or waiting for further tests F14M1, Freq= 6662.52-6674.52 MHz; Mode=1024 ch-128 Mbps-16x2MHz-2sec; obs date=01/03/2014 |Ef|Jb|Wb|On|Mc|Nt|Tr |Ys|Sh|T6|Td|Yd|Md |Sd| Observed |Y |Y |Y |Y |Y |Y |Y |Y |Y |Y |Y |Y |Y |Y | Frng.(ftp) |Y |Y |Y |Y |Y |Y |Y |Y |Y |N |Y |Y |Y |Y | Record. OK |Y |Y |Y |Y |Y |Y |Y |Y |Y |Y |Y |Y |Y |Y | Crs.pols OK|Y |Y |Y |Y |Y |Y |Y |Y |Y |Y |Y |Y |Y |Y | Check VCs | | | | | | |7,8| | | | | |1,2,3,4,5| | See notes | | | | | | |a | | | |b |c |d |e | Notes: general: the calibrator used in scan02 (3C48) is much weaker than calibrator used in scan21 and scan59 (DA193). This is the reason of the lack of fringes for scan02. a) Tr: no fringes in BBCs 7 and 8. b) Tr also sent DBBC data designated as 'Td'. c) Ys also sent DBBC data designated as 'Yd'. d) Mc also sent DBBC data designated as 'Md'. Md shows weaker fringes than Mc, this could be the reason of no fringes in DBBCs 1 (LL-USB & LSB), 2 (RR-LSB), 3(LL-USB & LSB), 4(RR-LSB), 5 (LL-LSB) e) T6 also sent DBBC data designated as 'Sd'. questions/comments - please email surcis@jive.nl Regards, -------------------------- ``` ## Automatic Fringe Test In ccs in the directory where *.vax* is you should type: ``` ssh -Y jops@ccs cd /ccs/expr/ # If needed: log2vex.pl - tapelog evlbi # no recommended anymore - EOP - Sched from Skdfile - Clock - Compile vexfile vex2db .vexfile schedule_fringe_test -h # if you need help schedule_fringe_test -e EXP -n No00## -s EfJbO8 -m email -t START -d DURATION | | | | name of the experiment | | | | | | scan you want | | to correlate | | | | list of station you | want to correlate | (don't use spaces) | (no vdif supported) | | your email address ``` example: `schedule_fringe_test -e N16M1 -n No0002 -s EfJbO8 -m surcis@jive.eu` You can specify either the scan number or the time from when you want the data in vex format (e.g. 2017y290d12h42m00s). The data are in `ssh -X jops@sfxc.h0.sfxc` in `/scratch/sfxc/fringetest//` if you wish to recorrelate, type in ccs ``` recorrelate_fringe_test -e name.experiment -n No0000 | | name of the experiment | | scan you want to recorrelate recorrelate_fringe_test -e N16M1 -n No0002 # for example ``` In case of questions contact eBob (eldering@jive.eu) ## Clock Search fit Some times the clocks for the stations are not accurately set. In this cases you may want to run a clock search fit (there may be a fringe but with a large clock offset). To do that, first you need to modify the `{exp}.scan##.ctrl` file to increase the number of channels (e.g. 4096 or 8192), and decrease the integration time to a small value (e.g. 0.2 or 0.5 s). In this way, we will be searching in a big window with enough temporal points to fit the rates. Then, run the correlation as usual (with `run_sfxc.py`). And run the simple clock search fit: ``` simple_fit.py -f ../../output/scan##.cor {exp}.vax {ref_station} ``` The output will contain the delays, rates, total SNR, and weights for each telescope.