JIVE Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Support Tools ======

=====Personal Computers=====
**schedgaps.py** checks the 'early' times in a sum file to make sure gap length is > 11 s every 15 min.
<code>
usage: schedgaps.py [-h] [-e EARLY] sum_file

List gaps for each telescope in a SCHED keyin file. Note, stations with
continuous cal will be reported but have an asterix in front of them

positional arguments:
  sum_file              the .sum file to read. Must have early as a 'bottom'
                        sumitem or by itself (last)

optional arguments:
  -h, --help            show this help message and exit
  -e EARLY, --early EARLY
                        seconds antenna must be on source before scan start to
                        count as sufficient for tsys. Defaults to 11
</code>

=====CCSBETA=====
**make_lis** - script that makes a lis file without having to use showlog:
<code>
usage: make_lis [-h] -e EXPERIMENT [-p PROFILE] [-m MPCSOURCE] [-s SAVE]

Create LIS files

optional arguments:
  -h, --help            show this help message and exit
  -e EXPERIMENT, --experiment EXPERIMENT
  -p PROFILE, --profile PROFILE
                        Profile to filter for. Default: use all profiles.
  -m MPCSOURCE, --mpcsource MPCSOURCE
                        Multi phase center source to include calibrators.
                        Default: all sources.
  -s SAVE, --save SAVE  Save the LIS file to file. Default: <experiment>.lis
                        in current directory. For multi phase center
                        experiments, a file will be created for each source,
                        where '{key}' will be replaced by the source name. In
                        this case the default is: <experiment>-{key}.lis.
</code>

**schedule_fringe_test** - script to schedule an automatic fringe test.

Inside the experiment name (/ccs/expr/<expname>).
<code>
log2vex.pl
    - tapelog evlbi
    - EOP
    - Sched from Skdfile
    - Clock
    - Compile vexfile


vex2db <exp>.vexfile

schedule_fringe_test [-h] -e EXPERIMENT -s STATIONS -m EMAIL [-n SCANNAME] [-t STARTTIME] [-d DURATION]

Either specify a scan or startime plus duration.

Arguments:
    -h              Show the help
    -e EXPERIMENT   Experiment name
    -s STATIONS     List with all stations (e.g. EfJbO8)
    -m EMAIL        Email where the results will be submitted
    -n SCANNAME     Name of the scan to obtain (e.g. No0040)
    -t STARTTIME    Starttime of the fringe test (e.g. 2017y290d12h42m00s)
    -d DURATION     Duration of the fringe test in seconds (default 4 s)
</code>


**recorrelate_fringe_test** - to correlate again a fringe test scheduled with schedule_fringe_test.

<code>
recorrelate_fringe_test -e EXPERIMENT -n SCANNAME
</code>





=====EEE(2)=====
**flag_weights.py** - flag all visibilities under weight threshold x:
<code>

usage: flag_weights.py [-h] <measurement set> <weight threshold>

Flag visibilities with weights below the provided threshold.

positional arguments:
  msdata         Measurement set containing the data to be corrected.
  threshold      Visibilities with a weight below this value will be flagged.
                 Must be positive.

optional arguments:
  -h, --help     show this help message and exit
  --version      show program's version number and exit
  -v, --verbose  Only checks the visibilities to flag (do not flag the data).
</code>

**fringeSelect.py** - script to help find you a fringe finder. Queries astrogeo rfc:
<code>
usage  fringeSelect.py [-h] [-b BAND] [-e MIN_EL] [-f MIN_FLUX]
                       timeStart duration stations [stations ...]

Provide some options for sources to use as fringe finders. Based on RfC
catalogue.

positional arguments:
  timeStart             The start date/time of your experiment. Format
                        ='DD/MM/YYYY HH:MM'
  duration              The duration of your experiment (in hours)
  stations              Space delimited list of stations

optional arguments:
  -h, --help            show this help message and exit
  -b BAND, --band BAND  The band you are searching for, one of l, s, c, m, x,
                        u, k, q. If not specified will default to using C-band
                        fluxes as priority.
  -e MIN_EL, --min-el MIN_EL
                        The minimum elevation to consider a source being 'up'.
                        Defaults to 20.
  -f MIN_FLUX, --min-flux MIN_FLUX
                        The mimimum flux density of sources to consider.
                        Defaults to 1.0 Jy
</code>

**multiphase.py** - script to setup a shell script (or multiple lis files) for multiple phase centre experiments. Should no longer be required with new showlog/make_lis:
<code>
usage: multiphase.py [-h] [-b] [-c] [-s]
                     experiment pcBaseName phaseCenters [phaseCenters ...]

This script will attempt to make a sane sfx2ms shell script file for multi-
phase center experiments. Expects a <exp>.vxsm and a base <exp>_sfxc2ms.sh
file in the working directory.

positional arguments:
  experiment          Experiment name
  pcBaseName          Base phase center name
  phaseCenters        list of phase centers (space delimited)

optional arguments:
  -h, --help          show this help message and exit
  -b, --include-base  Include the base phase center in the output?
                      Default=False
  -c, --include-cal   Include the calibrator sources in each measurement set.
                      Defaults to False, which means calibrator sources will
                      be included in the first measurement set only
  -s, --separate-lis  Writes a lis file for each phase centre. Can then be run
                      with j2ms2 -v lisfile
</code>

**parallel - multiple phase centre speedups** - parallel is a linux utility to run tasks in parallel.
Here's an example for running flag_weights.py, tConvert for multiple phase centres in eg078f. Similar can be done for j2ms2 etc:

flagging:
''parallel --eta -j 10 flag_weights.py {} 0.7 ::: *.ms_HD*''

tConvert:
Make a wrapper script:
<code>
#!/bin/bash
####tConv-wrap.sh
#!/bin/bash
tConvert $1 $2
echo "${1#eg078f.ms_*} -> $2" >> pc-idi.txt
mv $1 msdone/
</code>
Run it in parallel:
''parallel --dry-run --eta --xapply ./tConv-wrap.sh {1} eg078f_{2}_1.IDI  ::: eg078f.ms* ::: {2..699}''

**parsePIletter.py** - reads a piletter and adds station comments therein into the feedback database. Should be run for every experiment.
<code>
usage: parsePIletter.py [-h] [-s SESSION] PIfile

This script will attempt to read a piletter and insert the station specific
comments into the feedback DB.

positional arguments:
  PIfile                PI letter to parse

optional arguments:
  -h, --help            show this help message and exit
  -s SESSION, --session SESSION
                        The session name in format xxxYY (e.g. oct17)
</code>

**ysfocus.py** - Sets the MOUNT field in the ANTENNA table correctly for Yebes/Hobart. 
<code>
usage: ysfocus.py [-h] <measurement set>

Change the MOUNT field in the ANTENNA table for Yebes to 'ALT-AZ-NASMYTH-RH'.
It allows tConvert to put MNTSTA=4 into the FITS AN table to handle the
parallactic angle correction for Ys's Nasmyth focus correctly. Also, changes
Hobart X_YEW to X-YEW expected by tConvert (MNTSTA=3).

positional arguments:
  msdata      Measurement Set containing the data to be corrected.

optional arguments:
  -h, --help  show this help message and exit
  --version   show program's version number and exit
</code>


**pipelet.py** - Parses the pipe letter (exp.pipelet) to be sent to the PI after pipelining the data.
<code>
usage: pipelet.py [-h] [-v] [-o OUTPUT] [-c CREDENTIALS] [-u USERNAME] [-p PASSWORD]
                   experiment jss

Creates a .pipelet file in the current directory. This letter is the content that must
be sent to the PI after pipelining an experiment. It contains information about the
pipeline output and the credentials to access the data.
It takes the credentials from the username_password.auth file that should be placed in
the current directory (otherwise specify its file, or the username and password as parameters).
The user must provide the following information:
- The experiment name (case insensitive).
- Who are you (the Support Scientist). Type your surname.

positional arguments:
  experiment            Experiment name.
  jss                   JIVE Support Scientist doing the post-processing (your surname).

optional arguments:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit
  -o OUTPUT, --output OUTPUT
                        Output directory where the file {experiment}.pipelet will be store
                        (by default the current directory)
  -c CREDENTIALS, --credentials CREDENTIALS
                        Auth file containing the username and password in its name (required
                        if *.auth is not in current directory).
  -u USERNAME, --username USERNAME
                        Username to access the data (required if no credential file exists)
  -p PASSWORD, --password PASSWORD
                        Password to access the data (required if no credential file exists)

</code>

=====Pipe (jop83)=====

**ampcal.sh** - Run in '$OUT/expr/', adds any .ampcal information to the amplitude calibration database.

**antab_check.py** - tool to do some basic sanity checking of antab information:
Do you have antabs/uvflg for every station, INDEX line is the right length.
Also plots the tsys if requested.
<code>
usage: antab_check.py [-h] [-p] [-s STATION] [-i] [-l] [-f]

Some basic sanity checking for ANTAB files (all uvflgs/antabfs present in
current directory, does INDEX line appear to have right number of
definitions?) Also will plot the tsys values if requested.

optional arguments:
  -h, --help            show this help message and exit
  -p, --plot            Shall we plot antab file(s)? Defaults to no
  -s STATION, --station STATION
                        Choose to only plot one station (default is to plot
                        all). Use 2 letter code (e.g. Ef). Auto adds -p,-i if
                        not given
  -i, --no-pause        Skips the input pauses. Defaults to no
  -l, --no-flag-line    Turns off the line plotted at -2
  -f, --no-flagged-data
                        Does not plot flagged data (data at -2)
</code>

**antabfs_interpolate.py** - Modifies an ANTAB file to write more Tsys values when the time separation between them is too long.
<code>
usage: antabfs_interpolate.py [-h] [-v] [-p] [-o OUTPUTFILE] [-tini STARTIME] [-tend ENDTIME] antabfile int


Given an ANTAB file, it creates a new one with more Tsys values that are interpolated from the
given ones. This will fill gaps when the time separation between Tsys measurements is too long
(and e.g. AIPS starts flagging scans because there is not Tsys information for them).

IMPORTANT CONSIDERATIONS:
Note that it interpolates data with a smooth function, avoiding outliers or zero values.
However, it does not consider scan boundaries, so if Tsys are recorded in different sources
it can introduce biases. Therefore, it assumes that the Tsys should not change quickly
(e.g. no change of sources).

positional arguments:
  antabfile             The antabfs file to be read.
  int                   The interval (in seconds) between the final Tsys measurements

optional arguments:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit
  -o OUTPUT, --output OUTPUT
                        Output filename. By default same as antabfile.
  -p, --plot            Produce plots (per column) with the original values and the interpolation.
  -tini TINI            Starttime of the Tsys measurements. In case you want to modify it from the original file. It will extrapolate the earliest Tsys original values. The format must be as DOY/HH:MM:SS.
  -tend TEND            Ending time of the Tsys measurements. In case you want to modify it from the original file. It will extrapolate the latest Tsys original values. The format must be as DOY/HH:MM:SS.
</code>


**antabfs_nominal.py** - makes a nominal (based on expected SEFD only) antabfs file for a telescope that did not provide antabfs:
<code>
usage: antabfs_nominal.py [-h] [-v] [-b BAND] [-d DURATION] [-fr FREQRANGE]
                          [-s SEFD] [-i INTERVAL] [-sb SUBBANDS]
                          antenna experiment start

Writes a nominal SEFD ANTAB file. Gain will be set to 1/SEFD, and all Tsys to
1.0. It will overwrite any previous antab file in the current path.
antabfs_nominal.py uses the SEFD information from sefd_values.txt to compute
the nominal values. Creates (or overwrites) a file called
<experiment><antenna>.antabfs, where <experiment> and <antenna> are the input
from the user.

positional arguments:
  antenna               Antenna name (two-letters syntax, except for Jb1 Jb2
                        Ro7 Ro3)
  experiment            Experiment name
  start                 Start time (DOY/HH:MM, YYYY/DOY/HH:MM or
                        YYYY/MM/DD/HH:MM)

optional arguments:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit
  -b BAND, --band BAND  Observed band (in cm). REQUIRED unless SEFD provided
  -d DURATION, --duration DURATION
                        Duration of the experiment (in hours). Default: 24 h
  -fr FREQRANGE, --freqrange FREQRANGE
                        Frequency range where the ANTAB is applicable (lower
                        and upper limit, in MHz). Default 100,100000 (please,
                        do not use spaces between the numbers).
  -s SEFD, --sefd SEFD  SEFD to be used (optional). Default values are loaded.
  -i INTERVAL, --interval INTERVAL
                        Interval between Tsys measurements (in min). Default:
                        0.5
  -sb SUBBANDS, --subbands SUBBANDS
                        Number of subbands in the experiment. Default: 8 =
                        L1|R1 L2|R2 ... L8|R8
</code>

**sftpvlbeer** - for the very lazy (y/w - Jay)

''alias sftpvlbeer="sftp evn@vlbeer.ira.inaf.it"''

**uvflgall.csh** - makes a uvflgfs file from any .log files in the current directory. Also changes flagr line to make uvflg.pl happier.

**comment_tasav_file.py** - tool to create the comment file to be located in the pipeline output before running the feedback.pl script (and after having run the pipeline):

<code>
usage: comment_tasav_file.py [-h] [-v] [-oc OUTPUT_COMMENT] [-ot OUTPUT_TASAV]
                       experiment

Creates a .comment file (in $OUT/exp directory) and a .tasav.txt file (in
$IN/exp). Given the default templates, it customizes them to include the basic
data from the given experiment. The script takes the information from
different locations (ccsbeta, pipe $IN and $OUT directories). The EVN Pipeline
must have been run before calling this script.

positional arguments:
  experiment            Experiment name. Note: in case of multiple passes
                        write {exp}_number (e.g. ev100_1

optional arguments:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit
  -oc OUTPUT_COMMENT, --output_comment OUTPUT_COMMENT
                        Output directory where the file {experiment}.comment
                        will be saved (by default in $OUT/{experiment})
  -ot OUTPUT_TASAV, --output_tasav OUTPUT_TASAV
                        Output directory where the file {experiment}.tasav.txt
                        will be saved (by default in $IN/{experiment})

</code>
All additional data are taken from different files: the script access ccsbeta to get the epoch of the observation (from the MASTER_PROJECT.LIS file), and it also reads the pipeline input file and the pipeline output .SCAN and .DTSUM files. The default output will be in $OUT/{exp}/{exp}.comment, unless other location is indicated by the -o option.