Update Mark-5 Software There is an updated version of a tar file now at: http://web.haystack.edu/Mark5/Mark5A.tgz. You can download this file with a browser. A copy is at: ftp://web.haystack.edu/dist/mark5/Mark5A.tgz. You can download this file with ftp or with the script Mark5Update. (Be sure that you have the latest version: 2005 May 26 or later.) This script uses wget to download the tar file from Haystack, unzips and untars the tar file, reinstalls the Jungo driver, and recompiles all the Mark-5 programs with various error checks. This is intended to make upgrading easier and less prone to error. After the latest upgrade, a copy of Mark5Update will be in the ~oper/bin directory for next time (but check the date). Shut down all Mark-5 programs, then execute Mark5Update as root. We'll try to keep this tar file updated whenever we make significant changes. You can see the date on this file using just the directory (in time order): http://web.haystack.edu/Mark5/. If your Mark-5 machine is unable to communicate with Haystack, then you may download Mark5A.tgz as noted above, move this tar file into /tmp on your Mark-5 machine, and then, as root, execute the alternative script Mark5Update.dev. (The .dev is important. Be sure that you have the latest version: 2005 May 26 or later.) Note that this tar file contains nothing like a full installation. This update procedure works only on an already functional Mark-5 system. Recent changes, additions, and fixes: ## 2005 May 27 (day 147); With this version, in the response from a 'DTS_id?' query, see "2005y147d13h" (among other things). Please also read the notes below for the previous Mark-5 release. This Mark-5 release is in substantial compliance with version 2.7 of the "Mark 5A Command Set" documentation but with certain extensions. 'DTS_id?' will return "2.7x" for "command set revision," that 'x' is for extension. Important: The scripts Mark5Update and Mark5Update.dev have been updated (to May 26). You might need to 'chmod 0755 Mark5Update' after downloading. You should use these updated versions, else the Jungo driver might fail to install, and your rc.local file might get trashed. If Mark5A works after updating but fails after a reboot, then check /etc/rc.d/rc.local (on RedHat) or /etc/rc.local (on Ubuntu). This version of Mark5A was tested on an Ubuntu-Debian machine with: !OS_rev1? 0 : "Linux version 2.6.10-5-386 ..." and !OS_rev2? 0 : "gcc version 3.5 ... Debian 1:3.3.5-8ubuntu2 ..." and on certain other machines. (But ssopen and sstest don't yet work on Ubuntu.) The Mark-5B parts of Mark5A are "Under Construction." The parsing of commands with arguments in Parse5A.c has been rewritten to correctly handle cases with skipped parameters to be defaulted. Formerly whitespace was needed (e.g., between colons) to make this work. Extra debug printing also has been added to this part of Parse5A.c. ## 2005 May 08 (day 128); With this version, in the response from a 'DTS_id?' query, see "2005y128d09h" (among other things). Please also read the notes below for the previous Mark-5 release. This version incorporates a new Conduant release, sslinux63C.tgz, received April 7. This release contains "ApiVersion 5.21, ApiDateCode Apr 7 2005" and "FirmwareVersion 10.84, FirmDateCode Apr 06 2005" from Conduant. This release adds features and fixes bugs, some of which are noted below. Important: The scripts Mark5Update and Mark5Update.dev have been updated (to May 26), and you should use these updated versions with this release, else the Jungo driver is likely to fail to install and|or your rc.local file may get truncated. You might need to 'chmod 0755 Mark5Update' after downloading. This Mark-5 release is in substantial compliance with version 2.7 of the "Mark 5A Command Set" documentation but with certain extensions as noted below. 'DTS_id?' will return "2.7x" for "command set revision," that 'x' is for extension. This version of Mark5A was tested on a RedHat-9 machine with: !OS_rev1? 0 : "Linux version 2.4.20-28.9 ... and !OS_rev2? 0 : "gcc version 3.2.2 20030222 (Red Hat Linux 3.2.2-5)) ..." and also on an Ubuntu-Debian machine (with a 2.6 kernel) and certain other machines. (ssopen and sstest don't yet work on Ubuntu.) Bug fix: The default file name for 'disk2file=...' at startup (i.e., before 'scan_set=...') and on changing banks is set to the scan label of the first scan, if any. Bug fix: In case no disks are available, 'status?' returns an error only the first time after Mark5A startup. No disks is no longer an error because in2net and net2out can now work without disks. Bug fix: SSErase no longer reports DMS "Error" when there is no error and DMS is about to be changed to "Erased". A new version of XLRRecoverData() from Conduant allows the 'recover=...' command to be augmented as follows: Try 'recover=0' to recover from a power failure or an accident that interrupts recording (as before), try 'recover=1' to recover data after some initial part of them has been overwritten (as before), and try 'recover=2' to recover data that were erased. Debug printing in setLabel() has been improved. Extra debuggery has been added for Mark5B-formated disks. Try 'data_check?'. These Mark5B parts of Mark5A are "Under Construction." Bug fix: 'disk_serial?', 'disk_model?', and 'disk_size?' no longer remember and return information for a disk module that was removed. The four make and update scripts, Mark5Update, Mark5Update.dev, MakeMark5, and MakeMark5.dev were updated to work on either RedHat, Fedora, or Ubuntu distributions. SSErase has been modified to give stats and timing information for msglev less than 2 (formerly less than 1). Printing stats in SSErase with conditioning has been modified to print a line with "-" for each missing or broken disk. Bug fix: Two DEBUG prints from Data_check() are now correct. ## 2005 March 18 (day 077); With this version, in the response from a 'DTS_id?' query, see "2005y077d10h" (among other things). This tar file contains a new complete API from Conduant, sslinux63B.tgz dated Feb 26, which contains an sspxf.bib file with "FirmwareVersion 10.83, FirmDateCode Feb 25" (from 'SS_rev2?'). This Conduant release is intended to fix some bugs as indicated in the following note from Conduant: [This is the] list of firmware changes between 10.79 and 10.83. ... 1. Fixed bug that caused crash when very small amount of data is attempted to be read. 2. Remove requirement that drive must support write cache and read lookahead for drive to be used (required for solid state media). 3. Fixed bug that caused endless loop if all drives are offline. 4. Improved accuracy of GetFifoLength during forking operations. 5. Complete rework of drive error handling. 6. Leave valid directory bit set to prevent flashing ready in VLBI operations during first recording after an erase. Please also read the notes below for the previous Mark-5 release. Important: The scripts Mark5Update and Mark5Update.dev have been updated (to February 28), and you should use these updated versions with this release, else the Jungo driver is likely to fail to install. You may need to 'chmod 0755 Mark5Update' after downloading. This Mark-5 release is in substantial compliance with version 2.7 of the "Mark 5A Command Set" documentation but with certain extensions as noted below. 'DTS_id?' will return "2.7x" for "command set revision"; that 'x' is for extension. This version of Mark5A was tested on a RedHat-9 machine with: !OS_rev1? 0 : "Linux version 2.4.20-28.9 ... and !OS_rev2? 0 : "gcc version 3.2.2 20030222 (Red Hat Linux 3.2.2-5)) ..." and also on certain other machines. Bug fixed: A (scratch) disk module is no longer needed for in2net or net2out. Bug fix: The queries 'in2net?', 'disk2net?', 'net2out?', and 'net2disk?' now again correctly report byte numbers. Bug fix: The intermittent failure to start data transfers in in2net and disk2net seems to have been fixed. Bug fix: An extra (not needed) 'net2disk=close' does not produce a spurious and incorrect scan starting from byte 0 but returns, instead, return code 6 and "Already closed". Bug fix: Disk2net now allows multiple instances of 'disk2net=on...' possibly interspersed with multiple 'scan_set=...'s to give multiple scans or parts of scans to the same socket connection. Depending on the receiving program, this can give multiple scans onto SS disks (with net2disk), into a correlator (with net2out), or into the same OS file (with Net2file). As previously noted, disk2net and net2disk append fill pattern onto each transfer, if necessary, to bring the byte count up to an integer multiple of "workbuf size" in net_protocol, default 131072 bytes. Functions 'scan_check?' and 'data_check?' have been rewritten. Mis-identifying a VLBA-format scan as Mark 4 seems to have been completely eliminated. There are also several additional checks on a VLBA-format frame header to detect and reject headers with certain errors that were previously ignored. This involves checking some of those bits that are expected to be the same across tracks. A candidate VLBA-format frame header that fails such a check is then checked to see whether it is Mark 4. If not, then an error message is returned. Bug fix: 'scan_check?', 'data_check?', and 'track_check?' now decode date and time correctly with data taken on the last day of a leap year. This problem had nothing to do with the operator out partying on New Year's evening. Following is an addition to the documentation under net2out: "inactive" means no socket open---doing nothing, "waiting" means socket open and waiting for a connection, "paused" means socket connected but no data waiting to be read, "active" means connected and data flowing. Note that none of these states are necessarily errors; this depends on context. In particular, "paused" return from a 'net2out?' query is now usual during normal operation because it corresponds to no bytes waiting to be read from the socket. This is an extension to version 2.7 of the documentation. The 'net2out?' query now returns also the number of bytes received and sent to the output board since the last open. This number should increase during normal net2out operation. This is an extension to version 2.7 of the documentation. There is some additional debug printing in in2net, net2out, disk2net, and net2disk to aid in understanding eVLBI transfers. SMARTstate ("Fault", "OK", or "NotSMART") was added as the last parameter (after replaced-block count) on the 'get_stats?' response. This is an extension to version 2.7 of the documentation. Bug fix: With net2out, when the other end of the network connection disconnects, then net2out shuts down and cleans up, so that, in this case, 'net2out=close' is not needed. (But net2disk always needs 'net2disk=close' before data on disk are useable.) Bug fix: In FFHeader(), which serves 'data_check?' and 'scan_check?', the VLBA cross-track check has been relaxed in response to empirical evidence that SSRH and TGBSCD (in Figure 2, page 28 of VLBI document 230.3) are not always the same track-to-track. This notwithstanding that these parameters are labeled "Station ID," "Recorder Parameters," and "DAR Parameters," which should not change track-to-track. Bug fix: The spin loop (consuming CPU cycles) while in2net was connected but off (status "connected") has been eliminated. Bug fix: Doing 'scan_set=;' no longer kills Mark5A. Bug fix: The return from 'file2disk?' in case "active" now agrees with the documentation. ## 2004 December 16 (day 351); With this version, in the response from a 'DTS_id?' query, see "2004y351d" (among other things). This tar file contains a new complete API from Conduant, sslinux63.tgz dated Nov 17, which contains a sspxf.bib file with "FirmwareVersion 10.79, FirmDateCode Nov 09". This bib file from Conduant fixes some bugs and implements automatic bank switching on a full disk during either record or playback. These new features are, however, still being tested. This tar file also contains a new sspxf-1.bib file from Conduant. This bib file supplies "FirmwareVersion 20.07, FirmDateCode Oct 26" as used only in disk-FIFO mode. This disk-FIFO-only mode is still being tested but is tentatively ready for beta testing---notes below. Also as noted below, this tar file has several other new features and bug fixes. Bug fix: 'scan_set=...' allows an offset time of 0. Several bug fixes in SSErase: DMS is now handled correctly on both banks and set to "Error" in case conditioning is interrupted or to "Erased" as appropriate on a normal end. The user directory is now properly cleared, and the check on NumBuses is now correct also in case of fewer than 8 disks. Disk statistics after conditioning now print correctly (with debuggery on) for up to 16 disks. Bug fix: Bank_switch status (on or off) is now kept correctly. (But there is no XLRGetBankMode().) Bug fix: 'track_check?' on 8- and 16-track recordings now reports "D" (instead of "?") for correctly duplicated tracks. Bug fix: 'data_check?' and 'scan_check?' now work with 8- and 16-track recordings made with Mark-III formatters. (These are actually 7- and 14-track recordings.) 'track_check?' works (as before) provided that you select a track that was actually recorded. But a 'track_check?' on a Mark-III-formatted scan shows "-1" for decoded track number, which is not an error (no "?") because Mark-III formatters do not write track numbers into the frame headers. Logging of scan name, experiment name, source name, and station name as acquired from 'record=on:...', 'net2disk=open:...', and 'file2disk=...' has been revised. In the output of the 'record?', 'scan_set?', and 'scan_check?' queries, and also in the stand-alone DirList program, the experiment, station, scan, and source parameters parameters are now separated by underlines (_) rather than spaces as in Version 2.6.3 of the documentation. Underlines (with no spaces) also serve to combine these parameters into a single token, which, with an appended ".mk5" is offered as the default file name to 'disk2file=...'. Also accepted in the 'record=on:...', 'net2disk=open:...', and 'file2disk=...' commands are subfields separated by underlines (_) instead of colons (:), but, in the underlined case, no embeded spaces are allowed. Also the total length of this combined scan label can not exceed 63 chars including the underlines. This can be satisfied by having no more than 15 chars in each subfield as before, but there can now be more than 15 in some subfields provided that the total is 63 or less. In case the parameters are entered with colons (:) as separators, then the parameter order will be interchanged as follows: record = on : field1 : field2 : field3 : field4 would be interchanged to make a scan label of: field2_field3_field1_field4 If, for example: record = on : Scan : Experiment : Station : Source ; would be interchanged to make a scan label of: Experiment_Station_Scan_Source This interchanging is intended only for legacy compatibility; users are urged to type the scan label in the proper order with underlines as separators. This is a revision to Version 2.6.3 of the documentation. The method for searching for scans in 'scan_set=...' has been augmented as follows: The four fields of the scan label are now expected to be scan name, experiment name, station, and aux info. But, for this discussion, these are just four fields now separated by underlines (_) in the scan label. The 'scan_set=...' command searches for a scan using its first parameter as a mask and template as follows: If this first parameter is not all numeric or, if numeric, then does not number an actual scan, and also is not 'inc', 'dec', or 'next', then it is a template used to search for a scan label. If this template contains no underlines (_), then Mark5A searches through the list of scans to find the first match to all or any part of a scan label just as before: This is the legacy behavior. If, instead, this template contains at least one underline, then Mark5A searches for a scan with subfields that start with the parameters in the corresponding subfields in the scan label. An example, perhaps would make this clearer. Consider these four scans as from a DirList: 4 Experiment_Station_Scan_Source 5 Field1_Field2_Field3_Field4 6 FieldA_FieldB_FieldC_FieldD 7 AField_BField_CField_DField Then 'scan_set=stat' (no underlines) would find scan 4 (not case sensitive), 'scan_set=_bf' would find scan 7, 'scan_set=__fi' would find (Field3 of) scan 5, 'scan_set=_BF_CF_DF' would find scan 7, and 'scan_set=eld1_' would return error "6 : No such scan" (and leave the scan_set parameters unchanged) because 'eld1' does not start the first subfield of scan 5. These are revisions to Version 2.6.3 of the documentation. An additional value, 'next', is now allowed as the first argument of 'scan_set=...'. 'Next' causes scan_set to use the last previous (remembered) first argument of this command and to search beginning with the next scan after the last one previously found. This feature could be used, for example, to find all scans with a prescribed source. This is an addition to Version 2.6.3 of the documentation. The queries 'net2disk?' and 'file2disk?', when "inactive," now show the scan number and scan label of the last scan created by the corresponding command, if any. This makes these queries similar to the 'record?' query. But this information is lost on a bank switch or on ending and restarting Mark5A. These are additions to Version 2.6.3 of the documentation. The query 'disk2file?' when "inactive," now shows also the name of the last file written by the corresponding command, 'disk2file=...', if any. But this information is lost on ending and restarting Mark5A. This is an addition to Version 2.6.3 of the documentation. Bug fix: 'record=...' now remembers and, on query, shows the last scan recorded (rather than taking credit for the last scan regardless of whence it came). But this information is lost on a bank switch or on ending and restarting Mark5A. Binding a socket for net2disk or net2out and also in the stand-alone program Net2file now allows reuse of the port address, m5data. This same update has now been made also for the m5drive socket used for command and control. The reuse of socket addresses helps recover from unexpected socket disconnects. Bug fix: A spurious warning message on 'in2net=disconnect' was removed. Bug fix: 'get_stats?' skips over non-existent disks (except disk 0) without making a fuss. This same fix was also incorporated into the stand-alone program SSErase for debug printing (-m 0) of disk statistics after conditioning. The 'in2net?' query in case "inactive" now appends (after the target computer's name) the value of nowbyte remembered from the end of the last previous in2net transfer, if any. This is an addition to Version 2.6.3 of the documentation. Bug fix: In 'bank_switch=on' mode, the incorrect doubling of the first scan on the switched-to bank is no more. Automatic bank switching is automatically turned off (equivalent to 'bank_switch=off') whenever the selected bank changes to the other bank. This is necessary to prevent some really bad interactions. Bug Fix: DMS is now set to "Recorded" on either 'record=on' or 'record=off'. This is to insure that with 'bank_switch=on', after an automatic bank switch during recording, then both banks will have DMS set to "Recorded". Bug not fixed: With 'bank_switch=on', a 'scan_check?' of the last scan, after completing, causes a switch to the other bank, which turns bank_switch off (equivalent to 'bank_switch=off') and other problems. Don't! Bug not fixed: With 'bank_switch=on' during recording, a spurious "WARNING: Record pointer not incrementing" sometimes occurs, just once, as the bank is automatically switching. Bug fix: 'net2out=...' and 'net2disk=...' now return code 1, delayed completion, when appropriate. Bug fix: DMS is written only when it changes. This avoids warning messages on 'play=arm' and 'play=on'. Bug fix: In VLBA-format scans, a 'track_check?' query now correctly identifies tracks 102--133 (i.e., from data-acquisition set two) and now also supplies 'D' (duplicated) correctly in the 16-track case. Bug fix: Overflow status reporting from an 'in2net?' query was improved. The status returned by a 'net2out?' query now distinguishes among "waiting" for a connection, "halted", which means connected but no data flowing, and "active", which means that data are flowing or at least available to flow from net to FIFO. This is an addition to Version 2.6.3 of the documentation. In 'scan_set=...', an offset that would result in a byte number outside the range of the scan no longer causes the command to return an error. Scan_set, instead, retains the default start or end of the scan, and an error "Scan_set offset falls outside scan", readable by 'status?' or 'error?' is posted. This is a revision to Version 2.6.3 of the documentation. A 'net2out?' query in case "active" or "waiting" has the number of bytes waiting in the FIFO appended. A non-zero value for this number means that the FIFO is still supplying bytes to the output even while "waiting" for more input from the network. This is an addition to Version 2.6.3 of the documentation. Bug Fix: 'net2out=...' FIFO overflow at 531 Mbytes is prevented by (temporarily) refusing additional bytes from the network. Bug (probably) fixed: Formerly, when net2out was connected to a source such as in2net that can be temporarily stopped ('in2net=off', then 'in2net=on'), and when skips are used with net2out, and while in2net is off (i.e., no data are flowing to the machine with net2out), then the number of bytes in the FIFO at the net2out end (from XLRGetFifoLength()) was sometimes dramatically wrong (much too large). This caused a (fake) FIFO overflow condition (over 500000000 bytes), and Mark5A then refused to accept more bytes from the network to the FIFO until the FIFO drains down. But, in this case, because of this bug, this drain never happens. We're still testing, but we think that this bug was fixed by the new sspxf.bib mentioned above. Bug fix: Code in 'skip=...' that checked to avoid overflow or underflow of FIFO in net2out has been removed again following instructions from Conduant; they are doing such a check inside XLRSkip(). Bug Fix: The unnecessary double pass of checkDisks() on startup in case bank A is empty (and bank B is not) has been eliminated. Bug workaround: In case net2out is active, the length of the data received from the socket (using recv()) is truncated to an integer times 8 before calling XLRWriteData(). Bug fix: This version with 'in2net=on' now works with either bank A or bank B selected. (This bug fix is due to a new sspxf.bib file.) Bug fix: During playing, turning the keyswitch of the not-selected bank off does not halt playing. (This bug fix is due to the new sspxf.bib file noted above.) Bug fix: Net2disk now sets the default scan name as prescribed in the documantation. The eVLBI functions, in2net, disk2net, net2out, and net2disk, have been much modified and are being tested. The goal is to speed up these network transfers. The changes visible to the user are in net_protocol, which has an added parameter, nbuf, number of blocks (range 1 to 16, default 8), each of "workbuf size" in bytes, in a circular FIFO. The range of allowed values for "workbuf size" has been expanded. The product of nbuf times "workbuf size" cannot exceed 134217728 bytes. We are now working to improve the default values for these paramters for typical situations. These changes in net_protocol are additions to Version 2.6.3 of the documentation. A separate writeup is being prepared with suggestions for optimizing the net_protocol parameters as a function of measured network data rates and round-trip delays. As a heuristic trial, set the "socbuf size" in net_protocol to the delay-rate product converted to bytes. As an example, for a network link with an effective data rate of 8 Mbaud and a ping time of 100 ms, try a "socbuf size" of 100000 bytes. The functions disk2net and net2disk now append fill pattern onto each transfer, if necessary, to bring the byte count up to an integer multiple of "workbuf size" in net_protocol, default 131072 bytes. (But disk2file and file2disk do not do this.) Disk-FIFO-only mode is ready for beta testing. A writeup of this mode is not yet available in the published documentation. For an overview and operation notes, see http://web.haystack.edu/Mark5/DiskFIFO.pdf. Bug fix: 'play=off...' sets DMS to "Played" only if playing was actually stopped by this command. Otherwise leave DMS unchanged. New features have been added to the DMS system: A 3-bit mask allows (1 = TRUE) or disallows (0 = FALSE) automatic update of DMS to "Erased", "Played", or "Recorded". This mask is set by a 'disk_state_mask=E:P:R;' command (default: 1:1:1), where E, P, and R are 1-bit boolean numbers as noted. The default gives legacy behavior. These three numbers are shown by the corresponding query, 'disk_state_mask?'. The most likely use of this new feature is to disallow a change to "Played" at field stations when a snippet of a recording is checked by playing to a decoder. This is an addition to Version 2.6.3 of the documentation. Bug fix: A 'position?' query no longer fails after two unsuccessful 'bank_set=inc' (or similar) commands. Bug fix: In file2disk, the scan label now defaults to the source file name less path (e.g., "/data/", if any, and less suffix (e.g., ".mk5") if any. In scan_set, the default filename for disk2file is now set to the scan_label with ".mk5" appended unless there is already a dot suffix. This is an addition to Version 2.6.3 of the documentation. A 'recover=...' command has been added to Mark5A to replace the Recover stand-alone program, which was used to partially recover from two errors: (1) a partial erasure of a disk pack from running sstest or WRSpeedTest or any similar program that writes over part of a disk module or, (2) recording interrupted during a scan perhaps by a power failure or accidental offing the keyswitch on the active mdoule. Try 'recover=1' for case (1) above, that is to recover any data on a disk module beyond the part written over. Or try 'record=off' first then 'recover=0' for case (2) to supply the equivalent of a 'record=off' that went missing at the time the recording was interrupted. This is an addition to Version 2.6.3 of the documentation. Some additional timing information is now debug printed for network transfers with msglev 0 or less. ## 2004 July 13 (day 195); This tar file contains a new release from Conduant, another version 6.3.beta in sslinux63.BETA.tgz. Our copy was received June 28, and there are internal dates also up to June 28. (Yes, that is the same name as a previous version. Sorry.) Some of the new or improved features from Conduant are still under test. This version of the Mark-5 software is in substantial compliance with Version 2.6.3 of the documentation, Mark 5A command set, with some extensions noted below. 'DTS_id?' will return "2.6.3x" ("x" for extensions) for "command set revision." IMPORTANT: This version of Mark5A is probably incompatible with the current Field System (FS), and, until the next release of the FS, field stations should keep a previous version: March 23 (day 83, see 2004y083d from 'DTS_id?'). For details, see "IMPORTANT:" under "2004 June 15" below. Bug fix: Mark5A no longer refuses to start in case a master disk in bank A is missing or faulty. In the return from a 'status?' query, bit 5 (0x20) now indicates that Mark5A is in the special disk-FIFO-only mode. This is an extension of version 2.6.3 of the documentation. In case disk2file or disk2net is active, the polling loop now checks about every 10 seconds that the current-byte number is incrementing and generates an error message if not. This is similar to the check on the record pointer during recording. Bug fix: 'VSN=...' now works multiple times on the same disk module allowing for the VSN to be changed. Bug fix: When a disk or disks are faulty or missing, 'disk_serial?', 'disk_model?', and 'disk_size?' do not report incorrect information from a removed disk pack or from the other bank. A utility program, LLogr, has been added to the tar file. LLogr is a filter that prepends the date and time to each line of its input from stdin and passes the augmented line to stdout. LLogr is intended to be an aid to interpreting logged Mark5A messages (debugs, warnings, errors) perhaps using script. Typical usage: script -f log_file_name Mark5A -m 0 |& LLogr & The -f tells script to flush (unbuffer) each line. Omit the log_file_name to default to "typescript". The date and time format prepended by LLogr is yyyydddhh:mm:ss, where yyyy is the year, ddd the day of the year, hh the hour, mm the minute, and ss the second of local time. (Check that your Mark-5 computer knows what time it is.) A new feature: The intention is to implement something like Section 3.2a, "Module Pre-testing", of "Mark 5A Disk-Module Labeling and Management Procedures" ("22 June 2004"). Please have a look at this section. Following is a description of a new feature to help make this possible. This is an extension of version 2.6.3 of the documentation. Disk-module state (DMS) is a new parameter to be written into the Label area (with the VSN) on each SS disk module. The DMS represents the last significant operation that happened on the disk module and, in particular, distinguishes among disk modules with recorded data (waiting to be correlated), with data that are being or have been correlated, or with no data (erased) and ready to be recorded. The DMS is saved in the Label area so as to be able to read the DMS from both selected and not-selected disk modules. The possible DMS values are: * Unknown: Note below. * Recorded: Last significant operation was record or a record-like function, net2disk or file2disk. * Played: Last significant operation was play. (But disk2net and disk2file do not affect DMS because these are read, not play operations and are usually done at field stations, not at correlators.) * Erased: Last significant operation was erase or condition from either 'reset=erase' or SSErase. * Error: An error from, for example, an interrupted conditioning attempt or a failure during one of the significant operations above. DMS will be Unknown if the last significant operation was performed using a version of Mark5A or SSErase with a date code prior to the implementation of this new feature. The DMS is intended to represent the last significant action (record, play, or erase) on the SS disk module and is normally set automatically. A command is provided, however, that allows the DMS to be overwritten manually: 'disk_state=DMS'. This command requires a preceding 'protect=off' and affects only the active module. The corresponding query, 'disk_state?', returns the DMS for both modules, if possible, in a format like 'bank_info?'. Read operations, such as scan_check, data_check, track_check, disk2net, and disk2file, do not affect DMS. Read operations are likely to be performed at both field stations and correlators. A separate issue is writing a script to automate "Module Pre-testing" as defined in the document cited above. This DMS implementation provides the information necessary for such a script. WARNING: If a module is processed through recording, playing or erasing with this version (or later) of Mark5A or SSErase and later read into an earlier version of Mark5A, then the DMS code will appear to be appended onto the VSN of the module in the results from the 'bank_set?' and 'VSN?' queries. (There will be no 'disk_state?' query in this case.) If you are processing these results in a script or program, then allow buffer space for the extra characters. Bug fix: Several of the error and warning messages in case of faulty or missing disk(s) are now more logical and sensible. ## 2004 June 15 (day 167); This tar file contains a new release from Conduant, their version 6.3.beta in sslinux63.tgz (our copy received June 14; internal dates also up to June 14). This new Conduant release contains several improvements and bug fixes including automatic bank switching on module full, support for disk FIFO, a 3-second delay from key-switch off to LED off (to discourage pulling the module too soon), augmented bank-status reporting, and a flashing "Ready" LED whenever the bank status prevents (additional) recording. But automatic bank switching and disk FIFO do NOT yet work. Sorry. More discussion of some of these items is below. This version of the Mark-5 software is in substantial compliance with Version 2.6.3 of the documentation, Mark 5A command set, and 'DTS_id?' will return "2.6.3" for "command set revision." Some exceptions are noted below. IMPORTANT: This version of Mark5A is probably incompatible with the current Field System (FS), and, until the next release of the FS, field stations should keep the previous version, that is March 23 (day 83, see 2004y083d from 'DTS_id?'). If you need to upgrade just to the March-23 version or downgrade to the March-23 version (because you accidentally upgraded to a later version), then proceed as follows: Download the script file Mark5Update.bak and, if necessary, change its permissions to permit execution: chmod 0755 Mark5Update.bak Then, as root, execute this script: ./Mark5Update.bak You should then have a working and complete March-23 version of Mark5A and its auxiliary programs. Bug fix: This new Conduant version correctly supplies fill pattern to read or play a disk pack with one or two missing disks (without a special-purpose hack). But there are some spurious error messages in this case, which we'll try to fix for the next release. Additional testing for the missing- or faulty-disk case is needed and is in progress. Bug not fixed: Mark5A refuses to start if you have a faulty disk pack with a missing master disk either in bank A or in bank B with nothing in bank A. Workaround: Put a good disk pack in bank A, or leave both banks empty at Mark5A startup; then install disks. The 'disk2net?' query now returns, in order, "status", "target hostname", "start byte", "current byte", and "end byte". (This is a change from version 2.6.1x of the documentation and also an update to the previous version.) And 'disk2net?' now returns "connected" instead of "waiting" in case the socket connection has been made but data are not yet flowing. (This is also a change from version 2.6.1x of the documentation.) Bug fix: The 'disk2file?' query now returns the file name as specified in version 2.6.1x and 2.6.3 of the documentation. The previous "botched bug fix" was, well, botched. The 'in2net?' query now returns, in order, "status", "remote hostname", "bytes received", and "bytes in buffer". (This is a change from version 2.6.1x of the documentation and also an update to the previous version.) Bug fix and improvement: A logic flaw in errors() (which prints error messages with debuggery, e.g., -m 0) was fixed, and it no longer prints recurring error messages over and over due to polling. Several changes were made to error messages and error-message formats to improve clarity and readability. Bug fix: In case there are no disks in either bank, set nscans (number of scans) to zero. Extra debuggery added in Dplay() (called from ROT-scheduled 'play=on:...') shows output-board status at start of play (at Haystack-designed correlators). Bug fix: The 'play=arm' 'play=on' sequence now works as advertised. A newly added query, 'bank_info?', (usually) returns '!bank_info? 0 : selected_bank : bytes_remaining : other_bank : bytes_remaining ;', where "bytes_remaining" is an estimate of the number of additional bytes that could be recorded onto that disk pack. (But this estimate is made without taking into account any slow or bad disks: When recording is not in progress, 'rtime?' gives a more precise estimate for the selected bank.) Numerious checks are included within 'bank_info?': A missing or faulty disk or a faulty disk pack will show '-' for "bank" or '0' for "bytes_remaining" or both. ('bank_info?' is an addition to version 2.6.1x of the documentation.) Improved 'skip=...': Conduant version 6.3.beta allows skip to work during net2out just as skip during normal playing. (The previous ugly hack is in the trashcan.) Bug fix: 'bank_set=...' checks for data transfers in progress (e.g., recording or playing), and returns an error if so. Bug fix: With a full disk pack, 'rtime?' returns zeros instead of negative numbers for "remaining time" and "remaining GB". The 'VSN=...' command now truncates the disk sizes down to the next integer multiple of 10 Gbytes. (This is an extension of version 2.6.1x of the documentation.) For ROT-scheduled play (Haystack-designed correlators), the commads to start arming were moved to the Dplay() thread because these commands delay 2 or 3 seconds. Thus the play command with ROT now returns that much sooner. This is intended to fix delays experienced whenever four or more Mark-5 units are in a correlation. The empirical constant offset in getrot() was adjusted by 17 msec to reduce the number of skips needed when processing 64-track, 16-MHz data with Haystack-designed correlators. outBoard() now does the CQ sequence (clear and acquire sync) on SET regardless of whether anything was changed. This seems to improve syncing with 64-track, 16-MHz playback (Haystack-designed correlators). The starting and ending byte numbers for a 'scan_check?' are now those set by 'scan_set=...' rather than always the actual start and end of the scan (which are still the defaults in 'scan_set=...'). This is useful at correlators to help recover parts of some faulty scans. (This is an extension of version 2.6.1x of the documentation.) About 10 milliseconds after a ROT-scheduled start of playing (Haystack-designed correlators), the status of sync in the output section of the I/O board is checked, and, if not synced, then an additional CQ sequence (clear and acquire sync) is sent. With only a few milliseconds of lost data, this recovers from a very occasional and obscure error that could otherwise cause no sync for a whole scan. Bug fix: A 'track_check?' on a Mark-III-formatted scan now shows "-1 ?" for decoded track number. (Formerly these were quasi-random numbers, incorrect except by accident. Mark-III formatters do not record track numbers in the frame headers.) The 'record=...' command has been augmented to accept also a "station code", that is a maximum of five command parameters: 'on' or 'off', "scan name", "experiment name", "source name", and "station code". All four of these names are now included in the directory and reported by various queries including 'record?', 'scan_set?', and 'scan_check?'. (This is an extension of version 2.6.1x of the documentation.) The 'file2disk=...' and the 'net2disk=...' commands have been similarly augmented to accept "experiment name", "source name", and "station code" after "scan name" and to report these with the corresponding queries. But net2disk no longer supports the undocumented "receive buffer size" argument; use 'net_protocol=...' instead. (These are extensions of version 2.6.1x of the documentation.) ## 2004 March 23 (day 083); This tar file contains StreamStor version 6.1 (sslinux61.tgz) from Conduant (no date but downloaded March 2) and a later revised sspxf.bib (also March 2). Many of the problems with in2net and net2out seem to be fixed with this Conduant update, but you'll still need a scratch-disk module. If you missed any of the intermediate Mark-5 updates, please read the descriptions down to the last update that your machine already has. Bug fix: Corrected error messages in case 'play=on' or "scan_play' and already playing (etc.). Bug fix: 'skip=...' now works correctly in case net2out and 8 or 16 tracks. (But this is not yet the XLR version.) In mksock(), all TCP sockets are set SO_KEEPALIVE in case a firewall needs this. The same now applies also to Net2file and File2net. The StreamStor utility program dumpram has been added to the programs available to oper from ~oper/bin/; just type its name to start it. Dumpram is a "Utility to dump memory of StreamStor card to a file, Linux version," and is one of the "Utility files and programs to be used under direction of Conduant support." See notes in "Mark 5 Newsletter," Issue #5, Section 8. The 'VSN?' query now returns a parameter (following the VSN) that is one of "OK", "unknown", or "fail", referring to the serial-number cross check. In case "fail", there are three following parameters (as specified in the documentation) and an error message, "Disk serial-number mismatch". This is an extension to version 2.6.1x of the documentation. Bug fix: In CheckBanks(), a faulty or incorrectly mounted disk does not cause an infinite loop retrying. Additional possible status returns from a 'record?' query are: "throttled", "overflow", and "waiting" (all errors). Use 'status?' to see more information. Note that the "throttled" error state can flicker on and off. This is an extension to version 2.6.1x of the documentation. Botched bug fix: The 'disk2file?' query now returns the file name as the first parameter after the return code, followed by the other parameters as specified in the documentation. That is, dest filename and status are interchanged in order. So this version does not agree with version 2.6.1x of the documentation. Sorry. The 'in2net?' query now returns the remote hostname as the first parameter after the return code, followed by the other parameters as specified in the documentation. This is an extension to version 2.6.1x of the documentation. The 'in2net?' query now returns "connected" instead of "waiting" in case the socket connection has been made but data are not flowing (i.e., off). This is a change to version 2.6.1x of the documentation. The 'disk2net?' query now returns the target hostname as the first parameter after the return code, followed by the other parameters in order as specified in the documentation. This is an extension to version 2.6.1x of the documentation. ## 2004 February 08 (day 039); This is a minor update as described below. If you have missed any of the intermediate updates, please read the descriptions down to the last last update that your machine already has. Temporary modifications make skip work (sort of) with net2out. What is actually done by 'skip=..." in this case is to modify the playRate---increase or decrease it depending on the sign of skip---for a time of about two seconds, then set it back to the previous value. The magnitude of this modification is chosen to produce the effect of the prescribed skip within this two-second period provided that this is possible within the useable range of playRate. 'Skip=...' in this case returns 1, delayed completion, and a 'skip?' query returns 5 ("try again later") while this pseudo skip is in progress. This is an extension to version 2.6.1 of the documentation. Bug fix: 'reset?' (no such query) returns a sensible error message. SSErase was modified so that it accepts 'Y' or 'y' (formerly 'Yes' or 'yes') as the affirmative response to its questions. This allows /usr/bin/yes to supply the answers automatically, for example: yes | SSErase -m 0 & With this construct, you may ignore the "Broken pipe" at the end and also ignore "XLR error 7 Invalid command" when you erase or condition several disk packs in succession. But be sure that you want to erase all the disks that SSErase can find in both banks. The received-ROT debug message now prints local time (instead of Linux time in seconds). This applies to Haystack-designed correlators in debug mode. The 'play?' and 'scan_play?' queries were rewritten to use XLRGetPlayBufferStatus() to determine "arming" or "armed" status (instead of calculated timings). Bug fix: 'rtime?' now works on an erased disk pack. Bug fix: 'rtime?' no longer generates spurious error messages in case of all masters, no slave drives (because these XLR errors were renumbered). Bug fix: 'VSN?' now gives a probably correct answer even in case of a missing disk and certain other disk errors. ## 2004 January 08 (day 008); This is a minor update as described below. If you have missed any of the intermediate updates, please read the descriptions down to the last last update that your machine has. Bug fix: 'record?' query in 'off' case now has colons separating scan name from source and experiment name. (And the colons have spaces around them.) Bug fix: The 'n' option (default) in disk2file now works. ## 2003 December 18 (day 352); This is a minor update as described below. If you have missed any of the intermediate updates, please read the descriptions down to the last last update that your machine has. SSErase now says what it is about to do and asks "Are you sure ... ?" before starting erasing or conditioning. The documentation in Mark5A Auxiliary Programs was updated on December 21. The option parameter in 'disk2file' now has three possible values: 'n', which is now the default, means create a new file or return an error if the file already exists; 'w' (erase or create) and 'a' (append or create) are the same as before. This is an extension to version 2.6.1 of the documentation. The 'VSN=...' command has been rewritten. It now accepts a string of just 8 characters comprising a 2- to 6-letter OwnerID followed by a '-' followed by a serial number to make up the 8 characters. (Add leading '0's in the serial number as needed.) The OwnerID must be all alphabetic and will be converted to upper case if need be. The Mark5A program then calculates and appends /Capacity,Gbytes/MaxDataRate,Mbaud to make up the standard VSN format to write to SS disks. This is an extension to version 2.6.1 of the documentation. The 'rtime?' query now checks to see whether enough information is available to enable the calculations to be done and returns an error if not. (Previous versions sometimes tried to divide by zero.) ## 2003 December 05 (day 339); This is also the date on the tar file, but the latest version of Mark5Update is December 10, which eliminates the manual operation at the end of the update. Please also read the notes below; this version of the tar file is almost identical with the December-04 version except for minor cosmetic changes and: Bug fix: SSErase, whenever it can erase two packs at a time, now saves and restores the VSN of the pack in bank B. This fixes the serious bug noted below. ## 2003 December 04 (day 338); This version of the Mark5A software substantially complies with version 2.6.1 of the documentation, "Mark 5A command set", and the 'DTS_id?' query with this version will return '2.6.1x', that is version 2.6.1 with extensions (x), for 'command set revision'. This version of Mark5A was tested with input design revision (IDR) 0xb8 and output design revision (ODR) 0x19 and should work with all previous IDRs and ODRs but without certain features as noted below. The 'DTS_id?' query shows this revision information for the I/O board in the machine. The current version of Mark5Update is September 29. Bug fix: A new pair of libraries, libssapi.a, and new include files, xlrapi.h and xlrtypes.h, from Conduant (December 3) seem to have fixed the glitches in net2out. Bug not fixed: 'net2out=close' does not correctly return to bypass mode. Work around: 'play=on ; play=off'. Serious bug not fixed: SSErase on two packs at a time copies the VSN from the pack in bank A onto the pack in bank B. Use Mark5A to reset the VSN of the pack in bank B in this case ('protect=off ; VSN=xxxx'). Bug fix: 'net_protocol=...' allows 'socbuf_size' to be 0 to reset to defaults. Bug fix: File2net (stand-alone program) has an added sndbufs parameter for "tuning" file transfers. Bug fix: 'protect=off', even if it fails, allows a following 'reset=...' or 'VSN=...'. Bug fix: File2net has "localhost" as default target computer. Bug fix: 'disk_serial?', 'disk_model?', and 'disk_size?' no longer remember and report the last previous data when both banks are empty. Bug fix: Mark5A now has an XLRCardReset() in its startup routine, which insures starting correctly even if SSErase was used just before. (SSErase does not do a reset at its end to avoid delays when it is used several times in sequence.) The command-line arguments of the stand-alone programs File2net and Net2file have been changed to use the so-called getopt() scheme: Each parameter to be changed from its default is preceded by a '-x', where 'x' is a letter designating which parameter. To see a list of the available prameters and their letters, try 'File2net -h' or 'Net2file -h'; that's h for help. The documentation in Mark5A Auxiliary Programs was updated again on December 21. The 'VSN?' query returns an 'OK' to indicate that all serial numbers have been checked and match the remembered serial numbers. (Neither 'OK' nor error message means that there are no remembered serial numbers to check against, which occurs, for example, just after an SSErase.) This 'OK' was actually present in previous versions but not documented. This is an extension to version 2.6.1 of the documentation. SSErase was further modified: If both disk packs are able to be erased or conditioned, then, after this is completed, each disk pack is separately erased so that each can (and must) be written as a separate disk pack in bank mode. These additional steps take less than two seconds. The documentation has not yet been revised to show this change. Sorry. ## 2003 October 31 (day 304); This release is almost the same as the two previous releases except for bug fixes. Please also read the description for the two previous releases below. Bug fix: This tar file includes a new sspxf.bib file from Conduant, dated October 31 (but "FirmDateCode Oct 30 2003"), and this fixes net2out. But bug not fixed: We see some millisecond glitches in the decoder coming from net2out. This is not a speed problem; it occurs at speeds well below network capabilities. The Conduant buffer is definitely functional as we infer from several seconds of delay after sending stops and before playing stops. We're working on it. Bug fix: The default buffer size for UDP with 'disk2net' or 'in2net' is now OK even if you have an MTU larger than 1500 bytes. Bug not fixed: The Mark5A debuggery report for rcvbufs or sndbufs (whichever is in use) is twice the value that you set for socbuf_size. We think that the reporting is correct and that the value set (in setsockopt()) is doubled before use. And we think that this is a Linux problem. Some additional error messages were appended to error responses from commands and queries especially those used by the Field System. Starting Mark5A with debuggery (-m 0) is still recommended because additional information about errors is sometimes available there. ## 2003 October 28 (day 301); This release is very similar to the previous release except that we have fixed all known errors in eVLBI transfers with UDP. Please read also the notes for the October-20 version below. Known problem: net2out is broken. Sorry. We're working on it. Bug fix: Most of the errors in using udp with 'in2net', 'disk2net', 'net2out', and 'net2disk' in Mark5A and in Net2file and File2net have been fixed. eVLBI notes: net_protocol = protocol : socbuf_size : workbuf_size ; Protocol is 'tcp' or 'udp'; default 'tcp'. (Note: Mark5A is case insensitive whenever it can be, but protocol is sent to getservbyname(), which is case sensitive. So don't use 'TCP' or 'UDP'.) Socbuf_size is used as recbufs for receiving (net2disk or net2out) or sndbufs for sending (disk2net or in2net). If socbuf_size is not specified or is set to 0, then Mark5A uses the system defaults for recbufs and sndbufs. The maximum sndbufs and recbufs are set by Linux system configuration. Workbuf_size is the size of the buffer sent to the socket on sending (disk2net or in2net). Workbuf_size defaults to 131072 bytes for TCP or to 54752 bytes for UDP. The maximum workbuf_size for TCP is 999424 bytes. The maximum workbuf_size for UDP is 65528 bytes, but, in our experience, does not work in practice if larger than the default. Mark5A, nevertheless, allows you to try up to this maximum. The maximum transmission unit (MTU) for IP networks is typically 1500 bytes as set by network configuration. (So-called jumbo frames, up to 9000 bytes or even more, are sometimes possible.) Each workbuf_size UDP datagram can be fragmented into frames whose size is given by the MTU minus one or two headers. The IP header, which is in every fragment, is 20 bytes (actually sizeof(struct iphdr) from /usr/include/linux/ip.h), and the UDP header, which is only in the first fragment, is 8 bytes (actually sizeof(struct udphdr) from /usr/include/linux/udp.h). So, if fragmentation, then the first fragment can have MTU-28 data bytes, and all subsequent fragments each can have MTU-20 data bytes. If the prescribed workbuf_size is more than the resulting 1472 bytes in this case, then the UDP packet gets fragmented. And, in this case, Mark5A truncates the effective workbuf_size to an integer number of fragments, for efficiency, by assuming MTU-28 bytes for the first fragment and MTU-20 bytes for all subsequent fragments. This effective buffer size is also truncated, if need be, to an integer times 8. The default working maximum (for MTU 1500 bytes), as determined empirically, is 37 fragments for a workbuf_size of 54752 bytes, which is the UDP default, as noted above. Known problem: In case of sending eVLBI transfers (disk2net or in2net), with UDP, with MTU larger than 1500 bytes, and with the default buffer (as set by 'net_protocol='), then the buffer size is calculated incorrectly. Work around: Do not take the default. Set workbuf_size in this case to 65528 bytes or smaller. Testing eVLBI: Our limited testing among three local machines connected over shared 100-Mbaud ethernet through routers and a switch gave UDP baud rates in the range 9.3e7 to 9.6e7 (i.e., 93% to 96% of maximum possible) and fractional data-loss rates in the range 0.0002 to 0.0064. Other users were not happy with our use of UDP. TCP over the same links gave baud rates in the range 6.8e7 to 8.9e7 (i.e., 68% to 89% of maximum possible) and, of course, no data loss. And no complaints from other users. ## 2003 October 20 (day 293); (The tar file is dated October 21.) This version of the Mark5A software substantially complies with version 2.6 of the writeup, "Mark 5A command set", and the 'DTS_id?' query with this version will return '2.6x', that is version 2.6 with extensions (x), for command set revision. Some extensions and revisions of this version of the Mark5A software relative to version 2.6 of the writeup are noted below. This version of Mark5A was tested with input design revision (IDR) 0xb8 and output design revision (ODR) 0x19 and should work with all previous IDRs and ODRs but without certain features as noted below. The 'DTS_id?' query shows this revision information for the I/O board in the machine. The current version of Mark5Update is September 29. This tar file contains a new beta release from Conduant, sslinux601Beta.tgz, that is version 6.01, of October 15, and a newly revised sspxf.bib file dated October 17. Bug fix: protect=off preceding reset=abort is not required. Bug fix: startbyte and endbyte default to the start and end of the first scan (if any) on startup of Mark5A. In the answer to a 'status?' query, the bank-status bits (i.e., nn in 0x0nn00000) are now set as appropriate even if data transfers are in progress. This is a change from version 2.6 of the writeup. The 'play_rate?' query now shows 5 after-decimal digits for each reported frequency. The 'bank_set?' query has been augmented. It now reports: the selected bank, the selected module's VSN (Label), the other (not-selected) bank or a '-' if the other bank is not ready, and the other module's VSN, if any. This is an extension to version 2.6 of the writup. WARNING: There are serious errors in using udp in the following. A new keyword, 'net_protocol', has been added. As a command, net_protocol = protocol : socbuf_size : workbuf_size ; accepts protocol to be tcp or udp (default tcp). (Additional protocols will be accepted in future versions.) Then socbuf_size, which is used as rcvbufs in 'net2out' and 'net2disk' or as sndbufs in 'in2net' and 'disk2net', defaults to 0, which means to use the Linux OS defaults. Also workbuf_size (previously called socbuf) defaults to 131072 or to the previously set value and is used as the buffer size to send to data sockets in 'in2net' and 'disk2net'. As a query, 'net_protocol?' reports the previously set or in-use values. Units for all these buffer sizes are bytes. This is an extension to version 2.6 of the writup. WARNING: There are serious errors in using udp in the following. The stand-alone programs File2net and Net2file have also been rewritten to allow either tcp or udp protocol. Try: File2net machinename [ filename [ startbyte [ endbyte [ protocol [ workbufsize [ msglev ] ] ] ] ] ] Or try: Net2file [ filename [ rcvbufsize [ protocol [ msglev ] ] ] ] A '.' (dot) works as a placeholder for defaults. ## 2003 October 08 (day 281); IMPORTANT: This version fixes a serious error in ROT-delayed play as used by Haystack-designed correlators. The previous version (October 03) will NOT work with Haystack-designed correlators. This version of the Mark5A software substantially complies with version 2.6 of "Mark 5A command set", and the 'DTS_id?' query with this version will return '2.6x', that is version 2.6 with extensions (x), for command set revision. This version of Mark5A was tested with input design revision (IDR) 0xb8 and output design revision (ODR) 0x19 and should work with all previous IDRs and ODRs but without certain features as noted below. The 'DTS_id?' query shows this revision information. The current version of Mark5Update is September 29. Bug fix: 'play=...' with a ROT-start argument (as used by Haystack-designed correlators) works again. Sorry. Formal parsing (i.e., without -f 0 on the command line) has been made even more formal: Mark5A now returns an error if a keyword is followed by neither '?' nor '='. (Formerly a keyword alone (no '?' or '=') that was only a query was executed as a query, and a keyword that could be a command was executed as a command with defaults, if any. This caused some significant problems when a user intended but forgot a '?'.) Informal parsing mode (i.e., with -f 0 on the command line) is unchanged. The protection scheme has been modified as follows: A 'protect=off' command is required immediately preceding a 'reset=...' or 'VSN=...' command, even if protection was already off. If not, an error return such as '!reset = 6 : Previous protect=off required ;' explains what is needed. This is an extension to version 2.6 of the writeup. Bug fix: The default for 'scan_play=...' is now 'on' as in the writeup. (But note that the default for 'play=...' is 'off', and 'record=...' has no default.) Bug fix: Several commands with error conditions were incorrectly returning '?' instead of '='. Bug fix: A 'play=on' command following a previous 'play=arm' is not permitted to have also a play pointer (byte number) (because the play pointer has already been used by the 'play=arm'). This case now returns an error. ## 2003 October 03 (day 276); IMPORTANT: Please read also the notes down through the August-29 release below. This tar file contains a new release from Conduant, sslinux60.tgz, dated September 24, which contains libraries that work with either version 2.x or 3.2 of the gcc compiler and its associated libraries. This version of the Mark5A software substantially complies with version 2.6 of "Mark 5A command set", and the 'DTS_id?' query with this version will return '2.6x', that is version 2.6 with extensions (x), for command set revision. This version of Mark5A was tested with input design revision (IDR) 0xb8 and output design revision (ODR) 0x19 and should work with all previous IDRs and ODRs but without certain features as noted below. The 'DTS_id?' query shows this revision information. The current version of Mark5Update is September 29. The cc5A compile script was updated to check which version of gcc (and g++) is present and update the symbolic link to the corresponding version of libssapi.a. This was done to allow this version to be installed on Red-Hat Linux versions 8.x and 9.0 (as well as 7.x). This was checked in a Red-Hat 9.0 machine. SSErase was rewritten to deal with write protection on either bank A or B. The operator will be asked whether to remove any write protection before erasing. Certain illegal situations now generate proper error messages. Additional information is now available from SSErase at msglev -1 (i.e., -m -1). The 'play' and 'scan_play' commands and queries were rewritten to comply with version 2.6 of the writeup except that distinguishing between between 'arming' and 'armed' in the query responses is determined by calculated timings rather than feedback from the Conduant buffer. Bug fix: In case a data transfer is in progress, 'scan_set=...' returns an error and does nothing else (because it might change the data-transfer parameters). Bug fix: 'bank_set=...' no longer wastes time by causing an unnecessary second call to CheckBanks() (and checkDisks() and readdir()). Bug fix: A warning message from 'rtime?' is now only in case of a significantly slow disk. Bug fix: 'reset=erase' and 'reset=erase_last_scan' now also reset the reference recording index used to determine the warning "Record pointer not incrementing." This prevents some spurious warning messages. The 'replaced_blks?' query in version 2.6 of the writeup is now implemented. It works only when no data transfers (e.g., play or record) are in progress. Bug fix (sort of): 'data_check?', 'track_check?', and 'scan_check?' now verify also the two required zero bits in the byte after sync in the frame header. (This byte contains two BCD numbers in both Mark-4 and VLBA formats.) Although counter-intuitive, this allows 'track_check?' to find frame headers, previously missed, in some partly flawed tracks with long strings of ones, which can be caused by no input to the video converter or no video converter connected to the formatter for this track. This change is either good or bad depending on whether you want 'track_check?' to complain about such partly flawed tracks (the track is written correctly; the data are flawed.) Bug fix: Play_rate now correctly defaults to clock 9 MHz. Bug fix: 'record=on', 'net2disk=...', and 'file2disk=...' now check for write protect and, if so, then return an error and do nothing else. (Previous versions returned an error but then tried to write to disks anyway and became very confused.) Bug fix: 'scan_check?' with "skipped" unknown now returns 0 but with '?'s to indicate this error (presumably a faulty scan). Commands 'scan_set=...' and 'track_set=...' now have a 'dec' (for decrement) argument (as well as an 'inc' for increment). Both wrap or unwrap as appropriate. This is an extension of the specifications in writeup versions 2.5 and 2.6. The maximum socbuf size was increased from 999424 to 3997696 bytes, where socbuf is the second undocumented additional parameter to 'in2net=...' and 'disk2net...'. An error is generated by any attempt to exceed this maximum. (The default socbuf size is unchanged at SOCBUF = 131072 bytes.) The larger size is intended to help "tuning" eVLBI transfers. ## 2003 September 29, IMPORTANT: An error was found in the Mark5Update script. The latest version, dated September 29, fixes a reboot problem. If you use the old version of this script, then your Mark 5 will work until you need to reboot, then may fail because the Jungo driver may not be properly installed during the reboot. This seems to depend on which version of Linux you have. You may also fix this by hand as follows: In the file /etc/rc.d/rc.local, change the line /sbin/insmod windrvr6 to /home/streamstor/linux/driver/redist/wdreg windrvr6. If you execute this line by hand, then also execute the following line: chmod 0666 /dev/windrvr6. ## 2003 September 04 (day 247); IMPORTANT: Please read also the notes for the August-29 release below. The current version of Mark5Update is September 09. The 'DTS_id?' query with this version will return '2.5x' for command set revision, that is version 2.5 with extensions (x). Except for 'play', 'scan_play', and 'replaced_blks' however, this version closely complies with version 2.6 of "Mark 5A command set". It was tested with input design revision (IDR) 0xb8 and output design revision (ODR) 0x19 and should work with all previous IDRs and ODRs but without certain features as noted below. The 'DTS_id?' query shows this revision information. This release contains a new sspxf.bib file (September 04) from Conduant, which fixes two problems: (1) Bank B now works with delayed play as used at correlators, and (2) bank A's "Ready" light does not fib about being ready on startup. Bug fix: 'File2disk' now sets source file name default to "save.data". Bug fix: In 'disk2net' and 'disk2file', if the starting and ending byte numbers are not integer multiples of 8, then they are truncated down and handled correctly. The inline copy of the directory is no longer written, but we still try to read it (e.g., at a correlator) if the separate directory can't be read. ## 2003 August 29 (day 241); IMPORTANT: This version is in a renamed tar file, namely Mark5A.tgz (rather than Mark5A.tar.gz), and a new version of Mark5Update (September 02 or 09) is needed to download and install it. Do not try to use the previous version of Mark5Update (June 02) with this tar file. This new Mark5Update script (September 02 or 09) contains several necessary changes, including editing /etc/rc.d/rc.local to accomodate the renamed Jungo driver, that is windrvr6 instead of windrvr. This all happens automatically using this new script. This tar file contains a new release from Conduant, sslinux60Beta, dated August 27. (But note that some older releases had the same name but now have "old" suffixes.) This version of the software agrees with version 2.5 of the writeup but with extensions (2.5x) as noted below. It was tested with input design revision (IDR) 0xb8 and output design revision (ODR) 0x19 and should work with all previous IDRs and ODRs but without certain features as noted below. The 'DTS_id?' query shows this revision information. The stand-alone SSErase program with conditioning (-c 1) now prints the disk statistics obtained during conditioning. This might allow identification of faulty disks, but a suspicious disk should always be retested because disks might have been improved by the conditioning process---that's the point of doing it. Conditioning is a process rather different from normal recording and playing. To use the statistics after conditioning, try to compare one disk with another in the same pack or at least after the same conditioning process. The SSErase program has also been modified to check for write protection (which applies only to a disk pack) and ask the user whether to remove the write protection, as is necessary, of course, for erasure or conditioning. Note that SSErase does not have a reset at the end as, for example, Mark5A does. This allows erasing multiple disk packs sequentially without waiting for the normal resetting and checking processes that accompany XLROpen. The stand-alone program SSReset performs the reset that is missing from SSErase and so causes resetting and checking at the next startup of Mark5A or any other SS-aware program. An optional experiment name and source name (or source name and experiment name) have been added as two additional parameters on 'record=on...' (no embedded spaces). (This is labeled as "NYI" in version 2.5 of the writeup.) DirList, 'scan_set?', and 'scan_check?' now show these added parameters separated by spaces from the scan name. DirList was modified to leave room to print the added parameters. The 'scan_set=...' command can now use abbreviated scan names. If its first parameter is all numeric, then 'scan_set' will first try to interpret this parameter as a scan number. Failing that (e.g., not that many scans), then 'scan_set' will find the first scan whose scan name, experiment name, or source name contains the string in this first parameter. (Like almost everything else in Mark5A, this search is case insensitive.) This is an extension of the specifications in writeup version 2.5. The start byte number that is set and reported by 'scan_set' was augmented as follows: It defaults to the start of the scan or can be set by 's', 's+', 'c', or 'e' as before. Or it can be set, instead, by an offset byte number, preceded by '+' (no space), to be added to the start of the scan, or preceded by '-' (no space), to be subtracted from the end of the scan to give the start byte number. Or it can be set by a time in standard notation (no embedded spaces): 'nnndnnhnnmnns'. Any of these time units can be omitted, and any omitted parameters to the right of any prescribed parameters are taken to be zero. As an example, '5h' is the same as '5h0m0s'. Any omitted parameters to the left of any prescribed parameters mean to ignore the corresponding time units in the data. Mark5A will then, if possible, set the start byte number to the first occurence of this time within the prescribed scan. (But note that '2m', which means '2m0s', will not start at 4m, 6m, and so forth.) Alternatively, a '+' preceding such a time (no space) tells Mark5A, instead, to just add this time onto the start of the scan to give the start byte number. The play pointer also is set to this same start byte number by the scan_set command, but the play pointer can be changed by the play command, and it increases while playing. These are changes in the specifications from writeup version 2.5. There is now also an additional parameter on 'scan_set', the end byte number, which defaults to the end of the prescribed scan. Or the end byte number can be set by an offset byte number, preceded by '+' (no space), to be added to the start byte number (determined as above---not necessarily the start of the scan), or preceded by '-' (no space) to be subtracted from the end of the scan. As another alternative, the end byte number can be offset from the start byte number by a time parameter in the same time format as above. And if a '+' (no space) preceds this time parameter, then the time offset is added to the start byte number determined as above. If a '-' (no space) preceds this time parameter, then the time offset is subtracted from the end of the scan. These are changes in the specifications from writeup version 2.5. These starting and ending byte numbers as set and reported by 'scan_set' are now used in scan_play. This is a change in the specifications from writeup version 2.5. In 'disk2net' and 'disk2file', the defaults for the start byte and end byte numbers are now the start and end byte numbers as set and reported by 'scan_set'. And, in 'disk2file', the destination file name defaults to the name of this scan (in the current directory). These defaults can, of course, be overwritten by explicit parameters. These are changes in the specifications from writeup version 2.5 and also changes from the previous update (below). Bug fixes: 'File2net' and 'file2disk=...' now correctly translate 0 for the length of a file into the actual length even for files over 3 Gbyte, and they also correctly deal with non-zero file starting bytes. Bug fix: In 'file2disk', the default scan name is the file name without the path. The 'bank_set=...' command was rewritten. If the bank that you ask for is not the bank already selected, then this command returns a code of 1 meaning delayed completion. Bank changing takes a variable amount of time up to almost 3 seconds. While bank changing is in progress, many commands and queries return a code of 5 meaning busy---try again later---or 6 meaning conflicting request. (In effect, neither bank is mounted during this transition.) If an attempt to change the bank fails (e.g., if there is no ready disk pack in the other bank), then a 'status?' or 'error?' query will show error 1006, "Bank change failed." And a 'bank_set?' query will show whether the bank has changed. Changing banks can also generate other errors if there are problems with the new bank. Bug fix: A signal handler for SIGPIPE now handles a broken pipe (e.g., talking to the Field System) to keep Mark5A alive. This generates error 1005, "Broken pipe". Bug fix: Now 'disk2file?' query works again. Bug fix: Now 'disk2net?' query does "waiting" correctly. Bug fix: Debuggery was added for pthread_create() failure, and pthread_detach() is now used to avoid leaking thread resources. This bug fix is needed at correlators that use delayed play (i.e., all Haystack-designed correlators). Bug fix: 'Disk_serial', 'disk_model', and 'disk_size' now work without generating error messages even during recording or playing. Bug fix: 'VSN?' query now correctly checks disk serial numbers against its stored serial numbers and reports any errors (except that these can't be checked after an SSErase). There is also added information in case of a mismatch. Bug fix: Reset=erase now uses XLRErase() (instead of XLRRecord() and XLRStop()) and also clears the user-directory area, which clears the remembered serial numbers. (There is no effect on VSN.) This fix is intended to eliminate the problem of remembered old data that were recorded with previous SS versions. Bug fix: In case of a bad or missing disk, the check-directory flag is turned back on when bank or disks are changed. Bug fix: On 'play=off:', the play pointer is set to this even if playing ended at a different byte number. And 'play=off' without a updates the play pointer only if playing was in progress and this command stopped it. The 'rtime?' query was modified to calculate a better approximation to the remaining time and capacity using the effectively reduced capacity of the disk set in case one or more of the disks is slow enough to impair performance. At this release, however, this better approximation does not work during recording: Expect more accurate numbers from 'rtime?' only when idle, and expect less accurate numbers during recording. Note also that this situation---one or more slow or faulty disks---also increases the rate at which the remaining disk space is used up and reduces the maximum data rate that the disk set can accept. (The 'dir_info?' query was not changed.) The stand-alone DirList program was modified to accept an alternative input file name. With no command-line arguments, DirList reads and lists /var/dir/Mark5A as before. There are now two setable parameters: -m sets the debug message level, msglev (range -1 to 3, default 1), and -f sets the file name, default /var/dir/Mark5A. This allows reading a copy of the directory file as sent, perhaps by email, from stations. The delayed-play scheme used at Haystack-designed correlators was changed to use the new "play-arm" option from Conduant. The SS buffer is set to start filling a few seconds in advance of the desired start ROT; then a play trigger starts actual playing with only sub-millisecond delays. This is intended to almost eliminate the need for skips. The 'get_stats' function now also shows a count of the number of SS blocks that were replaced with fill pattern during the previous playing. This count, for each disk, appears as the last number on the printed line. Anything other than 0 for this count indicates a problem with this disk. This is an extension of the specifications in writeup version 2.5. The 'in2net' function has given us a lot of trouble and was rewritten. Among several changes, clocking in the input board is now turned off and on by in2net using "notclock" to define the start (on) and end (off) of each scan. Version 2.5 of the writeup is still correct except that the "#bytes sent" as returned by the 'in2net?' query is now the number of bytes received (including the bytes in the FIFO waiting to be sent) since 'in2net=connect'. And the 'in2net?' query has an added int parameter, the number of bytes in the FIFO waiting to be sent. After 'in2net=off', you should wait for this last parameter to drain to 0 before 'in2net=disconnect' to avoid losing data. After 'in2net=off' but before 'in2net=disconnect', "#bytes received" (formerly "#bytes sent") can be logged as an index into the received file or scan at the target computer. This is a good approximation even if the "#bytes remaining in buffer" is not 0 and becomes exact when this number drains to 0. This is an extension of the specifications in writeup version 2.5. Proper operation of the 'in2net=off' feature requires version 0xb7 or later of the "Input design revision" (IDR, which is the next-to-last parameter returned by the 'DTS_id?' query). Otherwise you'll need to turn off the formatter clock between scans to stop data flow to the I/O board. (The FS can do this.) Bug NOT yet fixed: 'in2net' wants a scratch disk pack in bank A even though it doesn't need or use it. ## 2003 June 12 (day 163); This version agrees with writeup version 2.5 but with extensions (2.5x) as noted below. It was tested with input design revision (IDR) 0xb8 and output design revision (ODR) 0x19 and should work with all previous IDRs and ODRs but without certain features as noted below. The query 'DTS_id?' shows this revision information. The 'dir_info?' query can now be used during record, play, and other data transfers. The stand-alone SSErase program was augmented to do disk conditioning. With no command-line arguments, SSErase does a simple erase as before. There are now two setable parameters: -m sets the msglev (range -1 to 3, default 1), and -c sets conditioning (0 for FALSE, 1 TRUE, default FALSE). So, for example: SSErase -m 0 -c 1 & goes through the long conditioning process on whatever disks it can access, up to 16 at a time in both banks. With msglev set to 0, debug prints about every minute to show what's happening. Conditioning disks is recommended before recording especially if they are to be recorded at or near their maximum data rate. Conditioning amounts to a write-read-write cycle through the whole set of disks, but here it is done in two rather than three passes. Printed with debuggery and counting down twice is the number of bytes per bus. With an 8-disk pack, for example, this count starts at a quarter of the total capacity, which is twice the capacity of a single disk. Conditioning one 120-Gbyte disk takes about 101 minutes; two 120-Gbyte disks (as two masters), 103 minutes; four 120-Gbyte disks (as four masters), 111 minutes; eight 120-Gbyte disks (an eight pack), 157 minutes; and sixteen 120-Gbyte disks (two eight packs), 278 minutes. 200-Gbyte disks take about 5/3rds as long---no surprise. Conditioning one 200-Gbyte disk takes about 160 minutes; eight 200-Gbyte disks, 286 minutes; and sixteen 200-Gbyte disks, 465 minutes. These times are approximate for the ideal case, but, if any of the disks has a problem, times can be much longer. After conditioning two disk packs at a time (e.g., more than eight disks), or in any case where you change the disks into a different configuration, then you'll also need to erase each disk pack separately before recording. When the recording speed (e.g., formatter clock) and mode are such that the disks can't keep up, either because of an improper setup or a faulty disk or disks, then the recording will be "throttled" (some data are lost), and a flag indicating this throttling (called a suspend flag because recording is momentarily suspended) is now available to Mark5A. This flag causes error 1003, "Recording throttled (can't keep up)" and, on 'status?', bit 0x400 set. In marginal conditions, this flag may flicker on and off, and this error, as with most errors, may be remembered after the error condition is fixed. Clear the error with 'status?' or 'error?'. This feature requires version 0x19 or later of the "Output design revision" (ODR, which is the last parameter returned by 'DTS_id?'). The added status bit 0x400 is an extension of the specifications in writeup version 2.5. Bug fix: With ODR 0x19 and later, a 'mode?' query reports the correct output mode and output submode. In disk2net and disk2file, if the end byte number is preceded by a + (no space), then this number will be interpreted, instead, as the byte length of the transfer. This is equivalent to saying that this number will then be added to the start byte number to give the effective end byte number. This is an extension of the specifications in writeup version 2.5. In disk2file, if the destination filename (minus path) matches an existing scan name, then the start and end byte numbers default to the start and end of this scan. These defaults can, of course, be overwritten by explicit numbers. This is an extension of the specifications in writeup version 2.5. We recommend that you do not use this feature because it will be changed in the next release. ## 2003 June 4 (day 155); This tar file contains a new partial release from Conduant (May 28), which fixes some problems with playing or reading with missing or failing disks. See notes below on fill pattern. The VSN command was augmented to record the list of serial numbers when the VSN command is issued and also at recording. The VSN query was augmented to cross check this list of serial numbers against the disks actually present and return an error, "Disk serial-number mismatch," if they differ. Bug fix: A 'status?' query now distinguishes between net2disk and net2out. Bug fix: 'get_stats?' doesn't generate spurious XLR errors on missing slave disks but does report disk number. Bug fix: 'reset=abort' now actually works as advertised. ## 2003 May 24 (day 144); An interim update from Conduant (May 23) is in this tar file. This fixes a bug in recording on bank B that affected the directory on bank A. To deal with a missing or faulty disk in a disk pack, this version sets a Conduant option that allows reading and playing in spite of this problem. Fill pattern replaces the missing data. (See notes on fill pattern below.) This might allow correlators to salvage some of the data from such a faulty disk pack. Data_check, track_check, and scan_check were modified to deal with certain kinds of disk-reading errors. If at least one valid frame header can be found (but not all five), then these queries report all the known information but without the track-frame period in time or in bytes. These two parameters are replaced by '?'s to indicate unknown and probably an error. This should allow correlators to try to use such flawed data. Bug (re)fix: net2disk and net2out do not generate incorrect error 1001 after the other machine has disconnected. ## 2003 May 20 (day 140); This is a very minor update to fix an error in the makefile for the Jungo driver (in $SS/driver). The 'position?' query was improved: It can now be used during playing, and its play pointer gives an (approximate) answer during playing in progress. Also during playing, polling now sets an error (1002) in case the play pointer is not incrementing. This is not the same as playing halted (end of scan or end of data), which is not considered to be an error. Check for either case with a 'status?' query. ## 2003 May 16 (day 136); This tar file contains a new release from Conduant, 6.0beta, dated May 15, which adds some new features and fixes some errors as noted below. Some of the following changes were made to be in accord with Revision 2.5 of the writeup. Bank mode is always on, and the 'bank_mode' command and query and the 'reset=dismount' and 'reset=mount' commands are no longer supported (except in the old Anova chassis). ROT rate read from ROT broadcasts is now used (instead of a fixed 32e6) to allow for speed_up. (This applies to correlators only and to delayed play only.) Play_rate is not affected. Bug fix: net2disk and file2disk do not generate incorrect error 1001. Bug fix: we think that EndM5 now works on all versions of Linux. Bug fix: disk2file now correctly does default file name. Bug fix: reset=erase now properly clears the directories (but also leaves a copy of the (empty) directory in line on disk, so the first scan starts at byte 81952.) Bug fix: record=off at end of medium (full disks) now works properly. Bug fix: the fourth argument of mode=... now correctly defaults. Most errors are now remembered (even if printed with debuggery) and printed (and cleared) by either a 'status?' or an 'error?' query. Thus errors may be remembered even after they have been corrected. (This may be regarded as either an improvement or a bug fix.) Following Linux tradition, the Mark5A command line now has a help option, -h, which lists the various startup parameters. The 'protect' command and query (write protection) now work except that the 'protect?' query does not work in non-bank mode (Anova chassis), and protection in non-bank mode has not been tested. The 'play=on' command and several other commands now check more carefully for other data transfers in progress and return errors if so. The extra debuggery from a 'bank_mode?' query was moved to 'bank_set?' (because bank_mode command and query are no longer supported). We have augmented the 'status?' query as follows: bit 12 disk2file 0x00001000 bit 13 file2disk 0x00002000 bit 14 disk2net 0x00004000 bit 15 net2disk 0x00008000 bit 16 in2net 0x00010000 bit 17 net2out 0x00020000 (Some of these are still partly ambiguous in this version.) When one of these bits is set, the corresponding command is active (transferring data) or waiting to become active. In this case, other data-transfer commands will return errors. We have further augmented the 'status?' query as follows: bit 20 0x00100000 Bank A selected bit 21 0x00200000 Bank A ready bit 22 0x00400000 Bank A full bit 23 0x00800000 Bank A write protected bit 24 0x01000000 Bank B selected bit 25 0x02000000 Bank B ready bit 26 0x04000000 Bank B full bit 27 0x08000000 Bank B write protected These add, of course, so a typical response might be: !status? 0 : 0x02300001 ; which means that bank A is selected and ready and bank B is ready. This bank-status augmentation of 'status?' works only when data transfers (e.g., recording and playing) are not in progress. Also 'status?' bit 0x000008, delayed completion, is now set whenever any data-transfer activity, such as recording, playing, or transfers to or from disk or net, is active or waiting. The 'rtime?' query now also returns the percentage remaining unrecorded disk space as its third parameter. A new reset command, 'reset=abort', aborts data transfers disk2net, disk2file, and file2disk. (We may expanded this list in the future to abort other operations also.) This command returns immediately, but there may be a delay of up to two seconds before the data transfer stops. During this delay, 'status?' queries will show what's happening. This version of Mark5A was upgraded to set and use the fill-pattern detection in the output section of the I/O board. On playing, fill pattern inserted by the SS card in lieu of unreadable data is converted to an equally long wrong-parity time sequence, which should enable the correlator to maintain synchronization through this unreadable-data segment. This requires version 0x18 or later of the "Output design revision" (ODR, which is the last parameter returned by 'DTS_id?'). A number of small changes in this version make it work better with the old Anova chassis (in non-bank mode, of course). These changes were only partly successful; this version in an Anova chassis generates some spurious error messages. Even though most things work, this is likely to be annoying. ## 2003 April 10 (day 100); The tar file with this date, Mark5A.100.tar.gz, remains available on web for downgrading. Important: This version of Mark5A implements bank mode and contains other major changes. This version of Mark5A can read and play SS disks recorded with older versions, however, older versions of Mark5A can not read or play SS disks recorded with this version. Thus correlators should be upgraded to this version before attempting to read or play SS disks recorded with this version. This version works with the older Anova chassis, but, of course, without bank mode. Important: This version has an upgrade from Conduant, sslinux452beta.tgz, which contains an updated Jungo driver; see notes above. The sspxf.bib file in $SS/bib and the library in $SS/lib are also special later updates. Important: The command-line options for Mark5A have been changed to be in accord with Linux conventions. There are three setable parameters: -m sets the msglev (range -1 to 3, default 1), -f sets formal parsing (0 for FALSE, 1 for TRUE, default TRUE), and -s sets the maximum number of simultaneous control socket connections (range 1 to 7, default 7). At field stations with old versions of the Field System, try: Mark5A -m 0 -f 0 & to give debug printing and informal parsing. At Haystack-designed correlators, try: Mark5A -m 0 & to give debug printing (and formal parsing). Note that parts of page 5 of Mark5A Software are thus out of date. Many of the following changes were made to be in accord with Revision 2.4 of the writeup. Track_check now has track data rate (as in scan_check and data_check) and decoded track number converted to 2--33 or 102--133 notation and followed by a 'D' to indicate a duplicated track (i.e., the track you asked for is correctly a duplicate of this track in this mode), or '?' to indicate an error (i.e., the wrong track was found). The '-' in place of missing bytes was deleted. DTS_id now also shows the command-set revision and the input- and output-design revisions from the I/O board. The input- and output-design revisions will both be 0 in case there is no I/O board (i.e., Mark5P). DTS-id now reads a file /etc/hardware_id for the serial number and uses the system name only if this file is not readable or not the correct length. Bug fix: Corrected error checking in mode command. Bug fix: Rtime now reloads the record pointer before calculating. This allows rtime to give an (approximate) answer during recording in progress. This new Conduant version contains working bank-mode functions, and the bank_select and bank_mode commands and queries have been rewritten to correspond. Also status polling now checks periodically on which bank is selected. This allows the operator to switch banks using just the key switches and disk insertions. (But automatic bank switching during record or play is not yet implemented.) This version from Conduant also contains label commands, which were used to implement VSN. (But the VSN serial-number cross-checking is not yet implemented.) To support VSN, tstMark5A now accepts key words as short as 3 chars. Bank_mode accepts just one parameter, which now defaults to "on", and bank_mode is "on" at startup if in a TK200 chassis. Bank mode can't work in the old Anova chassis. Bug fix: Bank_set now re-reads status and disk information whenever the bank changes. Previous errors newly reinserted in informal-parsing mode: '=' in response to disc_check and disc_serial queries. Bug fix: Dplay() (delayed play, correlator only) now writes an error message and starts playing immediately if the requested start ROT is in the past or less than 0.1 second in the future. Reset has another option for its parameter: "erase_last_scan". Bug fix: Reset=... tries to re-read device status only if disks are mounted. Reset=dismount is illegal in bank mode. Bug fix: Bank_mode=on now redoes the initialization. Bank_set has another option for its parameter: "inc" changes to the other bank. Scan_check, data_check, and track_check now calculate the frame period in bytes using an algorithm that is more likely to show any errors. Scan_check, in particular, is now more persnickety. The point is, after all, to show up errors and problems. Scan_set now also accepts "s+" as its second argument to set the play pointer to 65536 bytes past the start of the scan. Various improvements were made in in2net, which should work now and should report overflow (i.e., can't keep up) to either an in2net or status query. Bug fix: The status query now works properly during recording or playing. During recording, polling now sets an error (1001) in case the record pointer is not incrementing. This is not the same as recording halted for end of medium, which is not considered to be an error. Check for either case with a 'status?' query. Various minor improvements were made especially in error messages and in minor housekeeping tasks associated with bank switching. ## 2003 March 5; Conduant has fixed a bug in their on-board software dealing with LBL for disks greater than 137 Gbytes. If you are using 200-Gbyte SS disks in a Mark 5, then you should upgrade from the February-20 version of Mark5A. The upgrade is in this sspxf.bib file. After upgrading to the February-20 tar file, then put this sspxf.bib file into $SS/bib overwriting the older file with the same name. Then SSReset and restart Mark5A. (Do not apply this sspxf.bib upgrade to later versions.) ## 2003 February 20; This version will show "2003y044d14h" in the response to "DTS_id?" (because the latest Mark-5 update therein is February 13, but the latest Conduant update and the date of the tar file is February 20). The tar file with this date, Mark5A.pre.tar.gz, remains available on web for downgrading. This tar file includes a new sspxf.bib file from Conduant, which should make the in2net command work again. This same bib file also contains a work around for the bug in LBA in Western Digital disk drives with "JB" in the model number. Bug Fix: The mode command now generates an error return on an attempt to go into 64-track mode when the resulting play_rate clockgen would be above 45.36 MHz. The play_rate command also checks this. The actual limit is for the track data rate not to exceed 40 MHz for up through 32 tracks or 20 MHz for 64 tracks. Many of the following changes were made to be in accord with Revision 2.3 of the writeup. On record=off, the scan number is now set to the just-recorded scan, and the play pointer is set to the beginning of this scan. Dir_info is a query only and no longer changes the scan number. The scan_dir query is no longer supported, and, instead, the scan's starting and ending byte numbers were added onto the return from the scan_set query in place of the number of scans. (The number of scans is available from dir_info.) A scan_check query no longer increments the scan pointer. Name changes for this new writeup: Changed track_select to track_set and changed bank_select to bank_set. (But the old names also still work.) Other changes: One of the errors previously fixed was put back in order to work with the old version of the Field System, namely an '=' after the '?' in the return from the position query. This incorrect '=' occurs only in INformal parsing mode (described below), which is necessary for this version of the Field System. The scan_set command now allows "inc" as its first argument, which increments the scan number or, after the last scan, loops back to the first. The track_select command now allows "inc" in place of either or both of the track numbers to mean increment to the next higher-numbered track or loop back to 2 at the end of the list. The second argument of scan_set now defaults to "S", so the play pointer is always set by this command (by default to the start of the scan). Bug fix: Scan_play is now more nearly correct, and play handles end-of-medium in a consistent way. Playing stops at the end of the scan (scan_play) or end-of-medium (either play or scan_play) and updates the play pointer, but the corresponding queries then return "halted" to show what happened. Bug fixes: More incorrect "=" deleted from query replies. Note that many of the comments below for the previous release apply also to this one. ## 2003 January 28; Scan_play now is also a query. Disk_serial, disk_model, and disk_size are now faster; they each should now come in under the one-second limit even with 16 disks. (The delay was moved to the reset=mount sequence, which now also has more debuggery.) Data_check, track_check, and scan_check in case SS or TVG now return three additional numbers in order: starting and ending word numbers (32 bits each) where the pattern was found, and the length of the buffer. In the ideal case, the first number would be 0, and the last two numbers would be equal. Bug fix: Device info is needed on reset=mount. Note that many of the comments below for the previous release apply also to this one. ## 2003 January 24; Dozens of small changes were made for this update. Most were made to be compatible with Revision 2.2 of the writeup. Note particularly that the arguments of the mode command and query were revised, and the returns from data_check, track_check, and scan_check were correspondingly revised. Verify that you have a compatible suman to use this version at a Haystack-designed correlator. A tar file with the previous version ( Mark5A.old.tar.gz, 2002 December 27) remains available on web. Mark5A now has formal and informal parsing modes. In formal parsing mode, most of the '=' and ':' prescribed in the writeup are required. This allows, for example, the sequence ": :" to mean to take the default or the previous value (depending on the specs for the command) for the missing parameter that would be between the two ':'s. In informal parsing mode (the only parsing mode in previous versions of Mark5A), many of these parsing rules are relaxed. In particular, just a space can be used in place of "=" and ':'. A '-' can now be used as a placeholder in informal parsing mode. Also a successful record=on returns 1 (delayed completion) in formal parsing mode but returns 0 in informal parsing mode. Informal parsing mode might be helpful when you are typing commands and queries by hand. At the field stations, you might have to use informal parsing mode: Check your version of the Field System to determine which is wanted. Haystack-designed correlator software expects Mark5A to be in formal parsing mode. This version of Mark5A normally comes up in formal parsing mode. To select informal parsing mode, add a second parameter of 0 on the command line. For example: Mark5A 0 0 & where the first 0 selects debuggery, and the second 0 selects formal parsing mode off, that is informal parsing mode. This version of Mark5A allows functions that don't need to talk with the SS board to work even when disks are dismounted or are being mounted or dismounted. This allows, for example, configuration and interrogation of the I/O board while disks are being changed. There is a new query, rtime, for recording time remaining to end of medium, and the corresponding parameter was deleted from dir_info. Bank_mode and bank_select are present but not yet functional (NYI). Data_check returns blank as its last parameter in case skipped cannot be determined, and this is probably not an error. But scan_check returns '?' in the same situation because this probably is an error. We now try to use '?' and blank consistently in this way. This tar file contains a slightly revised SDK version (sslinux541.tgz, January 16) from Conduant but with the same Jungo driver. Bug fixes: Many changes were made to the handling of play_rate for various modes to make these consistent. These changes are intended to be helpful, but, if in doubt, set play_rate after changing modes. Bug fix: Ignore formatter serial number in VLBA mode. Bug fix: Delete '=' sign in the return from various queries. Bug fix: The spurious error message during reset=mount was programmed around (we still don't know where it's coming from). Bug fix: Mark5A now does a CardReset before an error halt. ## 2002 December 27; This version incorporates almost all the updates and changes in Revision 2.1 of the writeup. Bug fix: Data_check, track_check, and scan_check now do not back over the start of the buffer and now correctly skip a tape-frame header that is truncated by the prescribed starting byte number. Bug fix: A copy of the directory is written to file /var/dir/Mark5A (for DirList to read) at record=off even if the SS disks are full (end of medium). ## 2002 December 18; This release back tracks on a recent change that is incompatible with current versions of the Field System, namely the return code from record=on. This was changed back to 0 pending a release of the FS that understands return code 1 for this case. Bug fix: Input_mode (obsolescent) now handles an odd-numbered formatter error correctly. ## 2002 December 16; Bug fix: A floating-point round-off error in scan_check, data_check, and track_check was fixed by forcing the time offsets between nearby tape frames to be exact integer multiples of 1.25 milliseconds and the time offsets between distant tape frames to be exact integer multiples of this (previously determined) tape-frame spacing. (The infamous quarter-millisecond offset was already being corrected.) Bug fix: At play=off after scan_play, PlaybackLength is set back to 0 to avoid affecting a subsequent play=on; If a reset=mount or reset=dismount fails, it can now be retried. This allows recovery from certain disk-mounting errors without exiting Mark5A. Bug fix: An extra record=off when not recording no longer gives a bogus scan containing only a copy of the directory. ## 2002 December 9; Mark5A functions disc2file and file2disc were updated to Large File Support (LFS), which allows reading and writing files larger than 3 Gbytes. A minor error in file2disc was also corrected. Mark5A on startup with msglev 0 now contains a desperation attempt to read the directory. Try this if you're sure that your disk(s) have a directory but Mark5A can't otherwise read it. This tar file also contains some minor upgrades from Conduant, namely sstest, ssopen, and sspxf.bib. The sstest and ssopen upgrades affect only users with eight-pack modules using both together. The sspxf.bib upgrade fixes the return to bypass after either recording or playing. At the field stations, if your decoder fails between scans (i.e., when you're not recording), then you need this upgrade. The return codes have been updated to agree with the latest draft revision of "VLBI Standard Software Interface Specification---VSI-S" (see VSI-S). Most notably, code 3, formerly "syntax or parameter error," was split into code 3, "syntax error," and code 8, "parameter error." Code 7 now means "no such keyword," for example, mispeld. And code 1, delayed completion, is now used in all appropriate cases including record and play. This might require changes in cooperating software such as the Field System. ## 2002 December 4; This tar file contains a slightly revised SDK version (sslinux54.tgz) from Conduant but with the same Jungo driver. The stand-alone programs Net2file and File2net now support files larger than 2 Gbytes using Large File Support (LFS) under Linux, HP-UX, and perhaps other operating systems. With blocks of data larger than 2 Gbytes, use Net2file with the Mark5A command disc2net to transfer data from SS disks to ordinary files, and use the Mark5A command net2disc with File2net to transfer data from ordinary files to SS disks. Use these schemes even on the same machine (localhost) because the Mark5A commands disc2file and file2disc do not (yet) support large files (LFS). Beware: There are some bugs in manipulating files larger than 2 Gbytes. Under Linux, for example, using '*' and '?' for file-name expansion fails under csh and tcsh but works under sh and bash. Track numbers for the 8- and 16-track cases were revised yet again. The track_check command now returns "-" as its last argument (in place of skipped) if the track number read from the tape-frame header does not agree with the number in track_select. (And this generates a Mark5A warning message if debuggery is on.) Tracks with data that are duplicated (fanned out) on play according to Mark 5A Track Mapping, tables 1 and 2, are now also checked by track_check, but, if you track_select a duplicated track, then "-" instead of skipped will show that the track number in the tape-frame header is not the same as the track number that you requested. For VLBA mode, track numbers are checked against table 18, page 24 in Mark IIIA/IV/VLBA Tape Formats, Recording Modes and Compatibility. If incorrect, the warning message will show the expected and found VLBA track numbers from this table. ## 2002 November 27; Commands to the I/O board are now sent only when the corresponding parameters have changed. This allows, in particular, changing track_select during play without losing sync. Scan_play is partly functional: It starts playing at the start of the scan, plays to the end, and stops. But it does not go into bypass mode at the end of playing, and it does not update the play pointer, unless or until a play=off. Readback of VLBA mode in the mode query was fixed. Scan_set will now accept either b (for beginning) or s (start) for the starting byte position of a scan. The test program, tstMark5A, now uses readline() instead of fgets() to read what you type. This allows line editing and history recalls of previous commands and queries. See the man or info pages on readline. But this readline() version of tstMark5A.c no longer compiles on HP-UX and other platforms, alas. The April-30 version, which works on HP-UX, is available as tstMark5A.c.bak alongside the tar file. Don't forget to re-install the Jungo driver if you're upgrading from a much older version. ## 2002 November 25; This tarball contains yet another new release from Conduant with yet another new driver from Jungo. Important: A Jungo driver re-installation is required after installing this tarball. Other changes include: Revision and SMART information were added to the debuggery that prints during initialization. A check for odd-numbered formatter serial numbers was added, and mode=... returns an error if an attempt is made to configure an odd-numbered formatter for a parity-striped mode. The track numbers in 8- and 16-track modes were changed again. Scan-set returns an error if it can't interpret its second argument. Data_check, scan_check, and track_check now all work correctly in 64-track mode provided that the paired tracks (i.e., the tracks with 100 added to or subtracted from the track numbers) are configured and have valid sync. If an incorrect track number is found by track_check and debuggery is on, then a warning message is printed. There are also several minor bug fixes. ## 2002 November 18; Important: The check for even-numbered formatter serial numbers (noted below) was removed from this tarball because some in-use formatters (notably at Westford) still have odd-numbered serial numbers. Most of the changes in Revision 1.9 of the writeup have been made in this tarball, but some of the obsolete commands and queries, such as input_mode and play_mode, have also been retained. There are several minor bug fixes. ## 2002 November 15; Bug fixes: data_check, track_check, and scan_check should now work correctly with any even-numbered formatter. Clock frequency should now be correct for 64-track mode. A messy business with a play=off (from suman on the correlator) interfering with a reset=dismount or reset=mount seems to have been fixed. Also some, but not all, of the changes in Revision 1.9 of the writeup have been incorporated. ## 2002 November 14; Bug fix: The track_check query was flawed in the previous version. The interactions between the results returned by data_check, track_check, and scan_check and the commands to the I/O board (input_mode and play_mode) have been removed. This update does not yet incorporate the changes in Revision 1.9 of the writeup. ## 2002 November 12; This tarball contains many updates summarized as follows: Conduant has now incorporated the new Jungo driver with scatter/gather. This improves the speed of eVLBI operations. A Jungo driver re-installation is required after installing this tarball. To be compatible with this new Jungo driver, mmap(), as used to write the directory file, was replaced by ordinary write()s. The directory reading at startup is now more robust. The play_rate query now returns two frequencies, the effective rate as specified, and a new third (last) parameter: the actual rate including parity bits (*9/8) and, with VLBA format, also the header bits (*126/125). (Is this correct for the VLBA correlator?) The dir_info query now returns the estimated recording time remaining (in seconds) as its fifth (last) parameter. This parameter is accurate provided that the input_mode, play_mode, and play_rate parameters are correctly set. The track numbers are now correct for the 16- and 8-track cases. The efficiency of scan_check, data_check, and track_check was improved. A test that the bit preceding sync must be 0 was added to reduce false sync detections. (The new I/O board also needs this.) Another stand-alone program, DirList, for directory listing, was added to this tarball; see the writeup on page 7 of Mark5A Software. ## 2002 October 10; This tarball contains many upgrades summarized as follows: Mark5A now talks with the new (under development) combined I/O board. On machines with the old separate input and output boards, Mark5A detects this and falls back to the old stub behavior. Many small changes are associated with these new capabilities. The new outBoard() and inBoard() are in a separate file, IOBoard.c. Other fixes include: Revised wait-for-start-play (used at the Haystack correlator) and improved interrupt of this wait on abort. TVG clock was deleted from input_mode (there is no such clock). Mark5A now checks for and rejects negative play pointer. The status query now works almost always. Get_stats query now increments the drive number even if error or no such drive. Day number in data_check, track_check, and scan_check was corrected (+1) in case VLBA format. But this release does NOT contain the new faster Jungo driver. ## 2002 September 23; Fixed track_select to accept and reject the correct track number ranges (accept 2 to 33 and 102 to 133). ## 2002 September 23; Several minor fixes, which should avoid "Segmentation violation" on scan_check, data_check, or track_check. ## 2002 September 19; Fixed status query so that it is mostly correct even when recording or playing. ## 2002 September 18; Fixed the spurious WARNING message on startup in case no slave drives. ## 2002 September 18; Added an unauthorized additional parameter to set_scan or scan_set: "start" or "s" sets the play pointer to the start of the scan, "c" or "center" sets the play pointer to the center of the scan, and "e" or "end" sets the play pointer to 1000000 bytes before the end of the scan, which allows data_check or track_check to start there. ## 2002 September 18; Increased read-buffer size to accomodate 64-track mode (in data_check, track_check, and scan_check). This takes some extra milliseconds in case a check fails. ## 2002 September 17; Scan_play partly works---it starts playing the proper scan but doesn't stop at the end of the scan. Use play=off to stop. ## 2002 September 16; Scan_check now increments the scan counter even if it can't decode VLBI data (but not on errors). ## 2002 September 16; Set_scan or scan_set does NOT update play pointer to that scan. (Reverse change below.) ## 2002 September 16; Record query now returns "halted" in lieu of "off" if end of medium. (Play query already had a "halted" if end of recording.) ## 2002 September 16; Record query now returns scan number and scan name. ## 2002 September 16; Set_scan or scan_set command now updates play pointer to that scan. ## 2002 September 16; Fixed short-file error on disc2file. ## 2002 September 16; Added time and speed debuggery for disc2net and disc2file. ## 2002 September 13; Fixed (actually a work-around for) the missing-directory-after-reboot problem. ## 2002 September 10; Fixed date and time "Error" in data_check, track_check, and scan_check with VLBA-format data. Other minor changes especially to debug prints. ## 2002 September 4; Scan_set and set_scan both work, are equivalent, and are now also queries. ## 2002 September 3; Scan_check now seems to be working. This query is recommended for Field System use after each scan in lieu of data_check. ## 2002 August 30; Not-erased disk hangs up trying to read directory: No fix, but now message says what's wrong. Try SSErase. ## 2002 August 30; Revised sspxf.bib (was faulty in 5.3.beta) from Conduant. Skip query results for negative skip are now correct. ## 2002 August 29; New API and bib files from Conduant: 5.3.beta. As a result, skip as query now works. Fixes also to skip command. ## 2002 August 28; Two fixes to play: Dplay now rejects waits greater than one minute, and play=off cancels a wait in Dplay, if any. ## 2002 August 28; Added buffer-size parameters as unauthorized additional parameters to in2net and disc2net or disk2net (sndbuf (default 16384) and socbuf (default SOCBUF = 131072) bytes) and to net2out and net2disc or net2disk (rcvbuf (default 87380) bytes). See setsockopt(). These are used for speed "tuning" of eVLBI. ## 2002 August 28; Scan_set and scan_dir are implemented, and next_scan is retained as a synonym for scan_dir (with added scan number). ## 2002 August 23; Minor fixes to net2disc and disc2net. ## 2002 August 22; Two new Mark-5-associated stand-alone programs, Net2file and File2net, are ready for beta testing and are included in this tarball. Instructions will be ready soon; meanwhile read the headers of the source files, Net2file.c and File2net.c. ## 2002 August 19; Added "skipped" as the last parameter on data_check and track_check. This shows the number of bytes skipped from the last previous data_check or track_check and should be zero if both are within the same scan. ## 2002 August 16; Changed ROT timing for Dplay() (used only at correlator). ## 2002 August 15; Fixed track_check to check just the prescribed track. (Formerly it wandered around on other tracks.) ## 2002 August 13; Net2disc now accepts and shows scan name. ## 2002 August 12; Track_check seems to be working. Use track_select to set which track to check (but with no effect on hardware). ## 2002 August 8; Modifications to date and time printed from data_check? We now attempt to calculate and print the year and day of the year as a best guess, but these will be wrong if data were taken more than about 10 years ago for Mark-4 format or 1000 days (about 2.74 years) ago for VLBA format. ## 2002 August 7; Another fix to VLBA time in data_check? (Day number will be 0.) ## 2002 August 6; Disc2file and file2disc seem to be working. ## 2002 August 5; Start_stats and get_stats now conform to the published specs except for default values. ## 2002 August 2; New libssapi.a from Conduant. Start_stats and get_stats now work but with counts instead of seconds for bin definitions. ## 2002 August 2; fixed VLBA time in data_check? New sspxf.bib from Conduant. ## 2002 August 1; fixed disc_size?, disc_serial?, and disc_model? for case multiple masters and no slaves. ## 2002 July 31; fixed disc_model? to include the whole string. ## 2002 July 31; fixed status check on data_check? Revised skip. ## 2002 July 30; fixed glitch at end of response to DTS_id? ## 2002 July 25; new functions: SS_rev1, SS_rev2, OS_rev1, OS_rev2 ## 2002 July 24; new libssapi.a from Conduant ## 2002 July 24; new API (5.3.alpha_forMIT.tgz) from Conduant ## 2002 July 12; new ssatap3.bib from Conduant Revised: 2005 May 29, JAB