dataRecv

Receive processed data from the telemetry stream and store it as runs.

Syntax:

dataRecv [path]

Options:

path

the path to a raw data directory of the form: /Data/$Experiment/Data.raw/$year/$month/$day

Examples:

dataRecv is ordinarily started up by drext over a network socket, which handles setting up the path argument for it. If for some reason the raw data stream gets stored but runs do not get created or get corrupted, the processing can be replayed:

Take the input from a hpf2 data stream started on Aug 07, 2000 and store it in run files:

cat raw.dat.bak > dataRecv /Data/hpf2/Data.raw/2000/Aug/07

Description:

The dataRecv script takes the telemetry stream after it has been processed by drext and stores it into the kinetics data heirachy. It additionally starts up various other processing and analysis on the data via other scripts.

The basic functionality of dataRecv is to take the text input stream and copy it into the correct stream/raw.dat file (based on the command-line or stream-embedded !Path command). It also watches for commands in the input, however, and when these commands begin runs (e.g. !Begin Run), dataRecv will create a second output file ($runNum/raw.dat, where $runNum is a three-digit number identifying the run) and duplicate the output to there.

dataRecv invokes several other scripts to perform additional processing:

• findCalibRuns
• makeDayLinks
• makeRunLinks
• checkRun
• processRun

Exit status:

For the most part, whenever dataRecv runs into trouble, it logs an error message to day.log and/or /tmp/$Experiment.err.

Files:

• path/stream/raw.dat: entire stream of raw data for the entire set of runs (the beginning of which marks a "day" which determines the date embedded in the path)
• path/stream/day.log: Significant features of the stream processing are logged to this file
• path/stream/current.rid: A file that just keeps track of the last-used run number
• path/stream/done.mark: When it seems that the set of runs completed without errors, this file is created; during the processing of the next set of runs, the superfluous stream/raw.dat will be deleted
• path/$runNum/raw.dat: This contains an exact duplicate of a portion of the stream/raw.dat file between !Begin and usually !End.

See also:

• drext

Caveats:

dataRecv is currently in a transition period between Neil's version and a hopefully more robust system. As the stream format is more tightly defined, dataRecv may change to autodetect certain features of the current stream and react differently.

---

last updated: Thu Aug 7 14:46:35 2003	webmaster@huarp.harvard.edu
Copyright 2002 by the President and Fellows of Harvard College
[Home] [People] [More Info] [Research Areas] [Field Missions] [Engineering]