The purpose of this paper is to define the ALERT/IFLOWS gage data format commonly called the Enhanced IFLOWS Format
(EIF). EIF was developed to provide more robust message transmissions from radio reporting gages
to base station computer systems. A key constraint was to maintain compatibility with existing message formats
so gages using any combination of formats can coexist in a given area on a single frequency.
Background
ALERT/IFLOWS gages (hereinafter simply gages) are electronic devices that transmit a message, generally by VHF or UHF radio, whenever a monitored event occurs. Messages contain two fields: an address, which indicates which sensor is reporting, and event data, which describes the event in some way. Gages can support one or more sensors whose addresses are clustered about a base address assigned to the gage when it is installed.
Messages consist of four 8-bit bytes transmitted asynchronously at 300 b/s using FSK modulation. Each byte is preceded by a single start bit (space or logical 0) and followed by a single stop bit (mark or logical 1). Bits within each byte are transmitted in order from the least to the most significant.
Format depictions in this paper show the bytes in the order transmitted, from top to bottom. The low-order bit in each byte appears rightmost. Bits within a field are subscripted to show their relative numeric significance with 0 being the least significant. Unadorned numeric constants are decimal; hexadecimal constants have a subscript of 16, e.g., 3016.
Two message formats have become traditional standards. The older (and less useful) is the ASCII Data Format (ADF) illustrated below.
|
x |
0 |
1 |
1 |
AU3 |
AU2 |
AU1 |
AU0 |
|
x |
0 |
1 |
1 |
AT3 |
AT2 |
AT1 |
AT0 |
|
x |
0 |
1 |
1 |
DU3 |
DU2 |
DU1 |
DU0 |
|
x |
0 |
1 |
1 |
DT3 |
DT2 |
DT1 |
DT0 |
x is ignored; it can be 0 or 1.
AUi is the ith bit of the units digit of the sensor address.
ATi is the ith of the tens digit of the sensor address.
DUi is the ith bit of the units digit of the event data.
DTi is the ith bit of the tens digit of the event data.
IFLOWS software ignores the high-order bit of each byte because gages from various vendors encode this bit in various ways. The remaining seven bits form an ASCII decimal digit, i.e., a byte in the range 3016 - 3916. Address and data fields are two digits each, limiting their values to the range 0 - 99.
The second traditional format is the Binary Data Format (BDF) shown below.
| 0 | 1 | A5 | A4 | A3 | A2 | A1 | A0 |
| 0 | 1 | A11 | A10 | A9 | A8 | A7 | A6 |
| 0 | 1 | D4 | D3 | D2 | D1 | D0 | A12 |
| 0 | 1 | D10 | D9 | D8 | D7 | D6 | D5 |
Ai is the ith bit of the sensor address.
Di is the ith bit of the event data
The two high-order bits of each byte identify the format, leaving 24 bits for the message. Thirteen bits are reserved for the address, allowing a range from 100 - 8191 (addresses below 100 are not permitted since most gages automatically transmit data from sensors with such addresses in ADF). Eleven bits are reserved for event data, allowing a range from 0 - 2047.
Introducing EIF
The ADF and BDF share a major shortcoming: they provide limited means for detecting errors introduced during message transmission. Base station software uses validation to ensure that a series of observations follows a "reasonable" pattern, but this technique has significant vulnerabilities, especially where multiple repeaters are used. Moreover, validation cannot be used by repeaters to suppress retransmissions once an error has occurred. EIF was designed to enhance validation algorithms and improve repeater efficacy by incorporating an error detection mechanism into each message. EIF message structure is shown below.
| 1 | 1 | A5 | A4 | A3 | A2 | A1 | A0 |
| D0 | A12 | A11 | A10 | A9 | A8 | A7 | A6 |
| D8 | D7 | D6 | D5 | D4 | D3 | D2 | D1 |
| C0 | C1 | C2 | C3 | C4 | C5 | D10 | D9 |
Di is the ith bit of the event data
Ci is the ith bit of the frame check sequence (FCS), defined below.
EIF uses the two high-order bits of the first byte to indicate the message format. Since no valid ADF or BDF message has 11 in these bits, all formats remain uniquely identifiable. Like BDF, EIF provides a 13-bit address and an 11-bit data field, so any BDF message can be converted to EIF without data loss. EIF requires that messages from sensors with an address below 100 conform to the ADF rule that data values range from 0 - 99. In this way, ADF messages can be converted to EIF, retransmitted, and properly interpreted by subsequent receivers, i.e., sequential event data values reset to 0 after 99 rather than after 2047.
The Frame Check Sequence (FCS)
The FCS is a six-bit cyclic code added to each message to help receivers determine whether the message suffered errors during transmission. Most gages transmit by VHF or UHF radio, so messages may be altered by mutual interference (many gages operate on a common frequency without coordination), multiple receiver paths, and random noise, such as that caused by lightning. A well-considered FCS can help detect such transmission errors.
Details of cyclic code theory are beyond the scope of this paper, but may be found in most advanced telecommunications textbooks. We do, however, present a basic primer herein. Throughout this discussion, an EIF message is assumed to comprise 32 bits; the 4 start and 4 stop bits present in each transmission are ignored because they are stripped by receiver hardware and consequently not seen by receiver software.
Consider each bit in an EIF message to be the coefficient of a 31st-order polynomial of some arbitrary variable x. The first bit transmitted (A0) is the highest-order coefficient, i.e., the coefficient of x31; the last bit transmitted (C0) is the lowest-order; i.e., the coefficient of x0. Polynomial arithmetic is done modulo-2 without carries or borrows so addition and subtraction are identical to exclusive or (XOR).
When an event occurs, the gage assembles the sensor address and event data into a message and clears the FCS bits, forming a polynomial called M(x). It then divides M(x) by a 6th-order generator polynomial, called G(x), described below. The remainder polynomial is then subtracted from M(x), yielding a transmission polynomial T(x) that is evenly divisible by G(x). It is T(x) that is transmitted by the gage.
To allow for errors, we express the message received as T(x) + E(x), where each non-zero coefficient (if any) in E(x) represents a single bit that became inverted during transmission. The receiver divides [T(x) + E(x)] by G(x) and considers the remainder. If it is zero, the receiver assumes the message experienced no transmission errors and processes the address and event data fields. Otherwise, the message is deemed invalid and discarded.
The remainder of [T(x) + E(x)] / G(x) equals the remainder of T(x) / G(x) plus the remainder of E(x) / G(x). The FCS calculated by the transmitter makes the first term 0, so the receiver calculation yields the remainder of E(x) / G(x). This result is 0 only if E(x) is 0 (meaning no errors occurred) or has G(x) as a factor. In the latter case, the message is accepted as valid even though errors occurred. By choosing G(x) carefully, we can minimize the number of such undetected errors and ensure that most common errors are detected.
The generator polynomial chosen for EIF, x6 + x4 + x3 + 1, satisfies these criteria. It detects the following errors:
Generating And Checking The FCS
Although the above discussion may appear mathematically complex, the generation and checking of the FCS is computationally simple, as the pseudocode below indicates. To generate an FCS prior to transmission, set Ai to the sensor address, Di to the event data value, and Ci to 0. Then calculate the remainder using the algorithm below and place it in Ci . To check a received FCS, process the 4 bytes as received. If the remainder is non-zero, the message experienced errors in transmission and should be discarded. Otherwise, it may he assumed to be correct.
This algorithm assumes the following registers are available.
Initialize:
BytePtr = address of first byte of message
ByteCnt = 4
Rem = 0
DoNextByte:
Byte = message byte at BytePtr
BytePtr = BytePtr + 1
BitCnt = 8
DoNextBit:
Shift all bits in Rem right one position
Rem7 = Byte0
Shift all bits in Byte right one position
If Rem1 = 1 then Rem - Rem XOR 9A16
BitCnt = BitCnt - 1
If BitCnt 0 then GoTo DoNextBit
ByteCnt = ByteCnt - 1
If ByteCnt0 then GoTo DoNextByte
EndOfRoutine
An examination of the pseudo-code beginning at DoNextBit reveals that the algorithm simply implements the well-known
method of manual long division. Successively lower-order dividend coefficients are "brought down" into
the low-order position of the remainder until the remainder has a non-zero x6 coefficient. At that time,
the generator polynomial (divisor) is subtracted (via exclusive or) from the remainder. The process continues,
bit by bit, until all 32 bits have been processed. The remainder coefficients are returned in the Rem register
as described above.
Implementation Issues
Present and future IFLOWS software will support any mix of ADF, BDF, and EIF gages. However, it is likely that EIF will see increasing use, especially as older gages are replaced and new gages installed, due to its enhanced error detection capabilities. EIF was designed for easy and inexpensive implementation by equipment manufacturers. For example, EIF uses the same signaling, frequencies, 4-byte message format, 13-bit address, and 11-bit event data as BDF, so no changes to hardware should be necessary. EIF simply rearranges the fields within a message and adds an easily calculated FCS, so only minor firmware changes should be required for its implementation.
Base station software developers should also be able to add EIF support readily. When the software has determined that a message follows neither ADF nor BDF specifications, the message should be tested for EIF (two high-order bits of first byte equal to 11 and a valid FCS). If the message passes, the address and event data fields are extracted and processed as if the message were ADF for addresses less than 100, or as if it were BDF otherwise.
To provide maximum vendor flexibility while ensuring compatibility among various devices, the following guidelines have been developed.
Differences Between EIF and EAF
The principal reason for creation of EAF and EIF was to incorporate a CRC, which has a demonstrated superiority over ADF and BDF. EIF and EAF differ in two areas, however--battery status and wind gust/CRC substitution. EAF provides a "battery status" bit by shortening the gage identifier field from 13 to 12 bits, consequently reducing the number of unique identifiers from 8192 to 4096. EIF has no such battery status bit. Instead, EIF suggests that battery condition should be treated like any other measurement; i.e. the sensor should be assigned an identifier and report battery voltage, rather than a simple good/bad indication. Aside from the obvious benefit of determining the health of the battery itself, the voltage measurement makes it possible to determine whether a solar panel is working by looking at changes in voltage during night and day, in much the same way one determines whether a stream is rising or falling by looking at consecutive reports of water depth.
The second difference between EAF and EIF relates to wind gusts. EAF allows wind sensors to substitute gust information for the CRC bits. EIF does not permit this, because it adversely impacts base station software. Under EAF, base station software cannot determine whether to validate the CRC until it determines whether the sensor is reporting wind gust data. The only way to do this is to check the identifier before validating the CRC. Any bit error that converts a rain gage identifier, for example, to a wind sensor identifier will go undetected even though the report was sent with error-control enabled. The situation becomes more complicated with single-frequency (aka store-and-forward) repeaters. Unless every EAF-aware repeater knows which identifiers represent wind sensors it cannot validate any CRC's and therefore must blindly repeat all EAF messages including those already crippled by transmission errors.
Effects of Random Noise on Gage Formats
EAF (discounting wind gust reporting), EIF, and binary formats all encode 24-bits of intelligence in a 32-bit (ignoring start and stop bits) envelope. All 16,777,216 data bit combinations are valid, but for each, the remaining 8 bits must form a singular pattern or the message is marked invalid. Thus in the presence of pure random noise, one spurious message in 256 (about 0 .4%) appears valid regardless of message format.
IFLOWS sites normally monitor a small number of local gages, usually fewer than 100, and rely on error-controlled wide area networks to span large geographic areas. Consider a site monitoring 128 sensors in a random noise environment. The site accepts only 128 of the 8192 (EIF and binary) or 4096 EAF possible identifiers. so only 1.6% (EIF and binary) or 3.1% (EAF) of properly formatted messages are processed further. Since 0.4% of spurious messages pass the format test, about 6 (EIF and binary) or 12 (EAF) spurious messages in 10,000 will be accepted. Of these, many will be subsequently rejected by data validation algorithms.
Because gage reports do not have a distinctive frame, maintaining a timer is required. IFLOWS starts a timer upon receipt of the first character of a (potential) message. If three more characters are not received before the timer expires, IFLOWS discards what was received and awaits the first character of the next message. Since the timer cannot be started until the first character arrives, its minimum duration is 100 ms (the time needed to receive 3 characters at 30 cps). Because of the coarseness of PC-DOS timers (resolution is about 55 ms), the PC IFLOWS window averages 275 ms and it has had little problem with random noise.
Perhaps the best way to prevent bad data arising from random noise is to detect the noise. As seen from the analysis above, random noise is a problem only if it is continuous, or nearly so, resulting in an excessively high number of spurious messages. By counting the number of messages received over a period of time (5 minutes, for example) and the ratio of invalid messages, software can determine whether noise is an issue. Long duration noise is most often caused by an open receiver squelch circuit, so an appropriate message to a user to investigate may be in order.