Introduction to Optical Transient Detector (OTD) Lightning Product

Table of Contents

The OTD Instrument and Its Data
OTD Data Format and the NCSA HDF Libraries
The LIS/OTD Read Software
Obtaining Software
OTD HDF File Structure
References
Ordering Optical Transient Detector Data Sets

The OTD Instrument and Its Data

The Optical Transient Detector is a scientific payload on the MicroLab-1 satellite, which was launched into orbit by a Pegasus rocket in April of 1995. The primary mission of the OTD instrument is to improve the understanding of thunderstorm distributions, cloud processes, and storm variability by detecting and locating lightning activity over large areas of the Earth's surface. The concept for this instrument was developed at NASA's Marshall Space Flight Center in the 1980s, and was selected for development as part of NASA's Earth Observing System (EOS). The OTD instrument is a flight-qualified engineering model for the Lightning Imaging Sensor (LIS), which was launched in November of 1997 on the Tropical Rainfall Measuring Mission (TRMM) satellite.

The OTD instrument detects and locates the lightning discharges that occur within its field-of-view, marks the time of occurrence of the lightning, and measures the radiant energy. The spatial resolution of the instrument is 10 km and the temporal resolution is 2 ms. The OTD detects both intra-cloud and cloud-to-ground discharges during day and night conditions with a high detection efficiency.

The orbital trajectory of the MicroLab-1 satellite has allowed the OTD to circle the earth once every 100 minutes at an altitude of 740 km. Using its 128 x 128 pixel photo-diode array and wide field-of-view lens, the OTD instrument is capable of viewing a total area of 1300 km x 1300 km. Given the field-of-view and the orbital trajectory, the OTD can monitor individual storms and storm systems for about 4 minutes, a period long enough to obtain a measure of the lightning flashing rate in these storms.

The data obtained by the OTD instrument includes the "background images" as well as "lightning data". A background image is roughly a "snapshot" of the field-of-view of the instrument. These images are recorded about once every minute by the instrument, although the time between images varies according to the amount of lightning data being processed. The "lightning data" is simply the optical pulses that OTD detects while passing over active thunderstorms. These optical pulses are assembled into individual classes, namely "EVENTS", "GROUPS", "FLASHES", and "AREAS".

An EVENT is the smallest granule of OTD data. An EVENT is the recorded by the data system whenever the value at a particular focal plane pixel exceeds the background count at that pixel by a threshold value. A GROUP is defined as one or more adjacent events in the same 2 ms time frame. A FLASH is defined as one or more GROUPS sufficiently close in space in time to be classified as a single FLASH. An AREA is defined as a grouping of one or more FLASHES that are sufficiently separate from existing AREAS. For more details of the lightning data classes, see the Algorithm Theoretical Basis Document (ATBD) for the Lightning Imaging Sensor (LIS) at URL http://eospso.gsfc.nasa.gov/eos_homepage/for_scientists/atbd/docs/LIS/atbd-lis-01.pdf.

OTD Geolocation

All of the lightning data recorded by the OTD was geolocated using the SPICE Toolkit software, which was developed at the Navigation Ancillary Information Facility at Caltech's Jet Propulsion Laboratory. The SPICE Toolkit provided a means to conveniently store and access the ephemeris data for the OTD instrument using data sets called "kernels". The OTD "kernels" were constructed with SPICE software by combining the fixed sensor/platform geometry with the continuous stream of navigational information supplied by the MicroLab-1 satellite.

Quality Control of the OTD Data

OTD exists in a noisy space environment. It also responds to a number of optical signals, not all of which are necessarily lightning-related. A significant amount of software filtering has gone into the production of science data distributed to the science community. The filters maximize both detection efficiency and confidence level so that each datum is signal, and not noise.

Each OTD lightning event in an OTD file is tagged with four low-level quality indicators, while each OTD data file is assigned four high-level flags that were designed to notify potential users of possible irregularities in the data file. An automated process is used to tag each optical event in the OTD data file with a set of four numbers that indicate the relative likelihood that the event was produced by lightning, as opposed to solar glint, energetic particles in the Van Allen radiation belts, or electronic noise. These low-level tags are as follows:

  1. Non-noise Probability (the probability that the event is not caused by random noise or energetic particles).
  2. Solar Glint Factor (a number that indicates the likelihood that the event was caused by direct reflected solar radiation).
  3. Event Rate Ratio (a number that represents the ratio of "accepted" events to the raw detected events during a one-second period at the time of the event).
  4. Probability Density (a number that indicates whether the event is geolocated in the vicinity of other events that are likely to be lightning).

In addition, each OTD data file is manually inspected for irregularities in the data set. The data files that fail specific quality assurance are flagged. The high-level quality flags assigned to each OTD HDF data file are contained in the file OTD_QA.dat included with the LISOTD software. The structure of each line in this ascii file is as follows:

YY JJJ XXXX A B C D E

where YY is year, JJJ is julian day of year, XXXX is orbit number, and A,B,C,D and E are as follows:

A. This file has no valid data and shouldn't be used (not supposed to be distributed)
B. The file contains an irregular number of events that appear not to have been caused by lightning.
C. The file contains an instrument view time that is inconsistent with the period of the orbit or the periods of data availability.
D. The file contains data beyond normal orbit length or the file is tagged with an irregular orbit start or stop time.
E. The ephemeris used to geolocate some or all lightning events in the file is questionable and/or the satellite is out of control.


It is recommended that if any values occur in A-E that that file not be used for climate applications since they may skew flash rate calculations. These files are included in the distribution because useful data occurs within some portion of the dataset.

The Format of the OTD Data and the NCSA HDF Libraries

The data obtained from the OTD instrument is stored in a Hierarchical Data Format (HDF) format, the standard format for EOS projects. HDF is a platform independent data format used for the storage and exchange of scientific data. To read files written in a HDF format, it is necessary to obtain software from The National Center for Supercomputing Applications (NCSA) which is available via the Internet at URL http://hdf.ncsa.uiuc.edu/. NCSA provides a public domain library supporting HDF on a wide variety of computer platforms. Instructions on how to obtain the NCSA HDF libraries are provided in 5 of this document.

The LIS/OTD Read Software

A new software package has been written which supersedes previous OTD read software. The LIS/OTD Read software was designed to read both OTD and LIS HDF files. While both datasets are in slightly different HDF formats, both can be read by this software package. The following is from the introductory chapter of the users manual for the software package:

This document serves as a guide to the software intended for use with satellite data from the Optical Transient Detector (OTD) and Lightning Imaging Sensor (LIS). The software suite consists of both fully featured GUI (Graphical User Interface) driven applications, and collections of high- and low-level APIs (Application Programming Interfaces). The software is designed to simplify, as much as possible, user access to the OTD and LIS lightning data sets, which are currently distributed in HDF (Hierarchical Data Format) files. The suite is designed with four goals in mind: simplicity, reusability, compatibility and deployment. By providing software strongly tailored to these goals, we hope to minimize each user's time spent accessing and managing the datasets, and maximize the time spent actually analyzing them.

OTD HDF File Structure

The HDF file structure for the OTD HDF files along with brief descriptions of the data fields are provided in TABLE 1. For the definitions and full descriptions of the data fields please refer to the Algorithm Theoretical Basis Document (ATBD) for the Lightning Imaging Sensor (LIS)".

OTD HDF Data File Structure

NOTE: TAI93 is SI seconds counted from UTC midnight Jan. 1, 1993. The SI second is derived from atomic clocks at the National Institute of Standards and Technology (NIST).

NOTE: [ ] indicates the number of times the variable is repeated.

NOTE: Intensity has the units of micro-J/m-m/sr/micro-m.

TABLE 1

Variable name Type Description
Vgroup 0: Area Statistics
Vdata Set 0: Area Statistics
seq int32 Area creation sequence number
TAI93 float64 TAI start time of the area
delta float32 Lifetime of the area (in seconds)
view float32 OTD viewing time to the area (in seconds)
s-z-a int16 Solar Zenith Angle
d-n-t character Flag for day(d), night(n) or twilight(t)
end status character Grouping algorithm status flag for area (D=naturally closed, K=unnaturally closed)
events int16 Number of events that make of the area
cent[2] float32 Average latitude/longitude of the area (to 0.02 deg)
stdev[2] float32 Latitude/Longitude 'size' of the area (to 0.02 deg)
loc count int16 Number of different lat/long pairs that make up area
rad float32 The total summation of the events intensity that make up the area
orbit id int32 The index of the parent orbit of this area
day int32 The day number of the area
children int32 The number of lashes that are part of the area
child seq int32 The creation sequence of the first child flash
child rec int32 The index of the first child flash
QA[4] int16 Quality flags derived from child flash QA
[The Vdata set is repeated for each area found during orbit.]
Vgroup 1: Flash Statistics
Vdata Set 0: Flash Statistics
seq int32 Flash creation sequence number
TAI93 float64 TAI start time of flash
delta float32 Lifetime of the flash (in seconds)
view float32 OTD viewing time of the flash (in seconds)
s-z-a int16 Solar zenith Angle
d-n-t character Flag for day(d), night(n) or twilight(t)
end status character Grouping algorithm status flag for flash (D=naturally closed, K=unnaturally closed)
events int16 Number of events that make up the flash
cent[2] float32 Average latitude/longitude of the flash (to 0.02 deg)
stdev[2] float32 Lat/Long 'size' of the flash (to 0.02 deg)
loc count int16 Number of different latitude/longitude pairs that make up the flash
rad float32 The total summation of the events intensity that make up the flash
parent seq int32 The index of the parent area of this flash
parent rec int32 The HDF record # of the parent area of this flash
children int32 The number of groups that are part of the flash
child seq int32 The creation sequence of the first child group
child rec int32 The index of the first child group
QA[4] int16 Quality flags derived from child group QA
[The Vdata set is repeated for each flash found during orbit.]
Vgroup 2: Group Statistics
Vdata Set 0: Group Statistics
seq int32 Group creation sequence number
TAI93 float64 TAI start time of group
delta float32 Lifetime of the group (in seconds)
view float32 OTD viewing time of the group (in seconds)
s-z-a int16 Solar zenith Angle
d-n-t character Flag for day(d), night(n) or twilight(t)
end status character Grouping algorithm status flag for group (D=naturally closed, K=unnaturally closed)
events int16 Number of events that make up the group
cent[2] float32 Average latitude/longitude of the group (to 0.02 deg)
stdev[2] float32 Lat/Long 'size' of the group (to 0.02 deg)
loc count int16 Number of different latitude/longitude pairs that make up the group
rad float32 The total summation of the events intensity that make up the group
parent seq int32 The index of the parent area of this group
parent rec int32 The HDF record # of the parent area of this group
children int32 The number of groups that are part of the group
child seq int32 The creation sequence of the first child event
child rec int32 The index of the first child event
QA[4] int16 Quality flags derived from child event QA
[The Vdata set is repeated for each group found during orbit.]
Vgroup 3: Event Statistics
Vdata Set 0: Event Statistics
event # int32 Event creation number
seq # int32 Event HDF record number
TAI93 Float64 TAI start time of event
s-z-a int16 Solar Zenith Angle
d-n-t character Flag for day(d), night(n) or twilight(t)
x pixel int16 X pixel value of the event
y pixel int16 Y pixel value of the event
raw radiance int16 Event amplitude in counts
cal radiance float32 The calibrated event intensity
parent seq int32 The index of the parent group of this event
parent rec int32 The HDF record # of the parent group of this event
QA[4] int16 Quality flags
location[2] float32 The Earth latitude/longitude of the event
[The Vdata set is repeated for each event found during orbit.]
Vgroup 4: Browse Statistics
Vdata Set 0: Browse Area
location[2] float32 The latitude/longitude of the 2.5 x 2.5 degree browse grid point
EGFAs[4] int16 Number of events/groups/flashes/areas with centroid at the ground location
EGFA cal rad[4] float32 Total summation of calibrated intensity of the event/groups/flashes/areas with centroids at the ground location
max rate float32 Maximum flash rate found at the ground location
1 event areas int16 Number of areas at the ground location with only one event
highres locs int16 The number of full instrument resolution points
location 1st highres int16 The index of the first full instrument resolution Vdata associated with this browse location
bkgnds int32 The number of background attributes associated with this browse location
iloper int16 The number of background attributes not associated with this browse location but within the range of the background images
bkgrnd pointer int32 The index of the first background attribute associated with this browse location
duration int16 The total time in seconds that this browse location was viewed by OTD
[The Vdata set is repeated for each browse location.]
Vdata Set 1: Vector Statistics
location[2] float32 The latitude/longitude of the full instrument resolution vector location
EGFA counts[4] int16 Number of events/groupd/flashes/areas with centroid at the ground location
EGFA cal intensity[4] float32 Total summation calibrated intensity of the events/groups/flashes/areas with centroids at the ground location
max flash rate float32 Maximum flash rate found at ground location
single event areas int16 Number of areas at the ground location with only one event
[The Vdata set is repeated for each latitude/longitude point on the earth that has OTD data (events, groups, flashes, areas, or backgrounds) associated with it.]
Vdata Set 2: Image Attributes
TAI93 float64 The TAI time of the background
NW location[2] float32 The latitude/longitude of the upper left corner of the background image
NE location[2] float32 The latitude/longitude of the upper right corner of the background image
SW location[2] float32 The latitude/longitude of the lower left corner of the background image
SE location[2] float32 The latitude/longitude of the lower right corner of the background image
nadir location[2] float32 The nadir latitude/longitude of the satellite
image pointer int32 The HDF record number of the background image
calibration number int32 The calibration sequence used for this background
Spice pointer int32 Pointer to the ephemeris data closest to the time of the background image
The Vdata set is repeated for each background image occurring during the orbit.]
Vgroup 5: Flash Density Statistics
Vdata Set 0: 2.5 Degree Grid (2.5 x 2.5 degree grid)
location[2] float32 The latitude/longitude of grid point
flash count int32 Number of flashes occurring in grid point
[The Vdata set is repeated for each 2.5 x 2.5 degree grid point location.]
Vdata Set 1: View Time Grid (0.5 x 0.5 degree grid)
grid point[2] float32 The latitude/longitude of grid point
TAI93 start float64 The start TAI time of OTD
duration int16 Time OTD viewed grid (in seconds)
[The Vdata set is repeated for each grid 0.5 degree point.]
Vdata Set 2: 500 km Grid (500 x 500 km grid)
NOTE: Zero filled: There were no data recorded for this group for OTD.
location[2] float32 The latitude/longitude of grid point
flash count int32 Number of flashes in the grid point
[The Vdata set is repeated for each grid point.]
Vgroup 6: Orbit Summary Statistics
Vdata Set 0: Summary Data
areas int32 The number of areas found during an orbit
flashes int32 The number of flashes found during an orbit
groups int32 The number of groups found during an orbit
events int32 The number of events found during an orbit
backgrounds int32 The number of backgrounds recorded during an orbit
total cal radiance float32 The total integrated intensity measured during an orbit
Vdata Set 1: Flash Rate Data
flash rate float32 The flash rate of each area found during the orbit
The Vdata set is repeated for each area found during the orbit.]
Vdata Set 2: Event Rates Data
count int16 House Keeping data used by the LIS Science Team
Vdata Set 3: Ephemeris Data
GPS time float64 The GPS time of the SPICE ephemeris kernel
spice kernel[15] float64 The SPICE ephemeris kernel
[The Vdata set is repeated for every second in the orbit.]
Vgroup (SDS) 10: Background images
Background images are stored in 128 x 128 arrays and can be displayed as images. [Repeated for each of the background images recorded for the orbit.]
Vgroup (SDS) 11: Core Metadata
VALUES[7739] character The Metadata required for the EOSDIS Core System Database for the ECS Project
Lone Vdata
Lone Vdata Group 0: Orbit Attributes
orbit ID int32 A unique designation of the OTD orbit
TAI93 start float64 Orbit start time
start location[2] float32 The latitude/longitude of the sub-satellite point at the start of the orbit
TAI93 end float64 Orbit end time
end location[2] float32 The latitude/longitude of the sub-satellite point at the end of the orbit
Lone Vdata Group 1: Metadata
metadata[80] character Text based description of the OTD parameters unique to this orbit
[Repeated 6 times in order to describe the unique OTD orbit parameters.]
Lone Vdata Group 2: Threshold Values
TAI93 float64 TAI time of instrument
values[16] unsigned Threshold values (8 bit count)
[Repeated for each threshold change made during the orbit.]

Obtaining Software

The LIS/OTD Software Package

Go to URL http://lightning.nsstc.nasa.gov/data. There, in the OTD information column, you will find the link to the software package, which is a PDF file.

The NCSA Library

The NCSA HDF library, tools, and complete documentation can be retrieved from the NCSA site at http://hdf.ncsa.uiuc.edu .

References

Christian, H. J., R. J. Blakeslee, and S. J. Goodman, The Detection of Lightning from Geostationary Orbit, J. Geophys. Res., Vol. 94, pp. 13329-13337, 1989.

Christian, H. J., R. J. Blakeslee, S. J. Goodman, and D. M. Mach, Algorithm Theoretical Basis Document (ATBD) For the Lightning Imaging Sensor (LIS), Earth Observing System (EOS) Instrument Product, only available at http://eospso.gsfc.nasa.gov/eos_homepage/for_scientists/atbd/docs/LIS/atbd-lis-01.pdf.

Christian, H.J., R.J. Blakeslee, and S.J. Goodman, Lightning Imaging Sensor (LIS) for the Earth Observing System, NASA Technical Memorandum 4350, MSFC, Huntsville, AL, February, 1992.

Ordering Optical Transient Detector Data Sets

To order these data or for further information, please contact:

Global Hydrology Resource Center
User Services
320 Sparkman Drive
Huntsville, AL 35805
Phone: 256-961-7932
E-mail: ghrcdaac@itsc.uah.edu
Web: http://ghrc.nsstc.nasa.gov/