Introduction to Optical Transient Detector (OTD) Lightning Product
Table of Contents
The OTD Instrument and Its Data
The Optical Transient Detector is a scientific payload on the MicroLab-1 satellite, which was launched into orbit by a Pegasus rocket in April of 1995. The primary mission of the OTD instrument is to improve the understanding of thunderstorm distributions, cloud processes, and storm variability by detecting and locating lightning activity over large areas of the Earth's surface. The concept for this instrument was developed at NASA's Marshall Space Flight Center in the 1980s, and was selected for development as part of NASA's Earth Observing System (EOS). The OTD instrument is a flight-qualified engineering model for the Lightning Imaging Sensor (LIS), which was launched in November of 1997 on the Tropical Rainfall Measuring Mission (TRMM) satellite.
The OTD instrument detects and locates the lightning discharges that occur within its field-of-view, marks the time of occurrence of the lightning, and measures the radiant energy. The spatial resolution of the instrument is 10 km and the temporal resolution is 2 ms. The OTD detects both intra-cloud and cloud-to-ground discharges during day and night conditions with a high detection efficiency.
The orbital trajectory of the MicroLab-1 satellite has allowed the OTD to circle the earth once every 100 minutes at an altitude of 740 km. Using its 128 x 128 pixel photo-diode array and wide field-of-view lens, the OTD instrument is capable of viewing a total area of 1300 km x 1300 km. Given the field-of-view and the orbital trajectory, the OTD can monitor individual storms and storm systems for about 4 minutes, a period long enough to obtain a measure of the lightning flashing rate in these storms.
The data obtained by the OTD instrument includes the "background images" as well as "lightning data". A background image is roughly a "snapshot" of the field-of-view of the instrument. These images are recorded about once every minute by the instrument, although the time between images varies according to the amount of lightning data being processed. The "lightning data" is simply the optical pulses that OTD detects while passing over active thunderstorms. These optical pulses are assembled into individual classes, namely "EVENTS", "GROUPS", "FLASHES", and "AREAS".
An EVENT is the smallest granule of OTD data. An EVENT is the recorded by the data system whenever the value at a particular focal plane pixel exceeds the background count at that pixel by a threshold value. A GROUP is defined as one or more adjacent events in the same 2 ms time frame. A FLASH is defined as one or more GROUPS sufficiently close in space in time to be classified as a single FLASH. An AREA is defined as a grouping of one or more FLASHES that are sufficiently separate from existing AREAS. For more details of the lightning data classes, see the Algorithm Theoretical Basis Document (ATBD) for the Lightning Imaging Sensor (LIS) at URL http://eospso.gsfc.nasa.gov/eos_homepage/for_scientists/atbd/docs/LIS/atbd-lis-01.pdf.
OTD Geolocation
All of the lightning data recorded by the OTD was geolocated using the SPICE Toolkit software, which was developed at the Navigation Ancillary Information Facility at Caltech's Jet Propulsion Laboratory. The SPICE Toolkit provided a means to conveniently store and access the ephemeris data for the OTD instrument using data sets called "kernels". The OTD "kernels" were constructed with SPICE software by combining the fixed sensor/platform geometry with the continuous stream of navigational information supplied by the MicroLab-1 satellite.
Quality Control of the OTD Data
OTD exists in a noisy space environment. It also responds to a number of optical signals, not all of which are necessarily lightning-related. A significant amount of software filtering has gone into the production of science data distributed to the science community. The filters maximize both detection efficiency and confidence level so that each datum is signal, and not noise.
Each OTD lightning event in an OTD file is tagged with four low-level quality indicators, while each OTD data file is assigned four high-level flags that were designed to notify potential users of possible irregularities in the data file. An automated process is used to tag each optical event in the OTD data file with a set of four numbers that indicate the relative likelihood that the event was produced by lightning, as opposed to solar glint, energetic particles in the Van Allen radiation belts, or electronic noise. These low-level tags are as follows:
- Non-noise Probability (the probability that the event is not caused by random noise or energetic particles).
- Solar Glint Factor (a number that indicates the likelihood that the event was caused by direct reflected solar radiation).
- Event Rate Ratio (a number that represents the ratio of "accepted" events to the raw detected events during a one-second period at the time of the event).
- Probability Density (a number that indicates whether the event is geolocated in the vicinity of other events that are likely to be lightning).
In addition, each OTD data file is manually inspected for irregularities in the data set. The data files
that fail specific quality assurance are flagged. The high-level quality flags assigned to each OTD HDF data file are
contained in the file OTD_QA.dat included with the LISOTD software. The structure of each line in this ascii file is as
follows:
YY JJJ XXXX A B C D E
where YY is year, JJJ is julian day of year, XXXX is orbit number, and A,B,C,D and E are as follows:
A. This file has no valid data and shouldn't be used (not supposed to be distributed)
B. The file contains an irregular number of events that appear not to have been caused by lightning.
C. The file contains an instrument view time that is inconsistent with the period of the orbit or the periods of data availability.
D. The file contains data beyond normal orbit length or the file is tagged with an irregular orbit start or stop time.
E. The ephemeris used to geolocate some or all lightning events in the file is questionable and/or the satellite is out of control.
It is recommended that if any values occur in A-E that that file not be used for climate applications since they may skew
flash rate calculations. These files are included in the distribution because useful data occurs within some portion of the dataset.
The Format of the OTD Data and the NCSA HDF Libraries
The data obtained from the OTD instrument is stored in a Hierarchical Data Format (HDF) format, the standard format for EOS projects. HDF is a platform independent data format used for the storage and exchange of scientific data. To read files written in a HDF format, it is necessary to obtain software from The National Center for Supercomputing Applications (NCSA) which is available via the Internet at URL http://hdf.ncsa.uiuc.edu/. NCSA provides a public domain library supporting HDF on a wide variety of computer platforms. Instructions on how to obtain the NCSA HDF libraries are provided in 5 of this document.
The LIS/OTD Read Software
A new software package has been written which supersedes previous OTD read software. The LIS/OTD Read software was designed to read both OTD and LIS HDF files. While both datasets are in slightly different HDF formats, both can be read by this software package. The following is from the introductory chapter of the users manual for the software package:
This document serves as a guide to the software intended for use with satellite data from the Optical Transient Detector (OTD) and Lightning Imaging Sensor (LIS). The software suite consists of both fully featured GUI (Graphical User Interface) driven applications, and collections of high- and low-level APIs (Application Programming Interfaces). The software is designed to simplify, as much as possible, user access to the OTD and LIS lightning data sets, which are currently distributed in HDF (Hierarchical Data Format) files. The suite is designed with four goals in mind: simplicity, reusability, compatibility and deployment. By providing software strongly tailored to these goals, we hope to minimize each user's time spent accessing and managing the datasets, and maximize the time spent actually analyzing them.
OTD HDF File Structure
The HDF file structure for the OTD HDF files along with brief descriptions of the data fields are provided in TABLE 1. For the definitions and full descriptions of the data fields please refer to the Algorithm Theoretical Basis Document (ATBD) for the Lightning Imaging Sensor (LIS)".
OTD HDF Data File Structure
NOTE: TAI93 is SI seconds counted from UTC midnight Jan. 1, 1993. The SI second is derived from atomic clocks at the National Institute of Standards and Technology (NIST).
NOTE: [ ] indicates the number of times the variable is repeated.
NOTE: Intensity has the units of micro-J/m-m/sr/micro-m. TABLE 1
Variable name |
Type |
Description |
Vgroup 0: Area Statistics |
Vdata Set 0: Area Statistics |
seq |
int32 |
Area creation sequence number |
TAI93 |
float64 |
TAI start time of the area |
delta |
float32 |
Lifetime of the area (in seconds) |
view |
float32 |
OTD viewing time to the area (in seconds) |
s-z-a |
int16 |
Solar Zenith Angle |
d-n-t |
character |
Flag for day(d), night(n) or twilight(t) |
end status |
character |
Grouping algorithm status flag for area (D=naturally closed, K=unnaturally closed) |
events |
int16 |
Number of events that make of the area |
cent[2] |
float32 |
Average latitude/longitude of the area (to 0.02 deg) |
stdev[2] |
float32 |
Latitude/Longitude 'size' of the area (to 0.02 deg) |
loc count |
int16 |
Number of different lat/long pairs that make up area |
rad |
float32 |
The total summation of the events intensity that make up the area |
orbit id |
int32 |
The index of the parent orbit of this area |
day |
int32 |
The day number of the area |
children |
int32 |
The number of lashes that are part of the area |
child seq |
int32 |
The creation sequence of the first child flash |
child rec |
int32 |
The index of the first child flash |
QA[4] |
int16 |
Quality flags derived from child flash QA |
[The Vdata set is repeated for each area found during orbit.] |
Vgroup 1: Flash Statistics |
Vdata Set 0: Flash Statistics |
seq |
int32 |
Flash creation sequence number |
TAI93 |
float64 |
TAI start time of flash |
delta |
float32 |
Lifetime of the flash (in seconds) |
view |
float32 |
OTD viewing time of the flash (in seconds) |
s-z-a |
int16 |
Solar zenith Angle |
d-n-t |
character |
Flag for day(d), night(n) or twilight(t) |
end status |
character |
Grouping algorithm status flag for flash (D=naturally closed, K=unnaturally closed) |
events |
int16 |
Number of events that make up the flash |
cent[2] |
float32 |
Average latitude/longitude of the flash (to 0.02 deg) |
stdev[2] |
float32 |
Lat/Long 'size' of the flash (to 0.02 deg) |
loc count |
int16 |
Number of different latitude/longitude pairs that make up the flash |
rad |
float32 |
The total summation of the events intensity that make up the flash |
parent seq |
int32 |
The index of the parent area of this flash |
parent rec |
int32 |
The HDF record # of the parent area of this flash |
children |
int32 |
The number of groups that are part of the flash |
child seq |
int32 |
The creation sequence of the first child group |
child rec |
int32 |
The index of the first child group |
QA[4] |
int16 |
Quality flags derived from child group QA |
[The Vdata set is repeated for each flash found during orbit.] |
Vgroup 2: Group Statistics |
Vdata Set 0: Group Statistics |
seq |
int32 |
Group creation sequence number |
TAI93 |
float64 |
TAI start time of group |
delta |
float32 |
Lifetime of the group (in seconds) |
view |
float32 |
OTD viewing time of the group (in seconds) |
s-z-a |
int16 |
Solar zenith Angle |
d-n-t |
character |
Flag for day(d), night(n) or twilight(t) |
end status |
character |
Grouping algorithm status flag for group (D=naturally closed, K=unnaturally closed) |
events |
int16 |
Number of events that make up the group |
cent[2] |
float32 |
Average latitude/longitude of the group (to 0.02 deg) |
stdev[2] |
float32 |
Lat/Long 'size' of the group (to 0.02 deg) |
loc count |
int16 |
Number of different latitude/longitude pairs that make up the group |
rad |
float32 |
The total summation of the events intensity that make up the group |
parent seq |
int32 |
The index of the parent area of this group |
parent rec |
int32 |
The HDF record # of the parent area of this group |
children |
int32 |
The number of groups that are part of the group |
child seq |
int32 |
The creation sequence of the first child event |
child rec |
int32 |
The index of the first child event |
QA[4] |
int16 |
Quality flags derived from child event QA |
[The Vdata set is repeated for each group found during orbit.] |
Vgroup 3: Event Statistics |
Vdata Set 0: Event Statistics |
event # |
int32 |
Event creation number |
seq # |
int32 |
Event HDF record number |
TAI93 |
Float64 |
TAI start time of event |
s-z-a |
int16 |
Solar Zenith Angle |
d-n-t |
character |
Flag for day(d), night(n) or twilight(t) |
x pixel |
int16 |
X pixel value of the event |
y pixel |
int16 |
Y pixel value of the event |
raw radiance |
int16 |
Event amplitude in counts |
cal radiance |
float32 |
The calibrated event intensity |
parent seq |
int32 |
The index of the parent group of this event |
parent rec |
int32 |
The HDF record # of the parent group of this event |
QA[4] |
int16 |
Quality flags |
location[2] |
float32 |
The Earth latitude/longitude of the event |
[The Vdata set is repeated for each event found during orbit.] |
Vgroup 4: Browse Statistics |
Vdata Set 0: Browse Area |
location[2] |
float32 |
The latitude/longitude of the 2.5 x 2.5 degree browse grid point |
EGFAs[4] |
int16 |
Number of events/groups/flashes/areas with centroid at the ground location |
EGFA cal rad[4] |
float32 |
Total summation of calibrated intensity of the event/groups/flashes/areas with centroids at the ground location |
max rate |
float32 |
Maximum flash rate found at the ground location |
1 event areas |
int16 |
Number of areas at the ground location with only one event |
highres locs |
int16 |
The number of full instrument resolution points |
location 1st highres |
int16 |
The index of the first full instrument resolution Vdata associated with this browse location |
bkgnds |
int32 |
The number of background attributes associated with this browse location |
iloper |
int16 |
The number of background attributes not associated with this browse location but within the range of the background images |
bkgrnd pointer |
int32 |
The index of the first background attribute associated with this browse location |
duration |
int16 |
The total time in seconds that this browse location was viewed by OTD |
[The Vdata set is repeated for each browse location.] |
Vdata Set 1: Vector Statistics |
location[2] |
float32 |
The latitude/longitude of the full instrument resolution vector location |
EGFA counts[4] |
int16 |
Number of events/groupd/flashes/areas with centroid at the ground location |
EGFA cal intensity[4] |
float32 |
Total summation calibrated intensity of the events/groups/flashes/areas with centroids at the ground location |
max flash rate |
float32 |
Maximum flash rate found at ground location |
single event areas |
int16 |
Number of areas at the ground location with only one event |
[The Vdata set is repeated for each latitude/longitude point on the earth that has OTD data (events, groups, flashes, areas, or backgrounds) associated with it.] |
Vdata Set 2: Image Attributes |
TAI93 |
float64 |
The TAI time of the background |
NW location[2] |
float32 |
The latitude/longitude of the upper left corner of the background image |
NE location[2] |
float32 |
The latitude/longitude of the upper right corner of the background image |
SW location[2] |
float32 |
The latitude/longitude of the lower left corner of the background image |
SE location[2] |
float32 |
The latitude/longitude of the lower right corner of the background image |
nadir location[2] |
float32 |
The nadir latitude/longitude of the satellite |
image pointer |
int32 |
The HDF record number of the background image |
calibration number |
int32 |
The calibration sequence used for this background |
Spice pointer |
int32 |
Pointer to the ephemeris data closest to the time of the background image |
The Vdata set is repeated for each background image occurring during the orbit.] |
Vgroup 5: Flash Density Statistics |
Vdata Set 0: 2.5 Degree Grid (2.5 x 2.5 degree grid) |
location[2] |
float32 |
The latitude/longitude of grid point |
flash count |
int32 |
Number of flashes occurring in grid point |
[The Vdata set is repeated for each 2.5 x 2.5 degree grid point location.] |
Vdata Set 1: View Time Grid (0.5 x 0.5 degree grid) |
grid point[2] |
float32 |
The latitude/longitude of grid point |
TAI93 start |
float64 |
The start TAI time of OTD |
duration |
int16 |
Time OTD viewed grid (in seconds) |
[The Vdata set is repeated for each grid 0.5 degree point.] |
Vdata Set 2: 500 km Grid (500 x 500 km grid)
NOTE: Zero filled: There were no data recorded for this group for OTD. |
location[2] |
float32 |
The latitude/longitude of grid point |
flash count |
int32 |
Number of flashes in the grid point |
[The Vdata set is repeated for each grid point.] |
Vgroup 6: Orbit Summary Statistics |
Vdata Set 0: Summary Data |
areas |
int32 |
The number of areas found during an orbit |
flashes |
int32 |
The number of flashes found during an orbit |
groups |
int32 |
The number of groups found during an orbit |
events |
int32 |
The number of events found during an orbit |
backgrounds |
int32 |
The number of backgrounds recorded during an orbit |
total cal radiance |
float32 |
The total integrated intensity measured during an orbit |
Vdata Set 1: Flash Rate Data |
flash rate |
float32 |
The flash rate of each area found during the orbit |
The Vdata set is repeated for each area found during the orbit.] |
Vdata Set 2: Event Rates Data |
count |
int16 |
House Keeping data used by the LIS Science Team |
Vdata Set 3: Ephemeris Data |
GPS time |
float64 |
The GPS time of the SPICE ephemeris kernel |
spice kernel[15] |
float64 |
The SPICE ephemeris kernel |
[The Vdata set is repeated for every second in the orbit.] |
Vgroup (SDS) 10: Background images |
Background images are stored in 128 x 128 arrays and can be displayed as images. [Repeated for each of the background images recorded for the orbit.] |
Vgroup (SDS) 11: Core Metadata |
VALUES[7739] |
character |
The Metadata required for the EOSDIS Core System Database for the ECS Project |
Lone Vdata |
Lone Vdata Group 0: Orbit Attributes |
orbit ID |
int32 |
A unique designation of the OTD orbit |
TAI93 start |
float64 |
Orbit start time |
start location[2] |
float32 |
The latitude/longitude of the sub-satellite point at the start of the orbit |
TAI93 end |
float64 |
Orbit end time |
end location[2] |
float32 |
The latitude/longitude of the sub-satellite point at the end of the orbit |
Lone Vdata Group 1: Metadata |
metadata[80] |
character |
Text based description of the OTD parameters unique to this orbit |
[Repeated 6 times in order to describe the unique OTD orbit parameters.] |
Lone Vdata Group 2: Threshold Values |
TAI93 |
float64 |
TAI time of instrument |
values[16] |
unsigned |
Threshold values (8 bit count) |
[Repeated for each threshold change made during the orbit.] |
Obtaining Software
The LIS/OTD Software Package
Go to URL https://ghrc.nsstc.nasa.gov/lightning/dataset-info.html. There, in the OTD information column, you will find the link to the software package, which is a PDF file.
The NCSA Library
The NCSA HDF library, tools, and complete documentation can be retrieved from the NCSA site at http://hdf.ncsa.uiuc.edu .
References
Christian, H. J., R. J. Blakeslee, and S. J. Goodman, The Detection of Lightning from Geostationary Orbit, J. Geophys. Res., Vol. 94, pp. 13329-13337, 1989.
Christian, H. J., R. J. Blakeslee, S. J. Goodman, and D. M. Mach, Algorithm Theoretical Basis Document (ATBD) For the Lightning Imaging Sensor (LIS), Earth Observing System (EOS) Instrument Product, only available at http://eospso.gsfc.nasa.gov/eos_homepage/for_scientists/atbd/docs/LIS/atbd-lis-01.pdf.
Christian, H.J., R.J. Blakeslee, and S.J. Goodman, Lightning Imaging Sensor (LIS) for the Earth Observing System, NASA Technical Memorandum 4350, MSFC, Huntsville, AL, February, 1992.
Ordering Optical Transient Detector Data Sets
To order these data or for further information, please contact:
Global Hydrology Resource Center
User Services
320 Sparkman Drive
Huntsville, AL 35805
Phone: 256-961-7932
E-mail: support-ghrc@earthdata.nasa.gov
Web: http://ghrc.nsstc.nasa.gov/
|