Operational Line Scan (OLS) Derived Lightning Data Sets
Table of Contents
Summary
This README file contains information for three data sets: 1) OLS Analog Derived Lightning; 2) OLS Digital Derived Lightning From DMSP F10 and 3) OLS Digital Derived Lightning From DMSP F12. The structure of the data files, instructions for accessing the HDF library, description of read software, and pertinent scientific references are included in this README. The derived lightning data are observations at local midnight, since the system was only capable of detecting lightning during night.
Introduction
The sun-synchronous U.S. Air Force Defense Meteorological Satellite Program (DMSP) series has provided one of the longest running nighttime global lightning data set. The Scanning Radiometer on DMSP-5B-F3 and the Operational Line Scanner (OLS) on DMSP-5D-1/F1, 5D-2/F7, 5D-2/F8, 5D-2/F9, 5D-2/F10, and 5D-2/F12 recorded high-resolution images in the visible wavelength range. The visible-band (0.4 to 1.1 micrometer) data from 1973 through 1991 are in the form of hard-copy positive transparency images. Beginning in summer of 1991, the production of hard-copy by the U.S. Air Force began to taper off as they transitioned to a digital data handling system. Currently, high resolution and low resolution data files are written to 8 mm tape which are sent to National Geophysical Data Center (NGDC) on a daily basis. At NGDC, OLS data are decompressed, reordered, restructured and De-interleaved. The National Snow and Ice Data Center (NSIDC) has maintained the archives of analog OLS data since 1979. On March 9, 1992, the full DMSP digital data stream started to be sent to NGDC. Initial tapes were prepared on an experimental basis until routine operations begin in February 1994.
OLS Analog Derived Lightning data (START DATE: 1973-06-01 STOP DATE: 1991-12-31) are derived from thousands of hard-copy positive transparency film strips DMSP visible (0.4 - 1.1 um) imagery archived at the NSIDC. Hence to denote the data was derived from film strips, the word Analog was included in the data set name. The hard-copy positive transparency film strips date back to 1973 and have been manually sorted for lightning signatures which are geolocated with date and time. Unfortunately, the visible-band imagery are not uniformly available nor of uniform quality. There are numerous gaps in the nighttime imagery including a period form July 1980 to January 1984. In spite of the significant limitations and the lack of alternative sources of global lightning data, the DMSP film archive represents the most comprehensive global lightning data set in existence for this time period.
The OLS Digital Derived Lightning From DMSP F10 (START DATE: 1994-02-01 STOP DATE: 1994-05-31) and OLS Digital Derived Lightning From DMSP F12 (START DATE: 1995-05-01 STOP DATE: 1995-11-31) were derived from OLS electronic images. A neural network and pattern recognition algorithm was used to automatically extracts the lightning discharge time and location from the data. Hence to denote the data came from the electronic stored image, the word Digital was added to the data set name.
The DMSP Satellites
The Defense Meteorological Satellite Program(DMSP) is a Department of Defense(DoD) program run by the Air Force Space and Missile Systems Center(SMC). The DMSP program designs, builds, launches, and maintains several near polar orbiting, sun synchronous satellites monitoring the meteorological, oceanographic, and solar-terrestrial physics environments.
DMSP satellites are in a near polar orbiting, sun synchronous orbit at an altitude of approximately 830 Km above the earth. Each satellite crosses any point on the earth twice a day and has an orbital period of about 101 minutes thus providing complete global coverage of clouds every six hours. Each DMSP satellite monitors the atmospheric, oceanographic and solar-geophysical environment of the Earth. The visible and infrared sensors collect images of global cloud distribution across a 3,000 km swath during both daytime and nighttime conditions.
The data from the DMSP satellites are received and used at operational centers on a continual basis. The data are sent daily to the National Geophysical Data Center (NGDC), Solar Terrestrial Physics Division(STPD) for archiving.
The DMSP Image Scanners
The DMSP satellites have gain controls which allow the satellite to record in light levels ranging from daylight to one-quarter moonlight. A change in gain will change the saturation threshold of the system. The primary sensor, Operational Linescan System (OLS), on the DMSP satellites consists of high-resolution scanners that produce visible- and infrared-spectrum photographs. The OLS has two pixel resolutions, fine and smooth. The visible smooth image data was used to derive the lightning data product. The smooth data pixels are approximately 2.7 x 2.7 km and each scan line contains 1465 pixels. Smooth data are collected from each operational satellite on a global basis twice a day.
The OLS instrument consists of two telescopes and a photomultiplier tube (PMT). The high resolution photomulitiplier tube detector was used for nighttime visible imagery. The PMT is sensitive to radiation from 0.47 - 0.95 um (0.51 - 0.86 um FWHM) at 10(-5) - 10(-9) Watts per m2 per steradian. The detectors sweep back and forth in a "whisk broom" or pendulum-type motion. The continuous signal is sampled at a constant rate so the Earth-located centers of each pixel are roughly equidistant, i.e., 0.5 km apart. 7,325 pixels are digitized across the 1080 swath from limb to limb. The instruments are built by Westinghouse Corporation.
Prior to 1976 the Scanning Radiometer was used on the DMSP satellites. It used a high-resolution silicon PN photodiode detector. The quantum efficiency of the photomulitiplier tubes is much lower than that of the silicon diodes. Although, the signal to noise characteristics at low light levels for the photomulitiplier tubes are far superior to that of the silicon diodes.
At night the gain on the instruments was set so that cities, gas and brush fires, and lightning flashes saturated the sensor system. Thus, lightning flashes are visible only during nighttime passes of the DMSP satellites. The result in the data, in the case of a lightning flash, is a horizontal streak on the image and is only one scan line thick associated with clouds. The satellite detector records both intra-cloud and cloud-to-ground lightning.
Derivation Lightning from the OLS Images
Derivation Lightning from the OLS Photographic Images
The lightning data consist of a series of positive transparencies which are archived at the National Snow and Ice Data Center in Boulder, CO. At latitudes pole ward of 60 degrees, the transparencies are generally of poor quality because the satellite is passing through the twilight. Maps are generated which contain the longitude and latitude lines and the orbit tracks for one day. Each lightning streak in a DMSP photograph is marked on the map with an error estimated to be no larger than 1 degree (corresponding to approximately 100km). The maps are then mounted on a digitizer board and the coordinates of the lightning flashes and orbital paths are recorded.
Derivation Lightning from the OLS Electronic Images
Beginning in summer of 1991, the production of hard-copy by the U.S. Air Force began to taper off as they transitioned to a digital data handling system. On March 9, 1992, the full DMSP digital data stream started to be sent to NGDC. Initial tapes were prepared on an experimental basis until routine operations begin in February 1994. At NGDC, OLS data are decompressed, reordered, restructured and De-interleaved. Satellite ephemeris are computed using a four body orbital mechanics program with observed, rather than predicted, orbital elements used as input. Missing scan lines are correctly positioned. Quality assessments are made of each pixel and characterized for the each complete scan line. A neural network and pattern recognition algorithm has been developed and implemented that automatically extracts the lightning discharge time and location. This is currently being done for the OLS data from both DMSP 5D-2/F10, and 5D-2/F12.
File Naming Convention
OLS Analog Derived Lightning
The OLS Analog Derived Lightning data product has the following file naming convention:
olslityy.mmm.analog.HDF (example, olslit77.sep.analog.hdf)
where, yy is the year and mmm is the month. Using the example file name, the file contains the lightning events data for the month of September 1977. The OLS Analog Derived Lightning browse product has the following file naming convention:
ols_yyyy_mm_world.gif (example, ols_1977_09_world.gif)
where yyyy is the year and mm is the month. Using the example file name, the file contains a GIF image overlaid with a world map and annotated for the month of September 1977. 5.2 OLS Digital Derived Lightning From DMSP F10 and DMSP F12 The OLS Digital Derived Lightning From DMSP F10 and DMSP 12 data product has the following file naming convention:
olslityy.mmm_digital_xx.hdf (example, olslit1994.feb_digital_10.hdf)
where, yyyy is the year, mmm is the month and xx is the satellite id number (DSMP F10 = 10 ; DSMP F12 = 12). Using the example file name, the file contains the lightning events data for the month of February 1994 from DMSP F10. The OLS Digital Derived Lightning browse product has the following file naming convention:
ols_yyyy_mmm_world_xx.gif (example, ols_1994_feb_world_10.gif)
where yyyy is the year, mmm is the month, and xx is the satellite id number (DSMP F10 = 10 ; DSMP F12 = 12). Using the example file name, the file contains a GIF image overlaid with a world map and annotated for the month of Feb. 1994 from DMSP F10.
OLS Analog Derived Lightning HDF Data File Structure
The three data sets (OLS Analog Derived Lightning, OLS Digital Derived Lightning From DMSP F10, and OLS Digital Derived Lightning From DMSP F12) have the same basic file structure. Data produced by Dr. Richard E. Orville [Orville and Spencer, 1979; Orville 1981; and Orville and Henderson, 1986] for September 1977 through August 1978 have a variant HDF file structure used by the other HDF files in this data set.6.1 Hierarchical Data Format HDF is a library and platform independent data format for the storage and exchange of scientific data. It includes Fortran and C calling interfaces, and utilities for analyzing and converting HDF data files. HDF is developed and supported by the National Center for Supercomputing Applications (NCSA) and is available in the public domain (http://hdf.ncsa.uiuc.edu/). HDF stands for Hierarchical Data Format. It is a multi-object file format for the transfer of graphical and numerical data between machines. HDF is a portable file format. HDF files can be shared across platforms. An HDF file created on one computer, say a Silicon Graphics Indy (SGI), can be read on another system, say IBM PC, without modification. The purpose of the HDF files are to contain data, a file descriptor, and the meta-data. The meta-data contains information regarding the satellite platform, the film strip identification, orbit information, latitude and longitude of each corner of the image and the number of lightning flashes on the image. 6.2 OLS Analog Derived Lightning HDF File Structure The data is stored in HDF format. Table 1. list the variable names, type, HDF object type and a brief description of the OLS Analog Derived Lightning HDF File Structure (excluding Data Compiled by Dr. Orville (Sept. 1977 - Aug. 1978)).
TABLE 1. HDF FILE FORMAT FOR OLS ANALOG DERIVED LIGHTNING DATA SET
| Variable name |
Type |
HDF object type |
Description |
| 1) Monthly Lightning Event Image |
| [900X420] |
int32 |
RIS8 image |
Image of all lightning events recorded for month annotated with color bar |
| 2) Strip Identification (The following is repeated for each image (film) strip recorded in the file) |
| strip id[12] |
character |
vdata |
satellite designation and image strip number |
| time[24] |
character |
vdata |
start date and time recorded on image strip |
| node type[10] |
character |
vdata |
descending or ascending |
| node value |
float32 |
vdata |
|
| event count |
int32 |
vdata |
number of events found on image strip |
| (The following is repeated for each event recorded) |
| location[2] |
float32 |
vdata |
latitude and longitude of lightning event |
| (Coverage Coordinates repeated for each corner[4] of image strip) |
| location[2] |
float32 |
vdata |
latitude and longitude of each corner of image strip |
| 3) Meta-data Description |
| fld[80] |
character |
vdata |
supplies information about file in text format (i.e. number of ascending nodes; number of descending nodes; total number of lightning events recorded in the file ; number of image strips with no lightning events) |
Note: RIS8 is a raster image which is a space-saving image storage technique. The satellite designation code is listed in TABLE 2.
TABLE 2. SATELLITE DESIGNATION CODE GIVEN IN THE "strip id" FIELD
| Code valueDMSP satellite |
Launch date |
| QDMSP-5B-F3 |
Nov. 9, 1972 |
| F1DMSP-5D-1/F1 |
Sept. 11, 1977 |
| F7DMSP-5D-2/F7 |
Nov. 18, 1983 |
| F8DMSP-5D-2/F8 |
June 6, 1987 |
| F9DMSP-5D-2/F9 |
Feb. 3, 1988 |
| F10DMSP-5D-2/F10 |
Dec. 1, 1990 |
OLS Analog Derived Lightning HDF File Structure for Data Derived by Dr. Orville (Sept. 1977 - Aug. 1978)
The lightning events derived by Dr. Richard Orville were tabulated without any information as to which image strip they were found on. Thus, this information in the HDF is void or has null values. Table 3. list the variable names, type, HDF object type and a brief description (including values assigned to fields which the information is missing) of the OLS Analog Derived Lightning HDF File Structure for data compiled by Dr. Orville from Sept. 1977 through Aug. 1978.
TABLE 3. HDF FILE FORMAT FOR OLS ANALOG DERIVED LIGHTNING DATA SET FOR DATA COMPILED BY DR. ORVILLE FROM SEPT. 1977 THROUGH AUG. 1978
| Variable name |
Type |
HDF object type |
Description |
| 1) Monthly Lightning Event Image |
| [900X420] |
int32 |
RIS8 image |
Image of all lightning events recorded for month annotated with color bar |
| 2) Strip Identification (The following is repeated ONLY ONCE) |
| strip id[12] |
character |
vdata |
ORVILLE 0000 is always listed in this field. |
| time[24] |
character |
vdata |
start date and time is recorded |
| node type[10] |
character |
vdata |
UNKNOWN is always listed in this field |
| node value |
float32 |
vdata |
0.0 is always listed in in this field |
| event count |
int32 |
vdata |
number of events found for the given month |
| (The following is repeated for each event recorded) |
| location[2] |
float32 |
vdata |
latitude and longitude of lightning event |
| (Coverage Coordinates repeated for each corner[4] of image strip) |
| location[2] |
float32 |
vdata |
0.0 and 0.0 are always listed |
| 3) Meta-data Description |
| fld[80] |
character |
vdata |
supplies information about file in text format (i.e. always list 1 as the number of ascending nodes and 0 for the number of descending nodes; total number of lightning events recorded in file; always list 0 as the number of image strips with no lightning events) |
Note: RIS8 is a raster image which is a space-saving image storage technique.
How to Obtain the HDF Library and Tools
The HDF library and tools can be down loaded from the National Center for Supercomputing Applications at:
http://hdf.ncsa.uiuc.edu/
OLS Read Software
A "C" program is available that is heavily based on the source code for the "vshow" HDF command-line utility for reading the vgroup information. Code was added at the beginning to extract the raster image from the OLS datasets. Otherwise, the code is basically the same as "vshow", with comments added to explain to novice users of HDF how the HDF calls are used. The code will work for all three data sets (OLS Analog Derived Lightning, OLS Digital Derived Lightning From DMSP F10, and OLS Digital Derived Lightning From DMSP F12).
Browse Images in GIF Files
Browse images associated with the monthly lightning events can be obtained in Graphical Interchange Format (GIF). The GIF files were created from the HDF raster image. The GIF file contains a map overlay of the world and is annotated with the year, month, number of image strips examined, and the number of lightning events recorded for the month. A color bar is provide to indicate the number of events for a given location. The GIF files containing data derived by Dr. Orville (Sept. 1977 - Aug. 1978) will only list a 1 as the number of strips, since the information on which image strip the lightning events were derived from is not available.
References
Goodman S.J. and H.J. Christian, Global Observations of Lightning, Atlas of Satellite Observations Related to Global Change, R. J. Gurney, Foster J.L. and C.L. Parkinson (eds.), Cambridge University Press, London, p.p. 191- 211, 1993.
Orville R.E. and R.W. Henderson, Global Distribution of Midnight Lightning: September 1977 to August 1978, Monthly Weather Review, Vol. 114, p.p. 2640-2653, 1986.
Kroehl, H.W., G.R. Scharfen, E.S. Arrance, and S.J. Goodman. "An archive of digital data from the Defense Meteorological Satellite Program (DMSP)", In 10th International Conference on Interactive Information and Processing Systems for Meteorology, Oceanography, and Hydrology, January 23-28, Nashville, TN, American Meteorological Society, p.p. 148-153, 1994.
Scharfen, G.R., H.W. Kroehl, and K.W. Knowles. "A user services capability for the Defense Meteorological Satellite Program (DMSP) digital data archive", In 10th International Conference on Interactive Information and Processing Systems for Meteorology, Oceanography, and Hydrology, January 23-28, Nashville, TN, American Meteorological Society, p.p. 154-159, 1994.
Contact Information
To order these data or for further information, please contact:
Global Hydrology Resource Center
User Services
320 Sparkman Drive
Huntsville, AL 35805
Phone: 256-961-7932
E-mail: support-ghrc@earthdata.nasa.gov
Web: http://ghrc.nsstc.nasa.gov/
|
