MODAPS PCR 06-016
| PCR Number | 06-016 |
|---|---|
| Date | 2006-09-25 |
| Initiator | Richard Hucek |
| Abstract | Chunking of internally compressed MODIS Terra and Aqua L1B products. |
| Problem Statement | MODIS Terra and Aqua L1B products (1KM, HKM, and QKM) are presently storedin LAADS in compressed format and distributed to users that way. Some usersmay not be aware that the L1B files are compressed or that they could cause their applications to fail (e.g. IDL). Another caveat is the extremely poorprocessing performance that occurs when running some MODIS PGEs and other high CPU applications with compressed L1B input. Slowdowns by factors of5 or 10 do occur. |
| Description of Change | MODAPS intends to add array chunking to the L1B products in the LAADS archive in addition to the compression format already there. The chunking approach takes a large array and breaks it into smaller pieces, preferably in the same data blocks as the processing code reads data. This enhances processing performance relative to a compressed, but not chunked, HDF product. The change is to be implemented in the PGE02 Perl script using the HDF 'hrepack' utility. |
| Products Affected | MOD021KM, MOD02HKM, MOD02QKM |
| Software Affected | PGE02 |
| Processing String to Receive the Change | Terra Forward, Terra Reprocessing, Aqua Forward, Aqua Reprocessing |
| Downstream Product Effects | In the production system, downstream PGEs and products are unaffected by the chunking change. PGEs that previously required decompressed L1B inputs will now also include de-chunking. This restores the L1B product to MCST standard, the most efficient processing format. |
| Data Granules to be Used for Science Testing | |
| Justification | The LAADS archive is to be populated with compressed and chunked L1B products.Compression reduces file sizes and that conserves network bandwidth. At the same time, chunking promotes data usability. Chunked and compressed L1B input data yield runtime factors of about 1.4 greater than full size standard L1B. Compared this to factors of 10 or more when using compressed, but not chunked, L1B products. In addition, IDL applications may fail on L1B data that is compressed but not chunked. |
| Effective Date for Implementation of Change | |
| Status |
