LAADS Capabilities
Last update: August 7, 2006
Version 3 LAADS Capabilities
The capabilities of LAADS have evolved considerably over the last couple of years driven by the significant input that has been received from the MODIS land and atmospheres science communities. The initial system (Version 1.3) was released in December 2004 and has been used extensively to support the MODIS algorithm developers in developing and refining the codes to be used for Collection 5 atmospheres and land reprocessing. The Version 2 LAADS was released in October 2005 specifically to atmospheres community. The current version, Version 3, was released in early April 2006 to both land the atmospheres communities and is used to support both atmospheres and land Collection 5 reprocess starting in April 2006. The Version 3 has many comprehensive and advanced functionalities, and some major capabilities are described here.
On-line Archive for Fast Data Access
The LAADS takes advantage of low cost of disk storage to retain hundreds of terabytes of data on-line, and now archives reprocess Collection 5 MODIS Geolocation, some L1B products, and L2+ granule and tile products, as well as Collection 4 forward process L2 and L3 atmosphere data. The LAADS will not archive all MODIS L1 products because of their huge data volume size, but a MODAPS Datapool of about 100 TB disks is set up to rolling archive the "fresh" MODIS L1B and Geolocation data from the forward process and reprocess. The details of Datapool capability can be seen in the LAADS and Data Pool README.
Users not only can browse, select, and order LAADS archive data through LAADS website, but also can directly access LAADS archive data by using LAADS HTTP server without having to submit orders.
Rapid and Simple Data Searches and Orders
The intuitive LAADS Web interface has many functionalities to support rapid, simple and flexible data searches and orders with quick responses (within seconds) to users.
LAADS Web search page will allow users to search for MODIS data by specifying the following, but not limited, criteria: ESDT name, file name, temporal and spatial ranges, ArchiveSet, Day/Night flags, orbital numbers, TileId, and QA metadata. Users can also search data by browsing data product images, including global, granule and tile images.
The LAADS also provides user the data availability information for all data archived in LAADS and DataPool, the Order Track page to track user's previous orders, and the Tools page that lists many useful tools and their download sites.
Various data download methods are provided in LAADS: wget, http, ftp-pull, and ftp-push. Once data are ordered, the data order confirmation and data download instructions will be immediately given to the user, and an data confirmation email will be sent to user shall user's email address is given.
Users can not only order archived data in LAADS and Datapool, but also order post process data. LAADS provides users flexible step-by-step order instructions to select post-process parameters. The post-process data will be available from minutes to hours depending on specific post-process orders. Users have to provide their email address so that the data ready notification can be sent to the users once the post-process data are generated.
Processing-On-Demand (POD)
LAADS uses Processing-On-Demand (POD) to recreate products ordered by users that are not being held in the LAADS archive, typically L1 and and L2, L2G and daily L3 land products. When such files are ordered, LAADS recreates the Plan and PGE instances that originally created the products, as well as the Plan and PGE instances that created any POD inputs that are not archived (all the way back to PGEs using only L0 and ancillary data if necessary). Normally LAADS will be configured to leave these plans on pause until approved by an operator. The system will then run the necessary PGEs until the ordered files have been produced, after which the user's order will be fulfilled. The products will remain in the archive for about a week after which they will again be deleted.
LAADS uses POD because the cost of the processing power needed to reproduce L1 and L2 and daily L3 products that are old enough that they are infrequently ordered, is much less than the cost of the disk space that would be required to store them. (Tape systems are generally as expensive as disks, as slow as reprocessing, and unreliable.) Thus at a cost of a day or so delay in getting products to users, Processing On Demand allows LAADS to provide these products in a cost effective way.
Post-Processes
The LAADS system provides, or will provide, tools and services that perform the following types of transformations to MODIS data sets: subset, mosaic, reproject, resample, re-bin, filter, and reformat. In the near future, the LAADS system will also supports running user supplied tools on MODIS data sets. Currently available features are:
- Subset by Region of Interest: Subsetting is the process of selecting a subset of the data within a data set. Users can specify a geographic region and request LAADS to remove from their selected product files all data outside the specified region. In addition to specifying a region by latitude and longitude, users can specify regions by pixel location (row, column in the image), and by a combination of latitude/longitude and row/column (i.e. 5 wide by 3 tall surrounding the pixel at -45.7 longitude, 45 latitude).
- Subset by Parameter: MODIS product files can contain many different data sets. Users can specify a list of the data sets they want to keep, and request LAADS to remove all others from their selected product files.
Data Export to DAACs
The LAADS can export the archive data to DAACs which is configured and generated by PGEs. Once the data are archived in LAADS, they shall be pushed to DAACs' PDR partitions within 20 minutes under normal conditions. The LAADS also performs reconciliation for all the data exported to DAACs by sending PDRs to DAACs,and pulling and processing the PAN/PDRD feefback files from DAACs. The unsuccessful PDRs will be automatically retried for certain data transfer errors or will be manually retried by operators through LAADS GUI.
The LAADS has number of controls on creating, scheduling, and running the DAAC-export tasks, so that they may be turned on or off to control the export tasks depending on the system's status or the users' requests, such as End User control, PGE control, User-ESDT control, User-host control, and User-diskspace control.
Some scripts are provided specifically to monitor DAAC export tasks and generate report reports on export tasks' State, files sent based on a specified date or a specified days ago, PAN/PDRD related errors, total number of files and sizes have been sent or to be sent based on given period of time, UserId and grouped by DAAC Order, Standing Order, and Web Order.
Data Subscriptions
The LAADS provides Data Subscription service, or called Standing Orders, to export defined sets of product types or files to a user on a regular basis. For example a user can specify that all products ingested into LAADS within a specified period (like HOURS=3), with ESDTs in a specified list, be sent to a user specified ftp address, or to a stage place in LAADS for 7 days for user to pull.
LAADS provides following options for users to make their subscription requests: Data subscription based on ESDTs and/or Tiles with optional lat/lon box boundaries; Data collection period; Data push (ftp push) or data stage (ftp pull); Send PDR, DN, or Email notification; Option to tar or compress all files for delivery; Options to run some or all Post-processes.
Depending on the system's status or the users' requests, the following controls may be turned on or off to control the StandingOrder task or the export tasks created by the StandingOrder task: User control, ArchiveSet control, User-FTP-host control, and User Diskspace control.
Metrics
LAADS provides detail metrics on data ingest, data archvie, data distribution, and User supports. See Metrics (password protected).
More Capabilities in Version 4 LAADS
In addition to the capabilities of Version 3, Version 4 LAADS, to be delivered in September 2006, will have following new capabilities:
Export Metadata and Browse Images to and Accept Orders from ECHO
LAADS will export and reconcile metadata for the products in its archive or which can be produced by Processing-On-Demand to ECHO, including the URL at which archived product files can be directly obtained. This will be done by LAADS as metadata is ingested or deleted and as files come on or go off disk, using ECHO interfaces for providing and querying metadata. LAADS will eventually accept and handle orders for Post Processed products, and products produced by Processing-On-Demand. This will be done using a LAADS interface to be produced by ECHO and LAADS developers.
These features will allow users to obtain LAADS products through ECHO instead of LAADS if they so chose, and to access at least some of the LAADS options for post processing and file delivery.
Reproject
The LAADS system can reproject MODIS data from the standard MODIS swath and grid projections into all projections supported by the US Geological Survey library (GCTP). When reprojecting data, users can specify the sampling resolution (pixel size) at which the data will be displayed. This allows users to scale data sets as well as reproject them.
Subset by Masking
Users will be able to specify irregularly shaped regions of interest (for example, the border of a state or county). In this case, all data outside the region of interest will be replaced with fill values. This can help focus attention on the region of interest by masking data that are not of interest.
MODIS L0 data Search and Order
In Version 4, users will be able to search and order 5-min L0 data products.
Enhanced Data Product Availability Chart
The data Product Availbility Chart in Version 4 will be modified to include a drill down capability and to include small graphic per product type.
Channel Subset
In V4, LAADS will be able to generate channel-subset data for L1B data. Users need to identify which channels they want and submit the order through LAADS web.
Metadata Searches for Land Products
Version 4 will be able to perform QA metadata searches for Land products.
Capabilities Beyond Version 4 LAADS
Filter, Smooth, and Gap-Fill Post-Process
In future, users can apply filtering algorithms based on QA flags. Data that do not meet the filtering criteria will be subject to user specified replacement algorithms, including replace with fill value, replace with nearest neighbor, bilinear, cubic convolution, or TIMESAT value. The LAADS system will implement the TIMESAT algorithm as well as others recommended by the community to allow users to generate a continuous gap-filled data set.
Mosaic
Mosaic is the process of combining data sets from different files onto the same map grid, creating one output file. In LAADS, the data sets to be mosaicked must be of the same product type (ESDT) and must be from the same data-day. The LAADS system allows users to select MODIS products (either Swath or Grid) that cover up to a global sized region, and mosaic them together into one new product file.
Resample
Users can specify algorithms that determine how data are resampled when scaling and reprojecting images. Resampling algorithms include nearest neighbor, bilinear, and cubic convolution. Most of the MODIS multi-day products are produced on 8 and 16 day time steps. Some users are interested in other time steps such as weekly, 10-day or monthly. Moving to these different time steps involves either interpolating between the MODIS time steps and/or aggregating multiple time steps. In a future release of LAADS, users will be able to specify nearest neighbor, linear, and fitted phenological curves to resample at different time intervals.
File Format Conversion
MODIS data products are stored in files conforming to the HDFEOS data format. The LAADS system can reformat MODIS products from HDFEOS to GeoTIFF. The LAADS system can convert MODIS Swath products to Grid products, so that downstream transformation operations (mosaic, reprojection) can be uniformly applied.
User Defined Post-Processing
The LAADS system will be able to run user supplied tools on MODIS data sets. Users may arrange to submit their tools and applications to the LAADS system. Users may then use the LAADS system to place orders for MODIS data sets, which are then processed by the LAADS system using the user's tools.
Reformat Ancillary Data to HDFEOS Gridded Products
In the future, the LAADS system will be able to convert ancillary products to HDFEOS gridded products so that all LAADS analysis services can be performed on ancillary products.
Re-bin MODIS products
Binning, to be implemented in LAADS, is the process of combining or aggregating data that overlap the same area. When mosaicking or reprojecting, users can specify binning algorithms, including first, last, average, max, min, and highest count.
Training Tutorial
Version 4 LAADS will have a training tutorial package for users.
