|
LAADS Capabilities
Last update: August 7, 2006
Version 3 LAADS Capabilities
The capabilities of LAADS have evolved considerably over the last couple of
years driven by the significant input that has been received from the MODIS
land and atmospheres science communities. The initial system (Version 1.3)
was released in December 2004 and has been used extensively to support the
MODIS algorithm developers in developing and refining the codes to be used
for Collection 5 atmospheres and land reprocessing. The Version 2 LAADS was
released in October 2005 specifically to atmospheres community. The current
version, Version 3, was released in early April 2006 to both land the
atmospheres communities and is used to support both atmospheres and land
Collection 5 reprocess starting in April 2006. The Version 3 has many
comprehensive and advanced functionalities, and some major capabilities are
described here.
On-line Archive for Fast Data Access
The LAADS takes advantage of low cost of disk storage to retain hundreds of
terabytes of data on-line, and now archives reprocess Collection 5 MODIS
Geolocation, some L1B products, and L2+ granule and tile products, as well
as Collection 4 forward process L2 and L3 atmosphere data. The LAADS will
not archive all MODIS L1 products because of their huge data volume size,
but a MODAPS Datapool of about 100 TB disks is set up to rolling archive
the "fresh" MODIS L1B and Geolocation data from the forward process and
reprocess. The details of Datapool capability can be seen in the
LAADS and Data Pool README.
Users not only can browse, select, and order LAADS archive data through LAADS
website, but also can directly access LAADS
archive data by using LAADS HTTP server without having to
submit orders.
Rapid and Simple Data Searches and Orders
The intuitive LAADS Web interface has many functionalities to support rapid,
simple and flexible data searches and orders with quick responses (within
seconds) to users.
LAADS Web search page will allow users to search for MODIS data by
specifying the following, but not limited, criteria: ESDT name, file name,
temporal and spatial ranges, ArchiveSet, Day/Night flags, orbital numbers,
TileId, and QA metadata. Users can also search data by browsing data
product images, including global, granule and tile images.
The LAADS also provides user the data availability information for all data
archived in LAADS and DataPool, the Order Track page to track user's
previous orders, and the Tools page that lists many useful tools and their
download sites.
Various data download methods are provided in LAADS: wget, http, ftp-pull,
and ftp-push. Once data are ordered, the data order confirmation and data
download instructions will be immediately given to the user, and an data
confirmation email will be sent to user shall user's email address is given.
Users can not only order archived data in LAADS and Datapool, but also order
post process data. LAADS provides users flexible step-by-step order
instructions to select post-process parameters. The post-process data will
be available from minutes to hours depending on specific post-process
orders. Users have to provide their email address so that the data ready
notification can be sent to the users once the post-process data are
generated.
Processing-On-Demand (POD)
LAADS uses Processing-On-Demand (POD) to recreate products ordered by users
that are not being held in the LAADS archive, typically L1 and and L2, L2G
and daily L3 land products. When such files are ordered, LAADS recreates
the Plan and PGE instances that originally created the products, as well as
the Plan and PGE instances that created any POD inputs that are not archived
(all the way back to PGEs using only L0 and ancillary data if necessary).
Normally LAADS will be configured to leave these plans on pause until
approved by an operator. The system will then run the necessary PGEs until
the ordered files have been produced, after which the user's order will be
fulfilled. The products will remain in the archive for about a week after
which they will again be deleted.
LAADS uses POD because the cost of the processing power needed to reproduce
L1 and L2 and daily L3 products that are old enough that they are
infrequently ordered, is much less than the cost of the disk space that
would be required to store them. (Tape systems are generally as expensive
as disks, as slow as reprocessing, and unreliable.) Thus at a cost of a day
or so delay in getting products to users, Processing On Demand allows LAADS
to provide these products in a cost effective way.
Post-Processes
The LAADS system provides, or will provide, tools and services that perform
the following types of transformations to MODIS data sets: subset, mosaic,
reproject, resample, re-bin, filter, and reformat. In the near future, the
LAADS system will also supports running user supplied tools on MODIS data
sets. Currently available features are:
-
Subset by Region of Interest: Subsetting is the process of selecting a
subset of the data within a data set. Users can specify a geographic
region and request LAADS to remove from their selected product files all
data outside the specified region. In addition to specifying a region by
latitude and longitude, users can specify regions by pixel location (row,
column in the image), and by a combination of latitude/longitude and
row/column (i.e. 5 wide by 3 tall surrounding the pixel at -45.7
longitude, 45 latitude).
-
Subset by Parameter: MODIS product files can contain many different data
sets. Users can specify a list of the data sets they want to keep, and
request LAADS to remove all others from their selected product files.
Data Export to DAACs
The LAADS can export the archive data to DAACs which is configured and
generated by PGEs. Once the data are archived in LAADS, they shall be
pushed to DAACs' PDR partitions within 20 minutes under normal conditions.
The LAADS also performs reconciliation for all the data exported to DAACs
by sending PDRs to DAACs,and pulling and processing the PAN/PDRD feefback
files from DAACs. The unsuccessful PDRs will be automatically retried for
certain data transfer errors or will be manually retried by operators
through LAADS GUI.
The LAADS has number of controls on creating, scheduling, and running the
DAAC-export tasks, so that they may be turned on or off to control the
export tasks depending on the system's status or the users' requests, such
as End User control, PGE control, User-ESDT control, User-host control,
and User-diskspace control.
Some scripts are provided specifically to monitor DAAC export tasks and
generate report reports on export tasks' State, files sent based on a
specified date or a specified days ago, PAN/PDRD related errors, total
number of files and sizes have been sent or to be sent based on given
period of time, UserId and grouped by DAAC Order, Standing Order, and Web
Order.
Data Subscriptions
The LAADS provides Data Subscription service, or called Standing Orders, to
export defined sets of product types or files to a user on a regular basis.
For example a user can specify that all products ingested into LAADS within
a specified period (like HOURS=3), with ESDTs in a specified list, be sent
to a user specified ftp address, or to a stage place in LAADS for 7 days for
user to pull.
LAADS provides following options for users to make their subscription
requests: Data subscription based on ESDTs and/or Tiles with optional
lat/lon box boundaries; Data collection period; Data push (ftp push) or
data stage (ftp pull); Send PDR, DN, or Email notification; Option to tar
or compress all files for delivery; Options to run some or all Post-processes.
Depending on the system's status or the users' requests, the following
controls may be turned on or off to control the StandingOrder task or the
export tasks created by the StandingOrder task: User control, ArchiveSet
control, User-FTP-host control, and User Diskspace control.
Metrics
LAADS provides detail metrics on data ingest, data archvie, data
distribution, and User supports. See Metrics (password protected).
More Capabilities in Version 4 LAADS
In addition to the capabilities of Version 3, Version 4 LAADS, to be
delivered in September 2006, will have following new capabilities:
Export Metadata and Browse Images to and Accept Orders from ECHO
LAADS will export and reconcile metadata for the products in its archive or
which can be produced by Processing-On-Demand to ECHO, including the URL at
which archived product files can be directly obtained. This will be done
by LAADS as metadata is ingested or deleted and as files come on or go off
disk, using ECHO interfaces for providing and querying metadata. LAADS will
eventually accept and handle orders for Post Processed products, and
products produced by Processing-On-Demand. This will be done using a LAADS
interface to be produced by ECHO and LAADS developers.
These features will allow users to obtain LAADS products through ECHO
instead of LAADS if they so chose, and to access at least some of the
LAADS options for post processing and file delivery.
Reproject
The LAADS system can reproject MODIS data from the standard
MODIS swath and grid projections into all projections supported by the
US Geological Survey library (GCTP). When reprojecting data, users can
specify the sampling resolution (pixel size) at which the data will be
displayed. This allows users to scale data sets as well as reproject
them.
Subset by Masking
Users will be able to specify irregularly shaped regions of interest (for
example, the border of a state or county). In this case, all data outside
the region of interest will be replaced with fill values. This can help
focus attention on the region of interest by masking data that are not of
interest.
MODIS L0 data Search and Order
In Version 4, users will be able to search and order 5-min L0 data products.
Enhanced Data Product Availability Chart
The data Product Availbility Chart in Version 4 will be modified to
include a drill down capability and to include small graphic per product
type.
Channel Subset
In V4, LAADS will be able to generate channel-subset data for L1B data. Users need to
identify which channels they want and submit the order through LAADS web.
Metadata Searches for Land Products
Version 4 will be able to perform QA metadata searches for Land products.
Capabilities Beyond Version 4 LAADS
Filter, Smooth, and Gap-Fill Post-Process
In future, users can apply filtering algorithms based on QA flags. Data
that do not meet the filtering criteria will be subject to user specified
replacement algorithms, including replace with fill value, replace with
nearest neighbor, bilinear, cubic convolution, or TIMESAT value. The LAADS
system will implement the TIMESAT algorithm as well as others recommended
by the community to allow users to generate a continuous gap-filled data
set.
Mosaic
Mosaic is the process of combining data sets from different files
onto the same map grid, creating one output file. In LAADS, the data sets
to be mosaicked must be of the same product type (ESDT) and must be from
the same data-day. The LAADS system allows users to select MODIS products
(either Swath or Grid) that cover up to a global sized region, and mosaic
them together into one new product file.
Resample
Users can specify algorithms that determine how data are
resampled when scaling and reprojecting images. Resampling algorithms
include nearest neighbor, bilinear, and cubic convolution. Most of the
MODIS multi-day products are produced on 8 and 16 day time steps. Some
users are interested in other time steps such as weekly, 10-day or
monthly. Moving to these different time steps involves either
interpolating between the MODIS time steps and/or aggregating multiple
time steps. In a future release of LAADS, users will be able to specify
nearest neighbor, linear, and fitted phenological curves to resample at
different time intervals.
File Format Conversion
MODIS data products are stored in files conforming
to the HDFEOS data format. The LAADS system can reformat MODIS products
from HDFEOS to GeoTIFF. The LAADS system can convert MODIS Swath products
to Grid products, so that downstream transformation operations (mosaic,
reprojection) can be uniformly applied.
User Defined Post-Processing
The LAADS system will be able to run user supplied tools on MODIS data sets.
Users may arrange to submit their tools and applications to the LAADS
system. Users may then use the LAADS system to place orders for MODIS data
sets, which are then processed by the LAADS system using the user's tools.
Reformat Ancillary Data to HDFEOS Gridded Products
In the future, the LAADS system will be able to convert ancillary products
to HDFEOS gridded products so that all LAADS analysis services can be
performed on ancillary products.
Re-bin MODIS products
Binning, to be implemented in LAADS, is the process of combining or
aggregating data that overlap the same area. When mosaicking or
reprojecting, users can specify binning algorithms, including first, last,
average, max, min, and highest count.
Training Tutorial
Version 4 LAADS will have a training tutorial package for users.
|
