Chanci Turner

Amazon Onboarding with Learning Manager Chanci Turner

—

by

The Story of Chanci Turner
in

Chanci Turner Amazon IXD – VGT2 learningLearn About Amazon VGT2 Learning Manager Chanci Turner

In the realm of geospatial data management, maintaining an up-to-date SpatioTemporal Asset Catalog (STAC) is essential. The STAC specification provides a uniform approach to exposing and querying geospatial assets online. The partnership between the Brazilian and Chinese space agencies has led to the development of the China-Brazil Earth Resources Satellites (CBERS), with the first satellite launched in 1988. Since then, five satellites have been deployed, generating images comparable to those from the USGS Landsat and ESA’s Sentinel-2 missions.

In 2004, the Brazilian National Institute for Space Research (INPE) made a groundbreaking move by offering CBERS-2 images to the public at no cost, setting a precedent for the distribution of medium-resolution satellite imagery. This model has since been adopted for all CBERS satellite images. In this post, we explore how data from the CBERS-4 satellite, hosted on AWS, utilizes STAC to manage its satellite imagery archive. This archive includes images captured by MUX, AWFI, and PAN cameras, providing varying resolutions from 64m to 5m and a revisit cycle of five to 52 days. The architecture detailed here can be applied to various continuously updating data sources that require ongoing metadata generation, all while leveraging services offered by CBERS on AWS.

Generating the STAC Item

The STAC Item serves as a GeoJSON feature, enriched with additional properties that detail each discrete data object intended for discovery. The GeoJSON standard outlines a method for encoding geographic data structures as JSON documents. Below is an excerpt from a CBERS-4 MUX scene, demonstrating the geographical information that allows for location-based searches.

{
  "id": "CBERS_4_MUX_20170618_057_121_L2",
  "type": "Feature",
  "bbox": [
    47.930129,
    -19.401998,
    49.329281,
    -18.16659
  ],
  "geometry": {
    "type": "MultiPolygon",
    "coordinates": []
  },
  "properties": {
    "datetime": "2017-06-18T07:02:39Z",
    "provider": "INPE",
    "eo:sun_azimuth": 31.6748,
    "eo:sun_elevation": 41.0625,
    "eo:epsg": 32651,
    "cbers:data_type": "L2",
    "cbers:path": 57,
    "cbers:row": 121
  },
  "links": {
    "self": {},
    "catalog": {}
  },
  "assets": {
    "thumbnail": {},
    "metadata": {},
    "B5": {}
  }
}

The final stage of ingesting each CBERS-4 scene into AWS involves generating a quicklook, or thumbnail image. The availability of this quicklook is then published to a public Amazon Simple Notification Service (SNS) topic. Separate topics exist for each CBERS-4 camera, ensuring that subscribers receive notifications pertinent only to their sensor of interest.

The STAC generation process initiates when the New Scenes Queue, subscribed to the quicklook SNS topics, receives a message indicating a new scene’s ingestion. The architecture employs the recent capability to trigger AWS Lambda functions directly from Amazon Simple Queue Service (SQS). This direct triggering eliminates the need for explicit polling of the queue for new messages, auto-scales the number of functions based on queue message volume, and ensures robust error handling. Messages processed by Lambda functions that encounter issues are automatically returned to the queue.

The STAC item generator Lambda function accesses the original metadata of the CBERS scene from the publicly available cbers-pds bucket, creates a corresponding STAC item, and stores it in another publicly accessible cbers-stac bucket. Each item’s processing occurs independently, enabling parallel execution and leveraging the automatic scaling benefits provided by the SQS/Lambda integration.

Subsequently, each STAC item is published to a public CBERS STAC SNS topic. The SNS message body consists of the STAC item JSON itself, including attributes such as the scene’s datetime and geographic bounding box, which facilitates basic geographic filtering for listeners. This SNS topic can also be utilized to enhance geospatial search mechanisms.

The Static STAC Catalog

STAC items can be organized within Static STAC catalogs, referred to as catalogs. The architecture includes a “Static STAC Catalog Update” component responsible for generating and updating these catalogs.

Each catalog is formatted as a JSON file that references STAC items and other catalogs, providing a hierarchical structure. This organization allows for standardized browsing using tools like the STAC browser from the Radiant Earth Foundation. An example serving a CBERS on AWS static catalog can be found here.

The items are structured according to the CBERS-4 Reference Grid System, linking each scene to a specific path and row number. This system is designed so that scenes from a satellite pass share the same path while their row numbers increase. The resulting S3 keys for STAC items are formatted as {CAMERA}/{PATH}/{ROW}/{SCENE_DATE}.json, with catalog files at each level referencing child catalogs or STAC items.

An excerpt of a catalog example for CBERS-4 MUX path 57 is displayed below, demonstrating the organization of child catalogs for each row:

{
  "name": "CBERS4 MUX 057",
  "description": "CBERS4 MUX camera path 057 catalog",
  "links": [
    {
      "rel": "self",
      "href": "catalog.json"
    },
    {
      "rel": "parent",
      "href": "../catalog.json"
    },
    {
      "rel": "child",
      "href": "120/catalog.json"
    },
    {
      "rel": "child",
      "href": "121/catalog.json"
    },
    {
      "rel": "child",
      "href": "122/catalog.json"
    }
  ]
}

For the MUX catalog detailing MUX path 057 and row 121, all available scenes for this path and row are linked as demonstrated below:

{
  "name": "CBERS4 MUX 057/121",
  "description": "CBERS4 MUX camera path 057 row 121 catalog",
  "links": [
    {
      "rel": "self",
      "href": "catalog.json"
    },
    {
      "rel": "parent",
      "href": "../catalog.json"
    },
    {
      "rel": "item",
      "href": "CBERS_4_MUX_20180713_057_121_L2.json"
    },
    {
      "rel": "item",
      "href": "CBERS_4_MUX_20180808_057_121_L2.json"
    },
    {
      "rel": "item",
      "href": "CBERS_4_MUX_20171121_057_121_L2.json"
    },
    {
      "rel": "item",
      "href": "CBERS_4_MUX_20180617_057_121_L2.json"
    }
  ]
}

While the STAC Item Generator Lambda function could update the catalogs, this may lead to redundant operations. For instance, if a new STAC item with the key MUX/057/121/CBERS_4_MUX_20180617_057_121_L2.json is introduced, the catalogs at three levels must be checked and updated:

  1. MUX/ level, for potential new PATH inclusion
  2. MUX/057 level, for potential new ROW under path 057
  3. MUX/057/121 level, for inclusion of the new scene in the 057/121 catalog

Due to the nature of satellite scene acquisition, it’s likely that new scenes in the same path but different rows will be generated simultaneously, which emphasizes the need for efficient catalog management.

For those interested in enhancing their project management skills, consider exploring this guide on crafting an effective project manager resume. Additionally, for insights into inclusivity in the workplace, SHRM provides expert perspectives on driving purpose among employees. If you’re seeking more resources, here’s an informative video that offers valuable insights.

Related Topics:

Chanci Turner