02 - External Data Receipt and Ingest SA Process
| Related page | link |
|---|---|
| Composite Application | 02 - External Data Management |
Scope
The scope of the External Data Receipt and Ingest process covers, as the name suggests, the ingestion of data external to the data center with the notable exception of Summit Data. The data are received via a push from the external systems/roles. Each type of data received by this process has a specific, defined use case.
Goals
Goals |
|---|
| Ingest Calibration OP associations to facilitate calibration |
| Ingest Embargo information |
| Ingest Parameter Values produced outside of the Data Center |
| Ingest Experiment Descriptions |
Key Concept: Processing Candidates
When the summit collects data frames, it does so while running an Observing Program (OP). The success/failure and other metadata associated with the OP is returned to the "Ops Tools" for review by the Resident Astronomer. The Resident Astronomer confirms the OP completion and associates the OP Execution IDs of any other OPs that will be necessary for the calibration of the "observe" OP. This package of OP confirmation and association data is ingested into the data center as a Processing Candidate. Through the Summit Receipt and Ingest process, the Frames from the OP Execution are received. These Frames are grouped by their Processing Candidate information for processing in the Science Data Processing system.
Key Concept: Embargo and Investigator Data
Ingested data frames contain many IDs, as are defined in SPEC-122. However, there are 2 key pieces of metadata (i.e., Investigator list and Embargo status) that are not included in SPEC-122 and are ingested from the Ops Tools system. This data can be joined with data frames via the unique proposal ID.
Ingest Data Acquisition Support Data
The Ingest Data Acquisition Support Data process is responsible for the receipt and ingest of Processing Candidates which represent the top level unit of processing work upon which the Data Center operates. Processing Candidates are free to be updated provided they are not "locked." Locking occurs when candidates are made officially eligible for processing later on, since steps are taken at that point that would cause data inconsistencies should they be updated. In the event an update to a locked Processing Candidate is received, the change is accepted but instead of proceeding with the update, a work ticket is generated to resolve the update and any necessary rework due to processing on an inaccurate Processing Candidate.
Ingest Proposals
The Ingest Proposal process is responsible for ingesting select proposal metadata that is used for maintaining embargo authorizations.
Ingest Proposal Completion
The Ingest Proposal Completion Process is responsible for ingesting the data when a proposal is completed.
Ingest Experiment Descriptions
The Ingest Experiment Description process is responsible for ingesting Experiment Descriptions associated with an Experiment ID. The descriptions are used in the presentation of search results.
Ingest Calibration Parameter Values
The Ingest Calibration Parameter Values process is responsible for ingesting values for already existing Parameters in the Data Center. The process for entering Parameter Values exists within the Data Center already, but this process provides an endpoint for external authorized systems to provide values as well.
