01 - Processing Support
- Anastasia Alexov (Unlicensed)
Design Diagram
The Processing Support data store consists of a relational database supported by the open source RDBMS, Postgres (https://www.postgresql.org/). The majority of the read/writes are infrequent, with one exception for the recording of actual counts when ingesting science frames.
| Table Name | Description |
|---|---|
| dataset_catalog_receipt_account | Aggregation table for expected and actual frames ingested, grouped by dataset_id |
| frame_receipt_account | View combining the expected and actual frame receipts via a left outer join. |
| frame_receipt_actual | Table accumulating the actual frame counts by observing program execution ID |
| frame_receipt_expected | Manifest of the expected frame counts per observing program execution ID |
| input_dataset | Input frames and parameter values either planned for use or used in a recipe run |
| instrument | Valid instruments |
| parameter | Parameters that can be used in processing |
| parameter_type | Valid parameter types |
| parameter_value | Values for processing parameters |
| processing_candidate | Observing program and related metadata that will be calibrated |
| processing_candidate_calibration | Observing program(s) that will be used to calibrate a processing candidate |
| processing_candidate_view | A view of the processing candidate that includes aggregation information about associated recipe instances |
| recipe | Specification of a workflow and parameters |
| recipe_configuration | Default configurations used by recipes |
| recipe_instance | An instance of a recipe for a particular set of input data |
| recipe_instance_view | A view of recipe instances that includes the aggregations of total and complete Recipe Runs associated with each instance |
| recipe_instrument | Instruments that are valid for a particular recipe |
| recipe_parameter | Parameters that are valid for a particular recipe |
| recipe_run | A record of the execution of a recipe instance |
| recipe_run_provenance | A record of code, workflow and input data version information for a Recipe Run referenced on all processed frames |
| recipe_run_status | A list of valid statuses for the Recipe Runs |
Key Concept: Processing Statuses
The Recipe Run is the only object with a state in the Processing Support data model, though it will be common to ask state questions of Processing Candidates. Processing Candidates are related to Recipe Runs via a Recipe Instance and can have their state questions answered either with a boolean attribute on the Processing Candidate or as a function of its related Recipe Run statuses.
| Entity | Status | ||||||
|---|---|---|---|---|---|---|---|
| Processing Candidate | Received (is_locked = False) | Locked (is_locked = True) | Inprogress (Calculated as having 1+ Recipe Instances without completed successfully Recipe Runs) | Complete (Calculated as having all Recipe Instances with 1+ completed successfully Recipe Runs) | |||
| Recipe Instance | NA | ||||||
Recipe Run | Ready | Scheduled | Failed | Completed Successfully | Completed Unsuccessfully | Cancelled | In Progress |