What processing is done to the data to create DKIST Level 1 products?
General Description of Processing
The DKIST Data Center calibration pipelines are the collection of tools that the DKIST Data Center uses to transform Level 0 (L0) [a.k.a. raw] data delivered from the DKIST summit, into Level 1 (L1) [a.k.a. calibrated] data, which has most of the effects of the telescope and instruments removed to the extent possible. This is performed by first taking the data from the summit and breaking it into sets of data with common characteristics, such as instrument and wavelength. Then each of these datasets are fed into the relevant automated processing pipelines along with the required calibration frames and parameters (such as dark frames, gain frames, and polarization matrices). The various calibration routines are applied to the data one by one, initially using the algorithms supplied by the DKIST instrument partners as part of the science verification process. As DKIST moves into operations, the Data Center will look to make improvements to these algorithms as more information about the data and instruments becomes available.
The primary calibration data product is the set of L1 science frames (L1 “datasets”), packaged into FITS v4.0 format compliant files with fully descriptive headers. In addition to the L1 datasets, a file is created using the DKIST Advanced Scientific Data Format (DKIST ASDF)(*) which contains all the headers of all the science frames in the dataset. In combination with the User Tools this file can be used for finer grained search, selection and download of L1 FITS files. Though all files will contain the processing provenance in their FITS headers, an additional quality report is created, to detail the results of each calibration step and note any issues that occurred during the calibration process. Finally, a browse movie, where appropriate, is also created to help guide users to datasets of scientific interest.
More detailed processing information
The following pages provide a more in depth overview of the calibration steps that are performed on each of the DKIST instruments.
Link to Visible Broadband Imager (VBI) processing information
Link to Visible SpectroPolarimeter (ViSP) processing information
Link to Cryogenic Near-Infrared Spectro-Polarimeter (Cryo-NIRSP) Processing Information
Even More Detailed
If you are interested in the code that the Data Center is using to process the L0 data, the code is publicly available at < DKIST Data Center Calibration Pipelines >.
The Data Center processes the raw DKIST data to create Level 1 products/datasets. Instrument specific processing and algorithm information can be found using the links provided on this page. If you cannot find the information you are looking for regarding data processing, please file a Help Desk ticket.
* Advanced Scientific Data Format (ASDF) information can be found in detail in the following scientific paper.