There are quality issues with the DKIST data I downloaded

Summary

The DKIST Data Center (DC) Archive provides calibrated Level 1 data to the community. To see what calibrations have been applied to make the Level 1 data, see What processing is done to the data to create DKIST Level 1 products? This page provides guidance on what to do if you think that there are quality issues with the data you have downloaded.

Instructions

There are a number of reasons data might have quality issues, both intrinsic and extrinsic. Please work through the following section to see if they match the issues you are seeing. The reasons for the issues are described in detail below. Please note this list is not exhaustive.

Possible Reasons for Quality Issues

1. The observations were taken during a period of bad seeing or the seeing does not match the requested seeing from my proposal.

2. The observations were not as long as expected.

3. Adaptive Optics (AO) was not running.

4. There was a problem with an instrument or camera.

5. There was a problem with the calibration process or there was limited calibration data.

6. There is a bug in the calibration code.

7. Additional calibration was needed (by the user).

Understanding the Quality Report

The Quality Report provides a record of the steps that were used to calibrate the data and any errors that may have occurred in the calibration process, e.g. a less than optimal number of dark frames available for the gain calculation. For each step certain values will be recorded for historical comparison. In addition to calibration and data parameters, the Quality Report provides insight on e.g. the variation of the Fried parameter during an observation run, the sky brightness during the observation, whether the adaptive optics were running, and the calculated noise in an image. The presence of the Quality Report indicates that the Data Center believes the data has been calibrated to the best of its ability. This may however include some compromises in order to calibrate and publish the data. The Quality Report is a good place to look first for hints about Quality Issues.

Possible Solutions

For the possible quality issues listed above, the following actions are suggested:

1. Please check the Quality Report or the headers in the dataset ASDF report to verify the weather conditions [Header Keywords: WIND_SPD, WIND_DIR, WS_TEMP, WS_HUMID, WS_DEWPT, WS_PRESS, SKYBRIGT] and the value of the Fried parameter, [ATMOS_R0]. Check the light level at the start of data acquisition [LIGHTLVL]. This will give you some indication of whether they are a likely cause of the issues you are seeing in the dataset. The DC will attempt to process all data taken, even if such data may not meet the requirements of the given proposal being executed.

   1. Although the DKIST operators endeavour to match the conditions requested for a particular experiment, this is not guaranteed, particularly during the OCP phase. It is likely that the conditions varied during the time the experiment was being taken, and in some cases may not match the requested conditions.

2. If the AO lock fails, if the seeing deteriorates rapidly, if other factors such as high winds arise, experiments may be terminated before there intended duration. Again the DC will attempt to process all data taken, even if such data may not meet the requirements of the given proposal being executed.

3. Check if the AO was running [AO_LOCK].

4. Check the values of the worst health status parameter [DSHEALTH].

5. See What processing is done to the data to create DKIST Level 1 products? to check the steps that are applied to the dataset. Check the Quality Report for notification of issues during calibration especially in relation to problems with insufficient or bad calibration frames. It is likely that the DC will know about these issues already and a decision was made to publish the dataset as it was the best calibration that could be done at the time. If you feel more could have been done, please file a Help Desk Ticket.

6. If you believe you have discovered a bug in the calibration code, please see I found a bug in the DKIST Data Center Calibration Pipeline - now what? Please submit a Help Desk Ticket following the instruction on that page.

7. If you needed to perform additional calibration in order to achieve the quality you wanted for the science you were doing, please submit a Help Desk Ticket that details the additional steps you had to take. As we better understand the calibration requirements of the instruments and receive input from users, we may reprocess the data with additional steps to enhance the quality of DKIST data.

Other Issues

If there are any other quality issues with the data not covered here, please submit a Help Desk Ticket and include the following information

1. Instrument(s) used.

2. Dataset ID(s) of the impacted dataset(s).

3. A detailed description of the quality issues you are seeing and why you believe they are not covered by the information in the Quality Report.

4. Attach screen shots, pictures, and/or analytic results that may be helpful and informative.

Fried Parameter Caveat

The Fried parameter gives an indication of the seeing at a particular moment. The average Fried parameter and its standard deviation are recorded in the dataset Quality Report. While the average Fried parameter can be used as a guide to the seeing quality in a dataset, it should not be done so in isolation and be checked together with the values of AO_LOCK and DSHEALTH.

Related articles