Please note that there are limitations inherent to the Operations Commissioning Phase (shared-risk environment). In the course of the last months we learned a lot more about the instruments and the environment that they are operated in and some prior unknown technical limitations were encountered. For ViSP for example, some of these limitations had an impact on the frame rate and as such the time spent on an individual slit position and map cadences the ViSP can achieve.
In general, summit science operations staff (i.e. resident scientists and science operations specialists) strive to match the requests of any observing proposal the best they can, but there are no guarantees that for example, the lengths of observations, cadences or requested seeing will be 100% consistent with the proposal. If you do have any questions about (summit) science operations and the execution of your observing program(s) you may also contact the DKIST Program Scientist for Operations atritschler@nso.edu or use the DKIST Help Desk.
These caveats cover all the ViSP datasets released so far through OCP 1.5. There have been many updates to the ViSP pipeline code during the OCP. Many of these have been to fix bugs or update the robustness of key routines such as the geometric correction. Some updates were in response to optical issues discovered during OCP while others were planned as part of ongoing polarization systems calibration work.
There is a problem with the ViSP CDELT2 keyword (see also the following issue), which is the keyword giving the pixel plate scale along the slit. In the early data that was taken with ViSP this keyword provides a wrong value. We have not yet had the chance to fix it in the files that we made available to you but that is something that may be done in the future. Christian Beck, the ViSP Instrument scientist, has supplied the following correct values for CDELT2.
arm 1 630 nm: 0.0298"/pixel
arm 2 397 nm: 0.0245"/pixel
arm 3 854 nm: 0.0194"/pixel
The keywords NAXIS1 and NAXIS2 give the lengths of the spatial and spectral axis of the data array, respectively. However, the keywords CDELT1, CDELT1A, CRPIX1, CRPIX1A, CRVAL1, CRVAL1A, CTYPE1, CTYPE1A, CRDATE1, and CRDATE1A all refer to spectral quantities as opposed to spatial quantities. There may be other keywords that are "transposed" as a result of this that issue. This results in the WCS axis 1 referring to pixel axis 2 in the data array and vice versa.
There are issues with the calculated wavelength and dispersion based on the header information in ViSP as illustrated in Figure 1. No attempt is made to correct these at the moment. A wavelength calibration for ViSP data is being investigated for inclusion in a future version of the calibration pipeline.
Figure 1: Observed and Atlas Spectrum for Ca II 8542 A.
The ViSP instrument scientist has supplied a better approximate calibration of the wavelength in the 3 arms, show below in pseudo code.
630nm = 629.495+findgen(1000)*1.285/1000. ; dispersion 1.285 pm /px
397nm = 396.418+findgen(1000)*0.77/1000. ; dispersion 0.77 pm / px
854nm = 853.182+findgen(1000)*1.882/1000. ; dispersion 1.882 pm / px
The ViSP data has incorrect pointing information in the WCS headers. This is not currently correctable.
Data analysis and testing has identified multiple sources of stray light in ViSP. One set of tests done with showed that a significant part of the stray light enters from the sides and top at the end of the camera arms that were open to the environment. Reduction of stray light to about the same level as during a dark current observation with the GOS dark shutter in the beam was only possible when the camera arms where baffled. Various baffling attempts were made between May and June and these were successful in reducing the stray light from external sources. Data taken through OCP 1.4 will be contain stray light from this source.
A second source of background light is identified to come from within the beam itself, and is only seen when the beam is allowed to pass through the slit. This second source cannot be mitigated with external baffles or enclosures, and must be mitigated using other means. It has a different signature (spatial and spectral) at different wavelengths and/or ViSP arms. It appears to be additive, much like a dark signal and is most easily seen in frames with overall low flux. In order to mitigate this, the Data Center is currently using an algorithm that uses the PolCal frames taken at a single slit position. By normalizing the raw PolCal spectra to the mean, we get
Notice that in many spectral regions the spectra overlap very well, which is expected. In some regions, however, there is still a large difference. The algorithm find a single spectrum, that when subtracted from all the individual spectra, minimizes the differences in those regions. If we then subtract that signal from all of the individual spectra, prior to normalization, then we get the following:
There’s still some residual difference between the spectra, but overall the spectra look much better. Ultimately the best way to deal with this issue is to fix it optically. The algorithm presented above isn’t a perfect solution. It is only intended to get data “good enough” at this point, and further tuning is necessary. You may notice that there are some line artifacts visible. If you do have any questions or if you see and issues with line signals, you are encouraged to ask (DKIST Help Desk.)
During calibration testing several issues were noted with both the lamp and solar gain tasks.
Lamp gains were being used directly. The lamp was designed to be bright, (1% to 15% of the full solar on-disk DKIST beam) but this came at the cost of spatial uniformity. Many of the lamp gains have their own strong optical response and thus this response was affecting all downstream data (PolCal and science).
The Solar Gain calculation attempted to preserve detector variations through some complicated interpolation steps. Not only was this fundamentally incorrect but the repeated interpolation of narrow solar spectral lines (especially present in 630nm data) left very large spectral residuals. These residuals also affected.
To remove the optical variations from the Lamp Gain we apply a High-Pass Filter (HPF). The detector variations are, by their very nature, at the highest frequency possible in the image so a HPF with a very high cutoff frequency can successfully remove all optical variations from a gain image, leaving only detector variations. Tuning the cutoff frequency of the HPF needs to be done on a per wavelength (arm) basis and researching the best frequencies to use is still being investigated.
To identify and characterize the solar spectrum in the Solar Gain images we need to account for variations in the spectral shape along the slit that occur as a result of physical non-uniformities in the actual slit construction. In other words we can’t simply take the median spectrum along the slit because the true solar spectrum actually does vary along the slit. To compute the “characteristic spectra” we run a moving 1D Gaussian average along the slit. The width of this Gaussian window is an important tunable parameter.
The core gain algorithms are pretty simple: use a filtered Lamp Gain to remove detector variations, and use a Solar Gain (with solar spectrum removed) to remove optical variations. There are some important parameters to tune and further work is necessary to determine the best values for each wavelength region. In some/many cases we may be unable to perfectly remove all optical variations.
Further improvements have been made to the geometric calculations: The algorithm for rotation between beams during the dual-beam merge was improved to fix some failures with some datasets. The algorithm to compute XY shifts between individual modulation states was updated.
A couple of issues with beam 2 / arm3 (currently 854nm) have been noted.
854nm Beam2 (Arm3) has anomalously low modulation efficiency (35% vs 50%). It is suspected that this is due to low polarization beam splitter contrast. Optical mitigation will be required.
854nm Lamp gain has a “step” seen in Beam2. Raw PolCal frames also show this issue. It is unclear as to the origin of this step but it is quite possibly, an internal reflection issue.
Spatial scale for demodulation sampling is yet to-be-finalized. The initial “checkerboard” interpolation pattern, seen in earlier releases of ViSP data has now been fixed. Further assessment is necessary to find the appropriate trade-off between errors and smoothness.
Recent analysis on ViSP PolCal data has shown the continuum polarization errors (residual QUV) are present at levels ~1% PV with strong correlation to solar atmosphere features. The data has spatial and temporal variability from solar granules, lanes, etc. One way to mitigate this is to take the same GOS sequence at N different slit positions and the average those slit positions prior to sending them off to the fitter code. This change to the PolCal implementation is being investigated.
The following issues have been found / are being worked on with VBI datasets.
A wrong value is stored in the CRPIX[N] fits header keywords for proposals carried out in OCP 1.2 / 1.3 (February 2022 through April 2022). This should be corrected for runs from April 2022 onward.
We are working on understanding the response of the DKIST Wavefront Correction System (WFC) to varying atmospheric seeing conditions. The WFC system performance is still in the process of being optimized. The Fried parameter keyword AO___001 within your data set headers provides an estimate of the prevailing seeing condition. Due to technical limitations in the way the estimate is generated by the WFC system, the value provided can be misleading. Therefore, the reconstruction process occasionally fails even if a good Fried parameter is estimated. Furthermore, unrealistic Fried parameters (in the meter range) are estimated whenever the WFC system encounters conditions that are too severe for operation. In that case, a complete image reconstruction is not attempted.
For a movie demonstrating these issues please visit: [LINK]
We have encountered unexpected technical issues with the VBI cameras leading to a variety of noise artifacts in the data. This includes an overall dynamic noise pattern in the images which is amplified by the reconstruction process, a vertical stripes pattern in the images, and increased noise at strong gradients in the images as seen in particular in high contrast images (see Ca II K image at LINK).
We have developed a variety of algorithms to improve the data quality. These issues continue to be approached from multiple angles to provide further improvements and solutions.
In a few rare cases you might notice an overexposure in the G-Band and the Blue Continuum images which worsens during the observing sequence (as the sun rises). If there is a second observe sequence on the observing day, the exposure time will have been adjusted for this second run.
Please don't hesitate to contact us if you have questions.
Alisdair Davey
DKIST Data Center Scientist
adavey@nso.edu
Alexandra Tritschler
DKIST Program Scientist for Operations
atritschler@nso.edu