| Table of Contents |
|---|
General Caveats
Please note that there are limitations inherent to the Operations Commissioning Phase (shared-risk environment). In the course of the last months we learned a lot During early operations, we continue to learn more about the instruments and the environment that they are operated operating in, and some prior unknown technical limitations were have been encountered. For ViSP for example, some of these limitations had an impact on the frame rate and as such the time spent on an individual slit position and the map cadences the that ViSP can achieve.
In general, summit science operations staff (i.e., resident scientists and science operations specialists) strive to match the requests of any observing proposal the best they can, but there are no guarantees that for example, the lengths of observations, cadences or the requested seeing will be 100% consistent with the proposal. If you do have any questions about (summit) science operations and the execution of your observing program(s) you may also contact the DKIST Program Scientist for Operations atritschler@nso.edu or use the DKIST Help Desk.
...
These caveats cover all the ViSP datasets released so far through OCP 1.58 (December 2023). There have been many major updates to the ViSP pipeline code during the OCP. Many of Most recently these have been to fix bugs or update the robustness of key routines such as the geometric correction. Some updates were in response to optical issues discovered during OCP while others were planned as part of ongoing polarization systems calibration work.
Metadata Issues
There is a problem with the ViSP CDELT2 keyword (see also the following issue), which is the keyword giving the pixel plate scale along the slit. In the early data that was taken with ViSP this keyword provides a wrong value. We have not yet had the chance to fix it in the files that we made available to you but that is something that may be done in the future. Christian Beck, the ViSP Instrument scientist, has supplied the following correct values for CDELT2.
arm 1 630 nm: 0.0298"/pixel
arm 2 397 nm: 0.0245"/pixel
arm 3 854 nm: 0.0194"/pixel
The keywords NAXIS1 and NAXIS2 give the lengths of the spatial and spectral axis of the data array, respectively. However, the keywords CDELT1, CDELT1A, CRPIX1, CRPIX1A, CRVAL1, CRVAL1A, CTYPE1, CTYPE1A, CRDATE1, and CRDATE1A all refer to spectral quantities as opposed to spatial quantities. There may be other keywords that are "transposed" as a result of this that issue. This results in the WCS axis 1 referring to pixel axis 2 in the data array and vice versa.
There are issues with the calculated wavelength and dispersion based on the header information in ViSP as illustrated in Figure 1. No attempt is made to correct these at the moment. A wavelength calibration for ViSP data is being investigated for inclusion in a future version of the calibration pipeline.
...
The ViSP instrument scientist has supplied a better approximate calibration of the wavelength in the 3 arms, show below in pseudo IDL code.
630nm = 629.495+findgen(1000)*1.285/1000. ; dispersion 1.285 pm /px
397nm = 396.418+findgen(1000)*0.77/1000. ; dispersion 0.77 pm / px
854nm = 853.182+findgen(1000)*1.882/1000. ; dispersion 1.882 pm / px
The ViSP data has incorrect pointing information in the WCS headers. This is not currently correctable.
Optical & Algorithm Improvements & Mitigations
Stray Light
Testing had identified multiple sources of stray light in ViSP. Some stray light enters from the sides and top at the end of the camera arms that were open to the environment. Installation of various baffles was done between May and June. These were successful in reducing the stray light from external sources. Data taken through at least OCP 1.4 (ended 06/17/2022) will be contain stray light from this source.
A second source of background light is identified to come from within the beam. Mitigation of this source is in progress but likely not until OCP2. An ad-hoc algorithm was developed and used to fit and subtract this background source using PolCal data (see below).
Polarization Accuracy
Spatial scale for demodulation sampling is yet to-be-finalized.
We are currently investigating several sources of spurious polarized background signals. The QUV continuum error levels are at variable levels around 1%.
Please check the quality report for your data set to note any warning flags and fit failures in PolCal fitting outputs. We have seen data sets where certain variables (transmission of polarization calibration optics) are far away from metrology expectations. There is an expectation of higher than nominal cross-talk levels in these data sets.
We also note that relatively low modulation efficiency is seen in one of the two dual beams for the 397 nm and 854 nm channels (likely due to optics in those arms). Assessment is in progress.
Detailed Description of Data Processing & Optical Issues
Stray Light
...
Data analysis and testing has identified multiple sources of stray light in ViSP. One set of tests done with showed that a significant part of the stray light enters from the sides and top at the end of the camera arms that were open to the environment. Reduction of stray light to about the same level as during a dark current observation with the GOS dark shutter in the beam was only possible when both the camera arms and camera lenses where completely covered. Installation of various baffles was done between May and June. These were successful in reducing the stray light from external sources. Data taken through at least OCP 1.4 (ended 06/17/2022) will be contain stray light from this source.
A second source of background light is identified to come from within the beam itself and is only seen when the beam is allowed to pass through the slit. It affects all ViSP data currently taken. This second source cannot be mitigated with external baffles or enclosures, and must be mitigated using other means. It has a different signature (spatial and spectral) at different wavelengths and/or ViSP arms. Analysis of this signal shows it to be additive and mostly unpolarized, much like a dark or background signal. This signal is a much more significant contribution in frames with overall low flux (e.g. 396 nm and 854 nm channels, and the dimmer of the two dual beams).
In order to mitigate this, the Data Center is currently using an algorithm created by the Polarization Scientist Dave Harrington that uses the PolCal frames taken at a single slit position. We use the assumption that the modulation should be spectrally constant over the ~1nm bandpass covered within a ViSP camera arm. By normalizing each of the raw PolCal intensity spectra to the mean over all spectral pixels, we get a spectrum compensated for the intensity modulation. Variation in these normalized intensities with wavelength is measure of the stray light impact. Spectral invariance of modulation has been confirmed in each camera arm, and also by comparing the intensity modulation curves of both orthogonally polarized beams recorded strictly simultaneously in the dual beams of each camera. The worst behavior has been observed in the 854nm channel, in one of the two beams as seen below:
...
The algorithm finds a single background unpolarized spectrum, that when subtracted from all the individual modulated spectra, minimizes the spectral variation of the mean-normalized intensities. These background intensities correlate well with a known stray light optical pathway. An example of the stray light background in the 854nm channel from June is below. We note that this background is recorded after, and is not impacted by installation of the external baffles discussed above.
...
If we then subtract the background signal from all the individual spectra, prior to normalization, then we get the following modulation-normalized spectral shape:
...
There’s still some residual difference between the normalized spectra, but overall, the normalized spectra look much more spectrally constant. The resulting modulation curves similarly agree much better in overall contrast and uniformity. Ultimately the best way to deal with this issue is to remove the stray light with appropriate optical aperture stops and masking. Work on this topic is ongoing. The algorithm presented above is not a perfect solution. It is only intended to get data “good enough” at this point, and further tuning of the algorithm settings is necessary. You may notice that there are some line artifacts visible. If you do have any questions or if you see any issues with line signals, you are encouraged to ask (DKIST Help Desk.)
Efficiency Drop in 854nm (Arm 3) & 397nm (Arm 2) Beam 2
A polarimetric efficiency drop with one of the two beams in both arms 2 and 3 has been noted.
854nm Beam2 (Arm3) has anomalously low modulation efficiency (35% vs 50%). 397nm Beam2 (Arm2) also has reduced efficiency. It is suspected that this is due to low polarization beam splitter contrast. Optical mitigation likely will be required, and modeling is ongoing. We note that subtraction of the intrinsic stray light (above) improved the modulation efficiency, but it remains at / below 38%.
Demodulation Sampling
Spatial scale for demodulation sampling is yet to-be-finalized. The initial “checkerboard” interpolation pattern, seen in earlier releases of ViSP data has now been fixed. However, several other issues create background signals. Further assessment is necessary to find the appropriate trade-off between errors and smoothness. The QUV continuum levels are at variable levels around 1%. We are currently investigating different observing techniques to also improve the polarization zero point performance.
Cross-talk Possibilities & PolCal Fitting Residuals
Please check the quality report for your data set to note any failures in PolCal fitting outputs. We have seen data sets where certain variables (transmission of polarization calibration optics) are far away from metrology expectations. For example, in some cases the variable representing the polarizer transmission is fitted to be near 100% transmission when the optic is known to be 91.5% transmission +/-0.3%. This fitting error can create cross-talk of all types through the correlation between fitted variables. The settings for polarization calibration are currently being investigated, and we expect improvements in the demodulation matrix accuracy as the algorithms are tuned.
Current ViSP Polarization Coordinates
In December 2022, Tetsu Anan (tanan@nso.edu), performed an analysis of the current polarization coordinates of the calibrated data from the ViSP instrument. Tetsu compared his analysis to both the Stokes Polarimeter (SP) data from the Hinode Solar Optical Telescope (SOT) and to the Helioseismic and Magnetic Imager (HMI) instrument on the Solar Dynamics Observatory (SDO). The polarization coordinates of Hinode/SOT/SP are detailed in Ichimoto et al. (2008), and is show below together with the derived polarization coordinates of ViSP.
...
Hinode/SOT/SP is consistent with SDO/HMI and previous instruments such as the Michelson Doppler Imager (MDI) on the Solar and Heliospheric Observatory (SOHO).
Conclusion
The signs of Stokes U and V in ViSP are opposite to Hinode/SOT/SP and SDO/HMI. This will be rectified in a future reprocessing to enable more easy comparison to data from other instruments.
References
Ichimoto, K., Lites, B., Elmore, D. et al. Polarization Calibration of the Solar Optical Telescope onboard Hinode . Sol Phys 249, 233–261 (2008). https://doi.org/10.1007/s11207-008-9169-9 included,
Improved polarization calibration algorithms to resolve modulation variation along the slit.
Dual-beam intensity balancing prior to combination.
Dual-beam alignment (geometric registration) improvements.
Polarization coordinate frame corrections
Modified flat field algorithms to remove residual gain artifacts.
A brand-new release of ViSP data was initiated on April 23, 2023, which incorporated the changes mentioned above. They are explained in detail in [ViSP pipeline calibration changes (Apr/May 2023)] and demonstrate considerable improvement in the calibration of 630 and 854 nm data. Existing Ca II 396 nm data has also been reprocessed, though the level of improvement is less pronounced.
Spatial Registration
Metadata Issues
The current ViSP L1 data has incorrect pointing information in the WCS headers. This is not currently correctable. When possible, users are recommended to use the VBI data as context for the VISP data, and, again when possible, perform spatial registration of VISP maps with other solar data sets.
The effects of atmospheric dispersion are inherent to all ground-based data. No attempt to correct or augment WCS information has been made to account for atmospheric dispersion.
In particular, with the earlier OCP data, there is a problem with the ViSP CDELT2 keyword (see also the following issue), which gives the pixel plate scale along the slit. We have not yet been able to fix it in all VISP files. The following are the recommended nominal values:
arm 1 630 nm: 0.0298"/pixel
arm 2 397 nm: 0.0245"/pixel
arm 3 854 nm: 0.0194"/pixel
The keywords NAXIS1 and NAXIS2 give the lengths of the spatial and spectral axis of the data array, respectively. In earlier data, the keywords CDELT1, CDELT1A, CRPIX1, CRPIX1A, CRVAL1, CRVAL1A, CTYPE1, CTYPE1A, CRDATE1, and CRDATE1A had referred to spectral quantities as opposed to spatial quantities. While this is believed to now be fixed, there may be other keywords (for example the PCi_j matrix elements) that are "transposed" as a result of this issue.
Slit Width and Stepping Width
Prior to March 2023, the units given in the header for the keywords ‘VSPWID’ and ‘VSPSLTSS’ were incorrect. VSPWID records the VISP slit width in arcsec (on Sky), and VSPSLTSS records the slit raster step size in mm. The equivalent step size in arcsec is determined by the plate scale at the VISP slit, which is 1.613 arcsec per mm. For example, the value of VSPSLTSS for the EID_1_118 data is 0.1326759 mm. In arcseconds, this corresponds to a 0.214 arcsec step size equivalent to the slit width used for this experiment.
Spatial Drift along Slit during Raster Scan
The position of the hairline fiducials is known to drift during the raster scan by a few pixels along the spatial direction. This remains uncorrected in the Level 1 data at this time.
Spectral Registration
Correcting the spectral axis
The wavelength axis information contained in the L1 data headers is not currently accurate. At this time, it is inherited from the raw data values without further corrections applied within the pipeline. Therefore, users must do their due diligence to ensure the wavelength axis is calibrated to an accuracy sufficient for their own purposes. In cycle 1, the VISP wavelengths of 397, 630, and 854 nm had nominal dispersions of 0.77 pm pixel-1, 1.285 pm pixel-1, and 1.882 pm pixel-1, respectively. However, users should be aware that the dispersion is nonlinear. At 630 nm and 854 nm, there is a ~1% change in the linear dispersion value across the observed bandwidth. The figures below demonstrate a spectral calibration of the first spatial step’s characteristic spectrum in two datasets from EID_1_118, as identified. The spectral calibration is achieved by fitting the positions of the strongest telluric lines in the VISP spectrum (in 0-indexed pixel units) and the NSO/FTS Telluric Atlas (in corrected wavelength units). A quadratic polynomial is then fit to these values, as given in the figures. At 397 nm, this comparison likely must rely on the solar lines themselves, requiring closer attention from the user.
...
Drifts of the spectral axis as a function of scan position
The spectral axis is also known to drift as a function of the raster step position. This is not currently corrected in the L1 VISP data. Below, the derived spectral shift in pixel units is given for the EID_1_118 BRWJV dataset as a function of the Slit Translation Position in millimeters.
...
Polarization Accuracy
General Comments
Ensuring very high polarization accuracy is a key objective for all VISP data, and it relies on both calibration data acquired during a specific experiment and system-level calibration data that is used to model the polarization properties of the telescope mirrors between M1 and the Coude laboratory. In addition, a number of factors can influence the polarization accuracy of a given data set. For example, stray light (discussed further below) and seeing-induced polarization can both degrade polarization accuracy in ways that can be difficult to quantify. This is especially the case during the current OCP phase as we continue to learn more about the instrument’s performance. Do proceed cautiously and please use the DKIST Help Desk to ask questions if and when they arise.
Polarization Coordinate Frame
L1 data distributed prior to April 2023 attempted to rotate the telescope’s polarization coordinate frame to the solar frame; however, it only included the parallactic angle (ignoring the P-angle) and applied the rotation in the wrong direction. This led to an incorrect and time-variable polarization reference frame. The revised data products now apply the correct rotation so that the polarization coordinate frame is stable in time and consistent with canonical reference coordinates used by SDO/HMI and Hinode-SP, as shown in the figure below (courtesy of Ichimoto et al. Sol Phys 249, 233–261 (2008). https://doi.org/10.1007/s11207-008-9169-9). The figure references the solar cardinal points where N/S is aligned with the solar rotation axis. The solar rotation axis is misaligned from the geocentric celestial frame by the P angle which varies from +/- the Earth’s obliquity. Additionally, the celestial frame is rotated from the Alt/Az local frame by the parallactic angle. The two combined are sometimes referred to as the solar orientation angle.
...
Residual Crosstalk Corrections
After polarization calibration, it is common to have small-scale residual I-> Q,U,V crosstalk still present in the data, as these crosstalk terms are the most difficult to reliably control. The typical approach used to mitigate this residual crosstalk is to assume the continuum is unpolarized, which is an increasingly valid assumption with proximity to the disk center.
The residual signal within the continuum for the I-normalized polarized state (X) gives a direct measure for the fraction of Stokes I to X crosstalk. Therefore, on an individual spectrum basis, one can apply a correction as follows (e.g. for Q):
...
As, an example, a single Stokes spectrum from EID_1_118 Dataset BRWJV is shown below, before and after applying the correction to Q. (NOTE: U and V corrections not shown). Note how the horizontal streaking and the signatures of the telluric lines are removed from the corrected Q state.
...
The above correction can be applied on a spectrum-by-spectrum basis. It is common for the median crosstalk value to evolve in time as a function of the raster step position number, as shown in the following figure.
...
Detailed Description of Optical Issues and Mitigation Efforts
Stray Light
Testing had identified multiple sources of stray light in ViSP. Some stray light had entered from the sides and top at the end of the camera arms that were open to the environment. Installation of various baffles was done between May and June 2022. These were successful in reducing stray light from external sources. Data taken through at least OCP 1.4 (ended 06/17/2022) will contain stray light from this source.
A second source of background light was identified to come from within the beam. Mitigation of this source has included the insertion of aperture masks within the relay arms, which has considerably improved data acquired later in OCP 1, especially during OCP 1.8 (Dec 2022).
...
For most OCP 1 data, an ad-hoc algorithm was developed and used to fit and subtract this background source using PolCal data (see below). Analysis of this signal shows it to be additive and mostly unpolarized, much like a dark or background signal. This signal has a much more significant contribution in frames with overall low flux (e.g. 396 nm and 854 nm channels, and the dimmer of the two dual beams).
In order to mitigate this, the Data Center is currently using an algorithm that uses the PolCal frames taken at a single slit position. We use the assumption that the modulation should be spectrally constant over the ~1nm bandpass covered within a ViSP camera arm. By normalizing each of the raw PolCal intensity spectra to the mean overall spectral pixels, we get a spectrum compensated for the intensity modulation. Variation in these normalized intensities with wavelength is a measure of the stray light impact. Spectral invariance of modulation has been confirmed in each camera arm, and also by comparing the intensity modulation curves of both orthogonally polarized beams recorded strictly simultaneously in the dual beams of each camera. The worst behavior has been observed in the 854nm channel, in one of the two beams as seen below:
...
The algorithm finds a single background unpolarized spectrum, that when subtracted from all the individual modulated spectra, minimizes the spectral variation of the mean-normalized intensities. These background intensities correlate well with a known stray light optical pathway. An example of the stray light background in the 854nm channel from June is below. We note that this background is recorded after, and is not impacted by the installation of the external baffles discussed above.
...
If we then subtract the background signal from all the individual spectra, prior to normalization, then we get the following modulation-normalized spectral shape:
...
There’s still some residual difference between the normalized spectra, but overall, the normalized spectra look much more spectrally constant. The resulting modulation curves similarly agree much better in overall contrast and uniformity. Ultimately the best way to deal with this issue is to remove the stray light with appropriate optical aperture stops and masking. Work on this topic is ongoing. The algorithm presented above is not a perfect solution. It is only intended to get data “good enough” at this point, and further tuning of the algorithm settings is necessary. You may notice that there are some line artifacts visible. If you do have any questions or if you see any issues with line signals, you are encouraged to ask (DKIST Help Desk.)
Efficiency Drop in 854 nm (Arm 3) & 397 nm (Arm 2) Beam 2
A polarimetric efficiency drop with one of the two beams in both arms 2 and 3 has been noted.
854 nm Beam 2 (Arm3) has anomalously low modulation efficiency (35% vs 50%). 397 nm Beam 2 (Arm2) also has reduced efficiency. It is suspected that this is due to low polarization beam splitter contrast. Optical mitigation likely will be required, and modeling is ongoing. We note that subtraction of the intrinsic stray light (above) improved the modulation efficiency, but it remains at/below 38%.
ViSP Camera Artifacts During OCP 1.8
An issue has been discovered that affects 397 nm data taken during OCP 1.8 (datasets covering Dec 27-30, 2022), and has also been seen in other wavelengths. The ViSP detectors (which are actively cooled) suffered from a condensation buildup which resulted in the pattern you can see as spots on the right-hand side of the detector (and zoomed in, inset left) in the gain image. These spots have such high central core values that they ruin the segmentation algorithm used to find the two hairlines. Without the hairlines being properly detected, the angle calculation used in determining the overlap of the two beams fails. While the routine for finding the hairlines can be (and is being) hardened against such occurrences, it should be noted that this artifact is also in the science data frames. The camera was subsequently allowed to warm to around 0 degrees C and the condensation disappeared. However, this in turn is likely to slightly increase the noise seen in the images.
...
A gain image in the 397 nm channel showing condensation effects. Inset, a zoomed-in image of the spots.
...
VBI Data Set Caveats
The following issues have been found/are being worked on with VBI datasets.
...
We are working on understanding the response of the DKIST Wavefront Correction System (WFC) to varying atmospheric seeing conditions. The WFC system performance is still in the process of being optimized. The Fried parameter keyword AO___001 within your data set headers provides an estimate of the prevailing seeing condition. Due to technical limitations in the way the estimate is generated by the WFC system, the value provided can be misleading. Therefore, the reconstruction process occasionally fails even if a good Fried parameter is estimated. Furthermore, unrealistic Fried parameters (in the meter range) are estimated whenever the WFC system encounters conditions that are too severe for operation. In that case, a complete image reconstruction is not attempted.
We have encountered unexpected technical issues with the VBI cameras leading to a variety of noise artifacts in the data. This includes an overall dynamic noise pattern in the images which is amplified by the reconstruction process, a vertical stripes pattern in the images, and increased noise at strong gradients in the images as seen in particular in high-contrast images. We have developed a variety of algorithms to improve the data quality. These issues continue to be approached from multiple angles to provide further improvements and solutions.
In a few rare cases, you might notice an overexposure in the G-Band and the Blue Continuum images which worsens during the observing sequence (as the sun rises). If there is a second observe observation sequence on the observing day, the exposure time will have been adjusted for this second run.
...
Alisdair Davey
DKIST Data Center Scientist
adavey@nso.edu
Alexandra Tritschler
DKIST Program Scientist for Operations
atritschler@nso.edu