Summary of Changes in the Reprocessed ViSP Data

Several improvements have recently been implemented in the ViSP data processing pipeline and applied during the reprocessing of existing datasets. These updates address issues related to the handling of characteristic spectra, interpolation artifacts, and array geometry in the Level-1 data products. Additionally, the ViSP Level 1 data now benefit from an absolute wavelength calibration of ViSP solar spectra using routines implemented in the solar wavelength calibration library developed for the DKIST instruments.

Modulation-State Characteristic Spectra Handling (ACP-2)

The algorithm used to derive and remove characteristic spectra in the ViSP gain calibration has been revised to reduce polarization artifacts introduced during flat-field processing. The updated implementation computes a single gain frame per beam rather than separate frames for each modulation state, removes a refinement step that previously shifted spectra during fitting, and separates solar spectral features from instrumental vignetting when deriving the characteristic spectra.

These changes prevent polarized structures in the modulation states from contaminating the derived characteristic spectra and reduce ripple-like artifacts observed in some Level-1 polarization data. By more accurately distinguishing solar spectral signals from optical response effects such as vignetting, the updated pipeline improves the polarimetric accuracy of ViSP spectropolarimetric data products.

Improved Interpolation Used During Beam Registration (ACP-5)

During beam registration, the pipeline shifts one beam relative to the other to align the two datasets before combining them. Previously, the interpolation method used to fill pixels exposed by these shifts could introduce small artifacts.

The reprocessing introduces a revised interpolation scheme that produces smoother, more accurate reconstructions of the affected pixels. This reduces interpolation artifacts and improves the fidelity of fine spatial and spectral structure in the final data products.

Corrected Cropping of Beam-Combined Arrays (ACP-6)

A bug in the pipeline that affected the cropping of beam-combined ViSP Level-1 arrays has been corrected. In some cases, the cropping step removed an incorrect number of pixels after beam combination.

The corrected implementation ensures that the final Level-1 arrays are cropped consistently and contain the intended spatial and spectral extent.

The images below are NaI D-line spectra from a dataset taken on 04/17/2024. As you can see in the “After” image, the whole data array has been shifted due to the fixed cropping of ACP-6. The ghost line bottom left) has been removed due to ACP-5.

Before:	After:

Before:	After:
Open	Open

Wavelength Calibration

The ViSP Level 1 data now benefit from an absolute wavelength calibration of ViSP solar spectra using routines implemented in the solar wavelength calibration library developed for the DKIST instruments. This substantially improves on the values from the telescope software present in the level 0 headers or those provided by the Instrument PI, as noted in the ViSP caveats. For more information on the wavelength calibration, please see the wavelength calibration page.

Overall Impact on the Data

Recent updates to the ViSP data processing pipeline improve the accuracy and quality of Level-1 data products. The reprocessing introduces an updated method for deriving characteristic spectra to reduce polarization artifacts, an improved interpolation scheme used during beam registration to minimize reconstruction artifacts, and a correction to the cropping of beam-combined arrays.

Together, these changes improve the polarimetric accuracy, reduce interpolation artifacts, and ensure consistent array geometry in the final ViSP datasets. Additionally, the wavelength calibration of ViSP data is now exact and reliable.

Dataset Extras

We have also introduced a new set of data products, called dataset extras. All dataset extras are included with a full download of the main dataset and live in a subdirectory called “extra”. The primary purpose of dataset extras is to provide a common context between the user and the DKIST Data Center when discussing issues seen in the L1 data. However, for experienced users, they provide a detailed look at the intermediate calibration products used to create Level 1 data. Additionally, they make it much easier for the DKIST team to monitor trends in various aspects of Telescope operation. A full description of the dataset extras is given at the link:https://docs.dkist.nso.edu/projects/visp/en/stable/dataset_extras.html