Daniel K. Inouye Solar Telescope

Monthly Operations Report

Status as of November 30, 2023

Table of Contents

1. Science/Technical Highlight (Anyone)

A preprint entitled: Magnetic fields and plasma heating in the Sun’s atmosphere by P. Judge et al. was sumitted to Astro-PH.

https://arxiv.org/pdf/2311.01286.pdf

The research, led by the High Altitude Observatory, USA, was carried out in partnership with experts from the University of Bern, Astronomical Institute, Switzerland and the National Solar Observatory, USA. From their abstract:

“We use the first publicly available data from the Daniel K. Inouye Solar Telescope (DKIST) to track magnetic connections from the solar photosphere into the corona. We scrutinize relationships between chromospheric magnetism and bright chromospheric, transition region and coronal plasmas. In June 2022, the Visible Spectro-Polarimeter (ViSP) instrument targeted unipolar network within a decaying active region. ViSP acquired rastered scans with longitudinal Zeeman sensitivities of 0.25 Mx/cm2 (Fe I 630.2 nm) and 0.5 Mx/cm2 (Ca II 854.2 nm). ViSP was operated in a "low" resolution mode (0.214" slit width, spectral resolution R ~ 70,000) to produce polarization maps over a common area of 105" x 50". Data from SDO and IRIS are combined to ask: Why is only a fraction of emerging flux filled with heated plasma? What is the elemental nature of the plasmas? No correlations were found between heated plasma and properties of chromospheric magnetic fields derived from the WFA, on scales below supergranules. Processes hidden from our observations control plasma heating. While improved magnetic measurements are needed, these data indicate that "the corona is a self-regulating forced system" (Einaudi et al. 2021). Heating depends on the state of the corona, not simply on boundary conditions. Heating models based upon identifiable bipolar fields, including cool loops, tectonics and observable magnetic reconnection, are refuted for these regions with unipolar chromospheric magnetic fields.”

Shown below is Figure 4 from their preprint with the DKIST ViSP data shown in the top two rows:

2. Photos (Anyone)

3. Directorate (Thomas and Dave)

3.1 Activities

3.2 Financial Status

• DKIST operations carried forward approximately $20.83M in unencumbered funds into FY 2024 from previous years.  A total of $0M in new funding has been obligated by NSF thus far in FY 2024 for a total funding amount of $20.83M.

• Financial status of 31 Oct 2023

  • DKIST operational expenditures for the current month are not yet available.

  • Total YTD expenditures for DKIST operations are not yet available.

  • The average DKIST spending for the past 12 months (burn rate; BR) is not yet available.

4. Science Operations (Ali and Andres)

4.1 Status of Observing

• The fifth observing window (OCP2.5) closed on 03 Nov 2023. Observing continues in spring 2024.

• Observing Summary 2023:

  • Total Planned Observing: 127 days.

    • 14 out of the 127 days had to be dedicated and scheduled days on-sun without observing in order to allow (a) the instrumentation group to execute necessary tasks without which observing cannot start or continue or (b) allow science operations to execute tasks that are necessary in preparation for the next observing window in collaboration with the instrumentation group and technical operations.

  • Total Executed Observing: 41 days (out of 113).

    • Total Executed PI Observing: 30 days ; 23 individual PI’s (out of 50) received data.

    • Total Executed Community Observing: 11 days (co-observing with PSP and/or Solar Orbiter).

  • Total Shake and Bake: 11 days ; time necessary to prepare for on-sun time after observatory has been down.

  • Total Snow Clean: 7 days.

4.1.1 Regular Cycle Observing

4.1.2 Solar Orbiter Co-Observing

• N/A.

4.1.3 Parker Solar Probe Co-Observing

• N/A.

4.2 Observing Planning

• Quarter 2024 Priority planning meeting discussing observing time for March and April 2024. The respective OCP2.6 observing window will tentatively open on March 11, 2024 and close in April 2024, some tbd time after the Eclipse. This window will cover Solar Orbiters Remote Sensing Window #13 which opens on March 13 (Sun-Earth line crossing configuration), RSW#14 which opens on March 27 (SO perihelion) and closes after the total Eclipse on April 12. Also covered in OCP2.6 is PSPs 19th perihelion encounter on March 30 coinciding with Solar Orbiters perihelion window in RSW#14 both in quadrature configuration with Earth on the East limb of the sun.

• Development of a DDT proposal for observing on the day of the Eclipse 2024 is converging and on schedule. Potentially, the program can/wi be executed on immediate days prior the eclipse (we do need testing of this program) and the day after the eclipse (pending observing conditions and weather any observations of the corona with this program on days before or after the eclipse are still immensely valuable even if on the day of the eclipse no observing is possible).

4.3 Science Operations Training Activities

• One Scientist from Boulder was in training at the summit as a Resident Scientist to support observations.

• Maui Scientists and Science Operations specialists participated in First Aid, CPR & AED training with the DKIST Safety Group. Most to the Maui Science Operations staff had completed the training with certification for 2 years.

4.4 Science Operations Specialists

• SOS Group held a discussion meeting about the OCP2.5 with the Science Operations Manager. As a result of the discussion, two new procedures were requested to be created, and the other four were requested to be updated on the experience. Pages for creation and update were assigned to different SOSs.

• The Science Operations Manager defined the tasks for the group for December, January, and February. The priorities are to update the training program for the SOSs and update the SOS Group Daily Report tool.

4.5 Cycle Preparation

• Preparations for Cycle 3 Call continue and include among other: FIDO configuration(s), Instrument modes/capabilities (spectral lines, filters, spectroscopy and/or spectropolarimetry), general systems capabilities (telescope and WFC), pointing targets, science priorities, etc.

4.6 Experiment Generation

• All Cycle 2 Experiments need to be revised for observing in 2024 to support (a) a new standard observing script allowing for more flexibility and functionality, (b) DL-NIRSP changes due to the image slicer upgrade, and (c) other instrument specific parameters in need of adjustment. The planning of that effort and the details thereof have started. This will be a phased effort prioritizing all Cryo-NIRSP experiments and those experiments that are compliant with the FIDO configuration selected for OCP2.6.

4.7 Science Operations Management

• Weekly meetings between Program Scientist and Science Operations Manager.

• Biweekly meetings between Science Operations and the Instrumentation Group Leads.

• The science Operations Specialist hiring process continues. The best 3 candidates were invited to Maui in January. We hope to have a final decision early in February.

4.8 Science Operations Tools

• Completed Terraform scripts that will create new AWS Virtual Private Clouds (VPC)s for the production and development stacks of the Ops tools. Currently the Prod and Dev stacks are located within the same VPC, which is an unwanted situation as they share resources, queues, and security tools. The new Terrraform script will be tested and hopefully deployed in December.

• With the hiring of a new Software Engineer, began the migration of the Experiment Architect from VUE 2 to React. VUE 2 is nearing en of life and the migration is required to keep the product on a well supported platform.

• Completed the second round of updates to the OT-OCS Interface Control Document (ICD) - The ICD was updated in significant ways as we learned from implementing and using the ICD during the OCP 1 and 2 Observing periods. The ICD change was major in that is modified, deleted, and added a number of messages in order to more efficiently transfer data and to add functionality that was not previously possible.

• Completed the Beta version of the OPMT migration to React, which include an extensive redesign of the user interface. The changes are now in user review.

4.8.1 Management

• Nothing new to report.

5 Technical Operations (Heather and Paul)

5.1 Site & Facility Status

• Science operations ended Nov. 3 followed by the start of an extended instrumentation and technical engineering period that will continue through Feb. 2024.

5.2 General Site & Facility Repairs, Improvements & Upgrades

• Install of Chiller #2 began and continued on throughout the month.

5.3 Electrical/Electronics/PLC and Facility Management System

5.3.1  Highlights of Recurring Preventative Maintenance Performed in November

• PM for CNC and lathe in the machine shop.

• Telescope mount control cabinet PM.

5.3.2 System Repairs & Upgrades

• All tech completed the installation of status monitors in e lab, mezzanine open office, control room and began install on utility level.

• Completed the electrical installation for chiller #2. Installed the voltage monitor in chiller #2.

• Relocated the conduit for chiller pumps #3 and #4. 

• In the process of installing hardware and wiring for the Vent Gate automation project on Cab. 402 and FMS Cab. L-3-1.

5.4 Mechanical/Thermal

5.4.1 Highlights of Recurring Preventive Maintenance performed in November

• Continuing crane inspection & maintenance plan documentation. 

• Pump Monthly PM’s & Vibration Testing in-progress.

5.4.2 System Repairs & Upgrades

• Worked on the alignment of the Azimuthal Rail – Rail 11 & 12.

• Chilled water side piping is 65% complete. All controls, reliefs, and specialty items are complete and ready for FMS integration and startup. 

5.5 Opto-Mechanical

• Clean OSS area and M4 trunnion

• Inspected M1.

• Sent interferometer to DSSC, setup in lab and test.

• Completed M7 cleaning; subsequent report and discussion.

• Generated M4 cleaning plan.

5.6 Management

5.6.1 Recruitment & Training

• Reviewed recent recruitment applications and request for virtual interviews of 2 candidates. 

5.6.2 Environmental Safety & Health

• No OSHA recordable injuries, first aid, damage or near hits reported. 

• Environmental Obligations: performed monthly rodent grid and annual arthropod inspection. Contract in place through the end of CY24. 

5.6.3 Observatory Community Service, Broadening Participation 

• Nothing new to report.

6 Instrumentation Operations (Bret, Friedrich, and Dave H.)

6.1 Optics

6.1.1 System Optics

• The Vibrometer was used with the GOS lamp November 21st collecting several types of jitter data at the ViSP slit plane focus. Many narrow, repeatable frequencies are present in the data. Impulse response of various mirrors is a work in progress, with several examples showing very clear and repeatable frequency “fingerprints”. This will be useful in upcoming vibration mitigation efforts.
  
  Image jitter of the DKIST beam path from the GOS lamp to the ViSP slit plane recorded over 10 minutes with minimal activity and most systems parked / off. Many narrow and repeatable frequencies are present in the DKIST beam:

  

  • Impulse response of the DKIST beam path at the ViSP slit plane when tapping the back of the M8 glass substrate:

  

• Impulse response of the DKIST beam path at the ViSP slit plane when tapping the VBI-Blue optical bench:

  

6.1.2 Facility Instrument Distribution Optics (FIDO)

• The GS50T950 was unpacked on the summit, shock load data analyzed, and the optic placed in the coude lab clean room, ready for use.
  

  

• The FIDO dichroic suite of coating samples was measured in an updated DSSC lab setup to assess their blocking density, and the coating design utility in upcoming Cryo-NIRSP out-of-band filter blocking improvements. The coatings were not optimized for high density blocking, but BS-680, -643, and -555 are better than OD3 to OD4.

  

• The coating stress assessment for the GS30T500 design continued. Stress values are relatively low, with the predicted surface power figure bias being below tolerances. This coating is particularly difficult as the front side dichroic coating has very low reflectivity at the interferometer wavelength (633nm). Discussions with both vendors are continuing to ensure good selection of a surface bias.

6.1.3 Polarization Analysis and Calibration (PA&C)

• Coating development is progressing on the fringe-suppressed Cryo-NIRSP polycarbonate modulator upgrade project for the 1080nm and 1430nm bands. The 5-layer anti-reflection base coating achieved 0.03% reflectivity, a factor of 3x better than the minimum specification. An initial test of an 11-layer coating adding UV blocking to protect the polycarbonate was successful, with at least 2 more test runs expected to dial in performance. The suite of windows for coating were ordered. The optical specifications were finalized and a Requisition approved (though no PO has been issued by AURA yet). A PO for two more modulator cells was submitted after a re-design of some internal threads caused difficulties in prior projects.

• The optically-contacted MgF2 for a fringe-suppressed modulator for infrared Cryo-NIRSP observations has made some steps forward. A new vendor quoted coatings after our current vendor has struggled to create a successful coating. The specifications, quote, and PO have been finalized and submitted. MgF2 coating samples have been ordered. Testing and development should begin next month on these new samples.

• The ViSP beam balancing retarder pair project had a vendor make progress in fixing an issue with retardance uniformity in polycarbonate layers. The DKIST team worked with the vendor to assess individual manufacturing process steps and update the polarization error budgeting. A new optic was completed with acceptance metrology at the vendor nearly completed. Next month we expect to have received and tested this new optic in the Maui DSSC labs.
  

6.2 Summit Instrumentation

6.2.1 Instrument Systems

• Vibration data was collected while Cryo-NIRSP chillers were turned off. Initial analysis does show that the Cryo-NIRSP is a contributor to vibration on the coude floor.

6.2.2 Wavefront Correction System (WFS)

• New fans were installed on the High Order Adaptive Optics camera to significantly reduce self-induced vibrations. GOS and on-sun testing confirmed that the camera fan vibration is no longer a contributing factor to WFS vibration issues.

• WFS data collected during the November on-sun campaign is being analyzed for vibration issues. The WFS data, along with associated accelerometer data, will be used as part of the ongoing vibration study.

6.2.3 Visible Spectro-Polarimeter (ViSP)

• Some internal ghosting assessments were discussed using calibration data from recent observations. Test observations were planned for future on-sun campaigns. Discussions of additional internal aperture masks and baffles are ongoing.

• The movement issues with the ViSP arm 3 have been investigated and a solution has been approved. New motor shaft collars have been proposed and will be fabricated and installed next year. With the servicing performed this summer, the original fault issue seems to have disappeared, but a more permanent solution is still desired.

• To improve operational efficiency, replacement filter holders for the blocking filters have been fabricated.

6.2.4 Diffraction Limited Near-Infrared Spectro-Polarimeter (DL-NIRSP)

• The DL-NIRSP Machined Image Slicer (MISI-36) was installed, aligned, and tested. The DL-NIRSP instrument was floated and move to adjust the focus for the new optic.

• Initial MISI-36 on-sun site acceptance testing shows a significant improvement in performance over the older Bi-FOIS-36 fiber optics.

• The IR cameras systems routine vacuum maintenance was performed.

• IR camera controller boards were removed and taken to Manoa for upgrades to improve synchronization between the two readout on each sensor. This should significantly reduce the bias jitter seen in some data

• Facility coolant connections were made to the visible camera and heat stop. A coolant to the modulator will be installed after modification to the modulator mount.

6.2.5 Cryogenic Near-Infrared Spectro-Polarimeter (Cryo-NIRSP)

• The Cryo-NIRSP was warmed up and opened for annual maintenance. All four cold heads were replaced. Repairs were made to a camera controller board.

• The new warm filter wheel design was approved and fabrication has begun.

• Some spectrally narrow filter leaks have been suspected of creating artifacts in CN 1079nm science data. The DSSC filter spatial mapping at high density and high spectral resolution revealed spatially variable leaks earlier this year, though filters passed specifications as written. Recent calibrations of data showed curved artifacts that change when using the CN density filters, which have strong spectral gradients in their transmission spectra, enhancing visible flux near the filter leak wavelengths. Models of these leaks by DKIST scientists show reasonable agreement in predicted stray light contamination. An additional dichroic-type blocking filter, based on the FIDO-BS-680 coating should suppress these CN filter leaks by >2 orders of magnitude. A filter optical design, temporary commercial mechanical mount, and coating design shifts from fused silica to a CaF2 substrate have been done this month. We expect specs, quotes, and POs for glass and coatings to be done next month, with filter delivery hopefully by mid February.
  

  

6.2.6 Visible Broadband Imager

• Dichroic blocking filters for VBI-Blue were ordered from a second vendor following last’s months spec development. Some commercial mounting hardware was ordered for both -Blue and -Red channel blocking filters. These filters should be installed before out-of-band flux is sent to VBI by the new Grey FIDO beam splitters.

• A new on-band H-alpha filter was installed. The older filter, which was slightly off the center wavelength, was designated as the off-band filter.

6.7 Multi-Conjugate Adaptive Optics (MCAO)

• The 9 MCAO high-speed cameras were completed by the vendor and arrived at DKIST Boulder for testing and integration into the lab system. These 2500 Hz cameras will provide the data necessary to perform high-speed image motion correction.

• The field stop mirror contract has been awarded.

• Mechanical and electrical parts for the lab system continue to be designed and fabricated.

6.8 High Level Software (HLS)

• The latest software release was deployed to the observatory and successfully tested. This release includes upgrades requested by the summit science team to improve operations.

• The new Data Handling System (DHS) storage system was installed on the summit. This system will provide much faster data transfer and processing capability, along with a storage increase for future VTF data.

6.9 Infrared Cameras

• Two short-wave IR sensors have been designated as potential sensors for the DL-NIRSP camera upgrade. Review of the test results for these sensors are ongoing.

• Two mid-wave IR sensors completed final assembly and began cold testing. One of these failed the cold test.

• Yields for all sensors continue to be lower than expected. Efforts are underway to identify the yield issues.

• The IR cameras controller requirements have been written and are in the final stages of review and approval.

7 Data Center Operations (Bob and Alisdair)

7.1 Hardware Infrastructure

• Continued integrating Buys #4 and #5 hardware into the Data Center. The equipment has been racked and stacked, and is awaiting the installation and configuration of switches by CU personnel in order to be integrated and functioning in the DC.

7.2 Software Infrastructure

7.2.1 Data Center

• As a precursor to updating the Quality Reporting Framework, the quality data was moved from the Postgres DB to another DB as the volume of quality data wa on the verge of causing latency issues with the rest of the data in the search support DB.

• Began working on the design of the Quality reporting framework. The updated framework is meant to make the quality data machine readable, extensible, and usable in the context of scientific data trend identification and dashboarding of Quality data such that history of quality data may be graphically displayed

7.2.2 Portal/Archive/Public

• Updated the Portal to include an alerting function such that breaking news relating to the Data Center, the Data, or the Portal may be displayed to users when such alerts are warranted.

7.3 Science Data Processing

7.3.1 Calibration Pipelines

• Completed the 1st iteration of the DL-NIRSP pipeline and delivered it to the DL-NIRSP science team. It is expected to stay static until the new DL-NIRSP hardware is installed on the summit and new data is taken during the DL-NIRSP verification activities on the summit.

• Updated and refactored VISP pipleplie code to account for and implement the lessons learned during the Cryo NIRSP pipeline development.

• Continued working with the Cryo -NIRSP science team in the Cryo-NIRSP pipeline acceptance. The DC has updated the pipeline based on reviews of the data by the science team, and continues to update and rerun data through the pipeline as improvements are implemented.

• Began working (design and implementation) on the Manual Processing Worker (MPW) - which is the tool that will allow the DC to Calibrate data sets that have one-off issues that cause them to fail when running normally through the pipeline. There are a handful of VISP data sets in the backlog that will need the MPW in order to get processed

7.3.1 Data Released

• There were 9 new Datasets that were created and 12 Datasets that were made public

DataSets Released to PIs

Datasets Made Public