Open

Table of Contents

1. Science/Technical Highlight

• Despite the Maui wildfires and due to the heroic efforts of our mechanical, electrical/electronics, and controls team members, the replacement Chiller 1 install is complete.

• Chiller 1 will be ready for the commissioning visit by our vendor, scheduled to begin September 11, 2023.

Open

Chiller Positioning & Seismic Bracing Installation. Photo by H. Marshall

 

 

 

2. Maui Wildfires

• All DKIST site operations were suspended and the observatory was closed from August 8-16 due to the impact of the devastating wildfires on Maui that impacted the entire community including DKIST staff.

• DKIST technical operations and safety staff made an inspection visit to the summit on August 11. Technical Operations and Instrument Scientist restarted key systems and resumed chiller installation Aug 17-18, re-inspected and started, and checked all systems on Aug 21.

• DKIST was back on-sun from August 22.

3. Directorate

3.1 Activities

• Thomas travelled to Maui August 13-25 to support Maui staff in their efforts to recover from the wildfires and get the observatory up and running again.

• NSF visit to Boulder August 28 - September 1.

  • Thomas prepared slides for the scheduled NSF visit with some input from Dave.

  • Thomas met with and presented DKIST budget information and projections to NSF over the course of the week.

• Cycle 3 call for proposals

  • Thomas and Dave met with Ali and Friedrich to discuss the timeline for the release of the Cycle 3 call.

  • The decision was made to push the Cycle 3 call out to Spring 2024.

  • Dave agreed to work on a release for Solar News notifying the community of the delay for Cycle 3.

3.2 Financial Status

• DKIST operations carried forward approximately $28.16M in unencumbered funds into FY 2023 from previous years.  A total of $4.90M in new funding has been obligated by NSF thus far in FY 2023 for a total of $33.06M.

• Financial status of Jul 31, 2023

  • DKIST operational expenditures for July 2023 were $2.75M

  • Total YTD expenditures for DKIST operations are $23.25M.

  • The average DKIST spending for FY 2023 (burn rate; BR) is $2.32M/month.

4. Science Operations

4.1 Cycle 2 Experiment Generation

• The Cycle 2 experiment generation process has been completed. Nothing else to report.

4.2 Cycle 2 Observing

• The fourth observing window of Cycle 2 (OCP2.4) started on Aug 4, 2023 and will end on Sep 8, 2023.

• OCP2.4 is dedicated to FIDO configuration 3a.

• There are a total of 13 experiments/proposals that requested the FIDO configuration 3a.

• Due to fires and other external conditions, science observations were very limited at the time of this report. The first OCP2.4 science observations were obtained on Aug 28, 2023.

• The fifth observing window of Cycle 2 (OCP2.5) will start on Sep 25, 2023 and will end on Nov 3, 2023

4.3 Operations schedule and training activities

• Daily schedule during the OCP2.4 observing window (Aug 4, 2023 - Sep 8, 2023):

  • Arrive at DSSC (carpools): 4:45am

  • Leaving DSSC: 5:00am

  • Arriving at site: 6:00am

  • Start Observatory Shutdown: 2:00pm

  • Leaving site: 2:30pm

  • Arrive at DSSC (end of day): 3:30pm

• Science Operations schedules two Science Operations Specialists (SOS), and one Resident Scientist (RS) to support daily science operations at the summit.

• Two Boulder Scientists participated as in training Resident Scientists.

5 Technical Operations

5.1 Site & Facility Status

• Two weeks of planned observing and one optical snow cleaning were cancelled due to hazards including high winds, wildfire disaster, and associated power outages.

• The Technical staff were distributed to provide coverage for both the early summer observing shift and core day shift for ongoing preventive maintenance inspections and concurrent maintenance activities. Staff also prepared for upcoming engineering priorities and major maintenance activities.

• On Friday, August 11, after hazardous wind conditions had abated, a small team of TechOps volunteers, with a safety rep, inspected the observatory to ensure its security.

• On Thursday August 17, after power was restored, a volunteer crew of TechOps, Safety, and the Cryo-NIRSP Instrument Scientist returned to the site to restart non-generator backed control systems, high-level software, and primary chillers facility cooling followed by Cryo-NIRSP cryopumps. Chiller 1 installation also resumed as our top priority for long-term observatory availability, and the ongoing biological monitoring continued as required, within 1 week of target grid monitoring date.

• Planning has continued for the upcoming engineering (September 11-22) and on-sun (September 25 - November 3) periods.

5.2 Site & Facility Repairs, Improvements & Upgrades

• The Vehicle Fleet has now completed the transition from Federally-owned property (FOP) to recipient-titled property (RTP) per NSF/AURA direction. Various challenges between documentation requirements of Maui County, NSF, and AURA resulted in substantial inefficiency in this process. Each of the vehicles had to have three safety inspections, five DMV visits were required, as well as, two extra rounds of title re-issue to correct errors and character limitations for owner, address, mileage at transfer, and so on.

• Remedy of Non-Conformances:

  • Wastewater System - the Site Plan for repair of the tanks has been submitted for approval to the authority having jurisdiction.

  • Elevator and LULA (Limited Use, Limited Access) Lifts - scheduled visit was postponed to September due to the fires and outages

• Equipment Refresh - several pieces of equipment have reached or are nearing end-of life.

  • JLG185 Aerial Work Platform - Quotations incoming

  • Forklift - PO placed, disposition of old asset underway

5.2.1 Electrical/Electronics

• Safety-Rated controllers are now available with sufficient significant digits to utilize ephemeris pointing information for at SIL3 (Highest Functional Safety Integrity Level) motion control. DKIST mechanical team deployed a parallel precision encoder on each of the mount altitude and azimuth axes during engineering time earlier this year. This encoder functionality has been tested in parallel with observing since May, and is now ready to be implemented to the Global Interlock System. This will augment the other prime focus safety devices.

5.2.2 Facility Management System

• The communications and instrumentation (e.g. temperature, pressure, flow) as well as the HMI (Human-Machine Interface) have been prepared for the Chiller 1 Commissioning.

5.3 Mechanical/Thermal

• During severe storms, several areas of the observatory experience water ingress. To remedy this, a vestibule design is underway. The architectural design is at 95% Completion Level, and estimation is in progress for fabrication and construction costs.

• The east-facing door, shown below, is problematic in storms coming from the northeast (trade wind / hurricane wind direction). Entry doors on the south side of the observatory also experience this problem to a lesser degree (Kona storm direction); if improvement is sufficient we would deploy similar design there.

• A TechOps Mechanical Engineer, in collaboration with Instrumentation Group, visited KIS in Freiburg Germany, to participate and witness VTF instrument disassembly and familiarize in preparation for reception and installation of the instrument at DKIST

• Technical Operations Mechanical/Thermal, with the support of the Electrical/Electronics team, completed mitigation of a vibration source affecting the delivered wavefront quality of the observatory. In August, testing by the WFC and the TechOps accelerometer array have confirmed that the fix has been effective.

5.4 Opto-Mechanical

• The M3 critical spare has been received and is undergoing inspect and acceptance testing for reflectivity, adhesion, and pressure testing of the coolant channels and fittings.

• The new Heat Stop reflector was installed in May. As of end August, it is unfortunately showing wear and degradation already. This is anticipated to become a long-term wear/replacement item. The TechOps optical team is in discussion with the Optical Systems Scientist about potentially viable cleaning techniques to maintain this reflector, and alternate, potentially more resilient reflective coatings that may be utilized in the future for this critical component.

5.5 Issues & Concerns

Environmental Safety & Health

• Diagnoses of Covid-19 are increasing globally as well as locally. This month ~6 staff members have each missed 5-10 days of work, with rearrangement of operational support assignments required to fulfill the observing shift rotations, perform ongoing preventive maintenance, and make progress on top priority upgrades.

• Distribution of partial information and misinformation by remote colleagues on site communication channels increased difficulty of work coordination for local leadership / incident command.

• A visiting scientist from went AWOL from the observatory site for nearly an hour and a half.

• A load in transit was improperly secured; refresher training on load handling was briefed by safety in the Plan-of-the-Day (POD)

Recruitment Status:

• Positions are currently open for:

  • Mechanical Technician (2 positions) - since March 2022

    • A Mechanical Technician offer has been accepted! The start date is planned as September 25, 2023.

    • One Mechanical Technician onsite interview was cancelled as it was determined we could not meet the candidate’s salary requirement.

    • Thus far in this recruitment, we have had 61 applicants, more than a dozen phone interviews, 4 onsite interviews conducted, 2 offers made, 1 accepted.

  • Electrical Technician (1) - since March 2022

    • In August, two Electrical Technicians were no-shows for scheduled phone interviews

  • Maintenance Technician (1) - since December 2022

    • In August, an onsite interview was postponed to September, due to the wildfire shutdown

  • Mechanical Technician Supervisor (1) - since June 2023

• We anticipate that the difficulties we have faced in recruiting will increase after the recent wildfires. Local cost of living is increasing due to demand for an even more limited supply of housing.

Communication / Outreach / Broadening Participation:

• TechOps hosted two Lunch and Learns (L&Ls):

  • Akamai Interns summer projects presentations

  • Light Path Safety Logic Overview

• In August, summer internships concluded for three Akamai Workforce Initiative interns, and one UH Maui College Intern

  • Designing Encoder-Based Position & Velocity Feedback System Concepts for DKIST’s Aperture Cover

  • Comparative Analysis of Maintenance Management Systems for the DKIST

  • Improving Vibration Data-Collection and Analysis Processes at DKIST

• Observatory and Maui Community Service:

  • SPIE Ground Based and Airborne Telescopes - Call for Papers for 2024 Symposium

  • CMB-S4 Review Panel

  • Informal collaboration with the Space Force / AEOS on repair of their dome mechanism

  • Unofficial team community service day Upcountry Strong - brush and debris cleanup in the Kula Fire area

6 Instrumentation Operations

Planning has continued throughout August for on-sun and GOS activities when observing conditions cause a facility handover from SciOps to InstOps. A new confluence page has been created to organize these tasks so that Resident Scientists and Operators can easily find and execute the tasks in priority order.

Work and schedules for the upcoming September and November/December engineering periods has been mostly completed, although some activities are still in process.

6.1 Optics

6.1.1 Facility Instrument Distribution Optics (FIDO)

• Beamsplitter GS50T950 has been completed by the manufacturer. Shipping to Maui will occur in September.

• Beamsplitter GS30T500 coating development has started.

• M9B spec reviewed with Zygo, ROM quote expected on mounting.

• M9B coatings revisited in preparation for fabrication.

6.1.2 Polarization Analysis and Calibration (PA&C)

• A replacement needle valve for the GOS coolant system has been ordered and Received. Replacement will occur in December during shutdown.

• All PA&C optics were cleaned in August to remove dust and debris.

• Elliptical Calibrator spare fluid jet polishing done, parts received, testing started.

• NSO custom spectral metrology tooling substantially upgraded (light source, filtering, calibrations).

• DSSC Lab bench installed, additional metrology capabilities being ordered & installed.

6.2 Summit Instrumentation

6.2.1 Wavefront Correction System (WFS)

• M3/M6 Look-Up Table (LUT) measurements continued and were determined to be reliable. A new LUT has been released that reduces the nutation seen by the instruments.

• Gathered failure modes for Deformable Mirror (DM) chassis boot-up issues.

• Continued to diagnose routine calibration script issues.

6.2.2 Visible Spectro-Polarimeter (ViSP)

• Acquired and analyzed data to diagnose image jitter issue. The data collection for jitter analysis will continue throughout the year.

• Developed final plans for the arm 3 gear head exchange in September. Dr Alfred de Wijn (NCAR) will support this activity on summit.

• The spare Andor Zyla camera has returned from the manufacturer where it was in for routine servicing. In November, it will be swapped with the next of two remaining VISP cameras that are due for servicing.

• The slit focus was tested and adjusted.

• AR coated, fluid jet polished large ViSP retarder spare received.

6.2.3 Diffraction Limited Near-Infrared Spectro-Polarimeter (DL-NIRSP)

• The MISI-36 Lab Acceptance Testing continued through August. The analysis will be completed in September.

• The MISI-36 Installation plan has been completed and is under review.

• Components for the November engineering period have been identified and ordered, including MISI-36 parts, compressor, and IR camera replacement boards.

• DL-NIRSP cold filters have been tested and accepted.

6.2.4 Cryogenic Near-Infrared Spectro-Polarimeter (Cryo-NIRSP)

• Warm filter wheel re-design has progressed, that will significantly reduce ghost reflections.

• Ordered hardware for maintenance planned in November.

• Started mitigation of a synchronization issue: development of new timing diagrams in collaboration with contractor.

• Cryo-NIRSP de-fringed modulator polishing done, AR coating tests in progress and being assessed.

• Started development of IR filter specifications to significantly reduce stray light at mid-IR wavelength range.

6.2.5 Visible Tunable Filter

• All ongoing engineering activities have been completed and disassembly of the instrument has begun at KIS in Germany.

• Shipping plans have progressed and are being implemented.

6.2.5 Visible Broadband Imager

• Enhanced out-of-band blocking filter designs were progressed to accommodate future FIDO beamsplitters (for example, the GS50T950 & GS30T500 that are ‘gray’ beamsplitters that potentially transfer a fraction of light of all wavelengths into both VBI channels).

  • VBI red channel samples were received and tested, final production filters ordered

6.7 Multi-Conjugate Adaptive Optics (MCAO)

• Progressed on manufacturing of MCAO wavefront sensor parts.

• Finalized design of the high-level software.

• Significantly progressed the preliminary design of the DM4km mount and tables on the coude floor.

• Finalized HOWFS balcony resdesign. Parts are being fabricated.

6.8 High Level Software (HLS)

• Facility Control System to Facility Management System interface updated

• Polarization Modulator Controller software performance improvements

• Data Handling System V2 hardware received

• Wavefront Correction Telemetry data export software interface designed

6.9 Infrared Cameras

• Lab equipment procurement and organization, in preparation of IR camera development and testing

• Discussion with potential consultant for support with camera controller specification and procurement strategy & proposal

7 Data Center Operations (Bob and Alisdair)

7.1 Hardware Infrastructure

• Integrated new FY 2023 HW into the three non-production stacks of the Data Center, i.e., Asgard, Stage , and Sim. These stacks are used for prototyping, development and testing of software, and system updates prior to going into production.

• Production equipment is waiting on the delivery of switches, which are long lead items.

7.2 Software Infrastructure

7.2.1 Data Center

• Proprietary period changes

  • Updated the proprietary period software to start the proprietary period clock when the first dataset of a proposal is generated and released to the PI.

  • Updated the Ops tools interface service to perform a “NoOp” (no operation to be performed) upon the receipt of a “Proposal Complete” message. This message had previously been the trigger for the proprietary period clock starting, but has been superseded by the new paradigm.

• Overhauled and released the Continuous Integration, Continuous Development CI/CD system (TurtleBot) with updated technologies to stay current with evolving development requirements and the state of the art.

• Updated and released the DKIST Data Simulator and DKIST inventory (metadata information database) to account for Cryo-NIRSP data characteristics.

7.2.2 Portal/Archive

• Completed tests of the functionality of the newly enhanced portal alerts system. The alert system will allow announcements related to published data to be posted to the portal as needed (e.g., “ViSP data for wavelength X will be reprocessed starting dd/mm/yy”)

7.3 Science Data Processing

7.3.1 Calibration Pipelines

• Completed the Beta version of the Cryo-NIRSP Pipeline. Preparing to run data through the pipeline and begin the science acceptance iterations.

• Updated the ViSP pipeline in order to run 2.X and previously failing data through the pipeline.

7.3.1 Data Released

• There were a total of 17 data sets released to their PIs and a total of 47 data sets released to the public this month.

DataSets Released to PIs

Datasets Made Public