DKIST Data Center Archive Search Manual

Owned by Alisdair Davey
Last updated: Jul 26, 2024 by Alisdair Davey
34 min read

Table of Contents

Table of Contents

1. Introduction

The DKIST Data Center Search Portal is one of the main tools which can be used to access DKIST data. It presents a clear, visual interface to the calibrated DKIST datasets. The portal enables search, filter and download functionality for all public datasets, and while the portal allows the metadata of all datasets to be searched, a user will only be allowed to download datasets for which they are a PI or Co-I during the exclusive access period. The portal makes use of search filters which present a rich interface to find datasets of interest, with granularity that stretches from individual datasets through all the datasets generated in an experiment for a proposal. In addition, searching using filters such as wavelength or spectral line, spatial coordinates or whether an observation contains polarization data, allows for broad searches across the entire archive. These filters are key to the operation of the portal, and are described in great detail below. Lastly, along with the ability to create new search filters, another key feature of the portal is being able to save the filters so that they can be reused e.g., to find previously searched for datasets.

1.1 DKIST Data Products

Datasets from the DKIST instruments are both large and complex. In order to maximize the scientific return from the data, it is necessary to provide calibrated data products to the user. The Data Center is responsible for calibrating the raw data received from the telescope into Level 1 data products. As the raw data is received, it is divided into processing candidates that will be used to created one processed dataset. A dataset is roughly defined by the following items:

  • Instrument

  • Broad wavelength

  • Similar instrument parameters

For example, an experiment that uses VBI with observations in Ca II K (393.327 nm) and H-beta (486.139 nm), and DL-NIRSP with observations in Fe XI (789 nm) and Ca II (854.2 nm), would result in 4 data sets. 

Each dataset contains a series of calibrated Level 4 compliant FITS files with image data and attached headers, and is provided as one calibrated frame per FITS file. In addition to the calibrated data files, a dataset also contains a quality report, a browse movie and one ASDF file to support the DKIST User Tools and enable more granular data discovery. All of these files, are available along with the datasets via the search portal and also the User Tools.

1.1.1 Level 1 Data

The calibrated (Level 1) datasets consist primarily of sets of FITS files that are part of the same instrument program. The aim of the calibration is to remove as many of the telescope and instrumental effects as possible. Initial calibration codes are based upon those supplied by the Instrument teams. Undoubtedly, upgrades to the calibration routines or additional routines will be needed as experience is gained in working with each instrument. Data users are encouraged to provide feedback to the Data Center on additional calibration steps they might find necessary to perform their science. Please see also I found a bug in the DKIST Data Center Calibration Pipeline - now what?

1.1.2 ASDF File

The Advanced Scientific Data Format (ASDF) is a next-generation interchange format for scientific data. Although it can be used to store image data as well, the Data Center is using it to store all of the header information as well as world coordinate information and file ordering for all of the FITS files contained in a dataset. The User Tools makes use of this information to present a unified view of the dataset, enable finer grained filtering on any header field (not possible with the Search Portal), and selective downloading of files.

1.1.3 Browse Movie

In order to give a user a snapshot of the data contained in a dataset, a browse movie is created for each dataset.

1.1.4 Quality Report

It is vitally important for both the Data Center and scientists to be confident in the files created through the calibration process. To this end, with each processed dataset, a Quality Report is created. This report details the steps that were used to create the calibrated dataset and any errors that were encountered in the process. This could be, for example, a smaller than optimal number of dark frames to use in averaging for the dark calibration step. In addition, the Quality Report will also contain information about the weather conditions, sky brightness and seeing, which may assist a scientist to find datasets or parts of datasets of interest more easily.

1.1.5 Level 2 Data Products

Separate from the Data Center calibration effort, there is an effort to create higher order data products, such as magnetograms, from the publicly available data. It is envisioned that this effort would regularly create higher order products for a small number of lines. In the future it is hoped that these higher order products will be able to be incorporated in the calibration pipeline and automatically be produced.

1.2 Downloading DKIST Data

When dealing with datasets that DKIST will create (on the order of multiple Terabytes), old methods of data transfer like FTP are just not suitable. In addition, given that the Data Center only has access to finite networking resources, the Data Center needs to be able to shape and limit requests in such a way as to ensure a fair distribution of the resources to all data requestors.

Delivering DKIST data to the scientist thus requires a system that:

  • will be stateful

  • can continue if interrupted without restarting

  • can deal with transfers that might last many days and

  • guarantees the veracity of the downloaded data 

In order to provide scientists with the best possible download experience, DKIST DC is using Globus to transfer files to users. (See Data Transfer With Globus .)

Downloading DKIST data will necessitate a few extra steps, including creating a Globus account and having to define an "endpoint" for a user’s data.

An "endpoint" is one of the two file transfer locations – either the source or the destination – between which files can move. Once a resource (server, cluster, storage system, laptop, or other system) is defined as an endpoint, it will be available to authorized users who can transfer files to or from this endpoint. Authorized Users are users (PI, Co-Is) that are part of the proposal team, for which data was taken. Only authorized users can download the data during the exclusive access period.

The DKIST Data Center will be the source endpoint, and the destination endpoint will be a place at a user's institution or research group facility designed for downloading large datasets. It will also be possible to download datasets/subsets to a user’s laptop, but these datasets will be very large. (See also Logging in to Globus below.)

Users will be able to search for datasets without being logged in, but users will need to authenticate with Globus in order to download datasets. Note that searches may display datasets that are currently in the exclusive access period. Although metadata is available with respect to these searches, a user will not be able to download the data unless they are part of the proposal team until the exclusive access period has expired.

2. Searching for DKIST Data

The primary search mechanism for DKIST data is the search portal. Figure 1. below displays the default view when a user accesses the search portal dashboard. By default, the portal will display data sets sorted by the experiment start data, oldest to newest. Note that while some datasets may show up in the search, they are only downloadable by the PIs or Co-Is of the particular proposal to which the dataset belongs during the exclusive access period.

Screen Shot 2024-04-09 at 12.09.22 PM.png
Figure 1. Default portal view

The search portal uses search filters to construct queries that return datasets that match the many search filter criteria options. One such example is given below, for data being searched between 2024/03/31 00:00 and 2024/03/31 23:59 UTC. An in depth look at the portal filters is given below.

2.1 Search Results 

Screen Shot 2024-04-09 at 1.56.52 PM.png
Figure 2. Basic date search with the portal

The standard search results are summarized below. As filters are applied, the number of matching datasets is continuously updated. The portal also lets a user know how many datasets in total have been processed and how many are currently selected for download.

Below the summary the user will find the results table. The fields are described below in table 1.

 

Clicking on the Dataset ID will give the user more information on that particular dataset. This is illustrated below.

 

The “Calibration Version” links to the latest version of the instrument specific calibration pipeline. Clicking on “Changelog” on this page will allow you to see the changes to the code that have been made between versions of the pipeline code.

2.2 DKIST Dataset Known Issues (Caveats)

 

As the Data Center has worked through the calibration of the ViSP and VBI data, we have worked with NSO scientists and instrument teams to capture the issues we are aware of with the processed datasets. These issues have been captured in the DKIST Data Center Archive Help Desk, in the DKIST Data Set Caveats (ViSP / VBI) page. This page will be continually updated to maintain the latest information about the datasets as they are processed.

2.3 Data Set Exclusive Access Periods

DKIST is implementing an exclusive access period for it scheduled observations. PIs will be given a total of 6 months from when the data is made available by the DC, to analyze their data before it is made public. In the case of datasets being primarily for the thesis work of a PhD student, this term is extended to 12 months. A PI may elect to waive the exclusive access period should they wish too. Although the data itself is subject to an exclusive access period, the metadata is not. As shown above (Figure 5), results may contain data which is not yet public. For more information see DKIST Access and Data Policy - NSO - National Solar Observatory

 

Clicking on the exclusive access icon (see Table 1) will let a user know how much longer the data will be within that period before becoming public.

2.4 Filtering Search Results

The primary mechanism for data selection is by construction of filters. By using the filters, a user may search on any number of parameters. Filters are additive. A dataset must match all filter criteria (including negated criteria) in order to be displayed. By default, the search portal will display the latest available data with no filtering applied. Figure 11, illustrates the default appearance of the portal showing the filter options.

These options are broken down into Basic and Advanced search filters, as well as provide the functionality to load previously saved filter combinations. The use and display of the filters are described in detail in the following sections.

Note that there are currently few sanity checks on the possible filter values or combinations of filters. It may be entirely possible for the user to specify a filter combination for which there are no valid datasets.

Fields that accept multiple values have their parameters OR'd together while values across parameters are AND'd. These are listed explicitly in the table below.

Search Parameters AND'd

Search Parameters OR'd

Date Range, ­­Spatial, Polarization Observations, Wavelength, Exposure, Fried Parameter, Polarimetric Accuracy, Has Spectral Axis, Spectral Sampling, Temporal Sampling, Spatial Sampling

Instruments, Spectral Line, Proposal ID, Experiment ID, Dataset ID, Target Type

Parameters that are OR'd such as Dataset ID may be added more than once to the filter to select multiple datasets.

2.4.1 The Filter Bar

The filter bar displays the current filters that are being used. If nothing is selected, it will appear as illustrated on the left. Within the filter bar there are four filter related options, listed below:

Save Filter. Choose a unique name (max 50 characters) for this filter. This will save the current filter as the name provided.

Load Filter. Select a previously saved filter to load. The drop-down box will list all the filters you have previously saved. (See also Previous Filters)

Share Filter. Will create a URL that you san share with other scientists. This URL will expand to the current filter selection. See Share Current Filter

Clear Filter. This option deletes the current set of filters and reverts back to the default search. (Display latest datasets)

Table 2. The Save, Load, Share and Clear filter options.

The filter bar contains a shortcut to enable users to find publicly available datasets.

Table 3. The Display Only Public Datasets short cut.

When a user adds a filter option, it appears to the right of the Current Filter. In the filter bar example below, the portal will search the latest datasets for those that were taken with the VBI instrument. As this filter was loaded from a previously saved filter, the name of the filter is also displayed.

2.4.2 Negating Filters

For certain filters, such as Has a Spectral Axis, adding that filter without a value sets it to false. The search will look for datasets which do not have a spectral axis, in this case finding VBI data.

2.5 Basic Search Filters

The “Basic” search filters are envisioned to be the primary filters to be used for searching DKIST data. They are Date Range, Instruments, Spectral Line, Spatial and Polarization Observations. These filters are now described in detail.

2.5.1 Publicly Available Data

This filter enables a user to specify if they want to toggle their search for all data publicly available datasets or for data that is currently within the exclusive access period.

Using the dataset access filter, a user can toggle between publicly available data or data that is still within the exclusive access period. The default (no filter) will search both kinds of datasets. Clicking on ADD ACCESS FILTER adds Dataset Access to a user’s current list of selected filters. Note that you can also select the shortcut on the Filter Bar to enable users to find publicly available datasets.

2.5.2 Experiment Start Date Range

This filter limits the results to datasets that intersect the specified Dataset Creation Datetime Range. DKIST data is available from the earliest calibrated observation up until the latest calibrated observation which may lag the current date considerably. Be aware that too wide a date range may end up running into the selection limit of 1000 datasets.

 

 

Unlike the Experiment Start Date, the Dataset Creation date is determined by when the data is calibrated and published to the portal. Especially for VBI datasets, this data may lag the Experiment Start Date considerably. It is useful if you want to see the latest data calibrated by the Data Center. Clicking on ADD DATASET CREATION DATE FILTER adds DATASET CREATION DATE to a user’s current list of selected filters.

2.5.3 Dataset Creation Date Range

This filter limits the results to datasets that intersect the specified Date Range. DKIST data is available from the earliest calibrated observation up until the latest calibrated observation, which may lag the current date considerably. Users should be aware that too wide a date range may end up running into the selection limit of 1000 datasets.

Using the date picker, a user selects a start and end date for the observations they are interested. If they do not specify one of the options, it defaults to either the start or end of the available calibrated data. Clicking on ADD DATE FILTER adds Date Range to a user’s current list of selected filters.

2.5.4 Instruments

This filter allows the selection of datasets from one or more of the current DKIST Instruments. This menu is built dynamically so that a user will only be able to choose instruments for which observations have already been taken (Note: VTF will likely be added in 2024.) As more instruments are commissioned and datasets from them become available, they will appear in the drop-down menu. Clicking ADD INSTRUMENT FILTER adds Instruments to the user’s current list of selected filters.

2.5.5 Spectral Line

This filter allows the selection of datasets by one or more of the spectral lines observed by the DKIST instruments. This is provided as a convenience to the user. Wavelength searches are also available. However, for the VBI, VTF, ViSP and DL-NIRSP instruments, the band-passes in which the instruments observe have already been defined to maximize the scientific return from the instrument. The Spectral Line filter provides an easy way to select these instrument band-passes. For the ViSP instrument, which doesn’t use specific wavebands, a selection of the most common lines has been made. The ViSP line list will be updated as the DC gains more experience with the ViSP wavelengths being searched. For ease of selection, a user can filter by one of the DKIST instruments (i.e., choose Instrument to filter lines) to limit the line selection to that particular instrument. Clicking ADD SPECTRAL FILTER adds Spectral Line to the user’s current list of selected filters.

See Appendix: Supported Spectral Line Searches for a complete list of the current supported spectral line list.

2.5.6 Spatial

Note that there is a known issue with the WCS headers in both ViSP and VBI calibrated datasets, that will affect the ability to doing spatial / bounding box searches. For observations in the OCP phase a spatial search may return incorrect results or no results at all. For more details, see the DKIST Data Set Caveats (ViSP / VBI) page.

This filter allows the selection of datasets using a Spatial Bounding Box. To select a spatial match for datasets, enter the upper and lower corners of a helio-projective cartesian Spatial Bounding Box in arcseconds.

A user may choose a bounds match such that the datasets Contain the bounding box, Intersect the bounding box or are Wholly Contained by the defined box. Effective limit for the bounding box is the maximum off-point for the telescope of ~1.5 RSUN, or -1440” to 1440” in Solar X and -1440” to 1440” in Solar Y. Clicking ADD SPATIAL FILTER allows the user to Add a spatial bounding box to the user’s current list of selected filters.

Figure 21, to the left, illustrates the available bounds matches. The blue rectangle labeled A represents the corners of the spatial bounding box. The red rectangles represent the spatial extent of the observations. Number 1 represents the Contains match, where the observation data contains the spatial bounding box. The red rectangle numbered 2 represents the Intersects match, where some part of the observation box overlaps the spatial bounding box. The red rectangle numbered 3 represents the Wholly Contained match, where the observation box is wholly contained by the spatial bounding box.

2.5.7 Polarization Observations

This filter allows the selection of datasets depending on whether they contain Polarization Observations. All DKIST instruments except for VBI have the ability to make observations at different polarization angles. Select Polarization Observations Only or Intensity Observations Only to choose between these observations that contain polarimetric data, and those that don’t (Intensity Observations Only). By default (no filter) both observation types (and spectroscopic data) will be selected. Clicking ADD OBSERVATION FILTER allows the user to add Polarization Observations to the user’s current list of selected filters.

 

2.6. Advanced Search Filters

The ‘Advanced’ search filters provide additional filters that may be used to refine the user’s search for DKIST data sets. They are Proposal ID, Experiment ID, Dataset ID, Numerical wavelength search (primarily for ViSP), Exposure, Fried parameter, Polarimetric accuracy, Target type, If data has a spectral axis, Spectral sampling, Temporal sampling and Spatial sampling. These filters are designed to allow users to locate datasets more quickly, such as if a PI knows their Proposal ID, or has interest in a specific scientific objective, such as a sunspot (Target type) observed with excellent seeing (Fried parameter).

2.6.1 Investigated By Me

This filter quickly allows a principal investigator to find all datasets associated with their login. A user will only see this option if they have logged in via Globus first. Click ADD INVESTIGATED BY ME FILTER to add Investigated By Me to the user’s current list of selected filters.

2.6.2 Proposal ID

 

This filter allows the selection of datasets containing a particular Proposal ID. All observations scheduled with DKIST have a Proposal with a unique ID associated with them. Entering a Proposal ID here, e.g., pid_1_50, and all datasets associated with that ID are displayed. To select multiple Proposal IDs, separate them with commas. Attempting to add a new Proposal ID filter when you already have one selected, will just replace the Proposal ID selection with the new one. Note that while all metadata is available for searching, a user will only be able to download data for which they are the PI or a Co-I during the exclusive access period. Clicking ADD PROPOSAL ID FILTER adds Proposal ID to the current list of selected filters. A user may add more than one Proposal ID to search for by adding a new search ID and clicking on ADD PROPOSAL ID FILTER again.

2.6.3 Experiment ID

This filter allows the selection of datasets containing a particular Experiment ID. Experiments are how DKIST transforms a proposal into scheduled observing plans. A proposal may contain one or more Experiments. Entering an Experiment ID, e.g., eid_1_50, displays all datasets associated with that ID. To select multiple Experiment IDs, separate them with commas. Attempting to add a new Experiment ID filter when you already have one selected, will just replace the Experiment ID selection with the new one. Note that while all metadata is available for searching, a user will only be able to download data for which they are the PI or a Co-I during the exclusive access period. Clicking ADD EXPERIMENT ID FILTER adds Experiment ID to the user’s current list of selected filters.

2.6.4 Dataset ID

This filter allows the selection of datasets containing a particular Dataset ID. For each dataset created by an Experiment, a unique ID, known as the Dataset ID, is assigned. Entering a Dataset ID, e.g., AMJJB, selects a particular dataset. Attempting to add a new Dataset ID filter when you already have one selected, will just replace the Dataset ID selection with the new one. Note that while all metadata is available for searching, a user will only be able to download data for which they are the PI or a Co-I during the exclusive access period. Clicking ADD DATSET ID FILTER adds Dataset ID to their current list of selected filters. Using the Dataset ID is the easiest way to share a link to a particular dataset.

2.6.5 Numerical Wavelength Search (Primarily for ViSP)

This filter allows the selection of datasets between a minimum and maximum Wavelength. To search on specific wavelengths or wavelength bands, a user enters a minimum and maximum wavelength in nanometers, e.g., 1000nm and 1300nm. Any intersection will match. This is designed to be of particular use for the ViSP instrument. Clicking ADD WAVELENGTH FILTER adds wavelength range in nm to a user’s current list of selected filters. Whereas the other DKIST instruments have predefined wavelength bands based on instrument filters, ViSP can image the sun in three bands simultaneously anywhere in the 380‒900 nm range. To search multiple wavelength regimes, add multiple min and max wavelength pairs to your search filters. such as shown in the image below.

2.6.6 Exposure Time

This filter allows the selection of datasets based on the minimum and maximum Exposure Time. To filter on Exposure Time, a user enters the minimum and maximum Exposure Times (in seconds) to search for datasets. The Exposure times should be common across a dataset. The user may enter either or both of the Exposure bounds. Any intersection will match.

The lower bound will be assumed to be zero if not specified and the upper bound infinite if not specified. Clicking ADD EXPOSURE TIME FILTER adds exposure time range in seconds to a user’s current list of selected filters.

2.6.7 Fried Parameter

This filter allows the selection of datasets based on the Fried Parameter. The Fried Parameter (commonly referred to as r0) is a measure of the quality of optical transmission through the atmosphere due to random inhomogeneities in the atmosphere's refractive index. A Fried parameter larger than 12-15 centimeters can be considered generally excellent seeing. To filter on the atmospheric seeing conditions, a user enters the minimum Fried parameter (worst seeing) in centimeters (minimum allowed value is 0.1 cm) they are willing to accept. All datasets with a Fried parameter larger (better seeing) than or equal to the minimum Fried parameter will match. Clicking ADD FRIED FILTER adds minimum Fried parameter in centimeters to the user’s current list of selected filters.

Caveat

The Fried parameter gives an indication of the atmospheric seeing at a particular moment. The average Fried parameter and its standard deviation are recorded in the dataset Quality Report. While the average Fried parameter can be used as a guide to the seeing quality in a dataset, it should not be done so in isolation and should be checked together with the values of AO_LOCK and DSHEALTH. It also does not take into account uncommon paths between instruments, or for example instrument vibration which will affect the observed data quality.

2.6.8 Polarimetric Accuracy

This filter allows the selection of datasets based on the Polarimetric Accuracy. Polarimetric Accuracy is the absolute error in the measured fractional polarization which is normally written as P/Ic or P/Icont. To filter on Polarimetric Accuracy, a user enters the maximum Polarimetric Accuracy they are willing to accept. All datasets with polarimetric accuracy smaller (improved polarimetric accuracy) than or equal to the specified polarimetric accuracy will match. Clicking ADD POLARIMETRIC ACCURACY FILTER adds maximum accepted accuracy P/Ic to the user’s current list of selected filters.

2.6.9 Target Type

This filter allows the selection of datasets based on the Target Type. Target Type is a designation of the feature type of the observed target as noted by the operators. Target Type is a controlled vocabulary and has values of unknown, quietsun, sunspot, pore, plages, spicules, filament, prominence, coronalhole, quietcorona or activecorona.

To filter datasets based on Target Type, a user selects one or more of the defined Target Types. All datasets tagged with one of these Target Types will be displayed. Clicking ADD TARGET TYPE FILTER adds target type to a user’s current list of selected filters.

2.6.10 Has Spectral Axis?

This filter allows the selection of datasets based on whether the dataset contains a spectral axis. This will include both spectroscopic and spectrapolarimetric data. Only datasets that contain a Spectral Axis will be displayed when selected. Clicking ADD HAS A SPECTRAL AXIS FILTER adds Has Spectral Axis to a user’s current list of selected filters.

2.6.11 Spectral sampling

This filter allows for selecting datasets based on their spectral sampling. To filter datasets based on Spectral Sampling, a user enters the maximum spectral step in nanometers they are willing to accept. All datasets with Spectral Sampling smaller than or equal to this value will match. Clicking ADD SPECTRAL SAMPLING FILTER adds Has Spectral Sampling to a user's current list of selected filters.

2.6.12 Temporal sampling

This filter allows for selecting datasets based on their temporal sampling. To filter datasets based on Temporal Sampling, a user enters the minimum and maximum temporal step values in seconds they are willing to accept. They may enter either or both of the Temporal Sampling bounds. Any intersection will match the subject. Clicking ADD TEMPORAL SAMPLING FILTER adds Temporal Sampling to a user's current list of selected filters.

2.6.13 Spatial sampling

This filter allows for selecting datasets based on their spatial sampling. To filter datasets based on Spatial Sampling, a user enters the minimum and maximum spatial step values in arcseconds they are willing to accept. They may enter either or both of the Spatial Sampling bounds. Any intersection will match subject to other filters. Clicking ADD SPATIAL SAMPLING FILTER adds Spatial Sampling to a user's current list of selected filters.

2.7 Loading, Saving and Sharing Search Filters

Functionality has been created that will allow a user to save the search filters that are created, load them again in the future and to share links to particular searches. Note that you must be logged in, in order to access this functionality.

2.7.1 Save Filter

 

Clicking on the Save Filter icon (shown to the left) pops up the dialog box on the left. This allows the user to save the current set of filters. Note that you will need to be logged in for this feature to work, but this will enable you to access this feature from any browser or browser session. After choosing a unique name (max 50 characters) for this filter, the current filter will be saved as the name provided in the user’s current browser. It will also update the current filter with the filter name, as shown below.

2.7.2 Load Saved Filter

Clicking on the Load Saved Filter icon (shown to the left) pops up the dialog box on the left. This allows a user to choose from the set of their previously saved filters. Note that you will need to be logged in for this feature to work. Choosing the filter name desired and clicking on LOAD FILTER will repeat the previous search with those same filters applied. Note that you can also load a filter by clicking on the filter name from the Previous Filters menu. See Section 2.8 below.

 

2.7.3 Share Current Filter

Once you have saved a filter, it will show up under Previous Filters on the left hand menu, as shown in Figure 41. From the Previous Filters menu you can either share or delete a particular filter, in this case one named “PSPEncounter12”.

Clicking on the Share Current Filter icon (shown to the left) allows you to share a filter that you have saved. This could be useful for example to give someone a direct link to a particular data set or proposal.

 

 

Click on the copy button (left) to create a URL that you can then share with people. Note that the person receiving this URL does not need to be logged in to access the link. You may need to wait a few seconds for the link to show up.

 

 

 

In this case you may then share https://dkist.data.nso.edu/?shareid=55L9 with your colleagues and this link will take them directly to the data you selected.

Note this is the publicly available ViSP and VBI data taken coincident with the Parker Solar Probe Encounter 12.

In order to download data, a user will need to be logged into Globus. If they are not logged in, they will be prompted to do so (Figure 44 below). See Logging in to Globus for more details.

3.1 Download Endpoint

 

Once a user has logged in, they will be prompted to select the Download Endpoint they want to use.  This is the Globus Endpoint that will be used to download their files to. The default is Endpoints owned by me.

There are a number of other endpoint choices. It is likely that a number of people will want to download the data to a personal machine as illustrated in Figure 46. While this is a traditional way to download data, users are warned that the datasets are likely to be very large and users should investigate whether institutional endpoints are available.

If a user does want to use a personal endpoint (Globus Connect Personal endpoints owned by me) they will need to already have created and activated the endpoint. If they select All endpoints, they will be prompted to filter their selection, since this includes all visible Globus endpoints around the world!

3.2 Download Directory Destination

Select the data you wish to download data to, or create a new folder for the data.

3.3 Dataset(s) for Download

Click on “Download selected files to” to initiate the download. You will see:

You can follow the link in the status message to watch the transfer in the Globus Management Console. You may dismiss this message without affecting the data transfer.

TASK DETAILS Task ID: 84a59178-5fb4-11ed-b0b5-bfe7e7197080 Task Type: TRANSFER Status: SUCCEEDED Source: DKIST Data Transfer (d26bd00b-62dc-40d2-9179-7aece2b8c437) Destination: ARD Personal (b25a84f6-1dac-11ed-8dbb-9f359c660fbd) Label: DKIST data transfer initiated 2022-11-08 sais https://app.globus.org/activity/84a59178-5fb4-11ed-b0b5-bfe7e7197080/overview

Once the transfer has completed you should receive an email confirmation such as the one above.

  • How do I find my data?

To find data from your proposal(s) quickly, use the Investigated By Me filter described in Section 2.5.1. This will allow you find all datasets associated with you as a PI or Co-I.

  • How do a share the data from my proposal with a colleague?

See the Share Current Filter functionality described in Section 2.8. This will allow you to share data down to the dataset level, based upon any filter criteria you may choose. Although you need to be logged in to save your filter, the URL you create when you share your current filter is accessible without logging. However a user must always be logged in to download data. Note that if the data you are trying to share is still within the exclusive access period, then only Co-Is will be allowed to download it. Once it is publicly accessible, then anyone may download the data

  • How do a find publicly available data?

There are two methods that allow you easily select only publicly accessible data (data which anyone logged in can download). The simplest method is just to click on the Only Public Datasets option in the filter bar.

This automatically sets the Dataset Access filter to Public. You may also set the Dataset Access filter explicitly from the first filter option under Basic Filters from the main dashboard. See also Publicly Available Data.

 

In order to download data, request raw data, or access non-public data linked to a username, a user must be logged in to Globus.

Clicking on LOGIN begins the Globus login process.

Globus provides a number of different methods by which a user may authenticate. They may use their existing organizational login, login with their Google or ORCID IDs, or create their own Globus ID. If the user has submitted a proposal, they should however use the same ID that they used for logging into the proposal architect tool.

Globus has linked with many organizations to provide the option of logging in via institutional credentials, in this case, the University of Colorado at Boulder.

Clicking “Continue” after selecting a user’s institutional provider will lead to them being prompted to enter their institutional credentials, as shown below:

If a user’s institution is not in the drop-down lists, or they do not want to login via Google or ORCID, they may use an @globusid.org account if they already have one.

If they want to use a Globus ID and don’t already have one, then clicking on “Sign Up” will start the process. Filling out the form illustrated below creates a new Globus ID.

Once the user has authenticated by any of the means listed above, they will be prompted to allow DKIST to access the details it needs to support data downloads and requests involving data sets within the exclusive access period. Clicking on Allow enables the access DKIST requires. The user will then be prompted for a name for this connection if they have not already supplied one.

Once the user has logged in, their login ID will appear in the top right corner. Should they need to logout, the user can do so by clicking on the arrow icon to the right of their login ID.

For an in depth explanation of Globus endpoints, see the Globus FAQ: Globus Connect and Endpoints. Globus provides detailed instructions for Mac, Windows and Linux personal endpoint creation. Rather than duplicate those documents here, we suggest the user consult the appropriate document for their operating system. Note that in each case, Globus provides software for the user to download and install.

NOTE: If you downloaded a Globus Personal Endpoint before before Sept 12, 2022 it must be updated. See Globus Transfer Service Certificate Authority Update for more details.

The DKIST Data Center Archive provides calibrated Level 1 data to the community via the User Portal for download. It does not provide access to raw (Level 0) data. However, special requests to the DC for raw data can be made using the request form below, provided users have a fully justified reason for requesting raw data. Raw data requests will be reviewed on a case by case basis, and there is no guarantee they will be granted. It is assumed that a user will already have downloaded the L1 data before requesting the raw data. The user is required to specify the dataset IDs from the L1 data that corresponds to the raw data they wish to download. Raw data is not searchable and cannot be filtered by instrument or other similar parameters. If the request is approved, the user will receive access to ALL the raw data for at least one observing program as executed at the telescope.

Note that a user must be logged in the request raw data. If they are not logged in, the first thing they will see is the box shown to the left, informing the user that they must login via Globus. See the section Logging into Globus for more details.

Once a user has logged in, they will be presented with the following page. The page will automatically pull in the user’s name and email from the Globus login.

To request raw data, a user must have already downloaded the Level 1 data sets. Then, the user enters the Level 1 dataset IDs (with the correct format), separated by commas, that they are requesting in the first box. Note that no filtering is possible in this situation. If granted access, the user will get access to all of the files in a dataset. In the second box, the user should enter the full justification for the request. The Data Center requests that users please be as explicit as possible in the reason for their request. This will then be evaluated. When a request is ready, hit SUBMIT REQUEST. An acknowledgement of the user’s request will be presented.

As a reminder, submitting a raw data request does NOT guarantee that the request will be granted. Requests will be vetted on a case by case basis. Note that data can only be released to the PI and proposal team members during the exclusive access period.

If the request is approved by the DKIST management, then the user will receive a notification from the DC that the request was approved. Data Center staff will contact the user for additional information on requested datasets. Data Center staff will provide the raw data endpoint details for this Globus download as well as the allotted time the data will be made available to them. The user must initiate the Globus data transfer within the allotted time - the requested data will then be transferred to the user endpoint. If the user has requested notifications from Globus upon transfer completion, the user will receive a notification when the transfer completed (status: Success/Error). If a raw data request is denied, the requester will be notified by email with the reason.

  • If a user has multiple portal pages open in their browser, changes in one page will not be reflected in another one until the user refreshes the other page.

  • Currently, the only option for filters such as the Wavelength is to OR the selected wavelength ranges. An example of this is shown below. The ability to specify both OR and AND is a feature that the DC plans to support in a future release.

  • There is a known issue calculating the observed VBI bounding box. Though the values are correct in the data files, they are being calculated incorrectly for display in the portal. This issue is being worked on. (01/10/2020)

In the unlikely event there is an issue with the DC portal, the following message will be displayed. The DC closely monitors the health and availability of the portal and are likely aware and working on the problem if a user sees this error.

The DC welcomes any feedback users may have on their experiences using the portal. For example, if a user:

  • encountered errors or thinks they may have discovered a bug,

  • were unable to find the dataset they were looking for,

  • the Data Portal is unavailable for an extended period,

  • or thinks of new features that would make the search they wanted to do easier.

We have created a ticketing system to receive valuable user input. Users are encouraged to submit a help desk ticket, such that the DC may investigate the issue. The ticket form is displayed in Figure 66. To submit a help desk ticket, follow the instructions listed below:

  • Enter your name.

  • Enter a summary of the issue you are having.

  • Enter a detailed description of the problem.

    • Be sure to include the filters you are using so that we may identify the datasets and search construction you are having problems with.

  • The ticket form allows you to upload any screen shots that may be useful to diagnosing the issues.

  • Lastly, identify the DC component you are having issues with. In this case it will be Data Center Portal.

Included is a full list of the currently supported spectral lines in the Spectra Line search. For the ViSP instrument, which does not have defined band-passes, the line list will likely be updated as the DC gains experience with the searches users are performing.

Instrument / Wavelengths Reference Names

Wavelengths (nm)

Notes

VBI-Blue

 

Ca II K

393.327

 

G-band

430.520

 

Blue Continuum

450.287

 

H-Beta

486.139

 

VBI-Red

 

H-alpha

656.282

 

Red Continuum

668.423

 

Ti O

705.839

 

Fe XI

789.186

 

ViSP

 

Ca II K

393.366

 

Ca II H

396.847

 

H-delta

410.174

 

H-gamma

434.050

 

Ca I

422.670

 

H-beta

486.139

 

Mg I b1

517.268

 

Mg I b2

518.362

 

He I D3

587.590

 

Na  I D2

588.995

 

Na  I D1

589.592

 

Fe I

630.200

 

H-alpha

656.280

 

Ca II

854.210

 

VTF

 

Na D

589.000

 

Fe I

630.250

 

H-alpha

656.280

 

Ca II

854.210

 

DL-NIRSP

 

Fe XI

789.186

 

Ca II

854.210

 

Fe XIII

1074.700

 

Si I / He I

1083.000

 

Si X

1430.000

 

Fe I

1564.800

 

Cryo-NIRSP

 

Fe XIII

1074.700

 

Fe XIII

1079.700

 

He I

1083.000

 

Si X

1430.000

 

Fe IX

2218.000

 

CO

2326.000

 

Si IX

2580.000

 

Mg VIII

3028.000

 

Si IX

3935.000

 

CO

4651.000

 

Cryo-NIRSP Context

 

Fe XIII

1074.700

 

He I

1083.000

 

J Band

1250.000

 

Si IX

3934.000

 

Si IX Cont.

3955.000

 

CO

4651.000