Summary
The DKIST Data Center Archive (DC) provides calibrated (a.k.a. Level 1, “L1”) data to the community via the User Portal for download. It does not generally provide general access to raw (a.k.a. Level 0, “L0”) data. However, special requests can be made using a request form to the DC for raw data files for certain instruments. If the user has a justified reason to get these raw data, they can make a request to the DC for access. Requests will be reviewed on a case by case basis. It is assumed that a user will already have downloaded the L1 data before requesting the raw data. You are required to specify the dataset IDs from the L1 data that corresponds to the raw data you wish to download. Raw data is not searchable and cannot be filtered by instrument or other similar parameters. If the request is approved, the user will receive ALL the raw data for at least one observing program as executed at the telescope.
| Info |
|---|
Note that if you are a PI who has been contacted by the data center regarding issues calibrating data from your proposal, do not use this form. Please respond to the DC using the email you were contacted by. |
| Note |
|---|
Warning: Raw data requests are for experts only. The dataset sizes are massive and working with these data requires a large amount of storage and processing power. |
| Note |
|---|
| Info |
Nearly all VBI data is speckle reconstructed at the telescope and this data is then downloaded transferred to the data center. In such cases raw unreconstructed VBI data will not be available. If you requested unreconstructed data in your proposal, then this data will still have gain and dark corrections applied. If you are going to need unreconstructed VBI data, you must request it in your proposal. |
| Info |
|---|
Submitting a raw data request does NOT guarantee that the request will be granted. Requests will be vetted on a case by case basis. Note that proprietary data can only be released to the PI and proposal team members. Requests will be reviewed by a DKIST committee that will vet the request based on:
|
...
The user must have an active Globus account (see: How to create a globus account (for DKIST data))
The user must have an active endpoint for Globus data transfers (see: Globus Connect & Endpoints)
The user should already have downloaded and looked at the Level 1 datasets
...
Visit the DKIST Data Portal Raw Data request form (link available soon)
Log into the Data Portal
Fill in the following mandatory fields on the request form; please be as descriptive as possible:
Datasets Requested: Dataset ID(s) where the FITS header keyword = “DSETID” in the L1 data
Justification: Statement for the reason for this raw data request
...
When the RAW data files are delivered to the data center, they are named like
atst.ics.vbiBlue.dc.vcc.xfer.73449.48574-1.fits.fz.
The the only useful bit of information that the raw data filename contains is the instrument name. The data center subsequently renames the files (e.g. 0056d294667a4bb5a552bfbda08a4dd3.fits) to ensure that if we happened to get the same filename twice from the summit there are no issues handling multiple copies of the same file. As the data center will not generally release raw data, our overriding goal is just to create unique filenames. When the raw data is ingested at the data center, meta-information about the file goes into frame inventory, so the data center subsequently knows what the file is and what it contains. However, the data center doesn’t have mechanism to expose frame inventory publicly. The only way for a user to extract the information you require, is to read the file headers and parse them. The L1 files are more usefully named, e.g.
...