Skip to end of banner
Go to start of banner

How long should it take to download DKIST data?

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Next »

Summary

The following article should be used when you have issues with the DKIST data download portion itself which is using the Globus interface to perform the data transfer, since this action can be very time consuming. If you’re looking for the Data Center (DC) User Portal not functioning, please go here.

Instructions

Datasets are generally large, and data transfers, even with a good connection, can take anywhere from a few hours up to multiple weeks. Long download times and slow transfer speeds are more often than not (~98% of the time) on the user’s side.

Globus data transfer speeds are limited by the following factors:

  1. Transmitting Endpoint bandwidth - the TOTAL bandwidth is 10 Gb/sec from the DKIST Data Center. A user should not expect to be able to pull data at this rate, as it is for all users.

  2. Receiving Endpoint bandwidth (the location the data lands, which is subject to the network speeds at the location of the receiving endpoint). This may be completely out of the control of the user.

  3. Receiving Endpoint storage Write speed.

Sample Download Times

Data Size

Transmitting Endpoint bandwidth

Receiving Endpoint bandwidth

Receiving Endpoint storage Write speed

If the transfer speed seems slow to you, please verify the download capability of the receiving endpoint as there is nothing the DKIST Data Center Archive can do to speed up the transmission. You can contact the cluster administrator about read and write speeds, or utilize a website such as Speedtest to check download rates of your machine.

For more information about how Globus transfers work, please see Globus' https://docs.globus.org/faq/transfer-sharing/. You can also run a data download calculator test using this service: https://downloadtimecalculator.com.

In the download time calculator, you are able to set both your download speed, and the size of what you are trying to download in order to receive a calculated estimate on the download time. If you do not know what your download speed is, there a link on the page, reading “I don’t know my internet speed” which will take you to automatically take you to the speedtest.net.

Slow downloads are usually network bound (although sometimes host bound). Please refer to the ESNet pages: https://fasterdata.es.net/ – specifically the host tuning and network tuning tabs at the top.

Slow downloads and slow transfer speeds are more often than not (~98% of the time) on the user’s side. Please check your internet connectivity, the Globus configuration and contact the Globus Help Desk (support@globus.org) for assistance before filing a DKIST Help Desk ticket.

Filter by label

There are no items with the selected labels at this time.

  • No labels