How long should it take to download DKIST data?
- Anastasia Alexov (Unlicensed)
Summary
The following article should be used when you have issues with the DKIST data download portion itself which is using the Globus interface to perform the data transfer, since this action can be very time consuming. If you’re looking for the Data Center (DC) User Portal not functioning, please go here.
Instructions
Datasets are generally large, and data transfers, even with a good connection, can take anywhere from a few hours up to multiple weeks. Long download times and slow transfer speeds are more often than not (~98% of the time) on the user’s side.
Globus data transfer speeds are limited by the following factors:
Transmitting Endpoint bandwidth - the TOTAL bandwidth is 10 Gb/sec from the DKIST Data Center. A user should not expect to be able to pull data at this rate, as it is for all users.
Receiving Endpoint bandwidth (the location the data lands, which is subject to the network speeds at the location of the receiving endpoint). This may be completely out of the control of the user.
Sample Download Times
Data Size | Receiving Endpoint bandwidth(A) | Approximate Download time |
|---|---|---|
250 GB | 250 Mb/s (See note 1.) | 16m 40s |
500 GB | 250 Mb/s | 33m 20s |
1 TB | 250 Mb/s | 66m 40s |
5 TB | 250 Mb/s | 5h 33m |
10 TB | 250 Mb/s | 11h 6m |
25 TB | 250 Mb/s | 27h 47m |
250 GB | 1 Gb/s (See note 2.) | 4m 10s |
500 GB | 1 Gb/s | 8m 20s |
1 TB | 1 Gb/s | 16m 40s |
5 TB | 1 Gb/s | 1h 23m |
10 TB | 1 Gb/s | 2h 46m |
25 TB | 1 Gb/s | 6h 55m |
250 GB | 10 Gb/s (See note 3.) (See note 4.) | 25s |
500 GB | 10 Gb/s | 50s |
1 TB | 10 Gb/s | 1m 40s |
5 TB | 10 Gb/s | 8m 20s |
10 TB | 10 Gb/s | 16m 40s |
25 TB | 10 Gb/s | 41m 40s |
(A) This table assumes you can get 100% of the receiving endpoint bandwidth. This will never happen! Therefore the times listed in the table will be underestimates and are included only as a rough guide to the user.
Typical download speed to a machine on a home internet.
Typical download speed to a machine on an office internet.
Possible download resource to an institution provided high speed end point or HPC center.
Assumes that for the sake of argument you could get every bit of the 10 Gb/s out of the DKIST DC.
If the transfer speed seems slow to you, please verify the download capability of the receiving endpoint as there is nothing the DKIST Data Center Archive can do to speed up the transmission. You can contact the cluster administrator about read and write speeds, or utilize a website such as Speedtest to check download rates of your machine.
For more information about how Globus transfers work, please see Globus' https://docs.globus.org/faq/transfer-sharing/. You can also run a data download calculator test using this service: https://downloadtimecalculator.com.
In the download time calculator, you are able to set both your download speed, and the size of what you are trying to download in order to receive a calculated estimate on the download time. If you do not know what your download speed is, there a link on the page, reading “I don’t know my internet speed” which will take you to automatically take you to the speedtest.net.
Slow downloads are usually network bound (although sometimes host bound). Please refer to the ESNet pages: https://fasterdata.es.net/ – specifically the host tuning and network tuning tabs at the top.
Slow downloads and slow transfer speeds are more often than not (~98% of the time) on the user’s side. Please check your internet connectivity, the Globus configuration and contact the Globus Help Desk (support@globus.org) for assistance before filing a DKIST Help Desk ticket.