Key Concept: Audit Sample Size Determination

Source information can be found here: https://www.statisticshowto.datasciencecentral.com/probability-and-statistics/find-sample-size/

"""
Calculate a sample size given population and desired confidence
"""
from scipy.stats import norm


def calculate_sample_size(population: int, confidence_level: float, confidence_interval) -> int:
    """
    Calculate the sample size using Cochran's sample size formula.
    https://www.statisticshowto.datasciencecentral.com/probability-and-statistics/find-sample-size/

    :param population: Integer representing the size of the population (e.g. 1000)

    :param confidence_level: Float representing the decimal percentage confidence in the sample
    result being representative of the population (e.g. .95 for 95% confidence)

    :param confidence_interval: Float representing the decimal percentage error acceptable on the
    result (e.g. .05 for +- 5%)

    :return: Integer representing the number of samples required to achieve the confidence level / interval (e.g. 278)

    Sample Size
    ss =	Z^2 * (p) * (1-p)
            -----------------
                   c^2
    Where:

    Z = Z value (e.g. 1.96 for 95% confidence level)
    p = percentage picking a choice, expressed as decimal
    (.5 used for sample size needed)
    c = confidence interval, expressed as decimal
    (e.g., .04 = ±4)


    Correction for Finite Population
                   ss
                --------
    new ss =
                1+	ss-1
                --------
                  pop


    Where: pop = population

    Example:
    >>>calculate_sample_size_v2(1000, .95, .05)
    278

	Example Answer Explained:
    If you sampled 278 elements of the 1000 element population and found 10% loss then you would be 95%
    confident that the population loss rate was between 5% and 15%
    """

    # TODO type check
    # TODO range check

    p = .5  # Worst case (i.e. largest sample size needed) probability where half the files are missing
    confidence_level += (1 - confidence_level)/2  # Update confidence level to include the entire left tail
    Z = norm.ppf(confidence_level)  # Determine Z score based upon confidence level

    ss = Z**2 * p * (1-p) / confidence_interval**2  # Determine sample size for infinite population

    sample_size = ss/(1 + (ss-1) / population) # Correct for finite population

    return int(sample_size)

User Registration

The User Registration process is responsible for the generation of a user account in Globus. A Globus user account is required in order to leverage the advanced transfer functionality within Globus.

User Authorization Management

The User Authorization Management process is primarily responsible for updates to the list of DKIST Authorized Agents. DKIST Authorized Agents are people who were authorized by the DKIST director to be exempt from embargo restrictions for the purpose of instrument analyses.

Tickets that have been approved relating to authorization changes to embargoed proposal group membership are also executed here.

Data Holding Audit

The Data Holding Audit process is responsible for the floor to book, book to floor, and object integrity verifications that occur on an ongoing basis.

Service Desk Management

The Service Desk Management process is responsible for managing the work tickets in the ticketing system. There are multiple types of tickets which get routed according to their type. In some cases (e.g., a Raw Data search), other processes must be executed in order to resolve a ticket. This process is intended as a guideline for the capabilities a Data Center component of a larger DKIST-wide Help Desk would need to perform, and as such, would be just a part of the DKIST-wide Service Desk Management process.

Raw Data Search

The Raw Data Search process is responsible for the fulfillment of two types of Raw Data searches. The first are those requested by users outside of DKIST operations and require prior approval. Should the external Raw Data search request be approved, the requested data is collated and prepared for distribution by support personnel. In the second case (i.e., users within DKIST Operations), an API is provided for the discovery of Raw Data via Observing Program ID. The API method puts load on systems critical to data processing so is limited in search criteria.

DQAC Reduction

The DQAC Reduction process is responsible for the execution of data reduction activities within the Data Center. The management of the request that surrounds the execution is handled by the Service Desk Management process. Removal of data requires the maintenance of any associated inventory information.

Processed Data Availability Notification

The Processed Data Availability Notification system is responsible for notifying Investigators when data for their Proposal has been processed. The notifications are delivered in a digest form. These processing events can occur over a long timescale, as reprocessing events would also trigger notification.

Monitor System Health

The Monitor System Health process is responsible for providing the capabilities to inspect the run-time operations of the Data Center. This includes logs, system telemetry, and Data Center custom events. This information degrades in utility over time so it is not retained forever. The culling process allows for the retention of aggregations of select events though to provide monitoring information over timescales longer than the retained history.

Code and Algorithm Documentation Distribution

The Code and Algorithm Documentation Distribution process is responsible for displaying for search queries the algorithm documentation and user libraries created by the Data Center.

Poison Message Management

The Poison Message Management process is responsible for the review and corrective action associated with un-processable messages that were on the Interservice Bus.