Introduction

The Global Interlock system protects the personnel and equipment of our observatory from death, serious injury or damage. Access to the controller is restricted; only certain trained and authorized personnel can make changes that may affect system safety. Personnel policies will be in place and enforced with regard to such changes.

Intended Audience

There are two groups of people that will use different sections of this manual—operators and maintenance personnel. For the purposes of this manual, “operator” is used to describe personnel qualified to operate systems and subsystems of the facility not necessarily a job title. “Maintenance personnel” refers to technical staff qualified to diagnose and repair equipment malfunctions.

Only personnel who have been specifically trained and authorized shall perform the procedures detailed in this manual.

Related Documents

Reference Documents

Many of the components of the GIS are covered by manufacturer’s user manuals.

Safety Symbols

Standardized safety alerts are used to denote procedures or activities that are potentially hazardous.

Glossary

See SPEC-0012 for terms and abbreviations not listed below.

Human Machine Interface (HMI)

Terms used on the HMI

It is necessary to define terms that are used with the operator interface.

Levels of authorization

2.1.3       Sample Screens

The representative basic operator troubleshooting screen is Figure 1. This screen shows that an emergency stop has been activated and has tripped the system. The system cannot be reset at this time as the interlock is still active.

Figure 1 HMI showing interlock tripped

If more information is required, such as the location of the emergency stop device that has been activated further troubleshooting screens trace the fault further back. Figure 2 shows that the emergency stop push button located at the Left Nasmyth Platform has been activated.

Figure 2 HMI Emergency Stop Screen

After the emergency stop push button has been deactivated, the interlock condition has been removed. A reset is now required to restart the system. Figure 3 shows that the interlock has been latched, the interlock is no longer active and that the GIS is now ready to be reset.

Figure 3 HMI showing reset required

Figure 4 shows the system has been restored to its normal operating condition.

Figure 4 HMI showing normal operation

Telescope Interlock Manager (TIM)

Main Screen

The main screen allows the selection of system information, subsystem information or alarm history. There is also a button for shutting down the HMI, for example to reload or reset the HMI itself.

Global Interlock Controller Screen

The Global Interlock Controller screen provides an overview of the GIC status and subsystem status. The information provided for each controller is the current program name, status of controller communications with HMI, the controller status itself, keyswitch mode, safety status, safety signature, and CIP synchronization status.

2.1.4.2.1      Program Name

2.1.4.2.2      Controller Communications

This represents whether the device is present on the network as detected by the HMI. If the device is not present, ensure that power is on to the controller and the network is connected.

2.1.4.2.3      Controller Status

OK       The controller has detected no faults.

2.1.4.2.4      Keyswitch mode

RUN    The keyswitch is in the run position. The PLC will run normally but cannot be programed or changed remotely.

PROG  The keyswitch is in the program position. The PLC is not running.

REM RUN        The keyswitch is in the remote position. And is in the run mode.

REM PROG      The keyswitch is in the remote position. And is in the program mode. The PLC is not running. Connect using RSLogix 5000 and change mode to Run.

2.2       Observatory Control System

The observatory control system has access to the status of the various interlocks employed by the GIS. It is intended to allow the operator to determine which interlock has tripped and is preventing the operation of the facility. It is not intended to replicate all information available at the GIS HMI or via a development computer on the safety network. The HMI can provide detailed troubleshooting information in the event of a fault.

2.3       Control Room Stack Light

A wall-mounted signal light is located in the control room. The signal lights are designed to communicate the status of various conditions within the safety system. It consists of five colored lamps (red, amber, green, blue, clear). It also contains a piezo sounder (70 to 90 dbA).

Multiple lamps could be lit depending on the condition of the safety system. The meaning of the lamps are listed in Table 2‑1.

Table 2‑1 Functions Indicated by Stack Lights

Cold Start up

When power is first applied or re-applied to the system, it will likely cause the system to generate a large number of faults. This is mainly due to the fact that the EtherNet switches of the safety system take substantially longer to boot than the PLCs themselves.

If the safety system was working properly prior to the loss of power and the reason for the loss of power is understood, the quickest way to restore the safety system to normal operations is to cycle the individual LICs power or perform a RUN-PROG-RUN reset by turning the keyswitch from ‘run’ to ‘program’ and back to ‘run’ (This can also be done remotely if the keyswitch is in the ‘remote’ mode).

Normal Operations

During normal operations, the GIS does not require any operator interaction. Operator intervention is required only when the GIS enters a tripped or faulted state.

Logging

The GIS logs various events for troubleshooting purposes. Events to be logged include trips and faults in the system. The logs are timestamped in UTC synchronized to the observatory-wide TRADS system. The logs are kept on the a FactoryTalk Historian module located in the GIC rack. This module can be remotely accessed. The HMI also stores a number of alarms.

Resetting an tripped Interlock

In most cases, trips of the GIS can be reset by using the HMI (Human Machine Interface) to access password-protected screens. Certain trips may be more serious than others and may require higher authorization before proceeding.

Procedures require that the reason for all trips be investigated and understood before proceeding with a reset. The reset does not cause hazardous motion to resume; rather it allows the subsystem that tripped to be restarted.

Hazardous Area Access

Hazardous areas may be entered for various routine reasons. Personnel doing so must ensure that all personnel have left the hazardous area before securing it.

Hazardous areas may also be entered during an emergency. For example, the doors leading to the external ladders may be used to evacuate the building. By their vary nature they cannot be locked with trapped key or must be equipped with an override device.

These two types of tripped interlocks are very different. Normal access using a trapped key may be reset by personnel returning the trapped key in the proper sequence. In the second non-routine case, it would be necessary to inspect the hazardous area and ensure no personnel or other hazards are present prior to attempting to reset this type of tripped interlock.

Trapped Keys

Trapped keys prevent unauthorized entry into hazardous areas and prevents unexpected start-up of equipment when persons are in hazardous areas.

Only one key exists for each device. This is by design and essential to maintaining a safe system. Trapped keys must never leave the facility.

Muting an Interlock

Periodically it may be required to mute an interlock. Interlock designed to be muted will be designated after a Job Hazard Analysis identifies a routine task that requires an interlock to be ignored.

An example of interlock to be muted would be the manlift stowed interlock. During operations it is foreseen that the manlift may be removed from the observing chamber. Rather than defeating the interlock mechanically or electrically the HMI will allow for a time-limited password-protected bypass to programmatically ignore the interlock.

The purpose of this is to prevent a situation where a jumper is left in place to defeat an interlock. Even if the operator does not manually restore the muted interlock the controller will programmatically restore the interlock after a designated timeout period.

Another example of muting an interlock occurs when the Enclosure Azimuth axis is required to used (albeit at a safely-limited speed) when the Bridge Crane is in use. In this case, the connection of the Enclosure Pendant mutes the interlock when it is connected.

Emergency Operations

Certain operations may be required and interlocks may need to be bypassed to allow emergency operations, such as freeing personnel entrapped in a pinch/crush hazard. [TRW2] 

Unlocking Guardlocking Switches

The hazard access guard locking switches are designed to require power to unlock them. In the event of a power failure all guards will lock in place. If personnel are in the hazardous access area they may exit by using the emergency release mechanism installed on the inside of each door or use a monitored only exit door.

If no personnel are inside the hazardous area but entry is required, then entry must be gained through a monitored only exit door. This requires a building key. Once inside the guard locked door may be unlocked using the emergency release mechanism.

Connecting a Computer

Specific steps must be taken to ensure that any computer that is connected to the GIS does not pose a potential hazard to the system. The network configuration of the GIS is to allow only a limited set MAC addresses to be connected. Two maintenance computers will be configured for use on the GIS network. They will have the OS and application programs updated regularly.

Up-to-date anti-virus software must be installed. The computer will also be scanned for viruses prior to connecting to the GIS network. No removable media is to be used without being scanned prior to insertion in the maintenance computers.

Patch Management

From time to time the manufacturer will likely release updates to the firmware used in the various components of the GIS. It is not the intention to upgrade firmware with each new release that is made available. Firmware will only be upgraded when the update is required due to a safety or security concern with the existing firmware or additional capabilities are required for changes to the GIS.

Prior to deploying any software patch, whether it is to the firmware of any component or to the operating system of a host computer, the patch will be qualified by Rockwell Automation for compatibility. After a firmware update the GIS will require a function test to verify proper operation before being returned to service.

Updates to Firmware

Updates to Operating System

Updates to Programming Environment 

By-passing an Interlock

In the event of an interlock requiring a bypass due to some unforeseen circumstance, alternate means must be taken to ensure the safety of personnel and the facility.

To ensure that while a bypass is in place that it can be easily tracked; interlocks will not be bypassed by using electronic or mechanical means. Rather, the appropriate tags will be forced in the controller. By forcing the tags in the controller it will be easier to view which interlock(s) have been bypassed and the controller itself will indicate that I/O is in the forced condition.

Since access to the controller is restricted, only certain trained and authorized personnel can make changes that may affect system safety. Personnel policies will be in place and enforced with regard to such changes.

To force data the software project will have to be safety-unlocked and the safety task signature deleted.

Because this results from an unforeseen circumstance a specific Job Hazard Analysis must be performed and approved before bypassing an interlock by forcing controller I/O.

Periodic Maintenance

Battery Replacement

The only regular maintenance required for the L6-series of GuardLogix Controllers is periodic replacement of the battery that backups volatile memory. The battery should be replaced at least every 3 years. The L7 GuardLogix Controllers have replaced the Lithium battery with a capacitor-based energy storage module that does not require period replacement.

Functional Operation Testing

As it is possible that some safety devices (such as e-stop switches) may not be actuated for extended periods of time, the various components of the GIS, while highly reliable, will be examined and functionally tested on an annual basis to reduce the probability that an undetected fault exists that could lead to a hazardous condition.

Repair

The GIS is designed to be maintained and repaired with standard hand tools (Appendix A) and standard test equipment (Appendix B).

No special tools are needed to install, service, maintain, or operate the GIS.

7.1       Replacing a Failed Component

Because of the various security measures in place to prevent changes to the system, specific steps must be taken prior to replacing a failed component. The replacement module will have to be configured with the IP address and Safety Network Number off-line prior to installation. Firmware revision level will be set to the same revision level of the failed component.

After replacement of a failed component the safety functions that are affected by that component will require a functional test see section 7.

8        Testing

There are three basic types of testing, functional testing, verification, and validation. Functional testing is actuating an interlock device (such as a limit switch) and observing that the safety system sees normal operation of the device itself. Verification is similar to a functional test but this test the entire safety-related function by observing that activating the device does in fact cause the subsystem under test to enter a safe state. Validation is testing a safety-related control function by injecting a fault and observing that the subsystem under test enters a safe state.

8.1       Functional Testing

When a fault occurs with the GIS, the system will default to a safe state, to return to normal operations a functional test may be required. Functional tests are required when the system detects a hardware fault such as a two channel discrepancy.

A functional test is performed by causing the unit under test to change from the active state to the non-active state and back again. For example if a hardware fault requires a functional test of an emergency stop push button, the button has to be depressed then pulled back out to ensure that it is functioning properly.

This testing is done without the actually enabling the function.

8.2       Verification and Validation

Verification and validation are required by IEC 62061 and must follow the requirements of that standard.

Verification of the GIS shall include design qualification, installation qualification, operational qualification, and performance qualification.

Design qualification shall consist of a review of the design by qualified reviewers at the final design review.

Installation qualification shall consist of a thorough inspection and test of each circuit to ensure that the installation meets quality standards and is correct according to the schematic drawings and documentation. Any changes found necessary during installation shall be red-lined and corrections to the final document set shall be made.

Operational qualification includes the testing of each circuit for correct response including testing for shorts across circuits, shorts to ground, and shorts to the power bus. For each of these tests, the verification is that the response is to fail to the safe state. These tests shall be documented. These tests shall be repeated for all the subsystems that could be affected whenever a hardware or wiring change is made.

Performance qualification shall be performed on the finished installed system and on the installed system whenever a significant addition or change is made (such as adding a new hardware module or updating firmware or programming). Performance qualification includes testing against specifications for response speed and error-free packet delivery.

8.2.1       Verification

Verification is checking to make sure the function works as intended. This is a simple functional test of the system.

An example would be adding an emergency stop switch. Verification would be pressing the button and checking to see if all systems trip to a safe state.

8.2.2       Validation

Validation is checking to make sure the function works in the event of a failure. In the case of the GIS, this refers to a single failure. The system is designed so that a single failure will be detected before or when a demand is placed on the function. Multiple failures are not considered.

An example of validation would be adding an emergency switch would be shorting each of the two inputs to ground, shorting both inputs together, shorting both inputs to 24V_DC and checking to see if all systems fault to a safe state.

9        rEFERENCE

9.1       GIC OPERATOR SCREENS

[1] IEC 61508-4 3.6.1

[TRW1]http://manualise.com/en/blog/law-and-legislation/ansi-z535-6/

[TRW2]GISFRR-22

[t3]GISFRR-112