HWP Supervisor Agent

The HWP supervisor agent monitors and can issue commands to hwp subsystems, and monitors data from other agents on the network that may be relevant to HWP operation. Session data from the supervisor agent’s monitor task can be used to trigger shutdown procedures in the HWP subsystems.

usage: python3 agent.py [-h] [--sleep-time SLEEP_TIME]
                        [--ybco-lakeshore-id YBCO_LAKESHORE_ID]
                        [--ybco-temp-field YBCO_TEMP_FIELD]
                        [--ybco-temp-thresh YBCO_TEMP_THRESH]
                        [--hwp-encoder-id HWP_ENCODER_ID]
                        [--hwp-pmx-id HWP_PMX_ID] [--hwp-pid-id HWP_PID_ID]
                        [--hwp-pcu-id HWP_PCU_ID] [--ups-id UPS_ID]
                        [--ups-minutes-remaining-thresh UPS_MINUTES_REMAINING_THRESH]
                        [--driver-iboot-id DRIVER_IBOOT_ID]
                        [--driver-iboot-outlets DRIVER_IBOOT_OUTLETS [DRIVER_IBOOT_OUTLETS ...]]
                        [--driver-power-cycle-twice]
                        [--driver-power-cycle-wait-time DRIVER_POWER_CYCLE_WAIT_TIME]
                        [--driver-power-agent-type {iboot,synaccess}]
                        [--gripper-iboot-id GRIPPER_IBOOT_ID]
                        [--gripper-iboot-outlets GRIPPER_IBOOT_OUTLETS [GRIPPER_IBOOT_OUTLETS ...]]
                        [--forward-dir {cw,ccw}]

Agent Options

--sleep-time

Default: 2.0

--ybco-lakeshore-id

Instance ID for lakeshore reading out HWP temp

--ybco-temp-field

Field name of lakeshore channel reading out HWP temp

--ybco-temp-thresh

Threshold for HWP temp.

--hwp-encoder-id

Instance id for HWP encoder agent

--hwp-pmx-id

Instance ID for HWP pmx agent

--hwp-pid-id

Instance ID for HWP pid agent

--hwp-pcu-id

Instance ID for HWP PCU agent

--ups-id

Instance ID for UPS agent

--ups-minutes-remaining-thresh

Threshold for UPS minutes remaining before a shutdown is triggered

--driver-iboot-id

Instance ID for IBoot-PDU agent that powers the HWP Driver board

--driver-iboot-outlets

Outlets for driver iboot power

--driver-power-cycle-twice

If set, will power cycle the driver board twice on enable

Default: False

--driver-power-cycle-wait-time

Wait time between power cycles on enable (sec)

Default: 300

--driver-power-agent-type

Possible choices: iboot, synaccess

Type of agent used for controlling the driver power

--gripper-iboot-id

Instance ID for IBoot-PDU agent that powers the gripper controller

--gripper-iboot-outlets

Outlets for gripper iboot power

--forward-dir

Possible choices: cw, ccw

Whether the PID ‘forward’ direction is cw or ccw

Default: “cw”

Configuration File Examples

OCS Site Config

Here is an example of a config block you can add to your ocs site-config file:

{'agent-class': 'HWPSupervisor',
 'instance-id': 'hwp-supervisor',
 'arguments': [
     '--ybco-lakeshore-id', 'cryo-ls240-lsa2619',
     '--ybco-temp-field', 'Channel_7',
     '--ybco-temp-thresh', 75,
     '--hwp-encoder-id', 'hwp-bbb-e1',
     '--hwp-pmx-id', 'hwp-pmx',
     '--hwp-pid-id', 'hwp-pid',
     '--ups-id', 'power-ups-az',
     '--ups-minutes-remaining-thresh', 45,
     '--iboot-id', 'power-iboot-hwp-2',
     '--driver-iboot-id', 'hwp-synaccess-1',
     '--driver-iboot-outlets', 4, 5,
     '--driver-power-agent-type', 'synaccess',
     '--driver-power-cycle-twice',
     '--driver-power-cycle-wait-time', 300,
 ]}

For SATP1, we use an ibootbar to power the driver, and it is not necessary to cycle the driver power twice, so the config will look like:

{'agent-class': 'HWPSupervisor',
 'instance-id': 'hwp-supervisor',
 'arguments': [
     '--ybco-lakeshore-id', 'cryo-ls240-lsa2619',
     '--ybco-temp-field', 'Channel_7',
     '--ybco-temp-thresh', 75,
     '--hwp-encoder-id', 'hwp-bbb-e1',
     '--hwp-pmx-id', 'hwp-pmx',
     '--hwp-pid-id', 'hwp-pid',
     '--ups-id', 'power-ups-az',
     '--ups-minutes-remaining-thresh', 45,
     '--iboot-id', 'power-iboot-hwp-2',
     '--driver-iboot-id', 'power-iboot-hwp-2',
     '--driver-iboot-outlets', 1, 2,
     '--driver-power-agent-type', 'iboot',
 ]}

Note

The --driver-iboot-id and --driver-iboot-outlets arguments are currently used to specify the agent-id and outlets for all power agent types, not just iboot.

Docker Compose

If you want to run this agent in a docker, you can use a configuration like the one below:

ocs-hwp-supervisor:
  image: simonsobs/socs:latest
  hostname: ocs-docker
  environment:
    - INSTANCE_ID=hwp-supervisor
  volumes:
    - ${OCS_CONFIG_DIR}:/config:ro

Description

This agent has two main purposes:

Monitor HWP subsystems and related agents to make high level determinations such as when subsystems should start their shutdown procedure
Serve as a host for high-level HWP operations that require coordinated control of various HWP subsystems, such as “Begin rotating at 2 Hz”

For the first point, the supervisor agent implements the monitor process, which monitors HWP-related processes to compile full state info for the HWP, and uses that to make a determination if the HWP should be shutdown.

For high-level control, the HWP supervisor implements a state machine that is used to perform complex operations with HWP agents that depend on the global state of the HWP and related hardware.

Control States and Actions

In the context of the HWP supervisor state machine, a Control State is a python dataclass that contains data to dictate what the supervisor will do on each update call. For example, while in the WaitForTargetFreq state, the supervisor will do nothing until the HWP frequency is within tolerance of a specified freq for a specified period of time, at which point it will transition into the Done state.

A Control Action is a user-requested operation, in which a starting control state is requested, that then transitions through any number of subsequent states before completing. The action object contains its current state, state history, completion status, and whether it considers itself successful. The action is considered “complete” when it transitions into a “completion state”, which can be Done, Error, Abort, or Idle, at which point no more state transitions will occur. In between update calls, a control action may be aborted by the state-machine, where the action will transition into the completed “Abort” state, and no further action will be taken.

OCS agent operations are generally one-to-one with control actions, where each operation begins a new action, and sleeps until that action is complete. If an operation is started while there is already an action is in progress, the current action will be aborted at the next opportunity and replaced with the new requested action. The abort_action task can be used to abort the current action without beginning a new one.

Examples

Below is an example client script that runs the PID to freq operation, and waits until the target freq has been reached.

supervisor = OCSClient("hwp-supervisor")

result = supervisor.pid_to_freq(target_freq=2.0)
print(result.session['data']['action'])

>> {'action_id': 6,
    'completed': True,
    'cur_state': {'class': 'Done', 'msg': None, 'success': True},
    'state_history': [{'class': 'PIDToFreq',
                        'direction': '0',
                        'freq_tol': 0.05,
                        'freq_tol_duration': 10.0,
                        'target_freq': 2.0},
                      {'class': 'WaitForTargetFreq',
                        'freq_tol': 0.05,
                        'freq_tol_duration': 10.0,
                        'freq_within_tol_start': 1706829677.7613404,
                        'target_freq': 2.0},
                  {'class': 'Done', 'msg': None, 'success': True}],
      'success': True}

Below is an example of a client script that starts to PID the HWP to 2 Hz, then aborts the PID action, and shuts off the PMX power supply. Note that the abort_action here is technically redundant, since starting the new action would abort the active action in the same manner.

supervisor = OCSClient("hwp-supervisor")

supervisor.pid_to_freq.start(target_freq=2.0)
supervisor.abort_action()
res1 = supervisor.pid_to_freq.wait()
res2 = supervisor.pmx_off()

print("Result 1:")
print(res1.session['data']['action'])
print("Result 2: ")
print(res2.session['data']['action'])

>>
Result 1:
{'action_id': 1,
'completed': True,
'cur_state': {'class': 'Abort'},
'state_history': [{'class': 'PIDToFreq',
                    'direction': '0',
                    'freq_tol': 0.05,
                    'freq_tol_duration': 10.0,
                    'target_freq': 2.0},
                  {'class': 'Abort'}],
'success': False}
Result 2:
{'action_id': 3,
'completed': True,
'cur_state': {'class': 'Done', 'msg': None, 'success': True},
'state_history': [{'class': 'PmxOff', 'success': True},
                  {'class': 'Done', 'msg': None, 'success': True}],
'success': True}

Agent API

class socs.agents.hwp_supervisor.agent.ControlAction(state)[source]

This is a class to contain data regarding a single HWP control action. This groups together states that are part of a single action, and whether the action is completed and successful.

set_state(state)[source]: Sets state for the current action. If this is a completed_state, will mark as complete.

encode()[source]: Encodes this as a dict

sleep_until_complete(session=None, dt=1)[source]

Sleeps until the action is complete.

Parameters:

session (OpSession, optional) – If specified, this will set session.data[‘action’] to the encoded action on each iteration to keep session data up to date
dt (float, optional) – Time to sleep between iterations.

class socs.agents.hwp_supervisor.agent.HWPSupervisor(agent, args)[source]

The HWPSupervisor agent is responsible for monitoring HWP and related components, and high-level control of the HWP. This maintains an updated HWPState containing state info pertaining to HWP agents. Additionally, it contains a state-machine for controlling the HWP based on the ControlState.

agent

OCS agent instance

Type:: OCSAgent

args

Argument namespace

Type:: argparse.Namespace

hwp_state

Class containing most recent state information for the HWP

Type:: HWPState

control_state

Current control_state object. This will be used to determine what commands to issue to HWP agents.

Type:: ControlState

forward_is_cw

True if the PID “forward” direction is clockwise, False if CCW.

Type:: bool

monitor()[source]

Process – Monitors various HWP related HK systems.

This operation has three main steps:

Query session data for all HWP and HWP adjacent agents. Session info for each queried operation will be stored in the monitored_sessions field of the session data. See the docs for the get_op_data function for information on what info will be saved.
Parse session-data from monitored operations to create the state dict, containing info such as ybco_temp and hwp_freq.
Determine subsystem actions based on the HWP state, which will be stored in the actions dict which can be read by hwp subsystems to initiate a shutdown

An example of the session data, along with possible options for status strings can be seen below:

>>> response.session['data']

    {'timestamp': 1601924482.722671,
    'monitored_sessions': {
        'encoder': {
            'agent_id': 'test',
            'data': <session data for test.acq>,
            'op_name': 'acq',
            'status': 'ok',  # See ``get_op_data`` docstring for choices
            'timestamp': 1680273288.6200094},
        },
        'rotation': {see above},
        'temperature': {see above},
        'ups': {see above}
        'iboot': {see above}},
    # State data parsed from monitored sessions
    'state': {
        'hwp_freq': None,
        'ybco_temp': 20.0,
        'ybco_temp_status': 'ok',  # `no_data`, `ok`, or `over`
        'ybco_temp_thresh': 75.0,
        'ups_battery_current': 0,
        'ups_battery_voltage': 136,
        'ups_estimated_minutes_remaining': 50,
        'ups_minutes_remaining_thresh': 45.0,
        'ups_output_source': 'normal'  # See UPS agent docs for choices
    },
     # Subsystem action recommendations determined from state data
    'actions': {
        'pmx': 'ok'  # 'ok', 'stop', or 'no_data'
        'gripper': 'ok'  # 'ok', 'stop', or 'no_data'
    }}

spin_control()[source]

Process - Process to manage the spin-state for HWP agents. This will issue commands to various HWP agents depending on the current control state.

Parameters:: test_mode (bool) – If True, spin_control loop will run a single update iteration before exiting. This is useful for testing actions.

Notes

Example of session.data:

>>> session['data']
{
    'current_action': <Encoded action>,
    'action_history': List[Encoded action]
    'timestammp': <time.time()>
}

pid_to_freq(target_freq=2.0, freq_thresh=0.05, freq_thresh_duration=10)[source]

Task - Sets the control state to PID the HWP to the given target_freq.

Parameters:

target_freq (float) – Target frequency of the HWP (Hz). This is aa signed float where positive values correspond to counter-clockwise motion, as seen when looking at the cryostat from the sky.
freq_thresh (float) – Frequency threshold (Hz) for determining when the HWP is at the target frequency.
freq_thresh_duration (float) – Duration (seconds) for which the HWP must be within freq_thresh of the target_freq to be considered successful.

Notes

Example of session.data:

>>> session['data']
{'action':
    {'action_id': 3,
    'completed': True,
    'cur_state': {'class': 'Done', 'msg': None, 'success': True},
    'state_history': List[ConrolState],
    'success': True}
}

set_const_voltage(voltage=1.0)[source]

Task - Sets the control state set the PMX to a constant voltage.

Parameters:

voltage (float) – Voltage to set the PMX to (V).
direction (str) – Direction of the HWP. Must be one of cw or ccw, corresponding to the clockwise and counter-clockwise directions of the HWP, as seen when looking at the cryostat from the sky.

Notes

Example of session.data:

>>> session['data']
{'action':
    {'action_id': 3,
    'completed': True,
    'cur_state': {'class': 'Done', 'msg': None, 'success': True},
    'state_history': List[ConrolState],
    'success': True}
}

brake(freq_thresh=0.05, freq_thresh_duration=10, brake_voltage=10)[source]

Task - Sets the control state to brake the HWP.

Parameters:

freq_thresh (float) – Frequency threshold (Hz) for determining when the HWP is at the target frequency.
freq_thresh_duration (float) – Duration (seconds) for which the HWP must be within freq_thresh of the target_freq to be considered successful.
brake_voltage (float) – Voltage to use when braking the HWP.

Notes

Example of session.data:

>>> session['data']
{'action':
    {'action_id': 3,
    'completed': True,
    'cur_state': {'class': 'Done', 'msg': None, 'success': True},
    'state_history': List[ConrolState],
    'success': True}
}

pmx_off()[source]

Task - Sets the control state to turn off the PMX.

Notes

Example of session.data:

>>> session['data']
{'action':
    {'action_id': 3,
    'completed': True,
    'cur_state': {'class': 'Done', 'msg': None, 'success': True},
    'state_history': List[ConrolState],
    'success': True}
}

abort_action()[source]

Task - Aborts the current action, setting the control state to Idle

Notes

Example of session.data:

>>> session['data']
{'action':
    {'action_id': 3,
    'completed': True,
    'cur_state': {'class': 'Idle'},
    'state_history': List[ConrolState],
    'success': False}
}

enable_driver_board()[source]

Task - Enables the HWP driver board

Notes

Example of session.data:

>>> session['data']
{'action':
    {'action_id': 3,
    'completed': True,
    'cur_state': {'class': 'Done', 'msg': None, 'success': True},
    'state_history': List[ConrolState],
    'success': False}
}

disable_driver_board()[source]

Task - Disables the HWP driver board

Notes

Example of session.data:

>>> session['data']
{'action':
    {'action_id': 3,
    'completed': True,
    'cur_state': {'class': 'Done', 'msg': None, 'success': True},
    'state_history': List[ConrolState],
    'success': False}
}

class socs.agents.hwp_supervisor.agent.get_op_data(agent_id, op_name, log=None, test_mode=False)[source]

Process data from an agent operation, and formats it for the monitor operation session data.

Parameters:

agent_id (str) – Instance ID of the agent
op_name (str) – Operation from which to grab session data
log (logger, optional) – Log object
test_mode (bool) – If True, this will run in test mode, and not try to connect to the specified agent.

Returns:

Returns a dictionary with the following fields
agent_id (str) – Instance id for the agent being queried
op_name (str) – Operation name being queried
timestamp (float) – Time the operation status was queried
data (dict) – Session data of the operation. This will be None if we can’t connect to the operation.
status (str) – Connection status of the operation. This can be the following:
- no_agent_provided: if the passed agent_id is None
- test_mode: if test_mode=True, this will be the status and no operation will be queried.
- op_not_found: This means the operation could not be found, likely meaning the agent isn’t running
- no_active_session: This means the operation specified exists but
  was never run.
- ok: Operation and session.data exist