Pysmurf Controller

The Pysmurf Controller OCS agent provides an interface to run pysmurf and sodetlib control scripts on the smurf-server through an OCS client.

usage: python3 agent.py [-h] [--monitor-id MONITOR_ID] [--slot SLOT]
                        [--poll-interval POLL_INTERVAL]

Agent Options

--monitor-id, -m

Instance id for pysmurf-monitor corresponding to this pysmurf instance.

--slot

Smurf slot that this agent will be controlling

--poll-interval

Time between check-state polls

Dependencies

The pysmurf controller requires the following packages:

These can be installed via pip:

$ python -m pip install 'pysmurf @ git+https://github.com/slaclab/pysmurf.git@main'
$ python -m pip install 'sodetlib @ git+https://github.com/simonsobs/sodetlib.git@master'
$ python -m pip install 'sotodlib @ git+https://github.com/simonsobs/sotodlib.git@master'

Additionally, socs should be installed with the pysmurf group:

$ pip install -U socs[pysmurf]

Configuration File Examples

For a detailed walkthrough of how to set up the smurf dockers, see the smurf-software-setup SO wiki page.

OCS Site Config

Example site-config entry:

{'agent-class': 'PysmurfController',
 'instance-id': 'pysmurf-controller-s2',
 'arguments': [['--monitor-id', 'pysmurf-monitor']]},

Docker Compose

The pysmurf-controller docker is built on the pysmurf base image instead of the standard socs image. The pysmurf base docker does not always include the most recent version of pysmurf, so it is common to mount a local dev branch of the repo into the docker container to /usr/local/src/pysmurf. Similarly, we mount in a development copy of SODETLIB so that we have the most recent version without having to rebuild this container.

The docker-compose for the pysmurf-controller that publishes to a container named ocs-pysmurf-monitor might look something like:

ocs-pysmurf:
    image: simonsobs/ocs-pysmurf-agent:${SOCS_TAG}
    user: cryo:smurf
    network_mode:  host
    container_name: ocs-pysmurf-controller
    security_opt:
        - "aparmor=docker-smurf"
    environment:
        - INSTANCE_ID=pysmurf-controller-s2
        - SITE_HUB=ws://${CB_HOST}:8001/ws
        - SITE_HTTP=ws://${CB_HOST}:8001/call
        - SMURFPUB_BACKEND=udp
        - SMURFPUB_ID=crate1slot2
        - SMURFPUB_UDP_HOST=ocs-pysmurf-monitor
        - DISPLAY
        - OCS_CONFIG_DIR=/config
        - EPICS_CA_ADDR_LIST=127.255.255.255
        - EPICS_CA_MAX_ARRAY_BYTES=80000000
        - SLOT=2
    volumes:
        - ${OCS_CONFIG_DIR}:/config
        - /data:/data
        - /home/cryo/repos/pysmurf/client:/usr/local/src/pysmurf/python/pysmurf/client
        - /home/cryo/repos/sodetlib:/sodetlib

where CB_HOST and SOCS_TAG are set as environment variables or in the .env file.

Pysmurf Publisher Options

The following options can be set through the use of environment variables. Note that if SMURFPUB_BACKEND is not set to “udp”, messages will be discarded instead of published.

SMURFPUB_ID

An ID string associated with this system in order to disambiguate it from others; e.g. “readout_crate_1”.

SMURFPUB_BACKEND

A string that selects the backend publishing engine. Options are: “null” and “udp”. The null backend is the default, and in that case the published messages are simply discarded.

SMURFPUB_UDP_HOST

the target host for UDP packets. Defaults to localhost.

SMURFPUB_UDP_PORT

the target port for UDP packets. Defaults to module DEFAULT_UDP_PORT.

Description

The Pysmurf Controller agent exposes many essential pysmurf and sodetlib operations so that they can be called using OCS. Tasks and processes will generate a new local pysmurf and det-config instance, and load in the tunefile specified in the device cfg. The session object will be saved to the pysmurf instance as S._ocs_session so that sodetlib functions can add logs and data directly.

Additionally, arbitrary SODETLIB scripts can be run as subprocesses using the run task. Data can still be added to the session by passing it using the PysmurfPublisher to communicate with pysmurf-monitor agent (see the Passing Session Data section for more info).

In order for the Pysmurf instance to accurately represent the smurf state, we must be careful about not using a second pysmurf instance to modify any variables while a persistant instance exists. For that reason, the run function and most tasks are protected by a single lock, preventing you from running multiple tasks at a time.

Example Clients

Running a Script

The run task will tell the agent to run a script as an external process. The run function takes params

  • script: path to the script (in the docker container)

  • args: list of command line arguments to pass to the script

  • log: True if using agent logger, path to log file, or False if you don’t want to log the script’s stdout and stderr messages.

For instance, to run a script sodetlib/scripts/tune.py, the client script would look like:

from ocs.matched_client import MatchedClient

controller = MatchedClient('pysmurf-controller-s2', args=[])

script_path = '/sodetlib/scripts/tune.py'
args=['--bands', '0', '1', '2']
controller.run.start(script=script_path, args=args)

Passing Session Data

The Pysmurf Publisher can be used to pass information back from a detector operation script to the ocs client script using the session_data and session_logs message types.

Below is a simple control script that demonstrates this

 active_channels = S.which_on(0)
 S.pub.publish("Starting data stream", msgtype='session_log')
 datafile = S.take_stream_data(10) # Streams data for 10 seconds
 S.pub.publish("Data stream is closed", msgtype='session_log')
 S.pub.publish({'datafile': datafile}, msgtype='session_data')

From the OCS Client you can then inspect the session data

from ocs.matched_client import MatchedClient

controller = MatchedClient('pysmurf-controller-s2', args=[])

script_path = '/config/scripts/pysmurf/stream.py'
controller.run.start(script=script_path))

ok, msg, sess = controller.run.wait()
print(sess['data'])

This prints the dictionary:

{
    'datafile': '/data/smurf_data/20200316/1584401673/outputs/1584402020.dat',
    'active_channels': [0,1,2,3,4]
}

Agent API

class socs.agents.pysmurf_controller.agent.PysmurfController(agent, args)[source]

Controller object for running pysmurf scripts and functions.

Parameters:

  • agent (ocs.ocs_agent.OCSAgent) – OCSAgent object which is running

  • args (Namespace) – argparse namespace with site_config and agent specific arguments

agent

OCSAgent object which is running

Type:

ocs.ocs_agent.OCSAgent

log

txaio logger object created by agent

Type:

txaio.tx.Logger

prot

protocol used to call and monitor external pysmurf scripts

Type:

PysmurfScriptProtocol

lock

lock to protect multiple pysmurf scripts from running simultaneously.

Type:

ocs.ocs_twisted.TimeoutLock

slot

ATCA Slot of the smurf-card this agent is commanding.

Type:

int

run(script, args=[], log=True)[source]

Task - Runs a pysmurf control script.

Parameters:

  • script (string) – Path of the pysmurf script to run.

  • args (list, optional) – List of command line arguments to pass to the script. Defaults to [].

  • log (string/bool, optional) – Determines if and how the process’s stdout should be logged. You can pass the path to a logfile, True to use the agent’s log, or False to not log at all.

Notes

Data and logs may be passed from the pysmurf control script to the session object by publishing it via the Pysmurf Publisher using the message types session_data and session_logs respectively.

For example, below is a simple script which starts the data stream and returns the datfile path and the list of active channels to the session:

active_channels = S.which_on(0)
datafile = S.stream_data_on()
S.pub.publish({
    'datafile': datafile, 'active_channels': active_channels
}, msgtype='session_data')

This would result in the following session.data object:

>>> response.session['data']
{
    'datafile': '/data/smurf_data/20200316/1584401673/outputs/1584402020.dat',
    'active_channels': [0,1,2,3,4]
}
abort()[source]

Task - Aborts the actively running script.

check_state(poll_interval=10, test_mode=False)[source]

Process - Continuously checks the current state of the smurf. This will not modify the smurf state, so this task can be run in conjunction with other smurf operations. This will continuously poll smurf metadata and update the session.data object.

Parameters:

  • poll_interval (float) – Time (sec) between updates.

  • test_mode (bool, optional) – Run the Process loop only once. This is meant only for testing. Default is False.

Notes

The following data will be written to the session.data object:

>> response.session['data']
{
    'channel_mask': Array of channels that are streaming data,
    'downsample_factor': downsample_factor,
    'agg_time': Buffer time per G3Frame (sec),
    'open_g3stream': True if data is currently streaming to G3,
    'pysmurf_action': Current pysmurf action,
    'pysmurf_action_timestamp': Current pysmurf-action timestamp,
    'stream_tag': stream-tag for the current g3 stream,
    'last_update':  Time that session-data was last updated,
    'stream_id': Stream-id of the controlled smurf instance
}
run_test_func()[source]

Task - Task to test the subprocessing functionality without any smurf hardware.

stream(duration=None, kwargs=None, load_tune=True, stream_type='obs', subtype=None, tag=None, test_mode=False)[source]

Process - Stream smurf data. If a duration is specified, stream will end after that amount of time. If unspecified, the stream will run until the stop function is called.

Parameters:

  • duration (float, optional) – If set, determines how many seconds to stream data. By default, will leave stream open until stop function is called.

  • kwargs (dict) – A dictionary containing additional keyword arguments to pass to sodetlib’s stream_g3_on function

  • load_tune (bool) – If true, will load a tune-file to the pysmurf object on instantiation.

  • stream_type (string, optional) – Stream type. This can be either ‘obs’ or ‘oper’, and will be ‘obs’ by default. The tags attached to the stream will be <stream_type>,<subtype>,<tag>.

  • subtype (string, optional) – Operation subtype used tot tag the stream.

  • tag (string, optional) – Tag (or comma-separated list of tags) to attach to the G3 stream. This has precedence over the tag key in the kwargs dict.

  • test_mode (bool, optional) – Run the Process loop only once. This is meant only for testing. Default is False.

Notes

The following data will be written to the session.data object:

>> response.session['data']
{
    'stream_id': Stream-id for the slot,
    'sid': Session-id for the streaming session,
}
uxm_setup(bands=None, kwargs=None, run_in_main_process=False)[source]

Task - Runs first-time setup procedure for a UXM. This will run the following operations:

  1. Setup Amps (~1 min)

  2. Estimate attens if attens are not already set in the device cfg (~1 min / band)

  3. Estimate phase delay (~1 min / band)

  4. Setup tune (~7 min / band)

  5. Setup tracking param (~30s / band)

  6. Measure noise (~45s)

See the sodetlib setup docs for more information on the sodetlib setup procedure and allowed keyword arguments.

Parameters:

  • bands (list, int) – Bands to set up. Defaults to all.

  • kwargs (dict) – Dict containing additional keyword args to pass to the uxm_setup function.

  • run_in_main_process (bool) – If true, run smurf-function in main process. Mainly for the purpose of testing without the reactor running.

Notes

SODETLIB functions such as uxm_setup and any other functions called by uxm_setup will add relevant data to the session.data object to a unique key. For example, if all is successful session.data may look like:

>> response.session['data']
{
    'timestamps': [('setup_amps', 1651162263.0204525), ...],
    'setup_amps_summary': {
       'success': True,
       'amp_50k_Id': 15.0,
       'amp_hemt_Id': 8.0,
       'amp_50k_Vg': -0.52,
       'amp_hemt_Vg': -0.829,
    },
    'setup_phase_delay': {
        'bands': [0, 1, ...]
        'band_delay_us': List of band delays
    },
    'noise': {
       'band_medians': List of median white noise for each band
    }
}
uxm_relock(bands=None, kwargs=None, run_in_main_process=False)[source]

Task - Relocks detectors to existing tune if setup has already been run. Runs the following operations:

  1. Setup Amps (~1 min)

  2. Relocks detectors, setup notches (if requested), and serial gradient descent / eta scan (~5 min / band)

  3. Tracking setup (~20s / band)

  4. Noise check (~45s)

See the sodetlib relock docs for more information on the sodetlib relock procedure and allowed keyword arguments.

Parameters:

  • bands (list, int) – Bands to set up. Defaults to all.

  • kwargs (dict) – Dict containing additional keyword args to pass to the uxm_relock function.

  • run_in_main_process (bool) – If true, run smurf-function in main process. Mainly for the purpose of testing without the reactor running.

Notes

SODETLIB functions such as uxm_relock and any other functions called will add relevant data to the session.data object to a unique key. For example, if all is successful session.data may look like:

>> response.session['data']
{
    'timestamps': [('setup_amps', 1651162263.0204525), ...],
    'setup_amps_summary': {
       'success': True,
       'amp_50k_Id': 15.0,
       'amp_hemt_Id': 8.0,
       'amp_50k_Vg': -0.52,
       'amp_hemt_Vg': -0.829,
    },
    'noise': {
       'band_medians': List of median white noise for each band
    }
}
take_noise(duration=30., kwargs=None, tag=None, run_in_main_process=False)[source]

Task - Takes a short timestream and calculates noise statistics. Median white noise level for each band will be stored in the session data. See the sodetlib noise docs for more information on the noise function and possible keyword arguments.

Parameters:

  • duration (float) – Duration of timestream to take for noise calculation.

  • kwargs (dict) – Dict containing additional keyword args to pass to the take_noise function.

  • tag (string, optional) – Tag (or comma-separated list of tags) to attach to the G3 stream. This has precedence over the tag key in the kwargs dict.

  • run_in_main_process (bool) – If true, run smurf-function in main process. Mainly for the purpose of testing without the reactor running.

Notes

Median white noise levels for each band will be stored in the session.data object, for example:

>> response.session['data']
{
    'noise': {
       'band_medians': List of median white noise for each band
    }
}
take_bgmap(kwargs=None, tag=None, run_in_main_process=False)[source]

Task - Takes a bias-group map. This will calculate the number of channels assigned to each bias group and put that into the session data object along with the filepath to the analyzed bias-step output. See the bias steps docs page for more information on what additional keyword arguments can be passed.

Parameters:

  • kwargs (dict) – Additional kwargs to pass to take_bgmap function.

  • tag (Optional[str]) – String containing a tag or comma-separated list of tags to attach to the g3 stream.

  • run_in_main_process (bool) – If true, run smurf-function in main process. Mainly for the purpose of testing without the reactor running.

Notes

The filepath of the BiasStepAnalysis object and the number of channels assigned to each bias group will be written to the session.data object:

>> response.session['data']
{
    'nchans_per_bg': [123, 183, 0, 87, ...],
    'filepath': /path/to/bias_step_file/on/smurf_server.npy,
}
take_iv(kwargs=None, tag=None, run_in_main_process=False)[source]

Task - Takes an IV. This will add the normal resistance array and channel info to the session data object along with the analyzed IV filepath. See the sodetlib IV docs page for more information on what additional keyword arguments can be passed in.

Parameters:

  • kwargs (dict) – Additional kwargs to pass to the take_iv function.

  • tag (Optional[str]) – String containing a tag or comma-separated list of tags to attach to the g3 stream.

  • run_in_main_process (bool) – If true, run smurf-function in main process. Mainly for the purpose of testing without the reactor running.

Notes

The following data will be written to the session.data object:

>> response.session['data']
{
    'bands': Bands number of each resonator
    'channels': Channel number of each resonator
    'bgmap': BGMap assignment for each resonator
    'R_n': Normal resistance for each resonator
    'filepath': Filepath of saved IVAnalysis object
}
take_bias_steps(kwargs=None, rfrac_range=(0.2, 0.9), tag=None, run_in_main_process=False)[source]

Task - Takes bias_steps and saves the output filepath to the session data object. See the sodetlib bias step docs page for more information on bias steps and what kwargs can be passed in.

Parameters:

  • kwargs (dict) – Additional kwargs to pass to take_bais_steps function.

  • rfrac_range (tuple) – Range of valid rfracs to check against when determining the number of good detectors.

  • tag (Optional[str]) – String containing a tag or comma-separated list of tags to attach to the g3 stream.

  • run_in_main_process (bool) – If true, run smurf-function in main process. Mainly for the purpose of testing without the reactor running.

Notes

The following data will be written to the session.data object:

>> response.session['data']
{
    'filepath': Filepath of saved BiasStepAnalysis object
    'biased_total': Total number of detectors biased into rfrac_range
    'biased_per_bg': List containing number of biased detectors on each bias line
    'Rtes_quantiles': {
        'Rtes': List of 15%, 25%, 50%, 75%, 85% Rtes quantiles,
        'quantiles': List of quantile labels
        'count': Total count of the distribution
    }
    'responsivity_quantiles': Same as above for responsivity
    'Rfrac_quantiles': Same as above for Rfrac

}
take_bias_waves(kwargs=None, rfrac_range=(0.2, 0.9), tag=None, run_in_main_process=False)[source]

Task - Takes bias_wave and saves the output filepath to the session data object.

Parameters:

  • kwargs (dict) – Additional kwargs to pass to take_bias_wave function.

  • rfrac_range (tuple) – Range of valid rfracs to check against when determining the number of good detectors.

  • tag (Optional[str]) – String containing a tag or comma-separated list of tags to attach to the g3 stream.

  • run_in_main_process (bool) – If true, run smurf-function in main process. Mainly for the purpose of testing without the reactor running.

Notes

The following data will be written to the session.data object:

>> response.session['data']
{
    'filepath': Filepath of saved BiasWaveAnalysis object
    'biased_total': Total number of detectors biased into rfrac_range
    'biased_per_bg': List containing number of biased detectors on each bias line
    'Rtes_quantiles': {
        'Rtes': List of 15%, 25%, 50%, 75%, 85% Rtes quantiles,
        'quantiles': List of quantile labels
        'count': Total count of the distribution
    }
    'responsivity_quantiles': Same as above for responsivity
    'Rfrac_quantiles': Same as above for Rfrac

}
overbias_tes(bgs=None, kwargs=None)[source]

Task - Overbiases detectors using S.overbias_tes_all.

Parameters:

  • bgs (List[int]) – List of bias groups to overbias. If this is set to None, it will use all active bgs.

  • kwargs (dict) – Additional kwargs to pass to the overbias_tes_all function.

set_biases(bg=None, bias)[source]

Task - Task used to set TES biases.

Parameters:

  • bg (int, list, optional) – Bias group (bg), or list of bgs to set. If None, will set all bgs.

  • bias (int, float, list) – Biases to set. If a float is passed, this will be used for all specified bgs. If a list of floats is passed, it must be the same size of the list of bgs.

zero_biases(session, params)[source]

Task - Zeros TES biases for specified bias groups.

Parameters:

bg (int, list, optional) – bg, or list of bgs to zero. If None, will zero all bgs.

bias_dets(rfrac=(0.3, 0.6), kwargs=None)[source]

Task - Biases detectors to a target Rfrac value or range. This function uses IV results to determine voltages for each bias-group. If rfrac is set to be a value, the bias voltage will be set such that the median rfrac across all channels is as close as possible to the set value. If a range is specified, the voltage will be chosen to maximize the number of channels in that range.

See the sodetlib docs page for biasing dets into transition for more information on the functions and additional keyword args that can be passed in.

Parameters:

  • rfrac (float, tuple) – Target rfrac range to aim for. If this is a float, bias voltages will be chosen to get the median rfrac of each bias group as close as possible to that value. If

  • kwargs (dict) – Additional kwargs to pass to the bias_dets function.

Notes

The following data will be written to the session.data object:

>> response.session['data']
{
    'biases': List of voltage bias values for each bias-group
}
all_off(disable_amps=True, disable_tones=True)[source]

Task - Turns off tones, flux-ramp voltage and amplifier biases

Parameters:

  • disable_amps (bool) – If True, will disable amplifier biases

  • disable_tones (bool) – If True, will turn off RF tones and flux-ramp signal

Supporting APIs

class socs.agents.pysmurf_controller.agent.PysmurfScriptProtocol(path, log=None)[source]

The process protocol used to dispatch external Pysmurf scripts, and manage the stdin, stdout, and stderr pipelines.

Parameters:

  • path (str) – Path of script to run.

  • log (txaio.tx.Logger) – txaio logger object, used to log stdout and stderr messages.

path

Path of script to run.

Type:

str

log

txaio logger object, used to log stdout and stderr messages.

Type:

txaio.tx.Logger

end_status

Reason that the process ended.

Type:

twisted.python.failure.Failure

connectionMade()[source]

Called when process is started

outReceived(data)[source]

Called whenever data is received through stdout

errReceived(data)[source]

Called whenever data is received through stderr

processExited(status: Failure)[source]

Called when process has exited.