Magpie Agent

The magpie is an incredibly intelligent bird, with a decent ability to mimic other bird calls, though not as good as the superb lyrebird. In the context of OCS, the job of the Magpie agent is to take detector data from the SMuRF Streamer and translate into G3Frames that are compatible with lyrebird. This requires a small bit of pre-processing and downsampling that is done whenever a new frame is received.

usage: python3 agent.py [-h] [--src SRC [SRC ...]] [--dest DEST]
                        [--stream-id STREAM_ID] [--target-rate TARGET_RATE]
                        [--delay DELAY] [--layout {grid,wafer}] [--xdim XDIM]
                        [--ydim YDIM] [--wafer-scale WAFER_SCALE]
                        [--det-map DET_MAP] [--fake-data]
                        [--offset OFFSET OFFSET] [--rotation ROTATION]
                        [--monitored-channels MONITORED_CHANNELS [MONITORED_CHANNELS ...]]
                        [--monitored-channel-rate MONITORED_CHANNEL_RATE]
                        [--demod-freq DEMOD_FREQ]
                        [--demod-bandwidth DEMOD_BANDWIDTH]

Agent Options

--src

Address of incoming G3Frames.

Default: “tcp://localhost:4532”

--dest

Port to serve lyrebird frames

Default: 8675

--stream-id

Stream-id to use to distinguish magpie streams.This will be prepended to data-val names in lyrebird.

Default: “none”

--target-rate, -t

Target sample rate for data being sent to Lyrebird. Detector data will be downsampled to this rate.

Default: 20

--delay

Delay (sec) between the timestamp of a G3Frame relative to the initial frame, and when the frame should be sent to lyrebird. This must be larger than the frame-aggregation time for smooth update times in lyrebird.

Default: 5

--layout, -l

Possible choices: grid, wafer

Focal plane layout style

Default: “grid”

--xdim

Number of pixels in x-dimension for grid layout

Default: 64

--ydim

Number of pixels in y-dimension for grid layout

Default: 64

--wafer-scale, --ws

scale of wafer coordinates

Default: 50.0

--det-map

Path to det-map csv file

--fake-data

If set, will stream fake data instead of listening to a G3stream.

Default: False

--offset

Offset of detector coordinates with respect to lyrebird coordinate system

Default: [0, 0]

--rotation

Rotation of wafer about its center (rad)

Default: 0.0

--monitored-channels

Readout channels to start monitoring on startup

Default: []

--monitored-channel-rate

Target sample rate for monitored channels

Default: 10

--demod-freq

Demodulation frequency

Default: 8

--demod-bandwidth

Demodulation bandwidth

Default: 0.5

Configuration File Examples

Below are example docker-compose and ocs configuration files for running the magpie agent.

OCS Site Config

Below is the site-config entry for the magpie instance we have running at UCSD on K2SO. We are displaying detectors using a wafer layout, determined by a detmap csv file, with a target sample rate of 20 Hz:

{'agent-class': 'MagpieAgent',
 'instance-id': 'magpie-crate1slot2',
 'arguments': [
   '--stream-id', 'crate1-slot2',

   '--src', 'tcp://localhost:4532',

   '--dest', 8675,
   '--delay', 5,
   '--target-rate', 20,

   '--layout', 'wafer',
   # Detmap CSV file
   '--det-map', '/home/jlashner/lyrebird_demo/detmap.csv',
   '--offset', 0, 0,
   '--rotation', 0,
   '--demod-freq', 8,
   '--demod-bandwidth', 0.5
 ]},

It is possible to tell magpie to stream data from existing G3Files instead of directly from the smurf-stream. To do this, simply set the src argument to be the filepath of the file you wish to stream. If you want to stream from multiple files in series, you can do this by putting multiple source filepaths, as is shown below. In this case, once the first file has finished streaming magpie will immediately start streaming from the second file:

'--src', '/path/to/file1.g3', '/path/to/file2.g3',

You can also tell the magpie agent to ignore the src argument all together and generate fake data by adding the --fake-data argument.

Docker Compose

Below is an example docker-compose entry for running a magpie corresponding to crate-id 1 and slot 2. If crossbar is being run on a different server, you’ll have to modify the site-hub and site-http args accordingly:

ocs-magpie-crate1slot2:
    image: simonsobs/socs:latest
    hostname: ocs-docker
    user: ocs:ocs
    network_mode: host
    environment:
        - INSTANCE_ID=magpie-crate1slot2
        - SITE_HUB=ws://localhost:8001/ws
        - SITE_HTTP=http://localhost:8001/call
    volumes:
        - ${OCS_CONFIG_DIR}:/config
        - /data:/data

Lyrebird

Installation

In order to build lyrebird, you first need a local build of spt3g-software. Since we require direct access to the G3 C++ interface, unfortunately we’re not able to use the spt3g / so3g pypi package. You can follow the instructions on the SPT3G github page to build.

Once that is successful, clone the simonsobs fork of lyrebird, and follow the build instructions on the lyrebird readme.

After lyrebird is built successfully, you’ll want to add the following lines to your bashrc:

export PATH=/path/to/lyrebird/build/lyrebird-bin:$PATH
export PATH=/path/to/lyrebird/bin:$PATH

This will add two scripts to your path:

• lyrebird, the main lyrebird executable which takes in the path to a cfg file specifying data vals, streamer ports, etc. and starts lyrebird from that

• run_lyrebird which just takes the ports to listen to as command line arguments, and will use that to generate a new cfg file and start lyrebird from that.

Startup

To get lyrebird running, you must bring software up in the following order:

1. Make sure smurf-streamers you wish to monitor are running, though data doesn’t have to actually be streaming

2. Bring up Magpie OCS agents. If streamers are not already running, this will currently fail when it begins the read process. If this happens, you can restart the process manually using an OCS client, or just restart the agent, which will begin the process on startup. Make sure each magpie instance you are running has a different stream-id and a different dest port.

3. Run lyrebird. The lyrebird executable takes in a config file that specifies data-vals and ports to monitor for G3Streams, but it is much easier to use the run_lyrebird script in the lyrebird/bin directory. This will generate a temporary config file determined by the arguments passed in, and then start lyrebird with that config file. Right now you need to pass in the ports that it should monitor. For instance, if you have two magpie agents running with dest ports 8675 and 8676, you can run:

   run_lyrebird --port 8675 8676
   

   and this will start lyrebird for the two corresponding slots.

Operation

This section will describe the main operation of the SO fork is lyrebird. This isn’t necessary for running magpie, but this isn’t well documented in the lyrebird repo so this may be useful for future development.

On startup, lyrebird will attempt to connect to a G3NetworkSender streaming on any ports specified in the lyrebird config file. The first frame read from the port must be a config frame containing details on how lyrebird should display the focal-plane.

A config file for a focal-plane with nchans channels will contain the following fields:

• x: Array of len nchans with the x-coords of the visual element for each channel

• y: Array of len nchans with the y-coords of the visual element for each channel

• cname: Array of len nchans the base name for each channel. This is something like <magpie-instance-id>/channel_10

• rotation: Array of len nchans containing how many rads each vis elem should be rotated (counter-clockwise) when drawn.

• templates: Array of len nchans containing the det template (defined in the

• values: Array of len n * nchans containing data values corresponding to a given channel. These must be unique, and are of the form <cname>/<value_name>

• eqs: Array of len m * nchans containing the equation descriptions for each channel. See below for a full description, but these will be something like / + 1 s <cname>/<valuename> 2.

• eq_labels: Array of len m * nchans containing the labels for each individual equation. These are what will be displayed in the lyrebird menu gui, and can be something like osc-tod or rms.

• color_is_dynamic: Array of len m * nchans containing bools that determined if the eq values should be dynamically adjusted before passing to the colormap. If this is False, the eq output will be sent directly to the colormap. If True, the eq output will first be mapped to a value in the range (0, 1) by interpolating between the min/max values in the lyrebird data buffer.

• cmaps: Array of len m * nchans containing the name of the colormap to use to display each equation.

Data Values

data values are named values that lyrebird buffers and tracks internally, and can be used in equation evaluations.

A number of data values can be stored per detector channel (as long as this number is the same for all detector channels). So for instance if you want to keep track of the detector TOD, rms, bias-group, and whether the channel is flagged, you’ll need to register the data values <cname>/tod, <cname>/bg, <cname>/rms, <cname>/flagged for each detector.

Equations

Lyrebird can also store any number of equations for each detector channel. This number can be different from the number of data values, but the number of equations must be the same for each detector. These equations are what are actually visualized for each channel in the lyrebird GUI.

The equations are registered as strings in the Polish Notation. This is a method of describing equations where the operator comes before the operands, and is a notation that is very easy to parse and evaluate. For example, the operation \(a + b\) will be + a b in polish notation, and \((a + b) / 2\) can be written as / + a b 2. Operands can either be numeric values or registered data values, including both channel-specific data values such as <cname>/tod and global data values that are registered in the lyrebird config variable.

Below is a full table of operators that can be interpreted by lyrebird:

Lyrebird Equation Operations

+ x y	\(x + y\)	| x y	\(x \;\mathrm{or}\; y\)
- x y	\(x - y\)	= x y	\(x == y\)
* x y	\(x * y\)	! x	\(\mathrm{not}\; x\)
/ x y	\(x / y\)	& x y	\(x \;\mathrm{and}\; y\)
% x y	\(x % y\)	c x	\(\cos(x)\)
a x	\(\left|x\right|\)	t x	\(\tan(x)\)
^ x y	\(x^y\)	T x	\(\arctan(x)\)
s x	\(\sin(x)\)	q x	\(\sqrt{x}\)

For a more complex equation you might want to do something like display the TOD value if a channel is not flagged, and if it is flagged send it to -1. This example is done with the string:

+ * ! flagged tod * -1 flagged

which describes the equation:

\[\mathrm{tod} \times (! \mathrm{flagged}) + -1 \times \mathrm{flagged}\]

The displayed equation can be selected by changing the displayed_eq idx in the lyrebird GUI.

Color Maps

After equations are evaluated, the results will be passed to the specified color map to determine what color the visual element should be drawn with.

Below are the colormaps available in lyrebird. Generally these take in arbitrary floats, but will clamp them into the range [0, 1]. Along with the standard cmaps, there are additional bolo_cmaps, that map different regions of the range [0, 1] to varying colors to for different purposes. If the value is 0 or infinite, the vis-elem will be colored grey. The range (0, 0.3) will be colored bright red, and then the range (.3, 1) will be a gradient from black to a base color to white as pictured below..

There are additional colormaps white_cmap_fs and rainbow_cmap_fs that are identical to their counterparts above but instead map onto the range [-1, 1]. The colormap phase_cmap is identical to rainbow_cmap but maps the range \([0, \pi]\).

If an equation’s color is set to be dynamic, each equation value will be mapped to the range [0, 1] by interpolating between the min and max value in the equation’s data buffer. This allows you to view changes equations that span a large range with a high dynamic-range, but makes direct channel-to-channel comparison impossible.

Detector Layouts

Grid Layout

The grid layout is a grid with 4096 elements (maximum number of channels a single smurf slot can stream)

This layout contains 8 rows containing 512 detectors each, with the bottom row being band 0, and the top row band 7. This is the easiest layout to set up and is useful for viewing detector response as a function of their resonator frequency or band / channel id.

Wafer Layout

The Wafer layout takes in a det-map CSV file generated from the simonsobs detmap package, and uses it to generate a focal-plane layout.

Agent API

class socs.agents.magpie.agent.MagpieAgent(agent, args)

Agent for processing streamed G3Frames, and sending data to lyrebird.

target_rate

This is the target sample rate of data to be sent to lyrebird. Incoming data will be downsampled to this rate before being sent out.

Type:

float

ds_offset

Offset for downsample to avoid hiccups at frame boundaries

Type:

int

fp

This is the FocalplaneConfig object that contains info about what channels are present in the focal-plane representation, and their positions.

Type:

FocalplaneConfig

mask

This is a channel mask that maps readout channel to absolute-smurf-chan. Before a status frame containing the ChannelMask is seen in the G3Stream, this defaults to being an identity mapping which just sends the readout channel no. to itself. Once a status frame with the channel mask is seen, this will be updated

Type:

np.ndarray

out_queue

This is a queue containing outgoing G3Frames to be sent to lyrebird.

Type:

Queue

delay

The outgoing stream will attempt to enforce this delay between the relative timestamps in the G3Frames and the real time to ensure a smooth flow of data. This must be greater than the frame-aggregation time of the SmurfStreamer or else lyrebird will update data in spurts.

Type:

float

avg1, avg2

Two Rolling Averagers which are used to calculate the rolling RMS data.

Type:

RollingAvg

monitored_channels

List of monitored channels whose data should be sent to grafana. This list will contain entries which look like (readout_chan_number, field_name).

Type:

list

monitored_chan_sample_rate

Sample rate (Hz) to target when downsampling monitored channel data for grafana.

Type:

float

demod

Demodulator used to calculate demod signal for incoming timestreams

Type:

Demodulator

self.demod_freq

Demodulation frequency

Type:

float

self.demod_bandwidth

Filter cutoff for demodulation lowpass

Type:

float

wlcalc

WhiteNoiseCalculator used to calculate white noise levels for incoming timestreams

Type:

WhiteNoiseCalculator

set_target_rate(target_rate)

Sets the target downsampled sample-rate of the data sent to lyrebird

Parameters:

target_rate – float Target sample rate for lyrebird (Hz)

set_delay(delay)

Sets the target downsampled sample-rate of the data sent to lyrebird

Parameters:

target_rate – float Target sample rate for lyrebird (Hz)

set_monitored_channels(chan_info, sample_rate=10)

Task - Sets channels which will be monitored and have their downsampled data sent to grafana.

Field names can be manually specified in the chan_info list because it may be helpful to set a consistent name for specific channels, such as “in_transition”, instead of using the autogenerated channel name, which can change between tunes. Any additional field names used will remain in the influx database, so be wary of programatically adding many of them.

Parameters:

• chan_info (list) –

  List of channel info corresponding to channels to monitor. Entries of this list can be:

  • ints: This will be interpreted as the readout-channel to monitor. The field-name will be set to “r<chan_no>”.

  • tuples of length 2: Here the first element will be the readout chan to monitor, and the second value will be the field name to use for that channel.

  This list can be no more than 6 elements to limit how much detector data is saved to the HK format.

• sample_rate (float) – Target sample rate (Hz) to downsample detector data to. This must be less than 20 Hz to limit how much detector data is saved to hk.

read(src='tcp://localhost:4532')

Process - Process for reading in G3Frames from a source or list of sources. If this source is an address that begins with tcp://, the agent will attempt to connect to a G3NetworkSender at the specified location. The src param can also be a filepath or list of filepaths pointing to G3Files to be streamed. If a list of filenames is passed, once the first file is finished streaming, subsequent files will be streamed.

stream_fake_data()

Process - Process for streaming fake data. This will queue up G3Frames full of fake data to be sent to lyrebird.

send(dest)

Process - Process for sending outgoing G3Frames. This will query the out_queue for frames to be sent to lyrebird. This will try to regulate how fast it sends frames such that the delay between when the frames are sent, and the timestamp of the frames are fixed.

Supporting APIs

class socs.agents.magpie.agent.FIRFilter(b, a, nchans=None)

Class for Finite Input Response filter. Filter state is preserved between lfilt calls so you can filter frame-based data.

From scipy docs, the output of the filter is determined by:

a[0]*y[n] = b[0]*x[n] + b[1]*x[n-1] + … + b[M]*x[n-M]

• a[1]*y[n-1] - … - a[N]*y[n-N]

lfilt(data, in_place=True)

Filters data in place

classmethod butter_highpass(cutoff, fs, order=5)

Creates an highpass butterworth FIR filter

Parameters:

• cutoff (float) – Cutoff freq (Hz)

• fs (float) – sample frequency (Hz)

• order (int) – Order of the filter

classmethod butter_lowpass(cutoff, fs, order=5)

Creates an lowpass butterworth FIR filter

Parameters:

• cutoff (float) – Cutoff freq (Hz)

• fs (float) – sample frequency (Hz)

• order (int) – Order of the filter

classmethod moving_avg(width)

Creates a moving avg filter

Parameters:

width (int) – number of samples to average together

classmethod differ(delay=1)

FIR filter to calculate a rolling diff of incoming data.

Parameters:

delay (int) – Sets the diff spacing

class socs.agents.magpie.agent.Demodulator(f, bw=1, fs=200)

Helper class for demodulating a live timestream

Parameters:

• f (float) – Demodulation frequency

• bw (float) – Bandwidth. This will be the filter-cutoff of the applied lowpass filter

• fs (float) – Sample rate of incoming data

apply(times, data)

Applies demodulation to data segment.

Returns:

demod – Array of (unnormalized) demodulated data in the same shape as the input data

Return type:

np.ndarray

class socs.agents.magpie.agent.WhiteNoiseCalculator(fs=200, navg=200)

Helper class for calculating white noise levels of incoming data

Parameters:

• fs (float) – Sample rate of incoming data

• navg (int) – Number of samples to average over in the RMS calc

apply(data)

Returns rms / sqrt(fsamp), which estimates the white noise level

class socs.agents.magpie.agent.VisElem(name, x, y, rot, template, abs_smurf_chan, cmap_idx=0)

Container for config info for Lyrebird visual elements.

name

Name of the channel

Type:

str

x

x-coord of the element

Type:

float

y

y-coord of the element

Type:

float

rot

Rotation angle of the element (rads)

Type:

float

vals

List containing the names of the data values corresponding to this visual element. All vis-elems must have the same number of data-values. Data-values must be unique, such as <channel_name>/rms

Type:

List[str]

eqs

List of equations to be displayed by the visual element. Equations are written in Polish Notation, and may contain a combination of numbers and data-values. Data-values can be global as defined in the lyrebird config file, or channel values as defined in the value_names argument. There must be the same number of eqs per visual element.

Polish notation is a method of writing equations where operators precede the operands, making it easier to parse and evaluate. For example, the operation \(a + b\) will be + a b in polish notation, and \((a + b) / 2\) can be written as / + a b 2. See the magpie docs page for a full list of operators that are accepted by lyrebird.

Type:

List[str]

eq_labels

List of strings used to label equations in the lyrebird gui. This list must have the same size as the eqs array.

Type:

List[str]

cmaps

List of colormaps to use for each equation. This must be the same size as the eqs array.

Type:

List[str]

template

Template used for this vis-elem. Templates are defined in the lyrebird cfg file.

Type:

str

abs_smurf_chan

Absolute smurf-channel corresponding to this visual element.

Type:

int

eq_color_is_dynamic

List of booleans that determine if each equation’s color-scale is dynamic or fixed. This must be the same size as the eqs array.

Type:

List[bool]

class socs.agents.magpie.agent.FocalplaneConfig

config_frame()

Generates a config frame for lyrebird

add_vis_elem(*args, **kwargs)

Adds a visual element to the focal-plane and updates the channel mask

classmethod grid(stream_id, xdim, ydim, ygap=0, offset=(0.0, 0.0))

Creates a FocalplaneConfig object for a grid of channels.

Parameters:

• stream_id (str) – Stream-id for the magpie agent. This will be prepended to all lyrebird data-val names.

• xdim (int) – Number of channels in the x-dim of the grid

• ydim (int) – Number of channels in the y-dim of the grid

• ygap (int) – A small gap will be added every ygap rows, to better organize channels.

• offset (Tuple(float, float)) – Global offset of the grid with respect to the lyrebird coordinate-system

classmethod from_csv(stream_id, detmap_file, wafer_scale=1.0, offset=(0, 0), rotation=0.0)

Creates a FocalplaneConfig object from a detmap csv file.

Parameters:

• stream_id (str) – Stream-id for the magpie agent. This will be prepended to all lyrebird data-val names.

• detmap_file (str) – Path to detmap csv file.

• wafer_scale (int) – Scalar to multiply against det x and y positions when translating to lyrebird positions. Defaults to 1, meaning that the lyrebird coordinate system will be the same as the det-map coordinate system, so x and y will be in um.

• offset (Tuple(float, float)) – Global offset of the grid with respect to the lyrebird coordinate-system. If wafer_scale is 1, this should be in um.