Pysmurf Monitor

The pysmurf_monitor agent listens to the UDP messages that the pysmurf publisher sends and acts on them. It will add newly registered filse to the pysmurf_files database, and send session info to pysmurf-controller agents through an OCS Feed.

usage: python3 agent.py [-h] [--udp-port UDP_PORT]
                        [--create-table CREATE_TABLE] [--db-path DB_PATH]
                        [--echo-sql] [--test-mode]

Agent Options

--udp-port

Port for upd-publisher

--create-table

Specifies whether agent should create or update pysmurf_files table if non exists.

Default: True

--db-path

Path to suprsync sqlite database

Default: “/data/so/databases/suprsync.db”

--echo-sql

Default: False

--test-mode

Specifies whether agent should run in test mode, meaning it shuts down after processing any file(s).

Default: False

Configuration File Examples

OCS Site Config

Example site-config entry:

{'agent-class': 'PysmurfMonitor',
 'instance-id': 'pysmurf-monitor',
 'arguments': [['--udp-port', 8200],
               ['--create-table', True]]},

Docker Compose

An example docker-compose entry might look like:

ocs-pysmurf-monitor:
    image: simonsobs/socs:latest
    hostname: ocs-docker
    user: cryo:smurf
    network_mode: host
    container_name: ocs-pysmurf-monitor
    environment:
        - INSTANCE_ID=pysmurf-monitor
        - SITE_HUB=ws://${CB_HOST}:8001/ws
        - SITE_HTTP=http://${CB_HOST}:8001/call
    volumes:
        - ${OCS_CONFIG_DIR}:/config
        - /data:/data

Where SOCS_TAG and CB_HOST are set in the .env file in the same dir as the docker-compose file.

Description

Interfacing with the SupRsync Database

The pysmurf-monitor agent now adds files to a specified suprsync database when a new pysmurf file is registered, or it receives a message from the publisher built into the smurf-streamer saying that a g3 file has been closed and finalized.

Pysmurf auxiliary files will be written using the archive name “smurf”, and g3 timestreams will have the archive name “timestreams”. The pysmurf-monitor will determine the remote relative path based on the pysmurf action / action timestamp, so it will conform to the directory structure previously used by the pysmurf archiver:

<remote_base_dir>/<5 ctime digits>/<pub_id>/<action_timestamp>_<action>/<plots or outputs>

The SupRsync agents on the smurf-server will use these database entries to know what files to copy over to a daq node or simons1.

Agent API

class socs.agents.pysmurf_monitor.agent.PysmurfMonitor(agent, args)[source]

Monitor for pysmurf UDP publisher.

This agent should be run on the smurf-server and will monitor messages published via the pysmurf Publisher.

Main functions are making sure registered files make their way to the pysmurf files database, and passing session info to pysmurf-controller agents via the pysmurf_session_data

Parameters:

agent (ocs.ocs_agent.OCSAgent) – OCSAgent object
args (Namespace) – argparse namespace with site_config and agent specific arguments

agent

OCSAgent object

Type:: ocs.ocs_agent.OCSAgent

log

txaio logger object created by agent

Type:: txaio.tx.Logger

file_queue

Queue containing metadata for registered files

Type:: queue.Queue

db_path

Path to the suprsync database where files should be entered

Type:: str

running

True if the main process is running.

Type:: bool

echo_sql

If True, will echo all sql statements whenever writing to the suprsync db.

Type:: bool

run(test_mode=False)[source]

Process - Main process for the pysmurf monitor agent. Processes files that have been added to the queue, adding them to the suprsync database.

Parameters:: test_mode (bool, optional) – Stop the Process loop after processing any file(s). This is meant only for testing. Default is False.

Supporting APIs

agent.create_remote_path(archive_name)

Creates “remote path” for file.

For pysmurf ancilliary files (in the smurf archive), paths are generated from the pysmurf action / action timestamp:

<archive_dir>/<5 ctime digits>/<pub_id>/<action_timestamp>_<action>/<plots or outputs>

For timestream files, the path will be the relative path of the file from whatever the g3_dir is.

Parameters:

(dict) (meta) – A dict containing file metadata that’s sent from the pysmurf publisher when a new file is registered. Contains info such as the pysmurf action, file timestamp, path, publisher id, etc.
(str) (archive_name) – Name of the archive the file belongs to. ‘smurf’ if it is a pysmurf ancilliary file, in which case the path will be generated based on the pysmurf action / timestamp

PysmurfMonitor.datagramReceived(_data, addr)[source]

Called whenever UDP data is received.

Parameters:

_data (str) – Raw data passed over UDP port. Pysmurf publisher will send a JSON string
addr (tuple) – (host, port) of the sender.