Bluefors Agent

The Bluefors Agent is an OCS Agent which tracks the contents of the Bluefors logs and passes them to the live monitor and to the OCS housekeeping data aggregator.

usage: agent.py [-h] [--log-directory LOG_DIRECTORY]

Agent Options

--log-directory

Configuration File Examples

Below are configuration examples for the ocs config file and for running the Agent in a docker container.

OCS Site Config

An example configuration for your ocs config file:

{'agent-class': 'BlueforsAgent',
 'instance-id': 'bluefors',
 'arguments': [['--log-directory', '/logs']]
}

Note

The --log-directory argument will need to be updated in your configuration if you are running outside of a Docker container.

Docker Compose

Example docker-compose configuration:

ocs-bluefors:
  image: simonsobs/socs:latest
  hostname: ocs-docker
  network_mode: "host"
  environment:
    - INSTANCE_ID=bluefors
    - LOGLEVEL=info
    - FRAME_LENGTH=600
    - STALE_TIME=2
    - MODE=follow
  volumes:
    - ${OCS_CONFIG_DIR}:/config:ro
    - /home/simonsobs/bluefors/logs/:/logs:ro

Depending on how you are running your containers it might be easier to hard code the OCS_CONFIG_DIR environment variable.

Environment Variables

There are several environment variables that can be used to configure the Bluefors Agent container.

Variable	Description
LOGLEVEL	Verbosity of the logs.
FRAME_LENGTH	.g3 frame length.
MODE	File tracking mode, either “follow” or “poll”, defaulting to “poll”. In “follow” mode the Tracker will read the next line in the file if able to. In “poll” mode stats about the file are used to determine if it was updated since the last read, and if it has been the file is reopened to get the last line. This is more I/O intensive, but is useful in certain configurations.
STALE_TIME	Time limit (in minutes) for newly opened files to be published to feeds. Data older than this time when read will not be published.

Description

The Bluefors Agent tracks the contents of the Bluefors log files, publishing data as they are written to the log files while the Agent is running. There are many ways this could be setup to run, depending on your system. In this section we describe several of those configurations.

The Bluefors software must run on a Windows OS. This may make integrating into OCS different than on a Linux box (where we run in a Docker container) depending on your computer configuration.

Running the Bluefors Agent in a Docker container is the recommended configuration, but you have options for whether that container runs on Windows or on Linux. If neither of those solutions works, running directly on the Windows Subsystem for Linux is possible.

Configuring the Bluefors Logs

You should configure your Bluefors software to all log to the same directory. This will place thermometry logs and system logs in the same date directories within this top level directory.

Docker Container on Windows

Running Docker on Windows can vary a lot depending on what version of Windows you have, what kind of hardware you are running on, and whether or not you are running Windows virtually. As such, the setup can change significantly based on your configuration. We’ll try to provide general guidelines for how to setup the system.

Note

If you run Windows 10 in a virtualized environment you might run into difficulty running Docker, depending on what virutalization software you are using and what your hardware supports.

Dependencies

There are various limitations to what you can run on Windows depending on your configuration. For instance, Docker for Windows doesn’t run on Windows 10 Home, but a legacy tool called Docker Toolbox works.

Windows 10 Pro/Enterprise
Docker for Windows

Or:

Windows 10 Home
Docker Toolbox for Windows

Setup

The general outline for setup of Docker Toolbox (which probably also works on Windows 10 Pro/Enterprise, but Docker recommends upgrading to more modern tools, i.e. Docker for Windows, if possible) is:

Install Docker Toolbox
Run docker terminal (this performs some Virtualbox setup)
Run docker login
Clone the ocs-site-configs repo and create a directory for your machine
Configure ocs/docker-compose files
Make sure your system clock is set to UTC
Bring up the container(s)

Windows Subsystem for Linux

Windows 10 has a feature called Windows Subsystem for Linux, which allows you to run Linux CLI tools in Windows. This can be used much like you would if you were to configure a Linux host to run an OCS Agent. While we recommend running OCS Agents in Docker containers, this configuration is still possible.

You should refer to the OCS documentation for how to install, but the general outline is:

Install git
Clone the ocs, socs, and ocs-site-configs repositories
Install ocs and socs
Configure your ocs-config file and perform the associated setup
Start the Bluefors agent and command it to acquire data via an OCS client

Docker Container on Linux

In some situations it might be better to run the Agent on Linux. This might be because you run Windows within a virtual environment and write the logs to a shared filesystem, or if Docker for Windows is difficult to install for whatever reason. In this later case, if you sync your logs to a Linux system regularly, i.e. at roughly the rate they are written, you can run the Agent on that Linux system (or one that also has access to that filesystem.)

In this setup, depending on the syncing mechanism the Agent might need to reopen the file regularly. It will grab the latest reading from the logs to publish. Since it’s possible that this could be quite old in some scenarios the timestamp is now checked before publishing. It can be at most STALE_TIME minutes old. This defaults to two minutes, but can be set with the STALE_TIME environment variable.

Agent API

class socs.agents.bluefors.agent.BlueforsAgent(agent, log_directory)[source]

Agent to track the Bluefors logs generated by an LD400.

Parameters:: log_directory (str) – Top level log directory for the Bluefors logs

acq()[source]: Process - Monitor and publish data from the Bluefors log files.

Supporting API

class socs.agents.bluefors.agent.LogTracker(log_dir)[source]

Log Tracking helper class. Always tracks current date’s logs.

Parameters:: log_dir (str) – Top level log directory

log_dir

Top level log directory

Type:: str

date

Today’s date. Used to determine the active log directory

Type:: datetime.date

file_objects

A dictionary with filenames as keys, and another dict as the value. Each of these sub-dictionaries has two keys, “file_object”, and “stat_results”, with the open file object, and os.stat results as values, respectively. For example:

{'CH6 T 21-05-27.log':
    {'file_object': <_io.TextIOWrapper name='CH6 T 21-05-27.log' mode='r' encoding='UTF-8'>,
    'stat_results': os.stat_result(st_mode=33188,
                                   st_ino=1456748,
                                   st_dev=65024,
                                   st_nlink=1,
                                   st_uid=1000,
                                   st_gid=1000,
                                   st_size=11013,
                                   st_atime=1622135813,
                                   st_mtime=1622135813,
                                   st_ctime=1622135813)
    }
}

Type:: dict

check_open_files()[source]

Check all log files are opened.

The Status logs don’t always exist, but we want to catch them when they do get generated. This rebuilds the file list and checks all the files in it are in the open file objects dictionary.

set_active_date()[source]: Set the active date to today.

reopen_file(filename)[source]

If a file is already open and a new inode is detected, reopen the file. Returns the last line of the file.

Parameters:: filename (str) – Full path to filename to open

open_all_logs()[source]: Open today’s logs and move to end of files.

close_all_files()[source]: Close all the files tracked by the LogTracker.

class socs.agents.bluefors.agent.LogParser(tracker, mode='follow', stale_time=2)[source]

Log Parsing helper class.

Knows the internal formats for each log type. Used to loop over all logs tracked by a LogTracker and publish their contents to an OCS Feed.

Parameters:

tracker (LogTracker) – log tracker that contains paths and file objects to parse
mode (str) – Operating mode for the log tracker. Either “follow” or “poll”, defaulting to “follow”. In “follow” mode the Tracker will read the next line in the file if able to. In “poll” mode stats about the file are used to determine if it was updated since the last read, and if it has been the file is reopened to get the last line. This is more I/O intensive, but is useful in certain configurations.
stale_time (int) – Time in minutes which represents how fresh data in the bluefors logs must be when we open them in order to publish to OCS. This ensures we don’t reopen a file much later than when they were collected and publish “stale” data to the OCS live HK system.

static timestamp_from_str(time_string)[source]

Convert time string from Bluefors log file into a UNIX timestamp.

Parameters:: time_string (str) – String in format “%d-%m-%y,%H:%M:%S”
Returns:: UNIX timestamp
Return type:: float

static identify_log(filename)[source]

Identify type of log return unique identifier

Parameters:

filename (str) – path to file for identification

Returns:

A tuple containing two strings:

The log type, i.e. ‘lakeshore’, ‘flowmeter’
Unique identifier based on filetype, i.e. ‘lakeshore_ch8_p’, ‘pressure_ch1_state’

Return type:

tuple

read_and_publish_logs(app_session)[source]

Read a new line from each log file if there is one, and publish its contents to the app_session’s feed.

Parameters:: app_session (ocs.ocs_agent.OpSession) – session from the ocs_agent, used to publish to bluefors feed