• Ability to cluster appliances and to scale by adding appliances to the cluster.
  • Limited support for redundancy.
• Multiple stages and an inbuilt process to move data between the stages.
  • This supports the ability to use faster storage (which is perhaps limited in size) to improve performance.
  • Ability to reduce (decimate) the data as it moves into a store.
• Focus on data retrieval performance.
• A management interface giving you the ability to manage and monitor the system using a browser. This includes
  1. The ability to add PVs to a cluster of appliances using a browser (perhaps by users).
  2. Various metrics to help with capacity planning.
  3. Ability to define system-wide defaults for archiving parameters using policies.
• Ability to script the business processes in the appliances using an external process in a language like Python.
• Ability to configure various archiving parameters on a per PV basis.
• Ability to customize the appliance to suit a different set of requirements. This includes
  • The ability to use alternate storage technologies that may better suit your needs or perform better in your environment.
  • The ability to define your own data reduction algorithms and to optionally cache data generated by these algorithms on a per PV basis.
  • Simple ways to add support for new MIME types in data retrieval responses.
• Support for EPICS aliases.
• Support for EPICS 7/PVAccess/Structured data.
• Support for retrieval of data using CS-Studio, the ArchiveViewer and Matlab.
• Limited integration with existing Channel Archiver data sources.

System requirements

• A recent version of Linux, definitely 64 bit Linux for production systems. If using RedHat, we should aim for RedHat 6.1.
• JDK 1.16+ - definitely the 64 bit version for production systems. We need the JDK, not the JRE.
• A recent version of Tomcat 9.x; preferably apache-tomcat-9.0.20 or later.
• The management UI works best with a recent version of Firefox or Chrome.
• By default, the EPICS archiver appliance uses a bundled versions of the Java CA and PVA libraries from EPICS base.

• A recent version of MySQL mysql-5.1 or later if persisting configuration to a database. We hope to add Postgres support soon.

Storage

• PlainPBStoragePlugin This plugin serializes samples using Google's ProtocolBuffers and stores data in chunks. Each chunk has a well defined key and stores data for one PV for a well defined time duration (for example, a month). Using Java NIO2, one can store each chunk in
  1. A file per chunk resulting in a file per PV per time partition.
  2. A zip file entry in a .zip file per chunk resulting in a .zip file per PV.
  3. This can be extended to use other storage technologies for which a NIO2 provider is available (for example, Amazon S3, a database BLOB per chunk or a key/value pair per chunk in any key/value store).
  By default, the PlainPBStoragePlugin maps PV names to keys using a simple algorithm that relies on the presence of a good PV naming convention. To use your own mapping scheme, see the Key Mapping section in the customization guide.

Architecture

1. Ramdisk for the short term store - in this storage stage, we'd store data at a granularity of an hour.
2. SSD/SAS drives for the medium term store - in this storage stage, we'd store data at a granularity of a day.
3. A NAS/SAN for the long term store - in this storage stage, we'd store data at a granularity of a year.

The long term store is shown outside the appliance as an example of a commonly deployed configuration. There is no necessity for the appliances to share any storage; so both of these configurations are possible.

Policies

Clustering

While each appliance in a cluster is independent and self-contained, all members of a cluster are listed in a special configuration file (typically called appliances.xml) that is site-specific and identical across all appliances in the cluster. The appliances.xml is a simple XML file that contains the ports and URLs of the various webapps in that appliance. Each appliance has a dedicated TCP/IP endpoint called cluster_inetport for cluster operations like cluster membership etc.. One startup, the mgmt webapp uses the cluster_inetport of all the appliances in appliances.xml to discover other members of the cluster. This is done using TCP/IP only (no need for broadcast/multicast support).

The business processes are all cluster-aware; the bulk of the inter-appliance communication that happens as part of normal operation is accomplished using JSON/HTTP on the other URLs defined in appliances.xml. All the JSON/HTTP calls from the mgmt webapp are also available to you for use in scripting, see the section on scripting.

The archiving functionality is split across members of the cluster; that is, each PV that is being archived is being archived by one appliance in the cluster. However, both data retrieval and business requests can be dispatched to any random appliance in the cluster; the appliance has the functionality to route/proxy the request accordingly.

Scripting

#!/usr/bin/env python

import requests

resp = requests.get("http://archappl.slac.stanford.edu/mgmt/bpl/getAllPVs?pv=VPIO:IN20:111:VRA*")
print("\n".join(resp.json()))

EPICS 7

Screenshots