The RTMC Project. (Real Time Modular Collection)

The RTMC project is a Linux system metrics collector that uses plugins to gather, process/aggregate, and send out system metrics information. RTMC's design is a light weight C++ metrics collection facility that is a multi-threaded plugin application to provide a simple API to allow greater versatility for developers and system administrators to gain direct access to metric data to fit their needs. The plugins allow tasks to be modular to perform specific actions; i.e., gathering a specific metric. The use of threading is to allow real time parallel processing to take place between plugins. No longer will developers or system adminsitrators be force to accept what limited interface metric tools provide. With RTMC, developers and system administrators can work directly with metric data collection to fit their needs and infrastructure.

The cluster wide utility is currently being worked on to meet the beta release. In order to get a properly working cluster utility up and running, I needed to re-vist the system calls sendto() and recvfrom(), to allow greater flexibility when sending large data packets over UDP, RDS, and TCP, or any other protocol for that matter. Little advancement in these calls has been made since the last I looked at those system calls in regards to message fragmentation and re-assembly. The code right now has the class Socket_Connector, which I'm currently developing. This will allow me to send and recieve very large messages. It will also allow me to send and recieve in parallel from multiple peers too. I'm almost done with it, I'm happy with the sending part, now onto the receiving part. Once done with this class I can continue writting the cluster wide utility. I thank you for your patience.

Here are some upcoming milestone tasks to achieve Beta release.

The RTMC application has been built from scratch and has no outside dependencies on third party applications or libraries, other than the shared libraries that were directly developed for RTMC and contained with in the RTMC project. The idea is to keep the code independent from outside influences to allow maximum freedom. With no outside influences the less there is to keep up with changing licenses or demands of other authors/distributers. Or worse, dependencies on tools and libraries that are no longer actively being developed or lost.

With that said, RTMC plugins can use any application or library the author sees fit. There are limitations or restrictions placed on RTMC plugins. Many of the RTMC plugins that are packaged with RTMC heavily use the Google Protocol Buffers library to store and share metric data. The need to pass metric data to different platforms and languages is crucile for the long term success of RTMC. XML is a traditional choice of passing such data, but it's very heavy weighted and a poor choice when encapsulating data. The need for parsing and constructing XML is what making XML a heavy weighted drawback. However if you find yourself in need of the data being represented as XML, a utility plugin can be written to transform Google Protocol Buffers into XML. The RTMC application itself does not use Google Protocol Buffers and it is not mandated by RTMC for plugins to use them. All the metric plugins that I will be writing will use this library as it has some very nice features in serializing and directly accessing data. Each plugin that uses a Google Protocol Buffer will contain a directory called proto and inside contain the .proto file for that plugin. You can then use that .proto file to generate the code to deserialize the messages from that plugin.

RTMC is based on the need for a way to collect system metrics to suite the needs of developers and system administrators. Most system metrics tools present data in flat console text, in flat files, or through a GUI where metric data is either slow to gather or to hard to get. A lot of effort is wasted when the metric data needs to be transported from one metric tool to another application that do not share the same data transport channel. This usually involves a developer to write a middleware solution in the form of a script or application to bridge both the metric collector and the application that consumed the data. RTMC does away with ad-hoc coding to bridge different connection points and provide a plugin interface, so one plugin can gather as much information from a single source as it sees fit in a fashion that is benificial for their needs.

RTMC is a C++ multi-threaded plugin application. Where each plugin is a separate thread that RTMC manages. The plugin does not have to worry about any thread issues, unless the plugin spawns its own threads. When RTMC loads a plugin, it starts the plugin as a seperate thread and then manages the thread; which is a plugin. The reason RTMC was written in C++ was for for the speed and control over threads. The ability to control and perform tasks in parallel as threads allows for faster processing. The reason for plugins was to allow for greater flexibility for users to use or create plugins that fit their needs.

RTMC has a class that manages the individual plugins that it loads. It is this class that periodically checks for and loads new plugins from the .plugins directory from where the RTMC process was started. When a plugin is loaded, the plugin manager first checks to see if the proper function call has been defined in the plugin. Once the plugin is seen as valid, a shared class is created for the plugin and the plugin manager to use together. Then plugin manager starts the plugin as a POSIX thread and the shared class is passed to the newly started plugin.

Once a plugin has started it uses the shared class to interact with the plugin manager and with other plugins. It's up to the individual plugins on how they use the shared class. Generally a plugin will setup a few method calls and then get right to work gathering metric data, aggregating metric data, or sending metric data to somewhere else.

There are three types of RTMC plugins that RTMC recognizes; e.g., metric, utility, and producer. Metric plugins gather just metrics from the system. A plugin should only be of one type, but not mandated to be.