Process groups (Hostproc)

2.1 Process groups

The detailed process listing supplies plenty of information, but it’s hard to keep track of. What’s really useful for monitoring is the ability to aggregate information about similar processes into a process group. The rudimentary grouping capability is provided by snmpd itself via the proc configuration file statement, which counts running instances of the same process. However, the list of its shortcomings is pretty long. First of all, it uses the process argv[0] for grouping, which means that if a process modifies it, it won’t be counted. Secondly, only the number of process’ instances is aggregated, other metrics (such as memory usage, etc) is omitted. This list can be continued.

The grouping feature in hostproc aims to compensate for these deficiencies.

A process group keeps track of processes matching a certain pattern and aggregates the following information: number of instances, total CPU usage, total time scheduled, virtual memory and resident set size. The processes can be grouped by their argv[0] name, or their executable file name, or the full command line. Either exact or regular expression matching can be used.

Process groups are defined in the configuration file. The configuration file is named hostproc.conf and is located in the default Net-SNMP configuration path.

Let’s illustrate the concept taking as an example running httpd instances. To aggregate information about them, add the following to your hostproc.conf:

  group httpd

This simplest grouping declaration instructs hostproc to create a process group named httpd, which will gather statistics about processes with argv[0] value equal to ‘httpd’. The selection criterion which identifies a process as belonging to a particular group can be modified using three substatements that can appear after the group statement: field, pattern, and match.

The field statement defines the member of the processEntry object used for selection. It can have three values: comm, which stands for the argv[0] and is the default, exe, which means full pathname of the executable file, and cmdline, which means full command line with all arguments.

The pattern statement defines the string that the selected field must match in order for the process to be included in the group. If the statement is absent, the argument to the group statement is assumed as the default.

Finally, the match statement specifies how field and pattern are compared. Its possible values are: exact, basename, and regex (or rx). If it is exact, the two values are compared byte-to-byte. This is the default. basename means that first the base file name is extracted from pattern and field, then the obtained two strings are compared byte-to-byte. This matching method is useful in conjunction with field exe setting. The regex setting instructs the program to treat pattern as an extended regular expression. It is often useful together with field cmdline.

Writing out all the defaults expressly, the above statement is equivalent to:

  group httpd
    field comm
    pattern httpd
    match exact

As a more complex example, the following group will comprise all running processes whose executable file name ends with one of ‘httpd’, ‘lighttpd’, ‘apache’, ‘apache2’, or ‘nginx’:

  group httpd
    field exe
    pattern .*/(((lig)?(httpd))|apache2?|nginx)$
    match regex

One can also add arbitrary limits, e.g.:

  group httpd
    field exe
    pattern .*/(((lig)?(httpd))|apache2?|nginx)$
    match regex
    min 1
    max 50
    vsize 3000M

This statement requires that at least 1 and no more than 50 instances of http daemon be running, and that they occupy no more than 3000 megabytes of virtual memory.

See Process group definition, for a detailed discussion of the group and associated statements.

If hostproc is started with this configuration, in addition to the detailed process listing described above, the processGroupTable table will be displayed. In our example, it will look like:

processGroupName.0 = STRING: httpd
processGroupPattern.0 = STRING: .*/(((lig)?(httpd))|apache2?|nginx)$
processGroupField.0 = INTEGER: ProcessExe(1)
processGroupMatch.0 = INTEGER: regex(0)
processGroupOK.0 = INTEGER: true(1)
processGroupMinCount.0 = INTEGER: 1
processGroupMaxCount.0 = INTEGER: 50
processGroupMaxVSZ.0 = Gauge32: 3072000 kB
processGroupMaxRSS.0 = Gauge32: 0 kB
processGroupMaxCPU.0 = Gauge32: 0
processGroupCount.0 = INTEGER: 8
processGroupCPUMax.0 = Gauge32: 30
processGroupCPUAvg.0 = Gauge32: 19
processGroupCPUInst.0 = Gauge32: 25
processGroupVSZ.0 = Gauge32: 99356 kB
processGroupRSS.0 = Gauge32: 1788 kB

Each conceptual row describes a single process group and contains instances of the following objects:

processGroupName: Name of the group. This is the argument to the group statement in the configuration file.
processGroupPattern: Pattern that selects processes for this group (the value of the pattern statement).
processGroupField: The processEntry object used for selection, as set by the field statement. The possible values are: ProcessComm, ProcessExe, and ProcessCmdline.
processGroupMatch: Comparison algorithm used. It can be exact, basename, or regex (rx).
processGroupOK: Status of the process group. It is true, if all parameters fall within the imposed limits (if any), or false if at least one of them doesn’t.
processGroupMinCount: Minimum number of running instances.
processGroupMaxCount: Maximum number of instances. 0 stands for infinity.
processGroupMaxVSZ: Maximum value for virtual memory size (kB).
processGroupMaxRSS: Maximum value for resident set size (kB).
processGroupMaxCPU: Maximum CPU usage (percent).
processGroupCount: Actual number of process instances.
processGroupCPUMax: The maximum value of the instantaneous CPU usage among all processes in this group.
processGroupCPUAvg: Average CPU usage for the group.
processGroupCPUInst: Average instantaneous CPU usage for the group.
processGroupVSZ: Total virtual memory size.
processGroupRSS: Total resident set size.