xStats Quick Start Guide - SevOne NMS 5.7

Create an xStats Collection Source

Perform the following steps to create an xStats Collection Source.

1. From the navigation bar, click Administration and select Monitoring Configuration, then xStats Source Manager.
   

2. Click Add Source at the top of the page to display the Add Source pop-up.

3. In the Name field, enter the xStats Collection Source name. This is how the name will appear within the SevOne NMS xStats Source Manager and has no bearing on other settings.

4. Click the Peer drop-down and select the peer to monitor data from the xStats Collection Source.

   Each xStats Collection Source has a different load impact on the appliance. If you're working with a larger cluster and you have multiple xStats Collection Sources, it may be beneficial to distribute them among different peers in SevOne NMS.

5. In the IP Address field, enter the IP address of the device that the xStats Collection Source will monitor data from. If you're using the local system, you can enter either of two standard options: 0.0.0.1 or 127.0.0.1. If you're using SFTP, enter the IP address where the files to be monitored reside.

6. Click the Source Type drop-down. You should see the name of your xStats adapter listed twice. One occurrence will be followed by localfilesystem, and the other is followed by sftp. Select the collection method for the xStats Collection Source to use.

7. In the Retrieval Frequency (sec) field, enter the number of seconds indicating how often the xStats Collection Source should attempt to receive new data. The retrieval duration for each xStats Collection Source can be set to any value, but it's most commonly set to 300 seconds (5 minutes).

   This setting applies to the sftp source type. While it doesn't apply to the localfilesystem source type, the field can't be left empty, so make sure to enter a number here regardless of which source type you chose in the previous step.

8. Click the Device Creation drop-down and select one of the following options.

   • Manual - lets you manually add devices to SevOne NMS after they've been discovered by the xStats Collection Source. You can also link devices discovered by the xStats Collection Source to devices that already exist in SevOne NMS.

   • Automatic - automatically adds devices discovered by the xStats Collection Source. These devices have the xStats plugin enabled by default.

9. Select the Unique Files Only check box to have the xStats Collection Source collect only files that haven't previously been collected. Leave it clear to collect all available files from the devices, including previously collected files, and to re-process all previous data from those files.

10. In the Username field, enter the user name SevOne NMS needs to authenticate onto the device that the source data is monitored from. You'll need this if you're using SFTP.

11. In the Password field, enter the password SevOne NMS needs to authenticate onto the device. You'll need this for SFTP, too.

12. In the Override Retrieval Directory field, enter the full path to the directory on the SevOne NMS appliance where you prefer to store data. If you leave this field blank, a default directory based on the current SevOne NMS appliance will be assigned.

13. Click Apply to save changes.

Edit an xStats Collection Source

Perform the following steps to edit an xStats Collection Source.

1. From the navigation bar, click Administration and select Monitoring Configuration, then xStats Source Manager.
   

2. Select the check box for the xStats Collection Source that you would like to edit and click to display the Edit Source pop-up.

   If you would like to add a new xStats Collection Source, click Add Source at the top of the page to display the Add Source pop-up, where you can create a new xStats Collection Source using the steps below.

   

3. In the Name field, enter the xStats Collection Source name. This is how the name will appear within the SevOne NMS xStats Source Manager and has no bearing on other settings.

4. Click the Peer drop-down and select the peer to monitor data from the xStats Collection Source.

   Each xStats Collection Source has a different load impact on the appliance. If you're working with a larger cluster and you have multiple xStats Collection Sources, it may be beneficial to distribute them among different peers in SevOne NMS.

5. In the IP Address field, enter the IP address of the device that the xStats Collection Source will monitor data from. If you're using the local system, you can enter either of two standard options: 0.0.0.1 or 127.0.0.1. If you're using SFTP, enter the IP address where the files to be monitored reside.

6. Click the Source Type drop-down. You should see the name of your xStats adapter listed twice. One occurrence will be followed by localfilesystem, and the other is followed by sftp. Select the collection method for the xStats Collection Source to use.

7. In the Retrieval Frequency (sec) field, enter the number of seconds indicating how often the xStats Collection Source should attempt to receive new data. The retrieval duration for each xStats Collection Source can be set to any value, but it's most commonly set to 300 seconds (5 minutes).

   This setting applies to the sftp source type. While it doesn't apply to the localfilesystem source type, the field can't be left empty, so make sure to enter a number here regardless of which source type you chose in the previous step.

8. Click the Device Creation drop-down and select one of the following options.

   • Manual - lets you manually add devices to SevOne NMS after they've been discovered by the xStats Collection Source. You can also link devices discovered by the xStats Collection Source to devices that already exist in SevOne NMS.

   • Automatic - automatically adds devices discovered by the xStats Collection Source. These devices have the xStats plugin enabled by default.

9. Select the Unique Files Only check box to have the xStats Collection Source collect only files that haven't previously been collected. Leave it clear to collect all available files from the devices, including previously collected files, and to re-process all previous data from those files.

10. In the Username field, enter the user name SevOne NMS needs to authenticate onto the device that the source data is monitored from. You'll need this if you're using SFTP.

11. In the Password field, enter the password SevOne NMS needs to authenticate onto the device. You'll need this for SFTP, too.

12. In the Override Retrieval Directory field, enter the full path to the directory on the SevOne NMS appliance where you prefer to store data. If you leave this field blank, a default directory based on the current SevOne NMS appliance will be assigned.

13. Click Apply to save changes.

Manage xStats Devices Created from an xStats Collection Source

The xStats data that comes from an xStats Collection Source creates devices. The xStats Source Manager includes a Devices column, which shows how many devices have been created from the xStats Collection Source. It also shows the number of new devices found by the latest poll. For example, in the screenshot below, the Devices column displays 25 (0 new) for the xStats Collection Source OpenStackCeilometerTransform_1. The first number refers to the number of devices created from the xStats Collection Source. The number within parentheses displays the number of new devices from the latest poll.

1. On the xStats Source Manager page, select an xStats Collection Source.
   

2. Click the link under Devices to display the Manage Devices pop-up.
   
   

3. The following information appears for devices in the list.

   • Watched - indicates that a device is being watched by the xStats plugin. indicates that a device is being ignored by the xStats plugin.

   • Name From Source - displays the name of the device as discovered by the xStats Collection Source.

   • IP Address - displays the IP address of the device.

   • First Seen - displays the date and time the device first appeared from the xStats Collection Source.

   • Name in SevOne - displays the name of the device as it appears in the Device Manager.

4. Select a device to perform one or more of the following actions under Actions.

   • - click to display the Edit Device page, where you can edit the device.

   • - click to display a link to the Device Summary and links to the report templates that are applicable for the device.

   • - click to link the new device to a device that's already in SevOne NMS. This option is available when you configure an xStats Collection Source to require manual device creation. It's useful when multiple xStats Collection Sources find the same xStats device.

   • - click to add the device as a new device. This option is available when you configure an xStats Collection Source to require manual device creation.

5. At the top of the pop-up, the following options are available.

   • Watch Selected - click to have the xStats plugin watch the device.

   • Ignore Selected - click to have the xStats plugin ignore the device.

6. Click Save to apply changes.

Remove an xStats Collection Source

When you remove an xStats Collection Source, the devices, objects, and indicators that were created from it aren't affected. Perform the following steps to delete an xStats Collection Source.

1. SSH into your PAS as root user.

2. Enter the following command to delete the cron file for your adapter.

   $ rm /etc/cron.d/<adapter cron file>
   Example

   $ rm /etc/cron.d/xstats-OpenStackCeilometerTransform

   For most adapters, there will be only one cron file to delete, as in the example above. In the case of this particular adapter (OpenStack Ceilometer), you would also need to delete a custom cron file using the following command:

   $ rm /etc/cron.d/xstats-OpenStackCeilometerTransformCustom

3. In SevOne NMS, delete the xStats Collection Source. From the navigation bar, go to Administration -> Monitoring Configuration -> xStats Source Manager. Select the xStats Collection Source and click under Actions to delete it.

   A couple of steps down, you'll need to delete the devices for this xStats Collection Source. To view those devices before deleting the xStats Collection Source, locate the xStats Collection Source in the xStats Source Manager and click the link under Devices. This will display a list of the devices that will need to be deleted.
   
   

4. Return to your SSH client and enter the following command to delete log files.

   $ rm -r /var/log/xstats/<adapter package name>/
   Example

   $ rm -r /var/log/xstats/OpenStackCeilometerTransform/

5. In SevOne NMS, perform the following actions to delete the object types.

   1. From the navigation bar, go to Administration -> Monitoring Configuration -> Object Types.

   2. On the right, under Object Types, click the Filter drop-down and select xStats.

      

   3. Select each Object Type for the xStats Collection Source and click under Actions to delete it.

      Before deleting an Object Type, verify that it does not affect other xStats sources. Otherwise, data loss can occur once that Object Type is removed.

6. Perform the following actions to delete the devices in SevOne NMS.

   1. From the navigation bar, go to Devices -> Device Manager.

   2. Select the check box for each device that you want to delete for the xStats Collection Source.

      

   3. Click and select Tools, then Delete selected to delete the devices.

7. Return to your SSH client. Enter the following command to delete the xStats adapter directory.

   $ rm -r /opt/sevone-xstats/<adapter package name>/
   Example

   $ rm -r /opt/sevone-xstats/OpenStackCeilometerTransform/

   This final step is a destructive process that removes the adapter from your NMS. Once it has been executed it cannot be undone.

xStats Atomic Indicator Types

Perform the following steps on the Object Types page to add and edit xStats atomic indicator types.

1. In the Object Types hierarchy on the left, select the object type that you would like to create an indicator type for.

2. On the right side of the page, under Indicator Types, click Add Atomic Indicator Type to display the Add xStats Indicator Type pop-up. To edit an existing indicator type, select the indicator type and click under Actions.
   
   

3. In the Indicator Name field, enter the name of the indicator type.

4. In the Description field, enter the name to display.

5. Click the Indicator Type drop-down and select one of the following options:

   • GAUGE - for indicators that have specific values when polled.
     

   • COUNTER32 - for 32-bit indicators that continue to increment. If you select this option, you can select the Has Precalculated Deltas check box to total the delta/differences between polls. This provides the ability to graph things like the number of errors in a day, for example.

   • COUNTER64 - for 64-bit indicators that continue to increment. If you select this option, you can select the Has Precalculated Deltas check box.

6. Click the Measure As drop-down and select a data unit.

7. Click the Display As drop-down and select a display unit.

8. Select the Maximum Value check box to indicate that the indicator type has a maximum value.

9. In the Field Identifiers field, enter the indicator type field identifiers.

10. Select the Default allowed for new devices check box to have the plugin poll this indicator type by default when the object type is enabled and when the plugin for a device is enabled.

11. Select the Ignore check box if you want to prevent the xStats plugin from processing data for the indicator type.

12. In the Note field, enter any additional information you would like to include.

13. Click Save As New.

xStats Synthetic Indicator Types

Using synthetic indicator types, you can create your own key performance indicators (KPIs) even when those KPIs–such as Percent Usage, Percent Loss, Percent Error, and Percent Idle–don't exist on a device. For example, let's say that you want to monitor voice gateways to find out which primary rate interface (PRI) is getting the most usage. Typical poll metrics can tell you the busy status of individual bearer channels, or B channels, but they can't tell you the sum of the statuses for all the B channels. This makes it difficult to find out the total usage of a particular PRI.

In SevOne NMS, you can create a single indicator type that tells you what percentage of a PRI is being used.

First, we need the following:

• An existing indicator type that tells us how many busy B channels there are for a given PRI. Let's call this BChannelsBusy.

• The total number of B channels for our PRI. That number is 23.

Using this information, we can create a synthetic indicator type with an expression to perform the following calculation:

(BChannelsBusy*23)/100

Now we have a new indicator type–a synthetic indicator type–that tells us what percentage of the PRI is being used. There's no need to look up the number of busy B channels and perform calculations manually.

Perform the following steps on the Object Types page to add and edit xStats synthetic indicator types.

1. In the Object Types hierarchy on the left, click on an object type to display its indicator types on the right. If the object type doesn't have any indicator types, the Add Synthetic Indicator Type button won't appear.

   

2. Click Add Synthetic Indicator Type to display the Add Synthetic Indicator Type pop-up. To edit an existing synthetic indicator type, select the indicator type and click under Actions.
   

3. In the Indicator Name field, enter the name of the synthetic indicator type.

4. In the Description field, enter the name to display.

   You'll see the indicator description displayed in reports and elsewhere in the UI. Keep this in mind when creating the indicator type description. For example, when you look at an object's indicators, you'll notice that they're listed based on the first word of their description. If you have the two indicators Queued Bytes and Transmitted Bytes, they may not appear together. However, if you use the descriptions Bytes Queued and Bytes Transmitted, the indicators will appear in close proximity to each other.

5. The Synthetic Indicator Expression field is where you define the calculation. Perform the following actions to create the expression for this field.

   A red border around the Synthetic Indicator Expression field indicates that your calculation is invalid. This also means that your graph results will be incorrect.

   1. In the Available Source Indicators field on the right, select an indicator type to use in the expression. Drag and drop it into the Synthetic Indicator Expression field.

      The Available Source Indicators field contains the indicator types for the object type that you selected a few steps ago.

   2. Next, enter the applicable operators to formulate your calculation in the Synthetic Indicator Expression field. For a list of available operators, see Acceptable Operators below.

   3. Drag any additional indicator types from the Available Source Indicators field and enter additional mathematical symbols to complete the expression in the Synthetic Indicator Expression field.

6. The Maximum Value Expression field lets you define a maximum value calculation for the indicator type. Perform the following actions to create the expression for this field.

   1. In the Available Source Indicators field on the right, select an indicator type to use in the expression. Drag and drop it into the Maximum Value Expression field.

   2. Next, enter the applicable operators to formulate your calculation in the Maximum Value Expression field. For a list of available operators, see Acceptable Operators below.

   3. Drag any additional indicator types from the Available Source Indicators field and enter additional mathematical symbols to complete the expression in the Maximum Value Expression field.

7. Click the Measure As drop-down and select a data unit.

8. Click the Display As drop-down and select a display unit.

9. Select the Default allowed for new devices check box to have the plugin poll this indicator type by default when the object type is enabled and when the plugin for a device is enabled.

10. In the Note field, enter any additional information you would like to include.

11. Click Save As New.

Acceptable Operators

Your expression formula can include the following characters:

• + add

• - subtract

• * multiply

• / divide

• && logical AND

• || logical OR

• <= less than or equal to

• >= greater than or equal to

• ! not equal to

• == equal to

• > greater than

• < less than

• ^ raise x to the power of y

• % modulus

• ?: if...then...else

• isnan is Not a Number. This evaluates to 1 if the value is not a number. Otherwise, it evaluates to 0.

• isValid is valid. This evaluates to 1 if the value has been discovered and is not isnan. Otherwise, it evaluates to 0.

• useIfValid use if valid. This evaluates to the value if it has been discovered and is not isnan. It evaluates to the second argument otherwise.

If your calculation results in either of the following invalid values, there will be a gap in your graph:

• Not a Number (NAN)

• Infinity (+/-INF).

The following describes how SevOne NMS attempts to prevent invalid values. These are listed in the order that they're processed.

• Zero divided by zero results in NAN.

• Any positive value divided by zero results in +INF.

• Any negative value divided by zero results in -INF.

• Zero multiplied by +/-INF results in NAN.

• Any value added to, subtracted from, multiplied by, divided by, or divided from NAN results in NAN.

• Any value compared to NAN (<, <=, ==, >=, >) results in 0. NAN != NAN.

• Any value compared to +INF is less than +INF, except that +INF == +INF

• Any value compared to -INF is greater than -INF, except that -INF == -INF

• Any value added to or subtracted from +INF results in +INF

• Any positive value multiplied by +/-INF results in +/-INF

• Any value divided by +/-INF results in 0

Check the Release Notes for any Known Issues related to xStats

If there are hotfixes for an xStats related daemon (SevOne-fcad, SevOne-dispatchd, SevOne-ingestion-resolved, or SevOne-ocd) on the version experiencing issues, install the hotfixes before proceeding further with this guide.

Is fcad retrieving files?

On the xStats Source Manager page, check that the Last Retrieval Time field is updating whenever new files are created in the retrieval directory.

Check the Logs

The first troubleshooting step is to check the logs for the various xStats daemons.

Common errors and their solutions

Is Kafka Running?

Kafka is the message bus introduced in Sevone NMS 5.6.0, used to pass messages through the pipeline without using the filesystem. Kafka also requires a daemon called Zookeeper to run in order to work properly. If both of these are not running, then you will not receive messages. To test if Kafka is running, execute one of the following commands.

Execute one of the following commands to verify if Zookeeper is running.

When successfully running, /etc/init.d/kafka start and /etc/init.d/zookeeper start show processes running Java with a significant number of options and commands.

Are the Kafka SSL keys/certs correct?

Execute the following command to verify if the SSL certs are consistent across the cluster.

If there is a problem, you can rerun SevOne-act kafka GenCerts. This overwrites your existing certs with new ones and deploys them across the cluster. Alternatively, you could deploy the keys manually by copying the contents of /etc/kafka/ from the master to any troublesome peers.

Check the database

The logs are not always clear about what sort of problem you are seeing. In this case, you can instead use the database to narrow down how far along the pipeline you have gotten. xStats includes a number of tables which provide information about how far along the xStats pipeline a particular file has gotten.

1. local.bulkdata_fetched_files contains all files that SevOne-fcad has downloaded into its own directories. The filename is stored in the name column.

2. The source type and file processor must be linked correctly between the bulkdata_sources, bulkdata_source_types, and bulkdata_source_type_file_processor table.

Check the Kafka topics

Execute the following command to check how many topics are running on the box.

This will list all the topics and the number of partitions. This script communicates with Zookeeper so if it returns any error, Zookeeper may be in a trouble state.

Kafka stores its messages on a variety of topics. It sends messages to and receive messages from these different topics. The distribution is as follows:

• SevOne-fcad sends messages to the dispatch topic.

• SevOne-dispatch receives messages from the dispatch topic, and sends messages to the ingest topic.

• SevOne-ingestion-resolved receives messages from the ingest topic and optionally sends messages directly to SevOne-datad over ZMQ if it is shortterm data, or otherwise sends messages to the backfill topic.

You can check if data is going to a topic by tailing it's log file in /var/kafka/<topic name>-0. For example, the dispatch topic log is in /var/kafka/dispatch-0/ and the ingestion-resolved topic log is in /var/kafka/ingest-0/. If these directories are not present, that means that data has never been sent to these topics. Directories are thus created as needed. There may be several *.log files in this directory because the last GB of data in each topic gets stored. ls -l helps you locate the most recently edited log file. With a tail -f on that file, you should see more data populated when messages are sent to that topic. The output will be in binary form just by tailing this log, but there are tools to help with this. SevOne NMS 5.7+ (and a later 5.6 service pack) uses the SevOne-kafka-inspector tool and all versions use the old SevOne-act kafka read script (which SevOne-kafka-inspector replaces).

Execute the following command to see the help for the Kafka inspector. This includes details on how to select topics, partitions, offsets, the format to use to decode the data, the ability to tail a topic and partition (automatically printing out any new messages), and more.

Alternatively, you can execute a SevOne-act read script to read from a topic.

For a full list of options, run SevOne-act kafka read --help. This available options are as follows.

Check the messages

Sometimes you may need to verify the current status of the the xStats daemons. In this case, you can check the following topics for information about the following daemons.

You can also read from these topics by entering the following command.

The metrics displayed are logged by default every 5 minutes, and they indicate the time since the previously displayed metrics.

These messages also display when running the daemons verbosely. There are a number of different messages as well as a count, sum, minimum, maximum, standard deviation and mean value for all of them.

Additional strategies for debugging Kafka topics

Verify Source Files / Parser

To verify the validity of the source files or the parser or the configuration, execute the following command to see the options available. This command is useful for troubleshooting as it can show you what was emitted, the device and object it should belong to, timestamp, etc. On failure it can also provide the last entries in the parser log for the source.