SIEM, Vulnerability Scanning, Server Monitoring and Compliance Training for IT Professionals
Table of Contents

Application and Text Log Consolidation Template

Application and text log consolidation is the process of saving text-based log file entries to a Data Provider, also known as a Log Database. Server Manager supports text log files, CSV files and W3C files (e.g. IIS logs).

Text logs can be consolidated using several API technologies.

APIDescription
Windows SharesMonitor text-based logs on Windows servers.
SFTP/SSHMonitor text-based files on any Unix flavor.
FTP/SMonitor text-based logs on any remote endpoint such as a hosted webserver.

How to Configure Text Log Consolidation:

  • From the Menu Bar, select File | New. The Create New Object View displays.
  • From the Create New Object view, expand Template | Log Management then select Log Consolidation. The New Log Consolidation Template Properties view displays.
  • From the New Log Consolidation Template Properties view, use the Sub type drop-down to select Text Log. The Template Properties view now contains 8 tabs.

Read Options

The Read Options tab enables you to configure how to read the log file.

  • Use the Log drop-down to select the target log or search criteria to configure. Select (All) to configure all files and search criteria at the same time.
  • Use the Logical filename text box when monitoring files that contain either dates or instance numbers within the log file name. When set, each unique file is saved to the same database table, otherwise each dated file or instance file will be added to its own database table.
Note
Use the {HOST} variable to insert the target hostname or IP in the table name.
Use the {ALIAS} variable to insert the configured host's alias in the table name.
  • Use the Use the filename or logical filename check box to save log entries to a database table that is named using either the filename or logical filename, if set.
Note
This option enables users to save log entries from multiple hosts to a single database table.
  • Use the Enable entry pattern recognition check box to configure a pattern that defines each unique log entry.
OptionDescription
Entries start withSpecify a unique character such as <.
Entries start with date or timeSpecify a date or time mask such as yyyy-MM-dd.
Entries end withSpecify a unique character such as >.
Entries end with (No CR, LF or CRLF)Specify a null terminating character such as |.
  • Use the Read method drop-down to select where to begin reading the file.
OptionDescription
Beginning of FileEach time a file is newly discovered, the entire file is read. The next time the file is read, it is read from the last known position.
End of FileEach time a file is newly discovered, the file is opened then the read position moved to the end and closed. The next time the file is read, it is read from the last known position.
Read AllEach a time a file is read, the entire file is read.
  • use the Open as Unicode option if the file is written in Unicode format otherwise the file is opened as UTF-8.
  • To test the entry pattern, use the Test host, device or endpoint drop-down to select the target host then click the Test button. The software applies your rules to the file then reads the first 10 entries. Update your entry pattern rules until you get the results you desire.
  • If you have specified to search using date search criteria, use the Read yesterday's file checkbox to target files with yesterday's date.
  • Use the Target files with a modified date and time span controls to target files that have a modified date and time that is either greater than or less than the current date and time minus the specified time span.

Options

The Rules tab enables you to configure the consolidation filter and Log Entry Retention Policy to enforce.

  • Use the Log drop-down to select the target log or search criteria to configure. Select (All) to configure all files and search criteria at the same time.
  • Optionally assign a Consolidation filter to dump entries you do not want saved to the Log Database. When assigned, only entries that pass the assigned consolidation filter are saved to the Log Database.
  • Use the Log Entry Retention Policy drop-down to select the retention policy. The retention policy is another template that defines the number of days to retain in the Primary and Archive Log Databases, for example, archive entries older than 30 days and retain entries for 150 days for a total of 180 days. Assign multiple retention policies to remove entries that match filter criteria defined in each retention policy. For more information see: Log Entry Retention Policy Template

Columns

In addition to defining regular expressions to extract attribute value pairs, several built-in variables are supported.

VariableDescription
_HOST_The hostname the log file resides.
_ALIAS_The configured host's alias.
_DATE_The date the log entry was read. The following date types are supported:
  • DateTime: Saves the date with a time of 00:00:00.
  • String: Saves the date using the following format yyyyMMdd.
_FILENAME_The name of the file being consolidated.
_PATH_The full path and filename being consolidated.

Monitor

The Monitor tab enables you to configure various monitors to apply when consolidating the log entries.

  • Use the Log drop-down to select the target log or search criteria to configure. Select (All) to configure all files and search criteria at the same time.
  • Use the File size exceeds controls to monitor the file size (e.g. The file exceeds 10 GBs).
  • Use the File size changes by controls to monitor changes in file size between scans (e.g. The file increases by 1 MB between scans indicating an ongoing attack).
  • Use the File has not been written to for controls to receive notifications when a file has been idle for a period of time (e.g. The file has not been written to indicating a webserver is down).
  • Use the Rules controls to configure log entry monitor rules. For more information see: Log Monitor

Related Topics

Data Provider

Log Consolidation Templates

Log Entry Retention Policy Template

Data Providers

Log Monitor