Icon

Log Parser and Usage Reporting

KNIME Server Log Parsing
KNIME Server Log ParsingThis workflow parses logfiles created during the operation of KNIME Server. The workflow was developed usingKNIME Server 4.6.2, based on Apache TomEE 7.0.3. The workflow is known to work on the KNIME Server 4.7.xrelease line using Apache TomEE 7.0.3 or 7.0.4.The workflow was designed to be run on the KNIME WebPortal.Please note that the structure of logfiles created by earlier or later versions of KNIME Server might not be compatiblewith this workflow.The purpose of this workflow is to read the logfiles created during Server operation and transform them to a formatthat is both easier to read for humans and easier to analyze for machines. The workflow extracts the most relevantevents that are logged. Reading the Logs and Data PreparationThe initial data preparation consists of six steps:1. Select sourceSelect from which source logfiles should be read. Possible options are:- KNIME Server REST API (requires admin privileges)- Directory on a local filesystem.2. Path to logsOption REST API requires entering the necessary credentials as well as the URL of the KNIME Server REST API. Reading from a directoryrequires entering the path to the directory.3. Select date rangeThe KNIME Server creates a new logfile every day. Since a large number of log data is accumulated over time, this metanode provides theopportunity to specify a start and end date in order to only read data within the selected timeframe.4. Read logsThis metanode loops over the individual logfiles and collects the results in a single table.5. Filter relevant eventsThis workflow focuses on a number of selected events related to the execution of workflows on the KNIME Server. Other events, such asevents related to Executor startup or configuration are therefore filtered out. To also consider additional events, please add them in the TableCreator node6. PreprocessingAt this stage, all data is stored in a single string column, with one row for each logged event. For further processing, this string is split intoseparate columns - date, event type, log level, source, and the actual log message. Parse Logfiles and Create Visualizations AdminNon-adminCompare execution:Client/API vs.WebPortal Filter relevantevents Preprocessing Select date range Read logs Get logfilesvia REST API CASE SwitchData (End) Select source Path to logs Login Upload Workflow Execute Job Preprocessingweekly aggregation Weekly totals Weekly totals Weekly totals WebPortal execute Weekly totals Preprocessingweekly aggregation Concatenate Preprocessingweekly aggregation Weekly totals Preprocessingweekly aggregation Preprocessingweekly aggregation KNIME Server Log ParsingThis workflow parses logfiles created during the operation of KNIME Server. The workflow was developed usingKNIME Server 4.6.2, based on Apache TomEE 7.0.3. The workflow is known to work on the KNIME Server 4.7.xrelease line using Apache TomEE 7.0.3 or 7.0.4.The workflow was designed to be run on the KNIME WebPortal.Please note that the structure of logfiles created by earlier or later versions of KNIME Server might not be compatiblewith this workflow.The purpose of this workflow is to read the logfiles created during Server operation and transform them to a formatthat is both easier to read for humans and easier to analyze for machines. The workflow extracts the most relevantevents that are logged. Reading the Logs and Data PreparationThe initial data preparation consists of six steps:1. Select sourceSelect from which source logfiles should be read. Possible options are:- KNIME Server REST API (requires admin privileges)- Directory on a local filesystem.2. Path to logsOption REST API requires entering the necessary credentials as well as the URL of the KNIME Server REST API. Reading from a directoryrequires entering the path to the directory.3. Select date rangeThe KNIME Server creates a new logfile every day. Since a large number of log data is accumulated over time, this metanode provides theopportunity to specify a start and end date in order to only read data within the selected timeframe.4. Read logsThis metanode loops over the individual logfiles and collects the results in a single table.5. Filter relevant eventsThis workflow focuses on a number of selected events related to the execution of workflows on the KNIME Server. Other events, such asevents related to Executor startup or configuration are therefore filtered out. To also consider additional events, please add them in the TableCreator node6. PreprocessingAt this stage, all data is stored in a single string column, with one row for each logged event. For further processing, this string is split intoseparate columns - date, event type, log level, source, and the actual log message. Parse Logfiles and Create Visualizations AdminNon-adminCompare execution:Client/API vs.WebPortal Filter relevantevents Preprocessing Select date range Read logs Get logfilesvia REST API CASE SwitchData (End) Select source Path to logs Login Upload Workflow Execute Job Preprocessingweekly aggregation Weekly totals Weekly totals Weekly totals WebPortal execute Weekly totals Preprocessingweekly aggregation Concatenate Preprocessingweekly aggregation Weekly totals Preprocessingweekly aggregation Preprocessingweekly aggregation

Nodes

Extensions

Links

Download

To use this workflow in KNIME, download it from the below URL and open it in KNIME:

Download Workflow
Created by: rb
Created at: 2017-02-06
On NodePit since: 2020-03-09
Last update: 2024-04-24
Created with KNIME version: v3.6.0
Tags: RESTAPIKNIMEServerLogsLogfilesAnalysis

Deploy, schedule, execute, and monitor your KNIME workflows locally, in the cloud or on-premises – with our brand new NodePit Runner.

Try NodePit Runner!