Redirecting Spark logs to Kafka

Spark logs can be redirected to Kafka which can be further used for analysis. These logs can be fed from kafka to elastic search and further to kibana to get data and resource related insights w.r.t our job.

In this article, we will be discussing about how to stream log4j application logs to Apache Kafka using maven artifact kafka-log4j-appender.

We need to add kafka-log4j-appender as a dependency in pom.xml.

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-log4j-appender</artifactId>
    <version>1.0.0</version>
</dependency>

Additionally, we need to add kafka properties in log4j.properties file for using KafkaLog4jAppender. Kafka broker list and topic need to specified. This topic will be used for storing logs.

# Root logger option
log4j.rootLogger=INFO, stdout, kafka 
log4j.logger.kafka=WARN 
log4j.logger.org.apache.kafka=WARN 
 
# Redirect log messages to console 
log4j.appender.stdout=org.apache.log4j.ConsoleAppender 
log4j.appender.stdout.Target=System.out 
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout 
log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n 
 
log4j.appender.kafka=org.apache.kafka.log4jappender.KafkaLog4jAppender
log4j.appender.kafka.brokerList=localhost:9093 
log4j.appender.kafka.topic=Kafkalogs 
log4j.appender.kafka.layout=org.apache.log4j.PatternLayout 
log4j.appender.kafka.layout.ConversionPattern=[%d] %p %m (%c)%n 
log4j.appender.kafka.level=INFO

This log4j.properties file can be used to redirect logs of any application to kafka(not just spark).

Logs stored in kafka can then be processed by elastic search and displayed on kibana to get log level insights like number of errors, type of errors etc.

Code can be found on the link below:-
https://github.com/parkar-12/spark-examples/tree/master/LoggingToKafka

Originally published at https://parkar12.blogspot.com on February 19, 2021.