How to Monitor Kafka with AppDynamics

This article is about the monitoring of a Kafka installation with AppDynamics. I will look in to the instrumentation with the JavaAgent 4.5 and MachineAgent 4.5. All this on a local Notebook installation. Maybe I will look in to an Opensift setup later.

A good source for Kafka knowledge for the beginners can be found here: http://cloudurable.com/blog/kafka-tutorial-kafka-from-command-line/index.html

What we need:
1. Kafka & Zookeeper
2. AppDynamics Controler (SAAS)
3. AppDynamics Java Agent and Machine Agent

Kafka was downloaded from https://kafka.apache.org/downloads. At the time of writing 2.3.0 was the latest version and used for this article. I installed it according to the tutorial of Clousurable mentioned above.

For AppDynamics I have registered the 15 days evaluation license. From my newly created account I downloaded the Machine Agent and Java Agent 4.5.X. Both where unziped to the home folder of my Fedora 29.

Configuration

1. Kafka

To enable JMX I have added this line export JMX_PORT=9988 just above the last line in kafka-server-start.sh located in the Kafka bin folder.

To support the AppDynamics Java Agent I have added $APPD_OPTS to the section #Launch mode of the shell script kafka-run.calss.sh located in the bin folder of Kafka. So it looks like this:

exec $JAVA $APPD_OPTS $KAFKA_HEAP_OPTS $KAFK_.........

Location is important. It must be the first argument after $JAVA.

2. AppDynamics

For AppDynamics we have two sections. The Java Agent and the Machine Agent. Let us start with the Java Agent.

2.1 AppDynamics Java Agent

I have unpacked the ZIP  AppServerAgent-4.5.15.27926.zip to /home/juerg/AppDynamics_JavaAgenet. Configuration of the agent takes place in /conf/controler-info.xml. Here the controler-info.xml. I used. Beside the conroler-info.xml. The someof the fields I passed via -D parameter in $APPD_OPTS to Kafka. I placed this in ~/.bash_profile. The export looks like this:

export APPD_OPTS=" -javaagent:/home/juerg/AppDynamics_JavaAgenet/javaagent.jar -Dappdynamics.agent.logs.dir=/tmp/logs -Dappdynamics.agent.uniqueHostId=$HOSTNAME -Dappdynamics.agent.reuse.nodeName=true -Dappdynamics.jvm.shutdown.mark.node.as.histical=true -Dappdynamics.agent.reuse.nodeName.prefix=Kafka-Broker- -Dappdynamics.agent.tierName=Kafka-Broker -Dappdynamics.agent.log4j2.disabled=true"

Here some Screenshots from AppDynamics UI:

2.2 Machine Agent

Like the Java Agent I unziped the Machine Agent to my home folder. To start it as a stand alone agent I have created a simple shell script:

agent-start.sh
#!/bin/bash
./bin/machine-agent -p /tmp/pidfile -Dappdynamics.agent.uniqueHostId=$HOSTNAME -Dappdynamics.agent.reuse.nodeName=true -D appdynamics.jvm.shutdown.mark.node.as.histical=true -Dappdynamics.agent.reuse.nodeName.prefix=Kafka-Broker- -Dappdynamics.agent.tierName=Kafka-Broker -Dappdynamics.agent.log4j2.disabled=true -Dappdynamics.agent.maxMetrics=5000

As you can see the tier and node name are the same in Machine and Java agent. This will in AppDynamics UI then been aggregated in the same place.

Here some Screenshots from AppDynamics UI: