Skip to content

Kafka

Kafka

Manage your brokers, consumers, producers, troubleshoot topics/queues

Description

Kafka integration gives you ability to manage your kafka cluster using maira commands. You can also troubleshoot issues by sending or receiving messages on specific topics, or by checking producer/consumer stats. You can also configure your brokers with this integration.

Installation

For integrating with kafka cluster, you need to provide the list of brokers and authentication information. Maira supports mTLS and SASL authentication methods. You can manage multiple clusters from Maira.

To monitor Kafka broker metrics, you need to expose the JMX port while instantiating the broker and provide the authentication information for the JMX Port. Each broker must have its own JMX address in the same sequence as the brokers. Further, if any authentication is added for remote jmx, path to the configuration files need to be provided

Helm Config

Below is sample configuration for enabling kafka integration through Helm Chart Values file

kafka:
  disabled: false
  clusters:
    default:
      broker_addresses:
        - localhost:9093
        - localhost:9094
        - localhost:9095
      jmx_addresses: [localhost:7026,localhost:7027,localhost:7028]
      tls_config:
        cert_file: ./certs/dev/localhost.crt
        key_file: ./certs/dev/localhost.key
        ca_cert: ./certs/dev/CA.crt
        insecure: true
      sasl_config:
        user: user1
        password: password1
  • disabled tag can be set to true to temporarily disable the service without removing the config
  • multiple clusters can be added under tag "clusters"
  • tags "tls_config" and "sasl_config" are optional

Direct Config

Below is sample configuration for enabling kafka integration

If running in cluster, you only need to enable k8s module and service in maira-agent config file

Make sure that there is one cluster named default. This will be used by maira commands if no cluster argument is specified.

modules:
  - name: maira_io.module.kafka
services:
  - name: clutch.service.k8s
  - name: maira_io.service.kafka
    typed_config:
      "@type": types.google.com/maira_io.config.service.kafka.v1.Config
      clusters:
        default:
          broker_addresses: [localhost:9093,localhost:9094,localhost:9095]
          jmx_addresses: [localhost:7026,localhost:7027,localhost:7028]
          tls_config:
            cert_file: ./certs/dev/localhost.crt
            key_file: ./certs/dev/localhost.key
            ca_cert: ./certs/dev/CA.crt
            insecure: true
        cluster1:
          broker_addresses: [localhost:9092]
          jmx_addresses: [localhost:7025]
          sasl_config:
            user: user1
            password: password1