Manage your brokers, consumers, producers, troubleshoot topics/queues
Kafka integration gives you ability to manage your kafka cluster using maira commands. You can also troubleshoot issues by sending or receiving messages on specific topics, or by checking producer/consumer stats. You can also configure your brokers with this integration.
For integrating with kafka cluster, you need to provide the list of brokers and authentication information. Maira supports mTLS and SASL authentication methods. You can manage multiple clusters from Maira.
To monitor Kafka broker metrics, you need to expose the JMX port while instantiating the broker and provide the authentication information for the JMX Port. Each broker must have its own JMX address in the same sequence as the brokers. Further, if any authentication is added for remote jmx, path to the configuration files need to be provided
Below is sample configuration for enabling kafka integration through Helm Chart Values file
kafka: disabled: false clusters: default: broker_addresses: - localhost:9093 - localhost:9094 - localhost:9095 jmx_addresses: [localhost:7026,localhost:7027,localhost:7028] tls_config: cert_file: ./certs/dev/localhost.crt key_file: ./certs/dev/localhost.key ca_cert: ./certs/dev/CA.crt insecure: true sasl_config: user: user1 password: password1
- disabled tag can be set to true to temporarily disable the service without removing the config
- multiple clusters can be added under tag "clusters"
- tags "tls_config" and "sasl_config" are optional
Below is sample configuration for enabling kafka integration
If running in cluster, you only need to enable k8s module and service in maira-agent config file
Make sure that there is one cluster named
default. This will be used by maira commands if no cluster argument is specified.
modules: - name: maira_io.module.kafka services: - name: clutch.service.k8s - name: maira_io.service.kafka typed_config: "@type": types.google.com/maira_io.config.service.kafka.v1.Config clusters: default: broker_addresses: [localhost:9093,localhost:9094,localhost:9095] jmx_addresses: [localhost:7026,localhost:7027,localhost:7028] tls_config: cert_file: ./certs/dev/localhost.crt key_file: ./certs/dev/localhost.key ca_cert: ./certs/dev/CA.crt insecure: true cluster1: broker_addresses: [localhost:9092] jmx_addresses: [localhost:7025] sasl_config: user: user1 password: password1