Kouncil lets you monitor and manage your Apache Kafka clusters using a modern web interface.It's an easy-to-set-up, feature-rich, free and open-source Kafka web UI. This simple Kafka tool makes your DATA detectible, helps troubleshoot problems and deliver optimal solutions. You can easily monitor brokers and their condition, consumer groups and their pace along with the current lag, or view the content of topics in real time.

Here are some of Kouncil's main features. For a more comprehensive list check out the features section.

• Advanced record browsing in table format

• Multiple cluster support

• Cluster monitoring

• Consumer group monitoring

• Event Tracking

Demo app

Check out Kouncil in action without installing it. We've prepared a demo site showcasing the main features of Kouncil, which can be found [here]. (https://kouncil-demo.web.app/)

Quick start

The easiest way to start working with Kouncil is by using Docker:

docker run -d -p 80:8080 -e bootstrapServers="kafka1:9092" -e kouncil.auth.active-provider="inmemory" consdata/kouncil:latest

There are only two required environment variables: bootstrapServers, which should point to one of the brokers in your Kafka cluster, and kouncil.auth.active-provider, which specifies authentication mode. For example, if your cluster consists of three brokers - kafka1:9092, kafka2: 9092, kafka3:9092 - you only have to specify one of them (-e bootstrapServers="kafka1:9092"), and you are good to go. Kouncil will automatically do the rest.

Additionally, Kouncil supports multiple clusters. Hosts specified in bootstrapServers may point to brokers in several clusters, and Kouncil will recognize that properly. Brokers should be separated using a comma, i.e.: docker run -d -p 80:8080 -e bootstrapServers="kafka1:9092,kafka1.another.cluster:8001" -e kouncil.auth.active-provider="inmemory" consdata/kouncil:latest

After the docker run command, head to http://localhost.

Images for Kouncil are hosted here: https://hub.docker.com/r/consdata/kouncil.

For more advanced configuration, consult the Deployment section.

Authentication

Default credentials to log in to Kouncil are admin/admin. For more authentication options, check out Authentication

Starting from version 1.9 you will be able to configure and secure your Kafka clusters from UI. To do it log in to the app and select Clusters menu item. You will see your clusters list. To add new cluster click Add new cluster button and cluster form will be opened.

Cluster SSL/TLS configuration

If your Kafka cluster requires any authentication you are able to configure SASL, SSL or AWS MSK authentication.

Schema registry

Within this form you are able to add Schema Registry to your cluster.

If your Schema Registry requires any authentication you are able to configure SSL or SSL with basic authentication security.

Below you can find configuration snippets for authentication using LDAP, LDAPS and AD:

• LDAPS authentication for all users.

kouncil:
  auth:
    active-provider: ldap
    ldap:
      provider-url: "ldaps:///kouncil.io"
      search-base: "ou=Users,dc=kouncil,dc=io"
      search-filter: "(uid={0})"

• LDAP authentication with a technical user for users who belong to a KOUNCIL group.

kouncil:
  auth:
    active-provider: ldap
    ldap:
      provider-url: "ldaps://kouncil.io"
      technical-user-name: "[email protected]"
      technical-user-password: "q1w2e3r4"
      search-base: "ou=Users,dc=kouncil,dc=io"
      search-filter: "(&(objectClass=user)(uid={0})(memberOf=CN=KOUNCIL,CN=Users,DC=kouncil,DC=io))"

• Active Directory authentication for users who belong to a KOUNCIL group.

kouncil:
  auth:
    active-provider: ad
    ad:
      domain: "kouncil.io"
      url: "ldap://kouncil.io:389"
      search-filter: "(&(objectClass=user)(userPrincipalName={0})(memberOf=CN=KOUNCIL,CN=Users,DC=kouncil,DC=io))"

Assuming your Kafka is secured and requires mTLS to connect, you need to provide a client truststore containing the CA's public certificate, and a keystore with both the client's private key and the CA-signed certificate. Then, add a kafka node to your YAML configuration with the following values:

kouncil:
  clusters:
    - name: transaction-cluster
      kafka:
        security:
          protocol: SSL
        ssl:
          truststore-location: file:///config/truststore/client.truststore.jks
          truststore-password: secret
          keystore-password: secret
          keystore-location: file:///config/keystore/client.keystore.jks
          key-password: secret
      brokers:
        - host: 192.10.0.1
          port: 9092

Kouncil supports multiple authentication methods along with LDAP, Active Directory, and SSO. There are a lot of different configuration scenarios. Here are examples of the most common ones:

Running the project

For the backend, run KouncilApplication passing parameter bootstrapServers=localhost:9092 pointing to any of your Kafka brokers and spring.config.name=kouncil.

For the frontend, with Node.js and Yarn installed, run yarn and yarn start

For the local Kafka, create docker-compose.yml (KAFKA_ADVERTISED_HOST_NAME should match your Docker host IP)

run docker-compose up -d

more info: https://hub.docker.com/r/bitnami/kafka/

By default, authentication is set to inmemory. The default users role configuration is described here . You can modify this to match your needs. For example if you want to assign the editor role to an admin user, you need to add admin-group to the role-editor.

Release

To release, simply push to release branch:

after a successful release, remember to merge back into the master:

Kouncil supports logging to an external log file, and we recommend using Logback. You can either use the provided logback.xml file as-is or use it as a reference to create your custom one. If you use the provided logback.xml, logs will be placed under logs/kouncil.log

If you want the logs to be accessible outside the Docker container, you can add another volume to the Docker run command like this:

Ensure that path_to_docker_container_logs matches the path specified in the appender/file parameter in your logback.xml.

If you want to expose Kouncil in a custom context path, you need to set Spring's kouncil.context-path parameter. In Docker run command it will look like this

After that, visit in your browser, and you should see a login screen.

If one of your brokers in a cluster environment requires SASL authentication, you should specify saslMechanism, saslProtocol and saslJassConfig for that broker. For example:

I don't see my topics, brokers.

First verify logs using:

docker logs <container_name>

If you see this kind of messages

 INFO 201079 --- [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Node -1 disconnected.
 WARN 201079 --- [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Connection to node -1 (/192.168.1.28:9092) could not be established. Broker may not be available.
ERROR 201079 --- [nio-8080-exec-3] com.consdata.kouncil.logging.CoreLogger  : Exception while invoking method=TopicsController.getTopics(..)
ERROR 201079 --- [nio-8080-exec-3] c.c.kouncil.KouncilControllerAdvisor     : Received Exception message=java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listTopics
com.consdata.kouncil.KouncilRuntimeException: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listTopics

follow one of the below fixes.

If you run Kafka in terminal (not using Docker) you can:

1. Use your host IP address - in this case you need to modify the IP address of listeners and advertised.listeners in server.properties. Kouncil Docker run command would look like this (replace <host_ip_address> with yours IP address):

   Copy

   docker run -p 80:8080 -e bootstrapServers="<host_ip_address>:9092" -e kouncil.auth.active-provider="inmemory" consdata/kouncil:latest

   Kouncil will be available via: http://localhost/login

2. Use the IP address of the Docker bridge network - to find that IP address, run Docker network, inspect the bridge in the terminal, and under Config, use the value assigned to Gateway. You also need to modify the IP addresses of listeners and advertised.listeners in server.properties to the IP address of the gateway. If you run Kouncil on Windows or Mac, you should be able to use host.docker.internal as the IP address of bootstrapServer.

   Copy

   docker run -p 80:8080 -e bootstrapServers="host.docker.internal:9092" -e kouncil.auth.active-provider="inmemory" consdata/kouncil:latest

   If you are using Linux, you will have to use --add-host:

   Copy

   docker run -p 80:8080 -e bootstrapServers="host.docker.internal:9092" --add-host=host.docker.internal:host-gateway -e kouncil.auth.active-provider="inmemory" consdata/kouncil:latest

   Kouncil will be available via: http://localhost/login

3. Add --network host to Kouncil Docker command. It will look like this (as you can see, I removed the publish flag as it is discarded by Docker when using host network mode):

   Copy

   docker run -e bootstrapServers="localhost:9092" --network host -e kouncil.auth.active-provider="inmemory" consdata/kouncil:latest

   Kouncil will be available via: http://localhost:8080/login

If you run Kafka using a Docker container, you have to put both containers in the same network, so they can communicate with each other. First, create a new network using

docker network create --driver bridge <network_name>

Then use this network name in run command/Docker compose files, for example, Kouncil Docker run command will look like this:

docker run -p 80:8080 -e bootstrapServers="<container_name>:9092" --network="<network_name>" -e kouncil.auth.active-provider="inmemory" consdata/kouncil:latest

Also, you should use Kafka Docker container name as an IP address (see <container_name>)

I logged in and I see only brokers and consumer groups.

You logged in as an administrator user, admin. In this case we have few possible solutions:

1. Log in as a user with the editor’s role (in default configuration, it will be the editor user)

2. Pass the environment variable, -e, kouncil.authorization.role-editor in the docker run command, with the value of the group name for the admin user. This value will be used instead of the value from default configuration. The Docker run command will look like this:

   Copy

   docker run -p 80:8080 -e kouncil.authorization.role-editor="editor_group;admin_group" consdata/kouncil:latest

3. In the Docker run command, mount a volume with a custom configuration file. The command will look like this:

   Copy

   docker run -p 80:8080 -v <path to kouncil.yaml config file>:/config/ consdata/kouncil:latest

I don't see any resolution to my issue

Please reach out to us

In Kouncil you can configure GitHub SSO by adding below snippet to your configuration file. You have to replace your-client-id, your-client-secret and your-application-url.

kouncil:
  auth:
    active-provider: sso
    sso:
      supported:
        providers: github
spring:
  security:
    oauth2:
      client:
        registration:
          github:
            client-id: your-client-id
            client-secret: your-client-secret
            redirect-uri: http://your-application-url/oauth
            scope: read:org

Since GitHub doesn't have user roles like other RBAC systems, we decided to use GitHub teams as objects to define access in Kouncil. With the configuration above, Kouncil will use teams across all users organizations. If you want to limit this to specific organizations provide the spring.security.oauth2.client.registration.github.organizations parameter. If you need to specify more than one organization, use a comma as the delimiter.

If one of your brokers in a cluster environment is located in an Amazon MSK cluster, you should specify saslMechanism, saslProtocol, saslJassConfig, and saslCallbackHandler for this broker, like this:

kouncil:
  clusters:
    - name: transaction-cluster
      brokers:
        - host: 192.10.0.1
          port: 9092
          saslMechanism: AWS_MSK_IAM
          saslProtocol: SASL_SSL
          saslJassConfig: software.amazon.msk.auth.iam.IAMLoginModule required awsProfileName="username";
          saslCallbackHandler: software.amazon.msk.auth.iam.IAMClientCallbackHandler
        - host: 192.10.0.2
          port: 9093
        - host: 192.10.0.3
          port: 9094

The above configuration uses IAM access to the Amazon MSK cluster, and you should provide AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID as environment variables to Kouncil. These two values should be generated for the user with access to the Amazon MSK cluster, and their user name should be provided in awsProfileName in the Kouncil configuration.

By default, WebSocket allowed origins are set to *, which can be insecure. You can easily narrow it down by setting the allowedOrigins environment variable as follows:

docker run -d -p 80:8080 -e bootstrapServers="kafka1:9092" -e allowedOrigins="http://localhost:*, https://yourdomain.com" consdata/kouncil:latest

Authorization

Authorization

Kouncil enables restricting access to pages and functions. To achieve that you should add appropriate properties to your configuration file. There are three: kouncil.authorization.role-admin, kouncil.authorization.role-editor, andkouncil.authorization.role-viewer. Each one of these allows users to do specific actions in Kouncil. Users with roles assigned to:

• kouncil.authorization.role-admin can view brokers and consumer groups pages

• kouncil.authorization.role-editor can view topics, event tracker pages, and send messages to topics

• kouncil.authorization.role-viewer can only view topics and event tracker pages

As a value in these parameters you should provide a list of groups defined in the selected authentication provider (inmemory, LDAP, AD, SSO), separated by a semicolon (;).

For the default configuration, we have defined the user groups:

kouncil:
  authorization:
    role-admin: admin_group
    role-editor: editor_group
    role-viewer: viewer_group

These will be used in any authentication method unless you override them in any of your configuration files used by Kouncil.

Example of role configuration:

kouncil:
  authorization:
    role-admin: KOUNCIL_ADMIN;ROLE_USER;admin_group
    role-editor: KOUNCIL_EDITOR;editor_group
    role-viewer: KOUNCIL_VIEWER;viewer_group

To get LDAP user groups in LDAP configuration, add a few additional parameters (group-search-base, group-search-filter, group-role-attribute)

kouncil:
  auth:
    active-provider: ldap
    ldap:
      provider-url: "ldaps://kouncil.io"
      technical-user-name: "[email protected]"
      technical-user-password: "q1w2e3r4"
      search-base: "ou=Users,dc=kouncil,dc=io"
      search-filter: "(&(objectClass=user)(uid={0})(memberOf=CN=KOUNCIL,CN=Users,DC=kouncil,DC=io))"
      group-search-base: "ou=Groups,dc=kouncil,dc=io"
      group-search-filter: "(memberuid={0})"
      group-role-attribute: "cn"

Configuration

Since version 1.9 configuration of user groups and user groups permissions is available from UI.

When permissions will change users will see the message that re-login is required.

By default, users with groups assigned to kouncil.authorization.role-admin will be able to manage user groups and user groups permissions.

Kouncil can be configured in two ways:

• simple - suitable for most cases, relying solely on docker run parameters

• advanced - suitable for larger configurations, provided as an external file, allowing version control tracking and providing additional configuration options not available in the simple setup

In the case of both simple and advanced configuration being present, the advanced configuration takes precedence.

Docker - simple configuration

Simple configuration is passed directly into docker run command using bootstrapServers environment variable, just as we've seen in :

bootstrapServers variable expects a comma-separated list of brokers, each belonging to a different cluster. Kouncil only needs to know about a single broker from the cluster in order to work.

The simplest possible configuration looks like this:

Next, visit in your browser, and you should be greeted with a list of topics from your cluster.

If you have multiple clusters and wish to manage them all with Kouncil, you can do so by simply specifying one broker from each cluster using a comma-separated list:

If you want to set Schema Registry URL, use schemaRegistryUrl environment variable, for instance:

This URL will be used for every cluster in boostrapServers variable. If you want to be more specific, go to .

If you want to set a list of headers to keep while resending events from one topic to another, you can use resendHeadersToKeep environment variable and pass the list of comma-separated header names, for example:

To change the port on which Kouncil listens for connections, just modify the -p argument, like so:

It will cause Kouncil to listen on port 7070.

Docker - advanced configuration

If you have many Kafka clusters, configuring them using bootstrapServers may become cumbersome. It is also impossible to express more sophisticated configuration options using such a simple configuration pattern.

To address these issues, Kouncil allows you to provide an external configuration in a YAML file.

Kouncil expects this configuration file to be named kouncil.yaml. It's only a matter of binding a directory containing that file with Docker. Let's say your kouncil.yaml lives in /home/users/test/Kouncil/config/ - this is what your docker run should look like:

The format of kouncil.yaml is described below.

Advanced configuration example

This example shows two clusters named transaction-cluster and kouncil. Each cluster requires a specified name, followed by a list of brokers that make up the cluster. Each broker entry includes the broker's host and the port it is listening on.

Okta configuration

In Okta, you have to create a new Web application with OIDC - OpenID Connect Sign-in method:

In the next step you have to configure callback URI, which should look like this: http://<your-application-url>/oauth

Before saving configuration you have to configure application assignments.

In the last step if you should configure groups claim filter to get user groups with access token. You can use any of the compare methods, Equals, Matches regex, etc. which will match your requirements and Okta groups configuration.

Kouncil configuration

In Kouncil you can configure Okta SSO by adding below snippet to your configuration file. You have to replace your-client-id, your-client-secret, your-application-url and your-domain-id

Currently, Kouncil supports two databases:

• PostgreSQL

• H2

If no database is specified with below properties, H2 in-memory database will be used.

Configuration properties

• spring.datasource.url JDBC URL of the database

• spring.datasource.username login username of the database

• spring.datasource.password login password of the database

• spring.jpa.hibernate.default_schema sets default schema

Example

Here are some of the most important features of Kouncil. This list is not exhaustive. Check out our or to experience all features first-hand.

Advanced record browsing in table format

Thanks to Kouncil's convenient way of presenting records in a table format, even large amounts of complex messages can be easily browsed. You can choose between browsing a single partition or a topic as a whole. If you wish to examine any of the messages more closely, you can view its source, copy it to a clipboard, or even post it again.

Multiple cluster support

If your config spans across multiple Kafka clusters, it's not a problem for Kouncil. You can switch between clusters at any time, without having to restart or reconfigure anything.

Consumer monitoring

Monitoring your consumer groups is one of the most important things when dealing with Kafka. Are my consumers even connected to Kafka? Do they process events? If so, how fast? How long until they finish their work? Kouncil can help you answer all those questions.

Cluster monitoring

Monitoring your cluster's health can be as important as monitoring your consumer groups. Kouncil shows the brokers that are currently connected to the cluster and their current resource consumption (using Kouncil's )

Event tracking

Event tracking enables monitoring and visualizing the path of a given event or process across the Kafka topics.

The simplest in-memory provider. It does not require any configuration. Only for test purposes! The default uses are admin, editor, and viewer. The default password for each of these users is user name.

Assuming your SchemaRegistry is secured and requires SSL for connection, you need to provide a client truststore that contains the CA's public certificate, and a keystore that includes both the client's private key and the CA-signed certificate.

kouncil:
  clusters:
    - name: transaction-cluster
      schemaRegistry:
        url: "https://schema.registry:8081"
        security:
          protocol: SSL
        ssl:
          truststore-location: file:///config/truststore/client.truststore.jks
          truststore-password: password
          trustStoreType: JKS
          keystore-location: file:///config/keystore/client.keystore.jks
          keystore-password: password
          key-password: password
          keyStoreType: JKS
      brokers:
        - host: 192.10.0.1
          port: 9092

If your Kafka brokers expose JMX metrics, Kouncil can leverage them to display additional metrics. This is done using advanced config, where you can specify JMX parameters for each broker, as follows:

This example assumes that the broker does not require authentication to access JMX metrics - you only need to specify the JMX port. If JMX authentication is enabled, you can also specify the JMX username and password:

It quickly becomes clear that, in many cases, the properties (jmxPort, jmxUser, jmxPassword) will be identical for all brokers within the cluster. For that reason, you can specify them at the cluster level, and they will be propagated to each broker:

All brokers within the transaction-cluster will share the same JMX configuration (jmxPort = 5088, jmxUser = jmxAdmin, jmxPassword = jmxPassword).

Propagation of JMX parameters works independently for each parameter. For example, while all brokers may share the same JMX user and password, they can have different ports.

Assuming your SchemaRegistry is secured and requires both SSL and BASIC authentication for connection, you need to provide a client truststore that contains the CA's public certificate, and a keystore that includes both the client's private key and the CA-signed certificate. For BASIC authentication, you need to provide user credentials that will be used to authenticate when Kouncil connects to the SchemaRegistry.

kouncil:
  clusters:
    - name: local-cluster
      schemaRegistry:
        url: "https://schema.registry:8081"
        auth:
          source: USER_INFO
          user-info: username:password
        security:
          protocol: SSL
        ssl:
          truststore-location: file:///config/truststore/client.truststore.jks
          truststore-password: password
          trustStoreType: JKS
          keystore-location: file:///config/keystore/client.keystore.jks
          keystore-password: password
          key-password: password
          keyStoreType: JKS
      brokers:
        - host: 192.10.0.1
          port: 9092