Kouncil lets you monitor and manage your Apache Kafka clusters using a modern web interface.It's an easy-to-set-up, feature-rich, free and open-source Kafka web UI. This simple Kafka tool makes your DATA detectible, helps troubleshoot problems and deliver optimal solutions. You can easily monitor brokers and their condition, consumer groups and their pace along with the current lag, or view the content of topics in real time.

Here are some of Kouncil's main features. For a more comprehensive list check out the features section.

• Advanced record browsing in table format

• Multiple cluster support

• Cluster monitoring

• Consumer group monitoring

• Event Tracking

Demo app

Check out Kouncil in action without installing it. We've prepared a demo site showcasing the main features of Kouncil, which can be found [here]. (https://kouncil-demo.web.app/)

Quick start

The easiest way to start working with Kouncil is by using Docker:

docker run -d -p 80:8080 -e bootstrapServers="kafka1:9092" -e kouncil.auth.active-provider="inmemory" consdata/kouncil:latest

There are only two required environment variables: bootstrapServers, which should point to one of the brokers in your Kafka cluster, and kouncil.auth.active-provider, which specifies authentication mode. For example, if your cluster consists of three brokers - kafka1:9092, kafka2: 9092, kafka3:9092 - you only have to specify one of them (-e bootstrapServers="kafka1:9092"), and you are good to go. Kouncil will automatically do the rest.

Additionally, Kouncil supports multiple clusters. Hosts specified in bootstrapServers may point to brokers in several clusters, and Kouncil will recognize that properly. Brokers should be separated using a comma, i.e.: docker run -d -p 80:8080 -e bootstrapServers="kafka1:9092,kafka1.another.cluster:8001" -e kouncil.auth.active-provider="inmemory" consdata/kouncil:latest

After the docker run command, head to http://localhost.

Images for Kouncil are hosted here: https://hub.docker.com/r/consdata/kouncil.

For more advanced configuration, consult the Deployment section.

Authentication

Default credentials to log in to Kouncil are admin/admin. For more authentication options, check out Authentication

Assuming your Kafka is secured and requires mTLS to connect, you need to provide a client truststore containing the CA's public certificate, and a keystore with both the client's private key and the CA-signed certificate. Then, add a kafka node to your YAML configuration with the following values:

kouncil:
  clusters:
    - name: transaction-cluster
      kafka:
        security:
          protocol: SSL
        ssl:
          truststore-location: file:///config/truststore/client.truststore.jks
          truststore-password: secret
          keystore-password: secret
          keystore-location: file:///config/keystore/client.keystore.jks
          key-password: secret
      brokers:
        - host: 192.10.0.1
          port: 9092

If one of your brokers in a cluster environment requires SASL authentication, you should specify saslMechanism, saslProtocol and saslJassConfig for that broker. For example:

kouncil:
  clusters:
    - name: transaction-cluster
      brokers:
        - host: 192.10.0.1
          port: 9092
          saslMechanism: PLAIN
          saslProtocol: SASL_PLAINTEXT
          saslJassConfig: org.apache.kafka.common.security.plain.PlainLoginModule required username="user" password="secret";
        - host: 192.10.0.2
          port: 9093
        - host: 192.10.0.3
          port: 9094

If one of your brokers in a cluster environment is located in an Amazon MSK cluster, you should specify saslMechanism, saslProtocol, saslJassConfig, and saslCallbackHandler for this broker, like this:

The above configuration uses IAM access to the Amazon MSK cluster, and you should provide AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID as environment variables to Kouncil. These two values should be generated for the user with access to the Amazon MSK cluster, and their user name should be provided in awsProfileName in the Kouncil configuration.

Kouncil supports multiple authentication methods along with LDAP, Active Directory, and SSO. There are a lot of different configuration scenarios. Here are examples of the most common ones:

By default, WebSocket allowed origins are set to *, which can be insecure. You can easily narrow it down by setting the allowedOrigins environment variable as follows:

docker run -d -p 80:8080 -e bootstrapServers="kafka1:9092" -e allowedOrigins="http://localhost:*, https://yourdomain.com" consdata/kouncil:latest

In Kouncil you can configure GitHub SSO by adding below snippet to your configuration file. You have to replace your-client-id, your-client-secret and your-application-url.

Since GitHub doesn't have user roles like other RBAC systems, we decided to use GitHub teams as objects to define access in Kouncil. With the configuration above, Kouncil will use teams across all users organizations. If you want to limit this to specific organizations provide the spring.security.oauth2.client.registration.github.organizations parameter. If you need to specify more than one organization, use a comma as the delimiter.

Authorization

Authorization

Kouncil enables restricting access to pages and functions. To achieve that you should add appropriate properties to your configuration file. There are three: kouncil.authorization.role-admin, kouncil.authorization.role-editor, andkouncil.authorization.role-viewer. Each one of these allows users to do specific actions in Kouncil. Users with roles assigned to:

• kouncil.authorization.role-admin can view brokers and consumer groups pages

• kouncil.authorization.role-editor can view topics, event tracker pages, and send messages to topics

• kouncil.authorization.role-viewer can only view topics and event tracker pages

As a value in these parameters you should provide a list of groups defined in the selected authentication provider (inmemory, LDAP, AD, SSO), separated by a semicolon (;).

For the default configuration, we have defined the user groups:

These will be used in any authentication method unless you override them in any of your configuration files used by Kouncil.

Example of role configuration:

To get LDAP user groups in LDAP configuration, add a few additional parameters (group-search-base, group-search-filter, group-role-attribute)

Configuration

Since version 1.9 configuration of user groups and user groups permissions is available from UI.

When permissions will change users will see the message that re-login is required.

By default, users with groups assigned to kouncil.authorization.role-admin will be able to manage user groups and user groups permissions.

Some of the messages in topics might contain personal and sensitive data. To secure this kind of information we added new feature, data masking.

To turn on this feature, you will have to modify user groups permissions as shown on the screen below:

In that case users with admin_group will have access to view, create, modify and remove data masking policies.

List of policies

To view list of all defined policies you should pick Data masking policies from the menu.

Add new policy

To create new policy you will have to click Add new policy button on the policies list screen. Then you will see create new policy form.

In that form you will have to provide:

• name - policy name

• fields - list of fields names and masking type for each of them

• resources - clusters and topics

  • list of selected clusters and topics names (as topic names you can provide any valid regex) for which policy will be applied

  • or, if you want, you can apply policy to all resources by ticking the checkbox below resources label

• user groups - list of user groups for which policy will be applied

Applying policy

Policies will be applied automatically. When you log in as a user with user group defined in the policy, topic messages for selected resource will have masked data. For masking, star sign, *, will be used.

If we follow above defined policy you should not be able to see all information in CustomerTopic.

Remove policy

Of course if you no longer need the policy you can remove it. To do it, navigate to policies list and click Delete button in the row of the policy you want to remove.

The simplest in-memory provider. It does not require any configuration. Only for test purposes! The default uses are admin, editor, and viewer. The default password for each of these users is user name.

Kouncil supports logging to an external log file, and we recommend using Logback. You can either use the provided logback.xml file as-is or use it as a reference to create your custom one. If you use the provided logback.xml, logs will be placed under logs/kouncil.log

If you want the logs to be accessible outside the Docker container, you can add another volume to the Docker run command like this:

Ensure that path_to_docker_container_logs matches the path specified in the appender/file parameter in your logback.xml.

Assuming your SchemaRegistry is secured and requires SSL for connection, you need to provide a client truststore that contains the CA's public certificate, and a keystore that includes both the client's private key and the CA-signed certificate.

kouncil:
  clusters:
    - name: transaction-cluster
      schemaRegistry:
        url: "https://schema.registry:8081"
        security:
          protocol: SSL
        ssl:
          truststore-location: file:///config/truststore/client.truststore.jks
          truststore-password: password
          trustStoreType: JKS
          keystore-location: file:///config/keystore/client.keystore.jks
          keystore-password: password
          key-password: password
          keyStoreType: JKS
      brokers:
        - host: 192.10.0.1
          port: 9092

Here are some of the most important features of Kouncil. This list is not exhaustive. Check out our demo app or quickly install Kouncil to experience all features first-hand.

Okta configuration

In Okta, you have to create a new Web application with OIDC - OpenID Connect Sign-in method:

In the next step you have to configure callback URI, which should look like this: http://<your-application-url>/oauth

Before saving configuration you have to configure application assignments.

In the last step if you should configure groups claim filter to get user groups with access token. You can use any of the compare methods, Equals, Matches regex, etc. which will match your requirements and Okta groups configuration.

Kouncil configuration

In Kouncil you can configure Okta SSO by adding below snippet to your configuration file. You have to replace your-client-id, your-client-secret, your-application-url and your-domain-id

If your Kafka brokers expose JMX metrics, Kouncil can leverage them to display additional metrics. This is done using advanced config, where you can specify JMX parameters for each broker, as follows:

This example assumes that the broker does not require authentication to access JMX metrics - you only need to specify the JMX port. If JMX authentication is enabled, you can also specify the JMX username and password:

It quickly becomes clear that, in many cases, the properties (jmxPort, jmxUser, jmxPassword) will be identical for all brokers within the cluster. For that reason, you can specify them at the cluster level, and they will be propagated to each broker:

All brokers within the transaction-cluster will share the same JMX configuration (jmxPort = 5088, jmxUser = jmxAdmin, jmxPassword = jmxPassword).

Propagation of JMX parameters works independently for each parameter. For example, while all brokers may share the same JMX user and password, they can have different ports.

Assuming your SchemaRegistry is secured and requires both SSL and BASIC authentication for connection, you need to provide a client truststore that contains the CA's public certificate, and a keystore that includes both the client's private key and the CA-signed certificate. For BASIC authentication, you need to provide user credentials that will be used to authenticate when Kouncil connects to the SchemaRegistry.

If you want to expose Kouncil in a custom context path, you need to set Spring's kouncil.context-path parameter. In Docker run command it will look like this

After that, visit in your browser, and you should see a login screen.

When you launch Kouncil and no user groups and permissions are defined you will see popup which will inform you that you will be logged in as temporary user to define them.

After logging in you will see banner at the top of the page which will have information that you are logged in as temporary user.

After saving user groups permissions, temporary user will not be logged out. This will happen only if you manually click logout button. This is done so you can verify, that you defined groups and permissions correctly. After manual logout temporary user will be removed. And you will be able to log in with:

• predefined users (admin, editor, viewer, superuser) in inmemory authentication

• real users for other authentication methods, ldap, ad, sso.

Below you can find configuration snippets for authentication using LDAP, LDAPS and AD:

• LDAPS authentication for all users.

• LDAP authentication with a technical user for users who belong to a KOUNCIL group.

• Active Directory authentication for users who belong to a KOUNCIL group.

I don't see my topics, brokers.

First verify logs using:

docker logs <container_name>

If you see this kind of messages

 INFO 201079 --- [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Node -1 disconnected.
 WARN 201079 --- [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Connection to node -1 (/192.168.1.28:9092) could not be established. Broker may not be available.
ERROR 201079 --- [nio-8080-exec-3] com.consdata.kouncil.logging.CoreLogger  : Exception while invoking method=TopicsController.getTopics(..)
ERROR 201079 --- [nio-8080-exec-3] c.c.kouncil.KouncilControllerAdvisor     : Received Exception message=java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listTopics
com.consdata.kouncil.KouncilRuntimeException: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listTopics

follow one of the below fixes.

If you run Kafka in terminal (not using Docker) you can:

1. Use your host IP address - in this case you need to modify the IP address of listeners and advertised.listeners in server.properties. Kouncil Docker run command would look like this (replace <host_ip_address> with yours IP address):

   Copy

   docker run -p 80:8080 -e bootstrapServers="<host_ip_address>:9092" -e kouncil.auth.active-provider="inmemory" consdata/kouncil:latest

   Kouncil will be available via: http://localhost/login

2. Use the IP address of the Docker bridge network - to find that IP address, run Docker network, inspect the bridge in the terminal, and under Config, use the value assigned to Gateway. You also need to modify the IP addresses of listeners and advertised.listeners in server.properties to the IP address of the gateway. If you run Kouncil on Windows or Mac, you should be able to use host.docker.internal as the IP address of bootstrapServer.

   Copy

   docker run -p 80:8080 -e bootstrapServers="host.docker.internal:9092" -e kouncil.auth.active-provider="inmemory" consdata/kouncil:latest

   If you are using Linux, you will have to use --add-host:

   Copy

   docker run -p 80:8080 -e bootstrapServers="host.docker.internal:9092" --add-host=host.docker.internal:host-gateway -e kouncil.auth.active-provider="inmemory" consdata/kouncil:latest

   Kouncil will be available via: http://localhost/login

3. Add --network host to Kouncil Docker command. It will look like this (as you can see, I removed the publish flag as it is discarded by Docker when using host network mode):

   Copy

   docker run -e bootstrapServers="localhost:9092" --network host -e kouncil.auth.active-provider="inmemory" consdata/kouncil:latest

   Kouncil will be available via: http://localhost:8080/login

If you run Kafka using a Docker container, you have to put both containers in the same network, so they can communicate with each other. First, create a new network using

docker network create --driver bridge <network_name>

Then use this network name in run command/Docker compose files, for example, Kouncil Docker run command will look like this:

docker run -p 80:8080 -e bootstrapServers="<container_name>:9092" --network="<network_name>" -e kouncil.auth.active-provider="inmemory" consdata/kouncil:latest

Also, you should use Kafka Docker container name as an IP address (see <container_name>)

I logged in and I see only brokers and consumer groups.

You logged in as an administrator user, admin. In this case we have few possible solutions:

1. Log in as a user with the editor’s role (in default configuration, it will be the editor user)

2. Pass the environment variable, -e, kouncil.authorization.role-editor in the docker run command, with the value of the group name for the admin user. This value will be used instead of the value from default configuration. The Docker run command will look like this:

   Copy

   docker run -p 80:8080 -e kouncil.authorization.role-editor="editor_group;admin_group" consdata/kouncil:latest

3. In the Docker run command, mount a volume with a custom configuration file. The command will look like this:

   Copy

   docker run -p 80:8080 -v <path to kouncil.yaml config file>:/config/ consdata/kouncil:latest

I don't see any resolution to my issue

Please reach out to us

Currently, Kouncil supports two databases:

• PostgreSQL

• H2

If no database is specified with below properties, H2 in-memory database will be used.

Configuration properties

• spring.datasource.url JDBC URL of the database

• spring.datasource.username login username of the database

• spring.datasource.password login password of the database

• spring.jpa.hibernate.default_schema sets default schema

Example

spring:
  datasource:
    url: jdbc:postgresql://localhost:5432/
    username: postgres
    password: password
  jpa:
    hibernate:
      default_schema: kouncil

Starting from version 1.9 you will be able to configure and secure your Kafka clusters from UI. To do it log in to the app and select Clusters menu item. You will see your clusters list. To add new cluster click Add new cluster button and cluster form will be opened.

Cluster SSL/TLS configuration

If your Kafka cluster requires any authentication you are able to configure SASL, SSL or AWS MSK authentication.

Schema registry

Within this form you are able to add Schema Registry to your cluster.

If your Schema Registry requires any authentication you are able to configure SSL or SSL with basic authentication security.

Kouncil can be configured in two ways:

• simple - suitable for most cases, relying solely on docker run parameters

• advanced - suitable for larger configurations, provided as an external file, allowing version control tracking and providing additional configuration options not available in the simple setup

In the case of both simple and advanced configuration being present, the advanced configuration takes precedence.

Docker - simple configuration

Simple configuration is passed directly into docker run command using bootstrapServers environment variable, just as we've seen in :

bootstrapServers variable expects a comma-separated list of brokers, each belonging to a different cluster. Kouncil only needs to know about a single broker from the cluster in order to work.

The simplest possible configuration looks like this:

Next, visit in your browser, and you should be greeted with a list of topics from your cluster.

If you have multiple clusters and wish to manage them all with Kouncil, you can do so by simply specifying one broker from each cluster using a comma-separated list:

If you want to set Schema Registry URL, use schemaRegistryUrl environment variable, for instance:

This URL will be used for every cluster in boostrapServers variable. If you want to be more specific, go to .

If you want to set a list of headers to keep while resending events from one topic to another, you can use resendHeadersToKeep environment variable and pass the list of comma-separated header names, for example:

To change the port on which Kouncil listens for connections, just modify the -p argument, like so:

It will cause Kouncil to listen on port 7070.

Docker - advanced configuration

If you have many Kafka clusters, configuring them using bootstrapServers may become cumbersome. It is also impossible to express more sophisticated configuration options using such a simple configuration pattern.

To address these issues, Kouncil allows you to provide an external configuration in a YAML file.

Kouncil expects this configuration file to be named kouncil.yaml. It's only a matter of binding a directory containing that file with Docker. Let's say your kouncil.yaml lives in /home/users/test/Kouncil/config/ - this is what your docker run should look like:

The format of kouncil.yaml is described below.

Advanced configuration example

This example shows two clusters named transaction-cluster and kouncil. Each cluster requires a specified name, followed by a list of brokers that make up the cluster. Each broker entry includes the broker's host and the port it is listening on.