Kouncil lets you monitor and manage your Apache Kafka clusters using a modern web interface.It's an easy-to-set-up, feature-rich, free and open-source Kafka web UI. This simple Kafka tool makes your DATA detectible, helps troubleshoot problems and deliver optimal solutions. You can easily monitor brokers and their condition, consumer groups and their pace along with the current lag, or view the content of topics in real time.
Here are some of Kouncil's main features. For a more comprehensive list check out the features section.
Advanced record browsing in table format
Multiple cluster support
Cluster monitoring
Consumer group monitoring
Event Tracking
Check out Kouncil in action without installing it. We've prepared a demo site showcasing the main features of Kouncil, which can be found [here]. (https://kouncil-demo.web.app/)
The easiest way to start working with Kouncil is by using Docker:
There are only two required environment variables: bootstrapServers
, which should point to one of the brokers in your Kafka cluster, and kouncil.auth.active-provider
, which specifies authentication mode. For example, if your cluster consists of three brokers - kafka1:9092, kafka2: 9092, kafka3:9092 - you only have to specify one of them (-e bootstrapServers="kafka1:9092"
), and you are good to go. Kouncil will automatically do the rest.
Additionally, Kouncil supports multiple clusters. Hosts specified in bootstrapServers
may point to brokers in several clusters, and Kouncil will recognize that properly. Brokers should be separated using a comma, i.e.: docker run -d -p 80:8080 -e bootstrapServers="kafka1:9092,kafka1.another.cluster:8001" -e kouncil.auth.active-provider="inmemory" consdata/kouncil:latest
After the docker run
command, head to http://localhost.
Images for Kouncil are hosted here: https://hub.docker.com/r/consdata/kouncil.
For more advanced configuration, consult the Deployment section.
Default credentials to log in to Kouncil are admin/admin. For more authentication options, check out Authentication
Kouncil can be configured in two ways:
simple - suitable for most cases, relying solely on docker run
parameters
advanced - suitable for larger configurations, provided as an external file, allowing version control tracking and providing additional configuration options not available in the simple setup
In the case of both simple and advanced configuration being present, the advanced configuration takes precedence.
Simple configuration is passed directly into docker run
command using bootstrapServers
environment variable, just as we've seen in Quick start:
bootstrapServers
variable expects a comma-separated list of brokers, each belonging to a different cluster. Kouncil only needs to know about a single broker from the cluster in order to work.
The simplest possible configuration looks like this:
Next, visit http://localhost in your browser, and you should be greeted with a list of topics from your cluster.
If you have multiple clusters and wish to manage them all with Kouncil, you can do so by simply specifying one broker from each cluster using a comma-separated list:
If you want to set Schema Registry URL, use schemaRegistryUrl
environment variable, for instance:
This URL will be used for every cluster in boostrapServers
variable. If you want to be more specific, go to Advanced configuration.
If you want to set a list of headers to keep while resending events from one topic to another, you can use resendHeadersToKeep
environment variable and pass the list of comma-separated header names, for example:
To change the port on which Kouncil listens for connections, just modify the -p
argument, like so:
It will cause Kouncil to listen on port 7070
.
If you have many Kafka clusters, configuring them using bootstrapServers
may become cumbersome. It is also impossible to express more sophisticated configuration options using such a simple configuration pattern.
To address these issues, Kouncil allows you to provide an external configuration in a YAML file.
Kouncil expects this configuration file to be named kouncil.yaml
. It's only a matter of binding a directory containing that file with Docker. Let's say your kouncil.yaml
lives in /home/users/test/Kouncil/config/
- this is what your docker run
should look like:
The format of kouncil.yaml
is described below.
This example shows two clusters named transaction-cluster
and kouncil
. Each cluster requires a specified name, followed by a list of brokers that make up the cluster. Each broker entry includes the broker's host and the port it is listening on.
For the backend, run KouncilApplication passing parameter bootstrapServers=localhost:9092
pointing to any of your Kafka brokers and spring.config.name=kouncil
.
For the frontend, with Node.js and Yarn installed, run yarn
and yarn start
For the local Kafka, create docker-compose.yml (KAFKA_ADVERTISED_HOST_NAME should match your Docker host IP)
run docker-compose up -d
more info: https://hub.docker.com/r/bitnami/kafka/
By default, authentication is set to inmemory. The default users role configuration is described here Authorization. You can modify this to match your needs. For example if you want to assign the editor role to an admin user, you need to add admin-group
to the role-editor
.
To release, simply push to release branch:
after a successful release, remember to merge back into the master:
Kouncil supports multiple authentication methods along with LDAP, Active Directory, and SSO. There are a lot of different configuration scenarios. Here are examples of the most common ones:
The simplest in-memory provider. It does not require any configuration. Only for test purposes! The default uses are admin, editor, and viewer. The default password for each of these users is user name.
Below you can find configuration snippets for authentication using LDAP, LDAPS and AD:
LDAPS authentication for all users.
LDAP authentication with a technical user for users who belong to a KOUNCIL group.
Active Directory authentication for users who belong to a KOUNCIL group.
In Kouncil you can configure GitHub SSO by adding below snippet to your configuration file. You have to replace your-client-id
, your-client-secret
and your-application-url
.
In Okta, you have to create a new Web application with OIDC - OpenID Connect
Sign-in method:
In the next step you have to configure callback URI, which should look like this: http://<your-application-url>/oauth
Before saving configuration you have to configure application assignments.
In the last step if you should configure groups claim filter to get user groups with access token. You can use any of the compare methods, Equals
, Matches regex
, etc. which will match your requirements and Okta groups configuration.
In Kouncil you can configure Okta SSO by adding below snippet to your configuration file. You have to replace your-client-id
, your-client-secret
, your-application-url
and your-domain-id
WARNING
This configuration will be deprecated in version 1.9 and removed in the future. In version 1.9 this configuration will be used to preload user groups and based on that create permissions for each group.
Kouncil enables restricting access to pages and functions. To achieve that you should add appropriate properties to your configuration file. There are three: kouncil.authorization.role-admin
, kouncil.authorization.role-editor
, andkouncil.authorization.role-viewer
. Each one of these allows users to do specific actions in Kouncil. Users with roles assigned to:
kouncil.authorization.role-admin
can view brokers and consumer groups pages
kouncil.authorization.role-editor
can view topics, event tracker pages, and send messages to topics
kouncil.authorization.role-viewer
can only view topics and event tracker pages
As a value in these parameters you should provide a list of groups defined in the selected authentication provider (inmemory
, LDAP
, AD
, SSO
), separated by a semicolon (;
).
For the default configuration, we have defined the user groups:
These will be used in any authentication method unless you override them in any of your configuration files used by Kouncil.
Example of role configuration:
To get LDAP user groups in LDAP configuration, add a few additional parameters (group-search-base
, group-search-filter
, group-role-attribute
)
Since version 1.9 configuration of user groups and user groups permissions is available from UI.
When permissions will change users will see the message that re-login is required.
By default, users with groups assigned to kouncil.authorization.role-admin
will be able to manage user groups and user groups permissions.
Starting from version 1.9 you will be able to configure and secure your Kafka clusters from UI. To do it log in to the app and select Clusters menu item. You will see your clusters list. To add new cluster click Add new cluster
button and cluster form will be opened.
If your Kafka cluster requires any authentication you are able to configure SASL
, SSL
or AWS MSK
authentication.
Within this form you are able to add Schema Registry to your cluster.
If your Schema Registry requires any authentication you are able to configure SSL
or SSL with basic authentication
security.
WARNING
This configuration will be deprecated in version 1.9 and removed in the future. In version 1.9 this configuration will be used to preload your clusters.
Assuming your Kafka is secured and requires mTLS to connect, you need to provide a client truststore containing the CA's public certificate, and a keystore with both the client's private key and the CA-signed certificate. Then, add a kafka node to your YAML configuration with the following values:
WARNING
This configuration will be deprecated in version 1.9 and removed in the future. In version 1.9 this configuration will be used to preload your clusters.
If one of your brokers in a cluster environment requires SASL authentication, you should specify saslMechanism
, saslProtocol
and saslJassConfig
for that broker. For example:
WARNING
This configuration will be deprecated in version 1.9 and removed in the future. In version 1.9 this configuration will be used to preload your clusters.
If one of your brokers in a cluster environment is located in an Amazon MSK cluster, you should specify saslMechanism
, saslProtocol
, saslJassConfig
, and saslCallbackHandler
for this broker, like this:
The above configuration uses IAM access to the Amazon MSK cluster, and you should provide AWS_SECRET_ACCESS_KEY
and AWS_ACCESS_KEY_ID
as environment variables to Kouncil. These two values should be generated for the user with access to the Amazon MSK cluster, and their user name should be provided in awsProfileName
in the Kouncil configuration.
WARNING
This configuration will be deprecated in version 1.9 and removed in the future. In version 1.9 this configuration will be used to preload your clusters.
Assuming your SchemaRegistry is secured and requires SSL for connection, you need to provide a client truststore that contains the CA's public certificate, and a keystore that includes both the client's private key and the CA-signed certificate.
WARNING
This configuration will be deprecated in version 1.9 and removed in the future. In version 1.9 this configuration will be used to preload your clusters.
Assuming your SchemaRegistry is secured and requires both SSL and BASIC authentication for connection, you need to provide a client truststore that contains the CA's public certificate, and a keystore that includes both the client's private key and the CA-signed certificate. For BASIC authentication, you need to provide user credentials that will be used to authenticate when Kouncil connects to the SchemaRegistry.
If your Kafka brokers expose JMX metrics, Kouncil can leverage them to display additional metrics. This is done using advanced config, where you can specify JMX parameters for each broker, as follows:
This example assumes that the broker does not require authentication to access JMX metrics - you only need to specify the JMX port. If JMX authentication is enabled, you can also specify the JMX username and password:
It quickly becomes clear that, in many cases, the properties (jmxPort
, jmxUser
, jmxPassword
) will be identical for all brokers within the cluster. For that reason, you can specify them at the cluster level, and they will be propagated to each broker:
All brokers within the transaction-cluster
will share the same JMX configuration (jmxPort
= 5088
, jmxUser
= jmxAdmin
, jmxPassword
= jmxPassword
).
Propagation of JMX parameters works independently for each parameter. For example, while all brokers may share the same JMX user and password, they can have different ports.
Kouncil supports logging to an external log file, and we recommend using Logback. You can either use the provided logback.xml
file as-is or use it as a reference to create your custom one. If you use the provided logback.xml
, logs will be placed under logs/kouncil.log
If you want the logs to be accessible outside the Docker container, you can add another volume to the Docker run command like this:
Ensure that path_to_docker_container_logs
matches the path specified in the appender/file
parameter in your logback.xml
.
By default, WebSocket allowed origins are set to *, which can be insecure. You can easily narrow it down by setting the allowedOrigins environment variable as follows:
If you want to expose Kouncil in a custom context path, you need to set Spring's kouncil.context-path
parameter. In Docker run command it will look like this
After that, visit http://localhost/console in your browser, and you should see a login screen.
Currently, Kouncil supports two databases:
PostgreSQL
H2
If no database is specified with below properties, H2 in-memory database will be used.
spring.datasource.url
JDBC URL of the database
spring.datasource.username
login username of the database
spring.datasource.password
login password of the database
spring.jpa.hibernate.default_schema
sets default schema
Here are some of the most important features of Kouncil. This list is not exhaustive. Check out our demo app or quickly install Kouncil to experience all features first-hand.
Thanks to Kouncil's convenient way of presenting records in a table format, even large amounts of complex messages can be easily browsed. You can choose between browsing a single partition or a topic as a whole. If you wish to examine any of the messages more closely, you can view its source, copy it to a clipboard, or even post it again.
If your config spans across multiple Kafka clusters, it's not a problem for Kouncil. You can switch between clusters at any time, without having to restart or reconfigure anything.
Monitoring your consumer groups is one of the most important things when dealing with Kafka. Are my consumers even connected to Kafka? Do they process events? If so, how fast? How long until they finish their work? Kouncil can help you answer all those questions.
Monitoring your cluster's health can be as important as monitoring your consumer groups. Kouncil shows the brokers that are currently connected to the cluster and their current resource consumption (using Kouncil's advanced config)
Event tracking enables monitoring and visualizing the path of a given event or process across the Kafka topics.
First verify logs using:
If you see this kind of messages
follow one of the below fixes.
If you run Kafka in terminal (not using Docker) you can:
Use your host IP address - in this case you need to modify the IP address of listeners and advertised.listeners in server.properties. Kouncil Docker run command would look like this (replace <host_ip_address> with yours IP address):
Kouncil will be available via: http://localhost/login
Use the IP address of the Docker bridge network - to find that IP address, run Docker network, inspect the bridge in the terminal, and under Config, use the value assigned to Gateway. You also need to modify the IP addresses of listeners and advertised.listeners in server.properties to the IP address of the gateway. If you run Kouncil on Windows or Mac, you should be able to use host.docker.internal as the IP address of bootstrapServer.
If you are using Linux, you will have to use --add-host:
Kouncil will be available via: http://localhost/login
Add --network host to Kouncil Docker command. It will look like this (as you can see, I removed the publish flag as it is discarded by Docker when using host network mode):
Kouncil will be available via: http://localhost:8080/login
If you run Kafka using a Docker container, you have to put both containers in the same network, so they can communicate with each other. First, create a new network using
Then use this network name in run command/Docker compose files, for example, Kouncil Docker run command will look like this:
Also, you should use Kafka Docker container name as an IP address (see <container_name>)
You logged in as an administrator user, admin
. In this case we have few possible solutions:
Log in as a user with the editor’s role (in default configuration, it will be the editor
user)
Pass the environment variable, -e
, kouncil.authorization.role-editor
in the docker run
command, with the value of the group name for the admin user. This value will be used instead of the value from default configuration. The Docker run command will look like this:
In the Docker run command, mount a volume with a custom configuration file. The command will look like this:
Please reach out to us
UI Tweaks Part 1 A handful of fixes based on the most frequently reported comments from our users
Better logs on broker unavailability
Broker config - table improvements
Consumer group - Improved lag tracking
Consumer group - colour coded
Event tracking - date format
Event tracking - topic order
Event tracking - web socket toggle
Tables - Stick it to the left
Released (1.1)
Not Only JSON Part 1 This version will bring the remaining popular message formats
Schema Registry
Protobuf consumer
Protobuf producer
Released (1.2)
Not Only JSON Part 2 This version will bring the remaining popular message formats
Avro consumer
Avro producer
Plaintext handling
Released (1.4)
Security Extended Kafka and Kouncil security support
SSL support
JAAS authentication
LDAP authentication
Logged users activity monitoring
Released (1.5)
Security Part 2 Authorization
Introducing admin, editor and viewer roles
Event tracking - additional columns
Tables - column auto adjustment
Released (1.6)
More versions of Kafka and Schema Registry compatible This version will allow you to connect to more versions and configurations of Kafka and Schema Registry
Connecting to AWS MSK cluster
Connecting to secured Schema Registry
Released (1.7)
Manage your cluster In this version you will be able to manage your schemas and topics
Monaco editor
Manage schemas
Manage topics
AVRO logical types
AVRO example generator
Custom base path for exposing application
Released (1.8)
External database and RBAC v2 This version will allow you to configure clusters, permissions in application. You also will be able to connect to external database
External database configuration
In application cluster configuration
More granular permissions
OKTA integration
Released (1.9)
Data masking This version will allow you to define rules which will mask sensitive data
Data masking
In Progress
Alerts & notifications This version will allow you to define alerts and notify users
Alerting system and notifications via email, slack
TODO
Cloud Easy way of deploying Kouncil to the cloud
K8s support - helm chart
Terraform (GCP, AWS, Azure)
TODO