Links

Deployment

There are two ways in which Kouncil can be configured:
  • simple - suitable for most cases, relying solely on docker run parameters
  • advanced - suitable for larger configurations. Provided as an external file, and thus can be tracked in version control. It also exposes additional configuration options, which are not available in the simple configuration

Docker - simple configuration

Simple configuration is passed directly into docker run command using bootstrapServers environment variable, just as we've seen in Quick start:
docker run -d -p 80:8080 -e bootstrapServers="kafka1:9092" consdata/kouncil:latest
bootstrapServers variable expects a comma-separated list of brokers, each belonging to a different cluster. Kouncil only needs to know about a single broker from the cluster in order to work.
The simplest possible configuration looks like this:
docker run -d -p 80:8080 -e bootstrapServers="kafka1:9092" consdata/kouncil:latest
After that, visit http://localhost in your browser, and you should be greeted with a list of topics from your cluster.
If you have multiple clusters and wish to manage them all with Kouncil, you can do so by simply specifying one broker from each cluster using comma-separated list:
docker run -d -p 80:8080 -e bootstrapServers="kafka1:9092,kafka1.another.cluster:8001" consdata/kouncil:latest
If you want to set Schema Registry url use schemaRegistryUrl environment variable, for instance:
docker run -d -p 80:8080 -e bootstrapServers="kafka1:9092" -e schemaRegistryUrl="http://schema.registry:8081" consdata/kouncil:latest
This url will be used for every cluster in boostrapServers variable. If you want to be more specific go to Advanced configuration.
If you want to set list of headers to keep while resending events from one topic to another you can use resendHeadersToKeep environment variable and pass list of comma-seperated header names, for example:
docker run -d -p 80:8080 -e bootstrapServers="kafka1:9092" -e resendHeadersToKeep="requestId,version" consdata/kouncil:latest
In order to change the port on which Kouncil listens for connections, just modify the -p argument, like so:
docker run -d -p 7070:8080 -e bootstrapServers="kafka1:9092" consdata/kouncil:latest
That will cause Kouncil to listen on port 7070.

Docker - advanced configuration

If you have many Kafka clusters, configuring them using bootstrapServers may become cumbersome. It is also impossible to express more sophisticated configuration options using such a simple configuration pattern.
To address these issues Kouncil allows you to provide an external configuration in a yaml file.
Kouncil expects this configuration file to be named kouncil.yaml. After that it's only a matter of binding a directory containing that file into docker - let's say your kouncil.yaml lives in /home/users/test/Kouncil/config/, that's how your docker run should look:
docker run -p 80:8080 -v /home/users/test/Kouncil/config/:/config/ consdata/kouncil:latest
Format of kouncil.yaml is described below.

Advanced config example

kouncil:
clusters:
- name: transaction-cluster
schemaRegistry:
url: "http://schema.registry:8081"
brokers:
- host: 192.10.0.1
port: 9092
- host: 192.10.0.2
port: 9093
- host: 192.10.0.3
port: 9094
- name: kouncil
schemaRegistry:
url: "http://another.schema.registry:8081"
brokers:
- host: kouncil.kafka.local
port: 8001
- host: kouncil.kafka.local
port: 8002
This example shows two clusters, named transaction-cluster and kouncil respectively. Each cluster needs to have its name specified. After that comes a list of brokers that make up this cluster - each of which consisting of the broker's host and port on which it's listening on.

Advanced config - JMX monitoring

If your Kafka brokers expose JMX metrics Kouncil can take advantage of that, displaying additional metrics. This is done using advanced config, where you can specify JMX parameters for each broker, like so:
kouncil:
clusters:
- name: transaction-cluster
brokers:
- host: 192.10.0.1
port: 9092
jmxPort: 5088
- host: 192.10.0.2
port: 9093
jmxPort: 5089
- host: 192.10.0.3
port: 9094
jmxPort: 5090
This example assumes that broker does not require any kind of authentication to access JMX metrics - you only need to specify JMX port. If that's not the case, and JMX authentication is turned on, you can also specify JMX user and password:
kouncil:
clusters:
- name: transaction-cluster
brokers:
- host: 192.10.0.1
port: 9092
jmxPort: 5088
jmxUser: jmxAdmin
jmxPassword: jmxPassword
- host: 192.10.0.2
port: 9093
jmxPort: 5088
jmxUser: jmxAdmin
jmxPassword: jmxPassword
- host: 192.10.0.3
port: 9094
jmxPort: 5088
jmxUser: jmxAdmin
jmxPassword: jmxPassword
It quickly becomes clear, that in many cases those properties (jmxPort, jmxUser, jmxPassword) will be identical for each of the brokers inside the cluster. For that reason, you can also specify them on a cluster level, and they will propagate to each broker:
kouncil:
clusters:
- name: transaction-cluster
jmxPort: 5088
jmxUser: jmxAdmin
jmxPassword: jmxPassword
brokers:
- host: 192.10.0.1
port: 9092
- host: 192.10.0.2
port: 9093
- host: 192.10.0.3
port: 9094
All brokers inside transaction-cluster will share the same JMX configuration (jmxPort = 5088, jmxUser = jmxAdmin, jmxPassword = jmxPassword).
Propagation of JMX parameters works independently for each of those parameters. For example, each of the brokers may have the same JMX user and password, but different port:
kouncil:
clusters:
- name: transaction-cluster
jmxUser: jmxAdmin
jmxPassword: jmxPassword
brokers:
- host: 192.10.0.1
port: 9092
jmxPort: 5088
- host: 192.10.0.2
port: 9093
jmxPort: 5089
- host: 192.10.0.3
port: 9094
jmxPort: 5090
In the case of both simple and advanced configuration being present, the advanced configuration takes precedence.

Advanced config - TLS

Let's assume that your Kafka is secured and you need mTLS to connect. You need to provide a client truststore, containing CA public certificate and keystore with both client private key and CA signed certificate. Then add "kafka" node to your yaml with the following values:
kouncil:
clusters:
- name: transaction-cluster
kafka:
security:
protocol: SSL
ssl:
truststore-location: file:///config/truststore/client.truststore.jks
truststore-password: secret
keystore-password: secret
keystore-location: file:///config/keystore/client.keystore.jks
key-password: secret
brokers:
- host: 192.10.0.1
port: 9092

Advanced config - SASL Authentication

If one of your brokers in cluster type environment needs SASL authentication You could specify saslUsername and saslPassword for each broker, like so:
kouncil:
clusters:
- name: transaction-cluster
brokers:
- host: 192.10.0.1
port: 9092
saslUsername: username
saslPassword: password
- host: 192.10.0.2
port: 9093
- host: 192.10.0.3
port: 9094

WebSocket allowed origins configuration

By default, WebSocket allowed origins are set to *, which can be inefficient from the security point of view. You can easily narrow it down, setting allowedOrigins environment variable like that:
docker run -d -p 80:8080 -e bootstrapServers="kafka1:9092" -e allowedOrigins="http://localhost:*, https://yourdomain.com" consdata/kouncil:latest

Authentication

Kouncil supports multiple authentication methods along with LDAP, Active Directory and SSO. There are a lot of different configuration scenarios. Here are examples of most common ones:
  • Simplest in-memory provider. Do not require any configuration. Only for the test purposes!
kouncil:
auth:
active-provider: inmemory
  • LDAPS authentication for all users.
kouncil:
auth:
active-provider: ldap
ldap:
provider-url: "ldaps:///kouncil.io"
search-base: "ou=Users,dc=kouncil,dc=io"
search-filter: "(uid={0})"
  • LDAP authentication with a technical user for users who belong to a KOUNCIL group.
kouncil:
auth:
active-provider: ldap
ldap:
provider-url: "ldaps://kouncil.io"
technical-user-name: "[email protected]"
technical-user-password: "q1w2e3r4"
search-base: "ou=Users,dc=kouncil,dc=io"
search-filter: "(&(objectClass=user)(uid={0})(memberOf=CN=KOUNCIL,CN=Users,DC=kouncil,DC=io))"
  • Active Directory authentication for users who belong to a KOUNCIL group.
kouncil:
auth:
active-provider: ad
ad:
domain: "kouncil.io"
url: "ldap://kouncil.io:389"
search-filter: "(&(objectClass=user)(userPrincipalName={0})(memberOf=CN=KOUNCIL,CN=Users,DC=kouncil,DC=io))"
  • Github SSO
kouncil:
auth:
active-provider: sso
spring:
security:
oauth2:
client:
registration:
github:
client-id: your-client-id
client-secret: your-client-secret
redirect-uri: http://your-application-url/oauth

Logging

Kouncil supports logging to external log file. We suggest to use logback. If you want you can use our provided logback.xml file as it is or use it as a reference to create your custom one. If you use provided logback.xml logs will be placed under logs/kouncil.log
docker run -d -p 80:8080 -e bootstrapServers="kafka1:9092" -e logging.config="path_to_your_logback_xml_file_in_docker_container" -v path_to_your_local_logback_xml_folder:/path_to_your_container_logback_xml_folder consdata/kouncil:latest
If you want the logs to be accessible outside the docker container, you could pass another volume in docker run command like this:
-v path_to_your_local_logback_xml_folder:path_to_docker_container_logs
Also path_to_docker_container_logs should be equal to path in appender/file parameter in logback.xml