Deployment
There are two ways in which Kouncil can be configured:
- simple - suitable for most cases, relying solely on
docker run
parameters - advanced - suitable for larger configurations. Provided as an external file, and thus can be tracked in version control. It also exposes additional configuration options, which are not available in the simple configuration
Simple configuration is passed directly into
docker run
command using bootstrapServers
environment variable, just as we've seen in Quick start:docker run -d -p 80:8080 -e bootstrapServers="kafka1:9092" consdata/kouncil:latest
bootstrapServers
variable expects a comma-separated list of brokers, each belonging to a different cluster. Kouncil only needs to know about a single broker from the cluster in order to work.The simplest possible configuration looks like this:
docker run -d -p 80:8080 -e bootstrapServers="kafka1:9092" consdata/kouncil:latest
After that, visit http://localhost in your browser, and you should be greeted with a list of topics from your cluster.
If you have multiple clusters and wish to manage them all with Kouncil, you can do so by simply specifying one broker from each cluster using comma-separated list:
docker run -d -p 80:8080 -e bootstrapServers="kafka1:9092,kafka1.another.cluster:8001" consdata/kouncil:latest
If you want to set Schema Registry url use
schemaRegistryUrl
environment variable, for instance:docker run -d -p 80:8080 -e bootstrapServers="kafka1:9092" -e schemaRegistryUrl="http://schema.registry:8081" consdata/kouncil:latest
This url will be used for every cluster in
boostrapServers
variable. If you want to be more specific go to Advanced configuration.If you want to set list of headers to keep while resending events from one topic to another you can use
resendHeadersToKeep
environment variable and pass list of comma-seperated header names, for example:docker run -d -p 80:8080 -e bootstrapServers="kafka1:9092" -e resendHeadersToKeep="requestId,version" consdata/kouncil:latest
In order to change the port on which Kouncil listens for connections, just modify the
-p
argument, like so:docker run -d -p 7070:8080 -e bootstrapServers="kafka1:9092" consdata/kouncil:latest
That will cause Kouncil to listen on port
7070
.If you have many Kafka clusters, configuring them using
bootstrapServers
may become cumbersome. It is also impossible to express more sophisticated configuration options using such a simple configuration pattern.To address these issues Kouncil allows you to provide an external configuration in a yaml file.
Kouncil expects this configuration file to be named
kouncil.yaml
. After that it's only a matter of binding a directory containing that file into docker - let's say your kouncil.yaml
lives in /home/users/test/Kouncil/config/
, that's how your docker run
should look:docker run -p 80:8080 -v /home/users/test/Kouncil/config/:/config/ consdata/kouncil:latest
Format of
kouncil.yaml
is described below.kouncil:
clusters:
- name: transaction-cluster
schemaRegistry:
url: "http://schema.registry:8081"
brokers:
- host: 192.10.0.1
port: 9092
- host: 192.10.0.2
port: 9093
- host: 192.10.0.3
port: 9094
- name: kouncil
schemaRegistry:
url: "http://another.schema.registry:8081"
brokers:
- host: kouncil.kafka.local
port: 8001
- host: kouncil.kafka.local
port: 8002
This example shows two clusters, named
transaction-cluster
and kouncil
respectively. Each cluster needs to have its name specified. After that comes a list of brokers that make up this cluster - each of which consisting of the broker's host and port on which it's listening on.If your Kafka brokers expose JMX metrics Kouncil can take advantage of that, displaying additional metrics. This is done using advanced config, where you can specify JMX parameters for each broker, like so:
kouncil:
clusters:
- name: transaction-cluster
brokers:
- host: 192.10.0.1
port: 9092
jmxPort: 5088
- host: 192.10.0.2
port: 9093
jmxPort: 5089
- host: 192.10.0.3
port: 9094
jmxPort: 5090
This example assumes that broker does not require any kind of authentication to access JMX metrics - you only need to specify JMX port. If that's not the case, and JMX authentication is turned on, you can also specify JMX user and password:
kouncil:
clusters:
- name: transaction-cluster
brokers:
- host: 192.10.0.1
port: 9092
jmxPort: 5088
jmxUser: jmxAdmin
jmxPassword: jmxPassword
- host: 192.10.0.2
port: 9093
jmxPort: 5088
jmxUser: jmxAdmin
jmxPassword: jmxPassword
- host: 192.10.0.3
port: 9094
jmxPort: 5088
jmxUser: jmxAdmin
jmxPassword: jmxPassword
It quickly becomes clear, that in many cases those properties (
jmxPort
, jmxUser
, jmxPassword
) will be identical for each of the brokers inside the cluster. For that reason, you can also specify them on a cluster level, and they will propagate to each broker:kouncil:
clusters:
- name: transaction-cluster
jmxPort: 5088
jmxUser: jmxAdmin
jmxPassword: jmxPassword
brokers:
- host: 192.10.0.1
port: 9092
- host: 192.10.0.2
port: 9093
- host: 192.10.0.3
port: 9094
All brokers inside
transaction-cluster
will share the same JMX configuration (jmxPort
= 5088
, jmxUser
= jmxAdmin
, jmxPassword
= jmxPassword
).Propagation of JMX parameters works independently for each of those parameters. For example, each of the brokers may have the same JMX user and password, but different port:
kouncil:
clusters:
- name: transaction-cluster
jmxUser: jmxAdmin
jmxPassword: jmxPassword
brokers:
- host: 192.10.0.1
port: 9092
jmxPort: 5088
- host: 192.10.0.2
port: 9093
jmxPort: 5089
- host: 192.10.0.3
port: 9094
jmxPort: 5090
In the case of both simple and advanced configuration being present, the advanced configuration takes precedence.
Let's assume that your Kafka is secured and you need mTLS to connect. You need to provide a client truststore, containing CA public certificate and keystore with both client private key and CA signed certificate. Then add "kafka" node to your yaml with the following values:
kouncil:
clusters:
- name: transaction-cluster
kafka:
security:
protocol: SSL
ssl:
truststore-location: file:///config/truststore/client.truststore.jks
truststore-password: secret
keystore-password: secret
keystore-location: file:///config/keystore/client.keystore.jks
key-password: secret
brokers:
- host: 192.10.0.1
port: 9092
If one of your brokers in cluster type environment needs SASL authentication You could specify
saslUsername
and saslPassword
for each broker, like so:kouncil:
clusters:
- name: transaction-cluster
brokers:
- host: 192.10.0.1
port: 9092
saslUsername: username
saslPassword: password
- host: 192.10.0.2
port: 9093
- host: 192.10.0.3
port: 9094
By default, WebSocket allowed origins are set to *, which can be inefficient from the security point of view. You can easily narrow it down, setting
allowedOrigins
environment variable like that:docker run -d -p 80:8080 -e bootstrapServers="kafka1:9092" -e allowedOrigins="http://localhost:*, https://yourdomain.com" consdata/kouncil:latest
Kouncil supports multiple authentication methods along with LDAP, Active Directory and SSO. There are a lot of different configuration scenarios. Here are examples of most common ones:
- Simplest in-memory provider. Do not require any configuration. Only for the test purposes!
kouncil:
auth:
active-provider: inmemory
- LDAPS authentication for all users.
kouncil:
auth:
active-provider: ldap
ldap:
provider-url: "ldaps:///kouncil.io"
search-base: "ou=Users,dc=kouncil,dc=io"
search-filter: "(uid={0})"
- LDAP authentication with a technical user for users who belong to a KOUNCIL group.
kouncil:
auth:
active-provider: ldap
ldap:
provider-url: "ldaps://kouncil.io"
technical-user-name: "[email protected]"
technical-user-password: "q1w2e3r4"
search-base: "ou=Users,dc=kouncil,dc=io"
search-filter: "(&(objectClass=user)(uid={0})(memberOf=CN=KOUNCIL,CN=Users,DC=kouncil,DC=io))"
- Active Directory authentication for users who belong to a KOUNCIL group.
kouncil:
auth:
active-provider: ad
ad:
domain: "kouncil.io"
url: "ldap://kouncil.io:389"
search-filter: "(&(objectClass=user)(userPrincipalName={0})(memberOf=CN=KOUNCIL,CN=Users,DC=kouncil,DC=io))"
- Github SSO
kouncil:
auth:
active-provider: sso
spring:
security:
oauth2:
client:
registration:
github:
client-id: your-client-id
client-secret: your-client-secret
redirect-uri: http://your-application-url/oauth
Kouncil supports logging to external log file. We suggest to use logback. If you want you can use our provided
logback.xml
file as it is or use it as a reference to create your custom one. If you use provided logback.xml
logs will be placed under logs/kouncil.logdocker run -d -p 80:8080 -e bootstrapServers="kafka1:9092" -e logging.config="path_to_your_logback_xml_file_in_docker_container" -v path_to_your_local_logback_xml_folder:/path_to_your_container_logback_xml_folder consdata/kouncil:latest
If you want the logs to be accessible outside the docker container, you could pass another volume in docker run command like this:
-v path_to_your_local_logback_xml_folder:path_to_docker_container_logs
Also
path_to_docker_container_logs
should be equal to path in appender/file
parameter in logback.xml
Last modified 17d ago