Confluent CLI + JQ JSON Parsing for Governance and Resource Monitoring
This post lists useful Confluent CLI commands combined with JQ (JSON parsing tool) to analyze and monitor Confluent Cloud resources such as API keys and Kafka topic partitions.
These commands help organizations manage quotas, track resource usage, and maintain governance in Confluent Cloud environments.
Confluent Cloud API Key Limits
Confluent Cloud limits the number of API keys that can be created per organization.
The official limits are documented here:
https://docs.confluent.io/cloud/current/quotas/service-quotas.html#core-resource-scopes
Monitoring API key usage is important because excessive API keys can:
- increase security risks
- complicate governance
- reach organizational limits
- create unnecessary operational overhead
List Confluent API Keys and Count by Resource Type
This command lists Confluent API keys and counts them by resource type.
confluent api-key list --output json \
| jq -r '.[] | .resource_type' \
| sort | uniq -c | sort -rn
What this does
-
Lists all API keys
-
Extracts resource type
-
Groups and counts usage
-
Sorts by highest usage
Example Output
120 kafka-cluster
40 environment
25 schema-registry
10 flinkThis helps identify which resource types consume the most API keys.Confluent Kafka Partition Limits
Confluent Cloud limits the number of partitions that can be allocated based on CKU (Confluent Kafka Unit).
In Dedicated clusters, each CKU provides approximately:
4500 partitions
You can check the official limits here:
https://docs.confluent.io/cloud/current/clusters/cluster-types.html#ecku-cku-comparison
Identify Partition Usage by Application Namespace
If your organization uses a single Kafka cluster with multiple application teams, you may want to identify which team or namespace is using the most partitions.
This can be achieved using Confluent CLI + JQ.
Assumption
This script assumes:
Topic naming follows a standard convention
Each application uses a unique namespace
Naming is aligned with Java package structure
Example
com.ibm.mq.* com.ibm.db2.* org.apache.flink.* com.xyz.orders.*
This allows grouping topics by namespace prefix.
Get Kafka Topics in JSON Format
confluent kafka topic list \
--cluster <YOUR_CLUSTER_ID> \
--environment <YOUR_ENV_ID> \
-o json
This command returns all Kafka topics in JSON format.
Parse Partition Usage by Namespace
confluent kafka topic list \
--cluster <YOUR_CLUSTER_ID> \
--environment <YOUR_ENV_ID> \
-o json \
| jq -r '
map({
prefix: (.name | split(".") | .[0:3] | join(".")),
partitions: .partition_count
})
| group_by(.prefix)
| map({
namespace: .[0].prefix,
totalPartitions: (map(.partitions) | add)
})
| sort_by(.totalPartitions)
| reverse
| .[:20]
| .[]
| "\(.namespace) \(.totalPartitions)"
' \
| tr -d '\r' \
| awk '{printf("%-30s %10s\n", $1, $2)}'What This Script Does
Step-by-step
- Retrieves Kafka topics in JSON format
- Extracts namespace prefix from topic name
- Groups topics by namespace
- Sums partition counts
- Sorts by highest usage
- Displays top 20 namespaces
- Aligns output for readability
Example Output
com.ibm.mq 8200
com.ibm.db2 6000
org.apache.flink 5200
com.xyz.orders 4100This helps identify:
high partition consumers
over-utilized namespaces
teams consuming most cluster capacity
partition allocation distribution
Optional Component
Reverse and Top 20
| reverse
| .[:20]This limits output to top 20 namespaces.
Optional if you want full list.
Why This is Useful
This approach helps organizations:
monitor Kafka partition usage
enforce governance
prevent CKU exhaustion
identify heavy users
plan cluster scaling
allocate partitions per team
optimize resource consumption
Especially useful in shared Confluent Cloud clusters.
