Confluent CLI + JQ JSON Parsing for Governance and Resource Monitoring

This post lists useful Confluent CLI commands combined with JQ (JSON parsing tool) to analyze and monitor Confluent Cloud resources such as API keys and Kafka topic partitions.

These commands help organizations manage quotas, track resource usage, and maintain governance in Confluent Cloud environments.

Confluent Cloud API Key Limits

Confluent Cloud limits the number of API keys that can be created per organization.
The official limits are documented here:
https://docs.confluent.io/cloud/current/quotas/service-quotas.html#core-resource-scopes

Monitoring API key usage is important because excessive API keys can:

• increase security risks
• complicate governance
• reach organizational limits
• create unnecessary operational overhead

List Confluent API Keys and Count by Resource Type

This command lists Confluent API keys and counts them by resource type.

confluent api-key list --output json \
| jq -r '.[] | .resource_type' \
| sort | uniq -c | sort -rn

What this does

• Lists all API keys

• Extracts resource type

• Groups and counts usage

• Sorts by highest usage

Example Output

120 kafka-cluster
40 environment
25 schema-registry
10 flink


This helps identify which resource types consume the most API keys.

Confluent Kafka Partition Limits
Confluent Cloud limits the number of partitions that can be allocated based on CKU (Confluent Kafka Unit).
In Dedicated clusters, each CKU provides approximately:
4500 partitions
You can check the official limits here:
https://docs.confluent.io/cloud/current/clusters/cluster-types.html#ecku-cku-comparison

Identify Partition Usage by Application Namespace
If your organization uses a single Kafka cluster with multiple application teams, you may want to identify which team or namespace is using the most partitions.
This can be achieved using Confluent CLI + JQ.

Assumption
This script assumes:
Topic naming follows a standard convention
Each application uses a unique namespace
Naming is aligned with Java package structure
Example
com.ibm.mq.*
com.ibm.db2.*
org.apache.flink.*
com.xyz.orders.*
This allows grouping topics by namespace prefix.
Get Kafka Topics in JSON Format
confluent kafka topic list \
--cluster <YOUR_CLUSTER_ID> \
--environment <YOUR_ENV_ID> \
-o json

This command returns all Kafka topics in JSON format.
Parse Partition Usage by Namespace
confluent kafka topic list \
--cluster <YOUR_CLUSTER_ID> \
--environment <YOUR_ENV_ID> \
-o json \
| jq -r '
map({
    prefix: (.name | split(".") | .[0:3] | join(".")),
    partitions: .partition_count
})
| group_by(.prefix)
| map({
    namespace: .[0].prefix,
    totalPartitions: (map(.partitions) | add)
})
| sort_by(.totalPartitions)
| reverse
| .[:20]
| .[]
| "\(.namespace) \(.totalPartitions)"
' \
| tr -d '\r' \
| awk '{printf("%-30s %10s\n", $1, $2)}'

What This Script Does
Step-by-step
Retrieves Kafka topics in JSON format
Extracts namespace prefix from topic name
Groups topics by namespace
Sums partition counts
Sorts by highest usage
Displays top 20 namespaces
Aligns output for readability
Example Output
com.ibm.mq                     8200
com.ibm.db2                    6000
org.apache.flink               5200
com.xyz.orders                 4100
This helps identify:
high partition consumers
over-utilized namespaces
teams consuming most cluster capacity
partition allocation distribution
Optional Component
Reverse and Top 20
| reverse
| .[:20]
This limits output to top 20 namespaces.
Optional if you want full list.

Why This is Useful
This approach helps organizations:
monitor Kafka partition usage
enforce governance
prevent CKU exhaustion
identify heavy users
plan cluster scaling
allocate partitions per team
optimize resource consumption
Especially useful in shared Confluent Cloud clusters.