Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[METRICS] 討論 MetricFetcher 拉取特定 metrics #1624

Open
chinghongfang opened this issue Mar 30, 2023 · 1 comment
Open

[METRICS] 討論 MetricFetcher 拉取特定 metrics #1624

chinghongfang opened this issue Mar 30, 2023 · 1 comment

Comments

@chinghongfang
Copy link
Collaborator

chinghongfang commented Mar 30, 2023

Related to #1615 (comment)

localReceiver 將所有目標的 metric 拉下來 -> 過濾 -> 儲存。
然而,當 kafka cluster 內有許多 partition 時,metric 的數量也會增大。(數量大約是 2000 partition broker 大約要拉 20615 metric, 0 partition broker 大約要拉 614 metrics)

所以,若是只拉取需要的 metric 將有機會大大降低 "metric 拉下來" 的時間。
數量的來源主要是 partition level 的 metric,所以我認為把可以剔除不必要的 "partition level metric" 就好。

這裡討論有哪些 metric 是需要拉取的,我先列出 NetworkCost 會用到的 metric ,以及所有 metric 的數量級,供大家參考。

NetworkCost
Name Type Number of beans
Size Log 2001
ReplicasCount Partition 2000
null Memory 1
ClusterId KafkaServer 1
All metrics
Name Type Number of beans
NumLogSegments Log 2001
LogEndOffset Log 2001
LogStartOffset Log 2001
Size Log 2001
LastStableOffsetLag Partition 2000
InSyncReplicasCount Partition 2000
AtMinIsr Partition 2000
ReplicasCount Partition 2000
UnderReplicated Partition 2000
UnderMinIsr Partition 2000
TotalTimeMs RequestMetrics 51
ResponseSendTimeMs RequestMetrics 51
RequestBytes RequestMetrics 51
RequestQueueTimeMs RequestMetrics 51
RemoteTimeMs RequestMetrics 51
ResponseQueueTimeMs RequestMetrics 51
LocalTimeMs RequestMetrics 51
ThrottleTimeMs RequestMetrics 51
null socket-server-metrics 34
ResponseQueueSize RequestChannel 9
IdlePercent Processor 8
RequestsPerSec RequestMetrics 7
NumDelayedOperations DelayedOperationPurgatory 7
PurgatorySize DelayedOperationPurgatory 7
ErrorsPerSec RequestMetrics 6
uncleanable-bytes LogCleanerManager 2
uncleanable-partitions-count LogCleanerManager 2
null app-info 2
TemporaryMemoryBytes RequestMetrics 2
LogDirectoryOffline LogManager 2
MessageConversionsTimeMs RequestMetrics 2
ReplicationBytesOutPerSec BrokerTopicMetrics 1
OfflineReplicaCount ReplicaManager 1
G1 Survivor Space MemoryPool 1
ReassignmentBytesInPerSec BrokerTopicMetrics 1
max-compaction-delay-secs LogCleaner 1
linux-disk-write-bytes KafkaServer 1
CodeCacheManager MemoryManager 1
PartitionCount ReplicaManager 1
ReassigningPartitions ReplicaManager 1
UnderMinIsrPartitionCount ReplicaManager 1
DeadThreadCount ReplicaAlterLogDirsManager 1
ProduceMessageConversionsPerSec BrokerTopicMetrics 1
NumGroupsEmpty GroupMetadataManager 1
null heartbeat-metrics 1
LeaderCount ReplicaManager 1
AtMinIsrPartitionCount ReplicaManager 1
G1 Old Gen MemoryPool 1
ExpiredConnectionsKilledCount SocketServer 1
max-clean-time-secs LogCleaner 1
CodeHeap 'profiled nmethods' MemoryPool 1
null Threading 1
NumGroupsStable GroupMetadataManager 1
null forwarding-metrics 1
null ControllerMutation 1
Metaspace Manager MemoryManager 1
null raft-channel-metrics 1
NumGroupsCompletingRebalance GroupMetadataManager 1
null Memory 1
mapped BufferPool 1
max-buffer-utilization-percent LogCleaner 1
NumIncrementalFetchPartitionsCached FetchSessionCache 1
ReplicationBytesInPerSec BrokerTopicMetrics 1
null kafka.Log4jController 1
MetadataBatchProcessingTimeUs BrokerMetadataListener 1
DeadThreadCount ReplicaFetcherManager 1
NetworkProcessorAvgIdlePercent SocketServer 1
cleaner-io Throttler 1
time-since-last-run-ms LogCleanerManager 1
FetchMessageConversionsPerSec BrokerTopicMetrics 1
null HotSpotDiagnostic 1
ClusterId KafkaServer 1
TotalFetchRequestsPerSec BrokerTopicMetrics 1
MaxLag ReplicaAlterLogDirsManager 1
MaxLag ReplicaFetcherManager 1
null group-coordinator-metrics 1
G1 Eden Space MemoryPool 1
null raft-metrics 1
IsrExpandsPerSec ReplicaManager 1
RequestQueueSize RequestChannel 1
ReassignmentBytesOutPerSec BrokerTopicMetrics 1
MinFetchRate ReplicaFetcherManager 1
UnknownDestinationQueueSize TransactionMarkerChannelManager 1
null txn-marker-channel-metrics 1
OfflineLogDirectoryCount LogManager 1
CodeHeap 'non-profiled nmethods' MemoryPool 1
Metaspace MemoryPool 1
AcceptorBlockedPercent Acceptor 1
null Produce 1
null transaction-coordinator-metrics 1
DeadThreadCount LogCleaner 1
FailedFetchRequestsPerSec BrokerTopicMetrics 1
NoKeyCompactedTopicRecordsPerSec BrokerTopicMetrics 1
null FlightRecorder 1
MemoryPoolUsed SocketServer 1
null broker-metadata-metrics 1
null ClassLoading 1
LogAppendRetryQueueSize TransactionMarkerChannelManager 1
direct BufferPool 1
BytesInPerSec BrokerTopicMetrics 1
null Runtime 1
TotalProduceRequestsPerSec BrokerTopicMetrics 1
NumGroups GroupMetadataManager 1
NumIncrementalFetchSessions FetchSessionCache 1
max-dirty-percent LogCleanerManager 1
FailedPartitionsCount ReplicaAlterLogDirsManager 1
G1 Young Generation GarbageCollector 1
Compressed Class Space MemoryPool 1
G1 Old Generation GarbageCollector 1
null Compilation 1
null Request 1
MemoryPoolAvailable SocketServer 1
InvalidOffsetOrSequenceRecordsPerSec BrokerTopicMetrics 1
FailedPartitionsCount ReplicaFetcherManager 1
BytesRejectedPerSec BrokerTopicMetrics 1
NumGroupsDead GroupMetadataManager 1
LogFlushRateAndTimeMs LogFlushStats 1
MetadataBatchSizes BrokerMetadataListener 1
null DiagnosticCommand 1
FailedProduceRequestsPerSec BrokerTopicMetrics 1
NumOffsets GroupMetadataManager 1
null MBeanServerDelegate 1
InvalidMessageCrcRecordsPerSec BrokerTopicMetrics 1
IncrementalFetchSessionEvictionsPerSec FetchSessionCache 1
null kafka-metrics-count 1
yammer-metrics-count KafkaServer 1
null Fetch 1
null Logging 1
cleaner-recopy-percent LogCleaner 1
RequestHandlerAvgIdlePercent KafkaRequestHandlerPool 1
MinFetchRate ReplicaAlterLogDirsManager 1
BytesOutPerSec BrokerTopicMetrics 1
PartitionsWithLateTransactionsCount ReplicaManager 1
UnderReplicatedPartitions ReplicaManager 1
null alterPartition-metrics 1
InvalidMagicNumberRecordsPerSec BrokerTopicMetrics 1
null OperatingSystem 1
CodeHeap 'non-nmethods' MemoryPool 1
BrokerState KafkaServer 1
NumGroupsPreparingRebalance GroupMetadataManager 1
MessagesInPerSec BrokerTopicMetrics 1
IsrShrinksPerSec ReplicaManager 1
linux-disk-read-bytes KafkaServer 1
FailedIsrUpdatesPerSec ReplicaManager 1
@chia7712
Copy link
Contributor

@chinghongfang 感謝建議,#1622 有提一個類似的作法,目前我們先以拉取 built-in 會用到的 beans就好,妳覺得如何

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants