o :/i @sUddlZddlZddlmZmZddlmZddlmZddl m Z m Z m Z ddl mZddlmZddlmZmZddlmZmZdd lmZdd lmZmZdd lmZdd lmZm Z dd l!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*m+Z+ee,Z-eee.gdfZ/e0dZ1e1e/BZ2GdddeZ3de4e2fddZ5Gddde3Z6Gddde3Z7Gddde7e6Z8Gddde6Z9Gddde6Z:e e Be BZ;eee.e;ffd#d$Z?d%e4e.d&e.de4e.fd'd(Z@d&e.de4e.fd)d*ZAGd+d,d,ZBdS)-N)ABCabstractmethod)Callable) TypeAlias)CounterGauge Histogram)CUDAGraphLogging)SupportsMetricsInfo VllmConfig)KVConnectorLoggingKVConnectorPrometheus) init_logger)STAT_LOGGER_PLUGINS_GROUPload_plugins_by_group) FinishReason)PerfMetricsLoggingPerfMetricsProm)unregister_vllm_metrics)CachingMetricsIterationStatsMultiModalCacheStatsPromptTokenStatsSchedulerStats)SpecDecodingLoggingSpecDecodingPromStatLoggerBaseAggregateStatLoggerBasec @s~eZdZdZeddedefddZe  ddedBd e dBd e dBd efd d Z eddZ ddZ dedefddZdS)raInterface for logging metrics. API users may define custom loggers that implement this interface. However, note that the `SchedulerStats` and `IterationStats` classes are not considered stable interfaces and may change in future versions. r vllm_config engine_indexcCdSN)selfrrr"r"d/lsinfo/ai/hellotax_ai/llm_service/venv_vllm/lib/python3.10/site-packages/vllm/v1/metrics/loggers.py__init__0zStatLoggerBase.__init__Nscheduler_statsiteration_statsmm_cache_stats engine_idxcCr r!r"r#r'r(r)r*r"r"r$record3szStatLoggerBase.recordcCr r!r"r#r"r"r$log_engine_initialized<r&z%StatLoggerBase.log_engine_initializedcCr r!r"r-r"r"r$log?r&zStatLoggerBase.logis_awakelevelcCr r!r")r#r0r1r"r"r$record_sleep_stateBr&z!StatLoggerBase.record_sleep_staterNr)__name__ __module__ __qualname____doc__rr intr%rrrr,r.r/r2r"r"r"r$r(s(  returncCsRg}ttD]\}}t|trt|ts!td|d|d||q|S)NzStat logger plugin z+ must be a subclass of StatLoggerBase (got z).) rritems isinstancetype issubclassr TypeErrorappend) factoriesname plugin_classr"r"r$!load_stat_logger_plugin_factoriesFs  rDc@s*eZdZdZededeefddZdS)rzNAbstract base class for loggers that aggregate across multiple DP engines.rengine_indexescCr r!r"r#rrEr"r"r$r%[r&z AggregateStatLoggerBase.__init__N) r5r6r7r8rr listr9r%r"r"r"r$rWsc @seZdZd"dedefddZddZdefd d Zd e fd d Z dede de fddZ e ddZ  d#dedBd e dBdedBdefddZddZddZddZd d!ZdS)$LoggingStatLoggerrrrcCs||_||_|tt|_t|_t|_ t|_ t |_ |jj }t||_d|_|jjjr>t|jjj|jjj|_d|_d|_d|_d|_|rUt||_dSdS)NF)rr_resettime monotonicrlast_scheduler_statsrprefix_caching_metrics connector_prefix_caching_metricsmm_caching_metricsrspec_decoding_loggingkv_transfer_configr kv_connector_loggingcudagraph_loggingobservability_configcudagraph_metricsr compilation_configcudagraph_modecudagraph_capture_sizeslast_prompt_throughputlast_generation_throughputengine_is_idle aggregated_enable_perf_statsrperf_metrics_logging)r#rrrRr"r"r$r%`s.  zLoggingStatLogger.__init__cCs"||_d|_d|_d|_d|_dSr4) last_log_timenum_prompt_tokensnum_generation_tokensnum_corrupted_reqsnum_preemptions)r#nowr"r"r$rJ~s  zLoggingStatLogger._resetr:cCs |jjjSr!)rrUenable_mfu_metricsr-r"r"r$r^s z$LoggingStatLogger._enable_perf_statsr(cCsF|j|jj7_|j|j7_|j|j7_|j|j7_dSr!)raprompt_token_statscomputedrbrcrdnum_preempted_reqs)r#r(r"r"r$_track_iteration_statssz(LoggingStatLogger._track_iteration_stats tracked_statsrecCs"||j}|dkr dSt||S)NrI)r`float)r#rkre delta_timer"r"r$_get_throughputs  z!LoggingStatLogger._get_throughputcCs d|jS)NzEngine {:03d}: )formatrr-r"r"r$ log_prefixs zLoggingStatLogger.log_prefixNr'r)r*cCs|r|||dur[|j|j|jdur|j|j|jdur*|j|j|j}r5|j ||j durF|j durF|j |j |j sL||_ |j}r[|r[|j||re|j|dSdS)zLog Stats to standard output.N)rjrNobserveprefix_cache_statsconnector_prefix_cache_statsrOspec_decoding_statsrQkv_connector_statsrSrTcudagraph_statsr]rM perf_statsr^r_rP)r#r'r(r)r*rurwr"r"r$r,s,        zLoggingStatLogger.recordcCsVt}||j|}||j|}||t|||j|jf |_ ||_||_dSr!) rKrLrnrarbrJanyrZr[r\)r#reprompt_throughputgeneration_throughputr"r"r$ _update_statss  zLoggingStatLogger._update_statscCr r!r"r-r"r"r$aggregate_scheduler_statsz+LoggingStatLogger.aggregate_scheduler_statscCsd|||jrtjntj}gd}|j|j|jj |jj g}|j dkr1| d| |j | ddg| |jjd|jjdgtjrU| d| |j|jjsg| d| |jjd|jjsy| d | |jjd||jd |g|R|jj|d |jj|d |jdur|jj|d |r|jj||jd dSdS) N)z$Avg prompt throughput: %.1f tokens/sz(Avg generation throughput: %.1f tokens/szRunning: %d reqszWaiting: %d reqsrzPreemptions: %dzGPU KV cache usage: %.1f%%zPrefix cache hit rate: %.1f%%dzCorrupted: %d reqsz&External prefix cache hit rate: %.1f%%zMM cache hit rate: %.1f%%z, )log_fn)rrp)r{r|r\loggerdebuginforZr[rMnum_running_reqsnum_waiting_reqsrdr@extendkv_cache_usagerNhit_rateenvsVLLM_COMPUTE_NANS_IN_LOGITSrcrOemptyrPrpjoinrQr/rSrTr^r_)r#r log_partslog_argsr"r"r$r/sV          zLoggingStatLogger.logcCs(|jjjrtd|j|jjjdSdS)NzSEngine %03d: vllm cache_config_info with initialization after num_gpu_blocks is: %d)r cache_confignum_gpu_blocksrrrr-r"r"r$r.s z(LoggingStatLogger.log_engine_initializedr3r4)r5r6r7r r9r%rJboolr^rrjrlrnpropertyrprrr,r{r|r/r.r"r"r"r$rH_s.   # :rHc @seZdZdedeefddZeddZde fdd Z dd e d Bd e d Bde d BdefddZddZddZddZd S)AggregatedLoggingStatLoggerrrEcCs2||_dd|jD|_tj||ddd|_dS)NcSsi|]}|tqSr")r.0idxr"r"r$ sz8AggregatedLoggingStatLogger.__init__..)rT)rElast_scheduler_stats_dictrHr%r]rFr"r"r$r%s  z$AggregatedLoggingStatLogger.__init__cCsdt|jS)Nz{} Engines Aggregated: )rolenrEr-r"r"r$rp&sz&AggregatedLoggingStatLogger.log_prefixr:cCsdS)NFr"r-r"r"r$r^*r}z.AggregatedLoggingStatLogger._enable_perf_statsNrr'r(r)r*cCsH||jvr td|dStj|||||d|dur"||j|<dSdSNzUnexpected engine_idx: %dr)r*)rErwarningrHr,rr+r"r"r$r,.s  z"AggregatedLoggingStatLogger.recordcCsht|_|jD]}|jj|j7_|jj|j7_|jj|j7_q |jjt|j_dSr!)rrMrvaluesrrrr)r#rMr"r"r$r|Bsz5AggregatedLoggingStatLogger.aggregate_scheduler_statscCst|dSr!)rHr/r-r"r"r$r/PszAggregatedLoggingStatLogger.logcCs,|jjjrtdt|j|jjjdSdS)NzR%d Engines: vllm cache_config_info with initialization after num_gpu_blocks is: %d)rrrrrrrEr-r"r"r$r.Ss z2AggregatedLoggingStatLogger.log_engine_initializedr4)r5r6r7r rGr9r%rrprr^rrrr,r|r/r.r"r"r"r$rs.    rc @sfeZdZdedeededdfddZ  dd edBd e dBd e dBd efd dZ ddZ ddZ dS)PerEngineStatLoggerAdapterrrEper_engine_stat_logger_factoryr:NcCs*i|_||_|D] }||||j|<qdSr!)per_engine_stat_loggersrE)r#rrErrr"r"r$r%^s z#PerEngineStatLoggerAdapter.__init__rr'r(r)r*cCs6||jvr td|dS|j|j||||ddSr)rrrr,r+r"r"r$r,ks    z!PerEngineStatLoggerAdapter.recordcC|jD]}|qdSr!)rrr/r#per_engine_stat_loggerr"r"r$r/| zPerEngineStatLoggerAdapter.logcCrr!)rrr.rr"r"r$r.rz1PerEngineStatLoggerAdapter.log_engine_initializedr4)r5r6r7r rGr9PerEngineStatLoggerFactoryr%rrrr,r/r.r"r"r"r$r]s.   rc @seZdZeZeZeZe Z e Z e Z ddedeedBfddZdedefdd Z  dd edBd edBd edBdefddZddedefddZddZdS)PrometheusStatLoggerNrrEc*s@|durdg}||_t||_|jj|_|jj|_ddg}|jj|jj }fdd|D}| |j |||_ | ||||_|||||_|jddd|d }t|||_|jd d d|d }t|||_|jd d |dgddi|_gd}|D]fdd|D|j<q}||jddd|d } t| ||_tjr|jdd|d} t| ||_|jdd|d} t| ||_|jdd|d} t| ||_|jdd|d} t| ||_|jdd|d}t|||_|jdd |d}t|||_ |jd!d"|d}t|||_!|jd#d$|d}t|||_"|jd%d&|d}t|||_#|jd'd(|d)gdi|_$t%j&D]fd*d|D|j$<q=|jd+d,|d}t|||_'|jd-d.|d}t|||_(|jd/d0|d}t|||_)i|_*|jd1d2|d3gdt+D]fd4d|D|j*<q|j,d5d&t-||d6}t|||_.|j,d7d0t-||d6}t|||_/|j,d8d9gd:|d6}t|||_0|j,d;dgd?|d6}t|||_2|j,d@dAt-||d6}t|||_3|j,dBdCgdD|d6}t|||_4|j,dEdFgdG|d6}t|||_5|j,dHdIgdG|d6}t|||_6gdJ}|j,dKdL||d6} t| ||_7|j,dMdN||d6}!t|!||_8|j,dOdP||d6}"t|"||_9|j,dQdR||d6}#t|#||_:|j,dSdT||d6}$t|$||_;|j,dUdVt-||d6}%t|%||_<|jrgdW}&|j,dXdY|&|d6}'t|'||_=|j,dZd[|&|d6}(t|(||_>|j,d\d]|&|d6})t|)||_?n i|_=i|_>i|_?d|_@|jAdurtB|jd^krtCDd_d`|_Eda|_Fdb|_G|jAjH|_I|jdcddde|jE|jF|jGgd |_@dSdS)fNr model_nameenginecsi|] }|t|gqSr")strr)rr"r$rsz1PrometheusStatLogger.__init__..zvllm:num_requests_runningz.Number of requests in model execution batches. mostrecentrB documentationmultiprocess_mode labelnameszvllm:num_requests_waitingz+Number of requests waiting to be processed.zvllm:engine_sleep_statezEngine sleep state; awake = 0 means engine is sleeping; awake = 1 means engine is awake; weights_offloaded = 1 means sleep level 1; discard_all = 1 means sleep level 2. sleep_state)rBrrr)awakeweights_offloaded discard_allcsi|] }|j|dqS))rrr)labelsr)gauge_engine_sleep_statersr"r$rs zvllm:kv_cache_usage_percz*KV-cache usage. 1 means 100 percent usage.zvllm:corrupted_requestszMCorrupted requests, in terms of total number of requests with NaNs in logits.)rBrrzvllm:prefix_cache_queriesz;Prefix cache queries, in terms of number of queried tokens.zvllm:prefix_cache_hitsz7Prefix cache hits, in terms of number of cached tokens.z"vllm:external_prefix_cache_querieszsExternal prefix cache queries from KV connector cross-instance cache sharing, in terms of number of queried tokens.zvllm:external_prefix_cache_hitszoExternal prefix cache hits from KV connector cross-instance cache sharing, in terms of number of cached tokens.zvllm:mm_cache_queriesz?Multi-modal cache queries, in terms of number of queried items.zvllm:mm_cache_hitsz;Multi-modal cache hits, in terms of number of cached items.zvllm:num_preemptionsz0Cumulative number of preemption from the engine.zvllm:prompt_tokensz#Number of prefill tokens processed.zvllm:prompt_tokens_by_sourcez"Number of prompt tokens by source.sourcecs i|] }|t|qSr"rrr)counter_prompt_tokens_by_sourcerrr"r$r_s  zvllm:prompt_tokens_cachedz2Number of cached prompt tokens (local + external).zvllm:prompt_tokens_recomputedz4Number of cached tokens recomputed for forward pass.zvllm:generation_tokensz&Number of generation tokens processed.zvllm:request_successz)Count of successfully processed requests.finished_reasonc s$i|]}|t|tqSr"rr)counter_request_success_baserreasonr"r$rs zvllm:request_prompt_tokens)rBrbucketsrzvllm:request_generation_tokenszvllm:iteration_tokens_totalz.Histogram of number of tokens per engine_step.)  @iiiii i@z&vllm:request_max_num_generation_tokensz;Histogram of maximum number of requested generation tokens.zvllm:request_params_nz%Histogram of the n request parameter.)r zvllm:request_params_max_tokensz.Histogram of the max_tokens request parameter.z vllm:time_to_first_token_secondsz,Histogram of time to first token in seconds.)MbP?{Gzt?{Gz?{Gz?g{Gz?gQ?g{Gz?皙?g????@@@$@4@D@T@gd@g@g@z vllm:inter_token_latency_secondsz,Histogram of inter-token latency in seconds.)rg?皙?g333333?rg333333?皙?333333?g?rrrrrrrrrrz*vllm:request_time_per_output_token_secondsz7Histogram of time_per_output_token_seconds per request.)rrg?rg?g@rrrg.@rg>@rgI@gN@g^@gn@g~@g@g@g@z vllm:e2e_request_latency_secondsz,Histogram of e2e request latency in seconds.zvllm:request_queue_time_secondsz5Histogram of time spent in WAITING phase for request.z#vllm:request_inference_time_secondsz5Histogram of time spent in RUNNING phase for request.z!vllm:request_prefill_time_secondsz5Histogram of time spent in PREFILL phase for request.z vllm:request_decode_time_secondsz4Histogram of time spent in DECODE phase for request.z'vllm:request_prefill_kv_computed_tokenszMHistogram of new KV tokens computed during prefill (excluding cached tokens).)rgMb`?rrrrrrrrrrrr<xi,iXiizvllm:kv_block_lifetime_secondsz|Histogram of KV cache block lifetime from allocation to eviction. Sampled metrics (controlled by --kv-cache-metrics-sample).z'vllm:kv_block_idle_before_evict_secondszqHistogram of idle time before KV cache block eviction. Sampled metrics (controlled by --kv-cache-metrics-sample).zvllm:kv_block_reuse_gap_secondszHistogram of time gaps between consecutive KV cache block accesses. Only the most recent accesses are recorded (ring buffer). Sampled metrics (controlled by --kv-cache-metrics-sample).rzfvllm:lora_requests_info prometheus metrics may be incorrect/misleading with data parallel deployments.max_lorawaiting_lora_adaptersrunning_lora_adapterszvllm:lora_requests_infozRunning stats on lora requests.sum)JrErrrUshow_hidden_metricskv_cache_metricskv_cache_metrics_enabled model_configserved_model_name max_model_len_spec_decoding_clsspeculative_configspec_decoding_prom_kv_connector_clskv_connector_prom_perf_metrics_clsperf_metrics_prom _gauge_clsmake_per_enginegauge_scheduler_runninggauge_scheduler_waitingrr2gauge_kv_cache_usagerr _counter_clscounter_corrupted_requestscounter_prefix_cache_queriescounter_prefix_cache_hits&counter_connector_prefix_cache_queries#counter_connector_prefix_cache_hitscounter_mm_cache_queriescounter_mm_cache_hitscounter_num_preempted_reqscounter_prompt_tokensrr ALL_SOURCEScounter_prompt_tokens_cached counter_prompt_tokens_recomputedcounter_generation_tokenscounter_request_successr_histogram_clsbuild_1_2_5_buckets#histogram_num_prompt_tokens_request'histogram_num_generation_tokens_requesthistogram_iteration_tokens+histogram_max_num_generation_tokens_requesthistogram_n_requesthistogram_max_tokens_requesthistogram_time_to_first_tokenhistogram_inter_token_latency'histogram_request_time_per_output_tokenhistogram_e2e_time_requesthistogram_queue_time_request histogram_inference_time_requesthistogram_prefill_time_requesthistogram_decode_time_request%histogram_prefill_kv_computed_requesthistogram_kv_block_lifetime$histogram_kv_block_idle_before_evicthistogram_kv_block_reuse_gapgauge_lora_info lora_configrrrlabelname_max_loralabelname_waiting_lora_adapterslabelname_running_lora_adapters max_lorasr)*r#rrErrper_engine_labelvaluesrrrrrrrrrrrrrrrrrrrrrr r r r request_latency_bucketsr rrrrrkv_cache_residency_bucketsrrrr")rrrrrrrr$r%s            zPrometheusStatLogger.__init__r= config_objcCs|}d|d<d\}}|dkrd}d}|dusJd||j||d|d }|jD]}|}t||d<|jd i|d q-dS) NrNNrzvllm:cache_config_infoz(Information of the LLMEngine CacheConfigzUnknown metrics info type rrrr") metrics_inforkeysrErrset)r#r=rr"rBr info_gaugerr"r"r$log_metrics_infos$  z%PrometheusStatLogger.log_metrics_inforr'r(r)r*cCs|dur|j||j|j||j|j||j|j||j j |j ||j j |j durL|j||j j |j||j j |jdurY|j|j||jdurf|j|j||jdurs|j|j||jr|jr|j|}|j|}|j|}|jD]}||j||j|jD]} || qq|jdurd |j!"} d |j#"} |j$| |j%| |j&|j'i} |jj(di| )|dur|j*||j |j+||j |durdSt,j-r|j.||j/|j0||j1|j2||j3|j4} t5j6D]}|j7||| 8|q|j9|| j:|j;|| j<|j=||j>|j?||j3|j>|j@D] }|jA||qL|jBD] }|jC||q[|jDD] }|jE||qj|jFD] }|jG||qy|jHD]s}|jI|jJ||jK||jL|jM||jN|jO||jP|jQ||jR|jS||jT|j3tU|jVd}|jW|||jX||j3|jY||j>|jZ||j[|j\r|j]||j\qdS)zLog to prometheus.N,rr")^rr$rrrrrrincrrqueriesrhitsrsrrrtrrqrurrwrrkv_cache_eviction_eventsrrrlifetime_seconds idle_secondsreuse_gaps_secondsrrrr#rrrrrrset_to_current_timerrrrrrcrrirrargrrr get_by_sourcer cached_tokensrrecomputed_tokensrrbrmax_num_generation_tokens_iterr n_params_iterrtime_to_first_tokens_iterr inter_token_latencies_iterr finished_requestsr finish_reasonr  e2e_latencyr queued_timer prefill_timerinference_timer decode_timemaxnum_cached_tokensrrrr mean_time_per_output_tokenmax_tokens_paramr )r#r'r(r)r* lifetime_hist idle_hist reuse_histeventgaprrlora_info_labelsptsrmax_gen_tokensn_paramttftitlfinished_requestprefill_kv_computedr"r"r$r,s                                           zPrometheusStatLogger.recordsleepr1cCs~d}d}d}|dkrd}|dkrd}n|dkrd}|jD] }|jd|||jd|||jd||qdS)Nrrrrrr)rErr$)r#rOr1rrrr*r"r"r$r2s  z'PrometheusStatLogger.record_sleep_statecCs|d|jjdS)Nr)r&rrr-r"r"r$r.sz+PrometheusStatLogger.log_engine_initializedr!r4rr)r5r6r7rrrrrrrrr rrrr rGr9r%rr r&rrrr,r2r.r"r"r"r$rs@  e  r PromMetricmetric engine_idxsrcsfdd|DS)Ncsi|] }|t|qSr"rrrRrr"r$rsz#make_per_engine..r")rRrSrr"rTr$rsr mantissa_lst max_valuecCsDd}g} |D]}|d|}||kr||q|S|d7}q)z Builds a list of buckets with increasing powers of 10 multiplied by mantissa values until the value exceeds the specified maximum. rTrr)r@)rUrVexponentrmvaluer"r"r$ build_bucketss  rZcCstgd|S)zR Example: >>> build_1_2_5_buckets(100) [1, 2, 5, 10, 20, 50, 100] )rrr)rZ)rVr"r"r$rsrc @seZdZdZ     ddedeedBdeedBd ed ed ef d d Z  dde dBde dBde dBdedBfddZ ddedefddZddZddZdS) StatLoggerManagera StatLoggerManager: Logging happens at the level of the EngineCore (per scheduler). * DP: >1 EngineCore per AsyncLLM - loggers for each EngineCore. * With Local Logger, just make N copies for N EngineCores. * With Prometheus, we need a single logger with N "labels" This class abstracts away this implementation detail from the AsyncLLM, allowing the AsyncLLM to just call .record() and .log() to a simple interface. NTFrrrScustom_stat_loggersenable_default_loggersaggregate_engine_logging client_countc Cs|r|ndg|_g|_g}|dur|||r3ttjr3|dkr(tdn |r,tnt }| |d} |D])} t | t rRt | trR| ||jd} t | trQd} nt||j| d} |j | q7| so|j t||jdSdS)NrrzfAsyncLLM created with api_server_count more than 1; disabling stats logging to avoid incomplete stats.F)rrET)rrEr)rE stat_loggersrr isEnabledForloggingINFOrrrHr@r<r=r>rrr) r#rrSr\r]r^r_stat_logger_factoriesdefault_logger_factorycustom_prometheus_loggerstat_logger_factoryglobal_stat_loggerr"r"r$r%sL      zStatLoggerManager.__init__r'r(r)r*cCs.|durd}|jD] }|j||||dq dS)Nrr)r`r,)r#r'r(r)r* stat_loggerr"r"r$r,s zStatLoggerManager.recordrrOr1cCs|jD]}|||qdSr!)r`r2)r#rOr1rr"r"r$r2+s z$StatLoggerManager.record_sleep_statecC|jD]}|qdSr!)r`r/)r#rr"r"r$r//  zStatLoggerManager.logcCrjr!)r`r.)r# agg_loggerr"r"r$r.3rkz(StatLoggerManager.log_engine_initialized)NNTFrr!rP)r5r6r7r8r rGr9StatLoggerFactoryrr%rrrr,r2r/r.r"r"r"r$r[sD   7  r[)CrbrKabcrrcollections.abcrtypingrprometheus_clientrrr vllm.envsrvllm.compilation.cuda_graphr vllm.configr r 4vllm.distributed.kv_transfer.kv_connector.v1.metricsr r vllm.loggerr vllm.pluginsrrvllm.v1.enginervllm.v1.metrics.perfrrvllm.v1.metrics.prometheusrvllm.v1.metrics.statsrrrrrvllm.v1.spec_decode.metricsrrr5rr9rr=AggregateStatLoggerFactoryrmrrGrDrrHrrrrQ__annotations__objectdictrrZrr[r"r"r"r$s`        ;D(7