o :/iE@sPddlZddlmZmZddlmZmZddlmZm Z ddl m Z ddl m Z ddlmZddlmZer@ddlmZmZmZeGd d d ZGd d d ZeGd ddeZeGdddeZeGdddZeGdddZeGdddZeGdddZeGdddZGdddZGdddZ Gdd d Z!dS)!N) defaultdictdeque) dataclassfield) TYPE_CHECKINGAny) CUDAGraphStat) PerfStats)SpecDecodingStats)EngineCoreEventEngineCoreOutput FinishReasonc@sHeZdZUdZdZeed< dZeed< dZ eed< dZ eed<dS) BaseCacheStatszStores cache hit statistics.FresetrrequestsquerieshitsN) __name__ __module__ __qualname____doc__rbool__annotations__rintrrrrb/lsinfo/ai/hellotax_ai/llm_service/venv_vllm/lib/python3.10/site-packages/vllm/v1/metrics/stats.pyrs     rcsfeZdZdZddeddffdd Zdefd d Zd d Ze de fd dZ e de fddZ ZS)CachingMetricszMetrics for caching with a hit rate of the most recent N requests. Args: interval: The number of the most recent requests to aggregate. Defaults to 1000. max_recent_requestsreturnNcs<t||_d|_d|_d|_tttttf|_ dSNr) super__init__raggregated_requestsaggregated_query_totalaggregated_query_hitrtupler query_queue)selfr __class__rrr"*s zCachingMetrics.__init__statscCs|jr||jdkrdS|j|j|j|jf|j|j7_|j|j7_|j|j7_t |jdkrk|j|j kro|j \}}}|j|8_|j|8_|j|8_t |jdkrm|j|j ks?dSdSdSdS)aObserve the prefix caching for a set of requests. This function is called with information gathered when new requests are being scheduled and are looking for computed blocks. When there are more than `max_recent_requests` requests, the oldest set of requests are removed from the metrics. Args: stats: The prefix cache stats. rN) rrr'appendrrr#r$r%lenrpopleft)r(r+ old_requests old_queriesold_hitsrrrobserve6s$  zCachingMetrics.observecCs d|_d|_d|_|jdS)zReset the metrics.rN)r#r$r%r'clearr(rrrr^szCachingMetrics.resetcCs |jdkS)z.Return true if no requests have been observed.r)r#r5rrremptyes zCachingMetrics.emptycCs|jdkrdS|j|jS)z/Calculate the hit rate for the past N requests.r)r$r%r5rrrhit_ratejs  zCachingMetrics.hit_rate)r)rrrrrr"rr3rpropertyrr6floatr8 __classcell__rrr)rr#s (rc@sVeZdZUdZdZeed< dZeed< dZeed< dedede d d fd d Z d S) PrefixCacheStatsz Stores prefix cache hit statistics. - `reset`: Whether `reset_prefix_cache` was invoked. - `queries`: Refers to the number of tokens that were queried. rpreempted_requestspreempted_queriespreempted_hits num_tokensnum_hits preemptedrNcCs`|r|jd7_|j|7_|j|7_dS|jd7_|j|7_|j|7_dSz-Aggregate request information into the stats.r,N)r=r>r?rrr)r(r@rArBrrrrecordszPrefixCacheStats.record) rrrrr=rrr>r?rrDrrrrr<rs    r<c@s&eZdZdZdededdfddZdS)MultiModalCacheStatsz Stores multi-modal cache hit statistics. - `reset`: Whether `reset_mm_cache` was invoked. - `queries`: Refers to the number of multi-modal data items that were queried. num_queriesrArNcCs.|jd7_|j|7_|j|7_dSrC)rrr)r(rFrArrrrDszMultiModalCacheStats.record)rrrrrrDrrrrrEsrEc@s2eZdZUdZeed<eed<eedfed<dS)KVCacheEvictionEventz&Single KV cache block eviction sample.lifetime_seconds idle_seconds.reuse_gaps_secondsN)rrrrr:rr&rrrrrGs rGc@seZdZUdZdZeed<dZeed<dZeed<dZ eed<dZ e ed<dZ e ed <e ed Zeed <d Zed Bed <e ed Zeeed<d Zed Bed<d Zeeefd Bed<e ed Zeeefed<e ed Zeeefed<d Zed Bed<d Zed Bed<d S)SchedulerStatsz$Stats associated with the scheduler.rnum_running_reqsnum_waiting_reqs step_counter current_waver7kv_cache_usageencoder_cache_usage)default_factoryprefix_cache_statsNconnector_prefix_cache_statskv_cache_eviction_eventsspec_decoding_statskv_connector_statswaiting_lora_adaptersrunning_lora_adapterscudagraph_stats perf_stats) rrrrrLrrrMrNrOrPr:rQrr<rSrTlistrUrGrVr rWdictstrrrXrYrZrr[r rrrrrKs"       rKc@sreZdZUdZdZeed<dZeed<dZ eed<dZ eed<dZ eed<dZ eed <dZ eed <d Zeed <d S)RequestStateStatsz3Stats that need to be tracked across delta updates.rnum_generation_tokensr7 arrival_time queued_ts scheduled_tsfirst_token_ts last_token_tsfirst_token_latencyF is_corruptedN)rrrrr`rrrar:rbrcrdrerfrgrrrrrr_s        r_c@seZdZUdZded<dZeed<dZeed<dZ eed<d Z ed Bed <dZ eed <dZ eed <dZ eed <dZeed<dZeed<dZeed<dZeed<d S)FinishedRequestStatsz)Stats associated with a finished request.r finish_reasonr7 e2e_latencyrnum_prompt_tokensr`Nmax_tokens_param queued_time prefill_timeinference_time decode_timemean_time_per_output_tokenFrgnum_cached_tokens)rrrrrrjr:rkrr`rlrmrnrorprqrgrrrrrrrrhs          rhc@seZdZUdZdZeedfed<dZe ed<dZ e ed<dZ e ed<dZ e ed <dZ e ed <dZe ed <d e d e de ddfddZdede fddZdS)PromptTokenStatsabBreakdown of prompt tokens by source. Fields: computed: Tokens prefilled locally (actual compute work). local_cache_hit: Tokens from local prefix cache. external_kv_transfer: Tokens from external KV transfer. cached_tokens: Tokens skipped during prefill (from scheduler). recomputed_tokens: Cached tokens that were recomputed (see below). total: Total prompt tokens. Invariants: computed + local_cache_hit + external_kv_transfer - recomputed_tokens = total local_cache_hit + external_kv_transfer - recomputed_tokens = cached_tokens  local_computelocal_cache_hitexternal_kv_transfer. ALL_SOURCESrcomputedrvrw cached_tokensrecomputed_tokenstotalrrnum_external_computed_tokens prompt_lenrNcCsx|d|krdnd}|j||7_|j|7_|j|||7_|j|7_|j|7_|j|7_dS)z#Update stats from a prefill output.r,rN)ryrwrvrzr{r|)r(rrr}r~ recomputedrrrupdate_from_outputs  z#PromptTokenStats.update_from_outputsourcecCs0|j|j|jd}||vrtd|||S)z Get token count by source label.rtzUnknown source: )ryrvrw ValueError)r(r source_maprrr get_by_source#szPromptTokenStats.get_by_source)rrrrrxr&r^rryrrvrwrzr{r|rrrrrrrss&        rsc@seZdZdZddZdefddZedefddZ d e de fd d Z d d de de dede dddedBfddZdededde de dddedBf ddZ d%ddd ed!edBde d"ef d#d$ZdS)&IterationStatsz8Stats associated with a single set of EngineCoreOutputs.cCsFt|_d|_t|_d|_g|_g|_g|_g|_ g|_ d|_ dSr ) timeiteration_timestampr`rsprompt_token_statsnum_preempted_reqsfinished_requestsmax_num_generation_tokens_iter n_params_itertime_to_first_tokens_iterinter_token_latencies_iternum_corrupted_reqsr5rrrr"2s  zIterationStats.__init__rcCs0dddt|D}|jjd|dS)Nz, css"|] \}}|d|VqdS)=Nr).0kvrrr ?s z*IterationStats.__repr__..())joinvarsitemsr*r)r(field_to_value_strrrr__repr__>szIterationStats.__repr__cCs|jjS)z1Total prompt tokens (for backward compatibility).)rr|r5rrrrkBsz IterationStats.num_prompt_tokensstartcCs |j|S)z=Calculate an interval relative to this iteration's timestamp.)r)r(rrrr _time_sinceGs zIterationStats._time_sinceoutputr engine_core_timestamp is_prefillingr~ req_stats lora_statesLoRARequestStates lora_nameNc Cst|j}|j|7_|r(|jj|j|j|d||j} |j | | |_ |j|7_t j r=|js=|jdkr=d|_|jdurN||j|j|||||rT||_n ||j} |j | ||_dS)N)rrr}r~rT)r. new_token_idsr`rrrrr}rrarr-rfenvsVLLM_COMPUTE_NANS_IN_LOGITSrgnum_nans_in_logitseventsupdate_from_events request_idrdrer) r(rrrr~rrrnum_new_generation_tokensrfitlrrrrKs@         z!IterationStats.update_from_outputreq_idrr c Csddlm}|D]<}|j|jkr|j|_|||q|j|jkr1|jdkr*|j|_| ||q|j|j krD|j d7_ |||qdS)Nr)EngineCoreEventTyper7r,) vllm.v1.enginertypeQUEUED timestamprbrequest_waiting SCHEDULEDrcrequest_running PREEMPTEDr) r(rrrrrrreventrrrrs      z!IterationStats.update_from_eventsrrir rkrlrrc Cs||j}|j|j}|j|j}|j|j} |j|j} |jddkr,| |jdnd} t||||j|||| | | |j|d } |j | |jrR|j d7_ dSdS)Nr,r) rirjrkr`rlrmrnrorprqrgrr) rrarcrbrdrer`rhrgrr-r) r(rirkrlrrrrjrmrnrprorq finished_reqrrrupdate_from_finished_requests6      z+IterationStats.update_from_finished_request)r)rrrrr"r^rr9rrkr:rrr_rr\rrrrrrr/s^  7 rc@s@eZdZdZddZdededefddZed efd d Z d S) LoRAStatsz9Tracks waiting and running request IDs for a single LoRA.cCst|_t|_dSN)setwaitingrunningr5rrrr"s zLoRAStats.__init__rrrcCsN|r|rJ|r|j|n|j||r|j|dS|j|dSr)radddiscardr)r(rrrrrrupdates  zLoRAStats.updatercCs|jp|j Srrrr5rrrr6szLoRAStats.emptyN) rrrrr"r^rrr9r6rrrrrs  rc@seZdZdZddefddZdededBd ed efd d ZdededBfd dZdededBfddZ dededBfddZ de dBfddZ dS)rz1A per-LoRA count of running and waiting requests.F log_statscCs||_tt|_dSr)rrrr)r(rrrrr"szLoRARequestStates.__init__rrNrrcCs@|jr|dur dS|j|}|||||jr|j|=dSdSr)rrrr6)r(rrrr lora_statsrrr_request_updates  z!LoRARequestStates._request_updatecC|j||ddddS)NTFrrr(rrrrrrz!LoRARequestStates.request_waitingcCr)NFTrrrrrrrrz!LoRARequestStates.request_runningcCs|j||ddddS)NFrrrrrrrequest_finishedrz"LoRARequestStates.request_finishedscheduler_statscCsJ|jr|dur dS|jD]\}}t|j|j|<t|j|j|<qdSr)rrrr.rrXrrY)r(rrr+rrrupdate_scheduler_statss z(LoRARequestStates.update_scheduler_stats)F) rrrrrr"r^rrrrrKrrrrrrs   r)"r collectionsrr dataclassesrrtypingrr vllm.envsrvllm.compilation.cuda_graphrvllm.v1.metrics.perfr vllm.v1.spec_decode.metricsr rr r r rrr<rErGrKr_rhrsrrrrrrrs>    O>!