o :/i(@sddlmZddlmZmZddlmZmZmZm Z ddl m Z ddl m Z e eZedZedZedZed Zed Ze Gd d d Zd S))field)ClassVarLiteral)FieldSkipValidationfield_validatormodel_validator)config) init_logger)autofloat16bfloat16fp8fp8_e4m3fp8_e5m2fp8_inc fp8_ds_mla)r float32r )allalignnone)sha256 sha256_cborxxhash xxhash_cbor)nativelmcachec@s eZdZUdZdZeeed<dZe eed< e dddZ e ed< e d d d d Zeed < dZeed< dZe ed< dZedBed< dZedBed< dZe ed< dZeed< dZe ed< dZedBed< dZedBed< e dd dZedBed< dZeed< dZeed< dZe ed< e dddZ!edBed < e dddZ"edBed!< dZ#e ed"< dZ$edBed#< dZ%edBed$< d%Z&e'ed&< d'e(fd(d)Z)d*d+Z*e dddZ+e ed,< e,d-d.d3d/d0Z-e.dd-d.e/ded'efd1d2Z0dS)4 CacheConfigzConfiguration for the KV cache.DEFAULT_BLOCK_SIZEN block_sizeF)defaultinituser_specified_block_sizeg?r)r!gtlegpu_memory_utilizationr cache_dtypeis_attention_freenum_gpu_blocks_overridesliding_windowTenable_prefix_cachingrprefix_caching_hash_algocalculate_kv_scalescpu_kvcache_space_bytesmamba_page_size_padded)r!r%mamba_block_sizemamba_cache_dtypemamba_ssm_cache_dtypermamba_cache_modenum_gpu_blocksnum_cpu_blockskv_sharing_fast_prefillkv_cache_memory_byteskv_offloading_sizerkv_offloading_backendreturncCs*hd}ddlm}m}|||}||S)a WARNING: Whenever a new field is added to this config, ensure that it is included in the factors list if it affects the computation graph. Provide a hash that uniquely identifies all the configs that affect the structure of the computation graph from input ids/embeddings to the final hidden states, excluding anything before input ids/embeddings and after the final hidden states. > r6r5r)_block_size_resolvedr,r'r0r/r7r*r-r#r)get_hash_factors hash_factors)vllm.config.utilsr=r>)selfignored_factorsr=r>factorsrC^/lsinfo/ai/hellotax_ai/llm_service/venv_vllm/lib/python3.10/site-packages/vllm/config/cache.py compute_hashs  zCacheConfig.compute_hashcCsdd|jDS)NcSsi|] \}}|t|qSrC)str).0keyvaluerCrCrD sz,CacheConfig.metrics_info..)__dict__itemsr@rCrCrD metrics_infoszCacheConfig.metrics_infor<after)modecCsH|jr|St|dd|jdurt|d|j|St|dd|S)Nr<Tr r#)r<object __setattr__r rrMrCrCrD_apply_block_size_defaults z%CacheConfig._apply_block_size_defaultcCs|dr td|S)NrzUsing fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor.) startswithloggerinfo)clsr(rCrCrD_validate_cache_dtypes z!CacheConfig._validate_cache_dtype)r;r)1__name__ __module__ __qualname____doc__rrint__annotations__r rrr#boolrr'floatr( CacheDTyper)r*r+r,r-PrefixCachingHashAlgor.r/r0r1r2 MambaDTyper3r4MambaCacheModer5r6r7r8r9r:KVOffloadingBackendrFrErNr<rrSr classmethodrXrCrCrCrDrsp             #  rN) dataclassesrtypingrrpydanticrrrrr?r vllm.loggerr rYrUrarcrdrbrerrCrCrCrDs