o :/iz@svdZddlZddlmZddlmZmZddlmZedZ eGdddZ eGd d d Z eGd d d Z dS) z*Configuration for model weight offloading.N)Literal)Fieldmodel_validator)config)autouvaprefetchc@s>eZdZUdZedddZeed< eedZ ee ed<dS)UVAOffloadConfigzConfiguration for UVA (Unified Virtual Addressing) CPU offloading. Uses zero-copy access from CPU-pinned memory. Simple but requires fast CPU-GPU interconnect. rdefaultgecpu_offload_gbdefault_factorycpu_offload_paramsN) __name__ __module__ __qualname____doc__rr float__annotations__setrstrrr`/lsinfo/ai/hellotax_ai/llm_service/venv_vllm/lib/python3.10/site-packages/vllm/config/offload.pyr s  r c@sjeZdZUdZedddZeed< edddZeed< edddZ eed< ee dZ e e ed <d S) PrefetchOffloadConfigz~Configuration for prefetch-based CPU offloading. Groups layers and uses async H2D prefetch to hide transfer latency. rr offload_group_sizeoffload_num_in_groupoffload_prefetch_steproffload_paramsN) rrrrrrintrrrrr rrrrrr/s rc@sjeZdZUdZdZeed< eedZ eed< ee dZ e ed< e dddd d Z d efd d ZdS) OffloadConfigzEConfiguration for model weight offloading to reduce GPU memory usage.roffload_backendrrrafter)modereturncCs|jdks |jjdkr2|jj|jjkr"td|jjd|jjd|jjdkr2td|jjd|jjdk}|jjdk}|jd krN|rNtj d d d |S|jdkr^|r^tj d d d |S|jdkrn|rn|rntj dd d |S)z+Validate offload configuration constraints.rrzoffload_num_in_group (z!) must be <= offload_group_size ()rzoffload_prefetch_step (zK) must be >= 1 when prefetch offloading is enabled (offload_group_size > 0)rz]Prefetch offload fields are set but offload_backend='uva'. Prefetch settings will be ignored.) stacklevelzXUVA offload fields are set but offload_backend='prefetch'. UVA settings will be ignored.rzBoth UVA and prefetch offload fields are set with offload_backend='auto'. Prefetch backend will be selected. Set offload_backend explicitly to suppress this warning.) r#rrr ValueErrorrrr warningswarn)self uva_activeprefetch_activerrrvalidate_offload_configasB     z%OffloadConfig.validate_offload_configcCs*ddlm}m}||td}||}|S)a Provide a hash that uniquely identifies all the offload configs. All fields are included because PrefetchOffloader patches module forwards and inserts custom ops (wait_prefetch, start_prefetch) into the computation graph. Changing any offload setting can alter which layers are hooked and how prefetch indices are computed, so the compilation cache must distinguish them. r)get_hash_factors hash_factors)ignored_factors)vllm.config.utilsr1r2r)r-r1r2factorshash_strrrr compute_hashs zOffloadConfig.compute_hashN)r&r")rrrrr#OffloadBackendrrr rrrrr0rr7rrrrr"Os   )r") rr+typingrpydanticrrr4rr8r rr"rrrrs