o :/i @s(ddlmZmZddlmZmZddlmZddlm Z m Z m Z m Z m Z mZddlZddlZddlmZe r`ddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd l m!Z!Gddde"eZ#GdddZ$GdddeZ%GdddZ&ede&dZ'eGdddZ(edZ)GdddeZ*Gdddee e)Z+Gdd d e Z,Gd!d"d"ee e'Z-Gd#d$d$e-e'e e'Z.Gd%d&d&e-e'e e'Z/Gd'd(d(e-e'e e'Z0d)e"d*e1fd+d,Z2d-e"d.e3e%d/e3e+e)d*e3e%fd0d1Z4d-e"d.e3e%d2e5e"e fd*e3e%fd3d4Z6dS)5)ABCabstractmethod) dataclassreplace)Enum) TYPE_CHECKINGAnyClassVarGenericProtocolTypeVarN) deprecated) VllmConfig) CacheDType)ColumnParallelLinear)QuantKey)DeviceCapability)KVCacheLayoutType) AttentionSpecc@&eZdZdZdZ dZ dZ dZdS) AttentionTypezO Attention type. Use string to be compatible with `torch.compile`. decoderencoder encoder_onlyencoder_decoderN)__name__ __module__ __qualname____doc__DECODERENCODER ENCODER_ONLYENCODER_DECODERr#r#f/lsinfo/ai/hellotax_ai/llm_service/venv_vllm/lib/python3.10/site-packages/vllm/v1/attention/backend.pyrsrc@s$eZdZUeed<defddZdS) MultipleOfbasecCs ||_dSN)r&)selfr&r#r#r$__init__* zMultipleOf.__init__N)rrrint__annotations__r)r#r#r#r$r%'s r%c@seZdZUdZdZeed<ejej gZ e e ej ed<gdZe e ded<dZeed <ed e eeBfd d Zeed efd dZeed edfddZeeddZee dUdededededed eedff ddZe dUdedededed ef ddZe dVded eedffd d!Zed eeeffd"d#Zed e efd$d%Z eded efd&d'Z!ed(ej d efd)d*Z"ed+d,d efd-d.Z#eded/Bd efd0d1Z$ed2ed efd3d4Z%ed efd5d6Z&ed efd7d8Z'ed efd9d:Z(ed efd;d<Z)ed efd=d>Z*ed efd?d@Z+edAed efdBdCZ,edDdEd efdFdGZ-eded(ej d+d,ded/BdHedIedJedKdEd ed/BfdLdMZ.eded(ej d+d,ded/BdHedIedJedNedOedKdEdAed e efdPdQZ/edWdSdTZ0d/S)XAttentionBackendz&Abstract class for attention backends.Faccept_output_buffersupported_dtypes)autofloat16bfloat16rsupported_kv_cache_dtypesT forward_includes_kv_cache_updatereturncCs tdgS)N)r%r#r#r#r$ get_supported_kernel_block_sizes? z1AttentionBackend.get_supported_kernel_block_sizescCtr'NotImplementedErrorr#r#r#r$get_nameCzAttentionBackend.get_nameAttentionImplBasecCr9r'r:r#r#r#r$ get_impl_clsHr=zAttentionBackend.get_impl_clscCr9r'r:r#r#r#r$get_builder_clsMr=z AttentionBackend.get_builder_clsr0 num_blocks block_size num_kv_heads head_sizecache_dtype_str.cCr9r'r:)rArBrCrDrEr#r#r$get_kv_cache_shapeR z#AttentionBackend.get_kv_cache_shapecCs"d}|j|||||d}||S)zhDiscover which tensor dim is the block index, since different backends lay out dims differently.i)rE)rFindex)clsrBrCrDrE_Sshaper#r#r$get_kv_cache_block_dim]s  z'AttentionBackend.get_kv_cache_block_diminclude_num_layers_dimensioncCr9)av Get the physical (memory layout) ordering of the kv cache dimensions. e.g. if the KV cache shape is [2, num_blocks, block_size, num_heads, head_size], and get_kv_cache_stride_order returns (1, 3, 0, 2, 4) then the physical ordering of dimensions is [num_blocks, num_heads, 2, block_size, head_size]. If this function is unimplemented / raises NotImplementedError, the physical layout of the KV cache will match the logical shape. Args: include_num_layers_dimension: if True, includes an additional num_layers dimension, which is assumed to be prepended to the logical KV cache shape. With the above example, a return value (2, 4, 0, 1, 3, 5) corresponds to [num_blocks, num_heads, num_layers, 2, block_size, head_size]. If an additional dimension is NOT included in the returned tuple, the physical layout will not include a layers dimension. Returns: A tuple of ints which is a permutation of range(len(shape)). r:)rMr#r#r$get_kv_cache_stride_orderqsz*AttentionBackend.get_kv_cache_stride_ordercCs |j|jfSr')rrrIr#r#r$ full_cls_names zAttentionBackend.full_cls_namecCsgSr'r#rOr#r#r$get_supported_head_sizesz)AttentionBackend.get_supported_head_sizescCs|}| p ||vSr')rQ)rIrDsupported_head_sizesr#r#r$supports_head_sizesz#AttentionBackend.supports_head_sizedtypecCs ||jvSr')r/)rIrUr#r#r$supports_dtyper8zAttentionBackend.supports_dtypekv_cache_dtypezCacheDType | NonecCs|durdS|j p||jvSNT)r3)rIrWr#r#r$supports_kv_cache_dtypes z(AttentionBackend.supports_kv_cache_dtypeNcCsL|durdS|}|sdS|D]}t|tr|j}||dkr#dSqdS)NTrF)r7 isinstancer%r&)rIrBsupported_kernel_block_sizessupported_sizer#r#r$supports_block_sizes  z$AttentionBackend.supports_block_sizedefault_block_sizecCs0|}|s|S||r|Stdd|DS)Ncss$|] }t|tr |jn|VqdSr')rZr%r&).0sr#r#r$ s"z.)r7r]min)rIr^supported_sizesr#r#r$get_preferred_block_sizes  z)AttentionBackend.get_preferred_block_sizecCdSNFr#rOr#r#r$is_mlarRzAttentionBackend.is_mlacCrerfr#rOr#r#r$ supports_sinkrRzAttentionBackend.supports_sinkcCrerfr#rOr#r#r$supports_alibi_sqrtrRz$AttentionBackend.supports_alibi_sqrtcCrerfr#rOr#r#r$supports_mm_prefixrRz#AttentionBackend.supports_mm_prefixcCrerfr#rOr#r#r$ is_sparserRzAttentionBackend.is_sparsecCrerfr#rOr#r#r$supports_per_head_quant_scalesrRz/AttentionBackend.supports_per_head_quant_scales attn_typecCs |tjkS)zCheck if backend supports a given attention type. By default, only supports decoder attention. Backends should override this to support other attention types. )rr)rIrmr#r#r$supports_attn_types z#AttentionBackend.supports_attn_type capabilityrcCrerXr#)rIror#r#r$supports_compute_capabilityrRz,AttentionBackend.supports_compute_capabilityuse_mlahas_sink use_sparsedevice_capabilityc CdSr'r#) rIrDrUrWrBrqrrrsrtr#r#r$supports_combination z%AttentionBackend.supports_combination use_mm_prefixuse_per_head_quant_scalesc  Cs@g} ||s | d||s| d||s | d||s*| d|r5|s5| d||krH|rC| dn| d|rS|sS| d||krf|ra| d n| d | rq| sq| d | | s{| d | | s| d | d| |||||||| } | dur| | | S)Nzhead_size not supportedzdtype not supportedzkv_cache_dtype not supportedzblock_size not supportedz5partial multimodal token full attention not supportedzMLA not supportedznon-MLA not supportedzattention sinks not supportedzsparse not supportedznon-sparse not supportedz#per-head quant scales not supportedz compute capability not supportedzattention type z not supported) rTappendrVrYr]rjrgrhrkrlrprnrv)rIrDrUrWrBrqrrrsrxryrtrminvalid_reasonscombination_reasonr#r#r$validate_configurationsT                       z'AttentionBackend.validate_configurationKVCacheLayoutType | NonecCrur'r#rOr#r#r$get_required_kv_cache_layout6rRz-AttentionBackend.get_required_kv_cache_layout)r0F)r5r~)1rrrrr.boolr,torchr1r2r/r listrUr3r4 staticmethodr+r%r7rstrr<typer?r@tuplerF classmethodrLrNrPrQrTrVrYr]rdrgrhrirjrkrlrnrprvr}rr#r#r#r$r-.s                  ;r-c@s eZdZdS)AttentionMetadataN)rrrr#r#r#r$r;srT)boundc@seZdZUdZejed<ejed< ejed< eed< eed< eed< eed< ejed <ejed <d Ze ed <d Z ejd Bed<d Z ed Bed<d Z ejd Bed<d Z ejd Bed<d Zejd Bed<d Zejd Bed< d Zejd Bed<d Zejd Bed<d Zejd Bed<defddZdejfddZd)ddZeeddejfdd Zeed!dejfd"d#Zdejfd$d%Zded&eddfd'd(Zd S)*CommonAttentionMetadataz Per-batch attention metadata, shared across layers and backends. AttentionMetadataBuilder instances use it to construct per-layer metadata. For many of the tensors we keep both GPU and CPU versions. query_start_locquery_start_loc_cpuseq_lensnum_reqsnum_actual_tokens max_query_len max_seq_lenblock_table_tensor slot_mappingTcausalNlogits_indices_paddednum_logits_indicesencoder_seq_lensencoder_seq_lens_cpudcp_local_seq_lensdcp_local_seq_lens_cpu _seq_lens_cpu_num_computed_tokens_cpu_num_computed_tokens_cacher5cCs |jjdS)Nr)rrKr(r#r#r$ batch_sizess z"CommonAttentionMetadata.batch_sizecCs|jdd|jddS)zENaive because it assumes that query ends where the next query starts.r6N)rrr#r#r$naive_query_lensvsz(CommonAttentionMetadata.naive_query_lenscKst|fi|Sr')r)r(kwargsr#r#r$rzszCommonAttentionMetadata.replacez Prefer using device seq_lens directly to avoid implicit H<>D sync. If a CPU copy is needed, use `seq_lens.cpu()` instead. Will be removed in a future release, please migrate as soon as possible. cCs|jdur |jd|_|jS)Ncpu)rrtorr#r#r$ seq_lens_cpu}s z$CommonAttentionMetadata.seq_lens_cpua Prefer using device seq_lens directly to avoid implicit H<>D sync which breaks full async scheduling. If a CPU copy is needed, it can be derived from query_start_loc_cpu and seq_lens. Will be removed in a future release, please migrate as soon as possible. cCs8|jdur|jdd|jdd}|j||_|jS)Nr6r)rrr)r(query_seq_lensr#r#r$num_computed_tokens_cpus  z/CommonAttentionMetadata.num_computed_tokens_cpucCs8|jdur|jdd|jdd}|j||_|jS)z>Compute num_computed_tokens on device (seq_lens - query_lens).Nr6r)rrr)r( query_lensr#r#r$compute_num_computed_tokenss  z3CommonAttentionMetadata.compute_num_computed_tokensnum_actual_reqscsfdd}tdid|jddd|jddd|jdd|jdur2|jdndd|jdurA|jdnFdd d |d |jd |jd |jdd|j d|d|j d|j d|j d||j d||jd||jd||jSd d |d |jd |jd |jdd|j d|d|j d|j d|j d||j d||jd||jd||jS)Ncs|dur |dSdSr'r#)xrr#r$sz2CommonAttentionMetadata.unpadded..rr6rrrrrrrrrrrrrrrrrr#)rrrrrrrrrrrrrrrrr)r(rrmaybe_slice_reqsr#rr$unpaddeds                   z CommonAttentionMetadata.unpadded)r5r)rrrrrTensorr,r+rrrrrrnpndarrayrrrrrrrrpropertyr rrrrr#r#r#r$rBsb        rMc@r)AttentionCGSupportzConstants for the cudagraph support of the attention backend Here we do not consider the cascade attention, as currently it is never cudagraph supported.r6rN)rrrrALWAYS UNIFORM_BATCHUNIFORM_SINGLE_TOKEN_DECODENEVERr#r#r#r$rsrc@sDeZdZUejZeeed<dZe dBed<dZ e ed<e ddde ed d d ejfd d Zededd d dddefddZ   d0de dBde de ddfddZe d1de dede defddZdedejdejdefdd Zdedefd!d"Zded#e defd$d%Zde d&ejd'e d(e d)e d*e d+e d,e d-e de fd.d/ZdS)2AttentionMetadataBuilder_cudagraph_supportNreorder_batch_thresholdFsupports_update_block_table kv_cache_specr layer_names vllm_configrdevicecCs||_||_||_||_dSr')rrrr)r(rrrrr#r#r$r)s z!AttentionMetadataBuilder.__init__rIr5cCs|jS)z6Get the cudagraph support level of this builder class.)r)rIrrr#r#r$get_cudagraph_supportsz.AttentionMetadataBuilder.get_cudagraph_supportr6supports_spec_as_decodesupports_dcp_with_varlencCsx||_|jdur*|r*|jj}|dur*|jdur*d|jrdnd|j}t|j||_|jjjdkr8|s:d|_dSdSdS)Nr6r)rrspeculative_confignum_speculative_tokensparallel_draftingmaxparallel_configdecode_context_parallel_size)r(rrrrmax_num_queries_for_specr#r#r$_init_reorder_batch_thresholds(   z6AttentionMetadataBuilder._init_reorder_batch_thresholdcommon_prefix_lencommon_attn_metadata fast_buildcCr9)a Central method that builds attention metadata. Some builders (MLA) require reorder_batch to be called prior to build. Args: common_prefix_len: The length of the common prefix of the batch. common_attn_metadata: The common attention metadata. fast_build: The meta-data will prioritize speed of building over then speed at execution. Can be used for spec-decode where the result of a build call may only be used for few layers/iters. r:)r(rrrr#r#r$buildszAttentionMetadataBuilder.buildmetadata blk_tablercCr9)a Update the block table for the attention metadata. Faster when theres multiple kv-cache groups that create virtually the same metadata but just with different block tables. Only needs to be implemented if supports_update_block_table is True. r:)r(rrrr#r#r$update_block_table+ z+AttentionMetadataBuilder.update_block_tablecCs|jd|dS)z Build attention metadata for CUDA graph capture. Uses build by default. Subclasses that override this method should call self.build or super().build_for_cudagraph_capture. r)rrr)r(rr#r#r$build_for_cudagraph_capture:sz4AttentionMetadataBuilder.build_for_cudagraph_capture draft_indexcCs|jd|ddS)a Build attention metadata for draft model. Uses build by default. Args: common_attn_metadata: The common attention metadata. draft_index: The index of the current draft operation. When speculating a chain of tokens, this index refers to the draft attempt for the i-th token. For tree-based attention, this index instead refers to the draft attempt for the i-th level in the tree of tokens. rT)rrrr)r(rrr#r#r$build_for_draftingFs z+AttentionMetadataBuilder.build_for_draftingrnum_query_headsrC use_alibiuse_sliding_windowuse_local_attentionnum_smsdcp_world_sizec Crerfr#) r(rrrrCrrrrrr#r#r$use_cascade_attention\rwz.AttentionMetadataBuilder.use_cascade_attention)r6FFr) rrrrrrr r,rr+rrrrrrrr)rrrrrrrrrrrrrrr#r#r#r$rs                rc @szeZdZUejed<ejed<ejed<eed<eed<eed<ejed<dejd ejd ejd ejd ed ejf ddZdS)AttentionLayer_q_scale_k_scale_v_scale_q_scale_float_k_scale_float_v_scale_float _prob_scalequerykeyvaluekv_cache attn_metadatar5cCrur'r#)r(rrrrrr#r#r$forwardtszAttentionLayer.forwardN) rrrrrr,floatrrr#r#r#r$rks*    rcseZdZUdZeed<eed<eed<dZeed<dZ eed<dZ eed<dZ eed <dZ eed <eed <eed <eed <eed<eed<eed<fddZ dejfddZZS)r>zBase class for attention implementations. Contains common attributes and initialization logic shared by both standard AttentionImpl and MLAAttentionImpl. Does not define a forward method - subclasses define their own forward interfaces. num_headsrDscaleFcan_return_lse_for_decode supports_pcp0supports_mtp_with_cp_non_trivial_interleave_sizeneed_to_return_lse_for_decodesupports_quant_query_inputrdcp_rankpcp_world_sizepcp_ranktotal_cp_world_size total_cp_rankcst|}zddlm}|j|_|j|_Wnty'd|_d|_Ynwzddlm }|j|_ |j|_ WntyId|_ d|_ Ynw|j |j|_ |j |j|j|_ |jdkob|j|_|S)Nr) get_dcp_groupr6) get_pcp_group)super__new__vllm.distributed.parallel_stater world_sizer rank_in_grouprAssertionErrorrrrrrrr)rIargsrr(rr __class__r#r$rs,         zAttentionImplBase.__new__ act_dtypecCrur'r#)r(r r#r#r$process_weights_after_loadingsz/AttentionImplBase.process_weights_after_loading)rrrrr+r,rrrrrrrrrrUr  __classcell__r#r#rr$r>~s$       r>c@seZdZdZedddddejdfdededededBde edBd edBd e d edBd e d e dBddfddZ e   d(de de jde jde jde jdede jdBde jdBde jdBde jfddZd)ddZd d!Zde de jde jde jd"e jd#e jd$ede jd%e jfd&d'ZdS)* AttentionImplz6Standard attention implementation with forward method.Nr0rrDrrC alibi_slopessliding_windowrWlogits_soft_caprmkv_sharing_target_layer_namer5c Cr9r'r:) r(rrDrrCrrrWrrmrr#r#r$r)szAttentionImpl.__init__layerrrrrroutput output_scaleoutput_block_scalec Cr9r'r:) r(rrrrrrrrrr#r#r$rrzAttentionImpl.forward quant_keyrcCre)ab Does this attention implementation support fused output quantization. This is used by the AttnFusionPass to only fuse output quantization onto implementations that support it. :param quant_key: QuantKey object that describes the quantization op :return: is fusion supported for this type of quantization Fr#)r(rr#r#r$fused_output_quant_supportedrGz*AttentionImpl.fused_output_quant_supportedcCre)z Does this attention implementation support RoPE+KVCache fusion. This is used by the RopeKVCacheFusionPass to only fuse the RoPE ops with the KV cache update for implementations that support it. Fr#rr#r#r$fused_rope_kvcache_supportedsz*AttentionImpl.fused_rope_kvcache_supported positions cos_sin_cacheis_neoxlayer_slot_mappingc Cr9)z If `fused_rope_kvcache_supported` returns True, this method will be called by torch.ops.vllm.fused_rope_and_unified_kv_cache_update to perform the inplace RoPE and KV cache update. r:) r(rrrrrrrrrr#r#r$do_rope_and_kv_cache_updatesz)AttentionImpl.do_rope_and_kv_cache_update)NNN)rr)rrrrrrrr+rrrr)rrrrrrrrrr#r#r#r$r s            r c*@sDeZdZdZe  d*dededededeedBdedBd ed edBd ed edBd edBdedededededdde dBdedBddf(ddZ ede j de j de j de j de de j d e j ddfd!d"Zede j ee j e j fBde j de d#edee j e j dBff d$d%Zde j de j d&e j d'e j d ede j ddfd(d)ZdS)+MLAAttentionImplzFMLA attention implementation with forward_mqa and forward_mha methods.NrrDrrCrrrWrrmr q_lora_rank kv_lora_rankqk_nope_head_dimqk_rope_head_dim qk_head_dim v_head_dim kv_b_projrindexerq_pad_num_headsr5cCr9r'r:r(rrDrrCrrrWrrmrrr r!r"r#r$r%r&r'r#r#r$r)zMLAAttentionImpl.__init__q kv_c_normedk_pekv_c_and_k_pe_cacherk_scalercCr9)zMHA-style prefill forward pass.r:)r(r*r+r,r-rr.rr#r#r$ forward_mha2rwzMLAAttentionImpl.forward_mharcCr9zMQA-style decode forward pass.r:r(r*r-rrr#r#r$ forward_mqa@rGzMLAAttentionImpl.forward_mqarrcC@|dkrdSddlm}|j||d||||ddSNr) _custom_opsr6)rWrnumelvllmr5concat_and_cache_mlasqueezeflattenr(r+r,rrrWr.opsr#r#r$do_kv_cache_updateK   z#MLAAttentionImpl.do_kv_cache_updateNN)rrrrrr+rrrobjectr)rrrr/rrr2r>r#r#r#r$rs          rc*@s eZdZdZe  d'dededededeedBdedBd ed edBd ed edBd edBdedededededdde dBdedBddf(ddZ ede j e e j e j fBde j dedede e j e j dBff ddZd e j d!e j d"e j d#e j d ed$e j ddfd%d&ZdS)(SparseMLAAttentionImplzSparse MLA attention implementation with only forward_mqa method. Sparse MLA implementations only support decode (MQA-style) attention. They do not support prefill (MHA-style) attention. NrrDrrCrrrWrrmrrr r!r"r#r$r%rr&r'r5cCr9r'r:r(r#r#r$r)ir)zSparseMLAAttentionImpl.__init__r*r-rrcCr9r0r:r1r#r#r$r2rGz"SparseMLAAttentionImpl.forward_mqar+r,rrr.cCr3r4r6r<r#r#r$r>r?z)SparseMLAAttentionImpl.do_kv_cache_updater@)rrrrrr+rrrrAr)rrrrrr2r>r#r#r#r$rBbs        rBrWr5cCs |dS)Nfp8) startswith)rWr#r#r$is_quantized_kv_cacher*rE name_prefixattention_backend_cls builder_clscs$||j}t||fdfddiS)zN Return a new subclass where `get_builder_cls` returns `builder_cls`. r@csSr'r#r#rHr#r$rsz,subclass_attention_backend..rr)rFrGrHnamer#rIr$subclass_attention_backends rL overridescCs||j}t||f|Sr'rJ)rFrGrMrKr#r#r$)subclass_attention_backend_with_overridess rN)7abcrr dataclassesrrenumrtypingrrr r r r numpyrrtyping_extensionsr vllm.configrvllm.config.cacher!vllm.model_executor.layers.linearr9vllm.model_executor.layers.quantization.utils.quant_utilsrvllm.platforms.interfacer vllm.v1.attention.backends.utilsrvllm.v1.kv_cache_interfacerrrr%r-rrrrrrrr>r rrBrrErrLdictrNr#r#r#r$sh           ~NIMC