o 4/iy&@sdZddlmZddlZddlmZmZmZmZm Z m Z ddl Z ddl m Z ddlmZmZmZddlmZdd lmZmZmZmZmZmZmZdd lmZd d Zed dGdddZedddddZ edddddZ!e e!gZ"ed dGdddZ#e#e dde#e dde#e!ddgZ$de j%de j%de&de&de&de j%d e&d!e j%fd"d#Z'ej(d$d%Z)ej(d&d'Z*ej(d(d)Z+Gd*d+d+Z,e   , ,    -d>de j%de j%d.e j%d/e&de&de&d0e j%d1e j%d2e&de&d3ee j%d4e e-e j%fd5e e-e j%fd6eee j%d7ee-d8e.dBd9e/d!e j%f$d:d;Z0e  , ,  d?de j%de j%d.e j%d/e&de&de&d0e j%d1e j%d2e&d3ee j%d4e e-e j%fd5e e-e j%fd6eee j%d8e.dBd!e j%fdr? querykv_cacher4r2sparse_mla_top_kr$ page_sizereturncCsd|jdkr td|j|jdkr|d}n |jdkr%td|j|tjko.|tjk}|tjko8|tjk}|sJ|sJtd|d|dt|j\} } } } |jd} ||}| |ksb| |krotd |d | d | |d kr|j}|| | |fkrtd ||S|j\}}|}| |krtd| d| d ||d|d krtd|d||S)NzExpected query.ndim == 4, got rr z&Expected kv_cache.ndim == 3 or 4, got z-Unsupported MLA dimensions, got kv_lora_rank=z and qk_rope_head_dim=z, supported dimensions are: zExpected head dim z for query and kv_cache, got rrzKExpected page_table.shape == (num_seqs, num_tokens, sparse_mla_top_k), got rz for query and block_table, got rrr) rr unsqueezedeepseek_mla_dimensionsr4r2smaller_mla_dimensionssupported_mla_head_dimensionsr )rArBr4r2rCr$rDis_deepseek_dimensionsis_smaller_mla_dimensionsnum_seqs num_tokens_ qk_head_dimckv_dimexpected_qk_head_dimpage_table_shaper)r*r+r,r,r-_check_trtllm_gen_mla_shapesR        rTcCst}|}t||SN)r build_and_loadr get_library_path)modopr,r,r-get_trtllm_gen_fmha_modules rZcCs tSrU)rrVr,r,r,r-get_mla_modules r[cGst|g|RSrU)r rV)backendargsr,r,r-get_batch_mla_modulesr^c@sDeZdZdZe      d*dejdedeejdeejd eejd eejd e d dfd dZ e d+dejdejd ejd ejde de de de dede dej dej ded dfddZe       d,dejdejdejdejdeejdeejd edd!eejd"eejd#eejd$ed ejfd%d&Ze   '    d-dejdejdejdejdeejdeejd ed'd!eejd"eejd#eejd$ed eejejffd(d&Ze       d,dejdejdejdejdeejdeejd ed!eejd"eejd#eejd$ed eejeejejfffd)d&ZdS).BatchMLAPagedAttentionWrappera Wrapper class for MLA (`Multi-head Latent Attention `_) PagedAttention on DeepSeek models. This kernel can be used in decode, and incremental prefill and should be used together with `Matrix Absorption trick `_: where :math:`W_{UQ}` is absorbed with :math:`W_{UK}`, and :math:`W_{UV}` is absorbed with :math:`W_{O}`. For MLA attention without Matrix Absorption (``head_dim_qk=192`` and ``head_dim_vo=128``, which is used in prefilling self-attention stage), please use :class:`flashinfer.prefill.BatchPrefillWithRaggedKVCacheWrapper`. More information about The Paged KV-Cache layout in MLA is explained in our tutorial :ref:`MLA Page Layout `. For more details about the MLA computation, Matrix Absorption and FlashInfer's MLA implementation, please refer to our `blog post `_. Example ------- >>> import torch >>> import flashinfer >>> num_local_heads = 128 >>> batch_size = 114 >>> head_dim_ckv = 512 >>> head_dim_kpe = 64 >>> page_size = 1 >>> mla_wrapper = flashinfer.mla.BatchMLAPagedAttentionWrapper( ... torch.empty(128 * 1024 * 1024, dtype=torch.int8).to(0), ... backend="fa2" ... ) >>> q_indptr = torch.arange(0, batch_size + 1).to(0).int() # for decode, each query length is 1 >>> kv_lens = torch.full((batch_size,), 999, dtype=torch.int32).to(0) >>> kv_indptr = torch.arange(0, batch_size + 1).to(0).int() * 999 >>> kv_indices = torch.arange(0, batch_size * 999).to(0).int() >>> q_nope = torch.randn( ... batch_size * 1, num_local_heads, head_dim_ckv, dtype=torch.bfloat16, device="cuda" ... ) >>> q_pe = torch.zeros( ... batch_size * 1, num_local_heads, head_dim_kpe, dtype=torch.bfloat16, device="cuda" ... ) >>> ckv = torch.randn( ... batch_size * 999, 1, head_dim_ckv, dtype=torch.bfloat16, device="cuda" ... ) >>> kpe = torch.zeros( ... batch_size * 999, 1, head_dim_kpe, dtype=torch.bfloat16, device="cuda" ... ) >>> sm_scale = 1.0 / ((128 + 64) ** 0.5) # use head dimension before matrix absorption >>> mla_wrapper.plan( ... q_indptr, ... kv_indptr, ... kv_indices, ... kv_lens, ... num_local_heads, ... head_dim_ckv, ... head_dim_kpe, ... page_size, ... False, # causal ... sm_scale, ... q_nope.dtype, ... ckv.dtype, ... ) >>> o = mla_wrapper.run(q_nope, q_pe, ckv, kpe, return_lse=False) >>> o.shape torch.Size([114, 128, 512]) FNautofloat_workspace_bufferuse_cuda_graph qo_indptr kv_indptr kv_indices kv_len_arrr\rEcCs||_|j|_|dkr||_dStjdtj|jd|_tj|jj|jjddd|_ ||_ ||_ ||_ ||_ ||_|dkrDt|j|_dS||_dS) anConstructor for BatchMLAPagedAttentionWrapper. Parameters ---------- float_workspace_buffer : torch.Tensor The user reserved workspace buffer used to store intermediate attention results in split-k algorithm. The recommended size is 128MB, the device of the workspace buffer should be the same as the device of the input tensors. use_cuda_graph : bool, optional Whether to enable CUDA graph capture for the prefill kernels, if enabled, the auxiliary data structures will be stored in provided buffers. The ``batch_size`` cannot change during the lifecycle of this wrapper when CUDAGraph is enabled. qo_indptr_buf : Optional[torch.Tensor] The user reserved buffer to store the ``qo_indptr`` array, the size of the buffer should be ``[batch_size + 1]``. This argument is only effective when ``use_cuda_graph`` is ``True``. kv_indptr_buf : Optional[torch.Tensor] The user reserved buffer to store the ``kv_indptr`` array, the size of the buffer should be ``[batch_size + 1]``. This argument is only effective when ``use_cuda_graph`` is ``True``. kv_indices_buf : Optional[torch.Tensor] The user reserved buffer to store the ``kv_indices`` array. This argument is only effective when ``use_cuda_graph`` is ``True``. kv_len_arr_buf : Optional[torch.Tensor] The user reserved buffer to store the ``kv_len_arr`` array, the size of the buffer should be ``[batch_size]``. This argument is only effective when ``use_cuda_graph`` is ``True``. backend : str The implementation backend, could be ``auto``/``fa2`` or ``fa3``. Defaults to ``auto``. If set to ``auto``, the function will automatically choose the backend based on the device architecture and kernel availability. If ``cutlass`` is provided, the MLA kernels will be generated by CUTLASS and only float_workspace_buffer is required and other arguments are ignored. cutlassN)dtypedeviceTcpu)rj pin_memoryrkr`)_float_workspace_bufferrk_backendtorchemptyuint8_int_workspace_bufferr rj _pin_memory_int_workspace_buffer_use_cuda_graph_qo_indptr_buf_kv_indptr_buf_kv_indices_buf_kv_len_arr_bufr)selfrarbrcrdrerfr\r,r,r-__init__s,-  z&BatchMLAPagedAttentionWrapper.__init__r? head_dim_ckv head_dim_kperDcausalsm_scale q_data_type kv_data_type use_profilerc Cst|j| | | |j||| |_|d}|d}|d}|jrG|jj|dd|jj|dd|j dt |j|dd|j j|ddn$|j|j dd|_|j|j dd|_|j|j dd|_ |j|j dd|_ | |_ ||_| |_| |_|j|j|j|j|||||| |_dS)aPlan the MLA attention computation. Parameters ---------- qo_indptr : torch.IntTensor The indptr of the query/output tensor, shape: ``[batch_size + 1]``. For decoding attention, the length of each query is 1, and the content of the tensor should be ``[0, 1, 2, ..., batch_size]``. kv_indptr : torch.IntTensor The indptr of the paged kv-cache, shape: ``[batch_size + 1]``. kv_indices : torch.IntTensor The page indices of the paged kv-cache, shape: ``[kv_indptr[-1]]`` or larger. kv_len_arr : torch.IntTensor The query length of each request, shape: ``[batch_size]``. num_heads : int The number of heads in query/output tensor. head_dim_ckv : int The head dimension of compressed-kv. head_dim_kpe : int The head dimension for rope k-cache. page_size : int The page size of the paged kv-cache. causal : bool Whether to use causal attention. sm_scale : float The scale factor for softmax operation. q_data_type : torch.dtype The data type of the query tensor. kv_data_type : torch.dtype The data type of the kv-cache tensor. use_profiler : bool, optional Whether to enable intra-kernel profiler, default is False. rlT) non_blockingN)r^rorj_cached_moduletorurvcopy_rwrxlenryrk_causal _page_size _sm_scale _use_profilerplanrnrsrt _plan_info)rzrcrdrerfr?r|r}rDr~rrrrqo_indptr_hostkv_indptr_hostkv_len_arr_hostr,r,r-rYsJ2   z"BatchMLAPagedAttentionWrapper.planq_nopeq_pe ckv_cache kpe_cacheoutlse return_lseprofiler_bufferr#r$return_lse_base_on_ec CdSrUr, rzrrrrrrrrr#r$rr,r,r-runz!BatchMLAPagedAttentionWrapper.runTc CrrUr,rr,r,r-rrc Cs|jdkr^|r td|durtdt|_|dur!t|}n t||j|j|j dtj ||gdd} tj ||gdd} t | | | | tj dtj |j d }|j|j||| | | | |S|duri|jritd |jd }|j}|j}|j}|r}tjjntjj}|j }|durt|}n t||j|j|j d|r|durtj |jdd tj |d }nt||jdd tj |j d |jr|fnd}|jj|j|j|j|||||j||||||| g|R|r||fS|S)aRun the MLA attention computation. Parameters ---------- q_nope : torch.Tensor The query tensor without rope, shape: ``[batch_size, num_heads, head_dim_ckv]``. q_pe : torch.Tensor The rope part of the query tensor, shape: ``[batch_size, num_heads, head_dim_kpe]``. ckv_cache : torch.Tensor The compressed kv-cache tensor (without rope), shape: ``[num_pages, page_size, head_dim_ckv]``. ``head_dim_ckv`` is 512 in DeepSeek v2/v3 models. kpe_cache : torch.Tensor The rope part of the kv-cache tensor, shape: ``[num_pages, page_size, head_dim_kpe]``. ``head_dim_kpe`` is 64 in DeepSeek v2/v3 models. out : Optional[torch.Tensor] The output tensor, if not provided, will be allocated internally. lse : Optional[torch.Tensor] The log-sum-exp of attention logits, if not provided, will be allocated internally. return_lse : bool, optional Whether to return the log-sum-exp value, default is False. profiler_buffer : Optional[torch.Tensor] The buffer to store the profiler data. kv_len : Optional[torch.Tensor] The query length of each request, shape: ``[batch_size]``. Required when ``backend`` is ``cutlass``. page_table : Optional[torch.Tensor] The page table of the paged kv-cache, shape: ``[batch_size, num_pages]``. Required when ``backend`` is ``cutlass``. rgz4return_lse does not support cutlass backend for now.Nz9profiler_buffer does not support cutlass backend for now.r)dimrriz5Profiler is enabled, profiler_buffer must be providedr rrr,)rorr[rrp empty_likerr rjrkcatr.rqfloat32cutlass_mla_paged_attentionrnrrrrrCAUSALvalue NON_CAUSALrrsrrx)rzrrrrrrrrr#r$rr!r"r?rDrr~ mask_moderk profiler_argsr,r,r-rs *    )FNNNNr`)F)NNFNNNF)NNTNNNF)r5r6r7r8r rpTensorboolrstrr{r9floatrjrrrrrrr,r,r,r-r_s8A F      Z               r_?r`workspace_bufferr1 block_tablesseq_lens max_seq_lenr bmm1_scale bmm2_scalesinks#skip_softmax_threshold_scale_factor enable_pdlr\cCsf|dkrt|jddkrdnd}t| tjr#| jtjksJ| t} t| tjr1| jtjks1J|dkrt|jddksJ|jtjksJ|jtjkrVt d|jd|j| d ur^t d | d d krot d | d |d urwt d t |||d|||||| | | | |S|dkr+|d urt |jn|}t j}t|j}| d}|dkr|dkrt d||d ur| dkrt dt||||| ||}| d ur|jd d|f}tj|tj|jd} n|j\}}}}t| |||gtj|jd| d}| d }|dd }|| d ||||||||| | ddd|d| ||||| d || St d|d)a Parameters ---------- query: [batch_size, q_len_per_request, num_heads, head_dim_qk], head_dim_qk = qk_nope_head_dim (kv_lora_rank) + qk_rope_head_dim, should be concated q_nope + q_rope; q_len_per_request is the MTP query length. kv_cache: [num_pages, page_size, head_dim_ckv + head_dim_kpe] or [num_pages, 1, page_size, head_dim_ckv + head_dim_kpe], should be concated ckv_cache + kpe_cache. Both 3D and 4D formats are supported for backward compatibility. workspace_buffer: [num_semaphores, 4], used for multi_block mode. Must be initialized to 0 for its first use. qk_nope_head_dim: qk_nope_head_dim, must be 128 or 64 kv_lora_rank: kv_lora_rank, must be 512 or 256 qk_rope_head_dim: qk_rope_head_dim, must be 64 sparse_mla_top_k: sparse MLA top k, must be 0 for non-sparse MLA. block_tables: page_table of kv cache, [batch_size, num_pages] seq_lens: query_len max_seq_len: max sequence length for kv_cache out: output tensor, if not provided, will be allocated internally bmm1_scale: fused scale for mla bmm1 input. when using trtllm-gen backend, it can be a torch.Tensor with dtype torch.float32. bmm2_scale: fused scale for mla bmm2 input. when using trtllm-gen backend, it can be a torch.Tensor with dtype torch.float32. sinks: additional value per head in the denominator of the softmax. skip_softmax_threshold_scale_factor: threshold scale factor for skipping softmax operations. Providing a value for this parameter enables skip-softmax sparsity as described in: https://arxiv.org/abs/2512.12087 If no value is provided, then standard attention is used. Setting the threshold to a higher value generally increases kernel performance at the cost of accuracy degradation. The actual threshold value equals the provided threshold_scale_factor divided by the context length. backend : str = "auto" The implementation backend, could be ``auto``/``xqa`` or ``trtllm-gen``. Defaults to ``auto``. When set to ``auto``, the backend will be chosen based on the device architecture and kernel availability. For sm_100 and sm_103 (blackwell architecture), ``auto`` will choose ``trtllm-gen`` backend. For sm_120 (blackwell architecture), ``auto`` will choose ``xqa`` backend. Note ---- In MLA, the actual BMM1 and BMM2 scales applied would be fused as: bmm1_scale = q_scale * k_scale * sm_scale / (head_dim_qk ** 0.5) bmm2_scale = v_scale * o_scale or, bmm1_scale = torch.Tensor([q_scale * k_scale * sm_scale / (head_dim_qk ** 0.5)) bmm2_scale = torch.Tensor([v_scale * o_scale]) The two scale factors should be static constant for cuda graph capture. Either (bmm1_scale, bmm2_scale) or (bmm1_scale_log2_tensor, bmm2_scale_tensor) should be provided. For static constant scale factors, the scale factors should be provided as float. - (bmm1_scale, bmm2_scale) For on-device fused scale tensors, which could dynamically change, the scale factors should be provided as torch.Tensor. - (bmm1_scale_log2_tensor, bmm2_scale_tensor) - Currently, only fp8 tensor core operation supports this mode. When both are provided, the dynamic scale factor tensors will be used. r`r z trtllm-genxqa z=XQA MLA only supports fp8 operation on SM120/SM121 GPUs, got rNXQA MLA does not support sinksr 2XQA MLA only supports q_len_per_request == 1, got z-skip_softmax is not supported for XQA backendrr@r;z(Supported block_size are 32 and 64, got z,skip_softmax is not supported for sparse MLArirzBackend z not supported)rrk isinstancerprrjrr float8_e4m3fnrsize"xqa_batch_decode_with_kv_cache_mlarrZtrtllm_paged_attention_decoderrTr rqbfloat16rflattennumel element_size)rArBrr1r4r2rrrrCrrrrrrr\run_funcsm_countr+ out_shape batch_sizerO num_q_heads max_q_lenr,r,r-%trtllm_batch_decode_with_kv_cache_mlaKsE           rcCs\| dur t|jn| } t|j}|d}|d}|dkr%td||jtjks1|jtjkr=td|jd|j| durEtdt||||d||}| durh|j dd |f}tj |tj |jd } n|j \}}}}t | |||gtj |jd | tj}|dd }|d d}|dd }|d}t|||||| |||| | || d | S)a Parameters: query: [batch_size, q_len_per_request, num_heads, head_dim_qk], head_dim_qk = qk_nope_head_dim (kv_lora_rank) + qk_rope_head_dim, should be concated q_nope + q_rope; q_len_per_request is the MTP query length. kv_cache: [num_pages, page_size, head_dim_ckv + head_dim_kpe] or [num_pages, 1, page_size, head_dim_ckv + head_dim_kpe], should be concated ckv_cache + kpe_cache. Both 3D and 4D formats are supported for backward compatibility. workspace_buffer: torch.Tensor. Must be initialized to 0 for its first use. qk_nope_head_dim: qk_nope_head_dim, must be 128 kv_lora_rank: kv_lora_rank, must be 512 qk_rope_head_dim: qk_rope_head_dim, must be 64 block_tables: page_table of kv cache, [batch_size, num_pages] seq_lens: query_len max_seq_len: max sequence length for kv_cache out: output tensor, if not provided, will be allocated internally bmm1_scale: fused scale for mla bmm1 input. Can be a float or a torch.Tensor. bmm2_scale: fused scale for mla bmm2 input. Can be a float or a torch.Tensor. sinks: additional value per head in the denominator of the softmax. Note: In MLA, the actual BMM1 and BMM2 scales applied would be fused as: bmm1_scale = q_scale * k_scale * sm_scale / (head_dim_qk ** 0.5) bmm2_scale = v_scale * o_scale The two scale factors should be static constant for cuda graph capture. Either (bmm1_scale, bmm2_scale) or (bmm1_scale_log2_tensor, bmm2_scale_tensor) should be provided. For static constant scale factors, the scale factors should be provided as float. - (bmm1_scale, bmm2_scale) For on-device fused scale tensors, which could dynamically change, the scale factors should be provided as torch.Tensor. - (bmm1_scale_log2_tensor, bmm2_scale_tensor) - Currently, only fp8 tensor core operation supports this mode. When both are provided, the dynamic scale factor tensors will be used. Nrr rz5XQA MLA only supports fp8 tensor core operation, got rrrrrirrhr)q_scalekv_scalerr)rrkrrrrjrprrTr rqrrviewrrsqueezerGr)rArBrr1r4r2rrrrrrrrrr+q_len_per_requestrrrOr workspace_u8 semaphorescratch kv_cache_new seq_lens_newr,r,r-rsn0        r)rNrrNNNr`)NrrNN)2r8 dataclassesr functoolstypingrrrrrrrp api_loggingr jitr r r jit.mlarutilsrrrrrrrrrr.r0rHrIrJr=supported_mla_layer_dimensionsrr9rTcacherZr[r^r_rrrrrr,r,r,r-s(    $   =   }        <