o 4/i}<@s dZ ddlZddlmZmZddlZddlZddlmZddl m Z ddl m Z ddl mmZz ddlmZdZWneyKd Zd d ZYnwz dd lmZdZWn eycd ZdZYnwd ZdZdZdZd ZdZdZdZ dZ!dZ"dZ#dZ$d Z%dZ&d Z'dZ(d Z)ee)Z*dZ+dZ,dZ-dede.de.de.fddZ/dede.de.de.fddZ0ej1dej2dej3dej4d ej5e.d!ej5e.d"ej3d#ej3d$ej3d%ej3d&ej3d'ej3d(ej3d)ej3d*ej3d+ej3d,ej5e6d-ej5e6d.ej5e6d/ej5e.d0ej5e.d1ej5e.d2ej5e.d3ej5e.d4ej5e.d5ej5e7d6ej5e7d7ej5e7f6d8d9Z8ej1dej2dej3dej4d ej5e.d!ej5e.d"ej3d#ej3d$ej3d%ej3d&ej3d'ej3d(ej3d)ej3d*ej3d+ej3d,ej5e6d-ej5e6d.ej5e6d/ej5e.d0ej5e.d1ej5e.d2ej5e.d3ej5e.d4ej5e.d5ej5e7d6ej5e7d7ej5e7f6d:d;Z9ej:dej3d"ej3d#ej3d$ej3d%ej3d&ej3d'ej3d(ej3d)ej3d*ej3d+ej3d,ej5e6d-ej5e6d.ej5e6d/ej5e.d0ej5e.d1ej5e.d2ej5e.d3ej5e.d4ej5e.d5ej5e7d6ej5e7d7ej5e7dZd0e.d1e.d2e.d/e.d3e.d4e.dAej?d.e6d6e7fdBdCZ@ej>d0e.d1e.d2e.d/e.d3e.d4e.dAej?d.e6d6e7fdDdEZAe     dfd%ej3d&ej3d'ej3dFeej3d"ej3d#ej3d$ej3d(ej3d.ee6dGeej3d6e7dHeej3dIeej3deej3ej3ffdJdKZBej1dej2dej3dej4d!ej5e.d%ej3d&ej3d'ej3d#ej3d(ej3d"ej3d$ej3d)ej3d*ej3d,ej5e6d-ej5e6d.ej5e6d2ej5e.d/ej5e.d6ej5e7f&dLdMZCej1dej2dej3dej4d!ej5e.d%ej3d&ej3d'ej3d#ej3d(ej3d"ej3d$ej3d)ej3d*ej3d,ej5e6d-ej5e6d.ej5e6d2ej5e.d/ej5e.d6ej5e7f&dNdOZDej:d+ej3d%ej3d&ej3d'ej3d#ej3d(ej3d"ej3d$ej3dej3d*ej3d)ej3d,ej5e6d-ej5e6d.ej5e6d0ej5e.d1ej5e.d2ej5e.d/ej5e.d3ej5e.d4ej5e.d5ej5e7d6ej5e7dd0e.d1e.d2e.d/e.d3e.d4e.d^e.d_e.dXe7dYe7d.e6d6e7dWe.d e.fd`daZJe     dhd%ej3d&ej3d'ej3dHej3dIej3d"ej3d#ej3d$ej3d(ej3d.ee6dGeej3dbeej3dXe7d6e7deej3ej3ffdcddZKdS)ia3 Copyright (c) 2025 by FlashInfer team. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. N)OptionalTuple)cpasync) from_dlpack)flashinfer_apiTFcCs|S)N)funcrrb/lsinfo/ai/hellotax_ai/llm_service/venv_vllm/lib/python3.10/site-packages/flashinfer/gdn_decode.pyr;sr)gated_delta_rule $ batch_sizeseq_lenreturncCsdS)zSelect vec_size for MTP kernel. Always use vec_size=4 (32 threads per group = full warp, 4 groups per block). Full warp shuffle is more efficient and achieves >= 1.0x speedup vs Triton. rrrrrrr get_vec_size_mtpusrcCs4|dkrdS|dkr dS|dkrdS|dkrdSdS)zSelect optimal TILE_V for MTP kernel based on batch size and sequence length. With vec_size=4, num_groups=4, rows_per_group = tile_v / 4. Tuned via grid search for optimal performance. rrr rr@rrrrr get_tile_v_mtp~srtiled_copy_load h0_sourcesmem_layout_stagedvec_size num_v_tilesA_logadt_biasqkvbo h0_indices cu_seqlens softplus_betasoftplus_thresholdscaleHVBTHKVuse_initial_state use_qk_l2norm is_varlenca Cstj\}}}|d}tj}tj|}tj\}}}|t} |t}!|t}"| |}#| |}$|$||}%d}&tj }'|' tj |d}(|' tj t |fd})|' tj t |fd}*ttj |fddtj }+ttj |fddtj },ttj |fddtj }-ttj |fddtj }.ttj |fddtj }/ttj |fddtj }0||}1t ||$}2t ||#|&|$f}3t ||$}4t | |#|&|$f}5tj|| ddf}6t|dttf| ddf}7t|6ttfd }8||}9|!|"}:ttd|"};t|:|:|;D]-}<|<|:t}=|8dd||(dd|=f}?|9|>}@|9|?}At||@|Atjqt|ddd|f|#|&|%|f}Bt| ddd|f|#|&|%|f}Ct|B|.t|C|/t|D]}Dt |.|D|,|D<t |/|D|+|D<qkt| ddd|f|#|&|$|f}Et|E|0t|D]}Dt |0|D|*|1|D<qtjd }Fd }G|dkr |3|4}H||H}Id }J|I|krtj|Id d }Kt d |K}Lt tj|Ld d }Mt t d ||M}Jn|H}Jtj|2d d  |J}Nd d tj|5 d d }Gtj|Nd d }Ftj|Fd}Ftj|Gd}G|rd }Od }Pt|D]}D|O|,|D|,|D7}O|P|+|D|+|D7}Pq%dD]}Q|Otjj |O|Qddd7}O|Ptjj |P|Qddd7}Pq?tj!|Odd d }Rtj!|Pdd d }St|D]}D|,|D|R|,|D<|+|D|S|+|D<qrt|D] }D|,|D||,|D<q|:|"}Tt|:|TD] }<|<|:t}=tj"dtj|<|;}U|U|Tkr|U|:t}V|8dd|Uf}W|(dd|Vf}X|9|W}@|9|X}At||@|AtjtdtdD]}Y|d}Zd }[t|(d|df|Y|Z||=f}\t|\|-t|D]}D|-|D|F|-|D<|[|-|D|+|D7}[qdD]}Q|[tjj |[|Qddd7}[q(|*|r)#cutearch thread_idxwarp_idxmake_warp_uniform block_idxNUM_BLOCKS_PER_STATEcutlassutils SmemAllocatorallocate_tensorFloat32BFloat16 make_layoutmake_rmem_tensorbarrier local_tileTILE_VTILE_K get_slicemin NUM_STAGESrange partition_S partition_Dcopycp_async_commit_group autovec_copyrange_constexprexplog shuffle_syncshuffle_sync_bflyrsqrtcp_async_wait_group)arrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6tidx_lane_idrJrL batch_idx batch_innernum_v_tiles_per_blocki_ni_hvi_hi_tsmemsDatasOutputsVr_kr_qr_hr_q_bf16r_k_bf16r_v_bf16k_startr_A_logr_a r_dt_biasr_b gSrc_batchgDstgSrc thr_copy_load start_v_tilesprefetch_countv_tilesstage gSrc_tile sData_stagethr_gSrc thr_sDataq_tilek_tileiv_tiler_gr_betaxbeta_x softplus_x exp_beta_x log_input log_result r_g_valuesum_qsum_krC inv_norm_q inv_norm_k end_v_tiles next_v_tiles next_stage gSrc_next sData_nextrow row_offsetsum_hk sData_tilev_newsum_hq gDst_tileo_idxrrr *gdn_decode_kernel_small_batch_pretransposes<                               .rc\ CsPtj\}}}|d}tj}tj|}tj\}}}||} ||}!|!||}"d}#t||!}$t|| |#|!f}%t||!}&t| | |#|!f}'tj }(|( tj|d})|( tj t |fd}*|( tjt |fd}+t tj |fddtj},t tj |fddtj}-t tj |fddtj}.t tj |fddtj }/t tj |fddtj }0t tj |fddtj }1||}2tj||ddf}3t|dttf|ddf}4t|3ttfd }5||}6ttd|}7t|7D]*}8|8t}9|5dd|8f}:|)dd|9f};|6|:}<|6|;}=t||<|=tjqt|ddd|f| |#|"|f}>t| ddd|f| |#|"|f}?t|>|/t|?|0t|D]}@t|/|@|-|@<t|0|@|,|@<qUt| ddd|f| |#|!|f}At|A|1t|D]}@t|1|@|+|2|@<qtjd }Bd }C|dkr|%|&}D||D}Ed }F|E|krtj|Ed d }Gtd |G}Httj|Hd d }Ittd ||I}Fn|D}Ftj|$d d  |F}Jd d tj|' d d }Ctj|Jd d }Btj|Bd}Btj|Cd}C|rpd }Kd }Lt|D]}@|K|-|@|-|@7}K|L|,|@|,|@7}LqdD]}M|Ktjj|K|Mddd7}K|Ltjj|L|Mddd7}Lq)tj |Kdd d }Ntj |Ldd d }Ot|D]}@|-|@|N|-|@<|,|@|O|,|@<q\t|D] }@|-|@||-|@<qut|D] }8|8t}9tj!dtj|8|7}P|P|kr|Pt}Q|5dd|Pf}R|)dd|Qf}S|6|R}<|6|S}=t||<|=tjtdtdD]}T|d}Ud }Vt|)d|df|T|U||9f}Wt|W|.t|D]}@|.|@|B|.|@<|V|.|@|,|@7}VqdD]}M|Vtjj|V|Mddd7}Vq |+|8t|T|U|V}X|X|C}Xd }Yt|D]}@|.|@|,|@|X7<|Y|.|@|-|@7}Yq0t|4dd|dfd|T|U||8f}Zt|.|ZdD]}M|Ytjj|Y|Mddd7}Yqc|8t|T|U}[|dkr|[|krt |Y|*|[<qϐqtj||kr|*|| | |#|!|f<dSdSr7)"rGrHrIrJrKrLrNrRrOrPrQrSrTrUrVrWrXrYrZr[r\r]r^r_r`rarbrcrdrerfrgrhri)\rrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6rjrkrlrJrmrprqrrrsrrrrrtrurvrwrxryrzr{r|r}r~rrrrrrrrrrrrrrrrrrrrrrrrrrrCrrrrrrrrrrrrrrrrr (gdn_decode_kernel_big_batch_pretransposes2                              . rstreamc#Cs |jjd|jjd|jjd}}}tjtjtjjdtj dd}tj ddd }t d }t |||}t |t }td } tj t ttftdt tfd }!d t ttd |d|d }"t|||!| |||||||||| | | | | |||||||||j|tddftddg|"|d dS)z>Launch original pipelined kernel for small batch pretranspose.rrr cache_moder num_bits_per_copyrrrrr9rrrrgridblockrtrN)layoutshaperGmake_copy_atomr CopyG2SOp LoadCacheModeGLOBALrNrRrTmake_tiled_copy_tvceil_divrXrYr\rlaunchrM NUM_THREADS#rr!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6rrv_dimk_dim copy_atom thread_layout val_layoutrr rr smem_bytesrrr .run_gdn_decode_kernel_small_batch_pretransposesr      $  rc#Cs|jjd|jjd|jjd}}}tjtjtjjdtj dd}tj ddd }t d }t |||}t |t }td } tj t ttftdt tfd }!d t ttd |d|d }"t|||!| |||||||||| | | | | |||||||||j|ddftddg|"|d dS)Nrrrrr rrrr9rrrr)rrrGrrrrrrNrRrTrrrXrYr\rrrrrrr ,run_gdn_decode_kernel_big_batch_pretransposesr      $ rdtypec CiS)zECache compiled kernel for given configuration (pretranspose version).r r/r0r1r.r2r3rr-r5rrr _get_compiled_decode_kernelrc Cr)zECache compiled kernel for given configuration (nontranspose version).rrrrr (_get_compiled_decode_kernel_nontransposerrstateoutput initial_stateinitial_state_indicesc 0Cs|j\} }}}|j\}}}}| du}|| duksJd|rA| jd}| j||||fks@Jd|d|d|d|d| j n%|dusIJd |j| |||fksfJd | d|d|d|d|j |rk| jn|j}to|tjko|d vo|d ko|d k}|r|jtjtjfvsJd |j|jtjksJd|j|dur|dn|}t|||dd|||||r| n|| | |d }| du}|r| jn|j}| dur| |n|} | j|kr| |} |r| n|}| |fS|rJd|dksJd||jtjks Jd|j|d ksJd||d ks%Jd||t dks6Jdt d||jtjtjfvsHJd |j|jtjksWJd|j|dur`|d}| du}|rj| jn|j}| durtj | |||ftj|j d} | | |||}| ||||||j|| f }t|}d|vs|dj |j krtj | tj|j d|d<tj | dtj|j d|d<|d} |d}!d|vrAttjj}"t|dd }#t|dd }$t|dd }%t|dd }&t|dd }'t|dd }(t|dd })t|dd }*t| dd }+t| dd },t|!dd }-t}.tj|.|#|$|%|&|'|(|)|*|+|,|-f dd||| ||||d!| d"|"d#d$}/|/|d<n|d}/ttjj}"|/||||||||| | |!|" | j|krh| |} |sx|| | |||| |fS)%u Gated Delta Rule Decode kernel for single-token generation. This implements the decode phase of gated delta rule linear attention, processing one token at a time and updating the recurrent state. Args: q (torch.Tensor): Current query of shape ``[B, 1, H, K]``. Must be float16/bfloat16. k (torch.Tensor): Current key of shape ``[B, 1, H, K]``. Must be float16/bfloat16. v (torch.Tensor): Current value of shape ``[B, 1, HV, V]``. Must be float16/bfloat16. state (Optional[torch.Tensor]): Current state of shape ``[B, HV, V, K]`` (v-major / K-last layout). Float32: legacy kernel (T=1 only). Bfloat16: gdn_decode_klast_bf16_state backend when T in 1..4 and K=V=128. Will be updated in-place. Pass ``None`` when using ``initial_state`` / ``initial_state_indices`` instead. A_log (torch.Tensor): Log decay parameter of shape ``[HV]``. Must be float32. a (torch.Tensor): Input-dependent decay of shape ``[B, 1, HV]``. Must be float16/bfloat16. dt_bias (torch.Tensor): Decay bias of shape ``[HV]``. Must be bfloat16 or float32. b (torch.Tensor): Update gate (beta) input of shape ``[B, 1, HV]``. Must be float16/bfloat16. scale (Optional[float]): Scale factor for queries. If None, defaults to ``1 / sqrt(K)``. output (Optional[torch.Tensor]): Pre-allocated output tensor of shape ``[B, 1, HV, V]``. If None, will be allocated automatically. use_qk_l2norm (bool): Whether to apply L2 normalization to q and k. Default: ``True``. initial_state (Optional[torch.Tensor]): State pool of shape ``[pool_size, HV, V, K]`` (K-last / K-contiguous, same layout as the per-batch ``state`` argument). When provided, the kernel gathers directly from the pool using ``initial_state_indices`` and writes updates back in-place — eliminating the caller-side gather/scatter overhead. Requires bfloat16 state with T in 1..4 and K=V=128 (bf16 fast path). initial_state_indices (Optional[torch.Tensor]): Per-batch indices of shape ``[B]`` (int32 or int64) mapping each batch entry to its slot in ``initial_state``. Required when ``initial_state`` is provided. Returns: Tuple[torch.Tensor, torch.Tensor]: - output: Output tensor of shape ``[B, 1, HV, V]`` - state or initial_state: Updated state (in-place). Note: - Requires SM90+ (Hopper, Blackwell, etc.) - State is always updated in-place; the pool path writes directly into ``initial_state`` memory (no separate scatter step needed) - State layout is v-major (K-last): [B, HV, V, K]. When state is bfloat16 and T in 1..4 with K=V=128, the gdn_decode_klast_bf16_state kernel is used (supports both the direct ``state`` path and the pool+indices path). - pool+indices (``initial_state``/``initial_state_indices``) only supported via the bf16 fast path; float32 state raises an error. - Legacy path (float32 state, T=1): K and V must be multiples of 4. NzAinitial_state and initial_state_indices must be provided togetherr(Expected initial_state shape [pool_size=, HV=, V=, K=], got z.Either state or initial_state must be providedExpected state shape [B=)rrrr q must be float16/bfloat16, got A_log must be float32, got r>4@) r!r"r#r+r,r$r%r&r'initial_state_sourceruse_qk_l2norm_in_kernelr-zpool+indices (initial_state/initial_state_indices) requires bfloat16 state with T in 1..4 and K=V=128 (the gdn_decode_klast_bf16_state fast path)r Decode only supports T=1, got T=state must be float32, got K must be at least 128, got K=V must be at least 128, got V=V must be divisible by ( to prevent out-of-bounds access, got V=rdevicer)r*compiledr assumed_alignTF--enable-tvm-ffi)r+r,r-r.r/r0r1r2r3r4r5r6roptions)rr&_GDN_DECODE_KLAST_BF16_STATE_AVAILABLEtorchbfloat16float16float32-_gated_delta_rule_gdn_decode_klast_bf16_statecopy_torXzerosrreshaperint32cudaCUstreamcurrent_stream cuda_streamrrrGcompile is_contiguous)0r$r%r&rr!r"r#r'r-rr5rrr/r0r1r2rkr.r3use_pool pool_size state_dtypeuse_gdn_decode_klast_bf16_state scale_valoutoutput_provided target_dtype return_stater cache_keycacher)r*rh0_source_tensor A_log_tensora_tensordt_bias_tensorq_tensork_tensorv_tensorb_tensoro_tensorh0_indices_tensorcu_seqlens_tensorrun_funcrrrr $gated_delta_rule_decode_pretransposesM "                          rci Cstj\}}}|d}tj}tj|}tj\}}}d}t|}d|}t|}|t}|t}|t}||}||} ||}!|!||}"| | }#|#dkr||}$||}%||}&|&|%}'t j }(|( t j |d})tjtfdd}*|( t j |*d}+tjtfdd},tjtfdd}-|( t j |,d}.|( t j |-d}/|tkrt || d|"|f|.|<t || d|"|f|/|<|#||!}0||0ddf}1t|1ttfd}2||}3t td |}4t|4D].}5||5}6|5t}7|2dd|6f}8|)dd|7f}9|3|8}:|3|9};t||:|;tjqt | |!}t || d|!f}?d }@d }A|dkr|>|=}B| |B}Cd }D|C|krttj|Cd d }Et d |E}Ft tj|Fd d }Gt t d | |G}Dn|B}Dtj|r?r@rArB)rrrFrunrollrrr)!rGrHrIrJrKrLTILE_V_SMALL_NT TILE_K_NTNUM_BLOCKS_PER_STATE_SMALL_NTrNrOrPrQrRrTrWrZr[ NUM_STAGES_NTr]r^r_r`rardrerfrVrgrhrirSrc)irrrr r$r%r&r"r'r!r#r(r)r+r,r-r1r.r5rjrk in_warp_tidrJrLNUM_WARPS_SMALLV_PER_WARP_SMALLROWS_PER_ITER_SMALLNUM_K_ITERS_SMALLrmrnro start_v_tilerprqrrpool_idxk_localv_localv_basev_idxrtru smem_o_layoutsmem_o smem_k_layout smem_q_layoutsKsQflat_idxrrrr v_tile_offsetrrrrrrrrrrrrrrrrrrr sum_q_partial sum_k_partialq_valk_valrCrr local_sum_q local_sum_knext_v_tile_offset next_v_tilerrrv_globalr_vrk_iterk_basek_idxh_valr_k_valrrh_oldr_q_valh_new v_global_outflat_tidk_writev_writev_global_writerrr *gdn_decode_kernel_small_batch_nontransposesr                                          wrLc_ CsJtj\}}}|d}tj}tj|}tj\}}}||}||}|||}| |}|dkr|t}|t}|t}||}tj } | tj |d}!tj t fdd}"| tj |"d}#tj tfdd}$tj tfdd}%| tj |$d}&| tj |%d}'|tkrt ||d||f|&|<t ||d||f|'|<|||}(||(ddf})t|)tt fd}*||}+ttd|},t|,D]*}-|-t}.|*dd|-f}/|!dd|.f}0|+|/}1|+|0}2t||1|2tjqt | |}3t | |}4t ||d|f}5t ||d|f}6d }7d }8|dkrq|5|4}9| |9}:d };|:|krRtj|:d d }t t d | |>};n|9};tj|3d d  |;}?d d tj|6 d d }8tj|?d d }7tj|7d}7tj|8d}8tj|rQd }@d }A|tkr|'|}B|&|}C|B|B}@|C|C}Ad D]}D|@tjj|@|Dddd7}@|Atjj|A|Dddd7}Aq|dkr|@|#|<|A|#|d<tjd }Ed }F|dkr'd }Gd }H|tkr|#|}G|#|d}HdD]}D|Gtjj|G|Dddd7}G|Htjj|H|Dddd7}Hq|dkr'tj|Gdd d |#d<tj|Hdd d |#d<tj|#d}E|#d}F|tkrK|&||F|&|<|'|||E|'|<tjn|tkr^|'|||'|<tjt|D]:}-|-t}.tjdtj|-|,}I|I|kr|It}J|*dd|If}K|!dd|Jf}L|+|K}1|+|L}2t||1|2tj|-t |}Mt ||d||Mf}Nd }Ott ddD]}P|Pt!}Q|Q|}R|!|R||.f|7}S|&|R}T|O|S|T7}OqdD]}D|Otjj|O|Dtddd7}Oq|N|O|8}Utj|U|}Ud }Vtt ddD]/}P|Pt!}Q|Q|}R|!|R||.f|7}W|&|R}T|'|R}X|W|T|U}Y|Y|!|R||.f<|V|Y|X7}Vq dD]}D|Vtjj|V|Dtddd7}Vq=|dkre|-t |}Zt"|V| |d||Zf<tjt#t D]*}P||Pd}[|[t }\|[t }]|\tkr|!|\|]|.f}S|-t |]}^|S||(|\|^f<qotjqgdSdS)zDLarge batch kernel for (N, 1, ...) format with K-major state layout.rrr r8r9Nrrr;Tr<r>r?r@rArBr rrFrr)$rGrHrIrJrKrL V_PER_WARP_NTrNrOrPrQrRrT TILE_V_NTrrWrZr[r!r]r^r_r`rardrerfrVrgNUM_WARPS_LARGE_NTrhriNUM_K_ITERS_NTROWS_PER_ITER_NTrSrc)_rrrr r$r%r&r"r'r!r#r(r)r+r,r-r1r.r5rjrkr"rJrmrprqrrr(r)r*r+r,rtrur-r.r/r0r1r2r3rrrrrrrrrrrrrrrrrrrrrrrr5r6r7r8rCrrr9r:r<rrrr=r>rr?r@rArBrCrrrDrErFrGrHrIrJrKrrr (gdn_decode_kernel_big_batch_nontransposesP                                            vrRc"Cs|jj\}}}| jjd|}tjtjtjjdtj dd}t |t }tj t t tftdt tfd}tj ddd}t d }t|||} d t ttd t d t d d }!t| ||||||||||| | | | | |||j|tddftddg|!|d dS)Nrrr rrr9)rr)rrrrrrr)rrrGrrrrrrNrRrrrTrr!TILE_V_SMALL_PADDED_NTrrLrr NUM_THREADS_NT)"r*r$r%r&r"r'r!r#rr)r(r+r,r-r/r0r1r.r2r3r4r5r batch_hv_dimrrrrnum_v_tiles_smallsmem_layout_smallthread_layout_smallval_layout_smalltiled_copy_load_smallsmem_bytes_smallrrr .run_gdn_decode_kernel_small_batch_nontransposezsj       r\c"Cs|jj\}}}| jjd|}tjtjtjjdtj dd}t |t }tj t t tftdt tfd}tj ddd}t d }t|||} d t ttd t d t d d }!t| ||||||||||| | | | | |||j|ddftddg|!|d dS)Nrrr rrr9)rr )r rrrrrr)rrrGrrrrrrNrRrrNrTrr!TILE_V_PADDED_NTrrRrNUM_THREADS_LARGE_NT)"r*r$r%r&r"r'r!r#rr)r(r+r,r-r/r0r1r.r2r3r4r5rrUrrrrr base_smem_layoutrrrrrrr ,run_gdn_decode_kernel_big_batch_nontransposesf      r`c )Csh|j\} } } }| dksJd| |j\}}}}|j| |||fks6Jd| d|d|d|d|j |dksAJd ||dksLJd ||td ks\Jd td ||jtjtjfvsmJd|j|jtjks{Jd|j|jtjksJd|j|dur|d}| du}|r| jn|j}| durtj| | ||ftj|jd} | }| | |||}| | | ||||j|| f }t |}d|vs|dj|jkrtj | tj |jd|d<tj| dtj |jd|d<|d}|d}d|vr{ttjj}| tk}|rt}nt}t|dd}t|dd}t|dd}t|dd} t|dd}!t|dd}"t|dd}#t|dd}$t| dd}%t|dd}&t|dd}'tj||'|!|"|#||$|| ||&|%f dd|| | | |||d| |dd }(|(|d<n|d}(ttjj}|(||||||||||| | ||kr||| j|kr| |} | |fS)aGated Delta Rule Decode kernel (K-major layout, no transpose needed). This implements the decode phase of gated delta rule linear attention, processing one token at a time and updating the recurrent state. This version uses K-major state layout [B, HV, K, V] which is more natural and doesn't require transposition. Args: q (torch.Tensor): Current query of shape ``[B, 1, H, K]``. Must be float16/bfloat16. k (torch.Tensor): Current key of shape ``[B, 1, H, K]``. Must be float16/bfloat16. v (torch.Tensor): Current value of shape ``[B, 1, HV, V]``. Must be float16/bfloat16. state (torch.Tensor): Current state of shape ``[B, HV, K, V]`` (k-major layout). Must be float32. Will be updated in-place. A_log (torch.Tensor): Log decay parameter of shape ``[HV]``. Must be float32. a (torch.Tensor): Input-dependent decay of shape ``[B, 1, HV]``. Must be float16/bfloat16. dt_bias (torch.Tensor): Decay bias of shape ``[HV]``. Must be bfloat16 or float32. b (torch.Tensor): Update gate (beta) input of shape ``[B, 1, HV]``. Must be float16/bfloat16. scale (Optional[float]): Scale factor for queries. If None, defaults to ``1 / sqrt(K)``. output (Optional[torch.Tensor]): Pre-allocated output tensor of shape ``[B, 1, HV, V]``. If None, will be allocated automatically. use_qk_l2norm (bool): Whether to apply L2 normalization to q and k. Default: ``True``. Returns: Tuple[torch.Tensor, torch.Tensor]: - output: Output tensor of shape ``[B, 1, HV, V]`` - state: Updated state tensor of shape ``[B, HV, K, V]`` Note: - Requires SM90 (Hopper) architecture - State is updated in-place - K and V must be multiples of 4 for vectorized loads - State layout is k-major: [B, HV, K, V] (no transpose needed) rrrrrrrr rrrrrrrrNrrr)r*rrrr>rTr) r+r,r-r/r0r1r.r2r3r4r5rr)rrNrrrrrrr contiguousviewrarangerrrrrSMALL_BATCH_THRESHOLD_NTr\r`rrGrdata_ptrrr))r$r%r&rr!r"r#r'r-rr5r/r0r1r2rkr.r3rr state_contiguousrr r r)r*ruse_small_batchrr rrrrrrrrrrrrrr gated_delta_rule_decodes;"                  rhintermediate_statestile_vdisable_state_updatecache_intermediate_statesc] CsTtj\}}}|d}tj} tj| } ||}!d|!}"d|"}#||!}$||!}%| |"|%}&tj\}'}}|'|}(|'|})|)|}*|)|}+|*||},| |+}-t||*}.t||*}/tj }0|0 tjtj ||f|ddfdd}1|0 tjtj ||f|ddfdd}2|0 tjt |fd}3|0 tjt |fd}4t tj |fddtj}5t tj |fddtj}6t tj |fddtj}7t tj |fddtj }8t tj |fddtj }9|-dkr|$|}:t|D]o};t|ddd|f|+|;|,|$f}t|8|>|5|><t|9|>|6|><q%t|rd }?d }@t|D]}>|?|5|>|5|>7}?|@|6|>|6|>7}@qJd D]}A|?tjj|?|Ad d d 7}?|@tjj|@|Ad d d 7}@qdtj|?ddd|}Btj|@ddd}Ct|D]}>|5|>|B|5|><|6|>|C|6|><qnt|D] }>|5|>||5|><q||!krt|D]}>|5|>|1|;|:|>f<|6|>|2|;|:|>f<qt||+|;|*f}Dt| |+|;|*f}E|D|/}F||F}Gtj|Gdd}Htd|tjtd|Hdd}I|G|kr!tdntd }J|J|Itd|J|F}Ktj|.dd |K}Ltdtdtj|E dd}Mtj|Ldd}N|dkrc|N|3|;<|M|4|;<qtj||#}Ot|OD]4}P|(||&|O|P}Q|Q|kr|-||*}Rt|dd|f|R|Q|$f}St|S|7t|D]};t|1d|f|;|$f}Tt|2d|f|;|$f}Ut|T|5t|U|6|3|;}N|4|;}Mt|D] }>|7|>|N|7|><qd }Vt|D] }>|V|7|>|6|>7}Vqd D]}A|Vtjj|V|Ad d d 7}Vqt| |+|;|*|Qf}W|W|V|M}Xt|D]}>|7|>|6|>|X7<qt|rQ|+|||;||*}Yt|dd|f|Y|Q|$f}Zt|7|Zd }[t|D] }>|[|7|>|5|>7}[qXd D]}A|[tjj|[|Ad d d 7}[qh|$dkrt |[| |+|;|*|Qf<qt| rt|dd|f|R|Q|$f}\t|7|\qrdSdS)av Parallel MTP kernel - each block handles one [TILE_V, TILE_K] tile. Grid: (B * HV * num_v_tiles, 1, 1) Each block: - Loads its v_tile of state into registers - Processes all T time steps with state in registers - Writes output and optionally updates state This matches Triton's parallelization strategy for better small-batch performance. rrr rr9rr8rr;r?r@rArBrFTr<r>N)rGrHrIrJrKrLrNrRrOrPrQrTrUrSrcrWrb const_exprrgrhrdrerV)]rrirr rjr!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6rkrlrjrkrlrJthreads_per_groupgroups_per_warp num_groups lane_in_group group_in_warp group_idxrmi_vtmprqrprr cache_idxrrrtr2r1sGsBetaryrxrzr{r|r~rsrrrrrrCinv_norm_q_scaledrrrrrr softplus_val use_softplusrrrrrows_per_group row_in_groupr,flat_state_idxh_tilesQ_tilesK_tilerr>rr3 inter_tiler h_tile_outrrr gdn_verify_kernel_mtpsH+                          \rc#Cs|jjd|jjd|jjd}}}t||} ||| }!d||dd||dd|d|d}"t|||| ||||||||| | | | | ||||||||||||j|!ddftddg|"|ddS)Nrrrrr r r)rrrGrrrNUM_THREADS_MTP)#rrir!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3rjrr4r5r6rkrlrrkrrr grid_sizerrrr run_gdn_verify_kernel_mtp sj !     rr cache_stepscCr)z2Cache compiled MTP kernel for given configuration.r)r/r0r1r.r2r3rrrkrlr-r5rjrrrr _get_compiled_mtp_kernel[ srintermediate_states_bufferc0Cs,|j\}}}}|j\}}}}|jd}t||}t||}|j||||fks:Jd|d|d|d|d|j |dksEJd||dksPJd |||dks`Jd |d ||jtjtjfvsqJd |j|jtjksJd |j|jtjksJd|j| dur|d} | du}|r| jn|j}| durtj||||ftj|j d} | tj ||||}| du}|r| jd}| jd}||ksJd|d|d| tj ||||| }n|}tjdddtj|j d}||||||||| || | ||f}t |} d| vs!| dj |j kr/tj|dtj|j d| d<| d}!d| vrttjj}"t|dd}#t|dd}$t|dd}%t|dd}&t|dd}'t|dd}(t|dd})t|dd}*t|dd}+t| dd},t|dd}-t|!dd}.tjt|#|$|%|&|'|(|)|*|+|,|-|.f iddddd| d|d |d!|d"|d#|d$|d%|d&|d'd(d)| d*d+d,| d-|d.|"d/d0}/|/| d<n| d}/ttjj}"|/|||||||||| ||!|" | s|s|| ||||| j|kr| |} | |fS)1am Gated Delta Rule MTP Kernel (Multiple Token Processing). This function processes multiple tokens (T > 1) in sequence, typically used for speculative decoding verification. It supports intermediate state caching for potential rollback scenarios. Args: q (torch.Tensor): Query tensor of shape ``[B, T, H, K]``. k (torch.Tensor): Key tensor of shape ``[B, T, H, K]``. v (torch.Tensor): Value tensor of shape ``[B, T, HV, V]``. initial_state (torch.Tensor): Initial state tensor of shape ``[pool_size, HV, V, K]`` (K-last layout). initial_state_indices (torch.Tensor): Indices mapping each batch to its initial state, shape ``[B]``. A_log (torch.Tensor): Log decay parameter of shape ``[HV]``. a (torch.Tensor): Input-dependent decay of shape ``[B, T, HV]``. dt_bias (torch.Tensor): Decay bias of shape ``[HV]``. b (torch.Tensor): Update gate input of shape ``[B, T, HV]``. scale (Optional[float]): Scaling factor for queries. If None, uses ``1/sqrt(K)``. output (Optional[torch.Tensor]): Pre-allocated output tensor of shape ``[B, T, HV, V]``. intermediate_states_buffer (Optional[torch.Tensor]): Buffer for caching intermediate states, shape ``[pool_size, T, HV, V, K]``. If None, intermediate states are not cached. disable_state_update (bool): If True, the initial state is not updated. Default: ``True``. use_qk_l2norm (bool): Whether to apply L2 normalization to q and k. Default: ``True``. Returns: Tuple[torch.Tensor, torch.Tensor]: - output: Output tensor of shape ``[B, T, HV, V]`` - initial_state: Updated state tensor (unchanged if disable_state_update=True) Note: - Requires SM90 (Hopper) architecture - Supports T > 1 (multiple token processing) - State layout is K-last: [pool_size, HV, V, K] - Optimized for speculative decoding verification scenarios rrrrrrr rrrrrz#initial_state must be float32, got rNrrrz9intermediate_states_buffer second dimension (cache_steps=z) must be at least T=z" to prevent out-of-bounds indexingr*rrrr+r>r,rr-r.r/r0r1r2r3rjrr4Tr5r6Frkrlrrr)rrrrrrrrrrrrrarrrrrrrrGrrrr)0r$r%r&rrr!r"r#r'r-rrrkr5r/r0r1r2rkr.r3rrjrrr rrl buffer_sizerrir r r*rr intermediate_states_tensorrrrrrrrrrrrrrr gated_delta_rule_mtpp s&C   "                    !  rr8)NNTNN)NNT)NNNTT)L__doc__ functoolstypingrrrrN cutlass.cuterGcutlass.cute.nvgpurcutlass.cute.runtimercuda.bindings.driverbindingsdriverr api_loggingr_FLASHINFER_AVAILABLE ImportError!gdn_kernels.gdn_decode_bf16_stater rrrXrYr\rrMrrNr]rrSr!rTr r^rOrMrQrPrd TILE_K_MTPrintrrkernel TiledCopyTensorLayout Constexprfloatboolrrjitrrrr rrrrrLrRr\r`rhrrrrrrrr s                            f      j               r      ^      P      S     L            Y