o :/iKb@sUdZddlZddlZddlZddlZddlZddlZddlmZddl m Z m Z ddl Z ddl Z ddlmZddlmZddlmZddlmZeeZejdd Zejd efd d Zejd efd dZde de d e fddZde d e dBfddZ!efde de dede ffddZ"e"ddZ#e"ddZ$e"ddZ%e"ddZ&e"d d!Z'e"d"d#Z(e"d"d$Z)e"d"d%Z*e"d"d&Z+e"d'd(Z,e"d"d)Z-e"d*d+d,d-d.Z.d/Z/ee0d0<ejd efd1d2Z1ejd efd3d4Z2ejd efd5d6Z3ejd efd7d8Z4ejd efd9d:Z5ejd efd;d<Z6ejd efd=d>Z7ejd efd?d@Z8ejd efdAdBZ9ejd efdCdDZ:d edBfdEdFZ;dGedPedQedBdRedSed efdTdUZ?erddVl@mAZAdWe jBdXe jBdYe jBd dfdZd[ZCdWe jBdXe jBdYe jBd dfd\d]ZDeAd^eCdWgeDd_e jEjFd`gdadbdce jBdde jBdee jBdfe jBdge jBdhe j>diedje d e jBfdkdlZGe jEHd`dce jBdde jBdee jBdfe jBdge jBdhe j>diedje d e jBfdmdnZIe jEjFdogdadbdce jBdde jBdee jBdfe jBdhe j>dje d e jBfdpdqZJe jEHdodce jBdde jBdee jBdfe jBdhe j>dje d e jBfdrdsZKe jEjFdtgdadbdue jBdve jBd eLe jBe jBffdwdxZMe jEHdtdue jBdve jBd eLe jBe jBffdydzZNe jEjFd{gdadb |ddce jBdde jBdee jBdfe jBd}e j>dje d e jBfd~dZOe jEHd{ |ddce jBdde jBdee jBdfe jBd}e j>dje d e jBfddZP |ddue jBde jBde jBde jBd}e j>dje d e jBfddZQdue jBde jBde jBde jBde jBd}e j>dje d e jBfddZR ddue jBde jBde jBde jBd}e j>de jBdBd e jBfddZSdue jBdve jBd eLe jBe jBffddZTe"ddZUejd efddZVejd efddZWdede j>de jBde jBfddZXgdZYdS)zoCompatibility wrapper for FlashInfer API changes. Users of vLLM should always import **only** these wrappers. N)Callable)AnyNoReturn) init_logger)vllm_is_batch_invariant)current_platformFLASHINFER_CUBINS_REPOSITORYzWhttps://edge.urm.nvidia.com/artifactory/sw-kernelinferencelibrary-public-generic-local/returncCs,tjrdStjddurdStddS)z7Return `True` if flashinfer-cubin package is available.Tflashinfer_cubinNz&flashinfer-cubin package was not foundF)envsVLLM_HAS_FLASHINFER_CUBIN importlibutil find_speclogger debug_oncerrb/lsinfo/ai/hellotax_ai/llm_service/venv_vllm/lib/python3.10/site-packages/vllm/utils/flashinfer.pyhas_flashinfer_cubin&s  rcCsDtjddurtddSts tddur tddSdS)z8Return `True` if flashinfer-python package is available. flashinferNz2FlashInfer unavailable since package was not foundFnvcczSFlashInfer unavailable since nvcc was not found and not using pre-downloaded cubinsT)r rrrrrshutilwhichrrrrhas_flashinfer1s r___cOstd)z/Placeholder for unavailable FlashInfer backend.zFlashInfer backend is not available. Please install the package to enable FlashInfer kernels: https://github.com/flashinfer-ai/flashinfer) RuntimeError)rrrrr_missingDsr module_namec Cs&zt|WSttfyYdSw)zBSafely import a submodule and return it, or None if not available.N)r import_module ImportErrorModuleNotFoundError)rrrr_get_submoduleMs  r" attr_name fallback_fn.cs&tjfddfdd}|S)z5Create a lazy import wrapper for a specific function.cs&tsdSt}|rt|dSdSN)rr"getattr)mod)r#rrr _get_impl[sz'_lazy_import_wrapper.._get_implcs*}|dur|i|S||i|Sr%r)argskwargsimpl)r(r$rrwrapperbsz%_lazy_import_wrapper..wrapper) functoolscache)rr#r$r,r)r(r#r$rr_lazy_import_wrapperVsr/flashinfer.fused_moetrtllm_bf16_moetrtllm_fp8_block_scale_moetrtllm_fp8_per_tensor_scale_moecutlass_fused_moe$flashinfer.cute_dsl.blockscaled_gemmgrouped_gemm_nt_maskedr fp4_quantizenvfp4_batched_quantize*silu_and_mul_scaled_nvfp4_experts_quantizescaled_fp4_grouped_quantizezflashinfer.fp4_quantizationblock_scale_interleavetrtllm_fp4_block_scale_moezflashinfer.autotunerautotunecOstSr%) contextlib nullcontext)r)r*rrrsr@)r$F_is_fi_autotuningcCto tjdduS)z5Return `True` if FlashInfer comm module is available.flashinfer.commNrr rrrrrrhas_flashinfer_commsrEcC@tsdSgd}|D]\}}t|}|rt||sdSq dS)z7Return `True` if FlashInfer mnnvl all2all is available.F))rCMapping)zflashinfer.comm.mnnvl MnnvlMemory)flashinfer.comm.trtllm_alltoallMnnvlMoe)rIMoEAlltoallInfoT)rEr"hasattrrequired_functionsrr#r'rrrhas_flashinfer_nvlink_two_sided rOcCstsdStjdduS)zDReturn `True` if FlashInfer trtllm_moe_alltoall module is available.Fz#flashinfer.comm.trtllm_moe_alltoallN)rEr rrrrrrhas_flashinfer_nvlink_one_sidedsrQcCrB)z4Return `True` if FlashInfer MoE module is available.r0NrDrrrrhas_flashinfer_moesrRcCrB)z:Return ``True`` if FlashInfer cutedsl module is available.zflashinfer.cute_dslNrDrrrrhas_flashinfer_cutedslsrScCrF)z:Return `True` if FlashInfer TRTLLM fused MoE is available.F))r0r2)r0r3r0r<)r0trtllm_mxint4_block_scale_moeTrRr"rLrMrrrhas_flashinfer_trtllm_fused_moes rWcCrF)z;Return `True` if FlashInfer CUTLASS fused MoE is available.F))r0r4)rr7)rnvfp4_block_scale_interleaverTTrVrMrrr has_flashinfer_cutlass_fused_moerPrYcCrF)z=Return ``True`` if FlashInfer CUTLASS fused MoE is available.F))r5r6)rr:)r&silu_and_scaled_nvfp4_experts_quantizeT)rSr"rLrMrrr-has_flashinfer_cutedsl_grouped_gemm_nt_maskeds r[c Cs~trdSz tjtdd}|jdk}|rtd|WStd|j|WSty>}z td|WYd}~d Sd}~ww) zReturn `True` if NVIDIA's artifactory is accessible. This checks connectivity to the kernel inference library artifactory which is required for downloading certain cubin kernels like TRTLLM FHMA. T)timeoutz NVIDIA artifactory is accessiblez2NVIDIA artifactory returned failed status code: %dz+Failed to connect to NVIDIA artifactory: %sNF) rrequestsgetr status_coderr warning_once Exception)response accessibleerrrhas_nvidia_artifactorys$   rgcCstrdStdo tS)z TRTLLM attention is supported if the platform is SM100, NVIDIA artifactory is accessible, and batch-invariant mode is not enabled. Fd)rris_device_capability_familyrgrrrrsupports_trtllm_attentionsrjcCsddlm}|}|jjS)a, This function should only be called during initialization stage when vllm config is set. Return `None` if --attention-config.use_trtllm_attention is not set, return `True` if TRTLLM attention is forced to be used, return `False` if TRTLLM attention is forced to be not used. r)get_current_vllm_config) vllm.configrkattention_configuse_trtllm_attention)rk vllm_configrrrforce_use_trtllm_attention-s rp num_qo_heads num_kv_headscCs$tdurdSt}|o||dkS)z=Check if the current configuration supports TRTLLM attention.Fr)rprj)rqrr has_trtllmrrrcan_use_trtllm_attention;s rt num_tokens max_seq_lendcp_world_sizekv_cache_dtypeq_dtype is_prefillforce_use_trtllm has_sinkshas_specc Cs|dur|sdS|dkrtddSts|rtddS||dkr.|r,tddS| r9|s9tdd S|tkrFtd d S| rOtd d S|durs|rb|d k} | r`td | S|dkoi|d k} | rqtd| Stdd S)z*Return `True` if TRTLLM attention is used.NFzcTrtllm does not support returning LSE and as a result does not support DCP, reverting to FlashInferzkTRTLLM attention is not supported on this platform, but --attention-config.use_trtllm_attention is set to 1rzTRTLLM attention is not supported for this combination of query and key heads, but --attention-config.use_trtllm_attention is set to 1z:Using TRTLLM attention (enabled for speculative decoding).Tz,Using TRTLLM attention (query is quantized).z6Using TRTLLM attention (required for attention sinks).autoz/Using TRTLLM prefill attention (auto-detected).z.Using TRTLLM decode attention (auto-detected).zLUsing TRTLLM attention (--attention-config.use_trtllm_attention is set to 1))rrbrj info_oncer fp8_dtype) rqrrrurvrwrxryrzr{r|r} use_trtllmrrrrnCsT        rn)direct_register_custom_opkk_nopek_pecCsddlm}||||dS)arCustom op wrapper for flashinfer's concat_mla_k. This is an in-place operation that concatenates k_nope and k_pe into k. The kernel is optimized for DeepSeek V3 dimensions: - num_heads=128 - nope_dim=128 - rope_dim=64 Key optimizations: - Warp-based processing with software pipelining - Vectorized memory access (int2 for nope, int for rope) - L2 prefetching for next row while processing current - Register reuse for rope values across all heads Args: k: Output tensor, shape [num_tokens, num_heads, nope_dim + rope_dim]. Modified in-place. k_nope: The nope part of k, shape [num_tokens, num_heads, nope_dim]. k_pe: The rope part of k (shared), shape [num_tokens, 1, rope_dim]. This is broadcast to all heads. r) concat_mla_kN)flashinfer.concat_opsr)rrrrrrr_flashinfer_concat_mla_ks rcCsdSr%r)rrrrrr_flashinfer_concat_mla_k_fakesrflashinfer_concat_mla_k)op_nameop_func mutates_args fake_implzvllm::flashinfer_mm_fp4cuda)r device_typesABA_scaleB_scaleg_scaledtypeuse_8x4_sf_layoutbackendc Cs&ddlm}|||||||d||d S)Nr)mm_fp4) block_sizerr)rr) rrrrrrrrflashinfer_mm_fp4_rrrflashinfer_mm_fp4s rcCs tj|jd|jd||jdSNrr~rdevicetorchemptyshaper)rrrrrrrrrrrflashinfer_mm_fp4_fakes rz vllm::bmm_fp8cCs ddlm}||||||d|S)Nr)bmm_fp8)rr)rrrrrrbmm_fp8_rrrrs rcCs(tj|jd|jd|jd||jdS)Nrr~rr)rrrrrrrrr bmm_fp8_fakes rzvllm::flashinfer_nvfp4_quantizea a_global_sfcCs*ddlm}ddlm}||||jddS)Nr)SfLayout)nvfp4_quantizeF)sfLayout do_shuffle)rrr layout_8x4)rrrnvfp4_quantize_rrrflashinfer_nvfp4_quantizes   rcCs^|j\}}dd}||d}|d}||d}tj||dtj|jdtj||tj|jdfS)NcSs||d||S)Nr~r)xyrrrr@+sz0flashinfer_nvfp4_quantize_fake..rrr)rrruint8r)rrmnround_up rounded_mscale_n rounded_nrrrflashinfer_nvfp4_quantize_fake#s    rzvllm::mm_mxfp8cutlass out_dtypec Cs"ddlm}|||||d||dS)Nr)mm_mxfp8)outrr)rr)rrrrrr mm_mxfp8_rrrr5s rcCs tj|jd|jd||jdSrr)rrrrrrrrr mm_mxfp8_fakeNs rb block_scale_a block_scale_bcCsf|jdkr |jdks J|jd|jdksJ|jdkr'tdt|jt||||||dS)aiMXFP8 MM helper - mirrors flashinfer_scaled_fp4_mm API. Takes non-transposed weights and handles transpose internally. CRITICAL: mm_mxfp8 CUTLASS kernel requires SWIZZLED 1D scales for optimal performance and accuracy. Both input and weight scales should be in swizzled format from FlashInfer's mxfp8_quantize(is_sf_swizzled_layout=True). rr~zBmm_mxfp8 expects 1D swizzled weight scales for CUTLASS; got shape=)r)ndimr ValueErrortuplert)rrrrrrrrrflashinfer_mm_mxfp8]s  ralphac Cs|jdkr |jdks J|jdkr|jdksJ|ddkr&|ddks(J|jd|jdks4J|dvrD|tj}|tj}|dkrQ|jddkrQdnd }t||||||||d S) Nrr~)rcudnntrtllmr TF)rr)rstriderviewrrrr)rrrrrrrrrrrflashinfer_scaled_fp4_mms$    rscale_ascale_bbiascCs|jdkr |jdks J|jd|jdksJ|dkr$|dks&J|jtjkr2|jtjks4J|jjdkr@|jjdksBJ|jtjkrN|jtjksPJ|jjdkr\|jjdks^Jt | d| d|||d |jd|jd}|dur||}|S)Nrr~rrr) rrnumelrr float8_e4m3fnrtypefloat32r unsqueezer)rrrrrroutputrrrflashinfer_scaled_fp8_mms( rcCs t||Sr%)r)rrrrr$flashinfer_quant_nvfp4_8x4_sf_layouts rflashinfer.gemmfp8_blockscale_gemm_sm90cCstotdottddS)z>Return `True` if FlashInfer block-scale FP8 GEMM is available.Zrr)rris_device_capabilityrLr"rrrr"has_flashinfer_fp8_blockscale_gemms  rcCs tjotS)z>Return `True` if FlashInfer block-scale FP8 GEMM is supported.)r #VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFERrrrrr+is_flashinfer_fp8_blockscale_gemm_supportedsris_flashinfer_supported output_dtypeinputweightc Csb|sdSd}d}|j}|j}|tjko.|tjko.|tjko.|jd|dko.|jd|dk}|S)NF@rr~)rrbfloat16rr) rrrr N_MULTIPLE K_MULTIPLE weight_dtype input_dtypeshould_use_flashinferrrr-should_use_flashinfer_for_blockscale_fp8_gemms  r)r%flashinfer_trtllm_fp8_block_scale_moeflashinfer_cutlass_fused_moe)flashinfer_cutedsl_grouped_gemm_nt_maskedflashinfer_fp4_quantizer9r:rXr<r=rRrErOrQrYr[rrgrjrtrnrrrflashinfer_fp8_blockscale_gemmrr)NFF)rr%)Z__doc__r>r-r importlib.utilosrcollections.abcrtypingrrr_r vllm.envsr vllm.loggerr*vllm.model_executor.layers.batch_invariantrvllm.platformsr__name__renvironr`rr.boolrrrstrr"r/flashinfer_trtllm_bf16_moer*flashinfer_trtllm_fp8_per_tensor_scale_moerrrr8r9r:rXr<r=rA__annotations__rErOrQrRrSrWrYr[rgrjrpintrtrrnvllm.utils.torch_utilsrTensorrrlibrary custom_opr register_fakerrrrrrrrrrrrrrrr__all__rrrrs                 T                  & &