o :/i6@sddlZddlZddlmZddlmZmZddlmZm Z ddl m Z ddl m Z mZmZddlZddlmZddlmZmZmZddlmZdd lmZdd lmZeeZed Zed Z ed edfddZ!GdddZ"dej#j$d dfddZ%dS)N)abstractmethod)Callable Generator)contextmanager nullcontext)CodeType)Any ParamSpecTypeVar)CompilationMode CUDAGraphModeget_current_vllm_config)DynamicShapesType) init_logger)layerwise_nvtx_marker_contextRPreturnNNNccsbtjjj}tjjj}zdtjj_dtjj_dVW|tjj_|tjj_dS|tjj_|tjj_w)aContext manager for compilation settings. This manager sets higher dynamo cache limits for compilation. (Needed for qwen2_5_vl see test_qwen2_5_vl_evs_functionality). Generally a recompilation can happen whenever we use a new backend instance in torch.compile. ii N)torch_dynamoconfigcache_size_limitaccumulated_cache_size_limit)original_cache_sizeoriginal_accumulated_cachere/lsinfo/ai/hellotax_ai/llm_service/venv_vllm/lib/python3.10/site-packages/vllm/compilation/wrapper.py_compilation_contexts     rc@seZdZdZdededefddZdeeefdej dej defdd Z dd d Z dededefd dZ dededefddZedededefddZdefddZdededd fddZededfddZd S)TorchCompileWithNoGuardsWrapperaM A wrapper class for torch.compile, it ensures that all guards are dropped when CompilationMode is not CompilationMode.STOCK_TORCH_COMPILE. When guards are dropped, the first time __call__ is invoked, a single compilation is triggered. Dynamo should never be traced again after that since we drop all guards. argskwargsrcOs.t|dsJ|j|i||j|i|S)N_check_shape_invariants)hasattrr"forwardselfr r!rrrcheck_invariants_and_forward8szs   z>TorchCompileWithNoGuardsWrapper._call_with_optional_nvtx_rangeNc Csd|_t}||_|jj}|jj|_|durtd|j |}i}t |t r/|dkr/|jj }d|_ |jjj|_|jjj}|tjkrf|jrStjrLJddd|d<nttjd r`tjj|d<nd d|d<|j}|tjkrtjruJd |jr|Jd |j}t}tjrttjj d rtjj j!dd}nd}|d7}|d7}t"#||tj$|dd||d|_%Wdn1swYtjr|tjkrtjj&'|j(d|_)dSdSdS)NFz)Compilation mode cannot be NO_COMPILATIONinductorTz\compilation_config.dynamic_shapes_config.evaluate_guards requires VLLM_USE_BYTECODE_HOOK=0. cSdd|DS)NcSsg|]}|jdkqS) SHAPE_ENV) guard_type).0entryrrr os NTorchCompileWithNoGuardsWrapper.__init__....rxrrrosz:TorchCompileWithNoGuardsWrapper.__init__..guard_filter_fnskip_all_guards_unsafecSr6)NcSsg|]}dqS)Fr)r9_rrrr;xsr<rr=rrrr?xsz;UNBACKED dynamic shapes requires VLLM_USE_BYTECODE_HOOK=0. z)UNBACKED dynamic shapes do not add guardsenable_aot_compile)rCz/torch._dynamo.config.enable_aot_compile is not z.available. AOT compile is disabled and please z+upgrade PyTorch version to use AOT compile.) fullgraphdynamicbackendoptions)*compiledr vllm_configcompilation_configmodeobservability_configenable_layerwise_nvtx_tracingr* RuntimeError init_backend isinstancestrinductor_compile_config first_compiledynamic_shapes_configevaluate_guardstyper STOCK_TORCH_COMPILEenvsVLLM_USE_BYTECODE_HOOKr#rcompilerrAr$rUNBACKEDr'rVLLM_USE_AOT_COMPILErrpatchloggerwarningcompile_compiled_callable convert_frameregister_bytecode_hook bytecode_hook_compiled_bytecode) r&rIrKrFrGds_type compiled_ptr aot_contextmsgrrr__init__Nsj          z(TorchCompileWithNoGuardsWrapper.__init__cOs.t|jdstdtjd|j||fS)N aot_compilezaot_compile is not supported by the current configuration. Please make sure torch.compile is enabled with the latest version of PyTorch (current using torch: ))r#rarNr __version__rkr%rrrrks z+TorchCompileWithNoGuardsWrapper.aot_compilec Os:tjrO|jjjtjkr|j|i|S|js,t j j | |j|jg|Ri|S||j|jg|Ri|WdS1sHwYdS|jsU|jsXtnt jd}d|_t/||j|jg|Ri|WdWdS1swYWddS1swYdS)Nfail_on_recompileF)rXrYrIrJrKr rWrarerr eval_frameremove_from_cacheoriginal_code_objectr4_dispatch_to_compiled_coder$rSrUrrZ set_stancer)r&r r!r3rrr__call__sN $ Rz(TorchCompileWithNoGuardsWrapper.__call__cOsdS)Nrr%rrrr$sz'TorchCompileWithNoGuardsWrapper.forwardcCs |jjjS)z6Return the original code object of the forward method.)r.r$__code__)r&rrrrqs z4TorchCompileWithNoGuardsWrapper.original_code_objectold_codenew_codec Cs`||urdSt}|r1|jr1|j}|jj}|jjtj j d}|dkr,|dkr,n|r1|js|j d}|j|ks=J|j d|urFdS||_ |j }|r|d}|sz+ddl}||} t|d  } | | Wdn1swwYtd |Wn tyYnw|j jjtjkrd |jvrddl}||} d | } t| dSdS) z8Hook to save the compiled bytecode for direct execution.N_compilezconvert_frame.pyframer&ztransformed_code.pyrwz#Dynamo transformed code saved to %supdateaBAssigning / modifying buffers of nn.Module during forward pass is not allowed when using cudagraph inside the compiler because it will cause silent errors. Please use eager mode or fix the code. The following code contains clues about which buffer is being modified (please search for the usage of the function `update`): )rqsys _getframef_backf_codeco_name co_filenamesplitospathsepf_localsrerIcompile_debug_dump_pathexistsdepyf decompileopenwriter^debug ExceptionrJcudagraph_moder NONEco_namesrN) r&rvrwrz code_name file_namerdecompiled_filersrcfrirrrrdsP          z-TorchCompileWithNoGuardsWrapper.bytecode_hookrccsH|}|jdus J|j|jj_z dVW||jj_dS||jj_w)a+ Context manager to dispatch to internally compiled code for torch<2.8. Why does this work? Because Dynamo guarantees that the compiled bytecode has exactly the same arguments, cell variables, and free variables as the original code. Therefore we can directly switch the code object in the function and call it. See https://dev-discuss.pytorch.org/t/what-is-the-relationship-requirement-among-original-bytecode-transformed-bytecode-and-bytecode-returned-by-hooks-in-dynamo/1693/7 for more details. N)rqrer.r$ru)r&originalrrrrr s  z:TorchCompileWithNoGuardsWrapper._dispatch_to_compiled_code)rN)r/ __module__ __qualname____doc__rr'rrrr r!r4rjrkrtrr$rrqrdrrrrrrrrr/s*  S  8rmodelcCst|ts t|dr |j}t|tsdSt|dr|jrdSddlm}d|_d|_d|_ d|_ d|_ d|_ d|_ d|_d|_d|_d|_d|_d|_d|_d|_t|drYd|_t|drad|_|jj}d |_d |_||jj_t |dS) zJ Clean up compiled model and captured CUDA graphs for elastic EP. rNdo_not_compiler)compilation_counteraot_compiled_fn#was_aot_compile_fn_loaded_from_diskF)!rPrr#rrvllm.compilation.counterrnum_models_seennum_graphs_seennum_piecewise_graphs_seen$num_piecewise_capturable_graphs_seennum_backend_compilationsnum_gpu_runner_capture_triggersnum_cudagraph_capturednum_inductor_compilesnum_eager_compilesnum_cache_entries_updatednum_compiled_artifacts_savedstock_torch_compile_countnum_aot_compilesnum_aot_artifacts_savednum_aot_artifacts_loadedrrrIrJ cache_dirlocal_cache_dirrqr.r$rurj)rrrJrrrreset_compile_wrappersB     r)&rr}abcrcollections.abcrr contextlibrrtypesrtypingrr r r vllm.envsrX vllm.configr r r vllm.config.compilationr vllm.loggerrvllm.utils.nvtx_pytorch_hooksrr/r^rrrrnnModulerrrrrs*      p