o :/ixL@sddlZddlmZmZmZddlmZddlmZm Z ddl Z ddl m Z ddl mZddlmZddlmZmZddlmZmZmZmZdd lmZmZmZmZmZmZdd l m!Z!dd l"m#Z#m$Z$m%Z%m&Z&m'Z'dd l(m)Z)m*Z*dd l+m,Z,ddl-m.Z.m/Z/m0Z0m1Z1ddl2m3Z3ddl4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:m;Z;mZ>ddl?m@Z@mAZAmBZBmCZCmDZDmEZEddlFmGZGddlHmIZIddlJmKZKddlLmMZMmNZNmOZOmPZPmQZQddlRmSZSddlTmUZUddlVmWZWmXZXddlYmZZZm[Z[m\Z\ddl]m^Z^m_Z_m`Z`ddlambZbddlcmdZdddlemfZfddlgmhZhdd limjZjdd!lkmlZldd"lmmnZndd#lompZpdd$lqmrZrer;dd%lsmtZteGeuZved&ePeOBePeOBd'Zwed(e`eUBdBd)Zxed*e d+ZyGd,d-d-ZzdS).N)CallableIterableSequence)Path) TYPE_CHECKINGAny)ValidationError)tqdm)TypeVaroverload)BeamSearchInstanceBeamSearchOutputBeamSearchSequencecreate_sort_beams_key_function)AttentionConfigCompilationConfig PoolerConfigProfilerConfigStructuredOutputsConfig is_init_field)CompilationMode) ConvertOption HfOverrides ModelDType RunnerOption TokenizerMode)WeightTransferInitRequestWeightTransferUpdateRequest) EngineArgs)ChatCompletionMessageParamChatTemplateConfigChatTemplateContentFormatOptionload_chat_template)init_pooling_io_processors) ScoreDataScoreMultiModalParam_cosine_similaritycompress_token_type_idscompute_maxsim_scoreget_score_promptscore_data_to_promptsvalidate_score_input)log_non_default_args) DataPromptProcessorInputs PromptTypeSingletonPrompt TextPrompt TokensPrompt) init_logger) LoRARequest)QuantizationMethods)ClassificationRequestOutputEmbeddingRequestOutputPoolingRequestOutput RequestOutputScoringRequestOutput)current_platform) PoolingParams) ChatParams merge_kwargs)conversation_to_seqparse_model_prompt prompt_to_seq)BeamSearchParamsRequestOutputKindSamplingParams) PoolingTask) TokenizerLike) UsageContext)Counter)is_mistral_tokenizer) maybe_tqdm) PauseMode) LLMEngine)LogitsProcessor)Metric_O)bounddefault_P)rP_R)rQcK@s eZdZdZddddddddddddddddddddddddddddddddddd "d ed ed ed edBdeeBdededede edBde de de dBdedBdedBde eBdBde dedede de de deedBd ed!ed"ed#eeBdBd$edBd%eeefdBd&edBd'eeefeBdBd(eeefeBdBd)eeefeBdBd*e dBd+e eeefBeBdBd,e eeeBdBd-ed.dfJd/d0Zd.efd1d2Zdd4ed.e fd5d6Zdd7d8Zd.efd9d:Z dd3dddd;dee#d?e$fBd@e"e%e%BdBdAe e dBdBeeefdBd.e e&fdCdDZ'    3 ddee#d?e$fBdBeeefdBd.e efdEdFZ(e)d3dGd>ee#d?e$fBd.e e&e*BfdHdIZ+e)d3dGdJee,e-ee,d?fBd>ee#d?e$fBd.e e,fdKdIZ+ dd3dGdJeee-eed?fBdBd>ee#d?e$fBd.e efdLdIZ+dMe.d@e%dBd.e%dBfdNdOZ/  P ddQee#d?e0fBdRedBdSe-d-eeefdBd.e e0f dTdUZ1dVe#e2j3ge0fd.e e0fdWdXZ4   ddedZe dBd.e e8f d[d\Z9 dd    3    ddie e<dedBdbe=dceeefdBddedeedfe eeefdBdBeeefdBd%eeefdBd.e.fdjdkZ?  3    3     ddle e<e"e e<Bd=ee"eBdBd>ee#d?e$fBd@e"e%e%BdBdedBdbe=ddedeedfe eeefdBdceeefdBdBeeefdBd%eeefdBd.e e&fdmdnZ@ dd3ddddodee#d?e$fBd@e e%e%BdBdqeCdBdBeeefdBd.e e*fdrdsZDd3ddddtdee#d?e$fBdpeBe"eBBdBd@e e%e%BdBdBeeefdBd.e eEf dudvZFdd3dddwdee#d?e$fBd@e e%e%BdBdBeeefdBd.e eGf dxdyZHdd3dddwdee#d?e$fBd@e e%e%BdBdBeeefdBd.e e*f dzd{ZId|e eJd}e eJd>ee#d?e$fBdpeBdBd@e e%e%BdBdBeeefd.e eKfd~dZLd|e eJd}e eJd>ee#d?e$fBdpeBdBd@e e%e%BdBdBeeefd.e eKfddZMd|e eJd}e eJd>ee#d?e$fBdpeBdBd@e e%e%BdBdBeeefdedBd.e eKfddZNd3dddddd|eOe"eOBePBe ePBd}eOe"eOBePBe ePBd>ee#d?e$fBdpeBdBd@e e%e%BdBdBeeefdBdedBd.e eKfddZQddedBd.dfddZRdddZS ddeded.efddZTdde deUfddZVdde edBfddZWd.e dfddZXdYeYe"eYBde d.e"eYfddZZd@e%dBe"e%dBBde d.e"e%dBfddZ[dAe e dBde d.e"e fddZ\d3dddd;dee#d?e$fBd@e"e%e%BdBdAe e dBdBeeefdBd.e efddZ]d3dddd;dee#d?e$fBd@e"e%e%BdBdAe e dBdBeeefdBfddZ^d3dddd3dddddd dle e<e"e e<BdYeeBBe"eeBBBdJee,d>ee#d?e$fBd@e"e%e%BdBdedBdbe=ddedeedfe eeefdBdceeefdBdBeeefdBd%eeefdBfddZ_ddd3ddee#d?e$fBf ddZaddddee#d?e$fBd.e e,fddZddeeeBd.dfddZfdegeBd.dfddZhd.efddZidS)LLMaxAn LLM for generating texts from given prompts and sampling parameters. This class includes a tokenizer, a language model (possibly distributed across multiple GPUs), and GPU memory space allocated for intermediate states (aka KV cache). Given a batch of prompts and sampling parameters, this class generates texts from the model, using an intelligent batching mechanism and efficient memory management. Args: model: The name or path of a HuggingFace Transformers model. tokenizer: The name or path of a HuggingFace Transformers tokenizer. tokenizer_mode: The tokenizer mode. "auto" will use the fast tokenizer if available, and "slow" will always use the slow tokenizer. skip_tokenizer_init: If true, skip initialization of tokenizer and detokenizer. Expect valid prompt_token_ids and None for prompt from the input. trust_remote_code: Trust remote code (e.g., from HuggingFace) when downloading the model and tokenizer. allowed_local_media_path: Allowing API requests to read local images or videos from directories specified by the server file system. This is a security risk. Should only be enabled in trusted environments. allowed_media_domains: If set, only media URLs that belong to this domain can be used for multi-modal inputs. tensor_parallel_size: The number of GPUs to use for distributed execution with tensor parallelism. dtype: The data type for the model weights and activations. Currently, we support `float32`, `float16`, and `bfloat16`. If `auto`, we use the `dtype` attribute of the Transformers model's config. However, if the `dtype` in the config is `float32`, we will use `float16` instead. quantization: The method used to quantize the model weights. Currently, we support "awq", "gptq", and "fp8" (experimental). If None, we first check the `quantization_config` attribute in the model config file. If that is None, we assume the model weights are not quantized and use `dtype` to determine the data type of the weights. revision: The specific model version to use. It can be a branch name, a tag name, or a commit id. tokenizer_revision: The specific tokenizer version to use. It can be a branch name, a tag name, or a commit id. chat_template: The chat template to apply. seed: The seed to initialize the random number generator for sampling. gpu_memory_utilization: The ratio (between 0 and 1) of GPU memory to reserve for the model weights, activations, and KV cache. Higher values will increase the KV cache size and thus improve the model's throughput. However, if the value is too high, it may cause out-of- memory (OOM) errors. kv_cache_memory_bytes: Size of KV Cache per GPU in bytes. By default, this is set to None and vllm can automatically infer the kv cache size based on gpu_memory_utilization. However, users may want to manually specify the kv cache memory size. kv_cache_memory_bytes allows more fine-grain control of how much memory gets used when compared with using gpu_memory_utilization. Note that kv_cache_memory_bytes (when not-None) ignores gpu_memory_utilization cpu_offload_gb: The size (GiB) of CPU memory to use for offloading the model weights. This virtually increases the GPU memory space you can use to hold the model weights, at the cost of CPU-GPU data transfer for every forward pass. offload_group_size: Prefetch offloading: Group every N layers together. Offload last `offload_num_in_group` layers of each group. Default is 0 (disabled). offload_num_in_group: Prefetch offloading: Number of layers to offload per group. Default is 1. offload_prefetch_step: Prefetch offloading: Number of layers to prefetch ahead. Higher values hide more latency but use more GPU memory. Default is 1. offload_params: Prefetch offloading: Set of parameter name segments to selectively offload. Only parameters whose names contain one of these segments will be offloaded (e.g., {"gate_up_proj", "down_proj"} for MLP weights, or {"w13_weight", "w2_weight"} for MoE expert weights). If None or empty, all parameters are offloaded. enforce_eager: Whether to enforce eager execution. If True, we will disable CUDA graph and always execute the model in eager mode. If False, we will use CUDA graph and eager execution in hybrid. enable_return_routed_experts: Whether to return routed experts. disable_custom_all_reduce: See [ParallelConfig][vllm.config.ParallelConfig]. hf_token: The token to use as HTTP bearer authorization for remote files . If `True`, will use the token generated when running `hf auth login` (stored in `~/.cache/huggingface/token`). hf_overrides: If a dictionary, contains arguments to be forwarded to the HuggingFace config. If a callable, it is called to update the HuggingFace config. mm_processor_kwargs: Arguments to be forwarded to the model's processor for multi-modal data, e.g., image processor. Overrides for the multi-modal processor obtained from `AutoProcessor.from_pretrained`. The available overrides depend on the model that is being run. For example, for Phi-3-Vision: `{"num_crops": 4}`. pooler_config: Initialize non-default pooling config for the pooling model, e.g., `PoolerConfig(seq_pooling_type="MEAN", use_activation=False)`. compilation_config: Either an integer or a dictionary. If it is an integer, it is used as the mode of compilation optimization. If it is a dictionary, it can specify the full compilation configuration. attention_config: Configuration for attention mechanisms. Can be a dictionary or an AttentionConfig instance. If a dictionary, it will be converted to an AttentionConfig. Allows specifying the attention backend and other attention-related settings. **kwargs: Arguments for [`EngineArgs`][vllm.EngineArgs]. Note: This class is intended to be used for offline inference. For online serving, use the [AsyncLLMEngine][vllm.AsyncLLMEngine] class instead. autoNFrg?)"runnerconvert tokenizertokenizer_modeskip_tokenizer_inittrust_remote_codeallowed_local_media_pathallowed_media_domainstensor_parallel_sizedtype quantizationrevisiontokenizer_revision chat_templateseedgpu_memory_utilizationcpu_offload_gboffload_group_sizeoffload_num_in_groupoffload_prefetch_stepoffload_params enforce_eagerenable_return_routed_expertsdisable_custom_all_reducehf_token hf_overridesmm_processor_kwargs pooler_configstructured_outputs_configprofiler_configattention_configkv_cache_memory_bytescompilation_configlogits_processorsmodelrXrYrZr[r\r]r^r_r`rarbrcrdrerfrgrhrirjrkrlrmrnrorprqrrrsrtrurvrwrxrykwargsreturnc"3 Ksd|$vr|$dddl}%|%jdtddd|$vrd|$d<d |$vr1|$d }&t|&tr1t|&|$d <d |$vrmt|$d trmdd l m }'|$d }(z |'d@i|(|$d <Wnt yl})zt d |(|)td |)|)d})~)ww|dursi}dtdttdtfdd}*t|"trtt|"d}+n|*|"t}+|*|t},|*|t}-|*| t}.t|$dd}/|$d}0|/dkr|0dkststd|/dtd@id|d|d|d|d|d|d |d!|d"| d#| d$| d%| d&| d'|d(|d)|d*|!d+|d,|d-|d.|d/|ptd0|d1|d2|d3|d4|d5|d6|d7|,d8|-d9|.d:|+d;|#|$}1t|1tj|1tj d<|_!t|j!|_"t#|_$d|_%|j!&}2t 'd=|2|2|_(|j!j)|_)|j!j*|_*t+||_,|j!j-|_-|j!j.|_.t/|j,d>|_0t1|2|j)|j*|j0d?|_2d|_3dSd0|d1|d2|d3|d4|d5|d6|d7|,d8|-d9|.d:|+d;|#|$}1t|1tj|1tj d<|_!t|j!|_"t#|_$d|_%|j!&}2t 'd=|2|2|_(|j!j)|_)|j!j*|_*t+||_,|j!j-|_-|j!j.|_.t/|j,d>|_0t1|2|j)|j*|j0d?|_2d|_3dS)AzLLM constructor. swap_spacerNz]The 'swap_space' parameter is deprecated and ignored. It will be removed in a future version.) stackleveldisable_log_statsT worker_clskv_transfer_config)KVTransferConfigz[Failed to convert 'kv_transfer_config' dict to KVTransferConfig object. Dict: %s. Error: %sz'Invalid 'kv_transfer_config' provided: valueclsr|cs<|durSt|trdifdd|DS|S)z0Convert dict/None/instance to a config instance.Ncs i|] \}}t|r||qS)r).0kvrra/lsinfo/ai/hellotax_ai/llm_service/venv_vllm/lib/python3.10/site-packages/vllm/entrypoints/llm.py 5s z6LLM.__init__.._make_config..r) isinstancedictitems)rrrrr _make_config0s   z"LLM.__init__.._make_config)modedata_parallel_sizerWdistributed_executor_backendexternal_launcherzLLM(data_parallel_size=z) is not supported for single-process usage and may hang. Please use the explicit multi-process data-parallel example at 'examples/offline_inference/data_parallel.py'.rzrXrYrZr[r\r]r^r_r`rarbrcrdrfrgrwrhrirjrkrlrmrnrorprqrrrsrtrurvrxry) engine_args usage_contextzSupported tasks: %s)re)supported_tasks model_configrendererchat_template_configr)4popwarningswarnDeprecationWarningrtype cloudpickledumpsrvllm.config.kv_transferrrloggererror ValueErrorrrSintrrrrrgetr;is_tpursetr,rLfrom_engine_argsrG LLM_CLASS llm_engine engine_classrHrequest_counterdefault_sampling_paramsget_supported_tasksinforrrr"re io_processorinput_processorr rr#pooling_io_processors _cached_repr)3selfrzrXrYrZr[r\r]r^r_r`rarbrcrdrerfrgrhrirjrkrlrmrnrorprqrrrsrtrurvrwrxryr{rrrraw_config_dictercompilation_config_instancestructured_outputs_instanceprofiler_config_instanceattention_config_instance_dp_size_distributed_executor_backendrrrrr__init__s,                 !"&          !"&         z LLM.__init__cC |jSN)r get_tokenizerrrrrrs zLLM.get_tokenizerT include_dpcCs|jjj}|r |jS|jS)aGet the world size from the parallel config. Args: include_dp: If True (default), returns the world size including data parallelism (TP * PP * DP). If False, returns the world size without data parallelism (TP * PP). Returns: The world size (tensor_parallel_size * pipeline_parallel_size), optionally multiplied by data_parallel_size if include_dp is True. )r vllm_configparallel_configworld_size_across_dp world_size)rrrrrrget_world_sizes zLLM.get_world_sizecCs|j|jdSr)rclear_mm_cacherreset_mm_cacherrrrrs zLLM.reset_mm_cachecCs4|jdur |j|_|jrtjdi|jStS)Nr)rrget_diff_sampling_paramrD from_optionalrrrrget_default_sampling_paramss  zLLM.get_default_sampling_params)use_tqdm lora_requestprioritytokenization_kwargspromptssampling_paramsr.rrrc Cs@|jj}|dkr td|dur|}|j||t||||dS)aGenerates the completions for the input prompts. This class automatically batches the given prompts, considering the memory constraint. For the best performance, put all of your prompts into a single list and pass it to this method. Args: prompts: The prompts to the LLM. You may pass a sequence of prompts for batch inference. See [PromptType][vllm.inputs.PromptType] for more details about the format of each prompt. sampling_params: The sampling parameters for text generation. If None, we use the default sampling parameters. When it is a single value, it is applied to every prompt. When it is a list, the list must have the same length as the prompts and it is paired one by one with the prompt. use_tqdm: If `True`, shows a tqdm progress bar. If a callable (e.g., `functools.partial(tqdm, leave=False)`), it is used to create the progress bar. If `False`, no progress bar is created. lora_request: LoRA request to use for generation, if any. priority: The priority of the requests, if any. Only applicable when priority scheduling policy is enabled. If provided, must be a list of integers matching the length of `prompts`, where each priority value corresponds to the prompt at the same index. tokenization_kwargs: Overrides for `tokenizer.encode`. Returns: A list of `RequestOutput` objects containing the generated completions in the same order as the input prompts. generatezLLM.generate() is only supported for generative models. Try passing `--runner generate` to use the model as a generative model.N)rparams output_typerrrr)r runner_typerr_run_completionr9)rrrrrrrrrrrrs )z LLM.generatecCs>|jj}|dkr td|dur|}|j||||||dS)aEnqueue prompts for generation without waiting for completion. This method adds requests to the engine queue but does not start processing them. Use wait_for_completion() to process the queued requests and get results. Args: prompts: The prompts to the LLM. See generate() for details. sampling_params: The sampling parameters for text generation. lora_request: LoRA request to use for generation, if any. priority: The priority of the requests, if any. use_tqdm: If True, shows a tqdm progress bar while adding requests. tokenization_kwargs: Overrides for `tokenizer.encode`. Returns: A list of request IDs for the enqueued requests. rz6LLM.enqueue() is only supported for generative models.Nrrrrrr)rrrr_add_completion_requests)rrrrrrrrrrrenqueuesz LLM.enqueuercCdSrr)rrrrrwait_for_completion szLLM.wait_for_completionrcCrrrrrrrrrr'scCs|durttf}|j||dS)aWait for all enqueued requests to complete and return results. This method processes all requests currently in the engine queue and returns their outputs. Use after enqueue() to get results. Args: output_type: The expected output type, defaults to RequestOutput. use_tqdm: If True, shows a tqdm progress bar. Returns: A list of output objects for all completed requests. Nr)r9r8 _run_enginerrrrr/spromptc Cs|ddkr|S|jjj}|durdn|j}|s|S|d}t||}|s-|St|dkr;t d||S| }||}t | |d} |rZ|j | krXt d|St|| |S)Nr multimodalmm_placeholdersrWzMultiple modality specific loras were registered and would be used by a single prompt consuming several modalities; currently we only support one lora per request; as such, lora(s) registered with modalities: %s will be skippedzA modality with a registered lora and a lora_request with a different ID were provided; falling back to the lora_request as we only apply one LoRARequest per prompt)rr lora_configdefault_mm_loraskeysr intersectionlenrwarningrsortedindex lora_int_idr4) rrrrrprompt_modalitiesr modality_namemodality_lora_pathmodality_lora_idrrr_resolve_mm_loraFs<     zLLM._resolve_mm_lorarmethodtimeoutargscCs|j||||S)a Execute an RPC call on all workers. Args: method: Name of the worker method to execute, or a callable that is serialized and sent to all workers to execute. If the method is a callable, it should accept an additional `self` argument, in addition to the arguments passed in `args` and `kwargs`. The `self` argument will be the worker object. timeout: Maximum time in seconds to wait for execution. Raises a [`TimeoutError`][] on timeout. `None` means wait indefinitely. args: Positional arguments to pass to the worker method. kwargs: Keyword arguments to pass to the worker method. Returns: A list containing the results from each worker. Note: It is recommended to use this API to only pass control messages, and set up data-plane communication to pass data. )rcollective_rpc)rrrrr{rrrrzszLLM.collective_rpcfunccCs |j|S)a Run a function directly on the model inside each worker, returning the result for each of them. !!! warning To reduce the overhead of data transfer, avoid returning large arrays or tensors from this method. If you must return them, make sure you move them to CPU first to avoid taking up additional VRAM! )r apply_model)rrrrrrs zLLM.apply_modelrconcurrency_limitc, Cs |j}|j}|j}|j} |j} |j} | j} t| | } | |}| |t |}|r6|dur6t dd}|dur>t |}td|d|dd}g}t||D]\}}|dd kr]td |t||dd qOtd t ||D]}||||}t|}|rt|d ddd}t d|D]}ttdd|Dg}d gttdd|D}tt|dd|dd}t |d krn|jdd|D||t |tdd|Ddd}t||D]m\\}}}g}t||D]Q} || }!|| }"|"jd jdur@|"jd jd }#|#D]1\}$}%t|!j |!j!|$g|!j|#g|!j"|!j#|%j$d}&|$| kr9| s9|j%|&q||&qqt&|| dd}'|'d||_'qqqpg}(|D]-}|j%(|j't&|j%| dd})|)d|}*|*D] }+| )|+j!|+_*qo|(t+|*dqV|(S)a Generate sequences using beam search. Args: prompts: A list of prompts. Each prompt can be a string or a list of token IDs. params: The beam search parameters. lora_request: LoRA request to use for generation, if any. use_tqdm: Whether to use tqdm to display the progress bar. concurrency_limit: The maximum number of concurrent requests. If None, the number of concurrent requests is unlimited. NzSProgress bar is not supported when using concurrency_limit. Disabling progress bar.Fr~rWT)logprobs max_tokens temperature skip_clonerembedsz.Embedding prompt not supported for beam search)rrrz Beam searchtoken)descunit unit_scalezThe progress bar shows the upper bound on token steps and may finish early due to stopping conditions. It does not reflect instance-level progress.css|]}|jVqdSr)beamsrinstancerrr sz"LLM.beam_search..cs|]}t|jVqdSr)rrrrrrr  css|]}|VqdSr) get_promptrbeamrrrr scSsg|]}|jqSr)rr rrr sz#LLM.beam_search..rrr lora_requestsr)tokensrr cum_logprob)keyreverse) sequences), beam_widthrr ignore_eoslength_penaltyrr eos_token_idr_preprocess_cmpl_lora_request_to_seqrrrrDzipNotImplementedErrorappendr ranger listsum itertools accumulate_render_and_run_requests_params_to_seqr9outputsrrr orig_promptrrrlogprob completedrrextenddecodetextr ),rrrrrrrrrrrrZrsort_beams_keyengine_promptsrr instanceslora_reqr prompt_startinstances_batch token_iter_ all_beamsposinstance_start_and_endoutputstartendrinstance_new_beamsi current_beamresultrtoken_id logprob_objnew_beam sorted_beamsr%sorted_completed best_beamsr rrr beam_searchs            ! zLLM.beam_searchcs@|j}|jfdd|D}|jjdi|pi}|||S)a@ Convert prompt inputs from LLM APIs (other than [LLM.chat][]) into a format that can be passed to `_add_request`. Refer to [LLM.generate][] for a complete description of the arguments. Returns: A list of `ProcessorInputs` objects ready to be passed into LLMEngine. csg|]}t|qSr)r@rrrrrr Us z(LLM._preprocess_cmpl..Nr)rrdefault_cmpl_tok_params with_kwargs render_cmpl)rrrrparsed_prompts tok_paramsrrFrrDs   zLLM._preprocess_cmplcCs||g|\}|Sr)r)rrr engine_promptrrr_preprocess_cmpl_one^szLLM._preprocess_cmpl_one conversationschat_template_content_formatchat_template_kwargsadd_generation_promptcontinue_final_messagetoolsc  Cs`|j} t||t|t|||t| jdd} | jjdi|pi} | j|| | d| id\} }|S)a2 Convert a list of conversations into prompts so that they can then be used as input for other LLM APIs. Refer to [LLM.chat][] for a complete description of the arguments. Returns: A list of `ProcessorInputs` objects ready to be passed into LLMEngine. )rQrRrStokenize)rerOrPrr) prompt_extrasNr) rr=r>rrIrZdefault_chat_tok_paramsrH render_chat)rrNrerOrPrQrRrSrrrr chat_paramsrKr3r-rrr_preprocess_chatfs0  zLLM._preprocess_chat conversationc Cs$|j|g|||||||| d \} | S)NrerOrPrQrRrSrrr)rY) rrZrerOrPrQrRrSrrrrLrrr_preprocess_chat_ones  zLLM._preprocess_chat_onemessagesc CsP|j} | j}|dkrtd|dur|}|j||t||||| ||| | | d S)a Generate responses for a chat conversation. The chat conversation is converted into a text prompt using the tokenizer and calls the [generate][vllm.LLM.generate] method to generate the responses. Multi-modal inputs can be passed in the same way you would pass them to the OpenAI API. Args: messages: A sequence of conversations or a single conversation. - Each conversation is represented as a list of messages. - Each message is a dictionary with 'role' and 'content' keys. sampling_params: The sampling parameters for text generation. If None, we use the default sampling parameters. When it is a single value, it is applied to every prompt. When it is a list, the list must have the same length as the prompts and it is paired one by one with the prompt. use_tqdm: If `True`, shows a tqdm progress bar. If a callable (e.g., `functools.partial(tqdm, leave=False)`), it is used to create the progress bar. If `False`, no progress bar is created. lora_request: LoRA request to use for generation, if any. chat_template: The template to use for structuring the chat. If not provided, the model's default chat template will be used. chat_template_content_format: The format to render message content. - "string" will render the content as a string. Example: `"Who are you?"` - "openai" will render the content as a list of dictionaries, similar to OpenAI schema. Example: `[{"type": "text", "text": "Who are you?"}]` add_generation_prompt: If True, adds a generation template to each message. continue_final_message: If True, continues the final message in the conversation instead of starting a new one. Cannot be `True` if `add_generation_prompt` is also `True`. chat_template_kwargs: Additional kwargs to pass to the chat template. tokenization_kwargs: Overrides for `tokenizer.encode`. mm_processor_kwargs: Overrides for `processor.__call__`. Returns: A list of `RequestOutput` objects containing the generated responses in the same order as the input messages. rz{LLM.chat() is only supported for generative models. Try passing `--runner generate` to use the model as a generative model.N) r]rrrrrerOrPrQrRrSrrr)rrrr _run_chatr9)rr]rrrrerOrQrRrSrPrrrrrrrrchats.BzLLM.chat)rr pooling_taskrpooling_paramsr`cs|durtdj}|j}|dkrtdt|trd|vrjdur(td|d} | dur5tdj| } jj| d}t |} fd d  |t | D} | D] } | j durad | _ qWj | | t|||d }jdustJj|}ttd |t|ddgddgS|durt}t |} |t | } | D]}|j dur||_ q|j |krd|j d|d}t|q|jvrj|}|| |}|t | }dt |}j|| ||dj|td}||}|Sj | | t|||d }|S)aApply pooling to the hidden states corresponding to the input prompts. This class automatically batches the given prompts, considering the memory constraint. For the best performance, put all of your prompts into a single list and pass it to this method. Args: prompts: The prompts to the LLM. You may pass a sequence of prompts for batch inference. See [PromptType][vllm.inputs.PromptType] for more details about the format of each prompt. pooling_params: The pooling parameters for pooling. If None, we use the default pooling parameters. use_tqdm: If `True`, shows a tqdm progress bar. If a callable (e.g., `functools.partial(tqdm, leave=False)`), it is used to create the progress bar. If `False`, no progress bar is created. lora_request: LoRA request to use for generation, if any. pooling_task: Override the pooling task to use. tokenization_kwargs: Overrides for `tokenizer.encode`. Returns: A list of `PoolingRequestOutput` objects containing the pooled hidden states in the same order as the input prompts. Napooling_task required for `LLM.encode` Please use one of the more specific methods or set the pooling_task when using `LLM.encode`: - For embeddings, use `LLM.embed(...)` or `pooling_task="embed"`. - For classification logits, use `LLM.classify(...)` or `pooling_task="classify"`. - For similarity scores, use `LLM.score(...)`. - For rewards, use `LLM.reward(...)` or `pooling_task="token_classify"` - For token classification, use `pooling_task="token_classify"` - For multi-vector retrieval, use `pooling_task="token_embed"`poolingzvLLM.encode() is only supported for pooling models. Try passing `--runner pooling` to use the model as a pooling model.datazNo IOProcessor plugin installed. Please refer to the documentation and to the 'prithvi_geospatial_mae_io_processor' offline inference example for more details.zThe 'data' field of the prompt is expected to contain the prompt data and it cannot be None. Refer to the documentation of the IOProcessor in use for more details.)rcsg|]}j|qSr)rmerge_pooling_params)rparamrrrr es zLLM.encode..plugin)rrrrrrrVnum_cached_tokensrT) request_idr%rgprompt_token_idsfinishedz You cannot overwrite param.task=z with pooling_task=!rrr prioritiesrr)rrrrrrr parse_data pre_processrAr$rtaskrr8 post_processrgetattrr<rpre_process_offliner_priority_to_seq_render_and_add_requestsrpost_process_offline)rrrarrr`rrr prompt_datavalidated_prompt prompts_seq params_seqpr%processed_outputsremsgrprocessor_inputsseq_lora_requests seq_priorityrrrencodes$             z LLM.encoderrarrcCs6d|jvr td|j||||d|d}dd|DS)a Generate an embedding vector for each prompt. This class automatically batches the given prompts, considering the memory constraint. For the best performance, put all of your prompts into a single list and pass it to this method. Args: prompts: The prompts to the LLM. You may pass a sequence of prompts for batch inference. See [PromptType][vllm.inputs.PromptType] for more details about the format of each prompt. pooling_params: The pooling parameters for pooling. If None, we use the default pooling parameters. use_tqdm: If `True`, shows a tqdm progress bar. If a callable (e.g., `functools.partial(tqdm, leave=False)`), it is used to create the progress bar. If `False`, no progress bar is created. lora_request: LoRA request to use for generation, if any. tokenization_kwargs: Overrides for `tokenizer.encode`. Returns: A list of `EmbeddingRequestOutput` objects containing the embedding vectors in the same order as the input prompts. embedz_Embedding API is not supported by this model. Try converting the model using `--convert embed`.rrarr`rcSg|]}t|qSr)r7 from_baseritemrrrr zLLM.embed..rrr)rrrrarrrrrrr ! z LLM.embed)rarrrcCs6d|jvr td|j||||d|d}dd|DS)a Generate class logits for each prompt. This class automatically batches the given prompts, considering the memory constraint. For the best performance, put all of your prompts into a single list and pass it to this method. Args: prompts: The prompts to the LLM. You may pass a sequence of prompts for batch inference. See [PromptType][vllm.inputs.PromptType] for more details about the format of each prompt. pooling_params: The pooling parameters for pooling. If None, we use the default pooling parameters. use_tqdm: If `True`, shows a tqdm progress bar. If a callable (e.g., `functools.partial(tqdm, leave=False)`), it is used to create the progress bar. If `False`, no progress bar is created. lora_request: LoRA request to use for generation, if any. tokenization_kwargs: Overrides for `tokenizer.encode`. Returns: A list of `ClassificationRequestOutput` objects containing the embedding vectors in the same order as the input prompts. classifyzgClassification API is not supported by this model. Try converting the model using `--convert classify`.rcSrr)r6rrrrrr rz LLM.classify..r)rrrarrrrrrrrrz LLM.classifycCs|j||||d|dS)a Generate rewards for each prompt. Args: prompts: The prompts to the LLM. You may pass a sequence of prompts for batch inference. See [PromptType][vllm.inputs.PromptType] for more details about the format of each prompt. pooling_params: The pooling parameters for pooling. If None, we use the default pooling parameters. use_tqdm: If `True`, shows a tqdm progress bar. If a callable (e.g., `functools.partial(tqdm, leave=False)`), it is used to create the progress bar. If `False`, no progress bar is created. lora_request: LoRA request to use for generation, if any. tokenization_kwargs: Overrides for `tokenizer.encode`. Returns: A list of `PoolingRequestOutput` objects containing the pooled hidden states in the same order as the input prompts. token_classifyrrrar`r)r)rrrarrrrrrrewardsz LLM.rewarddata_1data_2cCs|}g}||D]} t| tstd|| q |j||||d|d} | dt|} | t|d} t| dkrB| t| } t|| | d} dd| DS) Nz;Embedding scores currently do not support multimodal input.rrrrW)rZembed_1embed_2cSrrr:rrrrrr mrz(LLM._embedding_score..)rrstrrrrrr&)rrrrrarrrZ input_textsr+encoded_outputencoded_output_1encoded_output_2scoresrrr_embedding_scoreDs4       zLLM._embedding_scorec Cs ddlm}|}|j} t|d| } t|d| } |j| | |||d|d} | dt| } | t| d}t| dkrB| t|} g}g}|j}durP|g}t| |D]3\}}|j j }|j j }t ||}|j ||j }| t|jd |j||d ||j|jd d qUd d|DS)z Late interaction scoring (ColBERT MaxSim). Encodes queries and documents into per-token embeddings, then computes MaxSim: sum over query tokens of max similarity to any document token. r) PoolingOutputquerydocument token_embedrNrWr3)rcT)rhr%rirgrjcSrrrrrrrr rz/LLM._late_interaction_score..) vllm.outputsrrrr*rr pad_token_idrr%rcr(rirr8rhrg)rrrrrarrrrZr prompts_1 prompts_2rrrrpaddingremb_1emb_2q_embd_emb maxsim_scorerrrr_late_interaction_scoreosH         zLLM._late_interaction_scorescore_templatec Cs|j}|} t| rtdt|dkr|t|}|dur%tdd}n|jdur-d|_tt} tt} ddt ||D} | D]5\} }t || || ||d\}}| dd}rn| }t |}d |i|_| |n| || |qC|j| | t||d }d d|DS) Nz0Score API is not supported for Mistral tokenizerrWscore)rqcSsg|]\}}||fqSrr)rt1t2rrrr sz-LLM._cross_encoding_score..)rrrrZrrtoken_type_idscompressed_token_type_ids)rrrrrcSrrrrrrrr r)rrrIrrr<rqrr/rr)rcloner' extra_kwargsrrr8)rrrrrarrrrrZpooling_params_listr input_pairsqdr3rLrr compressedr%rrr_cross_encoding_scoresJ              zLLM._cross_encoding_score)rrarrrec s|j}|j} | dkrtd|j|jj} | dk} | dk} | s.tfdddDr.td| r=t|jd d d kr=td | sG|d urGtd|j} |j }t ||| |d\}}|j }|j j di|pbi}|}| rw|j|||||||dS| r|j||||||dS|j||||||dS)a]Generate similarity scores for all pairs `` or ``. The inputs can be `1 -> 1`, `1 -> N` or `N -> N`. In the `1 - N` case the `data_1` input will be replicated `N` times to pair with the `data_2` inputs. The input pairs are used to build a list of prompts for the cross encoder model. This class automatically batches the prompts, considering the memory constraint. For the best performance, put all of your inputs into a single list and pass it to this method. Supports both text and multi-modal data (images, etc.) when used with appropriate multi-modal models. For multi-modal inputs, ensure the prompt structure matches the model's expected input format. Args: data_1: Can be a single prompt, a list of prompts or `ScoreMultiModalParam`, which can contain either text or multi-modal data. When a list, it must have the same length as the `data_2` list. data_2: The data to pair with the query to form the input to the LLM. Can be text or multi-modal data. See [PromptType] [vllm.inputs.PromptType] for more details about the format of each prompt. pooling_params: The pooling parameters for pooling. If None, we use the default pooling parameters. use_tqdm: If `True`, shows a tqdm progress bar. If a callable (e.g., `functools.partial(tqdm, leave=False)`), it is used to create the progress bar. If `False`, no progress bar is created. lora_request: LoRA request to use for generation, if any. chat_template: The chat template to use for the scoring. If None, we use the model's default chat template. tokenization_kwargs: Overrides for `tokenizer.encode`. Returns: A list of `ScoringRequestOutput` objects containing the generated scores in the same order as the input prompts. rbzuLLM.score() is only supported for pooling models. Try passing `--runner pooling` to use the model as a pooling model.zlate-interactionz cross-encoderc3s|]}|vVqdSrr)rtrrrr8s zLLM.score..)rrzsScore API is not supported by this model. Try converting the model using `--convert embed` or `--convert classify`. num_labelsrrWz.Score API is only enabled for num_labels == 1.Nz9chat_template is only supported for cross-encoder models.)is_multimodal_model architecture)rrarrrrr)rrrr score_typeallrs hf_configrrr+rrGrHget_encode_kwargsrrr)rrrrrarrrerrris_late_interactionis_cross_encoderrr score_data_1 score_data_2rrK encode_kwargsrrrrs|8     z LLM.scoreprofile_prefixcC|j|dS)a<Start profiling with optional custom trace prefix. Args: profile_prefix: Optional prefix for the trace file names. If provided, trace files will be named as "_dp_pp_tp". If not provided, default naming will be used. N)r start_profile)rrrrrrvszLLM.start_profilecCs|jdSr)r stop_profilerrrrrszLLM.stop_profilereset_running_requestsreset_connectorcCs|j||Sr)rreset_prefix_cache)rrrrrrrszLLM.reset_prefix_cacheabortlevelrcCs|jj||ddS)a Put the engine to sleep. The engine should not process any requests. The caller should guarantee that no requests are being processed during the sleep period, before `wake_up` is called. Args: level: The sleep level. - Level 0: Pause scheduling but continue accepting requests. Requests are queued but not processed. - Level 1: Offload model weights to CPU, discard KV cache. The content of kv cache is forgotten. Good for sleeping and waking up the engine to run the same model again. Please make sure there's enough CPU memory to store the model weights. - Level 2: Discard all GPU memory (weights + KV cache). Good for sleeping and waking up the engine to run a different model or update the model, where previous model weights are not needed. It reduces CPU memory pressure. mode: How to handle any existing requests, can be "abort", "wait", or "keep". )rrN)rsleep)rrrrrrrsz LLM.sleeptagscCr)a( Wake up the engine from sleep mode. See the [sleep][vllm.LLM.sleep] method for more details. Args: tags: An optional list of tags to reallocate the engine memory for specific memory allocations. Values must be in `("weights", "kv_cache", "scheduling")`. If None, all memory is reallocated. wake_up should be called with all tags (or None) before the engine is used again. Use tags=["scheduling"] to resume from level 0 sleep. N)rwake_up)rrrrrrs z LLM.wake_uprNcCr)aReturn a snapshot of aggregated metrics from Prometheus. Returns: A `MetricSnapshot` instance capturing the current state of all aggregated metrics from Prometheus. Note: This method is only available with the V1 LLM engine. )r get_metricsrrrrrs zLLM.get_metrics num_requestscCs>t|trt||krtd|dt|d|S|g|S)NThe lengths of prompts (z) and params () must be the same.rrrr)rrrrrrr$   zLLM._params_to_seqcCs>t|trt||krtd|dt|d|S|g|S)Nrz) and lora_request (rr)rrrrrrrrzLLM._lora_request_to_seqcCs<|durt||krtd|dt|d|Sdg|S)Nrz) and priority (rr)rr)rrrrrrrus  zLLM._priority_to_seqc sdt|}|t|}|t|} |t|} jfddt||ddD|| | dS)Nc3s|] }|VqdSr)rMrErrrrrs  z/LLM._add_completion_requests..zRendering promptsrrrl)rAr$rrrurvrJ) rrrrrrr seq_prompts seq_paramsrrrrrrs   zLLM._add_completion_requestscCs$|j||||||d|j||dS)Nrrn)rr)rrrrrrrrrrrr s zLLM._run_completion) rrrerOrQrRrSrPrrrc  sdt|}|t|}|t|}jf ddt||ddD||||dS)Nc 3s,|]}j|d VqdS)r[N)r\)rrZ rQrerOrPrRrrrrrSrrr=s  z LLM._run_chat..zRendering conversationsrr)r?r$rrr#rJ)rr]rrrrrerOrQrRrSrPrrr seq_convsrrrrrr^$s  z LLM._run_chat)rrmrrrmcCs8t|ttfr td|j||||d|j||dS)Na>Rendering all prompts before adding them to the engine is less efficient than performing both on the same prompt before processing the next prompt. You should instead pass a generator that renders one prompt per iteration, as that allows engine execution to begin for the first prompt while processing the next prompt.rlr)rrtupler warning_oncervr)rrrrrrmrrrrr#Us  zLLM._render_and_run_requests)rrmc Csg}z0t|D](\}}|j||||||durdn|||dur$dn||d}||qW|StyJ} z |rD|jj|dd| d} ~ ww)NrrrT)internal) enumerate _add_requestrr Exceptionr abort_request) rrrrrmadded_request_idsr;rrhrrrrrvrs* zLLM._render_and_add_requestscCs6t|tr tj|_tt|j}|jj |||||dS)Nr) rrDrC FINAL_ONLY output_kindrnextrr add_request)rrrrrrhrrrrs zLLM._add_requestc CsZ|r |j}t|r |nt}||dddddddddd}g}d}d}|jr|j} | D]g} t| |s;J| jr|| |rt| t rt | j } | j dusVJ|t | j | 7}||j d } |td d | j D7}||j d } d| dd| dd|_|| n|d |j|kr|q2|js+|r|t|d ddS)NzProcessed promptsTzest. speed input: rz.2fz toks/s, output: z toks/s)totalr dynamic_ncolspostfixelapsedcsrr)r token_ids)rstprrrrrz"LLM._run_engine..rWcSs t|jSr)rrh)xrrrs z!LLM._run_engine..)r)rget_num_unfinished_requestscallabler has_unfinished_requestssteprrjrr9rr%ri format_dictr rupdatenrefreshcloser)rrrr tqdm_funcpbarr% total_in_tokstotal_out_toks step_outputsr7rin_spdout_spdrrrrsR             zLLM._run_enginerequestcC0t|tr |dn|j}|jjdd|iddS)z Initialize weight transfer for RL training. Args: request: Weight transfer initialization request with backend-specific info init_infoinit_weight_transfer_enginer{N)rrrrr)rrinit_info_dictrrrrs   zLLM.init_weight_transfer_enginecCr)z Update the weights of the model. Args: request: Weight update request with backend-specific update info update_infoupdate_weightsr N)rrr rr)rrupdate_info_dictrrrr s  zLLM.update_weightscCsB|jdur|jd}|r|d|_|jSd|jjd|_|jS)z;Return a transformers-style hierarchical view of the model.Nget_model_inspectionrz LLM(model=))rrrrrz)rresultsrrr__repr__s   z LLM.__repr__)T)r|Nr)NNNTN)NrN)NFN)NrUNTFNNN) NTNNrUTFNNNN)FF)rWr)Nr)j__name__ __module__ __qualname____doc__rrrrboolrrrr5rfloatrrrrrrrrrrrMrrFrrrrDrr/rrr r4r9rrr r8rrOrr.rrSrnnModulerr2r1rBr rDrrMrr!rYr\r_r-r<rErr7rr6rrr$r:rrrr0r%rrrrrKrrrrRr$rrurrr^rr#rvrrrrrr rrrrrrTosFm         #$%&'() A     A   *      7 "   !       4          a /  6  7  '  +  D   I                '              7   "   !  7 rT){r!collections.abcrrrpathlibrtypingrrrtorch.nnrpydanticr tqdm.autor typing_extensionsr r vllm.beam_searchr r rr vllm.configrrrrrrvllm.config.compilationrvllm.config.modelrrrrr%vllm.distributed.weight_transfer.baserrvllm.engine.arg_utilsrvllm.entrypoints.chat_utilsrr r!r"/vllm.entrypoints.pooling.io_processor_factoriesr#$vllm.entrypoints.pooling.score.utilsr$r%r&r'r(r)r*r+vllm.entrypoints.utilsr,vllm.inputs.datar-r.r/r0r1r2 vllm.loggerr3vllm.lora.requestr4'vllm.model_executor.layers.quantizationr5rr6r7r8r9r:vllm.platformsr;vllm.pooling_paramsr<vllm.renderersr=r> vllm.renderers.inputs.preprocessr?r@rAvllm.sampling_paramsrBrCrD vllm.tasksrEvllm.tokenizersrFvllm.usage.usage_librGvllm.utils.counterrHvllm.utils.mistralrIvllm.utils.tqdm_utilsrJvllm.v1.enginerKvllm.v1.engine.llm_enginerLvllm.v1.sample.logits_processorrMvllm.v1.metrics.readerrNrrrOrRrSrTrrrrsb        (