o :/iT1@shddlmZddlmZddlmZddlmZmZddl Z ddl Z ddl m Z ddlmZddlmZmZdd lmZdd lmZeeZeGd d d ZeGd ddZGdddZeddddgddZe dedZGdddeeZeGdddZGdddeeZ eGdddZ!Gdddee!Z"eGd d!d!Z#Gd"d#d#ee#Z$dS)$)MutableSequence)Sequence) dataclass)AnyGenericN)TypeVar) init_logger)PromptLogprobsSampleLogprobs) LoRARequest)RequestStateStatsc@seZdZUdZeed<eed<eeed<edBed<e dBed<dZ e j dBed<dZ edBed <dZeeBdBed <dZedBed <d efd dZd efddZdS)CompletionOutputa!The output data of one completion output of a request. Args: index: The index of the output in the request. text: The generated output text. token_ids: The token IDs of the generated output text. cumulative_logprob: The cumulative log probability of the generated output text. logprobs: The log probabilities of the top probability words at each position if the logprobs are requested. finish_reason: The reason why the sequence is finished. stop_reason: The stop string or token id that caused the completion to stop, None if the completion finished for some other reason including encountering the EOS token. lora_request: The LoRA request that was used to generate the output. indextext token_idsNcumulative_logproblogprobsrouted_experts finish_reason stop_reason lora_requestreturncCs |jduSN)rselfrY/lsinfo/ai/hellotax_ai/llm_service/venv_vllm/lib/python3.10/site-packages/vllm/outputs.pyfinished2s zCompletionOutput.finishedcCsFd|jd|jd|jd|jd|jd|jd|jd|jd S) NzCompletionOutput(index=z, text=z , token_ids=z, routed_experts=z, cumulative_logprob=z , logprobs=z, finish_reason=z, stop_reason=))rrrrrrrrrrrr__repr__5s zCompletionOutput.__repr__)__name__ __module__ __qualname____doc__int__annotations__strGenericSequencefloatr rnpndarrayrrrr boolrrrrrrr s    r c@s<eZdZUdZejed<defddZde de fddZ d S) PoolingOutputznThe output data of one pooling output of a request. Args: data: The extracted hidden states. datarcCd|jdS)NzPoolingOutput(data=r)r-rrrrrLzPoolingOutput.__repr__othercCs t||jot|j|jkSr) isinstance __class__r+r-all)rr0rrr__eq__OszPoolingOutput.__eq__N) r r!r"r#torchTensorr%r&robjectr+r4rrrrr,Bs  r,c@seZdZdZ     ddddededBdeedBdedBdeed e d e dBd e dBd edBd eedBdedBde ee fdBde ddfddZddde ddfddZdefddZdS) RequestOutputakThe output data of a completion request to the LLM. Args: request_id: The unique ID of the request. prompt: The prompt string of the request. For encoder/decoder models, this is the decoder input prompt. prompt_token_ids: The token IDs of the prompt. For encoder/decoder models, this is the decoder input prompt token ids. prompt_logprobs: The log probabilities to return per prompt token. outputs: The output sequences of the request. finished: Whether the whole request is finished. metrics: Metrics associated with the request. lora_request: The LoRA request that was used to generate the output. encoder_prompt: The encoder prompt string of the request. None if decoder-only. encoder_prompt_token_ids: The token IDs of the encoder prompt. None if decoder-only. num_cached_tokens: The number of tokens with prefix cache hit. kv_transfer_params: The params for remote K/V transfer. N)kv_transfer_params request_idpromptprompt_token_idsprompt_logprobsoutputsrmetricsrencoder_promptencoder_prompt_token_idsnum_cached_tokensr9kwargsrc Ks`| r tdt| ||_||_||_||_||_||_||_ ||_ | |_ | |_ | |_ | |_dS)Nz+RequestOutput: Ignoring extra arguments: %s)logger warning_oncer&r:r;r<r=r>rr?rr@rArBr9)rr:r;r<r=r>rr?rr@rArBr9rCrrr__init__ms  zRequestOutput.__init__ next_output aggregatecCs|j|jO_|j|_|jD]Z}t|jD]L\}}|j|jkrb|r[|j|j7_t|jts6t |j|_|j |j|j rN|j dusGJ|j |j |j |_ |j |_ |j|_n||j|<nq|j|qdS)z,Merge subsequent RequestOutput into this oneN)rr9r> enumeraterrr1rrlistextendrrrrappend)rrGrHnext_completioni completionrrradds.       zRequestOutput.addcCs^d|jd|jd|jd|jd|jd|jd|jd|jd |jd |j d |j d S) NzRequestOutput(request_id=z , prompt=, prompt_token_ids=z, encoder_prompt=z, encoder_prompt_token_ids=z, prompt_logprobs= , outputs= , finished=z , metrics=z, lora_request=, num_cached_tokens=r) r:r;r<r@rAr=r>rr?rrBrrrrrs,   zRequestOutput.__repr__)NNNNN)r r!r"r#r&rJr$r r r+r r dictrrFrPrrrrrr8UsP      $r8T)r:r;r<r=r>r_O)defaultc @s:eZdZdZdededeededef ddZ d d Z d S) PoolingRequestOutputa The output data of a pooling request to the LLM. Args: request_id (str): A unique identifier for the pooling request. outputs (PoolingOutput): The pooling results for the given input. prompt_token_ids (list[int]): A list of token IDs used in the prompt. num_cached_tokens: The number of tokens with prefix cache hit. finished (bool): A flag indicating whether the pooling is completed. r:r>r<rBrcCs"||_||_||_||_||_dSr)r:r<rBrr>)rr:r>r<rBrrrrrFs  zPoolingRequestOutput.__init__c Cs8t|jd|jd|jd|jd|jd|jd S)Nz (request_id=rRrQrTrSr)typer r:r>r<rBrrrrrrszPoolingRequestOutput.__repr__N) r r!r"r#r&rWrJr$r+rFrrrrrrYs  rYc@PeZdZUdZeeed<edefddZ e de fddZ de fd d Zd S) EmbeddingOutputzThe output data of one embedding output of a request. Args: embedding: The embedding vector, which is a list of floats. Its length depends on the hidden dimension of the model. embeddingpooling_outputcC$|j}|jdkr tdt|S)Nz,pooled_data should be a 1-D embedding vector)r-ndim ValueErrorr\tolistr^ pooled_datarrr from_bases  zEmbeddingOutput.from_basercC t|jSr)lenr]rrrr hidden_size zEmbeddingOutput.hidden_sizecCr.)NzEmbeddingOutput(hidden_size=r)rirrrrrr/zEmbeddingOutput.__repr__N)r r!r"r#rJr(r% staticmethodr,rfpropertyr$rir&rrrrrr\s  r\c@eZdZedefddZdS)EmbeddingRequestOutputrequest_outputcC"t|jt|j|j|j|jdSN)r:r>r<rBr)rnr:r\rfr>r<rBrrorrrrf  z EmbeddingRequestOutput.from_baseNr r!r"rkrYrfrrrrrn rnc@r[) ClassificationOutputzThe output data of one classification output of a request. Args: probs: The probability vector, which is a list of floats. Its length depends on the number of classes. probsr^cCr_)Nr`z.pooled_data should be a 1-D probability vector)r-rarbrvrcrdrrrrf"s  zClassificationOutput.from_basercCrgr)rhrwrrrr num_classes+rjz ClassificationOutput.num_classescCr.)Nz!ClassificationOutput(num_classes=r)rxrrrrr/r/zClassificationOutput.__repr__N)r r!r"r#rJr(r%rkr,rfrlr$rxr&rrrrrrvs  rvc@rm)ClassificationRequestOutputrocCrprq)ryr:rvrfr>r<rBrrrrrrrf4rsz%ClassificationRequestOutput.from_baseNrtrrrrry3ruryc@s:eZdZUdZeed<edefddZde fddZ d S) ScoringOutputzThe output data of one scoring output of a request. Args: score: The similarity score, which is a scalar value. scorer^cCs(|j}|jdkrtdt|S)Nrz$pooled_data should be a scalar score)r-squeezerarbrzitemrdrrrrfIs   zScoringOutput.from_basercCr.)NzScoringOutput(score=r)r{rrrrrTr/zScoringOutput.__repr__N) r r!r"r#r(r%rkr,rfr&rrrrrrz?s  rzc@rm)ScoringRequestOutputrocCrprq)r~r:rzrfr>r<rBrrrrrrrfYrszScoringRequestOutput.from_baseNrtrrrrr~Xrur~)%collections.abcrrr' dataclassesrtypingrrnumpyr)r5typing_extensionsr vllm.loggerr vllm.logprobsr r vllm.lora.requestr vllm.v1.metrics.statsr r rDr r,r8STREAM_FINISHEDrWrYr\rnrvryrzr~rrrrsH       ,k $