o :/i8&@sPddlZddlmZddlmZmZddlmZddlm Z m Z ddl Z ddl mZddlmZddlmZddlmZd d lmZd d lmZmZmZmZmZd d lmZmZm Z m!Z!e rcddl"Z#ned e$d Z#ddde j%de&de'de'fddZ(ddde j%de&de'de'fddZ)ddddejde'de'de'fddZ*ddddejde'de'de'fdd Z+d!dd"ej,de'de'fd#d$Z-d!dd"ej,de'de'fd%d&Z.d'ede/e0e'e&ffd(d)Z1d*efd+d,Z2dd-d.d/eed0e#j3j4d1e5fd2d3Z6dd-d.d/eed0e#j3j4d1e5dee0e&effd4d5Z7dd-d.d6e/e0e'efd0e#j3j4d1e5dee0e'e&efddffd7d8Z8ed9dd-d.d6e/e0e'efd0e#j3j4d1e5dee0e'e&efddffd:d;Z9 dHdd?Z< dHd@e'dAe:e'e fdBdejfdBdCZ= dHdDe'dEe:e'e fdBde0ej,e:e'e fffdFdGZ>dS)IN) defaultdict) GeneratorSequence)groupby) TYPE_CHECKINGAny)Image) deprecated) LazyLoader)MultiModalHasher)BatchedTensorInputsMultiModalFieldElemMultiModalKwargsItemMultiModalPlaceholderDictMultiModalSharedField) AudioMediaIO ImageMediaIOMediaConnector VideoMediaIOtorchWAVformataudio sampling_raterreturncCst}|j||f|dS)zEncode audio as base64.) audio_format)r encode_base64)rrraudio_ior b/lsinfo/ai/hellotax_ai/llm_service/venv_vllm/lib/python3.10/site-packages/vllm/multimodal/utils.pyencode_audio_base64!sr"cC4t|||d}tjd|d}d|d|S)zEncode audio as a data URL.r.rdata:;base64,)r" mimetypes types_mapgetlower)rrr audio_b64mimetyper r r!encode_audio_url,sr-RGBPNG image_moderimager1cCst|d}|j||dS)z Encode a pillow image to base64 format. By default, the image is converted into RGB format before being encoded. )r1) image_format)rr)r2r1rimage_ior r r!encode_image_base648s r5cCr#)z| Encode a pillow image as a data URL. By default, the image is converted into RGB format before being encoded. r0r$r2r%r&)r5r'r(r)r*)r2r1r image_b64r,r r r!encode_image_urlGs r7JPEGframescCst}t|}|j||dS)N) video_format)rrr)r9rr4video_ior r r!encode_video_base64Wsr<cCsDt||d}|dkrd}n tjd|d}d|d|S)Nrjpegz video/jpegr$videor%r&)r<r*r'r(r))r9r video_b64r,r r r!encode_video_urlas  r@ mm_positionscCs0dd|D}t|ddd}dd|DS)a/ Given a `MultiModalPlaceholderDict`, output a sequence of keys to sort the dictionary by `offset` (starting index in the input sequence) in ascending order. Returns: A list of `(modality, idx)`, which can be used to access an item by `mm_positions[modality][idx]`. css0|]\}}t|D] \}}|||fVq qdSN) enumerate).0modalityitemsidxitemr r r! |s z'argsort_mm_positions..cSs |djS)N)offsetxr r r!s z&argsort_mm_positions..keycSsg|] \}}}||fqSr r )rDrErG_r r r! sz(argsort_mm_positions..)rFsorted)rA flat_itemssorted_flat_itemsr r r!argsort_mm_positionsps  rVelemcCst|jtsdStj|jdS)N)data) isinstancefieldrr hash_kwargsrX)rWr r r!_get_group_hashs r\Fdevice pin_memoryrFr^r_csVttttft}|D]}|D] \}}|||qq fdd|DS)Ncs(i|]\}}||djj|dqS)rr])rZ reduce_data)rDrPelemsr]r r! s z#_batch_mm_items..)rstrlistrrFappend)rFr^r_rarHrPrWr r]r!_batch_mm_itemss rfccspdd|D}ddt|D}d}|D]}t||||||d}||fV||7}q|t|ks6JdS)a Group consecutive items (possibly from different requests) into batches. Items must be split across groups if any of the following occurs, as the batch would otherwise be invalid: - They have different fields (e.g. mixed image and embedding inputs). - They have different values in `MultiModalSharedField`. Args: items: List of `MultiModalKwargsItem`. device: The device to place the grouped tensors on. pin_memory: Whether to pin memory for faster host-to-device transfer. Yields: A tuple `(num_items, grouped_kwargs)`, where: - `kwargs` is a dictionary of keyword arguments to pass to the model; - `num_items` is the corresponding number of items. cSs.g|]}tddt|dddDqS)css |] \}}|t|fVqdSrB)r\)rDrPrWr r r!rIs  6group_and_batch_mm_items...cS|dSNrr )kvr r r!rNz5group_and_batch_mm_items...rO)tuplerSrF)rDrHr r r!rRs z,group_and_batch_mm_items..cSs"g|] \}}tdd|DqS)css|]}dVqdS)r Nr )rDrQr r r!rIsrg)sum)rDrQgroupr r r!rRs"rr]N)rrflen)rFr^r_ group_ids group_sizes start_idx group_size group_datar r r!group_and_batch_mm_itemss  ru mm_kwargsccsRt|dddD]\}}dd|D}t|||dD] \}}|||fVqq dS)a Group consecutive items (possibly from different requests) into batches. Items must be split across groups if any of the following occurs, as the batch would otherwise be invalid: - They have different fields (e.g. mixed image and embedding inputs). - They have different values in `MultiModalSharedField`. To simplify the implementation of `embed_multimodal`, we add another restriction that the items in a batch must belong to the same modality. Args: mm_kwargs: List of `(modality, item)`. device: The device to place the grouped tensors on. pin_memory: Whether to pin memory for faster host-to-device transfer. Yields: A tuple `(modality, num_items, grouped_kwargs)`, where: - `modality` is the modality of the batch; - `kwargs` is a dictionary of keyword arguments to pass to the model; - `num_items` is the corresponding number of items. cSrhrir rLr r r!rNrkz+group_and_batch_mm_kwargs..rOcSsg|]\}}|qSr r )rDrQrHr r r!rRsz-group_and_batch_mm_kwargs..r]N)rru)rvr^r_rErn items_lst num_itemsmm_kwargs_batchr r r!group_and_batch_mm_kwargssrzzu`group_mm_kwargs_by_modality` has been renamed to `group_and_batch_mm_kwargs`. The old name will be removed in v0.19.cCst|||dS)Nr])rz)rvr^r_r r r!group_mm_kwargs_by_modalitys r{ audio_urlaudio_io_kwargscC&|sdnd|i}t|dd}||S)a+ Args: audio_url: URL of the audio file to fetch. audio_io_kwargs: Additional kwargs passed to handle audio IO. Warning: This method has direct access to local files and is only intended to be called by user code. Never call this from the online server! Nr/media_io_kwargsallowed_local_media_path)r fetch_audio)r|r}rmedia_connectorr r r!r   r image_urlimage_io_kwargscCr~)a+ Args: image_url: URL of the image file to fetch. image_io_kwargs: Additional kwargs passed to handle image IO. Warning: This method has direct access to local files and is only intended to be called by user code. Never call this from the online server! Nr2rr)r fetch_image)rrrrr r r!rrr video_urlvideo_io_kwargscCr~)a+ Args: video_url: URL of the video file to fetch. video_io_kwargs: Additional kwargs passed to handle video IO. Warning: This method has direct access to local files and is only intended to be called by user code. Never call this from the online server! Nr>rr)r fetch_video)rrrrr r r!r1rrrB)?r' collectionsrcollections.abcrr itertoolsrtypingrrnumpynp numpy.typingnptPILrtyping_extensionsr vllm.utils.import_utilsr hasherr inputsr rrrrmediarrrr torch.typesrglobalsndarrayintrcr"r-r5r7NDArrayr<r@rdrlrVr\typesDeviceboolrfrurzr{dictfloatrrrr r r r!s*                   3 '