o :/i$@sddlZddlmZddlmZddlmZmZmZddl Z ddl mZ ddl mZddlmZddlmZz ddlZddlmZWneyVedZedd ZYnweeZd e jd eeefd e jfd dZd e jded e jfddZ d e jded e jfddZ!GdddeZ"GdddeZ#GdddZ$eZ%GdddZ&e%'dGddde$e&Z(e%'d Gd!d"d"e$e&Z)e%'d#Gd$d%d%e$e&Z*e%'d&Gd'd(d(e(Z+e%'d)Gd*d+d+e$e&Z,dS),N)abstractmethod)BytesIO)Any NamedTuplecast) init_logger)PlaceholderModule)ExtensionManagercv2videoio_registryframessizereturnc Cs\|j\}}}}|\}}tj||||f|jd}t|D]\}} t| ||f} | ||<q|S)Ndtype)shapenpemptyr enumerater resize) r r num_frames_channels new_height new_widthresized_framesiframe resized_framerb/lsinfo/ai/hellotax_ai/llm_service/venv_vllm/lib/python3.10/site-packages/vllm/multimodal/video.py resize_videos r! size_factorcCs4|j\}}}}t||}t||}t|||fSN)rintr!)r r"rheightwidthrrrrr rescale_video_size's  r'rcCs<|jd}|dkr |Stjd|d|td}||df}|S)Nrr.)rrlinspacer$)r r total_frames frame_indicessampled_framesrrr sample_frames_from_video/s  r.c@*eZdZUdZeed<eed<eed<dS)VideoTargetMetadataz!Metadata represents target video.rfps max_durationN__name__ __module__ __qualname____doc__r$__annotations__floatrrrr r09  r0c@r/)VideoSourceMetadataz!Metadata represents source video.total_frames_num original_fpsdurationNr3rrrr r;Ar:r;c @steZdZedededeefddZee de de e j eeefffddZeded eed efd d Zd S) VideoLoadersourcetargetrcKt)z:Return the list of frame indices to sample from the video.NotImplementedError)clsr@rAkwargsrrr compute_frames_index_to_sampleJz*VideoLoader.compute_frames_index_to_sampledatacKrB)zFLoad video frames from bytes and return (frames_array, metadata_dict).rC)rErIrFrrr load_bytesTrHzVideoLoader.load_bytesvalid_frame_indices video_backendcCs"|j|j|j||t||jkdS)N)total_num_framesr1r>rLframes_indicesdo_sample_frames)r<r=r>len)rEr@rKrLrrr create_hf_metadata^s zVideoLoader.create_hf_metadataN)r4r5r6 classmethodr;r0listr$rGrbytestuplenptNDArraydictstrrrJrQrrrr r?Is0  r?c@seZdZeddZededdfddZedddefd d Z ed e d e e d e e e fde de f ddZeddde e de deeje e e e e fffddZedee de deeje e ffddZeddddde e de de deeje e ff ddZdS)OpenCVVideoBackendMixincCs^d}tD]&}t|sqt|s(t|\}}}|dks'|dkr(|dkr(q|}|S|S)Nr))vrgetStreamBufferedBackends hasBackendisBackendBuiltIn%getStreamBufferedBackendPluginVersion)api_prefbackendrabiapirrr get_cv2_video_apiss   z)OpenCVVideoBackendMixin.get_cv2_video_apirIrzcv2.VideoCapturecCs.|}tt||g}|std|S)NzCould not open video stream)rer VideoCapturerisOpened ValueError)rErIrbcaprrr open_video_captures z*OpenCVVideoBackendMixin.open_video_capturericCs>t|tj}|tj}|dkr||nd}t|||dS)Nrr<r=r>)r$getr CAP_PROP_FRAME_COUNT CAP_PROP_FPSr;)rir<r=r>rrr get_video_metadatas z*OpenCVVideoBackendMixin.get_video_metadataidx failed_framesnext_target_mapr+cCs$|sdS|d}|||}||kS)z;Check if current frame can recover the oldest failed frame.Fr)rl)rErprqrrr+ oldest_failedlimitrrr _can_use_for_recoverys   z-OpenCVVideoBackendMixin._can_use_for_recoveryr,cCst|tj}t|tj}|dkr|dks"Jd|d|t|}|r,|dnd}i}tt|dD] } || d||| <q8|||d<g} g} g} i} d}t|dD]o}||v}|}|su|rtt d|| |q[| || ||}|s|r| \}}|r|dur|jdkrt|tj}| || ||d7}|r| d}|| |<t d||||q[|rt d || |q[| D]}t d |q| rt| }n tjd||d ftjd }|| | fS) a Read frames with dynamic window forward-scan recovery. When a target frame fails to load, the next successfully grabbed frame (before the next target frame) will be used to recover it. Args: cap: OpenCV VideoCapture object frame_indices: Sorted list of target frame indices to load total_frames: Total number of frames in the video Returns: Tuple of (frames_array, valid_frame_indices, recovered_map) - frames_array: Array of loaded frames - valid_frame_indices: List of frame indices that were loaded - recovered_map: Dict mapping recovered_idx -> source_idx rz Invalid video frame size: width=z , height=r(r)z-Failed to grab frame %d during video loading.Nz-Recovered frame %d using frame %d (delay: %d)z1Failed to retrieve frame %d during video loading.z/Frame %d could not be recovered (end of video).r)r$rlr CAP_PROP_FRAME_WIDTHCAP_PROP_FRAME_HEIGHTsetrangerPgrabloggerwarningappendruretriever cvtColor COLOR_BGR2RGBpopinforstackruint8)rErir,r+r&r% frame_idx_set max_frame_idxrrk frames_listrKfailed_frames_idx recovered_maprrpis_target_frameok can_recoverretr rgb_frame recovered_idx failed_idxr rrr _read_frames_with_recoverys~         z2OpenCVVideoBackendMixin._read_frames_with_recoveryrcCst|}t|tj}t|tj}tj|||dftjd}d}g} t |dD]8} | } | s=| |vrrr1minmathfloorrrSrzrr*r$tolist) rEr@rArFr<r>rr1rruniform_sampled_framesrrr rGhs$   z1OpenCVVideoBackend.compute_frames_index_to_sampler(,FrIrr1r2rcK`||}t|}t|||d} |j|| d} |j|| |j|d\} } |j|d| d} | | fS)a Load video frames from bytes. Args: data: Raw video bytes num_frames: Target number of frames to sample (-1 for all) fps: Target FPS for sampling (-1 for original) max_duration: Maximum duration (unused in base backend) frame_recovery: Enable forward-scan recovery for failed frames Returns: Tuple of (frames_array, metadata_dict) rr1r2r@rAr<rrr@rLrKrjrZror0rGrr<rQ)rErIrr1r2rrFrir@rArr rKmetadatarrr rJs.   zOpenCVVideoBackend.load_bytesNr(r(rFr4r5r6rRr;r0rSr$rGrTrrUrVrWrXrYrrJrrrr rfs6 ropencv_dynamicc@leZdZedededeefddZe    dd e d ed ed ede de e j eeefff ddZdS)OpenCVDynamicVideoBackendr@rArc  s|j}|j}|j|j}|j|jd||kr2tt|}tfddt |D}|St|} | |krDt t |}|St j d|| dd} tfdd| D}|S)Nr)c s(h|]}ttt|qSrrr$rceil).0rr1rr=rr szKOpenCVDynamicVideoBackend.compute_frames_index_to_sample..rT)endpointc s$h|]}ttt|qSrrrt)rr=rr rs) r<r>r=r2r1r$rrsortedrzrSrr*) rEr@rArFr<r>r2nframe_indices_list num_samplestarget_secondsrrr rGs2     z8OpenCVDynamicVideoBackend.compute_frames_index_to_sampler(r[rFrIrr1r2rcKs||}t|}|jd} |jpt| |jd} t|j|j| d} t|||d} |j | | d} |j || | j|d\}}|j | d|d}||fS) Load video frames with dynamic sampling based on duration. Args: data: Raw video bytes num_frames: Not used in dynamic backend fps: Target FPS for sampling (default: 2) max_duration: Maximum video duration to process (default: 300s) frame_recovery: Enable forward-scan recovery for failed frames Returns: Tuple of (frames_array, metadata_dict) r)rkrrrrr) rjrZror<r>roundr=r;r0rGrrQ)rErIrr1r2rrFri orig_sourcerr>r@rArr rKrrrr rJs>    z$OpenCVDynamicVideoBackend.load_bytesNr(r[rFrrrrr rs6 &rmolmo2c@s^eZdZe d#dedededeefddZededed ed ed eeded Bf d dZ eded Bd ededede ed Be j ff ddZ e d$deded eded Bd eed Bde j f ddZededefddZe    d%ded ed Bdedededede e j eeefffdd Ze d&dedede e j eeefffd!d"Zd S)'Molmo2VideoBackend @ video_fps sampling_fpsmax_fpsrcCst|}t|}t|}|durtd|dks|dkr'td|d|d||dkr8td|d|d g}t||d |D]}||krK|S||dkrX|t|qB|S) a Return the subset of `video_fps` factors that remain multiples of `sampling_fps`. Examples: >>> get_candidate_target_fps(video_fps=6, sampling_fps=2) [2, 6] >>> get_candidate_target_fps(video_fps=5, sampling_fps=1) [1, 5] >>> get_candidate_target_fps(video_fps=2, sampling_fps=2) [2] >>> get_candidate_target_fps(video_fps=5, sampling_fps=2) Traceback (most recent call last): ... ValueError: sampling_fps=2 must divide video_fps=5 to produce consistent frame steps. Nzsampling_fps must be providedrz1video_fps and sampling_fps must be positive (got z, )z sampling_fps=z must divide video_fps=.r))r$rhrzr~r9)rErrr candidates candidaterrr get_candidate_target_fps)s4  z+Molmo2VideoBackend.get_candidate_target_fps max_framesr+frame_sample_modecandidate_target_fpsNc Csd}d}|D]8}tt||d} t|| } |dkr+d|vr&| |kr&|S|}| }q|| ks1J| |kr6q| |kr>|}| }q|S)z] Get the target fps that best spans the videoand has the most frames sampled rNr)uniform)rr$) rErrr+rrnum_frames_sampledselected_target_fps target_fps step_sizenum_frames_sampled_at_fpsrrr get_target_fpsZs(   z!Molmo2VideoBackend.get_target_fpsrcCs^|durtjd||dtd}ntt||d}td||}t||kr+|d|}||fS)NrF)rrr))rr*r$rarangerP)rErr+rrr,rrrr get_frame_times_and_chosen_fpss   z1Molmo2VideoBackend.get_frame_times_and_chosen_fpsr>c Ks|dkr1|dus J|d}|ddD] }|||krn|}qtd||} | | |k} | S|dkr{|durn|d|} | |krPtjd||dtjd} | Stjd|d|d} tj| |ggdd } t| |kslJ| Stjd||dtjd} | St|) Nr1rr)uniform_last_frameT)numrrstopstepaxis)rrr*float64 concatenaterPrD) rEr>rrrrrFr candidate_fpstimesr2rrr sample_timess:      zMolmo2VideoBackend.sample_timesr@rAcKs|d}|d}|durttd|jS|dvr!td||j}|j}|j}|j} |j} |dkr|dur|dkrHt | t } | S|| d|kret j d|dt| |d d  t } | St j d |dt||d } t | d |dkrt j| |dggdd} t |  t } | d |ksJt| | ksJ| S|dkrt j d|dt| |d d  t } | S|dkr||| } ||| ||| }|||| |\}} | S)Nrrr>r1rzUnsupported frame_sample_mode: rr[r)T)rrrrr(rr1)rlrSrzr<rDr>r=rr1rrastyper$r*rr9rrrPrrrr)rEr@rArFrrr>rrMrrindices float_indicesrrrrrr rGs  ,)  z1Molmo2VideoBackend.compute_frames_index_to_sampler(r[FrIrrcKsf||}t|} t||| jd} |j| | ||d} |j|| | j|d\} } |j| d| d}| |fS)Nr)r@rArrrrr) rjrZror0r>rGrr<rQ)rErIrrrrrrFrir@rArr rKrrrr load_bytes_opencvs2  z$Molmo2VideoBackend.load_bytes_opencvcKsXttdB|dd}tt|dd}tt|dd}|j|||||fi|}|S)Nrrr[r)rrYrr$r)rErIrrFrrroutrrr rJ0szMolmo2VideoBackend.load_bytes)rr#)Nr(r[r[F)r()r4r5r6rRr9rSrr$rYrrUrVrWrrr;r0rGrTrrXrrrJrrrr r's 0 '   ) F )r nemotron_vlc@sNeZdZe    d dededededed eej e e e fff d d Z d S)NemotronVLVideoBackendr(rFrIrr1r2rrc Ks8tj|f||||d|\}}t|}||d<||fS)N)rr1r2roriginal_video_bytes)rrJrX) rErIrr1r2rrFr rrrr rJGs   z!NemotronVLVideoBackend.load_bytesNr)r4r5r6rRrTr$rrUrVrWrXrYrrJrrrr rEs&r openpanguc@r)"OpenCVDynamicOpenPanguVideoBackendr@rArc  s|j|j|j}|j}dkrdkrd}nd}|dkr=|t||dkrszUOpenCVDynamicOpenPanguVideoBackend.compute_frames_index_to_sample..) r<r=rr1r$rrhrr*r9) rEr@rArFrr1total_durationsample_frame_timestampsrNrrr rGbs. zAOpenCVDynamicOpenPanguVideoBackend.compute_frames_index_to_sampler(r[rFrIrr1r2rcKr)rrrrrrr)rErIrr1r2rrFrir@rArr rKrrrr rJs.   z-OpenCVDynamicOpenPanguVideoBackend.load_bytesNrrrrrr r`s6 )r)-rabcriortypingrrrnumpyr numpy.typingrV vllm.loggerrvllm.utils.import_utilsrvllm.utils.registryr r cv2.videoio_registryr r\ ImportErrorplaceholder_attrr4r|rWrUr$r!r9r'r.r0r;r?VIDEO_LOADER_REGISTRYrZregisterrrrrrrrrr sL       "  &uVi