ujb XddlZddlmZmZddlmZeZGddZdS)N)ListTuple) get_loggerceZdZdZdedefdZedefdZdedefdZded e fd Z ded e d e fd Z defdZdedeee ee ffdZdedefdZdS) BatchTrackerzThread-safe tracker for pre-assigned upload batches. Files are assigned to batches by sorted index (file_index // batch_size). Upload threads record results; main thread waits for batches in order. total_files batch_sizec||_|dkr |dz |zdznd|_dt|jD|_dt|jD|_g|_t|jD]7}||z}t ||z|}|j||z 8dt|jD|_tj |_ dS)Nrcg|]}gSr .0_s n/lsinfo/ai/hellotax_ai/data_center/backend/venv/lib/python3.11/site-packages/modelscope/hub/upload_pipeline.py z)BatchTracker.__init__..s%1 1 1 B1 1 1 cg|]}gSr r rs rrz)BatchTracker.__init__..s%3 3 3 B3 3 3 rc4g|]}tjSr ) threadingEventrs rrz)BatchTracker.__init__..!s-5 5 5 "#IO  5 5 5 r) _batch_size _num_batchesrange_batch_results_batch_failures_batch_expectedminappend _batch_eventsrLock_lock)selfrr istartends r__init__zBatchTracker.__init__s8%7BQ) !%/02344LM 1 1 d/001 1 1 3 3 d/003 3 3 +-t()) 5 5A NEej(+66C  ' 'e 4 4 4 45 5 ',T->'?'?5 5 5 ^%% rreturnc|jSN)r)r#s r num_batcheszBatchTracker.num_batches&s   r file_indexc||jzSr*)r)r#r,s r batch_indexzBatchTracker.batch_index*sT---rresultc||}|j5|j||||r|j|ddddS#1swxYwYdSr*)r.r"rr_is_batch_completer set)r#r,r/idxs rrecord_successzBatchTracker.record_success-sz** Z . .   $ + +F 3 3 3&&s++ ."3'++--- . . . . . . . . . . . . . . . . . .sAA??BBitemerrorc||}|j5|j|||f||r|j|ddddS#1swxYwYdSr*)r.r"rrr1r r2)r#r,r5r6r3s rrecord_failurezBatchTracker.record_failure4sz** Z . .   % , ,dE] ; ; ;&&s++ ."3'++--- . . . . . . . . . . . . . . . . . .sABBBc||}|j5|j|xxdzcc<||r|j|ddddS#1swxYwYdS)zMark a file as skipped (already committed). Decrements the batch's expected count so _is_batch_complete uses the correct target. When all files in a batch are skipped, the batch event is set automatically. r N)r.r"rr1r r2)r#r,r3s rmark_file_skippedzBatchTracker.mark_file_skipped;sz** Z . .   % % % * % % %&&s++ ."3'++--- . . . . . . . . . . . . . . . . . .sA A44A8;A8 batch_idxc|j||j5t|j|t|j|fcdddS#1swxYwYdS)axWait for a batch to complete. Blocks indefinitely until all files in the batch have reported success or failure. Per-blob timeouts (UPLOAD_BLOB_TIMEOUT) prevent individual uploads from hanging forever. Args: batch_idx: Index of the batch to wait for. Returns: Tuple of (successful_results, failures). N)r waitr"listrr)r#r;s rwait_for_batchzBatchTracker.wait_for_batchHs 9%**,,, Z 1 1+I677$Y/:1:11 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1s4A((A,/A,ct|j|t|j|z}||j|kS)z Must be called under self._lock.)lenrrr)r#r;counts rr1zBatchTracker._is_batch_completeZsFD' 233c   +7-7--,Y777rN)__name__ __module__ __qualname____doc__intr'propertyr+r.dictr4tuple Exceptionr8r:rrr?boolr1r rrrr sE &C&S&&&&(!S!!!X!.c.c......d......E.).... .C . . . .11d4j$u+6M0N1111$8C8D888888rr)rtypingrrmodelscope.utils.loggerrloggerrr rrrPs...... S8S8S8S8S8S8S8S8S8S8r