jddlZddlZddlmZddlZgdZdZdZdZ dZ dZ dej eej zd dfd Z deddfd Zddeddfdd deej d ej eej zdzdededeejjdzdeej dzd dfdZ ddeej ded dfdZ ddeej deej d dfdZeddfdeej deej ded dfdZdS)N)Sequence) all_reducereduce broadcast all_gatherreduce_scattercFttjdstjdddSt }|D]]}|jrdS|sdS|jsdS| }||vrdS| |^dS)N_nccl_all_reducez)PyTorch is not compiled with NCCL support stacklevelFT) hasattrtorch_Cwarningswarnset is_sparse is_contiguousis_cuda get_deviceadd)tensorsdevicestensordevices Y/lsinfo/ai/hellotax_ai/base_platform/venv/lib/python3.11/site-packages/torch/cuda/nccl.py is_availablers 58/ 0 0 AaPPPPueeG     55##%% 55~ 55""$$ W  55 F 4ctj}|dz }|dz dz}|dz}tjd}|dkr|||fS||||fS)a Returns the version of the NCCL. This function returns a tuple containing the major, minor, and patch version numbers of the NCCL. The suffix is also included in the tuple if a version suffix exists. Returns: tuple: The version information of the NCCL. izutf-8)rr _nccl_version_nccl_version_suffixdecode)vermajorminorpatchsuffixs rversionr,#s} ( " "C 2IE BY% E %KE X * * , , 3 3G < tjSN)rr_nccl_unique_idrr unique_idr18s 8 # # % %%rcDtj|||Sr.)rr_nccl_init_rank) num_ranksuidranks r init_rankr7<s 8 # #IsD 9 99rinputsreturnct|tjjrt|tjrt ddS)Nz(Inputs should be a collection of tensors) isinstance collectionsabc ContainerrTensor TypeError)r8s r_check_sequence_typerA@sP fko7 8 8DJ ==DBCCCDDrct|||}t|tj|||||dSr.)rArrr r8outputsopstreamscommss rrrGsL   !!! Hfgr7EBBBBBr)rDoutputrootrErFrDct||6|tdtjdtd||}njt |t jsDt |tj j r%tjdtd||}n |||n|}t j ||||||dS)Nz'output' and 'outputs' can not be both specified. 'outputs' is deprecated in favor of 'output', taking in a single output tensor. The signature of reduce is: reduce(inputs, output=None, root=0, op=SUM, streams=None, comms=None).z`nccl.reduce` with an output tensor list is deprecated. Please specify a single output tensor with argument 'output' instead instead.r r z\nccl.reduce with an output tensor list is deprecated. Please specify a single output tensor.) rA ValueErrorrr FutureWarningr;rr?r<r=rr _nccl_reduce)r8rHrIrErFrGrD_outputs rrrQs     Y  M`      dmGG  - - =* (33 =   5      ,"(.&,,f H&'4WeDDDDDrcht|tj||||dSr.)rArr_nccl_broadcast)r8rIrFrGs rrr|s4    HVT7E:::::rct|t|tj||||dSr.)rArr_nccl_all_gather)r8rDrFrGs rrrsB    !!! Hfgw>>>>>rct|t|tj|||||dSr.)rArr_nccl_reduce_scatterrCs rrrsD   !!! H!!&'2wFFFFFr)rNN)NN)r<rcollections.abcr torch.cudar__all__SUMrr,r1r7r?rArintcudaStreamrrrrr0rrr\s$$$$$$ P O O*---*&&&:::D0F!FD4DDDD $TCCCC<@26  (E.2(E(E(E U\ "(E L8EL1 1D 8(E (E (E ej' (4 / (Eel #d *(E (E(E(E(EXHL;; U\ ";*-; ;;;;   ?? U\ "? el #?  ????   G G U\ " G el # G  G  G G G G G Gr