zj<ddlZddlmZmZmZmZgZdZdZdZ dS)N) get_clusterget_cluster_from_argsget_gpusloggerc tjd}| Jdtjd}| Jdtjd}| Jdttjd}| Jd |d }t |}t|}|d kr!||krt jd |d |d|d kr4|d |krt jd|d|dtjd} | |} |dkr} ttjdd} |t |kr | |krt jd| d| } n&#t$r} t| Yd} ~ nd} ~ wwxYw| d} tt| | t |z} g} |D]#| fd| D$ng| d }g} ||zt |ksJt|D](}| |||z|dz|z)t j d|d|d|d| t||| |\}}||j|fS)zZ args_node_ips:string, args_node_ip:string, args_port: int, selected_devices:list PADDLE_TRAINERSNz"PADDLE_TRAINERS should not be NonePOD_IPzPOD_IP should not be NonePADDLE_TRAINER_IDz$PADDLE_TRAINER_ID should not be NoneTRAINER_PORTS_NUMz$TRAINER_PORTS_NUM should not be None,z 127.0.0.1zcPlease NOTE: When using paddlecloud, node_ip is automatically got from POD_IP. Your input node_ip: z doesn't equals to node_ip: z from paddlecloud environment.zPlease NOTE: When using paddlecloud, cluster_node_ips is automatically got from PADDLE_TRAINERS(multi nodes) or POD_IP(single node).Your input cluster_node_ips: z doesn't equals to IPs: DISTRIBUTED_TRAINER_ENDPOINTS PADDLE_PORTzUse Cloud specified port:.icg|] }d| S):).0portips n/lsinfo/ai/hellotax_ai/data_center/backend/venv/lib/python3.11/site-packages/paddle/distributed/cloud_utils.py z%get_cloud_cluster..Ws#%G%G%Gnndnn%G%G%Gzparsed from args: node_ips:z node_ip:z node_rank:z trainer_endpoints:)osgetenvintsplitlenrwarningjoin Exceptionprintlistrangeappenddebugrpods) args_node_ips args_node_ip args_portselected_devicesnode_ipsnode_ip node_rankpaddle_ports_num num_nodestrainer_endpoints started_port paddle_porteportstrainer_endpoints_oriiclusterpodrs @rget_cloud_clusterr;s y*++H   !E   i!!G    ;    -..I  "H 29%899::  ' ')O ' ' '~~c""HH III+'\"9"9 24@ 2 2  2 2 2    ## (9K9K(K(K +  EM       "ABB  q== !")M2">">?? %,<(=(===#y00N#M{#M#M#MNNN#.L   a   LU<LGS GL+ ++sAF F;"F66F;cFttjddS)NPADDLE_TRAINERS_NUM1)rrrrrr_get_trainers_numr?ns ry.44 5 55rcpt|j}t}tjd|d|d}d}|jrB|dkrrIs  P,P,P,f666r