zjPddlZddlmZddlmZddlmZmZddlm Z d dZ dZ dS) N)fleet)in_dynamic_mode)HeterParallelOptimizerHybridParallelOptimizer)loggerctj}||_|4|jrtjdt j||_i|_| dkr|jj sqt||j |j}|jj djrd|_d|_|jj djrd|_|jr Jd|St'||jS|S)a Optimizer for distributed training. For the distributed training, this method would rebuild a new instance of DistributedOptimizer. Which has basic Optimizer function and special features for distributed training. Args: optimizer(Optimizer): The executor to run for init server. strategy(DistributedStrategy): Extra properties for distributed optimizer. It is recommended to use DistributedStrategy in fleet.init(). The strategy here is for compatibility. If the strategy in fleet.distributed_optimizer() is not None, then it will overwrite the DistributedStrategy in fleet.init(), which will take effect in distributed training. Returns: Fleet: instance of fleet. Examples: .. code-block:: python >>> import paddle >>> import paddle.distributed.fleet as fleet >>> fleet.init(is_collective=True) >>> strategy = fleet.DistributedStrategy() >>> linear = paddle.nn.Linear(10, 10) >>> optimizer = paddle.optimizer.SGD(learning_rate=0.001, parameters=linear.parameters()) >>> optimizer = fleet.distributed_optimizer(optimizer, strategy=strategy) Na$It is recommended to use DistributedStrategy in fleet_env.init(). The strategy here is only for compatibility. If the strategy in fleet_env.distributed_optimizer() is not None, then it will overwrite the DistributedStrategy in fleet_env.init(), which will take effect in distributed training.r pp_configsFz7sep parallel can not coexist with sharding_comm_overlap)ruser_defined_optimizer_is_collectiverwarningcopydeepcopy_user_defined_strategy_context worker_numheter_ccl_moder_hcghybrid_configsdp_comm_overlap _dp_enable _sep_enablesharding_comm_overlap_sharding_enabler) optimizerstrategy fleet_envhp_optims r/lsinfo/ai/hellotax_ai/data_center/backend/venv/lib/python3.11/site-packages/paddle/distributed/fleet/optimizer.py_dygraph_distributed_optimizerr s34 I'0I$  #  NB   ,0=+B+B (I!!/> .9>9+KH/> -',#',$/># -2)#/M/O)9; cbtr t|i|Stjj|i|SN)rr rdistributed_optimizer)argskwargss rr$r$`s<B-t>v>>>{0$A&AAAr!r#) rpaddle.distributedrpaddle.frameworkrmeta_optimizersrrutils.log_utilrr r$r!rr,s $$$$$$,,,,,,LLLLLLLL""""""EEEEPBBBBBr!