|j'y6ddlmZddlZddlZddlmZddlmZddlm Z m Z m Z ddl m Z ddlZddlmZd d lmZe rdd lmZdd lmZdd lmZddlmZd dlmZgZGdde ZGdde ZdZdZddZ ddZ!GddeZ"dS) ) annotationsN) defaultdict)reduce) TYPE_CHECKINGNoReturn TypedDict) NotRequired) framework) Optimizer)Sequence)Tensor)GradientClipBase)WeightDecayRegularizer)_ParameterConfigc~eZdZUded<ded<ded<ded<ded<ded <ded <ded <ded <d ed<ded<dS) _LbfgsStateint func_evalsn_iterrdalphaz list[Tensor]old_ykold_skroH_diagprev_flat_gradfloat prev_losszNotRequired[list[Tensor]]alN__name__ __module__ __qualname____annotations__f/lsinfo/ai/hellotax_ai/data_center/backend/venv/lib/python3.11/site-packages/paddle/optimizer/lbfgs.pyrr*sOOOKKK IIIMMMNNN!!!!!!r(rceZdZUded<dS)_LbfgsStateDictrstateNr"r'r(r)r+r+8sr(r+ctjr.tjddkrt jddSdSdS)z-Check and warn about TF32 acceleration statusNVIDIA_TF32_OVERRIDE0zWarning! TF32 Tensor Cores are enabled by default on some NVIDIA GPUs for faster computation, but may compromise numerical precision in specific cases, particularly with the L-BFGS optimizer.To disable it, set: NVIDIA_TF32_OVERRIDE=0N)paddledeviceis_compiled_with_cudaosgetenvwarningswarnr'r(r)check_tf32_overrider7<sc  ++-- I, - - 4 4  9       4 4r(c4||zdS)z NOTE: This is a temporary workaround for unstable result computed by `paddle.dot`, which will be reverted when the problem is fixed." axis)sumxys r)dotr@Is E;;B;  r(ct||\}}n||kr||fn||f\}}||zd||z z||z z z } | dz||zz } | dkrs| } ||kr|||z || z| z ||z d| zzz zz } n|||z || z| z ||z d| zzz zz } tt| ||S||zdz S)a]Cubic interpolation between (x1, f1, g1) and (x2, f2, g2). Use two points and their gradient to determine a cubic function and get the minimum point between them in the cubic curve. Reference: Jorge Nocedal, Stephen J. Wright, Numerical Optimization, Second Edition, 2006. pp59: formula 3.59 Args: x1, f1, g1: point1's position, value and gradient. x2, f2, g2: point2's position, value and gradient. bounds: bounds of interpolation area Returns: min_pos: the minimum point between the specified points in the cubic curve. Nr rg@)sqrtminmax) x1f1g1x2f2g2bounds xmin_bound xmax_boundd1 d2_squared2min_poss r)_cubic_interpolaterSQs $!' JJ-/2XX"bB8 J b1R=BG, ,BARIA~~ ^^   88BGb2"r'AF:J(KLLGGBGb2"r'AF:J(KLLG3w ++Z888Z'3..r(-C6??& .>c B|} |}||||\} } d}t| |}d|||f\}}}}d}d}|| kr| |||z|zzks |dkr)| |kr#||g}|| g}|| g}||g}nt|| |zkr |g}| g}| g}d}n|dkr#||g}|| g}|| g}||g}nt|d||z zz}|dz}|}t ||||| |||f}|}| }| }|}||||\} } |dz }t| |}|dz }|| k|| kr d|g}|| g}|| g}d}|d|dkrd nd \}}|sc|| kr\t|d|dz | z| krn5t |d|d|d|d|d|d}d t|t |z z} t t||z |t |z | kr|s&|t|ks|t |krht|t|z t|t |z krt|| z }nt || z}d}nd}nd}||||\} } |dz }t| |}|dz }| |||z|zzks | ||kr@|||<| ||<| ||<|||<|d|dkrd nd \}}nt|| |zkrd}nD|||||z zdkr,||||<||||<||||<||||<|||<| ||<| ||<|||<|s|| k\||}||} ||} | | ||fS) ag Implements of line search algorithm that satisfies the strong Wolfe conditions using double zoom. Reference: Jorge Nocedal, Stephen J. Wright, Numerical Optimization, Second Edition, 2006. pp60: Algorithm 3.5 (Line Search Algorithm). Args: obj_func: the objective function to minimize. ```` accepts a multivariate input and returns a scalar. xk (Tensor): the starting point of the iterates. alpha (Scalar): the initial step size. d (Tensor): search direction. loss (scalar): the initial loss grad (Tensor): the initial grad c1 (Scalar): parameter for sufficient decrease condition. c2 (Scalar): parameter for curvature condition. tolerance_change (Scalar): terminates if the change of function value/position/parameter between two iterations is smaller than this value. max_ls(int): max iteration of line search. alpha_max (float): max step length. Returns: loss_new (Scaler): loss of obj_func at final alpha. grad_new, (Tensor): derivative of obj_func at final alpha. alpha(Tensor): optimal step length, or 0. if the line search algorithm did not converge. ls_func_evals (Scaler): number of objective function called in line search process. Following summarizes the essentials of the strong Wolfe line search algorithm. Some notations used in the description: - `func` denotes the objective function. - `obi_func` is a function of step size alpha, restricting `obj_func` on a line. obi_func = func(xk + alpha * d), where xk is the position of k'th iterate, d is the line search direction(decent direction), and a is the step size. - alpha : substitute of alpha - a1 is alpha of last iteration, which is alpha_(i-1). - a2 is alpha of current iteration, which is alpha_i. - a_lo is alpha in left position when calls zoom, which is alpha_low. - a_hi is alpha in right position when calls zoom, which is alpha_high. Line Search Algorithm: repeat Compute obi_func(a2) and derphi(a2). 1. If obi_func(a2) > obi_func(0) + c_1 * a2 * obi_func'(0) or [obi_func(a2) >= obi_func(a1) and i > 1], alpha= zoom(a1, a2) and stop; 2. If |obi_func'(a2)| <= -c_2 * obi_func'(0), alpha= a2 and stop; 3. If obi_func'(a2) >= 0, alpha= zoom(a2, a1) and stop; a1 = a2 a2 = min(2 * a2, a2) i = i + 1 end(repeat) zoom(a_lo, a_hi) Algorithm: repeat aj = cubic_interpolation(a_lo, a_hi) Compute obi_func(aj) and derphi(aj). 1. If obi_func(aj) > obi_func(0) + c_1 * aj * obi_func'(0) or obi_func(aj) >= obi_func(a_lo), then a_hi <- aj; 2. 2.1. If |obi_func'(aj)| <= -c_2 * obi_func'(0), then alpha= a2 and stop; 2.2. If obi_func'(aj) * (a2 - a1) >= 0, then a_hi = a_lo a_lo = aj; end(repeat) reference: https://github.com/pytorch/pytorch r rFTg{Gz? )rLr9)rr )r rg?)absrEcloner@rSrD)!obj_funcxkrrlossgradgtdc1c2tolerance_changemax_lsd_normloss_newgrad_new ls_func_evalsgtd_newt_prevf_prevg_prevgtd_prevdonels_iterbracket bracket_f bracket_g bracket_gtdmin_stepmax_steptmpinsuf_progresslow_poshigh_posepss! r) _strong_wolfer{us\pUUWW[[]]F ::<222:"      h'   !!%Xb%33( h""1 a F  f&e*8$ 8$ N"+A,)B-"?"?VGXF+w'' wqzGAJ& ' '& 03C C C # AJ aL N AJ aL N   "S\\CLL01 s7||e#US\\%9 : :S @ @ &#g,,!6!6%3w<<:O:Ous7||+,,s53w<<3G/H/HHHLL3.EELL3.E!&!%"N%Xb%33( h""1  rEzC// 0 09W---!&GH "*Ih "*.."2"2Ih $+K !#A,)A,66F GXX7||sSy((GH-0@@AQFF$+G$4!&/&8 (#&/&8 (#(3G(< H% %GG !)Ig !)!1!1Ig #*K MF+w''R G E!H!H Xum 33r(ceZdZdZ d)d*fd Zd+dZd,dZd Zd!Zd"Z d#Z d$Z e j d-d&Z d.d/d(ZxZS)0LBFGSaJ The L-BFGS is a quasi-Newton method for solving an unconstrained optimization problem over a differentiable function. Closely related is the Newton method for minimization. Consider the iterate update formula: .. math:: x_{k+1} = x_{k} + H_k \nabla{f_k} If :math:`H_k` is the inverse Hessian of :math:`f` at :math:`x_k`, then it's the Newton method. If :math:`H_k` is symmetric and positive definite, used as an approximation of the inverse Hessian, then it's a quasi-Newton. In practice, the approximated Hessians are obtained by only using the gradients, over either whole or part of the search history, the former is BFGS, the latter is L-BFGS. Reference: Jorge Nocedal, Stephen J. Wright, Numerical Optimization, Second Edition, 2006. pp179: Algorithm 7.5 (L-BFGS). Args: learning_rate (float, optional): learning rate .The default value is 1. max_iter (int, optional): maximal number of iterations per optimization step. The default value is 20. max_eval (int|None, optional): maximal number of function evaluations per optimization step. The default value is max_iter * 1.25. tolerance_grad (float, optional): termination tolerance on first order optimality The default value is 1e-5. tolerance_change (float, optional): termination tolerance on function value/parameter changes. The default value is 1e-9. history_size (int, optional): update history size. The default value is 100. line_search_fn (string|None, optional): either 'strong_wolfe' or None. The default value is strong_wolfe. parameters (list|tuple|None, optional): List/Tuple of ``Tensor`` names to update to minimize ``loss``. \ This parameter is required in dygraph mode. The default value is None. weight_decay (int|float|WeightDecayRegularizer|None, optional): The strategy of regularization. \ It can be a int or float value as coeff of L2 regularization or \ :ref:`api_paddle_regularizer_L1Decay`, :ref:`api_paddle_regularizer_L2Decay`. If a parameter has set regularizer using :ref:`api_paddle_ParamAttr` already, \ the regularization setting here in optimizer will be ignored for this parameter. \ Otherwise, the regularization setting here in optimizer will take effect. \ Default None, meaning there is no regularization. grad_clip (GradientClipBase|None, optional): Gradient clipping strategy, it's an instance of \ some derived class of ``GradientClipBase`` . There are three clipping strategies \ ( :ref:`api_paddle_nn_ClipGradByGlobalNorm` , :ref:`api_paddle_nn_ClipGradByNorm` , \ :ref:`api_paddle_nn_ClipGradByValue` ). Default None, meaning there is no gradient clipping. name (str|None, optional): Normally there is no need for user to set this property. For more information, please refer to :ref:`api_guide_Name`. The default value is None. Return: loss (Tensor): the final loss of closure. Examples: .. code-block:: python >>> import paddle >>> import numpy as np >>> paddle.disable_static() >>> np.random.seed(0) >>> np_w = np.random.rand(1).astype(np.float32) >>> np_x = np.random.rand(1).astype(np.float32) >>> inputs = [np.random.rand(1).astype(np.float32) for i in range(10)] >>> # y = 2x >>> targets = [2 * x for x in inputs] >>> class Net(paddle.nn.Layer): ... def __init__(self): ... super().__init__() ... w = paddle.to_tensor(np_w) ... self.w = paddle.create_parameter(shape=w.shape, dtype=w.dtype, default_initializer=paddle.nn.initializer.Assign(w)) ... ... def forward(self, x): ... return self.w * x ... >>> net = Net() >>> opt = paddle.optimizer.LBFGS(learning_rate=1, max_iter=1, max_eval=None, tolerance_grad=1e-07, tolerance_change=1e-09, history_size=100, line_search_fn='strong_wolfe', parameters=net.parameters()) >>> def train_step(inputs, targets): ... def closure(): ... outputs = net(inputs) ... loss = paddle.nn.functional.mse_loss(outputs, targets) ... print('loss: ', loss.item()) ... opt.clear_grad() ... loss.backward() ... return loss ... opt.step(closure) ... >>> for input_np, target_np in zip(inputs, targets): ... input = paddle.to_tensor(input_np) ... target = paddle.to_tensor(target_np) ... train_step(input, target) ?NHz>rVd learning_ratermax_iterrmax_eval int | Nonetolerance_gradrc history_sizeline_search_fn str | None parameters4Sequence[Tensor] | Sequence[_ParameterConfig] | None weight_decay%float | WeightDecayRegularizer | None grad_clipGradientClipBase | NonenamereturnNonec @t||dzdz}||_||_||_||_||_||_||_t|tj rtdt|ztt|_t!d|| | | t|jdts |j|_n't)|jD]\} } | d|_d|_dS)Nz^parameters argument given to the optimizer should be an iterable of Tensors or dicts, but got r~)rrrrrrparams)r7rrrrrcrr isinstancer0r TypeErrortyperdictr,super__init___parameter_list_params enumerate _param_groups _numel_cache)selfrrrrrcrrrrrridx param_group __class__s r)rzLBFGS.__init__s;   !|q(H*    , 0(, j&- 0 0 <>B:>N>NO  !&&  !%    $.q1488 5/DLL$-d.@$A$A 5 5 [*84  r(r+czi}|jD]\}}|||id|iS)apReturns the state of the optimizer as a :class:`dict`. Return: state, a dict holding current optimization state. Its content differs between optimizer classes. Examples: .. code-block:: python >>> import paddle >>> paddle.disable_static() >>> net = paddle.nn.Linear(10, 10) >>> opt = paddle.optimizer.LBFGS( ... learning_rate=1, ... max_iter=1, ... max_eval=None, ... tolerance_grad=1e-07, ... tolerance_change=1e-09, ... history_size=100, ... line_search_fn='strong_wolfe', ... parameters=net.parameters(), >>> ) >>> def train_step(inputs, targets): ... def closure(): ... outputs = net(inputs) ... loss = paddle.nn.functional.mse_loss(outputs, targets) ... opt.clear_grad() ... loss.backward() ... return loss ... ... opt.step(closure) ... >>> inputs = paddle.rand([10, 10], dtype="float32") >>> targets = paddle.to_tensor([2 * x for x in inputs]) >>> n_iter = 0 >>> while n_iter < 20: ... loss = train_step(inputs, targets) ... n_iter = opt.state_dict()["state"]["func_evals"] ... print("n_iter:", n_iter) r,)r,itemsupdate)r packed_statekvs r) state_dictzLBFGS.state_dictsQ\ J$$&& ( (DAq   A ' ' ' '&&r(cV|jtd|jd|_|jS)Nc0||zSN)numel)totalps r)zLBFGS._numel..+s!2r(r)rrrrs r)_numelz LBFGS._numel's4   $ &22DL!!!D   r(cg}|jD]b}|j)tj|dg}n|jdg}||ctj|dS)Nr9rr:)rr_r0 zeros_likereshapeappendconcat)rviewsrviews r)_gather_flat_gradzLBFGS._gather_flat_grad0s  Av~(++33RD99v~~rd++ LL    }U++++r(c >d}|jD]x}|jgkrtd|jnd}tj|||||z|j|z|}||z }y||ksJdS)Nrc ||zSrr'r=s r)rz!LBFGS._add_grad..>s Ar(r )rshaperr0assignaddrr)rr directionoffsetrrs r) _add_gradzLBFGS._add_grad;s  A;<7b==F--qw777aE fv~56>>qwGG%O A eOFF&&&&&&r(c$d|jDS)Nc6g|]}|Sr')r[).0rs r) z&LBFGS._clone_param..Is 000a 000r()rrs r) _clone_paramzLBFGS._clone_paramHs004<0000r(cft|j|D]\}}tj||dSr)ziprr0r)r params_datarpdatas r) _set_paramzLBFGS._set_paramKs@DL+66 $ $HAu M% # # # # $ $r(c|||t|}|}||||fSr)rrrr)rclosurer>rrr^ flat_grads r)_directional_evaluatezLBFGS._directional_evaluateOsV ua   WWYY**,,  Yr(rc f tj5tjj}j}j}j}j}j}j }j } | dd| dd} t| } d} | dxxdz cc< } | |k}|r| cdddS| d}| d}| d}| d }| d }| d }| d }| d }d}||kr|dz }| dxxdz cc<| ddkr7| }g}g}g}tjd| j}n+| |}|tj||j}t-||}|dkrt/||kr?|d|d|d|||||d|z |t-||z }t/|}d| vr dg|z| d<| d}| }t5|dz ddD]\}t-|||||z||<tj||||| z|]tj||x}}t5|D][}t-|||||z} tj|||||| z z|\|| }ntj| || }| ddkr;t=dd| z |z}n|}t-| |}!|!| krnd}"||dkrtAd!}#fd}$tE|$|#||| | |!\} } }}"#||| |k}n#||||krtj5t} dddn #1swxYwY } | |k}d}"| |"z } | dxx|"z cc<|rnZ||z|krn,t| |z |krn| |krn||krn||k|| d<|| d<|| d<|| d <|| d <|| d <|| d <|| d <dddn #1swxYwY| S)aPerforms a single optimization step. Args: closure (callable): A closure that reevaluates the model and returns the loss. Examples: .. code-block:: python >>> import paddle >>> paddle.disable_static() >>> inputs = paddle.rand([10, 10], dtype="float32") >>> targets = paddle.to_tensor([2 * x for x in inputs]) >>> net = paddle.nn.Linear(10, 10) >>> opt = paddle.optimizer.LBFGS( ... learning_rate=1, ... max_iter=1, ... max_eval=None, ... tolerance_grad=1e-07, ... tolerance_change=1e-09, ... history_size=100, ... line_search_fn='strong_wolfe', ... parameters=net.parameters(), >>> ) >>> def closure(): ... outputs = net(inputs) ... loss = paddle.nn.functional.mse_loss(outputs, targets) ... print("loss:", loss.item()) ... opt.clear_grad() ... loss.backward() ... return loss ... >>> opt.step(closure) rrrr Nrrrrrrrr r~)dtypeg|=r!r9 strong_wolfez only 'strong_wolfe' is supportedc4|||Sr)r)r>rrrrs r)r\zLBFGS.step..obj_funcs$#'#=#= 'E1$$r()$r0no_grad enable_gradrrrrrcrrr, setdefaultrrrZrEgetneg to_tensorrsubtractmultiplyr@lenpoprrangerrr[rDr< RuntimeErrorrr{r)%rrrrrrrcrrr, orig_lossr^ current_evalsropt_condrrrrrrrr rr?sysnum_oldr!qirbe_ir`rhx_initr\s%`` r)stepz LBFGS.stepVsR^  s +s +*f(**733G .M}H}H!0N#4 !0N,LJE   \1 - - -   Xq ) ) )  I##DM ,   1 $   ..00I }}**,,>H ! 7s +s +s +s +s +s +s +s +< #AIIg&&EYYx((FYYx((F4BYYx((F"YY'788N +..IF8##! h1$ ?a''! AFFB#-cIIIFF"**>::A 6#3E#I#I#IJJAQBEzzv;;,66"JJqMMM"JJqMMMFF1III a((( a((( #(+++"$c!Qii"&kkG5(('+f|&;d tB" A"7Q;B77FF #F1Iq 1 1BqE 91 aeeF1I"Q%,@&A&A1EEEE#OAv666A"7^^LL"6!9a002a58 aeeF1IA,F&G&GKKKK!)%.__%6%6NNM)^<<< ?a''Cy}}':':'<'>>ti'((+;;;!H,,X%%C8##FE#J"E'N$E(O$E(OE$K$E(O&4E" #!*E+ gs +s +s +s +s +s +s +s +s +s +s +s +s +s +s +js>C-Z&RZ&V5) Z&5V9 9Z&<V9 =CZ&&Z*-Z*rc td)z}Empty method. LBFGS optimizer does not use this way to minimize ``loss``. Please refer 'Examples' of LBFGS() above for usage.zeLBFGS optimizer does not use this way to minimize loss. Please refer 'Examples' of LBFGS() for usage.)NotImplementedError)rr^startup_programr no_grad_sets r)minimizezLBFGS.minimize6s" s   r() r~rNrrVrNNNNN)rrrrrrrrrcrrrrrrrrrrrrrrr)rr+)rr)rr)NNN)rr)r#r$r%__doc__rrrrrrrrr non_static_onlyrr __classcell__)rs@r)r}r}es0XXx ## $"&%)KO>B-11!1!1!1!1!1!1!f2'2'2'2'h!!!!,,, ' ' '111$$$]]]]@HL         r(r}r)rTrUrVrW)# __future__rr3r5 collectionsr functoolsrtypingrrrtyping_extensionsr r0baser optimizerr collections.abcrrpaddle.nn.cliprpaddle.regularizerrr__all__rr+r7r@rSr{r}r'r(r)rs#""""" ######5555555555))))))  ,((((((//////999999++++++  " " " " ") " " "i       !/!/!/!/X   m4m4m4m4`W W W W W IW W W W W r(