jddlmZmZmZmZmZddlmZddl m Z ddl m Z m Z mZmZmZGddZGddeZGd d eZGd d eZGd deZGddeZGddeZGddeZGddeZGddeZGddeZGddeZdZdS))CallableListOptionalTupleUnionN)Module) tree_flattentree_map tree_merge tree_reducetree_unflattencveZdZdZddZdedefdZdefdZd e j d efd Z dedefd Z d e j d e j d efdZ edZejd efdZedZedZejdeee j ffdZdedeeee j ge j fffdZdS) OptimizerzThe base class for all optimizers. It allows us to implement an optimizer on a per-parameter basis and apply it to a parameter tree. Ncd|_dtjdtji|_d|piD|_dS)NFsteprci|]\}}|| Sr).0kvs c/lsinfo/ai/hellotax_ai/base_platform/venv/lib/python3.11/site-packages/mlx/optimizers/optimizers.py z&Optimizer.__init__..sHHHTQAqHHH) _initializedmxarrayuint64_stateitems _schedulers)self schedulerss r__init__zOptimizer.__init__sQ!rx29556 HHj.>B-E-E-G-GHHHrmodel gradientscX||||dS)aPApply the gradients to the parameters of the model and update the model with the new parameters. Args: model (mlx.nn.Module): An mlx module to be updated. gradients (dict): A Python tree of gradients, most likely computed via :func:`mlx.nn.value_and_grad`. N)updateapply_gradients)r!r$r%s rr'zOptimizer.updates,  T)))U;;<<<<>> optimizer = optim.SGD(learning_rate=1e-1, momentum=0.9) >>> model = nn.Linear(2, 2) >>> optimizer.init(model.trainable_parameters()) >>> optimizer.state.keys() dict_keys(['step', 'learning_rate', 'weight', 'bias']) c rt|ttfrt|}tt |D]}||||||<t |t |kr9|t d|t |dt||St|trJ| D]3\}}||vrt d|||<|||||<4|S|S)NciSNr_s rz6Optimizer.init..update_state..;sBrciSr-rr.s rr0z6Optimizer.init..update_state..@sbr) isinstancelisttuplerangelenextendr typedictr)paramsstateirr update_states rr=z$Optimizer.init..update_state5s.&4-00 U s5zz**AAA+|F1IuQx@@E!HHu::V,,LL,,s5zz||8L!M!MNNN#tF||E***FD)) "LLNN==DAq~~#+LL!#<#<a#/<58#<#<a  rc4|p||Sr-) init_single)psr!s rr0z Optimizer.init..Hsa94#3#3Aq#9#9rTN)rr r)r!r)r=s` @rinitzOptimizer.inits_,     $  Z---9999:t{SSS r parameterr;ct)zTo be extended by the children classes to implement each optimizer's state initialization. Args: parameter (mx.array): A single parameter that will be optimized. state (dict): The optimizer's state. NotImplementedErrorr!rCr;s rr?zOptimizer.init_singleK"###rc|js|||jD]\}}||j|j|<|jdz|jd<t |j|||jS)aApply the gradients to the parameters and return the updated parameters. Can be used to update a model via ``model.update(opt.apply_gradients(grads, model))`` which is precisely how :meth:`Optimizer.update` is implemented. Args: gradients (dict): A Python tree of gradients. parameters (dict): A Python tree of parameters. It can be a superset of the gradients. In that case the returned python tree will be of the same structure as the gradients. r)rrBr rrr;r apply_single)r!r%r)param schedulers rr(zOptimizer.apply_gradientsUs  ! IIi !% 0 6 6 8 8 5 5 E9 ) $) 4 4DJu  "Y] 6)9j$*MMMrgradientct)a To be extended by derived classes to implement the optimizer's update. Args: gradient (mx.array): The ``parameter`` gradient. parameter (mx.array): The ``parameter`` to update. state (dict): The optimizer's state. rE)r!rNrCr;s rrKzOptimizer.apply_singleorHrc|jS)z!The optimizer's state dictionary.)rr!s rr;zOptimizer.stateys {rc"d|_||_dS)NF)rr)r!r;s rr;zOptimizer.state~s! rc|jdS)Nrr;rQs rrzOptimizer.stepsz&!!rc|jdSN learning_raterTrQs rrWzOptimizer.learning_ratesz/**rrWc>tj||jd<dSrV)rrr;)r!rWs rrWzOptimizer.learning_rates&(h}&=&= ?###rnamerLct|tr||j|<||j}nt j|}||j|<dS)z\ To be used by derived classes to optionally put a parameter on a schedule. N)r2rr rrrr;)r!rYrLrCs r_maybe_schedulezOptimizer._maybe_schedulesU eX & & (%*D T "di((III$ 4rr-)__name__ __module__ __qualname____doc__r#rr9r'rBrrr?r(rKpropertyr;setterrrWrfloatstrrr[rrrrr sIIII =F =t = = = =*!t*!*!*!*!X$RX$d$$$$NN4NNNN4$RX$"($4$$$$X \4\""X"++X+>5+A>>>> % % %eXrxj"(6J-K&K L % % % % % %rrceZdZdZgfdeffd ZdefdZdefdZdedefdZ e d Z e j d efd Z e d Z e j d eeejffdZ xZS)MultiOptimizeralWraps a list of optimizers with corresponding weight predicates/filters to make it easy to use different optimizers for different weights. The predicates take the full "path" of the weight and the weight itself and return True if it should be considered for this optimizer. The last optimizer in the list is a fallback optimizer and no predicate should be given for it. Args: optimizers (list[Optimizer]): A list of optimizers to delegate to filters (list[Callable[[str, array], bool]): A list of predicates that should be one less than the provided optimizers. filtersc(ti|_t|t|dz kr3t dt|dt|dz d||_|dgz|_dS)NrJzGiven z filters but z needed.cdS)NTr)argskwargss rr0z)MultiOptimizer.__init__..s$r)superr#rr6 ValueError optimizersrf)r!rmrf __class__s rr#zMultiOptimizer.__init__s  w<<3z??Q. . .OWOOC OOA4EOOO %">">!?? rr%c^t|jdkr|gSdtt|jD}t|}|D]J\}}t |jD]0\}}|||r||||fn1Kd|DS)NrJcg|]}gSrr)rr/s r z4MultiOptimizer._split_dictionary..s999999rc,g|]}t|Sr)r )rr@s rrqz4MultiOptimizer._split_dictionary..s 111aq!!111r)r6rmr5r enumeraterfappend)r!r%partsflat_gradientsrgr<fns r_split_dictionaryz MultiOptimizer._split_dictionarys t  1 $ $; 99U3t#7#788999%i00"  DAq"4<00  22a88!HOOQF+++E2151111rr)ct|j||D]\}}||dSr-)ziprmryrB)r!r)or@s rrBzMultiOptimizer.initsJ)?)? )K)KLL  DAq FF1IIII  rci}t|j||D])\}}t||||}*|Sr-)r{rmryr r()r!r%r)treer|rws rr(zMultiOptimizer.apply_gradientss\)?)? )J)JKK F FDAqdA$5$5a$D$DEEDD rc(dd|jDiS)Nstatescg|] }|j SrrT)rr|s rrqz(MultiOptimizer.state..s<<>X> \4\00X0,5+A,,,,,,,,rrec eZdZdZ ddeeeejgejffdededede f fd Z d ejd e fd Z d ejd ejd e fdZ xZS)SGDaThe stochastic gradient descent optimizer. Updates a parameter :math:`w` with a gradient :math:`g` as follows .. math:: v_{t+1} &= \mu v_t + (1 - \tau) g_t \\ w_{t+1} &= w_t - \lambda v_{t+1} Args: learning_rate (float or callable): The learning rate :math:`\lambda`. momentum (float, optional): The momentum strength :math:`\mu`. Default: ``0`` weight_decay (float, optional): The weight decay (L2 penalty). Default: ``0`` dampening (float, optional): Dampening for momentum :math:`\tau`. Default: ``0`` nesterov (bool, optional): Enables Nesterov momentum. Default: ``False`` FrWmomentum weight_decay dampeningnesterovc|r|dks|dkrtdt|d|||_||_||_||_dS)Nrz9Nesterov momentum requires a momentum and zero dampening.rW)rlrkr#r[rrrr)r!rWrrrrrns rr#z SGD.__init__s  Q)q..K   _m<<<  ("  rrCr;c4tj||d<dSInitialize optimizer staterNr zeros_likerGs rr?zSGD.init_single ]9--c rrNc|jdkr ||j|zz }|jdkr%||j|j|zz S|j|dz}|jdkr|d|jz |zz }n||z }|jr||j|zz}n|}||d<||j|j|zz S)zVPerforms the SGD parameter update and stores :math:`v` in the optimizer state.rrrJ)rrrWastypedtypegetrr)r!rNrCr;rr's rrKzSGD.apply_singles   ! ! )I5 5H =A  t188HH8SS S MEIIcNN * >A   !dn$0 0AA MA =   11FFFc 4-44X^DDvMMMr)rrrF)r\r]r^r_rrbrrrboolr#r9r?rKrrs@rrrs(! !!UHbhZ-A$BBC!! !  !  !!!!!!(.RX.d....NRXN"(N4NNNNNNNNrrceZdZdZ d deeeejgejffdedeffd Z dejd e fd Z d ejdejd e fd Z xZ S)RMSpropagThe RMSprop optimizer [1]. [1]: Tieleman, T. and Hinton, G. 2012. Lecture 6.5-rmsprop, coursera: Neural networks for machine learning .. math:: v_{t+1} &= \alpha v_t + (1 - \alpha) g_t^2 \\ w_{t+1} &= w_t - \lambda \frac{g_t}{\sqrt{v_{t+1}} + \epsilon} Args: learning_rate (float or callable): The learning rate :math:`\lambda`. alpha (float, optional): The smoothing constant :math:`\alpha`. Default: ``0.99`` eps (float, optional): The term :math:`\epsilon` added to the denominator to improve numerical stability. Default: ``1e-8`` Gz?:0yE>rWalphaepsct|d|||_||_|jdkrt d|jd|jdkrt d|jddS)NrWrzRMSprop alpha should be >=0,  was provided insteadzRMSprop epsilon should be >0, )rkr#r[rrrl)r!rWrrrns rr#zRMSprop.__init__;s  _m<<<  :  Q QQQ  8c>>PPPP  >rrCr;c4tj||d<dSrrrGs rr?zRMSprop.init_singlePrrrNc|j|j}|j}|j}|d}||zd|z t j|zz}||d<|||zt j||zz z S)zRPerforms the RMSprop parameter update and stores :math:`v` in the optimizer state.rrJ)rWrrrrrsquaresqrt)r!rNrCr;lrrrrs rrKzRMSprop.apply_singleTs|   & &x~ 6 6 h #J AIUbi&9&99 9c 2=BGAJJ,<===r)rrr\r]r^r_rrbrrrr#r9r?rKrrs@rrr)s( UHbhZ-A$BBC *.RX.d.... >RX >"( >4 > > > > > > > >rrceZdZdZ d deeeejgejffdeffd Z dejde fdZ d ejdejde fd Z xZ S) AdagradaXThe Adagrad optimizer [1]. Our Adagrad implementation follows the original paper. In detail, [1]: Duchi, J., Hazan, E. and Singer, Y., 2011. Adaptive subgradient methods for online learning and stochastic optimization. JMLR 2011. .. math:: v_{t+1} &= v_t + g_t^2 \\ w_{t+1} &= w_t - \lambda \frac{g_t}{\sqrt{v_{t+1}} + \epsilon} Args: learning_rate (float or callable): The learning rate :math:`\lambda`. eps (float, optional): The term :math:`\epsilon` added to the denominator to improve numerical stability. Default: ``1e-8`` rrWrct|d|||_|jdkrt d|jddS)NrWrzAdagrad epsilon should be >0, r)rkr#r[rrl)r!rWrrns rr#zAdagrad.__init__tsl  _m<<< 8c>>PPPP  >rrCr;c4tj||d<dSrrrGs rr?zAdagrad.init_singlerrrNc|j|j}|j}|dt j|z}||d<|||zt j||zz z S)zZPerforms the Adagrad parameter update and stores :math:`v` in the optimizer state.r)rWrrrrrr)r!rNrCr;rrrs rrKzAdagrad.apply_singlese  & &x~ 6 6h #J8,, ,c 2=BGAJJ,<===r)rrrs@rrras*  UHbhZ-A$BBC       .RX.d.... >RX >"( >4 > > > > > > > >rrceZdZdZ d deeeejgejffdedeffd Z dejd e fd Z d ejdejd e fd Z xZ S)AdaDeltaahThe AdaDelta optimizer with a learning rate [1]. Our AdaDelta implementation follows the original paper. In detail, [1]: Zeiler, M.D., 2012. ADADELTA: an adaptive learning rate method. arXiv preprint arXiv:1212.5701. .. math:: v_{t+1} &= \rho v_t + (1 - \rho) g_t^2 \\ \Delta w_{t+1} &= \frac{\sqrt{u_t + \epsilon}}{\sqrt{v_{t+1} + \epsilon}} g_t \\ u_{t+1} &= \rho u_t + (1 - \rho) \Delta w_{t+1}^2 \\ w_{t+1} &= w_t - \lambda \Delta w_{t+1} Args: learning_rate (float or callable): The learning rate :math:`\lambda`. rho (float, optional): The coefficient :math:`\rho` used for computing a running average of squared gradients. Default: ``0.9`` eps (float, optional): The term :math:`\epsilon` added to the denominator to improve numerical stability. Default: `1e-8` ?ư>rWrhorct|d|||_||_|jdkrt d|jd|jdkrt d|jddS)NrWrzAdaDelta rho should be >=0, rzAdaDelta epsilon should be >0, )rkr#r[rrrl)r!rWrrrns rr#zAdaDelta.__init__s  _m<<< 8c>>NtxNNN  8c>>Q$(QQQ  >rrCr;cbtj||d<tj||d<dS)rruNrrGs rr?zAdaDelta.init_single,]9--c ]9--c rrNc|j|j}|j}|j}|d}|d}||zd|z t j|zz}t j||zt j||zz |z} ||zd|z t j| zz}||d<||d<||| zz S)ziPerforms the AdaDelta parameter update and stores :math:`v` and :math:`u` in the optimizer state.rrrJ)rWrrrrrrr) r!rNrCr;rrrrrds rrKzAdaDelta.apply_singles  & &x~ 6 6hh #J #J !Gq3w")H"5"55 5 GAG  rwq3w// /( : !Gq3w")A,,. .c c 26!!r)rrrrs@rrrs0 UHbhZ-A$BBC (.RX.d.... "RX""("4""""""""rrc eZdZdZddgddfdeeeejgejffde eded e ffd Z d ejd e fd Z dejd ejd e fdZxZS)AdamaThe Adam optimizer [1]. In detail, [1]: Kingma, D.P. and Ba, J., 2015. Adam: A method for stochastic optimization. ICLR 2015. .. math:: m_{t+1} &= \beta_1 m_t + (1 - \beta_1) g_t \\ v_{t+1} &= \beta_2 v_t + (1 - \beta_2) g_t^2 \\ w_{t+1} &= w_t - \lambda \frac{m_{t+1}}{\sqrt{v_{t+1}} + \epsilon} Args: learning_rate (float or callable): The learning rate :math:`\lambda`. betas (Tuple[float, float], optional): The coefficients :math:`(\beta_1, \beta_2)` used for computing running averages of the gradient and its square. Default: ``(0.9, 0.999)`` eps (float, optional): The term :math:`\epsilon` added to the denominator to improve numerical stability. Default: ``1e-8`` bias_correction (bool, optional): If set to ``True``, bias correction is applied. Default: ``False`` r+?rFrWbetasrbias_correctionct|d|||_||_||_dSrV)rkr#r[rrr)r!rWrrrrns rr#z Adam.__init__sM  _m<<< .rrCr;cbtj||d<tj||d<dSrmrNrrGs rr?zAdam.init_singlerrrNcJ|j|j}|j\}}|j}|j}|j} |d} |d} || zd|z |zz} || zd|z tj|zz} | |d<| |d<|r||d|| zz z |j} tj d|| zz |j} | | z}tj | | z|z}|||z z S||| ztj | |zz z S)zePerforms the Adam parameter update and stores :math:`v` and :math:`m` in the optimizer state.rrrJ) rWrrrrrrrrrsqrtr)r!rNrCr;rb1b2rrrrrc1c2 numerator denominators rrKzAdam.apply_singles:  & &x~ 6 6Bh.y #J #J Fa"f( ( Fa"f ( 3 33 3c c  ;BH %--hn==B!b$h,''..x~>>BQI'!**r/C/Ky;66 6rAvc)9:: :r)r\r]r^r_rrbrrrrrr#r9r?rKrrs@rrrs2"5\ % / /UHbhZ-A$BBC /E{ / /  / / / / / /.RX.d.... ;RX;"(;4;;;;;;;;rrc eZdZdZddgdddfdeeeejgejffde ed ed ed e f fd Z d ejdejde ffd Z xZS)AdamWa2The AdamW optimizer [1]. We update the weights with a weight_decay (:math:`\lambda`) value: [1]: Loshchilov, I. and Hutter, F., 2019. Decoupled weight decay regularization. ICLR 2019. .. math:: m_{t+1} &= \beta_1 m_t + (1 - \beta_1) g_t \\ v_{t+1} &= \beta_2 v_t + (1 - \beta_2) g_t^2 \\ w_{t+1} &= w_t - \alpha (\frac{m_{t+1}}{\sqrt{v_{t+1}} + \epsilon} + \lambda w_t) Args: learning_rate (float or callable): The learning rate :math:`\alpha`. betas (Tuple[float, float], optional): The coefficients :math:`(\beta_1, \beta_2)` used for computing running averages of the gradient and its square. Default: ``(0.9, 0.999)`` eps (float, optional): The term :math:`\epsilon` added to the denominator to improve numerical stability. Default: ``1e-8`` weight_decay (float, optional): The weight decay :math:`\lambda`. Default: ``0.01``. bias_correction (bool, optional): If set to ``True``, bias correction is applied. Default: ``False`` rrr{Gz?FrWrrrrc`t||||||_dS)N)rWrrr)rkr#r)r!rWrrrrrns rr#zAdamW.__init__4sB '+    )rrNrCr;c|j|j}t||d||jzz z|S)zbPerforms the AdamW parameter update by modifying the parameters passed into Adam. rJ)rWrrrkrKr)r!rNrCr;rrns rrKzAdamW.apply_singleDsR   & &x~ 6 6ww## i1rD,='=#=>   r)r\r]r^r_rrbrrrrrr#r9rKrrs@rrrs8"5\" % ))UHbhZ-A$BBC)E{) )  )  ))))))  RX "( 4          rrceZdZdZddgdfdeeeejgejffde edeffd Z d ejd e fd Z d ejd ejd e fd Z xZS)AdamaxaThe Adamax optimizer, a variant of Adam based on the infinity norm [1]. Our Adam implementation follows the original paper and omits the bias correction in the first and second moment estimates. In detail, [1]: Kingma, D.P. and Ba, J., 2015. Adam: A method for stochastic optimization. ICLR 2015. .. math:: m_{t+1} &= \beta_1 m_t + (1 - \beta_1) g_t \\ v_{t+1} &= \max(\beta_2 v_t, |g_t|) \\ w_{t+1} &= w_t - \lambda \frac{m_{t+1}}{v_{t+1} + \epsilon} Args: learning_rate (float or callable): The learning rate :math:`\lambda`. betas (Tuple[float, float], optional): The coefficients :math:`(\beta_1, \beta_2)` used for computing running averages of the gradient and its square. Default: ``(0.9, 0.999)`` eps (float, optional): The term :math:`\epsilon` added to the denominator to improve numerical stability. Default: ``1e-8`` rrrrWrrct|||d|kstd|jddS)NrzEpsilon value should be >=0, r)rkr#rlr)r!rWrrrns rr#zAdamax.__init__gsS s333czzOOOO zrrCr;cbtj||d<tj||d<dSrrrGs rr?zAdamax.init_singlesrrrNc"|j|j}|j\}}|j}|d}|d} ||zd|z |zz}t j|| zt j|} ||d<| |d<|||z| |zz z S)zgPerforms the Adamax parameter update and stores :math:`v` and :math:`m` in the optimizer state.rrrJ)rWrrrrrmaximumabs) r!rNrCr;rrrrrrs rrKzAdamax.apply_singlexs  & &x~ 6 6Bh #J #J Fa"f( ( JrAvrvh// 0 0c c 26QW---rr\r]r^r_rrbrrrrr#r9r?rKrrs@rrrOs4"5\  UHbhZ-A$BBC E{       .RX.d.... .RX."(.4........rrceZdZdZddgdfdeeeejgejffde edeffd Z d ejd e fd Z d ejd ejd e fd Z xZS)Liona&The Lion optimizer [1]. Since updates are computed through the sign operation, they tend to have larger norm than for other optimizers such as SGD and Adam. We recommend a learning rate that is 3-10x smaller than AdamW and a weight decay 3-10x larger than AdamW to maintain the strength (lr * wd). Our Lion implementation follows the original paper. In detail, [1]: Chen, X. Symbolic Discovery of Optimization Algorithms. arXiv preprint arXiv:2302.06675. .. math:: c_{t + 1} &= \beta_1 m_t + (1 - \beta_1) g_t \\ m_{t + 1} &= \beta_2 m_t + (1 - \beta_2) g_t \\ w_{t + 1} &= w_t - \eta (\text{sign}(c_t) + \lambda w_t) Args: learning_rate (float or callable): The learning rate :math:`\eta`. betas (Tuple[float, float], optional): The coefficients :math:`(\beta_1, \beta_2)` used for computing the gradient momentum and update direction. Default: ``(0.9, 0.99)`` weight_decay (float, optional): The weight decay :math:`\lambda`. Default: ``0.0`` rrrrWrrct|d|||_||_dSrV)rkr#r[rr)r!rWrrrns rr#z Lion.__init__sF  _m<<< (rrCr;c4tj||d<dS)rrNrrGs rr?zLion.init_singlerrrNc|j|j}|j\}}|j}|d}||zd|z |zz} ||zd|z |zz|d<|dkr d||zz |z}||t j| zz S)zWPerforms the Lion parameter update and stores :math:`m` in the optimizer state.rrJr)rWrrrrrsign) r!rNrCr;rrrrrcs rrKzLion.apply_singles  & &x~ 6 6B( #J Fa"f( (!Vq2v11c !  R,..);I2 ?**rrrs@rrrs:"4[! ) )UHbhZ-A$BBC )E{ ) ) ) ) ) ) ).RX.d.... +RX +"( +4 + + + + + + + +rrceZdZdZ dd eeeejgejfdfd e eefd ed ed e edede de de ffd Z dejde fdZdZdZdZdejdejde fdZxZS) AdafactoraThe Adafactor optimizer. Our Adafactor implementation follows the original paper: `Adafactor: Adaptive Learning Rates with Sublinear Memory Cost `_ Args: learning_rate (float or callable, optional): The learning rate. Default: ``None``. eps (tuple(float, float), optional): The first term :math:`\epsilon_1` added to the square of the gradients to improve numerical stability and the second term :math:`\epsilon_2` is used for parameter scaling if ``parameter_scale`` is set to ``True``. Default: ``(1e-30, 1e-3)``. clip_threshold (float, optional): Clips the unscaled update at ``clip_threshold``. Default: ``1.0``. decay_rate (float, optional): Coefficient for the running average of the squared gradient. Default: ``-0.8``. beta_1 (float, optional): If set to a value bigger than zero then first moment will be used. Default: ``None``. weight_decay (float, optional): The weight decay :math:`\lambda`. Default: ``0.0``. scale_parameter (bool, optional): If set to ``True`` the learning rate will be scaled by :math:`\max(\epsilon_1, \text{RMS}(w_{t-1}))`. Default: ``True``. relative_step (bool, optional): If set to ``True`` the ``learning_rate`` will be ignored and relative step size will be computed. Default: ``True``. warmup_init (bool, optional): If set to ``True`` then the relative step size will be calculated by the current step. Default: ``False``. NgKH9gMbP??皙rTFrWrclip_threshold decay_ratebeta_1rscale_parameter relative_step warmup_initc t||d|||_||_||_||_||_||_||_ | |_ dSrV) rkr#r[rrrrrrrr) r!rWrrrrrrrrrns rr#zAdafactor.__init__sz   $  - @ @ @,$ (.*&rrCr;cB|jdkr\|j}|j}tj|dd||d<tj|dd|ddz||d<ntj||d<|jtj||d <dSdS) rN)rexp_avg_sq_rowexp_avg_sq_col exp_avg_sqexp_avg)ndimshaperrzerosrr)r!rCr;rrs rr?zAdafactor.init_singles >Q  OEOE&(huSbSz&G&G&GE" #&(huSbSzE"##J/Fe&T&T&TE" # #"$- ":":E,  ; "!}Y77E)    # "rcrtjtjtj|Sr-)rrmeanr)r!inputss r _compute_rmszAdafactor._compute_rms s&wrwry0011222rc|jr6|jrd|znd}tj|tj|}n|j}||j}d}|jr tj |j d|}||zS)NrrrrJ) rrrminimumrrWrrrrr)r!r parameter_rmsmin_steprelative_step_sizeparameter_scales r_compute_learning_ratez Adafactor._compute_learning_rates   4&*&6@td{{DH!#Hbhtnn!E!E  !%!3 /66}7JKK   E j!mDDO!333rctj|tj|ddz }tj|}tjtj|dtj|dS)NrT)axiskeepdimsrr)rrrmatmul expand_dims)r!rrr_factorc_factors r_approximate_exp_moving_avgz%Adafactor._approximate_exp_moving_avgst8 RW^"tLLL L  8N++y N8" - - -r~hQ/O/O/O   rrNc||jdk}|j}|jdu}||}|||}d||jz|jz } tj ||j dz} |rz|d} |d} | | zd| z tj | d zz} | | zd| z tj | d zz} | |d<| |d<| | | } | |z} n2|d } | | zd| z | zz} | |d <tj | |z} | tjd|| |jz z } || z} |r'|d }|j|zd|jz | zz}||d <|} |jdkr|||j |zzz }|| z S) z2Performs the Adafactor parameter and state update.rNrrrrrJrrrrr)rrrrrrrrrrrrrrrrr)r!rNrCr;factoredruse_first_momentrrWbeta_2r'rrrrs rrKzAdafactor.apply_single$s"=A%y;d2)))44 33D-HH do-55m6IJJJ8$$tx{2  5"#34N"#34N$~5VrwvB7777N%~5VrwvB7777N'5E" #&4E" #55nnUUFh&FF|,J :-1v:2GHJ",E, Xj))H4F"* ""6**T-@@   '  I&G{W,!dk/V1KLG&E) F   ! ! t'8&8=&HI II6!!r) NrrrNrTTF)r\r]r^r_rrbrrrrrrr#r9r?rrrrKrrs@rrrskFMQ#0 # "&! $"!''UHbhZ-A$BDHI'5%< ' '  '  ''''''''''0 8RX 8d 8 8 8 8333 4 4 4   ,"RX,""(,"4,",",",",",",","rrc eZdZdZ ddeeeejgejffdeded e d e f fd Z d ejd e fdZ de fdZdejd ejd e fdZxZS)Muona%The Muon optimizer. Our Muon (MomentUm Orthogonalized by Newton-schulz) optimizer follows the original implementation: `Muon: An optimizer for hidden layers in neural networks `_ Note: - Muon may be sub-optimal for the embedding layer, the final fully connected layer, or any 0D/1D parameters. Those should be optimized by a different method (e.g., :class:`AdamW`). - For 4D convolutional filters, it works by flattening their last dimensions. Args: learning_rate (float or callable): The learning rate. momentum (float, optional): The momentum strength. Default: ``0.95`` weight_decay (float, optional): The weight decay (L2 penalty). Default: ``0.01`` nesterov (bool, optional): Enables Nesterov momentum. Recommended for better performance. Default: ``True`` ns_steps (int, optional): Number of Newton-Schulz iteration steps for orthogonalization. Default: ``5`` ffffff?rTrWrrrns_stepsct|d|||_||_||_||_dSrV)rkr#r[rrrr)r!rWrrrrrns rr#z Muon.__init__lsR  _m<<<  (    rrCr;c4tj||d<dSrrrGs rr?zMuon.init_single|rrstepsc|jdksJd|jdd\}}}|jd|jdk}|r|j}|tj|dd zz }t |D]D}||jz}tj||z||d | } tj||z| |d d }E|r|j}|S) Nrz;Expected a 2D array for Newton-Schulz iteration, got shape z instead.)guV @ggn@@rrT)rgHz>r)betar)rrTrlinalgnormr5addmm) r!Xrabrtranspose_neededr/ABs r_zeropower_via_newtonschulz5z!Muon._zeropower_via_newtonschulz5s FaKKK [ [ [ [ KK+1a72;4  A D11D8 9u ; ;AACAQ13a888AQ13c:::AA  ArrNcT|jdkr ||j|zz}|j|dz}|d|jz |zz}||d<|jr|d|jz z||jzz}n|}|j|j}|jdkr|j}|jdk}|r"tj ||jddf}| ||j }|rtj ||}|td|jd|jdz dzz}|||zz S) z"Performs the Muon parameter updaterrrJrr)rrg?) rrrrWrrrrrreshaper!rmax) r!rNrCr;rr'roriginal_shapereshape_neededs rrKzMuon.apply_singlesK   ! !$"3i"??H ME#J & T]"h. .c = T]!23a$-6GGFFF   & &x~ 6 6 ;!  #\N#[1_N CFV\!_b,ABB66vT]6SSF <FN;; #ab)FL,<<==D DB2;&&r)rrTr)r\r]r^r_rrbrrrrintr#r9r?r!rKrrs@rrrSs6" !!UHbhZ-A$BBC!! !  !  !!!!!! .RX.d....S*'RX'"('4''''''''rrctd|d}tj|}tj||dzz dt fd|}||fS)aClips the global norm of the gradients. This function ensures that the global norm of the gradients does not exceed ``max_norm``. It scales down the gradients proportionally if their norm is greater than ``max_norm``. Example: >>> grads = {"w1": mx.array([2, 3]), "w2": mx.array([1])} >>> clipped_grads, total_norm = clip_grad_norm(grads, max_norm=2.0) >>> print(clipped_grads) {"w1": mx.array([...]), "w2": mx.array([...])} Args: grads (dict): A dictionary containing the gradient arrays. max_norm (float): The maximum allowed global norm of the gradients. Returns: (dict, float): The possibly rescaled gradients and the original gradient norm. cT||zSr-)rsum)accrws rr0z clip_grad_norm..scAHHJJNN4D4D.Drrrrc|zSr-r)rw normalizers rr0z clip_grad_norm..s q:~r)r rrrr )gradsmax_norm norm_squared total_norm clipped_gradsr-s @rclip_grad_normr3sh*DDeSQQL&&JH T(9:C@@J5555u==M * $$r)typingrrrrrmlx.corecorermlx.nnr mlx.utilsr r r r r rrerrrrrrrrrrr3rrrr9s:9999999999999UUUUUUUUUUUUUUP%P%P%P%P%P%P%P%fF,F,F,F,F,YF,F,F,R@N@N@N@N@N)@N@N@NF5>5>5>5>5>i5>5>5>p/>/>/>/>/>i/>/>/>d@"@"@"@"@"y@"@"@"FA;A;A;A;A;9A;A;A;H2 2 2 2 2 D2 2 2 j8.8.8.8.8.T8.8.8.v7+7+7+7+7+97+7+7+tL"L"L"L"L" L"L"L"^a'a'a'a'a'9a'a'a'H%%%%%r