j:b~ddlmZddlZddlmZmZmZmZmZm Z ddl m Z ddl mZmZGddeZdZdZdS) ) annotationsN)AnyCallableListOptionalTupleUnion) tree_flattentree_unflattenceZdZUdZded<dZedZedZdHd Z d Z dIfd Z dJfd Z fdZ dKdLdZdMdZedZedZedZedZ dNdOd$Zd%Zd&Zd'Zd(ZdKdPd+Z dQdRd-ZdKdSd/ZdTd2Zd3Zd4Zd5Z ddd6d7dUd;Z!ddd6d7dUd<Z"dVd?Z#dKdWd@Z$dXdAZ%dBfdYdGZ&xZ'S)ZModuleaBase class for building neural networks with MLX. All the layers provided in :mod:`mlx.nn.layers` subclass this class and your models should do the same. A ``Module`` can contain other ``Module`` instances or :class:`mlx.core.array` instances in arbitrary nesting of python lists or dicts. The ``Module`` then allows recursively extracting all the :class:`mlx.core.array` instances using :meth:`mlx.nn.Module.parameters`. In addition, the ``Module`` has the concept of trainable and non trainable parameters (called "frozen"). When using :func:`mlx.nn.value_and_grad` the gradients are returned only with respect to the trainable parameters. All arrays in a module are trainable unless they are added in the "frozen" set by calling :meth:`freeze`. .. code-block:: python import mlx.core as mx import mlx.nn as nn class MyMLP(nn.Module): def __init__(self, in_dims: int, out_dims: int, hidden_dims: int = 16): super().__init__() self.in_proj = nn.Linear(in_dims, hidden_dims) self.out_proj = nn.Linear(hidden_dims, out_dims) def __call__(self, x): x = self.in_proj(x) x = mx.maximum(x, 0) return self.out_proj(x) model = MyMLP(2, 1) # All the model parameters are created but since MLX is lazy by # default, they are not evaluated yet. Calling `mx.eval` actually # allocates memory and initializes the parameters. mx.eval(model.parameters()) # Setting a parameter to a new value is as simply as accessing that # parameter and assigning a new array to it. model.in_proj.weight = model.in_proj.weight * 2 mx.eval(model.parameters()) r__call__c:t|_d|_dS)z1Should be called by the subclasses of ``Module``.TN)set_no_grad _trainingselfs \/lsinfo/ai/hellotax_ai/base_platform/venv/lib/python3.11/site-packages/mlx/nn/layers/base.py__init__zModule.__init__=s c|jS)z4Boolean indicating if the model is in training mode.rrs rtrainingzModule.trainingBs ~rc|S)akThe module's state dictionary The module's state dictionary contains any attribute set on the module including parameters in :meth:`Module.parameters` Unlike :meth:`Module.parameters`, the :attr:`Module.state` property is a reference to the module's state. Updates to it will be reflected in the original module. rs rstatez Module.stateGs  rreturnstrcdS)Nrrs r _extra_reprzModule._extra_reprTsrrc 6t||j}t|jd|}|D]6\}}|dz }|t jd|dt|dz }7|r|dz }|dz }|S)N)is_leaf( z): z )prefix)) r children is_moduletype__name__r"textwrapindentrepr)rr)valuekvs r__repr__zModule.__repr__Ws HHH::&==)9)9););== G GDAq TME X_%8%8%8tAww%8%8FFF FEE   TME   rkeyc||dx}|Stt||dSN)getsuperr __getattribute__)rr4r0 __class__s r __getattr__zModule.__getattr__csEXXc4(( (E 5L &$   0 0 5 5 5 5 5rvalrc6t|tjttt fr+t ||r||vrt|||||<dStt| ||| |ddSr6) isinstancemxarraydictlisttuplehasattrdelattrr8r __setattr__pop)rr4r<r:s rrFzModule.__setattr__is cBHdD%8 9 9 tS!! #cooc"""DIII &$   + +C 5 5 5 HHS$     rc||dx}||=dSt|dSr6)r7r8 __delattr__)rnamer<r:s rrIzModule.__delattr__usE88D$'' 'C 4T GG   % % % % %rTfile_or_weights&Union[str, List[Tuple[str, mx.array]]]strictboolc Z|}t|tr3ttj|}|rt |}t|i}| | z x}rGt|}d t|}td|d|d| | z x}rGt|} d t|}td| d|d|D]{\} } || } t| tjs"tdt| d | | j| jkr"td | jd | jd | |t|d kr$|t%|d |S)ay Update the model's weights from a ``.npz``, a ``.safetensors`` file, or a list. Args: file_or_weights (str or list(tuple(str, mx.array))): The path to the weights ``.npz`` file (``.npz`` or ``.safetensors``) or a list of pairs of parameter names and arrays. strict (bool, optional): If ``True`` then checks that the provided weights exactly match the parameters of the model. Otherwise, only the weights actually contained in the model are loaded and shapes are not checked. Default: ``True``. Returns: The module instance after updating the weights. Example: .. code-block:: python import mlx.core as mx import mlx.nn as nn model = nn.Linear(10, 10) # Load from file model.load_weights("weights.npz") # Load from .safetensors file model.load_weights("weights.safetensors") # Load from list weights = [ ("weight", mx.random.uniform(shape=(10, 10))), ("bias", mx.zeros((10,))), ] model.load_weights(weights) # Missing weight weights = [ ("weight", mx.random.uniform(shape=(10, 10))), ] # Raises a ValueError exception model.load_weights(weights) # Ok, only updates the weight but not the bias model.load_weights(weights, strict=False)  destinationz, z Received z parameters not in model: .zMissing z parameters: zExpected mx.array but received z for parameter zExpected shape z but received shape rF)rM)r>rrBr?loaditemsrAr parameterskeyslenjoinsorted ValueErrorr@r+shapeupdater ) rrKrMweights new_weights curr_weightsextras num_extramissing num_missingr1r2v_news r load_weightszModule.load_weights{sZh" gs # # 5277++113344G  w--K'(9(9rJJJL%**,,|/@/@/B/BBCv KK F6NN33 P PPvPPP(,,..1A1A1C1CCDw T!'ll **VG__55 !RK!R!R!R!R!RSSS$**,,  1#A!%22$;;;;;78;;;!'))$A!'AA!&AA=>AA* w<<1   KKw//K > > > rfilect|i}|drtj|fi|dS|drtj||dSt d|d)z Save the model's weights to a file. The saving method is determined by the file extension: - ``.npz`` will use :func:`mx.savez` - ``.safetensors`` will use :func:`mx.save_safetensors` rPz.npzz .safetensorszUnsupported file extension for z. Use '.npz' or '.safetensors'.N)r rUendswithr?savezsave_safetensorsrZ)rrf params_dicts r save_weightszModule.save_weightss #4??#4#4"EEE ==   HT ) )[ ) ) ) ) ) ]]> * *   k 2 2 2 2 2W$WWW rc,t|tSr6r>r )r0s rr*zModule.is_modules%(((rc:t|ttfSr6)r>rArBmoduler4r0s rvalid_child_filterzModule.valid_child_filters%$...rc|t|tttjfo|d S)N_)r>rArBr?r@ startswithrps rvalid_parameter_filterzModule.valid_parameter_filters0%$bh!788TPSATAT=TTrcLt|||o||jvSr6)r rvrrps rtrainable_parameter_filterz!Module.trainable_parameter_filters-  ) )&#u = = +6?* rN filter_fn"Callable[[Module, str, Any], bool]map_fnOptional[Callable] is_leaf_fn,Optional[Callable[[Module, str, Any], bool]]cdpdpdfdDS)atRecursively filter the contents of the module using ``filter_fn``, namely only select keys and values where ``filter_fn`` returns true. This is used to implement :meth:`parameters` and :meth:`trainable_parameters` but it can also be used to extract any subset of the module's parameters. Args: filter_fn (Callable): Given a value, the key in which it is found and the containing module, decide whether to keep the value or drop it. map_fn (Callable, optional): Optionally transform the value before returning it. is_leaf_fn (Callable, optional): Given a value, the key in which it is found and the containing module decide if it is a leaf. Returns: A dictionary containing the contents of the module recursively filtered c|Sr6rxs rz'Module.filter_and_map..sarcHt|tttf Sr6)r>r rArBmr1r2s rrz'Module.filter_and_map..s 1vtT.B C CCrc Zi|]'\}}|||t||(Sr_unwrap).0r1r2ryr}r{rs r z)Module.filter_and_map..sU   1yq!$$ wtQ9fjAA   r)rT)rryr{r}s````rfilter_and_mapzModule.filter_and_mapsi2(KK C C             rc6||jS)zoRecursively return all the :class:`mlx.core.array` members of this Module as a dict of dicts and lists.)rrvrs rrUzModule.parameterss""4#>???rc6||jS)zzRecursively return all the non frozen :class:`mlx.core.array` members of this Module as a dict of dicts and lists.)rrxrs rtrainable_parameterszModule.trainable_parameterss""4#BCCCrc<||jdS)z6Return the direct descendants of this Module instance.c,t|tSr6rnrs rrz!Module.children..%s 1f@U@Urr}rrrrs rr)zModule.children"s+""  #0U0U#   rc@d}||j|S)z8Return the submodules that do not contain other modules.ct|to1tt|dkS)Nr)r>r rWr r)rs r_is_leaf_modulez,Module.leaf_modules.._is_leaf_module+s5a((QSajjll1K1K-L-LPQ-Q Qrrr)rrs r leaf_moduleszModule.leaf_modules(s2 R R R""4#:"WWWrrUrAc.fd|||S)aLReplace the parameters of this Module with the provided ones in the dict of dicts and lists. Commonly used by the optimizer to change the model to the updated (optimized) parameters. Also used by the :meth:`mlx.nn.value_and_grad` to set the tracers in the model in order to compute gradients. The passed in parameters dictionary need not be a full dictionary similar to :meth:`parameters`. Only the provided locations will be updated. Args: parameters (dict): A complete or partial dictionary of the modules parameters. strict (bool): If ``True`` checks that ``parameters`` is a subset of the module's parameters. Default: ``True``. Returns: The module instance after updating the parameters. c t|tr|D]}||vr~||}||}t|tjrGr?t|tjs%t dt |jd|||<w||rt d|ddSt|trtt|D]}|t|kr&r#t d|dt|d;||}||}t|tjrGr?t|tjs%t dt |jd|||<||dSr%t dt |jddS)NReceived invalid type: rRz&Module does not have parameter named "".z List index z, is out of bounds for destination of length ) r>rAr?r@rZr+r,rBrangerW)dstrUr1 current_value new_valueiapplyrMs rrzModule.update..applyEsS*d++# Y# Y YACxx(+A $.qM %mRX>><%"jBH.M.M"&0$Yd9oo>V$Y$Y$Y'"'"!"&/CFF!E-;;;;Y()WRS)W)W)WXXXY Y YJ-- Ys://88ACHH}}!",!Ea!E!E9!>  D'' 6::;;; rmodulesc(t||||S)aJReplace the child modules of this :class:`Module` instance with the provided ones in the dict of dicts and lists. It is the equivalent of :meth:`Module.update` but for modules instead of parameters and allows us to flexibly edit complex architectures by programmatically swapping layers. The passed in parameters dictionary need not be a full dictionary similar to :meth:`modules`. Only the provided locations will be updated. Args: modules (dict): A complete or partial dictionary of the module's submodules. strict (bool): If ``True`` checks that ``modules`` is a subset of the child modules of this instance. Default: ``True``. Returns: The module instance after updating the submodules. )_update_modules)rrrMs rupdate_moduleszModule.update_moduless( gv... rapply_fnCallable[[str, Module], Any]cd|fg}|rj|\}}||||rd|znd}|t|||j|j|S)aApply a function to all the modules in this instance (including this instance). Args: apply_fn (Callable): The function to apply to the modules which takes two parameters. The first parameter is the string path of the module (e.g. ``"model.layers.0.linear"``). The second parameter is the module object. Returns: The module instance after updating submodules. r!rR)r'r$)rGextendr r)r*)rr module_stackr'mods rapply_to_moduleszModule.apply_to_modulessT |  &**,,KFC HVS ! ! !%+3S6\\F   S\\^^FDNSSS      rc<g|fdS)zReturn a list with all the modules in this instance. Returns: A list of :class:`mlx.nn.Module` instances. c.|Sr6appendr1r modulelists rrz Module.modules..s:+<+.s:+<+rBKeyError)rrVrMr1s r_validate_keyszModule._validate_keyssc!$--9ttD6  J J JD=="#HA#H#H#HIII! rF)recurserVrMrrVOptional[Union[str, List[str]]]c^fd}|r||n |d||S)atFreeze the Module's parameters or some of them. Freezing a parameter means not computing gradients for it. This function is idempotent i.e. freezing a frozen model is a no-op. Example: For instance to only train the attention parameters from a Transformer: .. code-block:: python model = nn.Transformer() model.freeze() model.apply_to_modules(lambda k, v: v.unfreeze() if k.endswith("attention") else None) Args: recurse (bool, optional): If True then freeze the parameters of the submodules as well. Default: ``True``. keys (str or list[str], optional): If provided then only these parameters will be frozen otherwise all the parameters of a module. For instance freeze all biases by calling ``module.freeze(keys="bias")``. strict (bool, optional): If set to ``True`` validate that the passed keys exist. Default: ``False``. Returns: The module instance after freezing the parameters. c}|/t|d}d|D}||}|j|dS)Nc\t|t o||||Sr6)r>r rvrs rrz5Module.freeze.._freeze_impl..s0Z6-B-B)B)>44Q1==rcg|]\}}|Srr)rr1r2s r z7Module.freeze.._freeze_impl..s999FQa999r)r rrrr\rtr local_keysrVrMs r _freeze_implz#Module.freeze.._freeze_implsJ!)$$>> :9j999 ))*f==J J  j ) ) ) ) )rr!r)rrrVrMrs `` rfreezez Module.freezes[F * * * * * *  #  ! !, / / / / LT " " " rc^fd}|r||n |d||S)aUnfreeze the Module's parameters or some of them. This function is idempotent ie unfreezing a model that is not frozen is a noop. Example: For instance to only train the biases of a Transformer one can do: .. code-block:: python model = nn.Transformer() model.freeze() model.unfreeze(keys="bias") Args: recurse (bool, optional): If True then unfreeze the parameters of the submodules as well. Default: ``True``. keys (str or list[str], optional): If provided then only these parameters will be unfrozen otherwise all the parameters of a module. For instance unfreeze all biases by calling ``module.unfreeze(keys="bias")``. strict (bool, optional): If set to ``True`` validate that the passed keys exist. Default: ``False``. Returns: The module instance after unfreezing the parameters. c|jdS|}|j|dSr6)rclearrdifference_updaters r_unfreeze_implz'Module.unfreeze.._unfreeze_impl+sU|   """""--dF;;  ,,Z88888rr!r)rrrVrMrs `` runfreezezModule.unfreezes[H 9 9 9 9 9 9  %  ! !. 1 1 1 1 N2t $ $ $ rmodeNonec||_dSr6rrrs r_set_training_modezModule._set_training_mode9s rc8|fd|S)aSet the model in or out of training mode. Training mode only applies to certain layers. For example :obj:`Dropout` applies a random mask in training mode, but is the identity in evaluation mode. Args: mode (bool): Indicate if the model should be in training or evaluation mode. Default: ``True``. Returns: The module instance after updating the training mode. c.|Sr6)r)rtrrs rrzModule.train..Js1+?+?+E+Errrs `rtrainz Module.train<s( EEEEFFF rc,|dS)zFSet the model to evaluation mode. See :func:`train`. F)rrs revalz Module.evalNs zz%   rc@tj|tjSr6)r? issubdtypefloatingrs rrzModule.XsBM r{E E rdtypemx.Dtype predicate$Optional[Callable[[mx.Dtype], bool]]cFd|fddS)amSet the dtype of the module's parameters. Args: dtype (Dtype): The new dtype. predicate (typing.Callable, optional): A predicate to select parameters to cast. By default, only parameters of type :attr:`floating` will be updated to avoid casting integer parameters to the new dtype. NcdS)NTr)rts rrz"Module.set_dtype..fs$rcR|jr|n|Sr6)rastype)rrrs rrz"Module.set_dtype..hs& !'0B0BIQXXe___r)r)rrrs ``r set_dtypezModule.set_dtypeUs8  &I IIIIIJJJJJr)rr)r4r)r4rr<r)T)rKrLrMrNrr )rfr)NN)ryrzr{r|r}r~)rUrArMrNrr r6)r{rryr~rr )rrArMrNrr )rrrr )rrNrVrrMrNrr )rrNrr)rrNrr )rr )rrrr)(r, __module__ __qualname____doc____annotations__rpropertyrrr"r3r;rFrIrerl staticmethodr*rrrvrxrrUrr)rr\rrrrrrrrrrrr __classcell__)r:s@rr r sZ,,\ X  X    666666       &&&&&TTTTTl"))\)//\/UU\U  \ &*CG ! ! ! ! ! F@@@ DDD    XXX<<<<<BCG...   04 555555t04 000000d$!!!!; ; KKKKKKKKKrr ct|tr|D]}||vr||}||}t|r t|r|||<Pt|ttfrt |||~|r+|ikr%t dt|jd|rt d|ddSt|trtt|D]}||}||}t|r t|r|||<Lt|ttfrt |||z|r+|ikr%t dt|jddS|r%t dt|jddS)NrrRz'Module does not have sub-module named "r) r>rAr r*rBrrZr+r,rrW)rrrMr1rrrs rrrks8'4  N R RACxx #A #AJ ##M22v7G7G 7R7R&CFF d|<<#M9fEEEE R$M$y//2JMMM R !P1!P!P!PQQQ R R R GT " " Ns7||$$ X XAFM I .. X63C3CI3N3N X"AMD$<88 X y&AAAA XIOO !V4 ??;S!V!V!VWWW X X NL4==3ILLLMMMNNrc ^||r Sttr#fdDSttrKi}D]2\}}|d|} || |rt || |ni||<3|Stt rVg} t D]B\} } |d| } | || | rt || | niC| Std)Nc Zi|]'\}}|||t||(Srr)rr1r2ryr}r{r0s rrz_unwrap..sU   1y1%% wuaIvzBB   rrRz1Unexpected leaf found while traversing the module) r>r rTrArrB enumerater RuntimeError) model value_keyr0ryr{r}ndr1r2tknlrvis ```` rrrsz%E**ve}} E6 " "            E4  KKMM  DAq####B9UB**r1iDDD qEE  E4  u%%  EAr####B II9UB++r2y&*EEE      J K KKr) __future__rr-typingrrrrrr mlx.corecorer? mlx.utilsr r rAr rrrrrrs#""""">>>>>>>>>>>>>>>>22222222\ K\ K\ K\ K\ KT\ K\ K\ K~NNN