j9ddlZddlmZmZmZddlmZddlm Z ddl m Z dZ dddddd e d e d e d ed edeeee geeefff dZGdde ZGdde ZGdde ZdS)N)CallableOptionalUnion)Module)tree_map_with_pathc6ddddd}||\}}|p||p|fS)N)@) r )r )r )affinemxfp4nvfp4mxfp8)mode group_sizebits mode_defaultsdefault_group_size default_bitss a/lsinfo/ai/hellotax_ai/base_platform/venv/lib/python3.11/site-packages/mlx/nn/layers/quantized.py_defaults_for_moder sB M (5T':$  ++T-A\ AArF)rquantize_inputclass_predicatemodelrrrrrcpdfd}|}t||tj}||dS)aQuantize the sub-modules of a module according to a predicate. By default all layers that define a ``to_quantized()`` method will be quantized. Both :obj:`Linear` and :obj:`Embedding` layers will be quantized. The module is updated in-place. Note: ``quantize_input=True`` is only supported for ``"nvfp4"`` and ``"mxfp8"`` modes and :obj:`Linear` layers. Args: model (mlx.nn.Module): The model whose leaf modules may be quantized. group_size (Optional[int]): The quantization group size (see :func:`mlx.core.quantize`). Default: ``None``. bits (Optional[int]): The number of bits per parameter (see :func:`mlx.core.quantize`). Default: ``None``. mode (str): The quantization method to use (see :func:`mlx.core.quantize`). Default: ``"affine"``. quantize_input (bool): Whether to quantize activations. Default: ``False``. class_predicate (Optional[Callable]): A callable which receives the :obj:`Module` path and :obj:`Module` itself and returns ``True`` or a dict of params for ``to_quantized`` if it should be quantized and ``False`` otherwise. If ``None``, then all layers that define a ``to_quantized()`` method are quantized. Default: ``None``. Example: Weight only quantization for all layers that define a ``to_quantized()`` method: >>> import mlx.nn as nn >>> nn.quantize(model, group_size=64, bits=4, mode="affine") Weight and input quantization for all linear layers: >>> predicate = lambda p, m: isinstance(m, nn.Linear) >>> nn.quantize(model, mode="nvfp4", quantize_input=True, class_predicate=predicate) c"t|dS)N to_quantized)hasattr)_ms rzquantize..Cswq.7Q7Qrc||x}rt|drt|trd}r|d<|jdi|St|tr.d|vr|ds|d|jdi|St dt dt||S)Nr!)rrrrzZ``class_predicate`` must return a bool or a dict of parameters to pass to ``to_quantized``z!Unable to quantize model of type r)r" isinstanceboolr!dictpop ValueErrortype) pathr$bool_or_paramskwargsrrrrrs r_maybe_quantizez!quantize.._maybe_quantizeEs,_T155 5> q.)) Pnd33,6dSSF%B3A/0)1>33F33355 (N::N(E:'**+;<<<)1>;;N;;;$O !!NT!WW!N!NOOOHr)is_leafN) leaf_modulesrr is_moduleupdate_modules)rrrrrrr0leavess ````` rquantizer6sZ&R*Q*QO0   ! !F AQ R R RF      rc eZdZdZ ddededededef fd Zd Zd Zd Z e dd e dededefdZ xZ S)QuantizedEmbeddinga8The same as :obj:`Embedding` but with a quantized weight matrix. :obj:`QuantizedEmbedding` also provides a :meth:`from_embedding` classmethod to convert embedding layers to :obj:`QuantizedEmbedding` layers. Args: num_embeddings (int): How many possible discrete tokens can we embed. Usually called the vocabulary size. dims (int): The dimensionality of the embeddings. group_size (Optional[int]): The group size to use for the quantized weight. See :func:`~mlx.core.quantize`. Default: ``None``. bits (Optional[int]): The bit width to use for the quantized weight. See :func:`~mlx.core.quantize`. Default: ``None``. mode (str): The quantization method to use (see :func:`mlx.core.quantize`). Default: ``"affine"``. Nrnum_embeddingsdimsrrrctt|||\|_|_||_t jd|z }tj ||f|}tj ||||^|_ |_ }|r|dnd|_||_||_|dS)N)shapescalerr)super__init__rrrrmathsqrtmxrandomnormalr6weightscalesbiasesr9r:freeze) selfr9r:rrrr>rGrI __class__s rrAzQuantizedEmbedding.__init__us &8j$%O%O"  !d(##!!(>e!LL,.K J4- - - ) T[6$*3fQiit ,  rc|d}tj|d||d||||nd|j|j|jS)NrIrGrH)rHrIrrr)getrD dequantizerrr)rKxrIs r__call__zQuantizedEmbedding.__call__sc(##} N1 >!$ & 26!99     rc tj||d|d|dd|j|j|jS)z Call the quantized embedding layer as a quantized linear layer. Use this for example when input embedding and output projection weights are tied. rGrHrITrHrI transposerrrrDquantized_matmulrNrrrrKrPs r as_linearzQuantizedEmbedding.as_linearsP" N>88H%%    rc T|jd|jd|jd|jd|j S)Nz, , group_size=, bits=, mode=)r9r:rrrrKs r _extra_reprzQuantizedEmbedding._extra_reprs\" P Pdi P P/ P P26) P PDHI P P rembedding_layerc|jj\}}||||||}tj|j|||^|_|_}|r|dnd|_|S)zHCreate a :obj:`QuantizedEmbedding` layer from an :obj:`Embedding` layer.r?rN)rGr=rDr6rHrI) clsr_rrrembedding_dimsr:qlrIs rfrom_embeddingz!QuantizedEmbedding.from_embeddingsy /5; Sz4d C C C(*  "   ) ) ) % 29v "(1F1IIT  rNNr)__name__ __module__ __qualname____doc__intstrrArQrXr^ classmethodrrd __classcell__rLs@rr8r8bs ,      6       $       [rr8c eZdZdZ ddedededed ed ef fd Zd Zd Z e dde ded ed efdZ xZ S)QuantizedLinearaIApplies an affine transformation to the input using a quantized weight matrix. It is the quantized equivalent of :class:`mlx.nn.Linear`. For now its parameters are frozen and will not be included in any gradient computation but this will probably change in the future. :obj:`QuantizedLinear` also provides a classmethod :meth:`from_linear` to convert linear layers to :obj:`QuantizedLinear` layers. Args: input_dims (int): The dimensionality of the input features. output_dims (int): The dimensionality of the output features. bias (bool, optional): If set to ``False`` then the layer will not use a bias. Default: ``True``. group_size (Optional[int]): The group size to use for the quantized weight. See :func:`~mlx.core.quantize`. Default: ``None``. bits (Optional[int]): The bit width to use for the quantized weight. See :func:`~mlx.core.quantize`. Default: ``None``. mode (str): The quantization method to use (see :func:`mlx.core.quantize`). Default: ``"affine"``. TNr input_dims output_dimsbiasrrrctt|||\|_|_||_t jd|z }tj | |||f}tj ||||^|_ |_ } | r| dnd|_|rtj|f|_|dS)Nr<lowhighr=r?r)r@rArrrrrBrCrDrEuniformr6rGrHrIzerosrsrJ) rKrqrrrsrrrr>rGrIrLs rrAzQuantizedLinear.__init__s &8j$%O%O"  !j.))"" +#   -/K J4- - - ) T[6$*3fQiit   1+00DI rc |jj\}}|dz|jz}d|d|dd|vd|jd|jd|j S) Nr input_dims=, output_dims=z, bias=rsrZr[r\)rGr=rrrrKout_dimsin_dimss rr^zQuantizedLinear._extra_reprs K-'R88H%%  T>>DL Ar linear_layerc|jj\}}|||d|||}tj|j|||^|_|_}|r|dnd|_d|vr |j|_|S)zACreate a :obj:`QuantizedLinear` layer from a :obj:`Linear` layer.Fr?rNrs)rGr=rDr6rHrIrs) rarrrrrrrqrcrIs r from_linearzQuantizedLinear.from_linears#/"5"; Z S[%T M M M(*     ) ) ) % 29v "(1F1IIT \ ! !"'BG r)TNNrre)rfrgrhrirjr(rkrAr^rQrlrrrmrns@rrprps 4               D          [rrpc eZdZdZ ddededededef fd Zd Zd Zd Z de ffd Z dZ e ddedededefdZxZS)QQLinearaQuantizes the input and applies an affine transformation using quantized weights. Two use cases are supported: 1) **Eval**: The weights are frozen and stored in quantized form together with their scales (``self.weight`` is quantized and ``self.scales`` is provided). 2) **Train**: The weights are stored in higher precision and are quantized on the fly during computation so that gradients with respect to the weights can be computed. To switch between the two cases, use ``layer.eval()`` and ``layer.train()`` respectively. Compared to the :class:`mlx.nn.QuantizedLinear` layer, this layer quantizes the input as well and includes weights in gradient computations. :obj:`QQLinear` also provides the class method :meth:`from_linear` to convert :class:`mlx.nn.Linear` layers to :obj:`QQLinear` layers. Note: This layer does not support a bias term yet. Args: input_dims (int): The dimensionality of the input features. output_dims (int): The dimensionality of the output features. group_size (Optional[int]): The group size to use for the quantized weight. See :func:`~mlx.core.quantize`. Default: ``None``. bits (Optional[int]): The bit width to use for the quantized weight. See :func:`~mlx.core.quantize`. Default: ``None``. mode (Optional[str]): The quantization method to use (see :func:`mlx.core.quantize`). Currently, only ``"nvfp4"`` and ``"mxfp8"`` are supported. Default: ``"nvfp4"``. Nrrqrrrrrc"tt|||\|_|_||_t jd|z }tj | |||f|_ d|_ dS)Nr<ruF) r@rArrrrrBrCrDrErxrG _quantized)rKrqrrrrrr>rLs rrAzQQLinear.__init__Rs &8j$%O%O"  !j.))i'' +(    rc |jj\}}|jjtjkr |dz|jz}d|d|d|jd|jd|j S)Nr r{r|rZr[r\)rGr=dtyperDuint32rrrr}s rr^zQQLinear._extra_reprhs K-' ;  ) )| 1G P' P P P P/ P P26) P PDHI P P rc|jsBtj|j|j|j|j\|_|_d|_dSdS)Nr?T)rrDr6rGrrrrHr]s rr6zQQLinear.quantizeqsV #'){  Y ((( $DK #DOOO # #rc|jrUtj|j|j|j|j|j|_|dd|_dSdS)NrHrrrrHF) rrDrOrGrHrrr __delattr__r]s rrOzQQLinear.dequantize{sh ? $- {?YY DK   X & & &#DOOO $ $rct||jr|dS|dS)N)r@_set_training_mode _trainingrOr6)rKrrLs rrzQQLinear._set_training_modesL ""4((( >  OO      MMOOOOOrctj||d|d|j|j|j}|S)NrGrHr)rDqqmmrNrrrrWs rrQzQQLinear.__call__sG G N88H%%    rrc|jj\}}|dtd||||||}|j|_||j|S)z:Create a :obj:`QQLinear` layer from a :obj:`Linear` layer.rsNz#QQLinear does not support bias yet.r?)rGr=rNNotImplementedErrortraintraining)rarrrrrrrqrcs rrzQQLinear.from_linearsu#/"5"; Z   F # # /%&KLL L S[*d F F F '  &''' r)NNr)rfrgrhrirjrkrAr^r6rOr(rrQrlrrrmrns@rrr1s?H                ,   ### $ $ $t       [rr)NN)rBtypingrrrmlx.corecorerDmlx.nn.layers.baser mlx.utilsrrrjrkr(r)r6r8rprrrrrs ,,,,,,,,,,%%%%%%((((((BBBI!  LPI!I!I! I!I! I!  I!  I!hV}eD$J6G'GHII!I!I!I!XccccccccLfffffffffRyyyyyvyyyyyr