jN ddlZddlmZddlmZmZmZddlmZ ddl m Z ddl m Z ddlmZddlmZedZd Z d d ed ed ee jjfd ZdZdZdZdddde deeefdeeefd ee jjfdZdddde dedeeefd ee jjfdZGdde Z Gdde Z!Gdde Z"Gdde Z#dS)!N) lru_cache)CallableOptionalUnion)Module)Linear)QuantizedLinear)tree_map_with_pathcdkrdStjd}|jfd}|S)Nc|SNxs c/lsinfo/ai/hellotax_ai/base_platform/venv/lib/python3.11/site-packages/mlx/nn/layers/distributed.pyzsum_gradients..sc|Srrrs rfzsum_gradients..fsrcFtj|S)Ngroup)mx distributedall_sum)rdx_rs rrzsum_gradients..fs~%%b%666r)sizercustom_functionvjp)rrs` r sum_gradientsr"si zz||q{U7777 U7 Hrct|tst|dtrtj|||S|j|fd|D}tj|||S)z:Equivalent to mx.split but allows for fractional segments.raxisc4g|]}t|zSr)int).0sNs r z_split..$s#,,,as1q5zz,,,r) isinstancer'rsplitshape)weightsegmentsr%indicesr*s @r_splitr2s}(C  5Jx{C$@$@5xt4444 TA,,,,8,,,G 8FG$ / / //r parameterssharding_predicaterc|ptj}||fd}t ||S)zReturns a new parameter tree with the weights sharded according to the sharding_predicate. The sharding predicate should return the sharding axis and optionally also the segments that comprise the weight. c tt|tjs|S||}||Sdd}t|tr|n*t|tr|\}nt dtjtjfdt||DS)Nr z;The sharding function should return int or tuple[int, list]c>g|]}t|Sr)r2)r(partr*r%rs rr+z-_shard.._shard_fn..Ls*UUUda&&q)UUUrr$) r,rarrayr'tuple ValueError contiguous concatenater2)pathr/r)r0r%r*r9r4s @r _shard_fnz_shard.._shard_fn7s&"(++ M  tV , , 9M a   DD 5 ! ! ND((M } NUUUUUUfVXt6T6TUUU      r)rrinitrrankr )r3r4rr@r*r9s ` @@r_shardrC(sn  *R^((**E A A       4 i 4 44rcfd}|S)zxSimple predicate to shard fully connected layers such that a common representation becomes a sharded representation.cj|drdfSt|jdz dfS)Nbiasr)endswithmaxndimr?r/r0s rr@z"_all_to_sharded.._shard_fnXs< ==  x< 6;?A&&00rrr0r@s` r_all_to_shardedrNTs$11111 rcfd}|S)zxSimple predicate to shard fully connected layers such that a sharded representation becomes a common representation.c:|drdSdfS)NrFrG)rIrLs rr@z"_sharded_to_all.._shard_fnds& ==  48|rrrMs` r_sharded_to_allrQ`s$ rc4|dvrtd|ddS)N)all-to-shardedsharded-to-allzSharding type sharding=zB not supported, choose one of 'all-to-sharded' or 'sharded-to-all')r<)shardings r_check_shardingrVlsB;;;EEEE   <;rr r0rmodulerUr0ct|tr3t||dkrt|nt |}|t |||dS)aDShard a module in-place by updating its parameter dictionary with the sharded parameter dictionary. The ``sharding`` argument can be any callable that given the path and the weight returns the sharding axis and optionally also the segments that comprise the unsharded weight. For instance if the weight is a fused QKV matrix the segments should be 3. .. note:: The module doesn't change so in order for distributed communication to happen the module needs to natively support it and for it to be enabled. Args: module (mlx.nn.Module): The parameters of this module will be sharded in-place. sharding (str or callable): One of "all-to-sharded" and "sharded-to-all" or a callable that returns the sharding axis and segments. segments (int or list): The segments to use if ``sharding`` is a string. Default: ``1``. group (mlx.core.distributed.Group): The distributed group to shard across. If not set, the global group will be used. Default: ``None``. rSN)r,strrVrNrQupdaterCr3)rXrUr0rs r shard_inplacer\vs<(C   !!!+++ H % % % **   MM&**,,h>>?????rct|tjtjt jt jd}||t|tf|||S)aCreate a new linear layer that has its parameters sharded and also performs distributed communication either in the forward or backward pass. .. note:: Contrary to ``shard_inplace``, the original layer is not changed but a new layer is returned. Args: module (mlx.nn.Module): The linear layer to be sharded. sharding (str): One of "all-to-sharded" and "sharded-to-all" that defines the type of sharding to perform. segments (int or list): The segments to use. Default: ``1``. group (mlx.core.distributed.Group): The distributed group to shard across. If not set, the global group will be used. Default: ``None``. ))rST)rSF)rTT)rTFrW) rVAllToShardedLinear from_linearQuantizedAllToShardedLinearfrom_quantized_linearShardedToAllLinearQuantizedShardedToAllLinearr,r)rXrUr0rfnss r shard_linearresl.H"4"@#>#T"4"@#>#T   C 53xFF333 4   rc eZdZdZ ddedededeejj ffd Z d e fd Z d ej d ej fd Zed dddedeeefdeejj fdZxZS)r^aEach member of the group applies part of the affine transformation such that the result is sharded across the group. The gradients are automatically aggregated from each member of the group. Args: input_dims (int): The dimensionality of the input features output_dims (int): The dimensionality of the output features bias (bool, optional): If set to ``False`` the the layer will not use a bias. Default is ``True``. group (mx.distributed.Group, optional): The sharding will happen across this group. If not set then the global group is used. Default is ``None``. TN input_dims output_dimsrFrcttjd|z }|ptj|_|j}||zdkrtd|d|dtj | |||z|f|_ |r.tj | |||zf|_ dSdS)N?r Cannot shard the output of size  across devices.lowhighr.super__init__mathsqrtrrrArrr<randomuniformr/rFselfrgrhrFrscaler* __class__s rrszAllToShardedLinear.__init__s  # *++3bn1133 JOO   !O ! !T;TTTTT i''!#Z0(     ))F"a')*DIII  rreturncv|jj\}}|j}||z}d|d|dd|vSN input_dims=, output_dims=, bias=rF)r/r.rr)ryout_dimsin_dimsr*s r _extra_reprzAllToShardedLinear._extra_reprsM K-' JOO  A UWUUHUUVt^UUUrrct|j|}d|vr(tj|d||dj}n||djz}|S)NrFr/)r"rraddmmTryrs r__call__zAllToShardedLinear.__call__s\ %M$* % %a ( ( T>>fq$x.*:;;AADN$$Arr rW linear_layerr0c"|ptj}|jj\}}|||t |d|}|t|t|||SNrF) rrrAr/r.hasattrr[rCr3rNclsrr0rrhrgsls rr_zAllToShardedLinear.from_linear.,,.."."5"; Z S[',*G*G O O &0022OH4M4MuUUVVV rTN__name__ __module__ __qualname____doc__r'boolrrrGrouprsrZrr:r classmethodrrlistr_ __classcell__r{s@rr^r^s+  &04   ,- >VSVVVV "( rx     &'04    T "  ,-   [     rr^c eZdZdZ ddedededeejj ffd Z d e fd Z d ej d ej fd Zed dddedeeefdeejj fdZxZS)rbaEach member of the group applies part of the affine transformation and then aggregates the results. All nodes will have the same exact result after this layer. :class:`ShardedToAllLinear` provides a classmethod :meth:`from_linear` to convert linear layers to sharded :obj:`ShardedToAllLinear` layers. Args: input_dims (int): The dimensionality of the input features output_dims (int): The dimensionality of the output features bias (bool, optional): If set to ``False`` the the layer will not use a bias. Default is ``True``. group (mx.distributed.Group, optional): The sharding will happen across this group. If not set then the global group is used. Default is ``None``. TNrgrhrFrcttjd|z }|ptj|_|j}||zdkrtd|d|dtj | ||||zf|_ |r+tj | ||f|_ dSdS)NrjrThe input of size  cannot be sharded across rmrnrqrxs rrszShardedToAllLinear.__init__%s   # *++3bn1133 JOO   Nq WZWW1WWW i'' a0(     ))F"n*DIII  rr|cv|j}|jj\}}||z}d|d|dd|vSr~)rrr/r.)ryr*rrs rrzShardedToAllLinear._extra_reprDsM JOO   K-'1 UWUUHUUVt^UUUrrc||djz}tj||j}d|vr ||dz}|S)Nr/rrF)rrrrrrs rrzShardedToAllLinear.__call__JsK X  N " "1DJ " 7 7 T>>DL Arr rWrr0c"|ptj}|jj\}}|||t |d|}|t|t|||Sr) rrrAr/r.rr[rCr3rQrs rr_zShardedToAllLinear.from_linearTrrrrrs@rrbrbs+,04   ,- >VSVVVV "(rx &'04    T "  ,-   [     rrbceZdZdZ ddeded ed ed ed ed eej j ffd Z fdZ defdZ dejdejfdZeddddedeeefd eej j fdZxZS)r`aEach member of the group applies part of the affine transformation with a quantized matrix such that the result is sharded across the group. It is the quantized equivalent of :class:`mlx.nn.AllToShardedLinear`. Similar to :class:`mlx.nn.QuantizedLinear` its parameters are frozen and will not be included in any gradient computation. Args: input_dims (int): The dimensionality of the input features. output_dims (int): The dimensionality of the output features. bias (bool, optional): If set to ``False`` then the layer will not use a bias. Default: ``True``. group_size (int, optional): The group size to use for the quantized weight. See :func:`~mlx.core.quantize`. Default: ``64``. bits (int, optional): The bit width to use for the quantized weight. See :func:`~mlx.core.quantize`. Default: ``4``. mode (str, optional): The quantization method to use (see :func:`~mlx.core.quantize`). Default: ``"affine"``. group (mx.distributed.Group, optional): The sharding will happen across this group. If not set then the global group is used. Default is ``None``. T@affineNrgrhrF group_sizebitsmoderc|t||_||_||_t jd|z }|ptj |_ |j } || zdkrtd|d| dtj | ||| z|f} tj| |||^|_|_} | r| dnd|_|rtj|| zf|_|dS)NrjrrkrlrmrnrrrrsrrrrtrurrrArrr<rvrwquantizer/scalesbiaseszerosrFfreeze ryrgrhrFrrrrrzr*r/rr{s rrsz$QuantizedAllToShardedLinear.__init__}sU %   # *++3bn1133 JOO   !O ! !T;TTTTT ""!#Z0#   -/K J4- - - ) T[6$*3fQiit   6+"2!455DI rcftj|i||ddSzlWrap unfreeze so that we unfreeze any layers we might contain but our parameters will remain frozen.F)recurseNrrunfreezerryargskwargsr{s rrz$QuantizedAllToShardedLinear.unfreeze; $)&))) E """""rr|c |jj\}}|dz|jz}||jz}d|d|dd|vd|jd|jd|j S N rrrrFz , group_size=z, bits=z, mode=r/r.rrrrrryrrs rrz'QuantizedAllToShardedLinear._extra_reprs K-'R88H%%  T>>DL Arr rWquantized_linear_layerr0c v|ptj}|jj\}}|dz|jz}|||t |d|j|jt|dd|}| t| t|||SNrrFrr)rrrr) rrrAr/r.rrrgetattrr[rCr3rNrrr0rrhrgrs rraz1QuantizedAllToShardedLinear.from_quantized_linear.,,.."8"?"E Z 2o*@*EE S   *F 3 3-8',/BB    &1133))      rTrrrNrrrrr'rrZrrrrrsrrr:rrrrrrarrs@rr`r`esg604**** *  *  **,-******X#####  S    "(rx$ &'04  &T "  ,- [rr`ceZdZdZ ddeded ed ed ed ed eej j ffd Z fdZ defdZ dejdejfdZeddddedeeefd eej j fdZxZS)rcaEach member of the group applies part of the affine transformation using the quantized matrix and then aggregates the results. All nodes will have the same exact result after this layer. It is the quantized equivalent of :class:`mlx.nn.ShardedToAllLinear`. Similar to :class:`mlx.nn.QuantizedLinear` its parameters are frozen and will not be included in any gradient computation. Args: input_dims (int): The dimensionality of the input features. output_dims (int): The dimensionality of the output features. bias (bool, optional): If set to ``False`` then the layer will not use a bias. Default: ``True``. group_size (int, optional): The group size to use for the quantized weight. See :func:`~mlx.core.quantize`. Default: ``64``. bits (int, optional): The bit width to use for the quantized weight. See :func:`~mlx.core.quantize`. Default: ``4``. mode (str, optional): The quantization method to use (see :func:`~mlx.core.quantize`). Default: ``"affine"``. group (mx.distributed.Group, optional): The sharding will happen across this group. If not set then the global group is used. Default is ``None``. TrrrNrgrhrFrrrrcvt||_||_||_t jd|z }|ptj |_ |j } || zdkrtd|d| dtj | |||| zf} tj| |||^|_|_} | r| dnd|_|rtj|f|_|dS)Nrjrrrrmrnrrrs rrsz$QuantizedShardedToAllLinear.__init__sN %   # *++3bn1133 JOO   Nq WZWW1WWW "" a0#   -/K J4- - - ) T[6$*3fQiit   1+00DI rcftj|i||ddSrrrs rrz$QuantizedShardedToAllLinear.unfreeze0rrr|c |jj\}}|dz|jz|jz}d|d|dd|vd|jd|jd|j Srrrs rrz'QuantizedShardedToAllLinear._extra_repr6s K-'Rs   N>88H%%  N " "1DJ " 7 7 T>>DL Arr rWrr0c v|ptj}|jj\}}|dz|jz}|||t |d|j|jt|dd|}| t| t|||Sr) rrrAr/r.rrrrr[rCr3rQrs rraz1QuantizedShardedToAllLinear.from_quantized_linearNrrrrrs@rrcrcsg:04**** *  *  **,-******X#####  S    "(rx  &'04  &T "  ,- [rrcr)$rt functoolsrtypingrrrmlx.corecorermlx.nn.layers.basermlx.nn.layers.linearrmlx.nn.layers.quantizedr mlx.utilsr r"r2dictrrrCrNrQrVrZr'rr\rer^rbr`rcrrrrs ,,,,,,,,,,%%%%%%''''''333333((((((       000-1)5)5)5 )5 BN( ))5)5)5)5X         "#,0 %@%@%@ %@CM"%@CI %@ BN( ) %@%@%@%@X"#,0      CI  BN( )    FNNNNNNNNbPPPPPPPPfBBBBB&BBBJAAAAA&AAAAAr