)j4ddlZddlmZddlmZmZmZmZddlm Z dddddddgfde de de d e d e d e d e d ee dee j ge j ffdZ d&deee e fdee dee dee dee dee dee fdZee je jje jjde j d e de j fdZee je jje jj d'de j de d e de j fdZee je jje jjde j de de j fdZee je jje jjde j d e d e d ee de j f dZee je jje jjd Zd(d!e d"e fd#Zd(d!e d"e fd$Zd(d!e d"e fd%ZdS))N)partial)CallableDictListOptionaltemptop_pmin_pmin_tokens_to_keeptop_kxtc_probability xtc_thresholdxtc_special_tokensreturnc< dkrdSg dkrdkr fddkr fddkr fddkr fd fd }|S) an Make a sampler function for use with ``generate_step``. Args: temp (float): The temperature for sampling, if 0 the argmax is used. Default: ``0``. top_p (float, optional): Nulceus sampling, higher means model considers more less likely words. min_p (float, optional): The minimum value (scaled by the top token's probability) that a token probability must have to be considered. min_tokens_to_keep (int, optional): Minimum number of tokens that cannot be filtered by min_p sampling. top_k (int, optional): The top k tokens ranked by probability to constrain the sampling to. xtc_probability (float, optional): The probability of applying XTC sampling. xtc_threshold (float, optional): The threshold the probs need to reach for being sampled. xtc_special_tokens (list(int), optional): List of special tokens IDs to be excluded from XTC sampling. Returns: Callable[mx.array, mx.array]: A sampler which takes log-probabilities and returns tokens. rc.tj|dS)Naxis)mxargmax)xs ]/lsinfo/ai/hellotax_ai/base_platform/venv/lib/python3.11/site-packages/mlx_lm/sample_utils.pyzmake_sampler../s12...?c$t|SN) apply_top_p)rr s rrzmake_sampler..4+a*?*?rrc&t|Sr ) apply_min_p)rr r s rrzmake_sampler..6s+a@R*S*Src(t|Sr ) apply_xtc)rrrrs rrzmake_sampler..9si?MCUVVrc$t|Sr ) apply_top_k)rrs rrzmake_sampler..<r"rcDD] }||}t|Sr )categorical_sampling)logprobsmethodsampling_methodsr s rsamplerzmake_sampler..sampler?s3& ( (Fvh''HH#Hd333r)append) r r r r rrrrr.r-s ```````` @r make_samplerr0 sH qyy... qyyUS[[ ? ? ? ?@@@ || S S S S STTT V V V V V V    qyy ? ? ? ?@@@444444 Nr logit_biasrepetition_penaltyrepetition_context_sizepresence_penaltypresence_context_sizefrequency_penaltyfrequency_context_sizec g}|rtjt| tjt| fd}||t ||ft||ft||fg} | D]-\} } } | %| dkr|| | | .|S)a Make logits processors for use with ``generate_step``. Args: repetition_penalty (float, optional): A (sign-aware) multiplicative penalty for repeating tokens. repetition_context_size (int, optional): The number of tokens to consider for repetition penalty. Default: ``20``. presence_penalty (float, optional): An additive penalty to reduce repeating tokens. presence_context_size (int, optional): The number of tokens to consider for the presence penalty. Default: ``20``. frequency_penalty (float, optional): An additive penalty to reduce repeating tokens. The tokens are penalized proportionally to their frequency. frequency_context_size (int, optional): The number of tokens to consider for the frequency penalty. Default: ``20``. logit_bias (dictionary, optional): Additive logit bias. Returns: List[Callable[[mx.array, mx.array], mx.array]]: A list of logits processors. Each processor in the list is a callable which takes an array of tokens and an array of logits and returns the updated logits. cL|jddfSr )atadd)_logitsindicesvaluess rlogit_bias_processorz4make_logits_processors..logit_bias_processoros&9QQQZ(,,V44 4rNr) rarraylistkeysr@r/make_repetition_penaltymake_presence_penaltymake_frequency_penalty)r2r3r4r5r6r7r8logits_processorsrArepetition_penalties make_penaltypenalty context_sizer?r@s @@rmake_logits_processorsrMHs D7(4  1 12233$z00223344 5 5 5 5 5 5   !5666 !"46MN  02GH !24JK 0DJJ+ g|  7a<<  $ $\\'<%H%H I I I r)inputsoutputsr+c Z|jd}t|trd|cxkr|ksntd|d|dt j| |dz dd|d f}t j||t jtd  |j d }|S) z Sample from only the top K tokens ranked by probability. Args: logprobs: A vector of log probabilities. top_k (int): Top k tokens to sample from. rrz(`top_k` has to be an integer in the (0, z] interval, but is .r )kthr.Ninfr) shape isinstanceint ValueErrorr argpartitionput_along_axisrBfloatdtype)r+r vocab_sizemask_idxmasked_logprobss rr(r(s#J eS ! ! !e*@*@*@*@j*@*@*@*@ z        yeaibAAA#uvv+NH'(BHeEll]HNCC"O rcxd|cxkrdksntd|t|tr|dkrtd|tj| d}tj||d}|ddddf}|t j|z}||k}d |d d|f<tj|td  |}tj tj ||tj |j d|j d} tj|| d} | S) ag Apply min-p sampling to the logprobs. Min-p keeps all tokens that are above a minimum probability, scaled by the probability of the most likely token. As a result, the filter is more aggressive given a very high-probability token. Args: logprobs: A vector of log probabilities. min_p (float): Minimum token probability. Typical values are in the 0.01-0.2 range, comparably selective as setting `top_p` in the 0.99-0.8 range. min_tokens_to_keep (int, optional): Minimum number of tokens that cannot be filtered. Default: ``1``. rrz9`min_p` has to be a float in the [0, 1] interval, but is r z:`min_tokens_to_keep` has to be a positive integer, but is rrNF.rSr[)rWrUrVrargsorttake_along_axismathlogwhererZrY zeros_likearangerTr[) r+r r sorted_indicessorted_logprobs top_logprobs scaled_min_ptokens_to_removeselected_logprobsinverse_indicesoriginal_order_logprobss rr$r$s,     #     O O O    (# . . 3E3I3I ]I[ ] ]   Z 333N(>KKKO#111ac6*L $(5//1L'516S----.!1E%LL=/RR' n%% .&r*.2FGGG  O!0? #"rctj|}tj|d}tj||d}tj|d}tjtj||tj|jd|j d}tj||d}tj |d|z k|td S)z Apply top-p (nucleus) sampling to logits. Args: logprobs: A vector of log probabilities. top_p: The cumulative probability threshold for top-p filtering. Returns: token selected based on the top-p criterion. rrr`r rS) rexprarbcumsumrYrfrgrTr[rerZ)r+r probsrh sorted_probscumulative_probsrns rr!r!s F8  EZr222N%e^"EEELyB777' n%% .&r*.2FGGG  O )*:ORTUUU 81u9$ u   rr>c d|cxkrdksntd|d|cxkrdksntd|tj|d}|tj||k|tjk}|rd|d|f<tjtjdd |k|tj|tj |S) a} Apply XTC sampling to the logits. Args: logits: The logits from the model's output. xtc_probability (float): Probability of XTC sampling to happen for each token xtc_threshold (float): The threshold the probs need to reach for being sampled. special_tokens_ids (list(int)): List of special tokens IDs to be excluded from XTC sampling. rg?z?`threshold` has to be a float in the [0, 0.5] interval, but is rz?`probability` has to be a float in the [0, 1] interval, but is rF.r )rWrsoftmaxrerSminrandomuniform)r>rrrrsmasks rr&r&s  % % % %# % % % % ]m ] ]     ' ' ' 'C ' ' ' ' _o _ _    Jvr " "E 28EM15"&AAEEGG GD.(-S$ $% 8 !Q/1 w''  rcLtj|d|z zS)Nr )rry categorical)r>r s rr*r*$s! 9 1t8!4 5 55rrKrLc~dkstttfstdfd}|S)ax Make repetition penalty processor. Paper: https://arxiv.org/abs/1909.05858 Args: penalty (float): The repetition penalty factor to be applied. context_size (int): The number of previous tokens to use. Default: ``20``. Returns: Callable[[mx.array, List[int]], mx.array]: The repetition penalty processor. rz*penalty must be a non-negative float, got ct|dkr@| d}|dd|f}tj|dk|z|z }||dd|f<|SNr)lenrre)tokensr>selected_logitsrLrKs rrepetition_penalty_processorz=make_repetition_penalty..repetition_penalty_processor;sv v;;??\MNN+F$QQQY/O h!#')')O !0F111f9  r)rUrVrZrW)rKrLrs`` rrErE)s^{{*WsEl;;{OgOOPPP       ('rcfd}|S)a Make a presence penalty processor. Corresponds to the OpenAI option with the same name. Namely, subtracts ``penalty`` from a logit if the token has occured at least once in the ``context_size`` previous tokens. Args: penalty (float): The presence penalty to be applied. context_size (int): The number of previous tokens to use. Default: ``20``. Returns: Callable[[mx.array, List[int]], mx.array] clt|dkr| d}|dd|fxxzcc<|Sr)rrr>rLrKs rpresence_penalty_processorz9make_presence_penalty..presence_penalty_processor[sK v;;??\MNN+F 111f9    (    r)rKrLrs`` rrFrFJs*" &%rcfd}|S)aL Make a frequency penalty processor. Corresponds to the OpenAI option with the same name. Namely, subtracts ``penalty`` from a logit for every time that the token has occured in the ``context_size`` previous tokens. The difference with the presence penalty is that the more often a token occurs the more it will be penalized. Args: penalty (float): The frequency penalty to be applied. context_size (int): The number of previous tokens to use. Default: ``20``. Returns: Callable[[mx.array, List[int]], mx.array] ct|dkr/| d}|jdd|f}|Sr)rr;subtractrs rfrequency_penalty_processorz;make_frequency_penalty..frequency_penalty_processorxsI v;;??\MNN+FYqqq&y)227;;F rr)rKrLrs`` rrGrGds*( '&r)NNr1Nr1Nr1)r )r1)rc functoolsrtypingrrrrmlx.corecorerrZrVrBr0rMcompilerystater(r$r!r&r*rErFrGrrrrs 111111111111 $&;; ; ; ; ;  ;  ;;S ;rxj"("#;;;;~.2*.-/(,+-)-,.66c5j)*6 6&c]6uo 6 $C= 6   6%SM6666r BIORY_EEEh XFE0 BIORY_EEE =#=#h=# =#=#X =#=#=#FE=#@ BIORY_EEE "( 5 RX   FE F BIORY_EEE! H!!!S ! X !!!FE!H BIORY_EEE66FE6((U(#((((B&&5&&&&&4''E'''''''r