j ddlZddlmZmZddlmZdededefdZdedede defdZ dded e d edefd Z d eedee defdZ ded ede defdZ dS)N)CallableListinit decay_ratereturncfd}|S)aMake an exponential decay scheduler. Args: init (float): Initial value. decay_rate (float): Multiplicative factor to decay by. Example: >>> lr_schedule = optim.exponential_decay(1e-1, 0.9) >>> optimizer = optim.SGD(learning_rate=lr_schedule) >>> optimizer.learning_rate array(0.1, dtype=float32) >>> >>> for _ in range(5): optimizer.update({}, {}) ... >>> optimizer.learning_rate array(0.06561, dtype=float32) c|zzSN)steprrs c/lsinfo/ai/hellotax_ai/base_platform/venv/lib/python3.11/site-packages/mlx/optimizers/schedulers.pyschedulez#exponential_decay..schedulesj$&&&r )rrrs`` r exponential_decayr s)&'''''' Or step_sizecfd}|S)a/Make a step decay scheduler. Args: init (float): Initial value. decay_rate (float): Multiplicative factor to decay by. step_size (int): Decay every ``step_size`` steps. Example: >>> lr_schedule = optim.step_decay(1e-1, 0.9, 10) >>> optimizer = optim.SGD(learning_rate=lr_schedule) >>> optimizer.learning_rate array(0.1, dtype=float32) >>> >>> for _ in range(21): optimizer.update({}, {}) ... >>> optimizer.learning_rate array(0.081, dtype=float32) c|zzzSr r )r rrrs r rzstep_decay..schedule7szdi&7899rr )rrrrs``` r step_decayr"s/*::::::: Or decay_stepsendcfd}|S)aMake a cosine decay scheduler. Args: init (float): Initial value. decay_steps (int): Number of steps to decay over. The decayed value is constant for steps beyond ``decay_steps``. end (float, optional): Final value to decay to. Default: ``0``. Example: >>> lr_schedule = optim.cosine_decay(1e-1, 1000) >>> optimizer = optim.SGD(learning_rate=lr_schedule) >>> optimizer.learning_rate array(0.1, dtype=float32) >>> >>> for _ in range(5): optimizer.update({}, {}) ... >>> optimizer.learning_rate array(0.0999961, dtype=float32) ctj|}ddtjtjz |zzz}|z zzS)Ng?g?)mxminimumcosmathpi)r sdecayrrrs r rzcosine_decay..scheduleSsM Jt[ ) )sRVTW{%:a$?@@@AUdSj)))rr )rrrrs``` r cosine_decayr!=s/,******* Or schedules boundariesctdkrtdttdzkr3tdtdtdz dfd}|S)agJoin multiple schedules to create a new schedule. Args: schedules (list(Callable)): A list of schedules. Schedule :math:`i+1` receives a step count indicating the number of steps since the :math:`i`-th boundary. boundaries (list(int)): A list of integers of length ``len(schedules) - 1`` that indicates when to transition between schedules. Example: >>> linear = optim.linear_schedule(0, 1e-1, steps=10) >>> cosine = optim.cosine_decay(1e-1, 200) >>> lr_schedule = optim.join_schedules([linear, cosine], [10]) >>> optimizer = optim.Adam(learning_rate=lr_schedule) >>> optimizer.learning_rate array(0.0, dtype=float32) >>> for _ in range(12): optimizer.update({}, {}) ... >>> optimizer.learning_rate array(0.0999938, dtype=float32) rz)Must provide at least 1 schedule to join.z Received z boundaries but expected .c d|}tddD]+\}}tj||k||||z },|S)Nrr%)ziprwhere)r outputboundaryrr#r"s r rz join_schedules..schedulezsj1d##"%j)ABB-"@"@ R R HhXdXovxxx7P7PQQFF r)len ValueError)r"r#rs`` r join_schedulesr.[s, 9~~DEEE 9~~Z1,,, .J . .I* . . .    OrstepscLdkrtddfd}|S)aLMake a linear scheduler. Args: init (float): Initial value. end (float): Final value. steps (int): Number of steps to apply the schedule over. The value is ``end`` for any steps beyond ``steps``. Example: >>> lr_schedule = optim.linear_schedule(0, 1e-1, 100) >>> optimizer = optim.Adam(learning_rate=lr_schedule) >>> optimizer.learning_rate array(0.0, dtype=float32) >>> for _ in range(101): optimizer.update({}, {}) ... >>> optimizer.learning_rate array(0.1, dtype=float32) r%z&steps must be greater than 0, but got r&cJtj|}|z z zzSr )rr)r rrr/s r rz!linear_schedule..schedules-z$&&d e+,t33r)r-)rrr/rs``` r linear_scheduler2sT( qyyJ%JJJKKK4444444 Or)r)rtypingrrmlx.corecorerfloatrintrr!r.r2r rr r8s4 !!!!!!!!Eu2U#(6u3UX<%d8n%$s)%%%%%P%eCHr