jpJUddlmZddlZddlZddlmZddlmZmZmZm Z ddl m Z m Z m Z ddlZddlmZerddlmZdd lmZgd Ze d Ze d Zeejd sHedejjd<edejjd<edejjd<ddlmZmZmZd0dZd1dZGddeZ GddZ!e dede"ffZ#de$d<e d2d3d)Z%e d2d4d,Z% d2d5d/Z%dS)6) annotationsN)Callable)overload TYPE_CHECKING TypeAliasUnion) ParamSpecSelfTypeVar)Tensor) _POOL_HANDLE) _dummy_type)is_current_stream_capturinggraph_pool_handle CUDAGraphgraphmake_graphed_callables_R_P_CudaStreamBase _CUDAGraph_graph_pool_handle_cuda_isCurrentStreamCapturing)rrrreturnboolctS)zReturn True if CUDA graph capture is underway on the current CUDA stream, False otherwise. If a CUDA context does not exist on the current device, returns False without initializing the context. )r[/lsinfo/ai/hellotax_ai/base_platform/venv/lib/python3.11/site-packages/torch/cuda/graphs.pyrr-s * + ++rr cXtjtS)zReturn an opaque token representing the id of a graph memory pool. See :ref:`Graph memory management`. .. warning:: This API is in beta and may change in future releases. )torchcudar rrrr rr6s! : " "#5#7#7 8 88rceZdZdZddfd Z ddfd Zd fd Zd fd Zd fd Zd fd Z d!fd Z d fd Z d"fd Z d#fd Z d#fd ZxZS)$ra-Wrapper around a CUDA graph. Arguments: keep_graph (bool, optional): If ``keep_graph=False``, the cudaGraphExec_t will be instantiated on GPU at the end of ``capture_end`` and the underlying cudaGraph_t will be destroyed. Users who want to query or otherwise modify the underlying cudaGraph_t before instantiation can set ``keep_graph=True`` and access it via ``raw_cuda_graph`` after ``capture_end``. Note that the cudaGraphExec_t will not be instantiated at the end of ``capture_end`` in this case. Instead, it will be instantiated via an explicit called to ``instantiate`` or automatically on the first call to ``replay`` if ``instantiate`` was not already called. Calling ``instantiate`` manually before ``replay`` is recommended to prevent increased latency on the first call to ``replay``. It is allowed to modify the raw cudaGraph_t after first calling ``instantiate``, but the user must call ``instantiate`` again manually to make sure the instantiated graph has these changes. Pytorch has no means of tracking these changes. .. warning:: This API is in beta and may change in future releases. F keep_graphrrr cHt||SN)super__new__)clsr% __class__s r r)zCUDAGraph.__new__]swwsJ///rNglobalpool_POOL_HANDLE | Nonecapture_error_modestrNonecNt||dS)aBegin capturing CUDA work on the current stream. Typically, you shouldn't call ``capture_begin`` yourself. Use :class:`~torch.cuda.graph` or :func:`~torch.cuda.make_graphed_callables`, which call ``capture_begin`` internally. Arguments: pool (optional): Token (returned by :func:`~torch.cuda.graph_pool_handle` or :meth:`other_Graph_instance.pool()`) that hints this graph may share memory with the indicated pool. See :ref:`Graph memory management`. capture_error_mode (str, optional): specifies the cudaStreamCaptureMode for the graph capture stream. Can be "global", "thread_local" or "relaxed". During cuda graph capture, some actions, such as cudaMalloc, may be unsafe. "global" will error on actions in other threads, "thread_local" will only error for actions in the current thread, and "relaxed" will not error on these actions. Do NOT change this setting unless you're familiar with `cudaStreamCaptureMode `_ )r-r/N)r( capture_begin)selfr-r/r+s r r3zCUDAGraph.capture_begin`s)& 4`_ and `cuda-python Graph Management bindings `_ )r(raw_cuda_graphr7s r rFzCUDAGraph.raw_cuda_graphs ww%%'''rcDtS)aReturns the underlying cudaGraphExec_t. ``instantiate`` must have been called if ``keep_graph`` is True, or ``capture_end`` must have been called if ``keep_graph`` is False. If you call ``instantiate()`` after ``raw_cuda_graph_exec()``, the previously returned cudaGraphExec_t will be destroyed. It is your responsibility not to use this object after destruction. See the following for APIs for how to manipulate this object: `Graph Execution `_ and `cuda-python Graph Execution bindings `_ )r(raw_cuda_graph_execr7s r rHzCUDAGraph.raw_cuda_graph_execs ww**,,,r)F)r%rrr )Nr,)r-r.r/r0rr1rr1rr )rAr0rr1)rrD)__name__ __module__ __qualname____doc__r)r3r6r9r;r=r-r@rCrFrH __classcell__)r+s@r rrBs40000000KSPPPPPPP*      ++++++......((((((----------rrcBeZdZUdZdZded< ddd ZddZddZdS)raContext-manager that captures CUDA work into a :class:`torch.cuda.CUDAGraph` object for later replay. See :ref:`CUDA Graphs ` for a general introduction, detailed use, and constraints. Arguments: cuda_graph (torch.cuda.CUDAGraph): Graph object used for capture. pool (optional): Opaque token (returned by a call to :func:`~torch.cuda.graph_pool_handle()` or :meth:`other_Graph_instance.pool()`) hinting this graph's capture may share memory from the specified pool. See :ref:`Graph memory management`. stream (torch.cuda.Stream, optional): If supplied, will be set as the current stream in the context. If not supplied, ``graph`` sets its own internal side stream as the current stream in the context. capture_error_mode (str, optional): specifies the cudaStreamCaptureMode for the graph capture stream. Can be "global", "thread_local" or "relaxed". During cuda graph capture, some actions, such as cudaMalloc, may be unsafe. "global" will error on actions in other threads, "thread_local" will only error for actions in the current thread, and "relaxed" will not error on actions. Do NOT change this setting unless you're familiar with `cudaStreamCaptureMode `_ .. note:: For effective memory sharing, if you pass a ``pool`` used by a previous capture and the previous capture used an explicit ``stream`` argument, you should pass the same ``stream`` argument to this capture. .. warning:: This API is in beta and may change in future releases. .. _cudaStreamCaptureMode: https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__STREAM.html#group__CUDART__STREAM_1g9d0535d93a214cbf126835257b16ba85 Ntorch.cuda.Stream | Nonedefault_capture_streamr, cuda_graphrr-r.streamr/r0cN|4|jj(tj|j_|dn|f|_||n |jj|_|jtdtj|j|_ ||_ ||_ dS)Nrzcapture_stream must not be None) r+rRr"r#Streamr-capture_streamAssertionErrorrT stream_ctxrSr/)r4rSr-rTr/s r __init__zgraph.__init__s >dnCK49J4E4E4G4GDN 1;? 3  JLLL     !!### !!###%% Y  $6      rargsobjectcV|j|jj|dSr')rSr6rY__exit__)r4rfs r rizgraph.__exit__s. ##%%%  $''''r)NNr,)rSrr-r.rTrQr/r0rI)rfrgrr1) rKrLrMrNrR__annotations__rZrerirrr rrs:8<;;;; %)+/"* 55555.    6((((((rrtorch.nn.Module.r_ModuleOrCallableF callables sample_argstuple[Tensor, ...]num_warmup_itersrDallow_unused_inputr-r.cdSr'rrnrorqrrr-s r rrs rtuple[_ModuleOrCallable, ...]tuple[tuple[Tensor, ...], ...]cdSr'rrts r rrs %(Cr1_ModuleOrCallable | tuple[_ModuleOrCallable, ...]3tuple[Tensor, ...] | tuple[tuple[Tensor, ...], ...]c )*tjr"tjrtdd}t |t s.d}|f}t jt tdf|f}n4t jt t tdfdf|}g)t||D]\}}t |tj j rt|j dkr0t|jdkrt|jdkst!dt#d|Dst!dtjjj|} )t | t#d | Dst!d d )D} d |D*)*fd t/t|D} dt/t|D} dt/t|D} |t1n|}tjtjtj5t||| D]\}}}d\}}}t/|D]}tjj||}t d|D}t|dkrRtj|t d|Dt d|Dd|}|||fD]}~ dddn #1swxYwYtjg}g}t||| D]\}}}tj ||5||}dddn #1swxYwYtjj!|\}}|t |||g}g}ttE| tE|tE| D]Z\}}}t d|D} t d|D}d}t|dkrtj ||5tj|t d|Dt d| Dd|}dddn #1swxYwYg}!d}"|D]A}#|#j#r#|!|!||"|"dz }",|!dBt |!}!|| ||!\|$|$d6d-}$g}%tK|D]\}&}|$| |&| |&*|&| |&||&| |&||&||&||& }'t |tj j r7d7d5}(|(||j&|'|j'|_'|%||%|'|r|%dSt |%S)8aAccept callables (functions or :class:`nn.Module`\ s) and returns graphed versions. Each graphed callable's forward pass runs its source callable's forward CUDA work as a CUDA graph inside a single autograd node. The graphed callable's forward pass also appends a backward node to the autograd graph. During backward, this node runs the callable's backward work as a CUDA graph. Therefore, each graphed callable should be a drop-in replacement for its source callable in an autograd-enabled training loop. See :ref:`Partial-network capture` for detailed use and constraints. If you pass a tuple of several callables, their captures will use the same memory pool. See :ref:`Graph memory management` for when this is appropriate. Arguments: callables (torch.nn.Module or Python function, or tuple of these): Callable or callables to graph. See :ref:`Graph memory management` for when passing a tuple of callables is appropriate. If you pass a tuple of callables, their order in the tuple must be the same order they'll run in the live workload. sample_args (tuple of Tensors, or tuple of tuples of Tensors): Samples args for each callable. If a single callable was passed, ``sample_args`` must be a single tuple of argument Tensors. If a tuple of callables was passed, ``sample_args`` must be tuple of tuples of argument Tensors. num_warmup_iters (int): The number of warmup iterations. Currently, ``DataDistributedParallel`` needs 11 iterations for warm up. Default: ``3``. allow_unused_input (bool): If False, specifying inputs that were not used when computing outputs (and therefore their grad is always zero) is an error. Defaults to False. pool (optional): Token (returned by :func:`~torch.cuda.graph_pool_handle` or :meth:`other_Graph_instance.pool()`) that hints this graph may share memory with the indicated pool. See :ref:`Graph memory management`. .. note:: The ``requires_grad`` state of each Tensor in ``sample_args`` must match the state that's expected for the corresponding real input in the training loop. .. warning:: This API is in beta and may change in future releases. .. warning:: ``sample_args`` for each callable must contain only Tensors. Other types are not allowed. .. warning:: Returned callables do not support higher order differentiation (e.g., double backward). .. warning:: In any :class:`~torch.nn.Module` passed to :func:`~make_graphed_callables`, only parameters may be trainable. Buffers must have ``requires_grad=False``. .. warning:: After you pass a :class:`torch.nn.Module` through :func:`~make_graphed_callables`, you may not add or remove any of that Module's parameters or buffers. .. warning:: :class:`torch.nn.Module`\s passed to :func:`~torch.cuda.make_graphed_callables` must not have module hooks registered on them at the time they are passed. However, registering hooks on modules *after* passing them through :func:`~torch.cuda.make_graphed_callables` is allowed. .. warning:: When running a graphed callable, you must pass its arguments in the same order and format they appeared in that callable's ``sample_args``. .. warning:: The automatic mixed precision is supported in :func:`~torch.cuda.make_graphed_callables` only with disabled caching. The context manager `torch.cuda.amp.autocast()` must have `cache_enabled=False`. z_make_graphed_callables does not support the autocast caching. Please set `cache_enabled=False`.FT.rzModules must not have hooks registered at the time they are passed. However, registering hooks on modules after passing them through make_graphed_callables is allowed.c3(K|] }|jduVdS)FN requires_grad.0bs r z)make_graphed_callables..s)EEAq%/EEEEEErzIn any :class:`~torch.nn.Module` passed to :func:`~make_graphed_callables`, only parameters may be trainable. All buffers must have ``requires_grad=False``.c3JK|]}t|tjVdSr') isinstancer"r )rargs r rz)make_graphed_callables..s.HHS:c5<00HHHHHHrzfIn the beta API, sample_args for each callable must contain only Tensors. Other types are not allowed.c,g|]}t|Sr)len)rrfs r z*make_graphed_callables..s!L!L!L#d))!L!L!Lrcg|]D}t|tjjr!t |ndES)r)rr"nnModuletuple parameters)rcs r rz*make_graphed_callables..sP""" ",Aux!?!?GallnnR"""rc2g|]}||zSrr)riflatten_sample_argsper_callable_module_paramss r rz*make_graphed_callables..s9***  A!;A!>>***rcJg|] }tj!Srr"r#rr_s r rz*make_graphed_callables..&HHHQ%*&&((HHHrcJg|] }tj!Srrrs r rz*make_graphed_callables..rrN)NNNc3(K|] }|j |VdSr'r|ros r rz)make_graphed_callables..s)$K$K11?$KQ$K$K$K$K$K$Krc3(K|] }|j |VdSr'r|rrs r rz)make_graphed_callables..s=%%"#q%%%%%%%rc3LK|]}|j tj|V dSr'r}r" empty_likers r rz)make_graphed_callables..sH++45AO+!,Q//++++++r)outputsinputs grad_outputs only_inputs allow_unused)r-c3PK|]!}|jrtj|ndV"dSr'rrs r rz)make_graphed_callables..sJ$ $ AB1? .s)JJ1!/JQJJJJJJrc3(K|] }|j |VdSr'r|rs r rz)make_graphed_callables..s) T TqAO T T T T T T Trc3K|]}||V dSr'rrs r rz)make_graphed_callables..s"&W&WQq&W&Wr fwd_graphr bwd_graph module_paramstuple[torch.nn.Parameter, ...] len_user_argsrDoutput_unflatten_spectorch.utils._pytree.TreeSpecstatic_input_surfacerpstatic_outputsstatic_grad_outputstuple[Tensor | None, ...]static_grad_inputsrCallable[..., object]c t Gfddtjj d fd } | S)NceZdZed fd Zeejjjd fd Z d S) Omake_graphed_callables..make_graphed_autograd_function..Graphedctxrgrr rrpctD]Y}|||kr!|||Zt t st dtt dDS)Nz"static_outputs must be tuple, got c3>K|]}|VdSr'detachrs r rzjmake_graphed_callables..make_graphed_autograd_function..Graphed.forward..s*@@AQXXZZ@@@@@@r)rangedata_ptrcopy_r;rrrXtype)rrrrrrrs r forwardzWmake_graphed_callables..make_graphed_autograd_function..Graphed.forwards}--AAA+A.7799VAY=O=O=Q=QQQ,Q/55fQi@@@  """!.%88(ST.=Q=QSS@@@@@@@@rgradsct|tkr/tdt|dtt|D]F\}}|?||kr||Gt tstdttdDS)Nz len(grads)=z != len(static_grad_outputs)=z&static_grad_inputs must be tuple, got c3FK|]}||n|VdSr'rr~s r rzkmake_graphed_callables..make_graphed_autograd_function..Graphed.backward..3sH#$-AHHJJJQr) rrXziprrr;rrr)rrggradrrrs r backwardzXmake_graphed_callables..make_graphed_autograd_function..Graphed.backwards$u::%8!9!999(ic%jjiisSfOgOgii ##6>>**GAt}::<<4==??::GGDMMM  """""4e<<([FXAYAY[[0rN)rrgrr rrp)rrgrr rrp) rKrLrM staticmethodrr"autogradfunctiononce_differentiabler)rrrrrrrsr Graphedrs  A A A A A A A A\ A ^ $ 8       9 8\   rr user_argsrgrctjjj|}jt |z}tjj|Sr')r"utils_pytreearg_tree_leavesapplyrtree_unflatten)rflatten_user_argsoutrrrs r functionalizedzVmake_graphed_callables..make_graphed_autograd_function..functionalized9sP!& 3 CY O '-%(9":":]"JLC;&55c;PQQ Qr)rrgrrg)r"rFunction) rrrrrrrrrrrs ````````` @r make_graphed_autograd_functionz>make_graphed_callables..make_graphed_autograd_functions( ( ( ( ( ( ( ( ( ( ( ( ( en-( ( ( T R R R R R R R Rrfuncrkgraph_training_statergraphedCallable[_P, _R]orig_fwdc dfd }|S) Nr_P.args user_kwargs _P.kwargsrrc:jkr|i|S|i|Sr')training)rrrrrrs r new_fwdzEmake_graphed_callables..make_graphed_forward..new_fwdZs=}(<<<&w A[AAA'xBkBBBr)rrrrrrr)rrrrrs```` r make_graphed_forwardz4make_graphed_callables..make_graphed_forwardTsC CCCCCCCCCr)rrrrrrrrDrrrrprrprrrrprr) rrkrrrrrrrr)(r"is_autocast_enabledis_autocast_cache_enabled RuntimeErrorrrtypingcastr rrrr_backward_hooks_forward_hooks_forward_pre_hooksrXallbuffersrrrappendrrr#r\rTrV tree_leavesrrr tree_flattenreversedr}reverse enumeraterr)+rnrorqrrr-just_one_callable _sample_argsrrf flatten_argper_callable_len_user_args"per_callable_static_input_surfaces fwd_graphs bwd_graphsmempoolrr grad_inputsr outputs_gradrvper_callable_static_outputs"per_callable_output_unflatten_specr func_outputsflatten_outputsspec per_callable_static_grad_outputsper_callable_static_grad_inputsrrrrgrad_idxrrretrrrrrs+ @@r rr%s R  "" u'F'H'H  m    i ' 'P L  E&#+$6 DDF {5vs{);S)@#A;OO y,//4 a ) ) A%&&!++())Q..,--22$aEEEEEEE $1 k)94@ ""5#5#5666HHKHHHHH  ^  "M!L8K!L!L!L"""""*****s9~~&&***& IH%I2G2GHHHJHH%I2G2GHHHJ%)\!!!tG  J   5:,,.. / /03 |%G1 1    ,D$,2B .K,+,,  +-99$$+FF$$K$K$K$K$KKK |$$q(("'."5"5 ,$%%';%%%  &+++9@+++&&%)%7#6 # #K|[9  A ' . J#%)+&!$Y j!I!I88dI Z  ig  6 6 ' '4;L ' ' ' ' ' ' ' ' ' ' ' ' ' ' '!& 3 @ @ N N#**5+A+ABBB*11$7777(*$&(#;>344,--<<%C%C7ni$$ $ FT$ $ $   JJJJJJJ  |  q !!)'!::  #n11( T T,@ T T TTT!&&W&W2E&W&W&W!W!W $!3 2                 ' 0 0C  0[%<"))+h*?@@@A "))$////"#566(//0CDDD'../ABBBB%,,...#++---====@$&CY''$ $ 400 qM qM &q ) &q ) .q 1 .q 1 ' * ,Q / +A .    dEHO , ,      0/dmWdlDL JJt     JJw    1v ::s8COOO&P88P< ?P< AV,,V0 3V0 )rrrJ)rmFN) rnrlrorprqrDrrrr-r.rrl) rnrurorvrqrDrrrr-r.rru) rnrxroryrqrDrrrr-r.rrx)& __future__rr`rcollections.abcrrrrrtyping_extensionsr r r r"r torch.cudar _utilsr__all__rrhasattrrc__dict__torch._Crrrrrrrrgrlrjrrrr rs""""""" $$$$$$<<<<<<<<<<<<6666666666 (''''''     WT]]Yt__wux*++&1k,&?&?EHl#.9k:N.O.OEH*+:E+(;;EH67TSSSSSSSSS,,,,9999q-q-q-q-q- q-q-q-hT(T(T(T(T(T(T(T(p %%6f8M%MNNNNN $ $   $ $ (((( ($ $ IIIIIIIr