yjjddlmZddlmZddlmZmZddlZddlm Z erddlm Z dZ Gdd Z Gd d e Z Gd d ZGddeZGddeZdZe d(d)dZe d(d*dZe d(d+dZe d(d,d Z d-d!Ze d(d.d"Ze d(d/d$Z d-d%Zd&Zd-d'ZdS)0) annotations)Sequence) TYPE_CHECKINGoverloadN) framework)Tensorct|tjr|St|trt |S|SN) isinstancerVariablertuple)xss h/lsinfo/ai/hellotax_ai/data_center/backend/venv/lib/python3.11/site-packages/paddle/autograd/autograd.py as_tensorsrs?"i()) B ! !Ryy ceZdZdZ d!d"d Zed#d Zd Zd$dZdZ dZ dZ dZ dZ dZdZdZdZdZdZdZdZdZdZd S)%JacobianacComputes the Jacobian matrix of given xs and ys. Once the Jacobian ``J`` is constructed, you can use a multidimensional index to retrieve the submatrix of ``J``, as same as slicing a Tensor. The submatrix is lazily evaluated along row axis, and will be cached once evaluated. you can retrieve the submatrix by following methods: * J[:], retrieving the full matrix. * J[:, :, j], retrieving the partial derivatives w.r.t. the j'th input variable. * J[:, i, :], retrieving the partial derivatives w.r.t. the i'th output variable. * J[:, i, j], retrieving the partial derivatives w.r.t. the i'th output variable and the j'th input variable. Notes: Ellipsis index is not supported currently. Args: ys (Tensor|Tuple[Tensor, ...]): The output derived from xs . xs (Tensor|Tuple[Tensor, ...]): The input tensor(s) . is_batched (bool): If true, the first axis is batch axis. Defaults to False. Returns: Jacobian (Object): A python object retains the Jacobian matrix. Fysrr is_batchedboolreturnNonec|sdt|jcxkrdks&ntdt|jdt|jcxkrdks&ntdt|jt|||_dSdt|jcxkrdks&ntdt|jdt|jcxkrdks&ntdt|jt |||_dS)Nrz7xs.ndim should be 0 or 1 when is_batched=False but got z7ys.ndim should be 0 or 1 when is_batched=False but got z6ys.ndim should be 1 or 2 when is_batched=True but got z6xs.ndim should be 1 or 2 when is_batched=True but got )lenshape ValueError_JacobianNoBatch _jacobian_JacobianBatchFirst)selfrrrs r__init__zJacobian.__init__Gs  9BH ******** 0 #BH 00BH ******** 0 #BH 00.b"55DNNNBH ******** 0 #BH 00BH ******** 0 #BH 001R88DNNNr list[int]c|jjS)z'The shape of flattened Jacobian matrix.)r rr"s rrzJacobian.shapefs~##rc|j|Sr )r )r"indexess r __getitem__zJacobian.__getitem__ks~g&&r_Jacobian__namestrc|dkrt|j|S|dkrt|j|St|j|S)Nr _evaluate_all)getattrr r-)r"r*s r __getattr__zJacobian.__getattr__nsZ W  4>622 2 _ $ $4>622 2t~3355v>>>rc|}t|tr|n|}||zSr r-r rr"otherlhsrhss r__add__zJacobian.__add__uB  ""'1%'B'BMe!!###Syrc|}t|tr|n|}||z Sr r1r2s r__sub__zJacobian.__sub__zr7rc|}t|tr|n|}||zSr r1r2s r__mul__zJacobian.__mul__r7rc|}t|tr|n|}||z Sr r1r2s r__div__zJacobian.__div__r7rc|}t|tr|n|}||z Sr r1r2s r __truediv__zJacobian.__truediv__r7rc|}t|tr|n|}||zSr r1r2s r__pow__zJacobian.__pow__sA  ""'1%'B'BMe!!###Cxrc|}t|tr|n|}||zSr r1r2s r__mod__zJacobian.__mod__r7rc|}t|tr|n|}||zSr r1r2s r __floordiv__zJacobian.__floordiv__sB  ""'1%'B'BMe!!###czrc|}t|tr|n|}||zSr r1r2s r __matmul__zJacobian.__matmul__r7rc|}t|tr|n|}||kSr r1r2s r__eq__zJacobian.__eq__B  ""'1%'B'BMe!!###czrc|}t|tr|n|}||kSr r1r2s r__ne__zJacobian.__ne__rJrc|}t|tr|n|}||kSr r1r2s r__lt__zJacobian.__lt__B  ""'1%'B'BMe!!###Syrc|}t|tr|n|}||kSr r1r2s r__le__zJacobian.__le__rJrc|}t|tr|n|}||kSr r1r2s r__gt__zJacobian.__gt__rOrc|}t|tr|n|}||kSr r1r2s r__ge__zJacobian.__ge__rJrN)F)rrrrrrrr)rr$)r*r+)__name__ __module__ __qualname____doc__r#propertyrr)r/r6r9r;r=r?rArCrErGrIrLrNrQrSrUrrrr#sa!!N! 99999>$$$X$'''????              rrceZdZdS)HessianN)rVrWrXr[rrr]r]sDrr]cZeZdZdZdZedZdZdZd dZ dZ d Z d Z d Z d S) _JacobianaThe base class for computing Jacobian matrix. ``_Jacobian`` implements the core logic of multidimensional index and lazy evaluation for Jacobian matrix, subclass only need to overwrite following methods: * ``_lazy_axis()``, return the axis along which will be lazy evaluating. * ``_flatten(xs)``, flattens the inputs ``xs``. * ``_evaluate(index)``, evaluates one slice along ``_lazy_axis`` . Notes: Because currently PaddlePaddle only support reverse differentiation by ``paddle.grad``, so lazy evaluation is only supported along the row of Jacobian matrix, which means that slicing along row will get better performance. c"|j|_|j|_||_||_t |jjdkr'|js |jdg|_t |jjdkr(|jr!|jddg|_|t|j|_ |t|j|_ i|_ dS)Nrr) roriginal_xs_shapeoriginal_ys_shape_xs_ysrrreshape_flattenr _flatten_xs _flatten_ys_cache)r"rrs rr#z_Jacobian.__init__s!#!# tx~  ! # #DO #x''DH tx~  ! # # #x''Q00DH==DH)=)=>>==DH)=)=>> rct)z "The axis of lazily evaluated.NotImplementedErrorr&s r _lazy_axisz_Jacobian._lazy_axiss "!rc||j}t|tr|fn,tt |j|j|jSr )rnr intr rangestartstopstep)r"r(idxs r _lazy_indexesz_Jacobian._lazy_indexessJdo&#s## =SFFuSY#(;;<< rctr rlr"rs rrgz_Jacobian._flattens!!rrc||j}t|trdntd|d}g|d|j|||jdzdRS)Nrr)rnr rpslice)r"r(lazy_axis_sizerushifted_lazy_axis_idxs r_shifted_indexesz_Jacobian._shifted_indexess}do&C%% FAA5NA+F+F  &t& ' ! T_q(** +   rc<jdur`tjdkrtdtjdkr tjdkrd|fn|df}ntjdkr|ddf}n{tjdkrct |t r|t dddf}n:tjdkr|dd|dfn|d|ddf}t|j}t |j trH|dj |j dzdz} |j |S |}tj}t||j <tjfd|Dj |}||t|}t|jtjkrNt%t|jtjz D]}|d}|S) NFrz0-D tensor can not be indexed.rrc:g|]}|Sr[)_cached_evaluate).0ir"s r z)_Jacobian.__getitem__..1s' < < ?? , ,++ rc|(|dgS|j|}|||}||j|<|SNr)rrfrjget _evaluate)r"kvs rrz_Jacobian._cached_evaluate=sb 9((++33B77 7 KOOA   9q!!ADKNrct)z&Evaluate one slice at along lazy axis.rl)r"indexs rrz_Jacobian._evaluateFs!!rcpt|jdkr|dS|ddSr)rrrr&s rr-z_Jacobian._evaluate_allJs6 tz??a  ((.. .7NrN)r)rVrWrXrYr#rZrnrvrgr}r)rrr-r[rrr_r_s($""X"   """     444l"""rr_cDeZdZdZfdZedZdZdZxZ S)rzCompute Jacobian matrix without batch dimension. Suppose the mapping is :math:`f: R^M o R^N`, the output shape is ``(N, M)`` . cd|_t||g|jjdd|jjdd|_g|jdd|jdd|_dS)NFrr) rsuperr#rirrhrrcrbr"rr __class__s rr#z_JacobianNoBatch.__init__Ws R    $QqS) $QqS)  $QqS) $QqS)  rcdSrr[r&s rrnz_JacobianNoBatch._lazy_axisdqrct|ts|dStjt d|DS)Nrac3@K|]}|dVdS)rN)rfrxs r z,_JacobianNoBatch._flatten..ks."@"@199U#3#3"@"@"@"@"@"@r)r rrfrrr rxs rrgz_JacobianNoBatch._flattenhsM"h'' %::e$$ $}U"@"@R"@"@"@@@AAArch|t|j||jSr rg_grad_for_jacobianrirdr" row_indexs rrz_JacobianNoBatch._evaluatems5}}  +     r rVrWrXrYr#rZrnrgr __classcell__rs@rrrQs~      XBBB        rrcDeZdZdZfdZedZdZdZxZ S)r!zCompute Jacobian matrix with batch at first axis. Suppose the mapping is :math:`f: R^{B,M} o R^{B,N}`, the output shape is ``(B, N, M)`` . cRd|_t||g|jjdd|jjdd|jjdd|_g|jjdd|jdd|jdd|_dS)NTrrr) rrr#rhrrirrcrbrs rr#z_JacobianBatchFirst.__init__|s R    $QqS) $QqS) $QqS)   $QqS) $QqS) $QqS)  rcdS)Nrr[r&s rrnz_JacobianBatchFirst._lazy_axisrrct|ts"||jddfSt jt dt|DdS)Nrrac3ZK|]&}||jddfV'dS)rraN)rfrrs rrz/_JacobianBatchFirst._flatten..s9FF!!))QWQZ,--FFFFFFrr)r rrfrrrr rrxs rrgz_JacobianBatchFirst._flattensg"h'' 1::rx{B/00 0} FFz"~~FFF F F   rcp|t|jdd|f|jSr rrs rrz_JacobianBatchFirst._evaluates7}} t/9 =tx H H   rrrs@rr!r!vs{      X          rr!c .t|tr|n|f}td|Drtd|t dddft |t |z zz}g}t |D]\}}t|trt |jpd|jp|||j pd}| t |jdkr|j||zn|j|jdkr|j||zn|j|j t|tr'| |dkr |||zn|td|dt|S)abA tool for parsing N-dimensional index into a standard format. Currently supporting following input format: * ([positive|negative|slice], ...), the right-most elements can be omitted. The standard format after converted is slice tuple which contains N elements: * ([positive|slice], ..., [positive|slice]) Notes: Ellipsis indexes such as ``(..., i), (i, ...)`` is not supported. Args: indexes (tuple): The input indexes. shape (tuple): The input shape. Returns: tuple: The standard format index as the above description. c3ZK|]&}t|ttV'dSr )r typeEllipsis)rrs rrz_multi_index..s2 : :Q:ah ( ( : : : : : :rz*Ellipsis index currently is not supported.rNrzNot supported index type .)r ranyrrzr enumeraterrrsrtappendrp TypeErrorr )r(rpositive_indexesrrs rrrs($GX66FggWJG : :' : : :::GEFFFq$--/3u::G 3LMMGg&&BB5 eU # # B  q%*"8a%*/E  # #.3kAooEK%(**5;-2Z!^^EJq))J     s # # B  # # EE!H$4$4u M M M M@@@@AA A ! " ""r.rrr batch_axis int | NonercdSr r[rrrs rjacobianrs srSequence[Tensor] tuple[tuple[Jacobian, ...], ...]cdSr r[rs rrrs (+srtuple[Jacobian, ...]cdSr r[rs rrr 3rcdSr r[rs rrrrrc||dkrtd|d|duttr2ttrtfdD}nttr2ttstfdD}nXtts2ttrtfdD}nt }|S)a Computes the Jacobian of the dependent variable ``ys`` versus the independent variable ``xs``. Where ``ys`` represents the output of ``xs`` after a certain operation, ``ys`` and ``xs`` can be Tensor or tuple of Tensors, ``batch_axis`` indicates the position of the batch dimension of the parameter data. When the input is a tuple Tensors, the returned result is a ``Jacobian`` object with the same number of nesting levels as ``xs``, and each Jacobian has the same shape as The ``xs`` tuples are identical in one-to-one correspondence. - When ``batch_axis=None``, only 0-dimensional Tensor or 1-dimensional Tensor is supported, assuming the shape of ``xs`` is ``[N, ]``, the shape of ``ys`` is ``[M, ]``, then the output Jacobian matrix shape is ``[M, N]``. - When ``batch_axis=0``, only 1-dimensional Tensor or 2-dimensional Tensor is supported, assuming the shape of ``xs`` is ``[B, N]``, The shape of ``ys`` is ``[B, M]``, then the output Jacobian matrix shape is ``[B, M, N]``. After the ``Jacobian`` object is created, the actual calculation process does not occur, but the lazy evaluation method is used for calculation. It can be multi-dimensional indexed to obtain the entire Jacobian matrix or sub-matrix, and the actual calculation will be performed at this time the value is calculated and the result is returned. At the same time, in the actual evaluation process, the calculated sub-matrix will be cached to avoid duplicate calculations in the subsequent indexing process. For example, assuming ``Jacobian`` instance ``J`` has shape ``[B, M, N]``, assuming ``M > 4`` , then ``J[:, 1:4:1, :]`` means to get the values from row ``1`` to row ``3`` of ``J``. In actual calculation, only the rows ``1`` to ``3`` are evaluated, and the calculation results of ``1`` to ``3`` will be cached at the granularity of the row, and will be used next time. When obtaining one or more rows of results above, the already calculated parts will not be recalculated. Args: ys (Union[paddle.Tensor, Tuple[paddle.Tensor, ...]]): Output or tuple of outputs derived from xs. xs (Union[paddle.Tensor, Tuple[paddle.Tensor, ...]]): Input or tuple of inputs. batch_axis (Optional[int], optional): Index of batch axis. Defaults to None. Returns: Union[Tuple[Tuple[Jacobian, ...], ...], Tuple[Jacobian, ...], Jacobian]: Jacobian(s) of ys derived from xs. Examples: .. code-block:: pycon >>> import paddle >>> x1 = paddle.randn([3]) >>> x2 = paddle.randn([3]) >>> x1.stop_gradient = False >>> x2.stop_gradient = False >>> y = x1 + x2 >>> J = paddle.autograd.jacobian(y, (x1, x2)) >>> J_y_x1 = J[0][:] # evaluate result of dy/dx1 >>> J_y_x2 = J[1][:] # evaluate result of dy/dx2 >>> print(J_y_x1.shape) paddle.Size([3, 3]) >>> print(J_y_x2.shape) paddle.Size([3, 3]) Nrz(batch_axis should be None or 0, but got rc3RK|] tfdDV!dS)c3:K|]}t|VdSr r)rrdrers rrz%jacobian...>s/??S(3Z00??????rN)r rrerrs @rrzjacobian..=sU  DGE?????B??? ? ?      rc3:K|]}t|VdSr rrs rrzjacobian..As/FFC(3J77FFFFFFrc3:K|]}t|VdSr r)rrdrrs rrzjacobian..Cs/FFC(2sJ77FFFFFFr)rr rr r)rrrr rs`` @rrrsNR*// Dz D D D   4'J"h 1Jr8$<$< 1     KM      B ! !1*R*B*B1FFFFF2FFFFF H % %1*R*B*B1FFFFF2FFFFF RZ00 rcdSr r[rs rhessianrJs crtuple[tuple[Hessian, ...], ...]cdSr r[rs rrrRs '*crcrS|dkr%td|d|d}nttrn|ddkr%td|ddkrtd |d }n td t d t |}ttst |}t|_ nxtfd |D}tt|D]?}tt|dD]}t|||_ @|S)a Computes the Jacobian of the dependent variable ``ys`` versus the independent variable ``xs``. Among them, ``ys`` means the output of ``xs`` after a certain operation, ``ys`` can only be a single Tensor, ``xs`` can be a Tensor or a Tensor tuple, and ``batch_axis`` means The position of the batch dimension of the parameter data. When the input ``xs`` is a Tensor tuple, the returned result is a ``Hessian`` tuple, assuming that the internal shape of the ``xs`` tuple is composed of ``([M1, ], [M2, ])``, the shape of the returned result consists of ``(([M1, M1], [M1, M2]), ([M2, M1], [M2, M2]))`` - When ``batch_axis=None``, only 0-dimensional Tensor or 1-dimensional Tensor is supported, assuming that the shape of ``xs`` is ``[N, ]``, and the shape of ``ys`` is ``[ ]`` (0-dimensional Tensor), the final output is a single Hessian matrix whose shape is ``[N, N]``. - When ``batch_axis=0``, only 1-dimensional Tensor or 2-dimensional Tensor is supported, assuming that the shape of ``xs`` is ``[B, N]``, and the shape of ``ys`` is ``[B, ]``, the final output Jacobian matrix shape is ``[B, N, N]``. After the ``Hessian`` object is created, the complete calculation process does not occur, but a partial lazy evaluation method is used for calculation. It can be multi-dimensionally indexed to obtain the entire Hessian matrix or sub-matrix. At this time, the actual Evaluates the computation and returns the result. At the same time, in the actual evaluation process, the calculated sub-matrix will be cached to avoid repeated calculations in the subsequent indexing process. Args: ys (paddle.Tensor): Output derived from xs which contain one element. xs (Union[paddle.Tensor, Tuple[paddle.Tensor, ...]]): Input or tuple of inputs. batch_axis (Optional[int], optional): Index of batch axis. Defaults to None. Returns: Union[Tuple[Tuple[Hessian, ...], ...], Tuple[Hessian, ...], Hessian]: Hessian(s) of ys derived from xs. Examples: .. code-block:: pycon >>> import paddle >>> x1 = paddle.randn([3]) >>> x2 = paddle.randn([4]) >>> x1.stop_gradient = False >>> x2.stop_gradient = False >>> y = x1.sum() + x2.sum() >>> H = paddle.autograd.hessian(y, (x1, x2)) >>> H_y_x1_x1 = H[0][0][:] # evaluate result of ddy/dx1x1 >>> H_y_x1_x2 = H[0][1][:] # evaluate result of ddy/dx1x2 >>> H_y_x2_x1 = H[1][0][:] # evaluate result of ddy/dx2x1 >>> H_y_x2_x2 = H[1][1][:] # evaluate result of ddy/dx2x2 >>> print(H_y_x1_x1.shape) paddle.Size([3, 3]) >>> print(H_y_x1_x2.shape) paddle.Size([3, 4]) >>> print(H_y_x2_x1.shape) paddle.Size([4, 3]) >>> print(H_y_x2_x2.shape) paddle.Size([4, 4]) NrzOnly support ys.numel()(z)==1 when batch_axis is None.r[rzOnly support ys[0].numel()(z)==1 when batch_axis is intzOnly support batch_axis=0 yet.rz*batch_axis should be None or int, but got rc3:K|]}t|VdSr )r)r_jrrs rrzhessian..s/IIR44IIIIIIr) numelrrfr rprrrr]rr rqr)rrrr rrjs `` rrrZsJ 88::>>T288::TTT ZZ^^ J $ $  a5;;==1  UbhhjjUUU  ??=>> > ZZ   Lj9I9I L L L   R,,I b( # # 29b*55$IIIIIyIIIIIs7||$$ 2 2A3wqz??++ 2 2*1 1 '' 2 Nrc|"tj}j|_|St|tr(t fdt |DS|S)Nc3JK|]\}}t||VdSr )_replace_none_with_zero_tensor)rrrrefss rrz1_replace_none_with_zero_tensor..sG  ;?1a *1d1g 6 6      r)r zeros_like stop_gradientr rr r)rrs `rrrs z  t $ $- B ! !    CLR==       rctjrntj|||dd}t|tjjjr0t|trt|dkr|d}nktj |||}t|t jr0t|trt|dkr|d}t||S)akA gradient function that can be used in dynamic graph and static graph. The ``grad`` combines ``paddle.grad`` used in dynamic graph and ``paddle.static.gradients`` used in static graph, and do following changes: * The ``allow_unused`` flag is removed and set defaults to true internally, none in outputs will be replaced by zero tensor. * The ``create_graph`` flag is removed and set defaults to true internally, only makes sense in dynamic graph. * When xs is a single Tensor, ``paddle.grad`` returns a list which only contains one Tensor. It may confuse users, thus in this case we improve to return a single Tensor in _grad_for_jacobian interface. Args: ys (Tensor|Sequence[Tensor]): The output tensor or tensor sequence of the graph to compute gradients. xs (Tensor|Sequence[Tensor]): The input tensor or tensor sequence of the graph to compute gradients. The returned values of this API are the gradients of inputs . v (Tensor|Sequence[Tensor]|None,optional): The initial gradient values of outputs . If grad_outputs is None, the initial gradient values of outputs would be Tensors filled with 1; if grad_outputs is not None, it must have the same length as outputs , and in this case, the initial gradient value of the i-th outputs would be: (1) a Tensor filled with 1 when the i-th element of grad_outputs is None; (2) the i-th element of grad_outputs when the i-th element of grad_outputs is a Tensor. Default None. Returns: Tensor|tuple[Tensor]: Tensor or a tuple of Tensors, whose length is the same as the Tensor number inside inputs, and the i-th returned Tensor is the sum of gradients of outputs with respect to the i-th inputs. T) create_graph allow_unusedr) rin_dynamic_modegradr baserr rrstatic gradientsr)rrrxs_grads rrrsF!+b"adNNN r6;09 : : !7H-- !G q  ajG-))"b!44 r9- . . !7H-- !G q  ajG )'2 6 66r).)rrrrrrrr)rrrrrrrr)rrrrrrrr)rrrrrrrrr )rrrrrrrr])rrrrrrrr) __future__rcollections.abcrtypingrrr paddle.baserrrrr]r_rr!rrrrrr[rrrs#"""""$$$$$$******** !!!!!![[[[[[[[|     h   IIIIIIIIX" " " " " y" " " J# # # # # )# # # L-#-#-#` !  !++++ + !  ! [[[[| !  !**** *ggggT   676767676767r