#j/ddlZddlmZmZddlmZddlmZddlm Z m Z m Z m Z m Z mZddlZddlmZddlmZdd lmZdd lmZmZmZdd lmZmZe r ddlZd d lm Z ddde!fdZ"eGddZ#ddde$fdZ%dS)N) dataclassfield)BytesIO)Path) TYPE_CHECKINGAnyClassVarDictOptionalUnion)config)DownloadConfig) array_cast) is_local_path is_remote_urlxopen)no_op_if_value_is_nullstring_to_dict) FeatureTypepdfpdfplumber.pdf.PDFreturnct5}|jD]!}||jj"|cdddS#1swxYwYdS)z-Convert a pdfplumber.pdf.PDF object to bytes.N)rpageswriterstreamgetvalue)rbufferpages _/lsinfo/ai/hellotax_ai/base_platform/venv/lib/python3.11/site-packages/datasets/features/pdf.py pdf_to_bytesr#s !fI * *D LL ) ) ) )  !!!!!!!!!!!!!!!!!!s=AA Ac eZdZUdZdZeed<eddZe e ed<dZ e e ed <e je je jd Ze eed <eddd Ze ed <dZdee eeedfdefdZddeddfdZdedee dfffdZdee je je jfde jfdZ dde jdedede jfdZ!dS)Pdfa **Experimental.** Pdf [`Feature`] to read pdf documents from a pdf file. Input: The Pdf feature accepts as input: - A `str`: Absolute path to the pdf file (i.e. random access is allowed). - A `pathlib.Path`: path to the pdf file (i.e. random access is allowed). - A `dict` with the keys: - `path`: String with relative path of the pdf file in a dataset repository. - `bytes`: Bytes of the pdf file. This is useful for archived files with sequential access. - A `pdfplumber.pdf.PDF`: pdfplumber pdf object. Args: decode (`bool`, defaults to `True`): Whether to decode the pdf data. If `False`, returns the underlying dictionary in the format `{"path": pdf_path, "bytes": pdf_bytes}`. Examples: ```py >>> from datasets import Dataset, Pdf >>> ds = Dataset.from_dict({"pdf": ["path/to/pdf/file.pdf"]}).cast_column("pdf", Pdf()) >>> ds.features["pdf"] Pdf(decode=True, id=None) >>> ds[0]["pdf"] >>> ds = ds.cast_column("pdf", Pdf(decode=False)) >>> ds[0]["pdf"] {'bytes': None, 'path': 'path/to/pdf/file.pdf'} ``` TdecodeNF)defaultrepridrdtypebytespathpa_type)r'initr(_typec|jSN)r.)selfs r"__call__z Pdf.__call__Ks |valuerctjrddl}nd}t|tr|ddSt|t r$t |ddSt|ttfrd|dS|)t||j j rt|S| d=tj|drd| ddS| d| d+| d| ddSt!d|d) zEncode example into a format for Arrow. Args: value (`str`, `bytes`, `pdfplumber.pdf.PDF` or `dict`): Data passed as input to Pdf feature. Returns: `dict` with "path" and "bytes" fields rNr-r,r-r+r,zRA pdf sample should have one of 'path' or 'bytes' but they are missing or None in .)rPDFPLUMBER_AVAILABLE pdfplumber isinstancestrrabsoluter, bytearrayrPDFencode_pdfplumber_pdfgetosr-isfile ValueError)r3r6r;s r"encode_examplezPdf.encode_exampleNso  &      J eS ! ! !D11 1 t $ $  0 011DAA A y1 2 2  511 1  # 5*.:L(M(M #(// / YYv   *rw~~eFm/L/L *!599V+<+<== = YYw   +uyy/@/@/L"YYw//69J9JKK Kmejmmm r5c|jstdtjrddl}nt d|i}|d|d}}||t d|dt|r|j|}n| d d }| tj r tj n tj } t||d } || } n#t $rd} YnwxYwt!| } t#|d | } |j| S|jt%|5} | }dddn #1swxYwY|S)aiDecode example pdf file into pdf data. Args: value (`str` or `dict`): A string with the absolute pdf file path, a dictionary with keys: - `path`: String with absolute or relative pdf file path. - `bytes`: The bytes of the pdf file. token_per_repo_id (`dict`, *optional*): To access and decode pdf files from private repositories on the Hub, you can pass a dictionary repo_id (`str`) -> token (`bool` or `str`). Returns: `pdfplumber.pdf.PDF` zKDecoding is disabled for this feature. Please use Pdf(decode=True) instead.rNz6To support decoding pdfs, please install 'pdfplumber'.r-r,z@A pdf should have one of 'path' or 'bytes' but both are None in r9::repo_idtokenrbdownload_config)r& RuntimeErrorrr:r; ImportErrorrEropensplit startswith HF_ENDPOINTHUB_DATASETS_URLHUB_DATASETS_HFFS_URLrrBrrr)r3r6token_per_repo_idr;r-bytes_r source_urlpatternrJrLrOfps r"decode_examplezPdf.decode_exampleqs&{ nlmm m  & X     VWW W  $ " V}eGnf >| !ldi!l!l!lmmm &&.)*/$//CC!%D!1!1"!5J&001CDD://#9 %"0W"E"Ei"P 1 5 5g > >%%%% $%&45&A&A&AOdD/JJJA*:?1--- 11 Q                s$+D DD"E11E58E5rcNddlm}|jr|n|d|ddS)zfIf in the decodable state, return the feature itself, otherwise flatten the feature into a dictionary.r)Valuebinarystringr+)featuresr`r&)r3r`s r"flattenz Pdf.flattensK######{ DDxh r5storagectj|jrrtjdgt |ztj}tj||gddg| }ntj |jrrtjdgt |ztj }tj||gddg| }n5tj |jr|j ddkr|d}n8tjdgt |ztj}|j ddkr|d}n8tjdgt |ztj }tj||gddg| }t||jS)aCast an Arrow array to the Pdf arrow storage type. The Arrow types that can be converted to the Pdf pyarrow storage type are: - `pa.string()` - it must contain the "path" data - `pa.binary()` - it must contain the image bytes - `pa.struct({"bytes": pa.binary()})` - `pa.struct({"path": pa.string()})` - `pa.struct({"bytes": pa.binary(), "path": pa.string()})` - order doesn't matter - `pa.list(*)` - it must contain the pdf array data Args: storage (`Union[pa.StringArray, pa.StructArray, pa.ListArray]`): PyArrow array to cast. Returns: `pa.StructArray`: Array in the Pdf arrow storage type, that is `pa.struct({"bytes": pa.binary(), "path": pa.string()})`. Ntyper,r-maskr)patypes is_stringrharraylenra StructArray from_arraysis_null is_binaryrb is_structget_field_indexrrr.)r3re bytes_array path_arrays r" cast_storagezPdf.cast_storages& 8  gl + + w(D6CLL#8ry{{KKKKn00+w1G'SYIZahapaparar0ssGG X   - - w4&3w<<"7bikkJJJJn00':1FRXHY`g`o`o`q`q0rrGG X   - - w|++G4499%mmG44  hvG '<29;;OOO |++F33q88$]]622 Xtfs7||&;")++NNN n00+z1JWV\L]dkdsdsdudu0vvG'4<000r5 local_files remote_filescitfdtjfd|Dtj}tjfd|dDtj}tj||gddg| }t||j S) a3Embed PDF files into the Arrow array. Args: storage (`pa.StructArray`): PyArrow array to embed. token_per_repo_id (`dict`, optional): Dictionary repo_id -> token to fetch the files bytes. local_files (`bool`, defaults to `True`) Whether to embed local files data in the array remote_files (`bool`, defaults to `True`) Whether to embed remote files data in the array. E.g. files with paths that start with hf:// or https:// Returns: `pa.StructArray`: Array in the PDF arrow storage type, that is `pa.struct({"bytes": pa.binary(), "path": pa.string()})`. Nc|dd}|tjr tjn tj}t ||}||dnd}t|}t|d|5}| cdddS#1swxYwYdS)NrHrIrJrKrMrN) rSrTrrUrVrWrrBrrread)r-rZr[source_url_fieldsrLrOr\rXs r" path_to_bytesz(Pdf.embed_storage..path_to_bytess D))"-J+5+@+@AS+T+Tv''Z`Zv !/z7 C C K\Kh%))*;I*FGGGnrE,5999OtT?CCC qvvxx                  s%CC  C cg|]U}|O|d?rt|dsr&t|dr|dn |dndVS)Nr,r-)rr).0xryrrzs r" z%Pdf.embed_storage..s   = z)%**7& *B*B*HT*YfghiogpYqYq*"M!F),,,7   r5rgcg|]I}|Crt|sr.t|rtj|n|ndJSr2)rrrCr-basename)rr-ryrzs r"rz%Pdf.embed_storage..s   #$(5d(;(;AMR_`dReReBG$$T***   r5r-r,ri) rrkrn to_pylistrarrbrprqrrrr.)r3rerXryrzrvrwrs ``` @r" embed_storagezPdf.embed_storagesI0  $ "     h      !**,,       X     $MM&11;;==       .,,k:-FRXHY`g`o`o`q`q,rr'4<000r5r2)NTT)"__name__ __module__ __qualname____doc__r&bool__annotations__rr)r r=r*r rkstructrarbr.rr0r4r r,r?dictrFr^r rd StringArrayrp ListArrayrxrr5r"r%r%s!!FFDd777B 7770E8C=///&RYibikk'R'RSSGXc]SSSu5u===E3===!E#uiG[*[$\!ae!!!!F88D8EY8888t  }d3 3E.FFG     #1E".".",*V$W#1\^\j#1#1#1#1LosC1C1~C1LPC1gkC1 C1C1C1C1C1C1r5r%ct|dr0t|jdr|jjr|jjddSdt|dS)aA Encode a pdfplumber.pdf.PDF object into a dictionary. If the PDF has an associated file path, returns the path. Otherwise, serializes the PDF content into bytes. Args: pdf (pdfplumber.pdf.PDF): A pdfplumber PDF object. Returns: dict: A dictionary with "path" or "bytes" field. rnameNr8)hasattrrrr#)rs r"rArA#s^sH:'#*f"="=:#*/: $777|C'8'8999r5)&rC dataclassesrriorpathlibrtypingrrr r r r pyarrowrkrdownload.download_configrtablerutils.file_utilsrrrutils.py_utilsrrr;rcrr,r#r%rrArr5r"rs ((((((((FFFFFFFFFFFFFFFF555555BBBBBBBBBBCCCCCCCC&%%%%%%!*!u!!!! A1A1A1A1A1A1A1 A1H:3:::::::r5