o z3i+@sddlZddlmZmZddlmZddlmZddlm Z m Z m Z m Z m Z mZddlZddlmZddlmZdd lmZdd lmZmZdd lmZmZe rZddlZd d lmZddde fddZ!eGdddZ"ddde#fddZ$dS)N) dataclassfield)BytesIO)Path) TYPE_CHECKINGAnyClassVarDictOptionalUnion)config)DownloadConfig) array_cast) is_local_pathxopen)no_op_if_value_is_nullstring_to_dict) FeatureTypepdfpdfplumber.pdf.PDFreturncCsLt}|jD] }||jjq|WdS1swYdS)z-Convert a pdfplumber.pdf.PDF object to bytes.N)rpageswriterstreamgetvalue)rbufferpagerc/lsinfo/ai/hellotax_ai/llm_service/venv_embed/lib/python3.10/site-packages/datasets/features/pdf.py pdf_to_bytess  $r!c@seZdZUdZdZeed<edddZe e ed<dZ e e ed <e e e d Ze eed <edddd Ze ed <ddZdee eeedfdefddZddeddfddZdedee dfffddZdee je je jfde jfddZ dde jde jfddZ!dS)Pdfa **Experimental.** Pdf [`Feature`] to read pdf documents from a pdf file. Input: The Pdf feature accepts as input: - A `str`: Absolute path to the pdf file (i.e. random access is allowed). - A `pathlib.Path`: path to the pdf file (i.e. random access is allowed). - A `dict` with the keys: - `path`: String with relative path of the pdf file in a dataset repository. - `bytes`: Bytes of the pdf file. This is useful for archived files with sequential access. - A `pdfplumber.pdf.PDF`: pdfplumber pdf object. Args: decode (`bool`, defaults to `True`): Whether to decode the pdf data. If `False`, returns the underlying dictionary in the format `{"path": pdf_path, "bytes": pdf_bytes}`. Examples: ```py >>> from datasets import Dataset, Pdf >>> ds = Dataset.from_dict({"pdf": ["path/to/pdf/file.pdf"]}).cast_column("pdf", Pdf()) >>> ds.features["pdf"] Pdf(decode=True, id=None) >>> ds[0]["pdf"] >>> ds = ds.cast_column("pdf", Pdf(decode=False)) >>> ds[0]["pdf"] {'bytes': None, 'path': 'path/to/pdf/file.pdf'} ``` TdecodeNF)defaultrepridrdtypebytespathpa_type)r$initr%_typecCs|jSN)r+)selfrrr __call__Ksz Pdf.__call__valuercCstjrddl}nd}t|tr|ddSt|tr"t|ddSt|ttfr.d|dS|dur=t||j j r=t |S| ddurTt j|drTd| ddS| ddusb| ddurm| d| ddStd|d) zEncode example into a format for Arrow. Args: value (`str`, `bytes`, `pdfplumber.pdf.PDF` or `dict`): Data passed as input to Pdf feature. Returns: `dict` with "path" and "bytes" fields rNr*r)r*r(r)zRA pdf sample should have one of 'path' or 'bytes' but they are missing or None in .)r PDFPLUMBER_AVAILABLE pdfplumber isinstancestrrabsoluter) bytearrayrPDFencode_pdfplumber_pdfgetosr*isfile ValueError)r/r1r5rrr encode_exampleNs$      zPdf.encode_examplecCs.|jstdtjrddl}ntd|duri}|d|d}}|dur{|dur2td|dt|r=||}|S| d d }| tj rMtj ntj }zt||d } || } Wn tyid} Ynwt| d } t|d | d} || S|t| } | }Wd|S1swY|S)aiDecode example pdf file into pdf data. Args: value (`str` or `dict`): A string with the absolute pdf file path, a dictionary with keys: - `path`: String with absolute or relative pdf file path. - `bytes`: The bytes of the pdf file. token_per_repo_id (`dict`, *optional*): To access and decode pdf files from private repositories on the Hub, you can pass a dictionary repo_id (`str`) -> token (`bool` or `str`). Returns: `pdfplumber.pdf.PDF` zKDecoding is disabled for this feature. Please use Pdf(decode=True) instead.rNz6To support decoding pdfs, please install 'pdfplumber'.r*r)z@A pdf should have one of 'path' or 'bytes' but both are None in r3::repo_idtokenrbdownload_config)r# RuntimeErrorr r4r5 ImportErrorr?ropensplit startswith HF_ENDPOINTHUB_DATASETS_URLHUB_DATASETS_HFFS_URLrr<rrr)r/r1token_per_repo_idr5r*bytes_r source_urlpatternrCrErHfprrr decode_exampleqsD       zPdf.decode_examplercCs(ddlm}|jr |S|d|ddS)zfIf in the decodable state, return the feature itself, otherwise flatten the feature into a dictionary.r)Valuebinarystringr()featuresrXr#)r/rXrrr flattens z Pdf.flattenstoragecCs<tj|jr%tjdgt|td}tjj||gddg| d}nstj |jrJtjdgt|t d}tjj||gddg| d}nNtj |jr|j ddkr_|d}ntjdgt|td}|j ddkr{|d}ntjdgt|t d}tjj||gddg| d}t||jS)aCast an Arrow array to the Pdf arrow storage type. The Arrow types that can be converted to the Pdf pyarrow storage type are: - `pa.string()` - it must contain the "path" data - `pa.binary()` - it must contain the image bytes - `pa.struct({"bytes": pa.binary()})` - `pa.struct({"path": pa.string()})` - `pa.struct({"bytes": pa.binary(), "path": pa.string()})` - order doesn't matter - `pa.list(*)` - it must contain the pdf array data Args: storage (`Union[pa.StringArray, pa.StructArray, pa.ListArray]`): PyArrow array to cast. Returns: `pa.StructArray`: Array in the Pdf arrow storage type, that is `pa.struct({"bytes": pa.binary(), "path": pa.string()})`. Ntyper)r*maskr)patypes is_stringr_arraylenrY StructArray from_arraysis_null is_binaryrZ is_structget_field_indexrrr+)r/r] bytes_array path_arrayrrr cast_storages     zPdf.cast_storagecsduritfddtjfdd|Dtd}tjdd|dDtd}tjj||gd dg| d }t ||j S) a4Embed PDF files into the Arrow array. Args: storage (`pa.StructArray`): PyArrow array to embed. Returns: `pa.StructArray`: Array in the PDF arrow storage type, that is `pa.struct({"bytes": pa.binary(), "path": pa.string()})`. Ncs|dd}|tjrtjntj}t||}|dur#|dnd}t|d}t |d|d }| WdS1s@wYdS)NrArBrCrDrFrG) rLrMr rNrOrPrr<rrread)r*rSrTsource_url_fieldsrErHrU)rQrr path_to_bytess  $z(Pdf.embed_storage..path_to_bytescs8g|]}|dur|ddur|dn|dndqS)Nr)r*r).0x)rrrr s*z%Pdf.embed_storage..r^cSs$g|]}|durtj|ndqSr.)r=r*basename)rsr*rrr rus$r*r)r`) rrbre to_pylistrYrrZrgrhrirr+)r/r]rQrmrnr)rrrQr embed_storages    zPdf.embed_storager.)"__name__ __module__ __qualname____doc__r#bool__annotations__rr&r r7r'rrbstructrYrZr+rr-r0r r)r9dictr@rWr r\ StringArrayrg ListArrayrorxrrrr r"s  #$ #:$ %r"cCs:t|drt|jdr|jjr|jjddSdt|dS)aA Encode a pdfplumber.pdf.PDF object into a dictionary. If the PDF has an associated file path, returns the path. Otherwise, serializes the PDF content into bytes. Args: pdf (pdfplumber.pdf.PDF): A pdfplumber PDF object. Returns: dict: A dictionary with "path" or "bytes" field. rnameNr2)hasattrrrr!)rrrr r;s r;)%r= dataclassesrriorpathlibrtypingrrrr r r pyarrowrbr download.download_configrtablerutils.file_utilsrrutils.py_utilsrrr5r[rr)r!r"rr;rrrr s&       h