jW6.UddlZddlmZmZmZmZmZmZddlZddl m Z ddl m Z m Z mZmZmZddlmZddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lm Z dd l!m"Z"ddl#m$Z$iZ%ee&efe'd<e$drddl(m)Z)ddl*m+Z+e)e%d<e+e%d<e$drddl,m-Z-ddl.m/Z/e-e%d<e/e%d<e$dre$dr ddl0m1Z1e1e%d<e$dr ddl2m3Z3e3e%d<e$dr!ddl4m5Z5ddl6m7Z7dd l8m9Z9e5e%d!<e7e%d"<e9e%d#<e$d$r dd%l:m;Z;e;e%d&<e$d'r dd(le?d+e>Dr dd,l@mAZAeAe%d-<e$d.r dd/lBmCZCeCe%d0<e$d1rdd2lDmEZEdd3lFmGZGeEe%d4<eGe%d1<e$dre$d5r dd6lHmIZIeIe%d7< d]d8e&d9eee&effd:ZJdddddd;e jKddigd<d=ddd;d;d;ddd;d;dddddfd>ee&d?ee&d@eeeLdAee&dBee&dCeMdDe&dEee&dFeee&ge&fdGee&e&fdHee&dIeMdJee&dKeee&dLeMdMeMdNeMdOeee&dPee&dQeMdReMdSee dTee&dUeeNdVee&dWee&f4dXZOdid=dfdBe&d?ee&dGee&e&fdIeMdUeeNdYeejPeeff dZZQd[eedHee&dMeMdYeMfd\ZRdS)^N)IOCallableDictListOptionalTuple)DataSourceMetadata)FILETYPE_TO_MIMETYPESTR_TO_FILETYPEFileTypedetect_filetypeis_json_processable)logger) exactly_one)partition_email)partition_html)partition_json)&convert_old_ocr_languages_to_languages)partition_text)PartitionStrategy) partition_xmldependency_existsPARTITION_WITH_EXTRAS_MAPpandas) partition_csv) partition_tsvcsvtsvdocx) partition_doc)partition_docxdocpypandoc) partition_odtodt)partition_epubepub) partition_org) partition_rst) partition_rtforgrstrtfmarkdown) partition_mdmd msg_parser) partition_msgmsg) pdf2imagepdfminerPILc#4K|]}t|VdSNr).0deps e/lsinfo/ai/hellotax_ai/base_platform/venv/lib/python3.11/site-packages/unstructured/partition/auto.py r=Ps+55#555555) partition_pdfpdfunstructured_inference)partition_imageimagepptx) partition_ppt)partition_pptxpptopenpyxl)partition_xlsxxlsxdoc_typepartition_with_extras_mapc x|t}||}|td|d|d|d|S)N partition_z is not available. Install the z- dependencies with pip install "unstructured[z]")rget ImportError)rKrL_partition_funcs r<_get_partition_with_extrasrRjss!($=!/33H==O 6 6 6# 6 6)1 6 6 6   r>F)r@jpgpngxlsrJTfilename content_typefile file_filenameurlinclude_page_breaksstrategyencodingparagraph_grouperheadersskip_infer_table_types ssl_verify ocr_languages languagesdetect_language_per_elementpdf_infer_table_structurepdf_extract_imagespdf_extract_element_typespdf_image_output_dir_pathpdf_extract_to_payload xml_keep_tagsdata_source_metadatametadata_filenamerequest_timeouthi_res_model_name model_namec $ t||||r|rtd||}tjd|d|| dkrd} | 4| tdt | } tjd|t||| | | \}}n.| ikrtjd t||||| }|| d t|| |}|tj krtd }|d,|||| |d|}n|tjkrtd}|d,|||| |d|}n|tjkrtd} | d,|||| |d|}n|tjkrt#d,|||| |d|}nw|tjkrtd}!|!d,||| |d|}nI|tjkrt)d,||||| |d|}n$|tjkrt-d,||||| |d|}n|tjkr td}"|"d,||||| |d|}n|tjkrtd}#|#d,|||| |d|}n|tjkr td}$|$d,||||| |d|}np|tjkr td}%|%d,||||| |d|}n@|tjkr(td}&|&d,||d|||| |||||p|d |}n|tjks |tjks|tjkrt?d,||d|||| |p|d|}n|tj krtCd,||||| |d|}n|tj"kr td }'|'d,||||| |d|}nj|tj#kr td!}(|(d,||||| |d|}n:|tj$kr td"})|)d,||||| |d|}n |tj%kr0tM||#std$tOd,||d#|}n|tj(ks|tj)krtd%}*|*d,|||| |d|}n|tj*krtd&}+|+d,|||| |d|}n^|tj+krtd'},|,d,||| |d|}n1|tj,krg}n|sd(nd)|}-t|-d*|d+|D]c}.||.j-_.||.j-_/|0taj1|}/|/ td|/nd|.j-_3Ltd||.j-_3d|S)-apPartitions a document into its constituent elements. Will use libmagic to determine the file's type and route it to the appropriate partitioning function. Applies the default parameters for each partitioning function. Use the document-type specific partitioning functions if you need access to additional kwarg options. Parameters ---------- filename A string defining the target filename path. content_type A string defining the file content in MIME type file A file-like object using "rb" mode --> open(filename, "rb"). metadata_filename When file is not None, the filename (string) to store in element metadata. E.g. "foo.txt" url The url for a remote document. Pass in content_type if you want partition to treat the document as a specific content_type. include_page_breaks If True, the output will include page breaks if the filetype supports it strategy The strategy to use for partitioning PDF/image. Uses a layout detection model if set to 'hi_res', otherwise partition simply extracts the text from the document and processes it. encoding The encoding method used to decode the text input. If None, utf-8 will be used. headers The headers to be used in conjunction with the HTTP request if URL is set. skip_infer_table_types The document types that you want to skip table extraction with. ssl_verify If the URL parameter is set, determines whether or not partition uses SSL verification in the HTTP request. languages The languages present in the document, for use in partitioning and/or OCR. For partitioning image or pdf documents with Tesseract, you'll first need to install the appropriate Tesseract language pack. For other partitions, language is detected using naive Bayesian filter via `langdetect`. Multiple languages indicates text could be in either language. Additional Parameters: detect_language_per_element Detect language per element instead of at the document level. pdf_infer_table_structure If True and strategy=hi_res, any Table Elements extracted from a PDF will include an additional metadata field, "text_as_html," where the value (string) is a just a transformation of the data into an HTML . The "text" field for a partitioned Table Element is always present, whether True or False. pdf_extract_images Only applicable if `strategy=hi_res`. If True, any detected images will be saved in the path specified by 'image_output_dir_path' or stored as base64 encoded data within metadata fields. Deprecation Note: This parameter is marked for deprecation. Future versions will use 'extract_element_types' for broader extraction capabilities. pdf_extract_element_types Only applicable if `strategy=hi_res`. Images of the element type(s) specified in this list (e.g., ["Image", "Table"]) will be saved in the path specified by 'image_output_dir_path' or stored as base64 encoded data within metadata fields. pdf_extract_to_payload Only applicable if `strategy=hi_res`. If True, images of the element type(s) defined in 'extract_element_types' will be encoded as base64 data and stored in two metadata fields: 'image_base64' and 'image_mime_type'. This parameter facilitates the inclusion of element data directly within the payload, especially for web-based applications or APIs. pdf_image_output_dir_path Only applicable if `strategy=hi_res` and `pdf_extract_to_payload=False`. The filesystem path for saving images of the element type(s) specified in 'extract_element_types'. xml_keep_tags If True, will retain the XML tags in the output. Otherwise it will simply extract the text from within the tags. Only applies to partition_xml. request_timeout The timeout for the HTTP request if URL is set. Defaults to None meaning no timeout and requests will block indefinitely. hi_res_model_name The layout detection model used when partitioning strategy is set to `hi_res`. model_name The layout detection model used when partitioning strategy is set to `hi_res`. To be deprecated in favor of `hi_res_model_name`. )rXrVrZzOnly one of metadata_filename and file_filename is specified. metadata_filename is preferred. file_filename is marked for deprecation.NzuThe file_filename kwarg will be deprecated in a future version of unstructured. Please use metadata_filename instead.rlz}Only one of languages and ocr_languages should be specified. languages is preferred. ocr_languages is marked for deprecation.zmThe ocr_languages kwarg will be deprecated in a future version of unstructured. Please use languages instead.)rZrWr_rarmzUThe headers kwarg is set but the url kwarg is not. The headers kwarg will be ignored.)rVrXrYrWr]rr#)rVrXinfer_table_structurercrdr r&)rVrXr]rcrdr4)rVrXrcrd)rVrXr[r]rcrd)rVrXr]rjrcrdr()rVrXr[rrrcrdr,)rVrXr[rcrdr-r1r@) rVrXrZr[rrr\rcextract_images_in_pdfextract_element_typesimage_output_dir_pathextract_to_payloadrn)rVrXrZr[rrr\rcrn)rVrXr]r^rcrdr.rGrD)rVrXzDetected a JSON file that does not conform to the Unstructured schema. partition_json currently only processes serialized Unstructured output.rJrrz Invalid filez Invalid file z. The z) file type is not supported in partition.)4r ValueErrorrwarn setdefaultrwarningfile_and_type_from_urlr seekdecide_table_extractionr DOCrRDOCXODTEMLrMSGHTMLrXMLrEPUBORGRSTMDPDFPNGJPGTIFFrBTXTrRTFPPTPPTXJSONrrXLSXXLSCSVTSVEMPTYmetadatarZ data_sourcer rOr filetype)0rVrWrXrYrZr[r\r]r^r_r`rarbrcrdrerfrgrhrirjrkrlrmrnrokwargsrrr_partition_docelements_partition_docx_partition_odt_partition_msg_partition_epub_partition_org_partition_rst _partition_md_partition_pdf_partition_rtf_partition_ppt_partition_pptx_partition_xlsx_partition_csv_partition_tsvr4element out_filetypes0 r< partitionrzs& XTH#6666 ]  W    )  4    )+<===    S  ?}MMI N0     /%!+    hh b== N5   #+%      ! 3! 8<3E::!> "7(C       X] " "4V<<"? "7(C       X\ ! !3E::!> "7(C       X\ ! !" (C       X\ ! !3E::!> (C       X] " "!  3(C      X\ ! !  '(C      X] " "4V<<"?  3"7(C      X\ ! !3E::!>  3(C       X\ ! !3E::!>  3"7(C      X[ 2488  =  3"7(C      X\ ! !3E::!>  3"7"4";";5/=:     hl " "HL(@(@hRZR_F_F_"   3"7/=:       X\ ! !! /(C      X\ ! !3E::!>  3"7(C      X\ ! !3E::!>  3"7(C      X] " "4V<<"?  3"7(C      X] " ""H4@@@ Z "I8$II&II hm # #X\)A)A4V<<"? "7(C       X\ ! !3E::!> "7(C       X\ ! !3E::!> (C       X^ # #$,Lnn2L(2L2LCZZxZZZ[[[ G G"';$  #*.|<returnctj||||}tj|j}|p|jd}|jdd}t |||}||fS)N)r_verifytimeoutz Content-TypezContent-Encodingzutf-8)rXrWr])requestsrOioBytesIOcontentr_r ) rZrWr_rarmresponserXr]rs r<r|r|s|C_]]]H :h& ' 'DG8#3#7#7#G#GL##$6@@HD|hWWWH >r>rc|r|jnd}|dkr'||vr|rtjd|d|d||vp|S||vS)Nr@z3Conflict between variables skip_infer_table_types: z and pdf_infer_table_structure: zK, please reset skip_infer_table_types to turn on table extraction for PDFs.)namelowerrr{)rr`rerKs r<r~r~'s )1:x}""$$$dH5 - - -2K - N\F\\\2K\\\    55R9RR 1 11r>r9)Srtypingrrrrrrrunstructured.documents.elementsr unstructured.file_utils.filetyper r r r runstructured.loggerrunstructured.partition.commonrunstructured.partition.emailrunstructured.partition.htmlrunstructured.partition.jsonrunstructured.partition.langrunstructured.partition.textr&unstructured.partition.utils.constantsrunstructured.partition.xmlrunstructured.utilsrrstr__annotations__unstructured.partition.csvrunstructured.partition.tsvrunstructured.partition.docr!unstructured.partition.docxr"unstructured.partition.odtr%unstructured.partition.epubr'unstructured.partition.orgr)unstructured.partition.rstr*unstructured.partition.rtfr+unstructured.partition.mdr0unstructured.partition.msgr3 pdf_importsallunstructured.partition.pdfr?unstructured.partition.imagerBunstructured.partition.pptrEunstructured.partition.pptxrFunstructured.partition.xlsxrIrRAUTObytesboolintrrr|r~rwr>r<rsx <<<<<<<<<<<<<<<<>>>>>>'&&&&&555555888888666666666666766666DDDDDD444444000000134X .333X5888888888888'4e$'4e$V7888888::::::'4e$(6f%V5!2!2:!>!>5888888'4e$Z  7::::::(6f%Z  5888888888888888888'4e$'4e$'4e$Z  3666666&2d#\""5888888'4e$/.. 355555555888888'4e$-..9<<<<<<)8g&V7888888::::::'4e$(6f%X7#4#4Z#@#@7::::::(6f% @D   'S(](;<    "#"& $#' %%*"8< (L(L(L#'%)(-&+$59/3#(9='+%)'+ $5YYsmY3-Y 2e9 YC= Y # Y  YYsmY # 45Y#s(^Y!IYYC=YS "Y"&Y $!Y"#Y$ (S 2%Y& (}'Y(!)Y*+Y,##56-Y. }/Y0c]1Y2 }3Y4 5YYYY| #' %)  3-#s(^  c]   2:x) )* "2x 2 I2 $2 222222r>