j:@UddlmZddlmZmZmZmZmZmZm Z m Z m Z ddl Z ddlZddlZddlmZddlmZddlmZddlmZmZmZmZmZmZm Z m!Z!ddl"m#Z#m$Z$dd l%m&Z&m'Z'm(Z(m)Z)dd l*m+Z+dd l,m-Z-m.Z.m/Z/m0Z0d Z1e2e3d <e!e$e#j4eddddddgddddf dee2dee ee5efdee2de6de6deee2de6dee2de6de6deefdZ7 d2dej8de6fdZ9d eee2efdeee2effd!Z:d"eeeefde e e;ed'ee;d(e e;e;fde ee;ee;ffd)Z?de ee;ee2ffd*Z@d+e2defd,ZA d3de6d.ee2d/ee;dee2d0e e2dfdef d1ZBdS)4)SpooledTemporaryFile) IOAnyBinaryIODictListOptionalTupleUnioncastN) fromstring)add_chunking_strategy) clean_bullets)ElementElementMetadataListItem NarrativeTextTableTextTitleprocess_metadata)FileTypeadd_metadata_with_filetype) exactly_oneget_last_modified_date get_last_modified_date_from_filespooled_to_bytes_io_if_needed)apply_lang_metadata)is_bulleted_textis_possible_narrative_textis_possible_numbered_listis_possible_titlexlsxDETECTION_ORIGINTautoFfilenamefilemetadata_filenameinclude_metadatainfer_table_structure languagesdetect_language_per_elementmetadata_last_modifiedinclude_header find_subtablereturnc <t||d} |rdnd} |r'tj|d| } t|} n\|rZt t t ttf|}tj|d| } t|} g}d}| D]\}}|dz }| s|r| d|dnd}t| }|r%t||||p||p| }t|_nt}t#|| }||t'|}|D] \}}|\}}}}|j||dz||dzf}t+|\}} t-||j\}!}"t1||||p||p| }|!8|"6t3|!|z }#t3||"z }$t5||#|$f}|!H| d|!dzD]:}%t7t9|%}&||&_||&;|[t=|dkrHt7t9|jdjd}&||&nw|u| d|d}t| }t#| }||_|r|nd|j_ |||!J|"H| |!dzdD]:}%t7t9|%}&||&_||&; tCtE||| }|S) aPartitions Microsoft Excel Documents in .xlsx format into its document elements. Parameters ---------- filename A string defining the target filename path. file A file-like object using "rb" mode --> open(filename, "rb"). include_metadata Determines whether or not metadata is included in the output. infer_table_structure If True, any Table elements that are extracted will also have a metadata field named "text_as_html" where the table's text content is rendered into an html string. I.e., rows and cells are preserved. Whether True or False, the "text" field is always present in any Table element and is the text content of the table (no structure). languages User defined value for metadata.languages if provided. Otherwise language is detected using naive Bayesian filter via `langdetect`. Multiple languages indicates text could be in either language. Additional Parameters: detect_language_per_element Detect language per element instead of at the document level. metadata_last_modified The day of the last modification include_header Determines whether or not header info is included in text and medatada.text_as_html )r&r'Nr) sheet_nameheaderF)indexr3na_rep) text_as_html page_name page_numberr& last_modified)textmetadatar<)elementsr+r,)#rpd read_excelrrr r rrritemsto_htmlsoupparser_fromstring text_contentrr$detection_originrappend_get_connected_componentsiloc_single_non_empty_rows(_find_first_and_last_non_consecutive_rowshape _get_metadataint_get_sub_subtable_check_content_element_typestrr=lenvaluesr8listr)'r&r'r(r)r*r+r,r-r.r/kwargslast_modification_dater3sheetsfr?r:r2sheet html_textr<r=table_connected_components_connected_component_min_max_coordsmin_xmin_ymax_xmax_ysubtablesingle_non_empty_rowssingle_non_empty_row_contentsfront_non_consecutivelast_non_consecutive first_rowlast_rowcontentelements' e/lsinfo/ai/hellotax_ai/base_platform/venv/lib/python3.11/site-packages/unstructured/partition/xlsx.pypartition_xlsxrm'seX----! *QQdF HxDHHH!7!A!A H ) x!556 = =  qT&AAA!A$!G!G HK#\\^^P1P1 Eq N 1) E. LLL  )33@@BBD -*!*( +.:("8"R,>9#hmA>N>UVW>X:Y:YZZGOOG,,,,) ( 0 0u^\^ 0 _ _I0;;HHJJD$$///H(0H%BW5aYY]aH%2OOH---(49M9Y#@-133$11#>c'll"K"K+3( 0000m6 1p(C   H OrYfilterc |j\}}tj||}tj||fj}|j}d||D}||tj|}g} |D]@} t| } t| \} } }}| | | | ||dA|rt| } d| DS)a Identify connected components of non-empty cells in an excel sheet. Args: sheet: an excel sheet read in DataFrame. filter (bool, optional): If True (default), filters out overlapping components to return distinct components. Returns: A list of tuples, each containing: - A list of tuples representing the connected component's cell coordinates. - A tuple with the min and max x and y coordinates bounding the connected component. Note: This function performs a depth-first search (DFS) to identify connected components of non-empty cells in the sheet. If 'filter' is set to True, it also filters out overlapping components to return distinct components. c,g|]}t|S)tuple).0pairs rl z-_get_connected_components..sGGGtuT{{GGGrn componentr_r`rarbcZg|](}|d|d|d|d|dff)Srwrr)rtconnected_components rlrvz-_get_connected_components..sY     ,#G,#G,#G,#G,      rn) rLnx grid_2d_graphnpindicesTisnaremove_nodes_fromconnected_componentsrT_find_min_max_coordrG_filter_overlapping_tables)rYromax_rowmax_colgraph node_array empty_cellsnodes_to_removeconnected_components_as_nodesr _componentrxr_r`rarbs rlrHrHs0,{GW&w88EWg.//1J**,,.KGGz+/FGGGO O,,,$&$;E$B$B!3   $$ %8%C%C"ueU##&       P9:NOO  $8    rnrct|d}g}d}|D]}||}|d|dkr|d|dt|d|d|d<t|d|d|d<t|d|d|d<t|d|d|d<|||}||||S) zT Filter out overlapping connected components to return distinct components. c|dS)Nr_rr)xs rlz,_filter_overlapping_tables..s 1W:rn)keyNr_rarxr`rb)sortedextendminmaxrG)rsorted_componentsmerged_componentscurrent_componentrxs rlrrsD 39M9MNNN$&&..  $ )  !%6w%???!+.55i 6LMMM-01B71KYW^M_-`-`!'*-01B71KYW^M_-`-`!'*-01B71KYW^M_-`-`!'*-01B71KYW^M_-`-`!'**"(():;;;$-!!$  !2333 rnrzctdtdtdtdf\}}}}|D]%\}}||kr|}||kr|}||kr|}||kr|}&||||fS)z[ Find the minimum and maximum coordinates (bounding box) of a connected component. infz-inf)float)rzr_r`rarb_x_ys rlrrs "'uuU||U6]]ERXMM!YE5%%B ::E ::E ::E ::E % %%rnrcfirst_and_last_rowc@|\}}||krdS|j||dzS)z] Extract a sub-subtable from a given subtable based on the first and last row range. Nr4)rI)rcrrhris rlrOrO/s4-Ix9t =X\1 22rn row_indices table_shapec |\}}t|dkst||kr|dkr|d|dfStj|}tdt t ||ddDd}|ddd}tdt t ||ddDd}||fS)z_ Find the indices of the first and last non-consecutive rows in a list of row indices. r4rc38K|]\}\}}|dz|k|VdSr4Nrrrtirys rl z;_find_first_and_last_non_consecutive_row..Js2GGyq&1aAEQJJJJJJGGrnNc38K|]\}\}}|dz |k|VdSrrrrs rlrz;_find_first_and_last_non_consecutive_row..Os4YYyq&1aaRSeWXjjjjjjYYrn)rRr}arraynext enumeratezip)rr table_rows table_colsarrrf reversed_arrrgs rlrKrK<s)J  ;1[!1!1Z!?!?JRSOO1~{1~-- (;  C GGIc#s122w&7&788GGG ttt9LYYIc, QRR8H&I&IJJYYY  !"6 66rncg}g}|D]d\}}|dkrG||||jde||fS)zb Identify single non-empty rows in a subtable and extract their row indices and contents. r4r)iterrowscountrGdropnarI)rcrdrer6rows rlrJrJUs$&!''))GG s 99;;!   ! ( ( / / / ) 0 01B11E F F F "? ??rnr<c4t|rtt|St|rt|St |rt |St |rt|St|S)zA Classify the type of content element based on its text. r>) rrrr!r rr"rrr>s rlrPrPbs t$$     #4 ( (      $D ) )       4          rnrr2r:rVcN|rt||||}nt}|S)z5Returns metadata depending on `include_metadata` flag)r9r:r&r;)r)r)r2r:r&rVr=s rlrMrM|s@%" #0    #$$ Orn)T)TNrNN)Ctempfilertypingrrrrrr r r r networkxr{numpyr}pandasr@lxml.html.soupparserr rDunstructured.chunkingrunstructured.cleaners.corerunstructured.documents.elementsrrrrrrrr unstructured.file_utils.filetyperrunstructured.partition.commonrrrrunstructured.partition.langr unstructured.partition.text_typerr r!r"r$rQ__annotations__XLSXbytesboolrm DataFramerHrrNrrrOrKrJrPrMrrrnrlrs")))))))NNNNNNNNNNNNNNNNNNNNNNDDDDDD777777444444                    RQQQQQQQ <;;;;;#HM**"=A'+!"&&,X(-,0 TTsmT 5E$889 :T }T T  T S " T"&T%SMTTT ']TTT+*Tr88 <8 8888vtCH~. $sCx.@&d38n-& 5e eCJ/sEz1BE#u*DU UV&&&&& 3 3%S/ 3VXVb 3 3 3 37c7sCx7 8C=(3- '(77772 @d3ic.B(C @ @ @ @ c g    6" $!#"/3  #sm  "#t),   rn