j$UdZddlmZddlZddlZddlmZmZmZm Z m Z m Z m Z m Z mZmZmZddlZddlmZmZddlmZmZmZmZmZmZmZmZddlmZeege fZ!de"d < d Z#de"d < Gd d Z$GddZ%GddZ&GddZ'GddZ(GddZ)GddZ*GddZ+d#dZ,d#dZ-d$d"Z.dS)%z@Chunking objects not specific to a particular chunking strategy.) annotationsN) AnyCallable DefaultDictDictIterableIteratorListOptionalSequenceTuplecast)Self TypeAlias)CompositeElementConsolidationStrategyElementElementMetadata RegexMetadataTable TableChunkTitle) lazypropertyrBoundaryPredicatezTablePreChunk | TextPreChunkPreChunkceZdZdZ d"d#dZe d"d$dZed%dZed%dZ ed&dZ ed%dZ ed%dZ ed'dZ ed(dZed)dZd*d!ZdS)+ChunkingOptionsa Specifies parameters of optional chunking behaviors. Parameters ---------- max_characters Hard-maximum text-length of chunk. A chunk longer than this will be split mid-text and be emitted as two or more chunks. new_after_n_chars Preferred approximate chunk size. A chunk composed of elements totalling this size or greater is considered "full" and will not be enlarged by adding another element, even if it will fit within the remaining `max_characters` for that chunk. Defaults to `max_characters` when not specified, which effectively disables this behavior. Specifying 0 for this argument causes each element to appear in a chunk by itself (although an element with text longer than `max_characters` will be still be split into two or more chunks). multipage_sections Indicates that page-boundaries should not be respected while chunking, i.e. elements appearing on two different pages can appear in the same chunk. combine_text_under_n_chars Provides a way to "recombine" small chunks formed by breaking on a semantic boundary. Only relevant for a chunking strategy that specifies higher-level semantic boundaries to be respected, like "section" or "page". Recursively combines two adjacent pre-chunks when the first pre-chunk is smaller than this threshold. "Recursively" here means the resulting pre-chunk can be combined with the next pre-chunk if it is still under the length threshold. Defaults to `max_characters` which combines chunks whenever space allows. Specifying 0 for this argument suppresses combining of small chunks. Note this value is "capped" at the `new_after_n_chars` value since a value higher than that would not change this parameter's effect. overlap Specifies the length of a string ("tail") to be drawn from each chunk and prefixed to the next chunk as a context-preserving mechanism. By default, this only applies to split-chunks where an oversized element is divided into multiple chunks by text-splitting. text_splitting_separators A sequence of strings like `(" ", " ")` to be used as target separators during text-splitting. Text-splitting only applies to splitting an oversized element into two or more chunks. These separators are tried in the specified order until one is found in the string to be split. The default separator is `""` which matches between any two characters. This separator should not be specified in this sequence because it is always the separator of last-resort. Note that because the separator is removed during text-splitting, only whitespace character sequences are suitable. NTrcombine_text_under_n_chars Optional[int]max_charactersintmultipage_sectionsboolnew_after_n_charsoverlaptext_splitting_separators Sequence[str]cZ||_||_||_||_||_||_dSN)_combine_text_under_n_chars_arg_max_characters_multipage_sections_new_after_n_chars_arg_overlap_text_splitting_separators)selfr r"r$r&r'r(s d/lsinfo/ai/hellotax_ai/base_platform/venv/lib/python3.11/site-packages/unstructured/chunking/base.py__init__zChunkingOptions.__init__Zs:0J,-#5 &7# *C'''returnrcN|||||||}||S)zrConstruct validated instance. Raises `ValueError` on invalid arguments like overlap > max_chars. ) _validate)clsr r"r$r&r'r(r2s r3newzChunkingOptions.newjs>s &     %     r5cX|j}|j}|j}|||kr|n|}||kr|n|S)aCombine consecutive text pre-chunks if former is smaller than this and both will fit. - Does not combine table chunks with text chunks even if they would both fit in the chunking window. - Does not combine text chunks if together they would exceed the chunking window. - Defaults to `max_characters` when not specified. - Is reduced to `new_after_n_chars` when it exceeds that value. )r-soft_maxr,)r2r"r<argr s r3r z*ChunkingOptions.combine_text_under_n_charssM-=28;{cNFZFZ^^`c"6@@xxF``r5c|jS)zThe maximum size for a chunk. A pre-chunk will only exceed this size when it contains exactly one element which by itself exceeds this size. Such a pre-chunk is subject to mid-text splitting later in the chunking process. )r-r2s r3hard_maxzChunkingOptions.hard_maxs ##r5c|jS)z0When False, break pre-chunks on page-boundaries.)r.r?s r3r$z"ChunkingOptions.multipage_sectionss ''r5c|jS)zThe number of characters to overlap text when splitting chunks mid-text. The actual overlap will not exceed this number of characters but may be less as required to respect splitting-character boundaries. )r0r?s r3r'zChunkingOptions.overlaps }r5cB|j}|j}| |dks||kr|n|S)zA pre-chunk of this size or greater is considered full. ??? Is a value of 0 valid? It would produce the behavior: "put each element into its own chunk". Nr)r-r/)r2 max_charsr&s r3r<zChunkingOptions.soft_maxsC(  7")->-B-BFWZcFcFc I" r5 Callable[[str], Tuple[str, str]]c t|S)aA text-splitting function suitable for splitting the text of an oversized pre-chunk. The function is pre-configured with the chosen chunking window size and any other applicable options specified by the caller as part of this chunking-options instance. ) _TextSplitterr?s r3splitzChunkingOptions.splitsT"""r5strcdS)a;The string to insert between elements when concatenating their text for a chunk. Right now this is just " " (a blank line in plain text), but having this here rather than as a module-level constant provides a way for us to easily make it user-configurable in future if we want to. z rr?s r3text_separatorzChunkingOptions.text_separators vr5Tuple[str, ...]c*t|jS)zLSequence of text-splitting target strings to be used in order of preference.)tupler1r?s r3r(z)ChunkingOptions.text_splitting_separatorssT4555r5Nonec |j}|dkrtd||j}||dkrtd||j}||dkrtd||j|krtd|jdS)z5Raise ValueError if requestion option-set is invalid.rz+'max_characters' argument must be > 0, got Nz8'combine_text_under_n_chars' argument must be >= 0, got z/'new_after_n_chars' argument must be >= 0, got z0'overlap' must be less than max_characters, got )r- ValueErrorr,r/r0)r2r"r r&s r3r8zChunkingOptions._validates- Q  _~__`` `&*%I" % 16PST6T6T5255 !7  (->-B-BYFWYY  =N * *cTXTaccdd d + *r5)NrTNrr) r r!r"r#r$r%r&r!r'r#r(r))r r!r"r#r$r%r&r!r'r#r(r)r6rr6r#)r6r%)r6rEr6rI)r6rLr6rO)__name__ __module__ __qualname____doc__r4 classmethodr:rr r@r$r'r<rHrKr(r8rr5r3rr0s''V59!#'+/35DDDDD 59!#'+/35[0aaa\a($$$\$(((\(\    \  ###\#\666\6eeeeeer5rcBeZdZdZddZdd Zedd ZddZdS)rGaProvides a text-splitting function configured on construction. Text is split on the best-available separator, falling-back from the preferred separator through a sequence of alternate separators. - The separator is removed by splitting so only whitespace strings are suitable separators. - A "blank-line" (" ") is unlikely to occur in an element as it would have been used as an element boundary during partitioning. This is a *callable* object. Constructing it essentially produces a function: split = _TextSplitter(opts) fragment, remainder = split(s) This allows it to be configured with length-options etc. on construction and used throughout a chunking operation on a given element-stream. optsrc||_dSr+)_optsr2r[s r3r4z_TextSplitter.__init__ s  r5srIr6Tuple[str, str]c|jj}t||kr|dfS|jD]k\}}||||\}}|r t|t|krB||fcS|d||||jjz dfS)aReturn pair of strings split from `s` on the best match of configured patterns. The first string is the split, the second is the remainder of the string. The split string will never be longer than `maxlen`. The separators are tried in order until a match is found. The last separator is "" which matches between any two characters so there will always be a split. The separator is removed and does not appear in the split or remainder. An `s` that is already less than the maximum length is returned unchanged with no remainder. This allows this function to be called repeatedly with the remainder until it is consumed and returns a remainder of "". N)r]r@len _patterns_split_from_maxlenrstriplstripr')r2r_maxlenpsep_lenfragment remainders r3__call__z_TextSplitter.__call__s$ q66V  b5L. 9 9JAw#'"9"9!Wa"H"H Hi y>>SVV++??$$i&6&6&8&88 8 8 8 &z  ""Aftz/A&A&C&C$D$K$K$M$MMMr5*Tuple[Tuple[regex.Pattern[str], int], ...]cL|jj}td|DS)aSequence of (pattern, len) pairs to match against. Patterns appear in order of preference, those following are "fall-back" patterns to be used if no match of a prior pattern is found. NOTE these regexes search *from the end of the string*, which is what the "(?r)" bit specifies. This is much more efficient than starting at the beginning of the string which could result in hundreds of matches before the desired one. c3bK|]*}tjd|t|fV+dS)z(?r)N)regexcompilerc).0seps r3 z*_TextSplitter._patterns..?s=SSemL3LL113s88<SSSSSSr5)r]r(rN)r2 separatorss r3rdz_TextSplitter._patterns3s+Z9 SS SSSSSSr5patternregex.Pattern[str]rjr#c|jj|jj}}|||dz||z}|d|fS|\}}d} |d|} ||d} |t| kr| | fS|t| z } | | d} | | z| z}| |fS)aReturn (split, remainder) pair split from `s` on the right-most match before `maxlen`. Returns `"", s` if no suitable match was found. Also returns `"", s` if splitting on this separator produces a split shorter than the required overlap (which would produce an infinite loop). `split` will never be longer than `maxlen` and there is no longer split available using `pattern`. The separator is removed and does not appear in either the split or remainder. )posendposNrb )r]r@r'searchspanrfrgrc)r2rwrjr_rhr'match match_start match_end separatorrk raw_remaindertail_lentailoverlapped_remainders r3rez _TextSplitter._split_from_maxlenAs*-tz/Aqgk&7:JKK =q5L"' Y \k\?))++)** ,,.. c)nn $ $]* *S^^+ #**,,#i/-?---r5N)r[r)r_rIr6r`)r6rn)rwrxrjr#r_rIr6r`) rUrVrWrXr4rmrrdrerr5r3rGrGs$#N#N#N#NJ T T T\ T(.(.(.(.(.(.r5rGcZeZdZdZddZedd Zdd Zedd Z ddZ dS)BasePreChunkeraRBase-class for per-strategy pre-chunkers. The pre-chunker's responsibilities are: - **Segregate semantic units.** Identify semantic unit boundaries and segregate elements on either side of those boundaries into different sections. In this case, the primary indicator of a semantic boundary is a `Title` element. A page-break (change in page-number) is also a semantic boundary when `multipage_sections` is `False`. - **Minimize chunk count for each semantic unit.** Group the elements within a semantic unit into sections as big as possible without exceeding the chunk window size. - **Minimize chunks that must be split mid-text.** Precompute the text length of each section and only produce a section that exceeds the chunk window size when there is a single element with text longer than that window. A Table element is placed into a section by itself. CheckBox elements are dropped. The "by-title" strategy specifies breaking on section boundaries; a `Title` element indicates a new "section", hence the "by-title" designation. elementsSequence[Element]r[rc"||_||_dSr+) _elementsr]r2rr[s r3r4zBasePreChunker.__init__s! r5r6Iterator[PreChunk]c>|||S)zEGenerate pre-chunks from the element-stream provided on construction.)_iter_pre_chunks)r9rr[s r3iter_pre_chunkszBasePreChunker.iter_pre_chunkss" s8T""33555r5c#,Kt|j}|jD][}||s||s|Ed{V||\|Ed{VdS)aGenerate pre-chunks from the element-stream provided on construction. A *pre-chunk* is the largest sub-sequence of elements that will both fit within the chunking window and respects the semantic boundary rules of the chunking strategy. When a single element exceeds the chunking window size it is placed in a pre-chunk by itself and is subject to mid-text splitting in the second phase of the chunking process. N)PreChunkBuilderr]r_is_in_new_semantic_unitwill_fitflush add_element)r2pre_chunk_builderelements r3rzBasePreChunker._iter_pre_chunkss,DJ77~ 3 3G,,W55 5=N=W=WX_=`=` 5,22444444444  ) )' 2 2 2 2%**,,,,,,,,,,,r5Tuple[BoundaryPredicate, ...]cdS)zBThe semantic-boundary detectors to be applied to break pre-chunks.rrr?s r3_boundary_predicatesz#BasePreChunker._boundary_predicatess rr5rrr%cHfd|jD}t|S)zITrue when `element` begins a new semantic unit such as a section or page.c&g|] }|Srr)rspredrs r3 z;BasePreChunker._is_in_new_semantic_unit..s!SSSttG}}SSSr5)rany)r2rsemantic_boundariess ` r3rz'BasePreChunker._is_in_new_semantic_units2 TSSS9RSSS&'''r5N)rrr[r)rrr[rr6rr6r)r6rrrr6r%) rUrVrWrXr4rYrrrrrrr5r3rrqs,666[6 ----,\((((((r5rc"eZdZdZd dZd d Zd S) TablePreChunkz/A pre-chunk composed of a single Table element.tablerr[rr6rOc"||_||_dSr+)_tabler])r2rr[s r3r4zTablePreChunk.__init__s  r5Iterator[Table | TableChunk]c#K|jj}|jj}|jjjpd}|jj}t||krt||kr |jVdSd}|s|rw||\}}t|tj |jj}|r |d|||d}}||j_|r d|j_ |Vd}|s|udSdS)zLSplit this pre-chunk into `Table` or `TableChunk` objects maxlen or smaller.rbNFtextmetadataT) r]rHrrr text_as_htmlr@rcrcopydeepcopyis_continuation) r2rHtext_remainderhtml_remainderrhr chunk_text table_chunk chunk_htmls r3 iter_chunkszTablePreChunk.iter_chunkss7  )-:@b$ ~  & ( (S-@-@F-J-J+    F # #).~)>)> &J$*t}T[Ma?b?bcccK ?-;GVG-DnU[U\U\F]N 4> $1 <7; $4   "O# # # # # # #r5N)rrr[rr6rO)r6r)rUrVrWrXr4rrr5r3rrsB99######r5rceZdZdZddZdd Zd dZd!dZed"dZ ed#dZ ed$dZ ed%dZ ed&dZ ed'dZdS)( TextPreChunkaoA sequence of elements that belong to the same semantic unit within a document. The name "section" derives from the idea of a document-section, a heading followed by the paragraphs "under" that heading. That structure is not found in all documents and actual section content can vary, but that's the concept. This object is purposely immutable. rIterable[Element]r[rr6rOc<t||_||_dSr+)listrr]rs r3r4zTextPreChunk.__init__sh r5otherrr%cPt|tsdS|j|jkS)NF) isinstancerr)r2rs r3__eq__zTextPreChunk.__eq__s(%.. 5~00r5other_pre_chunkcHt|j|jz|jS)zCReturn new `TextPreChunk` that combines this and `other_pre_chunk`.)r[)rrr])r2rs r3combinezTextPreChunk.combines!DN_-FFTZXXXXr5Iterator[CompositeElement]c#K|jj}|j}|j}|r%||\}}t ||V|#dSdS)zSSplit this pre-chunk into one or more `CompositeElement` objects maxlen or smaller.rN)r]rH_consolidated_metadata_textr)r2rHrrlr_s r3rzTextPreChunk.iter_chunkssq  .J  > 5++LAy"H=== = = = > > > > >r5r#c*t|jS)zDLength of concatenated text of this pre-chunk, including separators.)rcrr?s r3 text_lengthzTextPreChunk.text_lengths4:r5Dict[str, List[Any]]cdd}tjt}|jD]3}||jD] \}}|||!4t |S)aCollection of all populated metadata values across elements. The resulting dict has one key for each `ElementMetadata` field that had a non-None value in at least one of the elements in this pre-chunk. The value of that key is a list of all those populated values, in element order, for example: { "filename": ["sample.docx", "sample.docx"], "languages": [["lat"], ["lat", "eng"]] ... } This preprocessing step provides the input for a specified consolidation strategy that will resolve the list of values for each field to a single consolidated value. rrr6Iterator[Tuple[str, Any]]cHd|jDS)zM(field_name, value) pair for each non-None field in single `ElementMetadata`.c3(K|] \}}|||fVdSr+r)rs field_namevalues r3ruzSTextPreChunk._all_metadata_values..iter_populated_fields..!s=%J$U#$$$$r5) known_fieldsitems)rs r3iter_populated_fieldsz@TextPreChunk._all_metadata_values..iter_populated_fieldss2)1)>)D)D)F)F r5)rrr6r) collections defaultdictrrrappenddict)r2r field_valueserrs r3_all_metadata_valuesz!TextPreChunk._all_metadata_values s$    5@4KD4Q4Q  7 7A%:%:1:%F%F 7 7! EZ(//6666 7L!!!r5rc$tdi|jS)aMetadata applicable to this pre-chunk as a single chunk. Formed by applying consolidation rules to all metadata fields across the elements of this pre-chunk. For the sake of consistency, the same rules are applied (for example, for dropping values) to a single-element pre-chunk too, even though metadata for such a pre-chunk is already "consolidated". r)r _meta_kwargsr?s r3rz#TextPreChunk._consolidated_metadata0s33!2333r5Dict[str, List[RegexMetadata]]ci}t|jj}d}d}|jD]}t|j}|s||r|ndz }|}||z }|jjs6tj|jj}| D]Z\}} | D]"} | dxx|z cc<| dxx|z cc<#| |g} | | | ||<[|S)aConsolidate the regex-metadata in `regex_metadata_dicts` into a single dict. This consolidated value is suitable for use in the chunk metadata. `start` and `end` offsets of each regex match are also adjusted for their new positions. rstartend) rcr]rKrrrregex_metadatarrrgetextend) r2chunk_regex_metadata separator_lenrunning_text_len start_offsetrtext_lenelement_regex_metadata regex_namematchesm chunk_matchess r3_consolidated_regex_metaz%TextPreChunk._consolidated_regex_meta=sH@BDJ566  ~ A AG7<((H  1A H q H +L  ( #2 &*]73C3R%S%S "'='C'C'E'E A A# G --AgJJJ,.JJJeHHH ,HHHH 4 8 8R H H $$W---3@$Z00  A$#r5Dict[str, Any]czttjdfd }t|S)a$The consolidated metadata values as a dict suitable for constructing ElementMetadata. This is where consolidation strategies are actually applied. The output is suitable for use in constructing an `ElementMetadata` object like `ElementMetadata(**self._meta_kwargs)`. r6rc 3 KjD]\}}|}|jur ||dfV0|jur3|t |t ttgfVl|j ur2d|D}|t| fV|j ur |j fV|jurtdt!|ddS)zKGenerate (field-name, value) pairs for each field in consolidated metadata.rci|] }|D]}|d Sr+r)rsval_listkeys r3 zGTextPreChunk._meta_kwargs..iter_kwarg_pairs..xs)*`*`*`W_*`*`PS3*`*`*`*`r5zmetadata field z& has no defined consolidation strategyN)rrrFIRSTLIST_CONCATENATEsumrr r LIST_UNIQUErkeysREGEXrDROPNotImplementedErrorrepr)rvaluesstrategyordered_unique_keysCSfield_consolidation_strategiesr2s r3iter_kwarg_pairsz3TextPreChunk._meta_kwargs..iter_kwarg_pairslsN&*&?&E&E&G&G  " F9==jIIrx''$fQi/////!444$c&$tCy"2E2E&F&FFFFFF//*`*`V*`*`*`'$d+>+C+C+E+E&F&FFFFFF))$d&CCCCCC((.b$z*:*:bbb'  r5)r6r)rrr)r2rrrs` @@r3rzTextPreChunk._meta_kwargsbs]#)>)])_)_&        2$$&&'''r5rIcb|jj}|d|jDS)zThe concatenated text of all elements in this pre-chunk. Each element-text is separated from the next by a blank line (" "). c32K|]}|j |jVdSr+)r)rsrs r3ruz%TextPreChunk._text..s+"L"LaQV"L16"L"L"L"L"L"Lr5)r]rKjoinr)r2rKs r3rzTextPreChunk._texts3 2"""L"L4>"L"L"LLLLr5N)rrr[rr6rO)rrr6r%)rrr6r)r6rrR)r6r)r6r)r6r)r6rrS)rUrVrWrXr4rrrrrrrrrrrr5r3rrs;1111 YYYY > > > >\  " " "\ "D 4 4 4\ 4"$"$"$\"$H"("("(\"(HMMM\MMMr5rcjeZdZdZddZdd Zdd Zdd ZeddZ ddZ eddZ dS)raAn element accumulator suitable for incrementally forming a pre-chunk. Provides the trial method `.will_fit()` a pre-chunker can use to determine whether it should add the next element in the element stream. `.flush()` is used to build a PreChunk object from the accumulated elements. This method returns an iterator that generates zero-or-one `TextPreChunk` or `TablePreChunk` object and is used like so: yield from builder.flush() If no elements have been accumulated, no `PreChunk` instance is generated. Flushing the builder clears the elements it contains so it is ready to build the next pre-chunk. r[rr6rOcp||_t|j|_g|_g|_d|_dS)Nr)r]rcrK_separator_lenr_text_segments _text_lenr^s r3r4zPreChunkBuilder.__init__s8 !$"566(**,r5rrc|j||jrC|j|j|xjt |jz c_dSdS)zAdd `element` to this section.N)rrrr rrcr2rs r3rzPreChunkBuilder.add_elementsc g&&& < 0   & &w| 4 4 4 NNc',// /NNNN 0 0r5rc#K|jsdSt|jdtr t|jd|jn&t t |j|j}||VdS)aGenerate zero-or-one `PreChunk` object and clear the accumulator. Suitable for use to emit a PreChunk when the maximum size has been reached or a semantic boundary has been reached. Also to clear out a terminal pre-chunk at the end of an element stream. Nr)rrrrr]rr _reset_stater2 pre_chunks r3rzPreChunkBuilder.flushs~  F$.+U33 @M$.+TZ 8 8 8d4>22DJ??  r5r%c&t|jdkrdSt|trdS|jr"t|jdtrdS|j|jjkrdS|jt|jkrdSdS)aUTrue when `element` can be added to this prechunk without violating its limits. There are several limits: - A `Table` element will never fit with any other element. It will only fit in an empty pre-chunk. - No element will fit in a pre-chunk that already contains a `Table` element. - A text-element will not fit in a pre-chunk that already exceeds the soft-max (aka. new_after_n_chars). - A text-element will not fit when together with the elements already present it would exceed the hard-max (aka. max_characters). rTF) rcrrr _text_lengthr]r<_remaining_spacerrs r3rzPreChunkBuilder.will_fits t~  ! # #4 gu % % 5 > j):EBB 5  tz2 2 25  3w|#4#4 4 45tr5r#ch|jt|jz}|jj|jz |z S)zMMaximum text-length of an element that can be added without exceeding maxlen.)r rcr r]r@r)r2separators_lens r3rz PreChunkBuilder._remaining_spaces3,s43F/G/GGz"T^3nDDr5cx|j|jd|_dS)zMSet working-state values back to "empty", ready to accumulate next pre-chunk.rN)rclearr rr?s r3rzPreChunkBuilder._reset_states6  !!###r5c`t|j}|r|dz nd}|j||jzzS)aLength of the text in this pre-chunk. This value represents the chunk-size that would result if this pre-chunk was flushed in its current state. In particular, it does not include the length of a trailing separator (since that would only appear if an additional element was added). Not suitable for judging remaining space, use `.remaining_space` for that value. rzr)rcr rr )r2nseparator_counts r3rzPreChunkBuilder._text_lengths; # $ $#$+!a%%!~43F!FGGr5Nr[rr6rO)rrr6rOrrrRrT) rUrVrWrXr4rrrpropertyrrrrr5r3rrs      0000*:EEEXE  HHHXHHHr5rc"eZdZdZd dZd d Zd S) PreChunkCombinerzDFilters pre-chunk stream to combine small pre-chunks where possible. pre_chunksIterable[PreChunk]r[rc"||_||_dSr+) _pre_chunksr])r2r#r[s r3r4zPreChunkCombiner.__init__s% r5r6rc#Kt|j}|jj}|jD]{}t |t s|j|ks|j|jkr|Ed{Vt |t r|Vf| |||Ed{VdS)zVGenerate pre-chunk objects, combining TextPreChunk objects when they'll fit in window.N) TextPreChunkAccumulatorr]r r&rrrremaining_spacer add_pre_chunk)r2accumr rs r3iter_combined_pre_chunksz)PreChunkCombiner.iter_combined_pre_chunkss' 33%)Z%J") / /I9m44 )$(BBB(9+@@@ ;;==((((((()]33 /##I....;;==         r5N)r#r$r[rr)rUrVrWrXr4r,rr5r3r"r" sBNN!!!!!!r5r"cZeZdZdZddZdd Zdd Zedd ZeddZ dS)r(aAccumulates, measures, and combines text pre-chunks. Used for combining pre-chunks for chunking strategies like "by-title" that can potentially produce undersized chunks and offer the `combine_text_under_n_chars` option. Note that only sequential `TextPreChunk` objects can be combined. A `TablePreChunk` is never combined with another pre-chunk. Provides `.add_pre_chunk()` allowing a pre-chunk to be added to the chunk and provides monitoring properties `.remaining_space` and `.text_length` suitable for deciding whether to add another pre-chunk. `.flush()` is used to combine the accumulated pre-chunks into a single `TextPreChunk` object. This method returns an interator that generates zero-or-one `TextPreChunk` objects and is used like so: yield from accum.flush() If no pre-chunks have been accumulated, no `TextPreChunk` is generated. Flushing the builder clears the pre-chunks it contains so it is ready to accept the next text-pre-chunk. r[rr6rOc"||_g|_dSr+)r]r&r^s r3r4z TextPreChunkAccumulator.__init__Bs /1r5rrc:|j|dS)zPAdd a pre-chunk to the accumulator for possible combination with next pre-chunk.N)r&rrs r3r*z%TextPreChunkAccumulator.add_pre_chunkFs  *****r5Iterator[TextPreChunk]c#K|j}|sdS|d}|ddD]}||}|V|dS)zCGenerate all accumulated pre-chunks as a single combined pre-chunk.Nrrz)r&rr)r2r#rrs r3rzTextPreChunkAccumulator.flushJsw%   FqM )!""~ ; ;O!))/::II r5r#ct|jj}|js|n#||jz t |jjz S)zEMaximum size of pre-chunk that can be added without exceeding maxlen.)r]r@r&rrcrK)r2rhs r3r)z'TextPreChunkAccumulator.remaining_space[sD$# LFF$**S1J-K-KK  r5ct|j}|dkrdStd|jD}t|jj|dz z}||zS)z;Size of concatenated text in all pre-chunks in accumulator.rc3$K|] }|jV dSr+)r)rsr_s r3ruz6TextPreChunkAccumulator.text_length..ns$HH! HHHHHHr5rz)rcr&rr]rK)r2rtotal_text_lengthtotal_separator_lengths r3rz#TextPreChunkAccumulator.text_lengthfsj  ! ! 661HHt7GHHHHH!$TZ%>!?!?1q5!I #999r5Nr)rrr6rO)r6r0rR) rUrVrWrXr4r*rr r)rrr5r3r(r(,s*2222++++"   X  : : :X : : :r5r(r6c dddfd }|S) aNot a predicate itself, calling this returns a predicate that triggers on each new section. The lifetime of the returned callable cannot extend beyond a single element-stream because it stores current state (current section) that is particular to that element stream. A "section" of this type is particular to the EPUB format (so far) and not to be confused with a "section" composed of a section-heading (`Title` element) followed by content elements. The returned predicate tracks the current section, starting at `None`. Calling with an element with a different value for `metadata.section` returns True, indicating the element starts a new section boundary, and updates the enclosed section name ready for the next transition. NTrrr6r%cL|jj}r|ddS|dS|krdS|dS)NFT)rsection)rr9current_sectionis_firsts r3section_changedz+is_in_next_section..section_changedsQ"*  %OH5 ?5 o % %5!tr5rr)r<r:r;s @@r3is_in_next_sectionr=s;&*OH4 r5c dddfd }|S) aNot a predicate itself, calling this returns a predicate that triggers on each new page. The lifetime of the returned callable cannot extend beyond a single element-stream because it stores current state (current page-number) that is particular to that element stream. The returned predicate tracks the "current" page-number, starting at 1. An element with a greater page number returns True, indicating the element starts a new page boundary, and updates the enclosed page-number ready for the next transition. An element with `page_number == None` or a page-number lower than the stored value is ignored and returns False. rzTrrr6r%cP|jj}r|pdddS|dS|krdS|dS)NrzFT)r page_number)rr@current_page_numberr;s r3page_number_incrementedz0is_on_next_page..page_number_incrementedsZ&2  "-"2 H5  5 - - -5*tr5rr)rBrAr;s @@r3is_on_next_pagerCs< !H6 #"r5rrr%c,t|tS)z:True when `element` is a `Title` element, False otherwise.)rr)rs r3is_titlerEs gu % %%r5)r6rr)/rX __future__rrrtypingrrrrrr r r r r rrqtyping_extensionsrrunstructured.documents.elementsrrrrrrrrunstructured.utilsrr%r__annotations__rrrGrrrrr"r(r=rCrErr5r3rLs^FFF""""""                            --------                    ,+++++' 488888X444443EeEeEeEeEeEeEeEePq.q.q.q.q.q.q.q.rC(C(C(C(C(C(C(C(V&#&#&#&#&#&#&#&#RiMiMiMiMiMiMiMiMhoHoHoHoHoHoHoHoHd!!!!!!!!BD:D:D:D:D:D:D:D:t****Z+#+#+#+#\&&&&&&r5