jf dZddlmZddlmZmZmZmZddlm Z m Z m Z m Z m Z mZmZddlmZddlmZ dddZGdde ZdS)zdImplementation of chunking by title. Main entry point is the `@add_chunking_strategy()` decorator. ) annotations)IteratorListOptionalTuple)BasePreChunkerBoundaryPredicateChunkingOptionsPreChunkCombineris_in_next_sectionis_on_next_pageis_title)Element) lazypropertyTNelements List[Element]multipage_sectionsboolcombine_text_under_n_chars Optional[int]new_after_n_charsmax_charactersintreturnctj||||}tt|||}d|DS)aUses title elements to identify sections within the document for chunking. Splits off into a new CompositeElement when a title is detected or if metadata changes, which happens when page numbers or sections change. Cuts off sections once they have exceeded a character length of max_characters. Parameters ---------- elements A list of unstructured elements. Usually the output of a partition function. multipage_sections If True, sections can span multiple pages. Defaults to True. combine_text_under_n_chars Combines elements (for example a series of titles) until a section reaches a length of n characters. Defaults to `max_characters` which combines chunks whenever space allows. Specifying 0 for this argument suppresses combining of small chunks. Note this value is "capped" at the `new_after_n_chars` value since a value higher than that would not change this parameter's effect. new_after_n_chars Cuts off new sections once they reach a length of n characters (soft max). Defaults to `max_characters` when not specified, which effectively disables any soft window. Specifying 0 for this argument causes each element to appear in a chunk by itself (although an element with text longer than `max_characters` will be still be split into two or more chunks). max_characters Chunks elements text and text_as_html (if present) into chunks of length n characters (hard max) )rrrr)optsc@g|]}|D]}|S) iter_chunks).0 pre_chunkchunks e/lsinfo/ai/hellotax_ai/base_platform/venv/lib/python3.11/site-packages/unstructured/chunking/title.py z"chunk_by_title..Es1 S S Si9;P;P;R;R S S%E S S S S)r newr _ByTitlePreChunkeriter_pre_chunksiter_combined_pre_chunks)rrrrrr pre_chunkss r$chunk_by_titler,syF  #=%-+    D"**8T::   T S: S S SSr&c*eZdZdZeddZdS)r(zPre-chunker for the "by_title" chunking strategy. The "by-title" strategy specifies breaking on section boundaries; a `Title` element indicates a new "section", hence the "by-title" designation. rTuple[BoundaryPredicate, ...]c>dfd }t|S)zBThe semantic-boundary detectors to be applied to break pre-chunks.rIterator[BoundaryPredicate]c3zKtVtVjjst VdSdS)N)rr _optsrr )selfsr$iter_boundary_predicateszI_ByTitlePreChunker._boundary_predicates..iter_boundary_predicatesSsQNNN$&& & & &:0 (%''''''' ( (r&)rr0)tuple)r3r4s` r$_boundary_predicatesz'_ByTitlePreChunker._boundary_predicatesOs; ( ( ( ( ( ( --//000r&N)rr.)__name__ __module__ __qualname____doc__rr6rr&r$r(r(Hs>  1 1 1\ 1 1 1r&r()TNNr) rrrrrrrrrrrr)r: __future__rtypingrrrrunstructured.chunking.baserr r r r r runstructured.documents.elementsrunstructured.utilsrr,r(rr&r$r@s( #"""""222222222222433333++++++ $04'+ .T.T.T.T.Tb1111111111r&