#j2dZddlmZddlZddlZddlmZmZddlm Z ddl m Z ddl m Z mZmZmZddlmZmZdd lmZmZe rdd lmZmZmZmZdd lmZ ddlZd Zn #e $rd ZYnwxYw ddl!m"Z"d Z#n #e $rd Z#YnwxYwej$e%Z&eddZ'GddeeZ(Gdde(Z)Gdde*e Z+e d GddZ,d dZ-dS)!zText splitter base interface.) annotationsN)ABCabstractmethod) dataclass)Enum) TYPE_CHECKINGAnyLiteralTypeVar)BaseDocumentTransformerDocument)Selfoverride)Callable CollectionIterableSequence)SetTF)PreTrainedTokenizerBaseTS TextSplitter)boundceZdZdZddedddfd8dZed9dZ d:d;dZdd&Z e d?d+Z e d,ded-fd@d5ZedAd7ZdS)Brz)Interface for splitting text into chunks.iFT chunk_sizeint chunk_overlaplength_functionCallable[[str], int]keep_separatorbool | Literal['start', 'end']add_start_indexboolstrip_whitespacereturnNonec|dkrd|}t||dkrd|}t|||krd|d|d}t|||_||_||_||_||_||_dS)aLCreate a new TextSplitter. Args: chunk_size: Maximum size of chunks to return chunk_overlap: Overlap in characters between chunks length_function: Function that measures the length of given chunks keep_separator: Whether to keep the separator and where to place it in each corresponding chunk `(True='start')` add_start_index: If `True`, includes chunk's start index in metadata strip_whitespace: If `True`, strips whitespace from the start and end of every document rzchunk_size must be > 0, got z chunk_overlap must be >= 0, got zGot a larger chunk overlap (z) than chunk size (z), should be smaller.N) ValueError _chunk_size_chunk_overlap_length_function_keep_separator_add_start_index_strip_whitespace)selfrrrr r"r$msgs g/lsinfo/ai/hellotax_ai/base_platform/venv/lib/python3.11/site-packages/langchain_text_splitters/base.py__init__zTextSplitter.__init__/s* ??===CS// ! 1  D]DDCS// ! : % %6}66666 S// !%+ /- /!1textstr list[str]cdS)z$Split text into multiple components.N)r/r4s r1 split_textzTextSplitter.split_textWsr3Ntexts metadataslist[dict[Any, Any]] | Nonelist[Document]c |pigt|z}g}t|D]\}}d}d}||D]} tj||} |jrE||z|jz } || td| }|| d<t| }t| | } | | |S)z9Create a list of `Document` objects from a list of texts.r start_index) page_contentmetadata) len enumerater9copydeepcopyr-r*findmaxr append) r/r:r; metadatas_ documentsir4indexprevious_chunk_lenchunkrAoffsetnew_docs r1create_documentszTextSplitter.create_documents[s32$U"3   '' * *GAtE!" .. * *=A77(4"%77$:MMF IIeSF^^<--i88 !#  K KA((++D [1A1AA1E1E 1M"##4+++NN'(  {##a''//+yAAC C((( $"555 [9I9IA9M9M STU*++!AII!6!6{1~!F!F-0-=-=-A-AMMq"'2!""o  $"555 [9I9IA9M9M STU*++!AII   q ! ! ! Tc+.>.>.B.B]]J JEEook955 ? KK    r3 tokenizerrkwargsr c tsd}t|ttsd}t|d fd }|d d|i|S) z?Text splitter that uses Hugging Face tokenizer to count length.z`Could not import transformers python package. Please install it with `pip install transformers`.zATokenizer received was not an instance of PreTrainedTokenizerBaser4r5r%rcHt|SrZ)rBtokenizer4rjs r1_huggingface_tokenizer_lengthzNTextSplitter.from_huggingface_tokenizer.._huggingface_tokenizer_lengthsy))$//00 0r3rr4r5r%rr8)_HAS_TRANSFORMERSr( isinstancer)clsrjrkr0rps ` r1from_huggingface_tokenizerz'TextSplitter.from_huggingface_tokenizers ! "E S// !)%<== "UCS// ! 1 1 1 1 1 1sKK#@KFKKKr3gpt2all encoding_name model_nameallowed_special!Literal['all'] | AbstractSet[str]disallowed_special Literal['all'] | Collection[str]rc  tsd}t||tj| ntj| d  fd }t |t r ||d}i||}|d d |i|S) z;Text splitter that uses `tiktoken` encoder to count length.zCould not import tiktoken python package. This is needed in order to calculate max_tokens_for_prompt. Please install it with `pip install tiktoken`.Nr4r5r%rcNt|SN)rzr|)rBencode)r4rzr|encs r1_tiktoken_encoderz=TextSplitter.from_tiktoken_encoder.._tiktoken_encoders4 $3'9 r3)rxryrzr|rrqr8) _HAS_TIKTOKEN ImportErrortiktokenencoding_for_model get_encoding issubclassTokenTextSplitter) rtrxryrzr|rkr0r extra_kwargsrs `` @r1from_tiktoken_encoderz"TextSplitter.from_tiktoken_encoders #A  c"" "  !-j99CC' 66C         c, - - 0!.(#2&8 L 0/,/Fs??#4????r3Sequence[Document]c F|t|S)z2Transform sequence of documents by splitting them.)rUlist)r/rJrks r1transform_documentsz TextSplitter.transform_documentss ##DOO444r3)rrrrrrr r!r"r#r$r#r%r&r4r5r%r6rZ)r:r6r;r<r%r=)rJrRr%r=)rVr6rWr5r%rX)r^r_rWr5r%r6)rjrrkr r%r) rxr5ryrXrzr{r|r}rkr r%r)rJrrkr r%r)__name__ __module__ __qualname____doc__rBr2rr9rQrUr]ri classmethodrusetrrrr8r3r1rr,sL33 039> %!%&2&2&2&2&2P333^3JN(AAAA ****XLLL[L($!%=@SUU?D (@(@(@(@[(@T555X555r3cFeZdZdZddedfdfd ZddZxZS)rz/Splitting text to tokens using model tokenizer.rvNrwrxr5ryrXrzr{r|r}rkr r%r&c tjdi|tsd}t||t j|}nt j|}||_||_||_ dS)zCreate a new TextSplitter.zCould not import tiktoken python package. This is needed in order to for TokenTextSplitter. Please install it with `pip install tiktoken`.Nr8) superr2rrrrr _tokenizer_allowed_special_disallowed_special) r/rxryrzr|rkr0r __class__s r1r2zTokenTextSplitter.__init__s ""6""" #A  c"" "  !-j99CC' 66C /#5   r3r4r6cdfd }tjjjj|}t ||S) arSplits the input text into smaller chunks based on tokenization. This method uses a custom tokenizer configuration to encode the input text into tokens, processes the tokens in chunks of a specified size with overlap, and decodes them back into text chunks. The splitting is performed using the `split_text_on_tokens` function. Args: text: The input text to be split into smaller chunks. Returns: A list of text chunks, where each chunk is derived from a portion of the input text based on the tokenization and chunking rules. _textr5r% list[int]cRj|jjSr)rrrr)rr/s r1_encodez-TokenTextSplitter.split_text.._encodes1?)) $ 5#'#;* r3)rtokens_per_chunkdecoderro)rr5r%r) Tokenizerr*r)rrsplit_text_on_tokens)r/r4rrjs` r1r9zTokenTextSplitter.split_textsc       -!-?)    $CCCCr3) rxr5ryrXrzr{r|r}rkr r%r&r)rrrrrr2r9 __classcell__)rs@r1rrsx99$!%=@SUU?D 66666664DDDDDDDDr3rceZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z d Zd ZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdS)Languagez"Enum of the programming languages.cppgojavakotlinjstsphpprotopythonrrstrubyrustscalaswiftmarkdownlatexhtmlsolcsharpcobolcluaperlhaskellelixir powershell visualbasic6N) rrrrCPPGOJAVAKOTLINJSrPHPPROTOPYTHONRRSTRUBYRUSTSCALASWIFTMARKDOWNLATEXHTMLSOLCSHARPCOBOLCLUAPERLHASKELLELIXIR POWERSHELL VISUALBASIC6r8r3r1rr/s,, C B D F B B C E F A C D D E EH E D C F E A C DG FJ!LLLr3r)frozencBeZdZUdZded< ded< ded< ded<d S) rzTokenizer data class.rrrzCallable[[list[int]], str]rzCallable[[str], list[int]]rN)rrrr__annotations__r8r3r1rrPsQ*,&&&&=&&&&==r3rr4r5rjr%r6cg}||}d}|j|jkrd}t||t |krt ||jzt |}|||}|sne||}|r|||t |krn%||j|jz z }|t |k|S)z6Split incoming text and return chunks using tokenizer.rz3tokens_per_chunk must be greater than chunk_overlap)rrrr(rBminrrH) r4rjr^ input_ids start_idxr0cur_idx chunk_idsdecodeds r1rr^sF  &&II!Y%<<<Coo c)nn $ $i)"<rs##"""""" ########!!!!!!GFFFFFFF,,,,,,,,3HHHHHHHHHHHH222222OOOMMMMMLLLLLL  8 $ $ WT(((B5B5B5B5B5*CB5B5B5J;D;D;D;D;D ;D;D;D|"""""sD"""B $ > > > > > > > >s$AA'&A'+A44A>=A>