j>`UddlmZddlZddlmZmZmZmZmZm Z m Z m Z ej dkrddl mZnddlmZddlmZddlmZmZddlmZdd lmZmZmZmZmZmZmZmZmZm Z dd l!m"Z"m#Z#dd l$m%Z%dd l&m'Z'm(Z(m)Z)m*Z*m+Z+dd l,m-Z-gdZ.de/d<ddgZ0de/d<gdZ1de/d<gdZ2de/d<gdZ3de/d<dgZ4de/d<dgZ5de/d<e5e4zZ6de/d<dd gZ7de/d!<d"d#gZ8de/d$<Gd%d&Z9Gd'd(e9eZ:Gd)d*e9eZ;Gd+d,e9eZ<Gd-d.e9e Z=Gd/d0e9eZ>Gd1d2e9eZ?Gd3d4e9eZ@dnd8ZAdnd9ZBdnd:ZCGd;d> "&( "57&*************r^rJceZdZdZdS)HTMLTextzText with tag information.Nr_r`rarbrKr^r]rfrfOs$$$$r^rfceZdZdZdS) HTMLAddresszAddress with tag information.NrgrKr^r]ririSs''''r^riceZdZdZdS)HTMLEmailAddressz!EmailAddress with tag informationNrgrKr^r]rkrkWs++++r^rkceZdZdZdS) HTMLTitlezTitle with tag information.NrgrKr^r]rmrm[s%%%%r^rmceZdZdZdS)HTMLNarrativeText#NarrativeText with tag information.NrgrKr^r]roro_----r^roceZdZdZdS) HTMLListItemrpNrgrKr^r]rsrscrqr^rsceZdZdZdS) HTMLTablez"NarrativeText with tag informationNrgrKr^r]rurugs,,,,r^ruelementreturnboolc>td|jDS)zChecks to see if an element has ancestors that are table elements. If so, we consider it to be a table element rather than a section of narrative text.c3(K|] }|tvVdSN)r=.0ancestors r] z%has_table_ancestor..ns'KK(x:%KKKKKKr^anyrMrvs r]has_table_ancestorrks$ KKg6JKKK K KKr^cFtd|jDrdSdS)zIChecks to see if an element is contained within a header or a footer tag.c3(K|] }|tvVdSr{)rEr|s r]rz&in_header_or_footer..ss( R R8, , R R R R R Rr^TFrrs r]in_header_or_footerrqs. R RW=Q R R RRRt 5r^c|jtvS)z&Checks to see if an element is a table)rLr=rs r]is_tablerxs ;* $$r^cDeZdZdZ ddfd Zdd Z dddZxZS) HTMLDocumentzuClass for handling HTML documents. Uses rules based parsing to identify sections of interest within the document.NT stylesheetrRparserrassemble_articlesrxc\||_t||dS)N)rr)assembled_articlesrYrZ)r[rrrr\s r]rZzHTMLDocument.__init__s0 #4 Jv>>>>>r^rw List[Page]c |jr|jStjdg}tj|jddgdt |j}t||j}d}t|}|D]}d }| D]}||vrt|rt|r@t|} | D]-} t| } | |j| .n+t|} | |j| t#|}t'|r_t)|} t+|} |jJt/|jd d d| | } | |j| !t1|rLt3|}|j|t#|}|t7|r@t9|\} }| *|j| t;||}|jt>vr[tA|} | |j| | s |jd kr!t#|}4|jtBvrBtE|jdkr*|||dz }t|}tE|jdkr*|||dz }t|}|S)aParse HTML elements into pages. A *page* is a subsequence of the document-elements parsed from the HTML document corresponding to a distinct topic. At present pagination is determined by `
` elements that surround something like a blog-post. Each article becomes its own page. If no article tags are present in the HTML the entire HTML document is a single page. zReading document ...scriptstyleF) with_tail)rrnumberrKNrF)depthrNrOr:)#_pagesrinforstrip_elements document_tree _find_main_find_articlesrriter _is_text_tag_has_break_tags_unfurl_break_tags _parse_tagelementsappendtupleiterdescendants_is_container_with_text_get_links_from_tag_get_emphasized_texts_from_tagtext_text_to_element_is_bulleted_table_bulleted_text_from_tableextendis_list_item_tag_process_list_item_get_bullet_descendantsrLr= _parse_HTMLTable_from_table_elemrAlen)r[pagesrootarticles page_numberpagearticledescendanttag_elemstag_elemflattened_elems _tag_elemrvrNrO bulleted_text next_elements r]_parse_pages_from_element_treez+HTMLDocument._parse_pages_from_element_trees ; ;  *+++ T/(G1DPUVVVV$,--!$$:QRRR ;'''C 0C 0G>@ #LLNN< 4< 4222))64&x00 :*>I&0&;&;G&2 $ 4 4W = = => #-X"6"6". M00999*/0H0H0J0J*K*K'',X66(4/99E'Eh'O'O$#=444. #)9 G* ,,W555'114$=h$G$GMM((777*/0H0H0J0J*K*K''%h//4,>x,H,H)G\* ,,W555.E$(//+ \Z//>xHHG* ,,W555P(,'"9"9.3H4L4L4N4N.O.O+\^33DM8J8JQ8N8NLL&&&1$K{333D4=!!A%% T"""q ;/// r^Fskip_headers_and_footers skip_tableinplacec  g}|r|t|r|tg}d}t|}|jD]}g} |jD] t ts'td|j dt t fd|Ds| |r!dt j  jfzvrn| r1| |_|||dz }t|}|r||_d|_|S|j |} t | t$std |j j| S) aFilters elements returning new instance based on the criteria specified. Note that the number of pages can change in the case that all elements on a page are filtered out. Parameters ---------- skip_table: If True, skips table element skip_headers_and_footers: If True, ignores any content that is within
or