o Ղig9 @s:UdZddlmZddlZddlZddlZddlZddlmZm Z m Z ddl m Z ddl mZddddd d d d dd Zd ed <d ZGdddZiZded<eZd:ddZddddddddddd dd!d"d dd#d$ddd%d&d dd'd(ddd)dd*dgZd+d,d-d.d/d0d1d2d3d Zd4ZGd5d6d6Zd;d8d9ZdS)@z(_TokenBucket.acquire..rg?u[rate-limit] 令牌桶满（/u/min），等待 .2fscr!rrr"r%rrr(Nr)) rtime monotonicrlenrrdebugrsleepappend)roldestwaitrr%racquire9s& .z_TokenBucket.acquireN)rr)__name__ __module__ __qualname____doc__r r5rrrrr,s  rzDict[tuple, _TokenBucket]_bucketsdomainstr category_idrreturnc s||f}t4IdH.|tvr+t|t}t|t|<td|d|d|dt|WdIdHS1IdHs?wYdS)Nu$[rate-limit] 新建令牌桶 domain=z category=z rate=z/min) _buckets_lockr:rget _DEFAULT_RATErrr0)r;r=keyrrrr _get_bucket[s  0rCzoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36z#zh-CN,zh;q=0.9,en-US;q=0.8,en;q=0.7Win32)ualangplatformzoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36zzh-CN,zh;q=0.9,en;q=0.8zuMozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36MacIntelzuMozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36zzh-CN,zh;q=0.9zPMozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0z;zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2zTMozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:121.0) Gecko/20100101 Firefox/121.0z#zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3z}Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 Edg/120.0.0.0z/zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6zeMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36z Linux x86_64z:https://fgk.chinatax.gov.cn/zcfgk/c100009/listflfg_fg.htmlz:https://fgk.chinatax.gov.cn/zcfgk/c100010/listflfg_fg.htmlz7https://fgk.chinatax.gov.cn/zcfgk/c102440/listflfg.htmlz?@rNfgk.chinatax.gov.cn delay_minfloat delay_maxr=r proxy_listOptional[List[str]]r;r<max_requests_per_minute Optional[int]cCsd||_||_||_|p g|_||_||d}||_||d|_||_g|_|p.t |t |_ dS)u Args: delay_min: 最小延迟（秒） delay_max: 最大延迟（秒） category_id: 当前采集的分类ID（0=未指定，使用默认速率） proxy_list: 代理列表（可选） domain: 目标域名（令牌桶分组键之一） max_requests_per_minute: 兼容旧参数（若提供则覆盖按分类的配置） r rN) rNrPr=rQr; _delay_mu _delay_sigma_override_rate request_timesrr@rArS)rrNrPr=rQr;rSmidrrrr s   zAntiBlockingStrategy.__init__urlcsd|rt|n|j}|j}t||IdH}|jr"|j|jkr"|j|_|IdH|IdHdS)u3请求前：先令牌桶限流，再随机延迟。N)_extract_domainr;r=rCrWrr5_apply_normal_delay)rr[r;r=bucketrrrbefore_requestsz#AntiBlockingStrategy.before_requestr>dictc CsLtt}|dur |n|j}t|t}|dd|dd|dddd d d d S) u 获取随机请求头，包含真实的 Accept-Language / Referer。 Args: category_id: 用于选择对应 Referer；None 时使用实例的 category_id NrEzUtext/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8rFzgzip, deflate, brz keep-alive1z max-age=0documentnavigatez same-origin) z User-AgentAcceptzAccept-LanguagezAccept-EncodingReferer ConnectionzUpgrade-Insecure-Requestsz Cache-ControlzSec-Fetch-DestzSec-Fetch-ModezSec-Fetch-Site)randomchoice _UA_PROFILESr= _REFERERSr@_DEFAULT_REFERER)rr=profilecidrefererrrr get_headerss  z AntiBlockingStrategy.get_headers Optional[str]cCs|jsdSt|jS)u2获取随机代理（无代理时返回 None）。N)rQrgrhrrrr get_proxys zAntiBlockingStrategy.get_proxycs|IdHdS)u9兼容旧接口（旧代码直接调用 apply_delay）。N)r]rqrrr apply_delaysz AntiBlockingStrategy.apply_delaycsdS)uM兼容旧接口 — 现在由令牌桶统一管理，此方法为空操作。Nrrqrrrcheck_rate_limitsz%AntiBlockingStrategy.check_rate_limitcsVt|j|j}t|jt|j|}t d|dd|j dt |IdHdS)u[正态分布随机延迟（μ=中点，σ=范围/4），截断到 [delay_min, delay_max]。u[anti-block] 延迟 r+z s (category=)N) rggaussrUrVmaxrNminrPrr0r=rr1)rdelayrrrr]s z(AntiBlockingStrategy._apply_normal_delay)rKrLrNrMN) rNrOrPrOr=rrQrRr;r<rSrT)rZ)r[r<r)r=rTr>r`)r>rp) r6r7r8r9ri USER_AGENTSr r_rorrrsrtr]rrrrrIs  ,    rIr[cCs(zt|jp|WSty|YSw)u5从 URL 提取域名，失败时返回原字符串。)rnetloc Exception)r[rrrr\*s  r\)r;r<r=rr>r)r[r<r>r<)r9 __future__rrmathrgr-typingrrr urllib.parserlogururr__annotations__rArr:rr?rCrirjrkrIr\rrrrs     + -