o ri>8@sddlZddlmZddlmZddlmZmZmZm Z m Z m Z m Z m Z mZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZddlmZddl m!Z!dd l"m#Z#dd l$m%Z%dd l&m'Z'dd l(m)Z)dd l*m+Z+ddl,m-Z-ddl.m/Z/ddl0m1Z1ddl2m3Z3ddl4m5Z5ddl6m7Z7ddl8m9Z9ddl:m;Z;ddlm?Z?ddl@mAZAddlBmCZCddlDmEZEddlFmGZGddlHmIZIddlJmKZKddlLmMZMide%d e)d!e+d"e9d#e!d$eCd%eEd&e5d'e7d(e/d)e;d*e=d+e-d,e1d-eKd.e'd/eMeIe3e#eAe?eGd0ZNided e d!e d(e d"ed#ed$ed%ed&ed'ed*ed+e d,e d)ed-ed.e d/eeeeeeed0ZOeeee e eeefZPeQeRZSGd1d2d2ZTGd3d4d4ZUd5eVfd6d7ZWd8eVfd9d:ZXd;d<ZYdS)=N) AutoConfig)logging) AqlmConfigAutoRoundConfig AwqConfigBitNetQuantConfigBitsAndBytesConfigCompressedTensorsConfig EetqConfigFbgemmFp8ConfigFineGrainedFP8ConfigFourOverSixConfig FPQuantConfig GPTQConfig HiggsConfig HqqConfig MetalConfig Mxfp4ConfigQuantizationConfigMixinQuantizationMethod QuantoConfig QuarkConfig SinqConfig SpQRConfig TorchAoConfig VptqConfig) HfQuantizer)AqlmHfQuantizer)AutoRoundQuantizer) AwqQuantizer)BitNetHfQuantizer)Bnb4BitHfQuantizer)Bnb8BitHfQuantizer)CompressedTensorsHfQuantizer)EetqHfQuantizer)FbgemmFp8HfQuantizer)FineGrainedFP8HfQuantizer)FourOverSixHfQuantizer)FPQuantHfQuantizer)GptqHfQuantizer)HiggsHfQuantizer)HqqHfQuantizer)MetalHfQuantizer)Mxfp4HfQuantizer)QuantoHfQuantizer)QuarkHfQuantizer)SinqHfQuantizer)SpQRHfQuantizer)TorchAoHfQuantizer)VptqHfQuantizerawqbitsandbytes_4bitbitsandbytes_8bitgptqaqlmquantoquark fouroversixfp_quanteetqhiggshqqzcompressed-tensors fbgemm_fp8torchaobitnetvptq)spqrfp8z auto-roundmxfp4metalsinqc@s.eZdZdZedefddZeddZdS)AutoQuantizationConfigz The Auto-HF quantization config class that takes care of automatically dispatching to the correct quantization config given a quantization config stored in a dictionary. quantization_config_dictcCs|d}|dds|ddr!|ddrdnd}tj|}n|dur)td|tvr;td|d ttt|}||S) N quant_method load_in_8bitF load_in_4bit_4bit_8bitThe model's quantization config from the arguments has no `quant_method` attribute. Make sure that the model has been correctly quantizedUnknown quantization type, got  - supported types are: ) getrBITS_AND_BYTES ValueError AUTO_QUANTIZATION_CONFIG_MAPPINGlistAUTO_QUANTIZER_MAPPINGkeys from_dict)clsrLrMsuffix target_clsr`j/lsinfo/ai/hellotax_ai/llm_service/venv_embed/lib/python3.10/site-packages/transformers/quantizers/auto.pyr\s    z AutoQuantizationConfig.from_dictcKsVtj|fi|}t|dddurtd|d|j}||}|jdi||S)Nquantization_configz)Did not found a `quantization_config` in z2. Make sure that the model is correctly quantized.r`)rfrom_pretrainedgetattrrWrbr\update)r]pretrained_model_name_or_pathkwargs model_configrLrbr`r`rarcs  z&AutoQuantizationConfig.from_pretrainedN)__name__ __module__ __qualname____doc__ classmethoddictr\rcr`r`r`rarKs rKc@s\eZdZdZedeeBfddZeddZedeeBdedBfd d Z e d d Z dS) AutoHfQuantizerz The Auto-HF quantizer class that takes care of automatically instantiating to the correct `HfQuantizer` given the `QuantizationConfig`. rbcKst|tr t|}|j}|tjkr't|tstd|j r#|d7}n|d7}|t vr9t d|dt t t |}||fi|S)NzZFound `quant_method=bitsandbytes` but `quantization_config` is not a `BitsAndBytesConfig`.rQrPrSrT) isinstancernrKr\rMrrVr TypeErrorrNrZrWrYr[)r]rbrgrMr_r`r`ra from_configs&      zAutoHfQuantizer.from_configcKstj|fi|}||S)N)rKrcrr)r]rfrgrbr`r`rarcs zAutoHfQuantizer.from_pretrainedquantization_config_from_argsNcCs|durd}nd}t|trt|trt|}nt|}|dur9|jj|jjkr9td|jjd|jjdt|trdt|trd| }| D] \}}t |||qK|rd|dt | d7}|dkrwt|tttfswt||St||S) z handles situations where both quantization_config from args and quantization_config from model config are present. NzYou passed `quantization_config` or equivalent parameters to `from_pretrained` but the model you're loading already has a `quantization_config` attribute. The `quantization_config` from the model will be used.zThe model is quantized with z but you are passing a z| config. Please make sure to pass the same quantization config class to `from_pretrained` with different loading attributes.z"However, loading attributes (e.g. z]) will be overwritten with the one you passed to `from_pretrained`. The rest will be ignored.)rprnrr\rK __class__rirWLOADING_ATTRIBUTES_CONFIG_TYPESget_loading_attributesitemssetattrrYr[rrr warningswarnloggerinfo)r]rbrs warning_msgloading_attr_dictattrvalr`r`ramerge_quantization_configss4        z*AutoHfQuantizer.merge_quantization_configscCs|dd}|dds|ddr"|ddrdnd}tj|}n|dur*td|tvr@td|d tt d dSd S) NrMrNFrOrPrQrRrSrTz~. Hence, we will skip the quantization. To remove the warning, you can delete the quantization_config attribute in config.jsonT) rUrrVrWrXr|warningrYrZr[)rLrMr^r`r`rasupports_quant_method s   z%AutoHfQuantizer.supports_quant_method) rirjrkrlrmrrnrrrcr staticmethodrr`r`r`raros  1romethodcfdd}|S)z-Register a custom quantization configuration.c6tvr tddt|tstd|t<|S)NzConfig '' already registeredz*Config must extend QuantizationConfigMixin)rXrW issubclassrrqr]rr`raregister_config_fn$  z8register_quantization_config..register_config_fnr`)rrr`rraregister_quantization_config!  rnamecr)zRegister a custom quantizer.cr)Nz Quantizer 'rz!Quantizer must extend HfQuantizer)rZrWrrrqrrr`raregister_quantizer_fn4rz1register_quantizer..register_quantizer_fnr`)rrr`rraregister_quantizer1rrcCst|d}|rt|jsd}|s|dur,|r t|j||_n||_tj|j|d}nd}|dur[|j||d||}||}| |}t |jdds[|jj }t |d||d<|||fS)NrbF) pre_quantized) device_map weights_only dequantizevaluequant) hasattrrorrbrrrvalidate_environmentupdate_device_mapupdate_tp_planupdate_ep_planrdrM)configrbrr user_agentr hf_quantizerrMr`r`raget_hf_quantizerAs4      r)Zrzmodels.auto.configuration_autorutilsrutils.quantization_configrrrrr r r r r rrrrrrrrrrrrrrrbaserquantizer_aqlmrquantizer_auto_roundr quantizer_awqr!quantizer_bitnetr"quantizer_bnb_4bitr#quantizer_bnb_8bitr$quantizer_compressed_tensorsr%quantizer_eetqr&quantizer_fbgemm_fp8r'quantizer_finegrained_fp8r(quantizer_fouroversixr)quantizer_fp_quantr*quantizer_gptqr+quantizer_higgsr, quantizer_hqqr-quantizer_metalr.quantizer_mxfp4r/quantizer_quantor0quantizer_quarkr1quantizer_sinqr2quantizer_spqrr3quantizer_torchaor4quantizer_vptqr5rZrXrv get_loggerrir|rKrostrrrrr`r`r`ras  h                                   )o