o diE$ @srddlZddlmZmZmZmZddlZddlZddl m Z m Z e r)ddl m Z  d&deeeejfdedeefd d Z d'd eeeejfd edeedeeejffddZd(dedededefddZd(dedededefddZd(dedededefddZd(dedededefddZd(dedededefddZ d)d"edededed#edef d$d%ZdS)*N)AnyDictListOptional)DATASETS_IMPORT_ERRORis_datasets_available) load_datasetexamples batch_size pad_token_idcsxg|D]}|d}|d}t|t|dqdkr)dur)tdfddtd tDS) a Prepare the dataset by making sure that we have the right format and `batch_size` Args: examples (`List[Dict[str, torch.LongTensor]]`): List of data to prepare batch_size (`int`, defaults to `1`): Batch size of the data pad_token_id (`Optional[int]`, defaults to `None`): Pad token id of the model Returns: ` List[Dict[str, torch.LongTensor]]`: Batched dataset input_idsattention_maskr rr NzfYou need to pass a `pad_token_id` in `quantize_model` if you want to have examples with batch size > 1cs&g|]}t||ddqS)F)contain_labelsr ) collate_data).0startr new_examplesr _/lsinfo/ai/hellotax_ai/llm_service/venv_embed/lib/python3.10/site-packages/optimum/gptq/data.py <sz#prepare_dataset..r)appendtorch LongTensor ValueErrorrangelen)r r r exampler rrrrprepare_dataset"sr FblocksrreturncCsPdd}dd|D}dd|D}|r$dd|D}tdd|D}t|}tdd|D} t|D]P} || j\} } | | } | d krd||| t| | f||| <||| t| | f|| <|r|| jd }||}|d kr||| t| |fd || <q5tj|d d tj|d d d }|rtj|d d |d<|S)a Collate data in `blocks` Args: blocks (`List[Dict[str, torch.LongTensor]]`): List of tensors that we need to batch together pad_token_id (`Optional[int]`, defaults to `None`): Pad token id of the model contain_labels (`bool`, defaults to `False`): Set True to also process the labels Returns: `Dict[str, torch.LongTensor]`: Batched data cSstj||j|fddS)Ndim)rcattodevicelong)blockpadsrrr pad_blockVszcollate_data..pad_blockcSg|]}|dqS)r rrr*rrrrYz collate_data..cSr-)rrr.rrrrZr/cSr-)labelsrr.rrrr\r/cSg|]}|dqSr#sizer.rrrr]cSr1r2r3r.rrrr`r5rr#ir$rr0) maxrrshaperoneszerosr&r))r!rr r,input_ids_blocksattention_mask_blocks label_blocks label_max_lenbsz inp_max_leni block_bsz block_inp_lenpad_numblock_label_len label_pad_numdatarrrrCs4   rtrain tokenizerseqlennsamplessplitc Csts ttd|dkrtdddd}n |dkr!tdddd}dd d |d dd D}||d d}g}t|D],}td|j j d|d} | |} |j dd| | f} t | } | | | dq=|S)N get_wikitext2rGwikitextzwikitext-2-raw-v1)rK validationtestcSsg|] }|dkr dn|qS)rPz r)rsrrrrsz!get_wikitext2..textiptreturn_tensorsrr r)r ImportErrorrformatrjoinrrandomrandintr r7r ones_liker) rHrIrJrKrFrRencdataset_r@jinprrrrrLxs     rLc Cts ttd|dkrtddddid}n |dkr%tddddid}g}t|D]K} td t|d }|||d d d }|j j d |krKnq.td |j j d |d }||} |j dd|| f} t | } | | | dq+|S)Nget_c4rG allenai/c4"en/c4-train.00000-of-01024.json.gzrK data_filesrN'en/c4-validation.00000-of-00008.json.gzTrr rRrSrTrrrVrrWrrrYrZrr r7rr[r rHrIrJrKrFr]r^r@r\r_r`rrrrrb0  rbc Cra)N get_c4_newrGrcrdrerNrgTrr rRrSrTrrhrirrrrkrjrkcCtdNz(Loading the `ptb` dataset was deprecated RuntimeErrorrHrIrJrKrrrget_ptbrqcCrlrmrnrprrr get_ptb_newrrrs dataset_nameseedcCst|tj|tj|tttd}|dvr"td||dvr1t|dt |||vrCtdt | d|||}|||||dS) a Get the dataset from the original paper of GPTQ Args: dataset_name (`str`): Dataset name. Available options are `['wikitext2', 'c4', 'c4-new']`. tokenizer (`Any`): Tokenizer of the model nsamples (`int`, defaults to `128`): Number of samples seqlen (`int`, defaults to `2048`): The sequence length of the model seed (`int`, defaults to `0`): Seed split (`str`, defaults to `train`): Split of the dataset. Can be either "train" or "validation" Returns: `List[Dict[str,torch.LongTensor]]`: The tokenized dataset. ) wikitext2c4zc4-new)rGrNz7The split need to be 'train' or 'validation' but found >ptb-newptbzD dataset was deprecated, only the following dataset are supported : zExpected a value in z but found )rHrJrIrK) rYrwnpr manual_seedrLrbrkrlistkeys)rvrHrJrIrwrKget_dataset_mapget_dataset_fnrrr get_datasets"   r)r N)FN)rG)rtrurrG)rYtypingrrrrnumpyr|roptimum.utils.import_utilsrrdatasetsrstrrintr boolrrLrbrkrqrsrrrrrs\  #  5