Hello, I encountered the following error when running finetuning Llama2:
Some indexes in the input tensor for the embedding layer are outside the vocabulary dimension. I guess the problem occurs when adding a special tokens to the HF Tokenizer vocabulary in tokenizer.py:
For myself, I solved this by changing the line:
Is this the right way?
Hello, I encountered the following error when running finetuning Llama2:
Some indexes in the input tensor for the embedding layer are outside the vocabulary dimension. I guess the problem occurs when adding a special tokens to the HF Tokenizer vocabulary in tokenizer.py:
For myself, I solved this by changing the line:
Is this the right way?