getting loss as 'nan' after 1st epoch only?

[shubhamk16]

Hello guys,
just like the Glove, I created a dictionary of all the possible words. with keys as words and values as 768 embedding vector for BERT.
But when I use this dictionary and train the model, the loss is getting nan in 1st epoch only.

1. How to handle this problem?
2. what are the possible reasons for getting a loss 'nan'?
3. Is this a good approach, to make a dictionary of embedding vectors?

[alkaideemo]

I got a similar problem. Here it's not numerical stable while computing the loss.

IRNet/src/model.py

Lines 308 to 309 in c329460

	sketch_prob_var = torch.stack(
	[torch.stack(action_probs_i, dim=0).log().sum() for action_probs_i in action_probs], dim=0)

IRNet/src/model.py

Lines 479 to 480 in c329460

	lf_prob_var = torch.stack(
	[torch.stack(action_probs_i, dim=0).log().sum() for action_probs_i in action_probs], dim=0)

I add a small number before log operation, problem solved.

[liguozhanglearner]

i have no idea about the loss function computing way

[ersaurabhverma]

Try reducing the learning rate.
Your gradient is exploding due to high learning rate