Weights Of Cnn Model Go To Really Small Values And After Nan

I am not able to understand the reason why the weights of following model are going smaller and smaller until NaN during training. The model is the following: def initialize_embed

Solution 1:

By your edits, it got a little easier to find the problem.

Those zeros passed unchanged to the warp_loss function. The part that went through the convolution remained unchanged at first, because any filters multiplied by zero result in zero, and the default bias initializer is also 'zeros'. The same idea applies to the dense (filters * 0 = 0 and bias initializer = 'zeros')

That reached this line: return numerator / denominator and caused an error (division by zero)

It's a common practice I've seen in many codes to add K.epsilon() to avoid this:

return numerator / (denominator + K.epsilon())