minimize(cost) operation takes a lot of time

 

I have the following bit of code to compute the cost for an LSTM in tensorflow. When I execute the code, the optimizer step takes a lot of time to go through, ~25 minutes. 

cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=pred, labels=y_class))
optimizer = tf.train.RMSPropOptimizer(learning_rate=lr).minimize(cost)

Why does this happen? Is this dependent on the sizes of my weights, biases, number of hidden layers in my LSTM? I'm training the network on GTX 840M

It is quite normal that RNN is slower in GPU. According to the answer given here (https://datascience.stackexchange.com/questions/19220/choosing-between-cpu-and-gpu-for-training-a-neural-network) sometimes its worse than the CPU depending on your GPU model.

RNN training time depends on the number of weight parameters (that depends on the number of inputs and hidden units). Therefore, if possible you can try to reduce those numbers.

Moreover, as suggested by this answer (How to speedup rnn training speed of tensorflow?) you can,

  • Train with different batch sizes.
  • Use sampled softmax instead of regular softmax.

    • Furthermore, you can read this article (http://svail.github.io/rnn_perf/) from Baidu on Optimizing RNN performance , which gives u good understanding.

    Hope this helps.

    链接地址: http://www.djcxy.com/p/5514.html

    上一篇: > SNS SQS通知不起作用

    下一篇: 最小化（成本）操作需要很长时间