how can I improve my LSTM code on tensorflow?
I am trying to predict the household power consumption using LSTM. The following is a small portion of input data(total training input is about ~1M records) that I am using to train my model followed by my LSTM code using TensorFlow. I need some help with:
(1) Verification of my model : I would like a network with 2 LSTM layers of size 512 with time_step: 10. I think I only have 1 LSTM hidden layer, but I am not sure how to add another.
(2) General improvement on my model : seems like the accuracy is not converging to any specific value and I am not quite sure where to look at this point. Any advice on modeling/stacking layers/choice of optimizer/etc will be much appreciated.
Input data (:
16/12/2006;17:24:00;4.216;0.418;234.840;18.400;0.000;1.000;17.000
16/12/2006;17:25:00;5.360;0.436;233.630;23.000;0.000;1.000;16.000
16/12/2006;17:26:00;5.374;0.498;233.290;23.000;0.000;2.000;17.000
16/12/2006;17:27:00;5.388;0.502;233.740;23.000;0.000;1.000;17.000
16/12/2006;17:28:00;3.666;0.528;235.680;15.800;0.000;1.000;17.000
16/12/2006;17:29:00;3.520;0.522;235.020;15.000;0.000;2.000;17.000
16/12/2006;17:30:00;3.702;0.520;235.090;15.800;0.000;1.000;17.000
16/12/2006;17:31:00;3.700;0.520;235.220;15.800;0.000;1.000;17.000
16/12/2006;17:32:00;3.668;0.510;233.990;15.800;0.000;1.000;17.000
16/12/2006;17:33:00;3.662;0.510;233.860;15.800;0.000;2.000;16.000
LSTM code :
from getdata_new import DataFromFile
import numpy as np
import tensorflow as tf
import datetime
# ==========
# MODEL
# ==========
# Parameters
learning_rate = 0.01
training_iters = 100000
batch_size = 1000
display_step = 10
# Network Parameters
seq_len = 10 # Sequence length
n_hidden = 512 # hidden layer num of features
n_classes = 1201 #
n_input = 13
num_layers = 2
trainset= DataFromFile(filename="/tmp/train_data.txt",delim=";")
# Define weights
weights = {
'out': tf.Variable(tf.random_normal([n_hidden, n_classes]))
}
biases = {
'out': tf.Variable(tf.random_normal([n_classes]))
}
x = tf.placeholder("float", [None, seq_len, n_input])
y = tf.placeholder("float", [None, n_classes])
def RNN(x, weights, biases):
x = tf.transpose(x, [1, 0, 2])
x = tf.reshape(x, [-1, n_input])
x = tf.split(0, seq_len, x)
lstm_cell = tf.nn.rnn_cell.BasicLSTMCell(n_hidden)
outputs, states = tf.nn.rnn(lstm_cell, x, dtype=tf.float32)
# Linear activation, using outputs computed above
return tf.matmul(outputs[-1], weights['out']) + biases['out']
pred = RNN(x, weights, biases)
# Define loss and optimizer
cost = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(pred, y))
optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate).minimize(cost)
# Evaluate model
correct_pred = tf.equal(tf.argmax(pred,1), tf.argmax(y,1))
accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))
#test
testPred = tf.argmax(pred,1)
# Initializing the variables
init = tf.initialize_all_variables()
# Launch the graph
with tf.Session() as sess:
sess.run(init)
acc = 0.0
for step in range(1,training_iters+1):
batch_x, batch_y = trainset.train_next(batch_size,seq_len)
# Run optimization op (backprop)
sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})
if step % display_step == 0:
# Calculate batch accuracy
acc = sess.run(accuracy, feed_dict={x: batch_x, y: batch_y})
# Calculate batch loss
loss = sess.run(cost, feed_dict={x: batch_x, y: batch_y})
print "Iter " + str(step) + ", Minibatch Loss= " +
"{:.6f}".format(loss) + ", Training Accuracy= " +
"{:.5f}".format(acc)
print "Optimization Finished!"
Result :
Iter 99860, Minibatch Loss= 0.015933, Training Accuracy= 0.29900
Iter 99870, Minibatch Loss= 0.015993, Training Accuracy= 0.26200
Iter 99880, Minibatch Loss= 0.015783, Training Accuracy= 0.30500
Iter 99890, Minibatch Loss= 0.016071, Training Accuracy= 0.27200
Iter 99900, Minibatch Loss= 0.015390, Training Accuracy= 0.40300
Iter 99910, Minibatch Loss= 0.015247, Training Accuracy= 0.43700
Iter 99920, Minibatch Loss= 0.015264, Training Accuracy= 0.42700
Iter 99930, Minibatch Loss= 0.015212, Training Accuracy= 0.43800
Iter 99940, Minibatch Loss= 0.016164, Training Accuracy= 0.26500
Iter 99950, Minibatch Loss= 0.015923, Training Accuracy= 0.30800
Iter 99960, Minibatch Loss= 0.016338, Training Accuracy= 0.22600
Iter 99970, Minibatch Loss= 0.016327, Training Accuracy= 0.19000
Iter 99980, Minibatch Loss= 0.016322, Training Accuracy= 0.22300
Iter 99990, Minibatch Loss= 0.016608, Training Accuracy= 0.15400
Iter 100000, Minibatch Loss= 0.016809, Training Accuracy= 0.10700
Optimization Finished!
Please refer following code which I have written for Multilayer Perceptron (2 hidden layers) in Tensorflow.
Declare Weights and biases for each layer.
weights = {
'h1': tf.Variable(tf.truncated_normal([n_input, n_hidden_1],stddev=0.1)), #weights for units in first hidden layer
'h2': tf.Variable(tf.truncated_normal([n_hidden_1, n_hidden_2],stddev=0.1)),
'out': tf.Variable(tf.truncated_normal([n_hidden_2, n_classes],stddev=0.1)) #weights for output layer ##Check n_hidden2
}
biases = {
'b1': tf.Variable(tf.constant(0.1,shape=[n_hidden_1])),#biases of units of hidden layer
'b2': tf.Variable(tf.constant(0.1,shape=[n_hidden_2])),
'out': tf.Variable(tf.constant(0.1,shape=[n_classes])) #biases of units of output layer
}
def multilayer_perceptron(x, weights, biases):
# Hidden layers with ELU activation
layer_1 = tf.add(tf.matmul(x, weights['h1']), biases['b1']])
layer_1 = tf.nn.elu(layer_1) #Activation function for layer 1 units
layer_2 = tf.add(tf.matmul(layer_1, weights['h2']), biases['b2'])
layer_2 = tf.nn.elu(layer_2) #Activation function for layer 2 units
out_layer = tf.matmul(layer_2, weights['out']) + biases['out']
return tf.sigmoid(out_layer)
In your case,
n_hidden_1=512
n_hidden_2=512
链接地址: http://www.djcxy.com/p/32124.html