Storing TensorFlow network weights in Python multi
I'm totally new to TensorFlow and Python, so please excuse me for posting such a basic question, but I'm a bit overwhelmed with learning both things at once. EDIT: I found a solution myself and posted it below, however, more efficient solutions are wellcome
Short version of the question: How can I extract every weight and bias at any point from a neural network using TensorFlow and store it into a Python array with the shape [layers][neurons-previous-layer][neurons-current-layer]. The goal is NOT to store on the hdd but in variables with the same shape and type as the one explained below the last code snipet. I'd also like to know, which is the most efficient way to do so.
The task I want to perform is to create a neural network with pre-trained weights and biases (not obtained from Tensor, but from totally different source), refine the training with Tensor and then return the refined weights to the program.
I've investigated how to create NN's in Tensor Flow as well as made my way through a way to initialize the weights of the network using previously created lists in Python based on some Tensor tutorials and some unrelated questions from StackOverflow.
So, my question is, given a trained network in TensorFlow, how can I extract every weight and bias to variables (my network has around 2,8 million weights and biases) in the fastest possible way? (keep in mind that this operation is going to be repeated over and over)
To clarify the question, here's some code:
First of all, the entire network creation and training process (except the network layout) is based on this post: Autoencoder Example.
The relevant parts of the code for this example are the following (I cut the output part because it is not necessary to explain the way I create the network):
num_hidden_1 = 256 # 1st layer num features
num_hidden_2 = 128 # 2nd layer num features (the latent dim)
num_input = 784 # MNIST data input (img shape: 28*28)
X = tf.placeholder("float", [None, num_input])
weights = {
'encoder_h1': tf.Variable(tf.random_normal([num_input, num_hidden_1])),
'encoder_h2': tf.Variable(tf.random_normal([num_hidden_1, num_hidden_2])),
}
biases = {
'encoder_b1': tf.Variable(tf.random_normal([num_hidden_1])),
'encoder_b2': tf.Variable(tf.random_normal([num_hidden_2])),
}
# Building the encoder
def encoder(x):
# Encoder Hidden layer with sigmoid activation #1
layer_1 = tf.nn.sigmoid(tf.add(tf.matmul(x, weights['encoder_h1']),
biases['encoder_b1']))
# Encoder Hidden layer with sigmoid activation #2
layer_2 = tf.nn.sigmoid(tf.add(tf.matmul(layer_1, weights['encoder_h2']),
biases['encoder_b2']))
return layer_2
The code I created to generate my neural network is the following:
def create_network(layout, funciones, pesos, biases):
capas = [tf.placeholder("float", [None, layout[0]])]
for i in range(0, len(pesos)):
#There's an element already, so capas[i] is the previous layer
capas.append(tf.layers.dense(capas[i], layout[i+1], activation=funciones[i],
kernel_initializer=tf.constant_initializer(pesos[i], dtype=tf.float32),
bias_initializer=tf.constant_initializer(biases[i], dtype=tf.float32)))
return capas
Code explanation: Pesos and biases three dimensional lists containing weights ("pesos") and biases in a format TensorFlow can accept. Capas is an array containing all the layers of the net. "funcciones[]", it is ATM a testing global variable to adjust the activation function of the NN, it should be part of "layout", but I'm just testing now. Layout is an array containing the size of every layer in the network. Layout[0] contains the number of neurons on the input layer, layout1 on the first hidden layer, and so on.
Pesos and Biases format: the first level has as many elements as network layers-1 (input doesnt need weights or biases). On the second level it has as many elements as neurons present in the PREVIOUS layer, and the third level has as much elements as neurons on this list. The 2nd and 3rd level have the same shape that the one generated in the examples by tf.random_normal([num_input, num_hidden_1])
My question is: Assuming I'm use the same structure to execute my NN than the one used in the example provided in the URL above, how can I store the trained weights and biases back in two three dimensional lists with the same exact structure than the ones provided in my code, and which is the fastest way to do so
SIDE QUESTION: Is there a more efficient way to build/execute my NN than the one mentioned above? Good references for this one are accepted as answers too.
How about using tf.trainable_variables() ?
This returns a list of all the trainable parameters and since it's a tensorflow model, I would asume it's optimized.
You can access specific weights from this list by tensorname:
variable = [weight for weights in tf.trainable_variables() if weight.name == name_my_var]
I found a solution and built a working function. The naming convention is the same as in my own question. I had to manually name the layers, otherwise it placed weights and biases on the previous layer (I checked the graph and it was connected correctly at the end, but the script couldn't extract them properly)
def extraer_pesos(red, session):
pesos = []
biases = []
for i in range(1, len(red)):
pesos.append(session.run(tf.get_default_graph().get_tensor_by_name(
os.path.split(red[i].name)[0] + '/kernel:0')).tolist())
biases.append(session.run(tf.get_default_graph().get_tensor_by_name(
os.path.split(red[i].name)[0] + '/bias:0')).tolist())
return pesos, biases
def create_network(layout, funciones, pesos,biases):
capas = [(tf.placeholder("float", [None, layout[0]])]
for i in range(0, len(pesos)):
#There's an element already, so capas[i] is the previous layer
capas.append(tf.layers.dense(capas[i], layout[i+1], activation=funciones[i],
kernel_initializer=tf.constant_initializer(pesos[i], dtype=tf.float32),
bias_initializer=tf.constant_initializer(biases[i], dtype=tf.float32),
name="layer"+str(i)))
return capas
Keep in mind thay the variables have to be initialized. If you're extracting'em after training the network there shouldn't be any problem.
链接地址: http://www.djcxy.com/p/32094.html