TensorFlow (5-3) Use TensorFlow Built-in RNN Functions

Use TensorFlow Built-in RNN Functions to classify MNIST images:

This is an good example of RNN in Learning TensorFlow [1]. The program is shorter and easier to write but exploits low-level optimizations.

Set the parameters:

#!/usr/bin/python3
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data

mnist = input_data.read_data_sets("data/", one_hot=True)
element_size = 28
time_steps = 28
num_classes = 10
batch_size = 128
hidden_layer_size = 128

2. Computational process:

Construct a RNN with built-in functions.

_inputs = tf.placeholder(tf.float32,shape=[None, time_steps, element_size], name='inputs')
y = tf.placeholder(tf.float32, shape=[None, num_classes],name='inputs')

# TensorFlow built-in functions
rnn_cell = tf.contrib.rnn.BasicRNNCell(hidden_layer_size)
outputs, _ = tf.nn.dynamic_rnn(rnn_cell, _inputs, dtype=tf.float32)

Wl = tf.Variable(tf.truncated_normal([hidden_layer_size, num_classes], mean=0, stddev=.01))
bl = tf.Variable(tf.truncated_normal([num_classes], mean=0, stddev=.01))

def get_linear_layer(vector):
    return tf.matmul(vector, Wl) + bl

# [batch, time, hsize]
last_rnn_output = outputs[:,-1,:]
final_output = get_linear_layer(last_rnn_output)

cross_entropy = tf.reduce_mean(
                       tf.nn.softmax_cross_entropy_with_logits(logits=final_output, labels=y))
train_step = tf.train.RMSPropOptimizer(0.001, 0.9).minimize(cross_entropy)

tf.contrib.rnn.BasicRNNCell() creates a RNN cell and tf.nn.dynamic_rnn() uses a tf.While loop to dynamically construct the graph.

3. Training and accuracy:

correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(final_output,1))
accuracy = (tf.reduce_mean(tf.cast(correct_prediction, tf.float32)))*100

sess = tf.InteractiveSession()
sess.run(tf.global_variables_initializer())

test_data = mnist.test.images[:batch_size].reshape((-1, time_steps, element_size))
test_label = mnist.test.labels[:batch_size]

for i in range(3001):
    batch_x, batch_y = mnist.train.next_batch(batch_size)
    batch_x = batch_x.reshape((batch_size, time_steps, element_size))
    sess.run(train_step,feed_dict={_inputs:batch_x, y:batch_y})
    if i % 1000 == 0:
        acc = sess.run(accuracy, feed_dict={_inputs: batch_x, y: batch_y})
        loss = sess.run(cross_entropy,feed_dict={_inputs:batch_x, y:batch_y})
        print ("Iter " + str(i) + ", Minibatch Loss= " + \
               "{:.6f}".format(loss) + ", Training Accuracy= " + \
               "{:.5f}".format(acc))
    print ("Testing Accuracy:", sess.run(accuracy, feed_dict={_inputs: test_data, y: test_label}))

Reference

[1] Tom Hope, Yehezkel S. Resheff, and Itay Lieder, Learning TensorFlow A Guide to Building Deep Learning Systems , O'Reilly Media (2017)

[2] Nikhil Buduma, Fundamentals of Deep Learning Designing Next-Generation Machine Intelligence Algorithms, O'Reilly Media (2017)

[3] Aurélien Géron, Hands-On Machine Learning with Scikit-Learn and TensorFlow Concepts, Tools, and Techniques to Build Intelligent Systems , O'Reilly Media (2017)