Skip to content

Latest commit

 

History

History
244 lines (178 loc) · 6.59 KB

TensorFlow.markdown

File metadata and controls

244 lines (178 loc) · 6.59 KB
Raw

General notes

  • img tensor shapes should be : N x H x W x NChannels

Standard flow

# usually have something like
X = tf.placeholder(tf.float32, shape=(None, n_features), name='X')

# usually build network in `with tf.variable_scope`
# set up variables with `tf.get_variable`

# establish the session and initialize variables
sess = tf.Session()
init = tf.global_variables_initializer()
# can also do a subset, e.g.
# init_ab = tf.variables_initializer([a, b], name='init_subset')
sess.run(init)
# can also run solo vars, e.g. `sess.run(W.initializer)`

# set up an optimizer (need a cost function)
train_op = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)
loss = ... some function

# run the training
for epoch in range(training_epochs):
    _, l = sess.run([train_op, loss] feed_dict={X: xs, Y: labels})
    # can get cost in a separate call instead
    # l = sess.run(loss, feed_dict={X: xs, Y: labels})
    if epoch % epoch_print_frequency == 0:
        print(epoch, l)

# examine learned model params
w_val = sess.run(w)
print('learned parameters', w_val)

sess.close()

Saving and loading models - checkpoints

  • Restore:

      # sess = tf.Session() ...
  saver = tf.train.Saver()
  if os.path.exists('./my_model.ckpt'):
      saver.restore(sess, './my_model.ckpt')
      print('model restored')

  • Save (every so often during training):

      # sess = tf.Session() ...
  saver = tf.train.Saver()
  save_path = saver.save(sess, './my_model.ckpt')
  print('model saved in file: %s' % save_path)

Saving and loading models - protobufs

  • Save:

      # saver = tf.train.Saver() ...
  path='./'
  ckpt_name = path + 'model.ckpt'
  fname = path + 'model.tfmodel'
  dst_nodes = ['Y']   # destination nodes
  g_1 = tf.Graph()
  with tf.Sesssion(graph=g_1) as sess:
      x = tf.placeholder(tf.float32, shape=(1, 224, 224, 3))
      net = create_network()
      sess.run(tf.initialize_all_variables())
      saver.restore(sess, ckpt_name)
      graph_def = tf.python.graph_util.convert_variables_to_constants(
          sess, sess.graph_def, dst_nodes
      )
  
  g_2 = tf.Graph()
  with tf.Session(graph=g_2) as sess:
      tf.train.write_graph(
          tf.python.graph_util.extract_sub_graph(graph_def, dst_nodes),
          path, fname, as_text=False
      )

  • Import:

      with open('model.tfmodel', mode='rb') as f:
      graph_def = tf.GraphDef()
      graph_def.ParseFromString(f.read())
  
  tf.import_graph_def(net['graph_def'], name='model')

Graph manipulation

  • Reset the operations in a graph:

      from tensorflow.python.framework import ops
  ops.reset_default_graph()

  • List the operations in a graph:

      [op.name for op in tf.get_default_graph().get_operations()]

  • Look at the graph definiton:

      print sess.graph.as_graph_def()

Note: constants are stored in the graph definition. This makes loading graphs expensive if constants are big. Therefore, only use constants for primitive types and use variables or readers for data that requires more memory.

Note: don't defer creating/initializing objects until needed in graph definiion. This can cause us to actually create an object more times than is needed (e.g., if done in a loop, the operation is created N times!). Always separate fully the definition of ops from their computation.

Creating variables

  • We generally want to work within a scope:

      def create_layer(X, n_inp, n_out, act=None, scope=None):
      with tf.variable_scope(scope or 'layer'):
          W = tf.get_variable(name='W',
                              shape=[n_inp, n_out],
                              initializer=tf.random_normal_initializer(
                                  mean=0.0, stddev=0.1
                              )
          h = tf.matmul(X, W)
          if activation is not None:
              h = activation(h)
          return h

Note that we also used a TF initializer here - this will create new random values every time we call sess.run(initialize_all_variables()).

Reshaping tricks

  • We need images to have shape Batch x H x W x N-Channels for convolution. To go from [None, n_features] (where None is letting us have an arbitrary batch size) we use -1 in the reshape():

      X = tf.placeholder(tf.float32, [None, n_features])
  X_tensor = tf.reshape(X, [-1, H, W, NChan])

Shape

tensor.shape
tensor.shape.as_list()
tensor.get_shale().as_list()

Evaluating tensors

print sess.run(W)
print W.eval()

TensorBoard

with tf.Session() as sess:
    writer = tf.summary.FileWriter('./graphs', sess.graph)
    ...   # computations

writer.close()

Then, externally:

$ python prog.py
$ tensorboard --logdir="./graphs" --port 6006

Then, go to http://localhost:6006

InteractiveSession

InteractiveSession makes itself the default:

sess = tf.InteractiveSession()
...
expr.eval()  # no need to specify `sess`
sess.close()

Feeding

  • placeholder must be fed.
  • Check others with tf.Graph.is_feedable(tensor)

Feeding values is useful for testing, etc.

a = tf.add(2, 5)
b = tf.multiply(a, 3)
with tf.Session() as sess:
    replace_dict = {a: 15}
    sess.run(b, feed_dict=replace_dict)  # returns 45

Simple tensor evaluation

import tensorflow as tf
w = tf.get_variable('weights', [2, 2], initializer=tf.random_normal_initializer())
[op.name for op in tf.get_default_graph().get_operations()]
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    t = tf.get_default_graph().get_tensor_by_name('weights:0').eval()
    print(t)
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    t = sess.run(w)
    print(t)

Interactive sessions

sess = tf.Session()
print sess.run(my_tensor)
sess.close()

Note, even when interactive:

sess = tf.Session()
print my_tensor.eval(session=sess)
sess.close()

i.e., need to sepecify the session with eval().

Tensorboard graphs

tf.reset_default_graph()
writer = tf.summary.FileWriter('./simple_example_graph')
x = 2
y = 3
op1 = tf.add(2, 3)
op2 = tf.multiply(2, 3)
op3 = tf.pow(op1, op2)
with tf.Session() as sess:
    writer.add_graph(sess.graph)
writer.close()

Suppress warnings

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'