Tensorflow
Tensorflow is the second generation machine learning system published by google. It is a successor for
google’s previous DistBelief system. Its compution is based on data flow graph with takes math operation
as node and multidimensional data arrays (tensors) flows through edges.
It is open-sourced and can be used in either single machine or multi-server clusters. It can be run in CPU or
GPU and even speciailized computation device such as TPU(Tensor Processing Units) which are used in
google. It enables the researchers to easily implement various deep learning algorithms and has attract
much attention from research communities.
The main components of tensorflow consists of client, master and working processes. Client sends request
to the master and the master schedules the working processes to do computation in available devices.
Tensorflow can be used both in single-machine and distributed clusters where client, master and working
processes run in different machines .
One of the many useful features is that tensorflow can differentiate symbolic expression and derive the
backpropagation automatically for neural network training which greatly reduce the work on programmer
and the chance to make mistakes.
The tensorflow is designed based on dataflow-graph model. It provides python and c++ interface for
programmers to easily construct the graph which makes architecture, algorithm and parameters
experimentation very easy.
After the user constructs the dataflow-graph, the tensorflow system will optimized the graph and actually
execute the operations in machines. Through this first constructing graph then actually executing approach,
it enables the tensorflow to know the whole information before executing and thus can do optimization as
much as possible.
All computations are encoded as nodes in data graph, the dependency of the data between different
operations are explicitly encoded in the graph, so the tensorflow can partition the graph according to the
dependencies and run the subgraph computations parallel in different devices.
The tensorflow allows the user to specify the subgraph that need to be computed. The user can feed
tensors to none or some of the input place holders. The tensorflow system only runs the computation that is
necessary and prune the irrelevant graph away.
Tensorflow
Introduction
Advantage
The tensor flow’s data graph model not only make it easy to run concurrently and also easy to distribute
computation to multiple devices.
In tensorflow, the data flowing through graph are called tensors. A tensor is a multi dimensional array of
primitive types such as int32. It represents the input and the output of the operations which is represented
in the vertex. Every operation has a type and none or more attributes. An operation that contain mutable
state is called stageful operation, Variable is one of such kind of operation. Another special operation is
queue operation
User can use tensor flow’s checkpointing to periodically save training models to file and reload the model
later. This facility not only improve the fault tolerance, it also can be used for transfer learning.
The TensorFlow adopts a layered architecture. On the top level are training and inference libraries. The
next level is python and c++ API which are built on the C API. Below C API level are distributed master and
dataflow executor.
The distributed master accepts a data flow graph as input, it will prune the unnecessary part of the graph
and divide the graph into subgraphs to distribute computation to different devices. Many optimization such
as constant folding and subexpression elimination are done by it.
The dataflow executor’s take is to execute the computation of the subgraph distributed by the distributed
master.
The next level is kernel implementations which has more than 200 operations implemented including often
used operation such as Const, Var, MatMul, Conv2D and ReLU.
Apart from above core components, the tensorflow system also includes several useful tools such as a
dashboard to visualize the data flow graph and training progress and a profiler that shows the running time
of different tasks in different devices.
In Chintala’s benchmark of convolutional models testing, the results show that TensorFlow has shorter
training step time than Caffe and similar with Torch. Experiments have shown that tensorflow can scale well
in problems such as image classification and language modeling.
Architecture
Performance