Explore tensorflow

preparation

Use pycharm to create a new project, then create a tf.py file, and download the basic data required for this work to the project folder. ( Download link).

Import related libraries

import numpy as np
import h5py
import matplotlib.pyplot as plt
import tensorflow as tf

# Add this when running some code written in version 1. Using version 2. Of tensorflow
tf.compat.v1.disable_eager_execution()
from tensorflow.python.framework import ops
import tf_utils
import time

Try to calculate the loss function

We know the formula of the loss function is as follows:
l o s s = L ( y ^ , y ) = ( y ^ ( i ) − y ( i ) ) 2 (1) loss = \mathcal{L}(\hat{y}, y) = (\hat y^{(i)} - y^{(i)})^2 \tag{1} loss=L(y^​,y)=(y^​(i)−y(i))2(1)
Note the version difference of tensorflow 2.0. The following code can be version 2.x.

# Calculate the loss function and define some variables
tf.compat.v1.disable_eager_execution()  # Ensure that session.run() can run normally
y_hat = tf.constant(36, name='y_hat')  # Define y_hat constant. Set to 36.
y = tf.constant(39, name='y')  # Define y. Set to 39

loss = tf.Variable((y - y_hat) ** 2, name='loss')
init = tf.compat.v1.global_variables_initializer()
with tf.compat.v1.Session() as session:  # Create a session and print the output
    session.run(init)  # Initializes the variables
    print(session.run(loss))

Writing and running a program in TensorFlow includes the following steps:

• Create variables that have not yet been executed.
• Write operations between these variables.
• Initialize variables.
• Create a session.
• Run the session, which will run the operations you wrote above.
  Therefore, when we create a variable for loss, we only define the loss as another number of functions, but do not verify its value. In order to verify it, we must run init = tf.compat.v1.global_variables_initializer() to initialize the loss variable. In the last line, we can finally verify the value of loss and print it.
  Test a simple example

# Test simple examples
a = tf.constant(2)
b = tf.constant(10)
c = tf.multiply(a, b)

print(c)

Finally, we didn't output what c is, but we got a variable of Tensor type, no dimension, and the number type is int32. All we did before was to put these things into a "calculation graph" To actually calculate these two numbers, we need to create a session and run it:
Add the following code:

sess = tf.compat.v1.Session ()
 
print(sess.run(c))

Finally, the loss value is 20

Next, we need to know about placeholders. A placeholder is an object whose value can only be specified later, that is, it occupies a position here first. To specify the value of a placeholder, you can use a feed Dictionary (feed_dict variable) Next, we create a placeholder for x, which allows us to pass in a number when we run the session later.

# Use feed_dict to change the value of x
x = tf.compat.v1.placeholder(tf.int64, name="x")
print(sess.run(2 * x, feed_dict={x: 3}))
sess.close()

The output is 6

linear function

Calculate the linear function Y=Wx+b. where W, X and b are extracted from the random normal distribution. The dimension of W is (4,3), X is (3,1), and b is (4,1).

# Define linear function
def linear_function():
    """
    Realize a linear function:
        initialization W，Type is tensor Random variable with dimension(4,3)
        initialization X，Type is tensor Random variable with dimension(3,1)
        initialization b，Type is tensor Random variable with dimension(4,1)
    return:
        result - It's running session The result of running is Y = WX + b

    """

    X = np.random.randn(3, 1)
    W = np.random.randn(4, 3)
    b = np.random.randn(4, 1)

    Y = tf.add(tf.matmul(W, X), b)  # tf.matmul is matrix multiplication
    # Y = tf.matmul(W,X) + b #It can also be written like this

    # Create a session and run it
    sess = tf.compat.v1.Session()# tf2. How many versions are written
    result = sess.run(Y)

    # After using the session, close it
    sess.close()

    return result
# Test it
print("result = " +  str(linear_function()))

The output is a matrix with four rows and one column. It is as follows:

Calculate sigmoid

TF provides us with a variety of commonly used neural network functions, tf.softmax, tf.sigmoid. Use the placeholder variable x to do this exercise. When running the session, you should use the feed dictionary to pass in the input z. perform the following steps.
(i) Create a placeholder x;
(ii) use tf.sigmoid to define the operation required to calculate Sigmoid;
(iii) then run the session
The following code is used to implement the above steps:

• tf.placeholder(tf.float32, name = "...")
• tf.sigmoid(...)
• sess.run(..., feed_dict = {x: z})

Note that there are two typical ways to create and use sessions in tensorflow:

//First kind
sess = tf.Session()
result = sess.run(...,feed_dict = {...})
sess.close()

//Second
with tf.Session as sess:
    result = sess.run(...,feed_dict = {...})

Next, let's implement the calculation of sigmoid

# Define sigmoid calculation
def sigmoid(z):
    """
    Implementation use sigmoid Function calculation z

    Parameters:
        z - The value entered, scalar or vector

    return:
        result - use sigmoid calculation z Value of

    """

    # Create a placeholder x called "X"
    x = tf.compat.v1.placeholder(tf.float32, name="x")

    # Calculate sigmoid(z)
    sigmoid = tf.sigmoid(x)

    # To create a session, use method 2
    with tf.compat.v1.Session() as sess:
        result = sess.run(sigmoid, feed_dict={x: z})

    return result
# Test the function
print ("sigmoid(0) = " + str(sigmoid(0)))
print ("sigmoid(12) = " + str(sigmoid(12)))

The output results of the test are as follows:

Calculate cost

Using the built-in function to calculate the loss of neural network, tensorflow can be completed in one line of code. The cost function J is shown below.

The codes to be used are as follows:

tf.nn.sigmoid_cross_entropy_with_logits(logits = ..., labels = ...)

Let's test a cost function calculation. The code should enter z, calculate sigmoid (get a), and then calculate J

# cost function 
def cost(logits, labels):
    """
    Computes the cost using the sigmoid cross entropy

    Arguments:
    logits -- vector containing z, output of the last linear unit (before the final sigmoid activation)
    labels -- vector of labels y (1 or 0)

    Note: What we've been calling "z" and "y" in this class are respectively called "logits" and "labels"
    in the TensorFlow documentation. So logits will feed into z, and labels into y.

    Returns:
    cost -- runs the session of the cost (formula (2))
    """

    # Create the placeholders for "logits" (z) and "labels" (y) (approx. 2 lines)
    z = tf.compat.v1.placeholder(tf.float32, name="z")
    y = tf.compat.v1.placeholder(tf.float32, name="y")

    # Use the loss function (approx. 1 line)
    cost = tf.nn.sigmoid_cross_entropy_with_logits(logits=z, labels=y)

    # Create a session (approx. 1 line). See method 1 above.
    with tf.compat.v1.Session() as sess:
    # Run the session (approx. 1 line).
        cost = sess.run(cost, feed_dict={z: logits, y: labels})

    # Close the session (approx. 1 line). See method 1 above.
    sess.close()

    return cost
# Test it
logits = sigmoid(np.array([0.2,0.4,0.7,0.9]))
cost = cost(logits, np.array([0,0,1,1]))
print ("cost = " + str(cost))

The output is as follows:

Use single heat coding (0, 1 Coding)

In deep learning, you will often get a Y vector whose number ranges from 0 to C-1, where C is the number of classes. For example, if C is 4, you may have the following y vector, and you will need to convert it in the following way:

This is called unique hot coding, also known as a bit valid coding, because in the converted representation, an element in each column is exactly "hot" (meaning set to 1). To perform this conversion in numpy format, you may need to write several lines of code. In tensorflow, you can use only one line of code: tf.one_hot(labels, depth, axis).
Define unique heat Code:

# Define unique heat code
def one_hot_matrix(lables, C):
    """
    Create a matrix where i Row corresponds to the second row i Class number, No j Column corresponds to the second column j Training samples
    So if the second j The first sample corresponds to the second sample i A label, then entry (i,j)It will be 1

    Parameters:
        lables - Label vector
        C - Classification number

    return:
        one_hot - Unique heat matrix

    """

    # Create a tf.constant, assign it to C, and call it c
    C = tf.constant(C, name="C")

    # Using tf.one_hot, pay attention to axis
    one_hot_matrix = tf.one_hot(indices=lables, depth=C, axis=0)

    # Create a session
    sess = tf.compat.v1.Session()

    # Run session
    one_hot = sess.run(one_hot_matrix)

    # Close session
    sess.close()

    return one_hot
# Test it
labels = np.array([1,2,3,0,2,1])
one_hot = one_hot_matrix(labels,C=4)
print(str(one_hot))

The output is as follows:

Initialize with 0 and 1

How to initialize vectors of 0 and 1. The function to be called is tf.ones(). To initialize with zero, you can use tf.zeros(). These functions take a dimension and return an array of dimensions containing 0 and 1 respectively.

# Initialization of definition 1
def ones(shape):
    """
    Create a dimension for shape Variables whose values are all 1

    Parameters:
        shape - The dimension of the array you want to create

    return:
        ones - Array containing only 1
    """

    # Use tf.ones()
    ones = tf.ones(shape)

    # Create session
    sess = tf.compat.v1.Session()
    # Run session
    ones = sess.run(ones)

    # Close session
    sess.close()

    return ones
# Test it
print ("ones = " + str(ones([3])))

The output is as follows:

Build a neural network using Tensorflow

Load dataset

Create a new file, shou_zhi.py, under the same project folder, import the code into the relevant library and load the dataset.

#Build the network and load the dataset
X_train_orig , Y_train_orig , X_test_orig , Y_test_orig , classes = tf_utils.load_dataset()

View the pictures sorted as 0 in the dataset.

# Show pictures sorted as 0
index = 0
plt.imshow(X_train_orig[index])
plt.show()
print("y = " + str(np.squeeze(Y_train_orig[:, index])))

The output is as follows:

As usual, we need to flatten the data set, and then divide it by 255 to normalize the data. In addition, we need to convert each label into a unique heat vector, as shown in the figure above.

# data processing
X_train_flatten = X_train_orig.reshape(X_train_orig.shape[0], -1).T  # Each column is a sample
X_test_flatten = X_test_orig.reshape(X_test_orig.shape[0], -1).T
# Normalized data
X_train = X_train_flatten / 255
X_test = X_test_flatten / 255
# Convert to heat independent matrix
Y_train = tf_utils.convert_to_one_hot(Y_train_orig, 6)
Y_test = tf_utils.convert_to_one_hot(Y_test_orig, 6)
print("Number of training set samples = " + str(X_train.shape[1]))
print("Number of test set samples = " + str(X_test.shape[1]))
print("X_train.shape: " + str(X_train.shape))
print("Y_train.shape: " + str(Y_train.shape))
print("X_test.shape: " + str(X_test.shape))
print("Y_test.shape: " + str(Y_test.shape))

The output is as follows:

Create placeholder

Create placeholders for X and X so that you can pass training data later when running a session.

# Create placeholder
def create_placeholders(n_x, n_y):
    """
    by TensorFlow Create placeholder for session
    Parameters:
        n_x - A real number, the size of the picture vector (64)*64*3 = 12288)
        n_y - A real number, classification number (from 0 to 5, so n_y = 6)

    return:
        X - A placeholder for data entry. The dimension is[n_x, None]，dtype = "float"
        Y - A placeholder corresponding to the entered label. The dimension is[n_Y,None]，dtype = "float"

    Tips:
        use None，Because it gives us the flexibility to handle the number of samples provided by placeholders. In fact, testing/The number of samples during training is different.

    """

    X = tf.compat.v1.placeholder(tf.float32, [n_x, None], name="X")  # Note here that the tf version is inconsistent with the original blog
    Y = tf.compat.v1.placeholder(tf.float32, [n_y, None], name="Y")

    return X, Y
# Test it
X, Y = create_placeholders(12288, 6)
print("X = " + str(X))
print("Y = " + str(Y))

The output is as follows:

Initialization parameters

Initialize parameters in tensorflow. Xavier initialization using weights and zero initialization of deviations. For example:
Note: since TF2.x deleted contrib, I used initializer = tf.initializers.glootuniform (seed = 1)) instead of initializer=tf.contrib.layers.xavier_initializer(seed=1))

W1 = tf.get_variable("W1", [25,12288], initializer = tf.contrib.layers.xavier_initializer(seed = 1))
b1 = tf.get_variable("b1", [25,1], initializer = tf.zeros_initializer())

Define initialization parameter function

# Initialization parameters
def initialize_parameters():
    
    """
    Initialize the parameters of the neural network. The dimensions of the parameters are as follows:
        W1 : [25, 12288]
        b1 : [25, 1]
        W2 : [12, 25]
        b2 : [12, 1]
        W3 : [6, 12]
        b3 : [6, 1]

    return:
        parameters - Contains W and b Dictionary of

    """

    W1 = tf.compat.v1.get_variable("W1", [25, 12288], initializer=tf.initializers.GlorotUniform(seed=1))
    b1 = tf.compat.v1.get_variable("b1", [25, 1], initializer=tf.zeros_initializer())
    W2 = tf.compat.v1.get_variable("W2", [12, 25], initializer=tf.initializers.GlorotUniform(seed=1))
    b2 = tf.compat.v1.get_variable("b2", [12, 1], initializer=tf.zeros_initializer())
    W3 = tf.compat.v1.get_variable("W3", [6, 12], initializer=tf.initializers.GlorotUniform(seed=1))
    b3 = tf.compat.v1.get_variable("b3", [6, 1], initializer=tf.zeros_initializer())

    parameters = {"W1": W1,
                  "b1": b1,
                  "W2": W2,
                  "b2": b2,
                  "W3": W3,
                  "b3": b3}

    return parameters
# Test it
ops.reset_default_graph()  # Used to clear the default graph stack and reset the global default graph.

with tf.compat.v1.Session()  as sess:
    parameters = initialize_parameters()
    print("W1 = " + str(parameters["W1"]))
    print("b1 = " + str(parameters["b1"]))
    print("W2 = " + str(parameters["W2"]))
    print("b2 = " + str(parameters["b2"]))

The output is as follows:

Forward propagation in tensorflow

We will implement forward propagation in TensorFlow. This function will accept a dictionary parameter and complete forward propagation. It uses the following code:

• tf.add(...): add
• tf.matmul(...,...): matrix multiplication
• tf.nn.relu(...): Relu activation function
  It is important that the forward propagation stop at Z3, because the output of the last linear output layer in TensorFlow is used as the input for calculating the loss function, so A3 is not required.

# Define forward propagation
def forward_propagation(X, parameters):
    """
    Realize the forward propagation of a model. The model structure is LINEAR -> RELU -> LINEAR -> RELU -> LINEAR -> SOFTMAX

    Parameters:
        X - Placeholder for input data, dimension is (input node quantity, sample quantity)
        parameters - Contains W and b Dictionary of parameters

    return:
        Z3 - the last one LINEAR Output of node

    """

    W1 = parameters['W1']
    b1 = parameters['b1']
    W2 = parameters['W2']
    b2 = parameters['b2']
    W3 = parameters['W3']
    b3 = parameters['b3']
                                        # You can also write it like this, using numpy
    Z1 = tf.add(tf.matmul(W1, X), b1)   # Z1 = np.dot(W1, X) + b1
    A1 = tf.nn.relu(Z1)                 # A1 = relu(Z1)
    Z2 = tf.add(tf.matmul(W2, A1), b2)  # Z2 = np.dot(W2, a1) + b2
    A2 = tf.nn.relu(Z2)                 # A2 = relu(Z2)
    Z3 = tf.add(tf.matmul(W3, A2), b3)  # Z3 = np.dot(W3,Z2) + b3

    return Z3
# Test it
ops.reset_default_graph() #Used to clear the default graph stack and reset the global default graph.
with tf.compat.v1.Session ()  as sess:
    X,Y = create_placeholders(12288,6)
    parameters = initialize_parameters()
    Z3 = forward_propagation(X,parameters)
    print("Z3 = " + str(Z3))

The output is as follows:

You may have noticed that forward propagation does not output any cache. When we start back propagation, you will understand why below.

Calculate cost

Use the following code to calculate the cost tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits = ..., labels = ...)).

• It is important to know tf.nn.softmax_ cross_ entropy_ with_ The "logits" and "labels" inputs of logits should have the same dimension (number of data, number of categories). Z3 and Y need to be converted.
• In addition, tf.reduce_mean is the sum of all data.

# Define calculation cost function
def compute_cost(Z3, Y):
    """
    Calculate cost

    Parameters:
        Z3 - Results of forward propagation
        Y - Label, a placeholder, and Z3 The dimensions of are the same

    return:
        cost - Cost value


    """
    logits = tf.transpose(Z3)  # Transpose
    labels = tf.transpose(Y)  # Transpose

    cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=labels))

    return cost
# Test it
ops.reset_default_graph()

with tf.compat.v1.Session() as sess:
    X, Y = create_placeholders(12288, 6)
    parameters = initialize_parameters()
    Z3 = forward_propagation(X, parameters)
    cost = compute_cost(Z3, Y)
    print("cost = " + str(cost))

The output is as follows:

Back propagation and parameter update

Thanks to the programming framework, all back propagation and parameter updates are handled in one line of code. After calculating the cost function, an optimizer object is created. When running tf.session, this object must be called together with the cost function. When called, it will optimize the given cost using the selected method and learning rate.
For example, for gradient descent, the optimizer would be:

optimizer = tf.train.GradientDescentOptimizer(learning_rate = learning_rate).minimize(cost)

To optimize, you can do the following:

_ , c = sess.run([optimizer,cost],feed_dict={X:mini_batch_X,Y:mini_batch_Y})

When writing code, we often use_ As a throw away variable to store values that we don't need to use later. Here_ Have the evaluation value of the optimizer we don't need (and c is the value of the cost variable)

Model building

Use the previously built functions to combine into a model

# Define model
def model(X_train, Y_train, X_test, Y_test,
          learning_rate=0.0001, num_epochs=1500, minibatch_size=32,
          print_cost=True, is_plot=True):
    """
    Implement a three-tier TensorFlow Neural network: LINEAR->RELU->LINEAR->RELU->LINEAR->SOFTMAX

    Parameters:
        X_train - Training set, dimension is (input size) (input number of nodes) = 12288, Number of samples = 1080)
        Y_train - Training set classification quantity, dimension is (output size)(Number of output nodes) = 6, Number of samples = 1080)
        X_test - Test set, dimension is (input size) (input number of nodes) = 12288, Number of samples = 120)
        Y_test - Test set classification quantity, dimension is (output size)(Number of output nodes) = 6, Number of samples = 120)
        learning_rate - Learning rate
        num_epochs - Traversal times of the whole training set
        mini_batch_size - Size of each small batch dataset
        print_cost - Print cost every 100 generations
        is_plot - Plot or not

    return:
        parameters - Parameters after learning

    """
    ops.reset_default_graph()  # Ability to rerun the model without overwriting tf variables
    tf.random.set_seed(1)
    # tf.set_random_seed(1)
    seed = 3
    (n_x, m) = X_train.shape  # Get the number of input nodes and samples
    n_y = Y_train.shape[0]  # Get the number of output nodes
    costs = []  # Cost set

    # Create placeholder s for X and Y
    X, Y = create_placeholders(n_x, n_y)

    # Initialization parameters
    parameters = initialize_parameters()

    # Forward propagation
    Z3 = forward_propagation(X, parameters)

    # Calculate cost
    cost = compute_cost(Z3, Y)

    # Back propagation, optimized using Adam
    optimizer = tf.compat.v1.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

    # Initialize all variables
    init = tf.compat.v1.global_variables_initializer()

    # Start session and calculate
    with tf.compat.v1.Session() as sess:
        # initialization
        sess.run(init)

        # Cycle of normal training
        for epoch in range(num_epochs):

            epoch_cost = 0  # Cost per generation
            num_minibatches = int(m / minibatch_size)  # Total number of minibatch
            seed = seed + 1
            minibatches = tf_utils.random_mini_batches(X_train, Y_train, minibatch_size, seed)

            for minibatch in minibatches:
                # Select a minibatch
                (minibatch_X, minibatch_Y) = minibatch

                # The data is ready to run the session
                _, minibatch_cost = sess.run([optimizer, cost], feed_dict={X: minibatch_X, Y: minibatch_Y})

                # Calculate the error of this minibatch in this generation
                epoch_cost = epoch_cost + minibatch_cost / num_minibatches

            # Record and print costs
            ## Record cost
            if epoch % 5 == 0:
                costs.append(epoch_cost)
                # Print:
                if print_cost and epoch % 100 == 0:
                    print("epoch = " + str(epoch) + "    epoch_cost = " + str(epoch_cost))

        # Map drawn
        if is_plot:
            plt.plot(np.squeeze(costs))
            plt.ylabel('cost')
            plt.xlabel('iterations (per tens)')
            plt.title("Learning rate =" + str(learning_rate))
            plt.show()

        # Save the learned parameters
        parameters = sess.run(parameters)
        print("Parameters have been saved to session. ")

        # Calculate current forecast results
        correct_prediction = tf.equal(tf.argmax(Z3), tf.argmax(Y))

        # Calculation accuracy
        accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))

        print("Accuracy of training set:", accuracy.eval({X: X_train, Y: Y_train}))
        print("Accuracy of test set:", accuracy.eval({X: X_test, Y: Y_test}))

        return parameters
# Test it
#start time
start_time = time.perf_counter()
#Start training
parameters = model(X_train, Y_train, X_test, Y_test)
#End time
end_time = time.perf_counter()
#Calculate time difference
print("CPU Execution time of = " + str(end_time - start_time) + " second" )

The output is as follows:

summary

• Tensorflow is a programming framework often used in deep learning
• The two main object categories in Tensorflow are variables and operators.
• When coding in Tensorflow, you must perform the following steps:
  • Create a calculation diagram containing tensors (variables, placeholders...) and operations (tf.matmul, tf.add,...)
  • Create session
  • Initialize session
  • Run the session to execute the graph
• You can execute the graph as many times as you can see in model ()
• When you run a session on the optimizer object, back propagation and optimization are done automatically.

Complete code

Python 3.6 interpreter used, tensorflow version 2.6.
The code in tf.py is as follows

import numpy as np
import h5py
import matplotlib.pyplot as plt
import tensorflow as tf

# Add this when running some code written in version 1. Using version 2. Of tensorflow
tf.compat.v1.disable_eager_execution()
from tensorflow.python.framework import ops
import tf_utils
import time




## Calculate the loss function and define some variables
# tf.compat.v1.disable_eager_execution()  # Ensure that session.run() can run normally
# y_hat = tf.constant(36, name='y_hat')  # Define y_hat constant. Set to 36.
# y = tf.constant(39, name='y')  # Define y. Set to 39
#
# loss = tf.Variable((y - y_hat) ** 2, name='loss')
# init = tf.compat.v1.global_variables_initializer()
# with tf.compat.v1.Session() as session:  # Create a session and print the output
#     session.run(init)  # Initializes the variables
#     print(session.run(loss))

# Test simple examples
# a = tf.constant(2)
# b = tf.constant(10)
# c = tf.multiply(a, b)
#
# print(c)
# sess = tf.compat.v1.Session()
#
# print(sess.run(c))
#
# # Use feed_dict to change the value of x
# x = tf.compat.v1.placeholder(tf.int64, name="x")
# print(sess.run(2 * x, feed_dict={x: 3}))
# sess.close()

# Define linear function
def linear_function():
    """
    Realize a linear function:
        initialization W，Type is tensor Random variable with dimension(4,3)
        initialization X，Type is tensor Random variable with dimension(3,1)
        initialization b，Type is tensor Random variable with dimension(4,1)
    return:
        result - It's running session The result of running is Y = WX + b

    """

    X = np.random.randn(3, 1)
    W = np.random.randn(4, 3)
    b = np.random.randn(4, 1)

    Y = tf.add(tf.matmul(W, X), b)  # tf.matmul is matrix multiplication
    # Y = tf.matmul(W,X) + b #It can also be written like this

    # Create a session and run it
    sess = tf.compat.v1.Session()# tf2. How many versions are written
    result = sess.run(Y)

    # After using the session, close it
    sess.close()

    return result
# Test it
# print("result = " +  str(linear_function()))

# Define sigmoid calculation
def sigmoid(z):
    """
    Implementation use sigmoid Function calculation z

    Parameters:
        z - The value entered, scalar or vector

    return:
        result - use sigmoid calculation z Value of

    """

    # Create a placeholder x called "X"
    x = tf.compat.v1.placeholder(tf.float32, name="x")

    # Calculate sigmoid(z)
    sigmoid = tf.sigmoid(x)

    # To create a session, use method 2
    with tf.compat.v1.Session() as sess:
        result = sess.run(sigmoid, feed_dict={x: z})

    return result
# Test the function
# print ("sigmoid(0) = " + str(sigmoid(0)))
# print ("sigmoid(12) = " + str(sigmoid(12)))

# cost function 
def cost(logits, labels):
    """
    Computes the cost using the sigmoid cross entropy

    Arguments:
    logits -- vector containing z, output of the last linear unit (before the final sigmoid activation)
    labels -- vector of labels y (1 or 0)

    Note: What we've been calling "z" and "y" in this class are respectively called "logits" and "labels"
    in the TensorFlow documentation. So logits will feed into z, and labels into y.

    Returns:
    cost -- runs the session of the cost (formula (2))
    """

    # Create the placeholders for "logits" (z) and "labels" (y) (approx. 2 lines)
    z = tf.compat.v1.placeholder(tf.float32, name="z")
    y = tf.compat.v1.placeholder(tf.float32, name="y")

    # Use the loss function (approx. 1 line)
    cost = tf.nn.sigmoid_cross_entropy_with_logits(logits=z, labels=y)

    # Create a session (approx. 1 line). See method 1 above.
    with tf.compat.v1.Session() as sess:
    # Run the session (approx. 1 line).
        cost = sess.run(cost, feed_dict={z: logits, y: labels})

    # Close the session (approx. 1 line). See method 1 above.
    sess.close()

    return cost
# # Test it
# logits = sigmoid(np.array([0.2,0.4,0.7,0.9]))
# cost = cost(logits, np.array([0,0,1,1]))
# print ("cost = " + str(cost))

# Define unique heat code
def one_hot_matrix(lables, C):
    """
    Create a matrix where i Row corresponds to the second row i Class number, No j Column corresponds to the second column j Training samples
    So if the second j The first sample corresponds to the second sample i A label, then entry (i,j)It will be 1

    Parameters:
        lables - Label vector
        C - Classification number

    return:
        one_hot - Unique heat matrix

    """

    # Create a tf.constant, assign it to C, and call it c
    C = tf.constant(C, name="C")

    # Use tf.one_hot, pay attention to axis
    one_hot_matrix = tf.one_hot(indices=lables, depth=C, axis=0)

    # Create a session
    sess = tf.compat.v1.Session()

    # Run session
    one_hot = sess.run(one_hot_matrix)

    # Close session
    sess.close()

    return one_hot
# # Test it
# labels = np.array([1,2,3,0,2,1])
# one_hot = one_hot_matrix(labels,C=4)
# print(str(one_hot))

# Initialization of definition 1
def ones(shape):
    """
    Create a dimension for shape Variables whose values are all 1

    Parameters:
        shape - The dimension of the array you want to create

    return:
        ones - Array containing only 1
    """

    # Use tf.ones()
    ones = tf.ones(shape)

    # Create session
    sess = tf.compat.v1.Session()
    # Run session
    ones = sess.run(ones)

    # Close session
    sess.close()

    return ones
# # Test it
# print ("ones = " + str(ones([3])))

shou_ The code in zhi.py is as follows:

import numpy as np
import h5py
import matplotlib.pyplot as plt
import tensorflow as tf

# Add this when running some code written in version 1. Using version 2. Of tensorflow
tf.compat.v1.disable_eager_execution()
from tensorflow.python.framework import ops
import tf_utils
import time

#Build the network and load the dataset
X_train_orig , Y_train_orig , X_test_orig , Y_test_orig , classes = tf_utils.load_dataset()
# # Show pictures sorted as 0
# index = 0
# plt.imshow(X_train_orig[index])
# plt.show()
# print("y = " + str(np.squeeze(Y_train_orig[:, index])))

# data processing
X_train_flatten = X_train_orig.reshape(X_train_orig.shape[0], -1).T  # Each column is a sample
X_test_flatten = X_test_orig.reshape(X_test_orig.shape[0], -1).T
# Normalized data
X_train = X_train_flatten / 255
X_test = X_test_flatten / 255
# Convert to heat independent matrix
Y_train = tf_utils.convert_to_one_hot(Y_train_orig, 6)
Y_test = tf_utils.convert_to_one_hot(Y_test_orig, 6)
# print("number of training set samples =" + str(X_train.shape[1]))
# print("number of test set samples =" + str(X_test.shape[1]))
# print("X_train.shape: " + str(X_train.shape))
# print("Y_train.shape: " + str(Y_train.shape))
# print("X_test.shape: " + str(X_test.shape))
# print("Y_test.shape: " + str(Y_test.shape))

# Create placeholder
def create_placeholders(n_x, n_y):
    """
    by TensorFlow Create placeholder for session
    Parameters:
        n_x - A real number, the size of the picture vector (64)*64*3 = 12288)
        n_y - A real number, classification number (from 0 to 5, so n_y = 6)

    return:
        X - A placeholder for data entry. The dimension is[n_x, None]，dtype = "float"
        Y - A placeholder corresponding to the entered label. The dimension is[n_Y,None]，dtype = "float"

    Tips:
        use None，Because it gives us the flexibility to handle the number of samples provided by placeholders. In fact, testing/The number of samples during training is different.

    """

    X = tf.compat.v1.placeholder(tf.float32, [n_x, None], name="X")  # Note here that the tf version is inconsistent with the original blog
    Y = tf.compat.v1.placeholder(tf.float32, [n_y, None], name="Y")

    return X, Y
# # Test it
# X, Y = create_placeholders(12288, 6)
# print("X = " + str(X))
# print("Y = " + str(Y))

# Initialization parameters
def initialize_parameters():

    """
    Initialize the parameters of the neural network. The dimensions of the parameters are as follows:
        W1 : [25, 12288]
        b1 : [25, 1]
        W2 : [12, 25]
        b2 : [12, 1]
        W3 : [6, 12]
        b3 : [6, 1]

    return:
        parameters - Contains W and b Dictionary of

    """

    W1 = tf.compat.v1.get_variable("W1", [25, 12288], initializer=tf.initializers.GlorotUniform(seed=1))
    b1 = tf.compat.v1.get_variable("b1", [25, 1], initializer=tf.zeros_initializer())
    W2 = tf.compat.v1.get_variable("W2", [12, 25], initializer=tf.initializers.GlorotUniform(seed=1))
    b2 = tf.compat.v1.get_variable("b2", [12, 1], initializer=tf.zeros_initializer())
    W3 = tf.compat.v1.get_variable("W3", [6, 12], initializer=tf.initializers.GlorotUniform(seed=1))
    b3 = tf.compat.v1.get_variable("b3", [6, 1], initializer=tf.zeros_initializer())

    parameters = {"W1": W1,
                  "b1": b1,
                  "W2": W2,
                  "b2": b2,
                  "W3": W3,
                  "b3": b3}

    return parameters
# # Test it
# ops.reset_default_graph()  # Used to clear the default graph stack and reset the global default graph.
#
# with tf.compat.v1.Session()  as sess:
#     parameters = initialize_parameters()
#     print("W1 = " + str(parameters["W1"]))
#     print("b1 = " + str(parameters["b1"]))
#     print("W2 = " + str(parameters["W2"]))
#     print("b2 = " + str(parameters["b2"]))

# Define forward propagation
def forward_propagation(X, parameters):
    """
    Realize the forward propagation of a model. The model structure is LINEAR -> RELU -> LINEAR -> RELU -> LINEAR -> SOFTMAX

    Parameters:
        X - Placeholder for input data, dimension is (input node quantity, sample quantity)
        parameters - Contains W and b Dictionary of parameters

    return:
        Z3 - the last one LINEAR Output of node

    """

    W1 = parameters['W1']
    b1 = parameters['b1']
    W2 = parameters['W2']
    b2 = parameters['b2']
    W3 = parameters['W3']
    b3 = parameters['b3']
                                        # You can also write it like this, using numpy
    Z1 = tf.add(tf.matmul(W1, X), b1)   # Z1 = np.dot(W1, X) + b1
    A1 = tf.nn.relu(Z1)                 # A1 = relu(Z1)
    Z2 = tf.add(tf.matmul(W2, A1), b2)  # Z2 = np.dot(W2, a1) + b2
    A2 = tf.nn.relu(Z2)                 # A2 = relu(Z2)
    Z3 = tf.add(tf.matmul(W3, A2), b3)  # Z3 = np.dot(W3,Z2) + b3

    return Z3
# # Test it
# ops.reset_default_graph() #Used to clear the default graph stack and reset the global default graph.
# with tf.compat.v1.Session ()  as sess:
#     X,Y = create_placeholders(12288,6)
#     parameters = initialize_parameters()
#     Z3 = forward_propagation(X,parameters)
#     print("Z3 = " + str(Z3))

# Define calculation cost function
def compute_cost(Z3, Y):
    """
    Calculate cost

    Parameters:
        Z3 - Results of forward propagation
        Y - Label, a placeholder, and Z3 The dimensions of are the same

    return:
        cost - Cost value


    """
    logits = tf.transpose(Z3)  # Transpose
    labels = tf.transpose(Y)  # Transpose

    cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=labels))

    return cost
# # Test it
# ops.reset_default_graph()
#
# with tf.compat.v1.Session() as sess:
#     X, Y = create_placeholders(12288, 6)
#     parameters = initialize_parameters()
#     Z3 = forward_propagation(X, parameters)
#     cost = compute_cost(Z3, Y)
#     print("cost = " + str(cost))

# Define model
def model(X_train, Y_train, X_test, Y_test,
          learning_rate=0.0001, num_epochs=1500, minibatch_size=32,
          print_cost=True, is_plot=True):
    """
    Implement a three-tier TensorFlow Neural network: LINEAR->RELU->LINEAR->RELU->LINEAR->SOFTMAX

    Parameters:
        X_train - Training set, dimension is (input size) (input number of nodes) = 12288, Number of samples = 1080)
        Y_train - Training set classification quantity, dimension is (output size)(Number of output nodes) = 6, Number of samples = 1080)
        X_test - Test set, dimension is (input size) (input number of nodes) = 12288, Number of samples = 120)
        Y_test - Test set classification quantity, dimension is (output size)(Number of output nodes) = 6, Number of samples = 120)
        learning_rate - Learning rate
        num_epochs - Traversal times of the whole training set
        mini_batch_size - Size of each small batch dataset
        print_cost - Print cost every 100 generations
        is_plot - Plot or not

    return:
        parameters - Parameters after learning

    """
    ops.reset_default_graph()  # Ability to rerun the model without overwriting tf variables
    tf.random.set_seed(1)
    # tf.set_random_seed(1)
    seed = 3
    (n_x, m) = X_train.shape  # Get the number of input nodes and samples
    n_y = Y_train.shape[0]  # Get the number of output nodes
    costs = []  # Cost set

    # Create placeholder s for X and Y
    X, Y = create_placeholders(n_x, n_y)

    # Initialization parameters
    parameters = initialize_parameters()

    # Forward propagation
    Z3 = forward_propagation(X, parameters)

    # Calculate cost
    cost = compute_cost(Z3, Y)

    # Back propagation, optimized using Adam
    optimizer = tf.compat.v1.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

    # Initialize all variables
    init = tf.compat.v1.global_variables_initializer()

    # Start session and calculate
    with tf.compat.v1.Session() as sess:
        # initialization
        sess.run(init)

        # Cycle of normal training
        for epoch in range(num_epochs):

            epoch_cost = 0  # Cost per generation
            num_minibatches = int(m / minibatch_size)  # Total number of minibatch
            seed = seed + 1
            minibatches = tf_utils.random_mini_batches(X_train, Y_train, minibatch_size, seed)

            for minibatch in minibatches:
                # Select a minibatch
                (minibatch_X, minibatch_Y) = minibatch

                # The data is ready to run the session
                _, minibatch_cost = sess.run([optimizer, cost], feed_dict={X: minibatch_X, Y: minibatch_Y})

                # Calculate the error of this minibatch in this generation
                epoch_cost = epoch_cost + minibatch_cost / num_minibatches

            # Record and print costs
            ## Record cost
            if epoch % 5 == 0:
                costs.append(epoch_cost)
                # Print:
                if print_cost and epoch % 100 == 0:
                    print("epoch = " + str(epoch) + "    epoch_cost = " + str(epoch_cost))

        # Map drawn
        if is_plot:
            plt.plot(np.squeeze(costs))
            plt.ylabel('cost')
            plt.xlabel('iterations (per tens)')
            plt.title("Learning rate =" + str(learning_rate))
            plt.show()

        # Save the learned parameters
        parameters = sess.run(parameters)
        print("Parameters have been saved to session. ")

        # Calculate current forecast results
        correct_prediction = tf.equal(tf.argmax(Z3), tf.argmax(Y))

        # Calculation accuracy
        accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))

        print("Accuracy of training set:", accuracy.eval({X: X_train, Y: Y_train}))
        print("Accuracy of test set:", accuracy.eval({X: X_test, Y: Y_test}))

        return parameters
# Test it
#start time
start_time = time.perf_counter()
#Start training
parameters = model(X_train, Y_train, X_test, Y_test)
#End time
end_time = time.perf_counter()
#Calculate time difference
print("CPU Execution time of = " + str(end_time - start_time) + " second" )