Keras can predict multiple values at the same time, like if you input a vector of 100 elements, Keras can compute one prediction for each element, giving 100 outputs. This computation can also be done in batches, defined by the batch_size.

This is just in case you cannot fit all the data in the CPU/GPU RAM at the same time and batch processing is needed.

The reason is the same , why you need batch size for training, because you cannot fit all data into one single batch

Similarly, if you have millions of data points to predict, it is obviously that you will not be able to pass at one go (single batch). 

After all, training and prediction both have a forward pass on the batch data.

Hence, you need the batch size to control/limit the data point in a single batch and distribute it across multiple batches of prediction.

So, I&#39;m diving deeper and deeper into **Clojure.Spec**.

One thing I stumbled upon is, **where to put my specs**. I see three options:


# Global Spec File

In most examples, I found online, there is one big `spec.clj` file, that gets required in the main namespace. It has all the `(s/def)` and `(s/fdef)` for all the &quot;data types&quot; and functions.

Pro: 

* One file to rule them all

Contra:

* This file can be big
* Single Responsibliy Principle violated?


# Specs in production namespaces

You could put your `(s/def)` and `(s/fdef)` right next to your production code. So that, the implementation and the spec co-exist in the same namespace.

Pro:

* co-location of implementation and spec
* one namespace - one concern?

Contra:

* production code could get messy
* one namespace - two concerns?

# Dedicated spec namespace structure

Then I thought, maybe Specs are a third kind of code (next to production and test). So maybe they deserve their own structure of namespaces, like this:

    ├─ src
    │&#160; └─ package
    │   &#160; ├─ a.clj
    │   &#160; └─ b.clj
    ├─ test
    │&#160; └─ package
    │   &#160; ├─ a_test.clj
    │   &#160; └─ b_test.clj
    └─ spec
     &#160; └─ package
        &#160; ├─ a_spec.clj
        &#160; └─ b_spec.clj

Pro:

* dedicated (but related) namespaces for specs

Contra:

* you have to source and require the correct namespaces

---

Who has experience with one of the approaches?  
Is there another option?  
What do you think about the different options?

Where to put specs for Clojure.Spec?

Seaborn offers a function called color_palette, which allows you to easily create new color_palettes for plots.

    colors = [&quot;#67E568&quot;,&quot;#257F27&quot;,&quot;#08420D&quot;,&quot;#FFF000&quot;,&quot;#FFB62B&quot;,&quot;#E56124&quot;,&quot;#E53E30&quot;,&quot;#7F2353&quot;,&quot;#F911FF&quot;,&quot;#9F8CA6&quot;]
        
    color_palette = sns.color_palette(colors)

I want to transform color_palette to a cmap, which I can use in matplotlib, but I don&#39;t see how I can do this. 

Sadly just functions like &quot;cubehelix_palette&quot;,&quot;light_palette&quot;,… have an &quot;as_cmap&quot; paramater. &quot;color_palette&quot; doesn&#39;t, unfortunately.

seaborn color_palette as matplotlib colormap

In Keras, to predict class of a datatest, the `predict_classes()` is used.

For example:

    classes = model.predict_classes(X_test, batch_size=32)

My question is, I know the usage of `batch_size` in training, but why does it need a `batch_size` for prediction? how does it work?

Why does prediction needs batch size in Keras?

In Keras, to predict class of a datatest, the <code>predict_classes()</code> is used.
For example:
<pre><code class="hljs language-ini">classes = model.predict_classes(X_test, batch_size=32)
</code></pre>
My question is, I know the usage of <code>batch_size</code> in training, but why does it need a <code>batch_size</code> for prediction? how does it work?

Is there a function call or another way to count the total number of parameters in a tensorflow model?

By parameters I mean: an N dim vector of trainable variables has N parameters, a `NxM` matrix has `N*M` parameters, etc. So essentially I&#39;d like to sum the product of the shape dimensions of all the trainable variables in a tensorflow session.



How to count total number of trainable parameters in a tensorflow model?

I am playing with a ANN which is part of Udacity DeepLearning course.

I have an assignment which involves introducing generalization to the network with one hidden ReLU layer using L2 loss. I wonder how to properly introduce it so that ALL weights are penalized, not only weights of the output layer.

Code for network *without* generalization is at the bottom of the post (code to actually run the training is out of the scope of the question).

Obvious way of introducing the L2 is to replace the loss calculation with something like this (if beta is 0.01):

    loss = tf.reduce_mean( tf.nn.softmax_cross_entropy_with_logits(out_layer, tf_train_labels) + 0.01*tf.nn.l2_loss(out_weights))
      
But in such case it will take into account values of output layer&#39;s weights. I am not sure, how do we properly penalize the weights which come INTO the hidden ReLU layer. Is it needed at all or introducing penalization of output layer will somehow keep the hidden weights in check also?


    #some importing
    from __future__ import print_function
    import numpy as np
    import tensorflow as tf
    from six.moves import cPickle as pickle
    from six.moves import range
    
    #loading data
    pickle_file = &#39;/home/maxkhk/Documents/Udacity/DeepLearningCourse/SourceCode/tensorflow/examples/udacity/notMNIST.pickle&#39;
    
    with open(pickle_file, &#39;rb&#39;) as f:
      save = pickle.load(f)
      train_dataset = save[&#39;train_dataset&#39;]
      train_labels = save[&#39;train_labels&#39;]
      valid_dataset = save[&#39;valid_dataset&#39;]
      valid_labels = save[&#39;valid_labels&#39;]
      test_dataset = save[&#39;test_dataset&#39;]
      test_labels = save[&#39;test_labels&#39;]
      del save  # hint to help gc free up memory
      print(&#39;Training set&#39;, train_dataset.shape, train_labels.shape)
      print(&#39;Validation set&#39;, valid_dataset.shape, valid_labels.shape)
      print(&#39;Test set&#39;, test_dataset.shape, test_labels.shape)
      
    
    #prepare data to have right format for tensorflow
    #i.e. data is flat matrix, labels are onehot
    
    image_size = 28
    num_labels = 10
    
    def reformat(dataset, labels):
      dataset = dataset.reshape((-1, image_size * image_size)).astype(np.float32)
      # Map 0 to [1.0, 0.0, 0.0 ...], 1 to [0.0, 1.0, 0.0 ...]
      labels = (np.arange(num_labels) == labels[:,None]).astype(np.float32)
      return dataset, labels
    train_dataset, train_labels = reformat(train_dataset, train_labels)
    valid_dataset, valid_labels = reformat(valid_dataset, valid_labels)
    test_dataset, test_labels = reformat(test_dataset, test_labels)
    print(&#39;Training set&#39;, train_dataset.shape, train_labels.shape)
    print(&#39;Validation set&#39;, valid_dataset.shape, valid_labels.shape)
    print(&#39;Test set&#39;, test_dataset.shape, test_labels.shape)
    
    
    #now is the interesting part - we are building a network with
    #one hidden ReLU layer and out usual output linear layer
    
    #we are going to use SGD so here is our size of batch
    batch_size = 128
    
    #building tensorflow graph
    graph = tf.Graph()
    with graph.as_default():
          # Input data. For the training data, we use a placeholder that will be fed
      # at run time with a training minibatch.
      tf_train_dataset = tf.placeholder(tf.float32,
                                        shape=(batch_size, image_size * image_size))
      tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, num_labels))
      tf_valid_dataset = tf.constant(valid_dataset)
      tf_test_dataset = tf.constant(test_dataset)
      
      #now let&#39;s build our new hidden layer
      #that&#39;s how many hidden neurons we want
      num_hidden_neurons = 1024
      #its weights
      hidden_weights = tf.Variable(
        tf.truncated_normal([image_size * image_size, num_hidden_neurons]))
      hidden_biases = tf.Variable(tf.zeros([num_hidden_neurons]))
      
      #now the layer itself. It multiplies data by weights, adds biases
      #and takes ReLU over result
      hidden_layer = tf.nn.relu(tf.matmul(tf_train_dataset, hidden_weights) + hidden_biases)
      
      #time to go for output linear layer
      #out weights connect hidden neurons to output labels
      #biases are added to output labels  
      out_weights = tf.Variable(
        tf.truncated_normal([num_hidden_neurons, num_labels]))  
      
      out_biases = tf.Variable(tf.zeros([num_labels]))  
      
      #compute output  
      out_layer = tf.matmul(hidden_layer,out_weights) + out_biases
      #our real output is a softmax of prior result
      #and we also compute its cross-entropy to get our loss
      loss = tf.reduce_mean( tf.nn.softmax_cross_entropy_with_logits(out_layer, tf_train_labels))
      
      #now we just minimize this loss to actually train the network
      optimizer = tf.train.GradientDescentOptimizer(0.5).minimize(loss)
      
      #nice, now let&#39;s calculate the predictions on each dataset for evaluating the
      #performance so far
      # Predictions for the training, validation, and test data.
      train_prediction = tf.nn.softmax(out_layer)
      valid_relu = tf.nn.relu(  tf.matmul(tf_valid_dataset, hidden_weights) + hidden_biases)
      valid_prediction = tf.nn.softmax( tf.matmul(valid_relu, out_weights) + out_biases) 
    
      test_relu = tf.nn.relu( tf.matmul( tf_test_dataset, hidden_weights) + hidden_biases)
      test_prediction = tf.nn.softmax(tf.matmul(test_relu, out_weights) + out_biases)



TensorFlow - regularization with L2 loss, how to apply to all weights, not just last one?

When you run a Keras neural network model you might see something like this in the console: 

    Epoch 1/3
       6/1000 [..............................] - ETA: 7994s - loss: 5111.7661

As time goes on the loss hopefully improves. I want to log these losses to a file over time so that I can learn from them. I have tried: 

    logging.basicConfig(filename=&#39;example.log&#39;, filemode=&#39;w&#39;, level=logging.DEBUG)

but this doesn&#39;t work. I am not sure what level of logging I need in this situation. 

I have also tried using a callback like in: 

    def generate_train_batch():
        while 1:
            for i in xrange(0,dset_X.shape[0],3):
                yield dset_X[i:i+3,:,:,:],dset_y[i:i+3,:,:]
    
    class LossHistory(keras.callbacks.Callback):
        def on_train_begin(self, logs={}):
            self.losses = []
    
        def on_batch_end(self, batch, logs={}):
            self.losses.append(logs.get(&#39;loss&#39;))
    logloss=LossHistory()
    colorize.fit_generator(generate_train_batch(),samples_per_epoch=1000,nb_epoch=3,callbacks=[&#39;logloss&#39;])

but obviously this isn&#39;t writing to a file. Whatever the method, through a callback or the logging module or anything else, I would love to hear your solutions for logging loss of a keras neural network to a file. Thanks! 

How to log Keras loss output to a file

I was running TensorFlow and I happen to have something yielding a NaN. I&#39;d like to know what it is but I do not know how to do this. The main issue is that in a &quot;normal&quot; procedural program I would just write a print statement just before the operation is executed. The issue with TensorFlow is that I cannot do that because I first declare (or define) the graph, so adding print statements to the graph definition does not help. Are there any rules, advice, heuristics, anything to track down what might be causing the NaN?

----

In this case I know more precisely what line to look at because I have the following:

    Delta_tilde = 2.0*tf.matmul(x,W) - tf.add(WW, XX) #note this quantity should always be positive because its pair-wise euclidian distance
    Z = tf.sqrt(Delta_tilde)
    Z = Transform(Z) # potentially some transform, currently I have it to return Z for debugging (the identity)
    Z = tf.pow(Z, 2.0)
    A = tf.exp(Z) 

when this line is present I have it that it returns NaN as declared by my summary writers. Why is this? Is there a way to at least explore what value Z has after its being square rooted?

----

For the specific example I posted, I tried `tf.Print(0,Z)` but with no success it printed nothing. As in:

    Delta_tilde = 2.0*tf.matmul(x,W) - tf.add(WW, XX) #note this quantity should always be positive because its pair-wise euclidian distance
    Z = tf.sqrt(Delta_tilde)
    tf.Print(0,[Z]) # &lt;-------- TF PRINT STATMENT
    Z = Transform(Z) # potentially some transform, currently I have it to return Z for debugging (the identity)
    Z = tf.pow(Z, 2.0)
    A = tf.exp(Z) 

I actually don&#39;t understand what `tf.Print` is suppose to do. Why does it need two arguments? If I want to print 1 tensor why would I need to pass 2? Seems bizarre to me.

----

I was looking at the function [tf.add_check_numerics_ops()][1] but it doesn&#39;t say how to use it (plus the docs seem to not be super helpful). Does anyone know how to use this?

----

Since I&#39;ve had comments addressing the data might be bad, I am using standard MNIST. However, I am computing a quantity that is positive (pair-wise eucledian distance) and then square rooting it. Thus, I wouldn&#39;t see how the data specifically would be an issue.



  [1]: https://www.tensorflow.org/versions/r0.9/api_docs/python/control_flow_ops.html#add_check_numerics_ops

How does one debug NaN values in TensorFlow?

I&#39;m training a network for image localization with Adam optimizer, and someone suggest me to use exponential decay. I don&#39;t want to try that because Adam optimizer itself decays learning rate. But that guy insists and he said he did that before. So should I do that and is there any theory behind your suggestion?

Should we do learning rate decay for adam optimizer

I&#39;m trying to use deep learning to predict income from 15 self reported attributes from a dating site.

We&#39;re getting rather odd results, where our validation data is getting better accuracy and lower loss, than our training data. And this is consistent across different sizes of hidden layers.
This is our model:
```python
for hl1 in [250, 200, 150, 100, 75, 50, 25, 15, 10, 7]:
    def baseline_model():
        model = Sequential()
        model.add(Dense(hl1, input_dim=299, kernel_initializer=&#39;normal&#39;, activation=&#39;relu&#39;, kernel_regularizer=regularizers.l1_l2(0.001)))
        model.add(Dropout(0.5, seed=seed))
        model.add(Dense(3, kernel_initializer=&#39;normal&#39;, activation=&#39;sigmoid&#39;))

        model.compile(loss=&#39;categorical_crossentropy&#39;, optimizer=&#39;adamax&#39;, metrics=[&#39;accuracy&#39;])
        return model

    history_logs = LossHistory()
    model = baseline_model()
    history = model.fit(X, Y, validation_split=0.3, shuffle=False, epochs=50, batch_size=10, verbose=2, callbacks=[history_logs])
```

And this is an example of the accuracy and losses:
![Accuracy with hidden layer of 250 neurons][1] and ![the loss][2].

We&#39;ve tried to remove regularization and dropout, which, as expected, ended in overfitting (training acc: ~85%). We&#39;ve even tried to decrease the learning rate drastically, with similiar results.

Has anyone seen similar results?


  [1]: https://i.stack.imgur.com/0TfSp.png
  [2]: https://i.stack.imgur.com/TH6Nr.png
  [3]: https://i.stack.imgur.com/0Em7K.png

Higher validation accuracy, than training accurracy using Tensorflow and Keras

When trying to get cross-entropy with sigmoid activation function, there is a difference between 

 1. `loss1 = -tf.reduce_sum(p*tf.log(q), 1)`
 2. `loss2 = tf.reduce_sum(tf.nn.sigmoid_cross_entropy_with_logits(labels=p, logits=logit_q),1)`

But they are the same when with softmax activation function.

Following is the sample code:

&lt;!-- language-all: lang-python --&gt;

    import tensorflow as tf
    
    sess2 = tf.InteractiveSession()
    p = tf.placeholder(tf.float32, shape=[None, 5])
    logit_q = tf.placeholder(tf.float32, shape=[None, 5])
    q = tf.nn.sigmoid(logit_q)
    sess.run(tf.global_variables_initializer())
    
    feed_dict = {p: [[0, 0, 0, 1, 0], [1,0,0,0,0]], logit_q: [[0.2, 0.2, 0.2, 0.2, 0.2], [0.3, 0.3, 0.2, 0.1, 0.1]]}
    loss1 = -tf.reduce_sum(p*tf.log(q),1).eval(feed_dict)
    loss2 = tf.reduce_sum(tf.nn.sigmoid_cross_entropy_with_logits(labels=p, logits=logit_q),1).eval(feed_dict)
    
    print(p.eval(feed_dict), &quot;\n&quot;, q.eval(feed_dict))
    print(&quot;\n&quot;,loss1, &quot;\n&quot;, loss2)

What is the difference between a sigmoid followed by the cross entropy and sigmoid_cross_entropy_with_logits in TensorFlow?

I&#39;m trying to train a classifier via PyTorch. However, I am experiencing problems with training when I feed the model with training data.
I get this error on `y_pred = model(X_trainTensor)`:

&gt; RuntimeError: Expected object of scalar type Float but got scalar type Double for argument #4 &#39;mat1&#39;

Here are key parts of my code:

```python
# Hyper-parameters 
D_in = 47  # there are 47 parameters I investigate
H = 33
D_out = 2  # output should be either 1 or 0
```
```python
# Format and load the data
y = np.array( df[&#39;target&#39;] )
X = np.array( df.drop(columns = [&#39;target&#39;], axis = 1) )
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size = 0.8)  # split training/test data

X_trainTensor = torch.from_numpy(X_train) # convert to tensors
y_trainTensor = torch.from_numpy(y_train)
X_testTensor = torch.from_numpy(X_test)
y_testTensor = torch.from_numpy(y_test)
```
```python
# Define the model
model = torch.nn.Sequential(
    torch.nn.Linear(D_in, H),
    torch.nn.ReLU(),
    torch.nn.Linear(H, D_out),
    nn.LogSoftmax(dim = 1)
)
```
```python
# Define the loss function
loss_fn = torch.nn.NLLLoss() 
```
```python
for i in range(50):
    y_pred = model(X_trainTensor)
    loss = loss_fn(y_pred, y_trainTensor)
    model.zero_grad()
    loss.backward()
    with torch.no_grad():       
        for param in model.parameters():
            param -= learning_rate * param.grad
```

How to fix RuntimeError &quot;Expected object of scalar type Float but got scalar type Double for argument&quot;?

I am trying to do some vanilla pattern recognition with an LSTM using Keras to predict the next element in a sequence.

My data look like this:

[![My data][1]][1]


  [1]: http://i.stack.imgur.com/NnEvI.png

where the label of the training sequence is the last element in the list: `X_train[&#39;Sequence&#39;][n][-1]`.

Because my `Sequence` column can have a variable number of elements in the sequence, I believe an RNN to be the best model to use. Below is my attempt to build an LSTM in Keras:

    # Build the model

    # A few arbitrary constants...
    max_features = 20000
    out_size = 128
    
    # The max length should be the length of the longest sequence (minus one to account for the label)
    max_length = X_train[&#39;Sequence&#39;].apply(len).max() - 1

    # Normal LSTM model construction with sigmoid activation
    model = Sequential()
    model.add(Embedding(max_features, out_size, input_length=max_length, dropout=0.2))
    model.add(LSTM(128, dropout_W=0.2, dropout_U=0.2))
    model.add(Dense(1))
    model.add(Activation(&#39;sigmoid&#39;))
    
    # try using different optimizers and different optimizer configs
    model.compile(loss=&#39;binary_crossentropy&#39;, optimizer=&#39;adam&#39;, metrics=[&#39;accuracy&#39;])

And here&#39;s how I attempt to train my model:

    # Train the model
    for seq in X_train[&#39;Sequence&#39;]:
        print(&quot;Length of training is {0}&quot;.format(len(seq[:-1])))
        print(&quot;Training set is {0}&quot;.format(seq[:-1]))
        model.fit(np.array([seq[:-1]]), [seq[-1]])

My output is this:

    Length of training is 13
    Training set is [1, 3, 13, 87, 1053, 28576, 2141733, 508147108, 402135275365, 1073376057490373, 9700385489355970183, 298434346895322960005291, 31479360095907908092817694945]

However, I get the following error:

    Exception: Error when checking model input: expected embedding_input_1 to have shape (None, 347) but got array with shape (1, 13)

I believe my training step is correctly setup, so my model construction must be wrong. Note that 347 is `max_length`.

How can I correctly build a variable-length input LSTM in Keras? I&#39;d prefer not to pad the data. Not sure if it&#39;s relevant, but I&#39;m using the Theano backend.

How do I create a variable-length input LSTM in Keras?

Keras documentation isn&#39;t clear what this actually is. I understand we can use this to compress the input feature space into a smaller one. But how is this done from a neural design perspective? Is it an autoenocder, RBM?

What is an Embedding in Keras?

I am trying to reconcile my understand of LSTMs and pointed out here in [this post by Christopher Olah][1] implemented in Keras. I am following the [blog written by Jason Brownlee][2] for the Keras tutorial. What I am mainly confused about is, 

1. The reshaping of the data series into `[samples, time steps, features]` and,
2. The stateful LSTMs 

Lets concentrate on the above two questions with reference to the code pasted below:

    # reshape into X=t and Y=t+1
    look_back = 3
    trainX, trainY = create_dataset(train, look_back)
    testX, testY = create_dataset(test, look_back)
    
    # reshape input to be [samples, time steps, features]
    trainX = numpy.reshape(trainX, (trainX.shape[0], look_back, 1))
    testX = numpy.reshape(testX, (testX.shape[0], look_back, 1))
    ########################
    # The IMPORTANT BIT
    ##########################
    # create and fit the LSTM network
    batch_size = 1
    model = Sequential()
    model.add(LSTM(4, batch_input_shape=(batch_size, look_back, 1), stateful=True))
    model.add(Dense(1))
    model.compile(loss=&#39;mean_squared_error&#39;, optimizer=&#39;adam&#39;)
    for i in range(100):
        model.fit(trainX, trainY, nb_epoch=1, batch_size=batch_size, verbose=2, shuffle=False)
        model.reset_states()
Note: create_dataset takes a sequence of length N and returns a `N-look_back` array of which each element is a `look_back` length sequence.    

# What is Time Steps and Features?
As can be seen TrainX is a 3-D array with Time_steps and Feature being the last two dimensions respectively (3 and 1 in this particular code). With respect to the image below, does this mean that we are considering the `many to one` case, where the number of pink boxes are 3? Or does it literally mean the chain length is 3 (i.e. only 3 green boxes considered). [![enter image description here][3]][3]

Does the features argument become relevant when we consider multivariate series? e.g. modelling two financial stocks simultaneously? 

# Stateful LSTMs
Does stateful LSTMs mean that we save the cell memory values between runs of batches? If this is the case, `batch_size` is one, and the memory is reset between the training runs so what was the point of saying that it was stateful. I&#39;m guessing this is related to the fact that training data is not shuffled, but I&#39;m not sure how.

Any thoughts?
Image reference: http://karpathy.github.io/2015/05/21/rnn-effectiveness/

## Edit 1: 
A bit confused about @van&#39;s comment about the red and green boxes being equal. So just to confirm, does the following API calls correspond to the unrolled diagrams? Especially noting the second diagram (`batch_size` was arbitrarily chosen.):
[![enter image description here][4]][4]
[![enter image description here][5]][5]

## Edit 2:
For people who have done Udacity&#39;s deep learning course and still confused about the time_step argument, look at the following discussion: https://discussions.udacity.com/t/rnn-lstm-use-implementation/163169

## Update:
It turns out `model.add(TimeDistributed(Dense(vocab_len)))` was what I was looking for. Here is an example: https://github.com/sachinruk/ShakespeareBot

## Update2:
I have summarised most of my understanding of LSTMs here: https://www.youtube.com/watch?v=ywinX5wgdEU


  [1]: http://colah.github.io/posts/2015-08-Understanding-LSTMs/
  [2]: http://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/
  [3]: http://i.stack.imgur.com/kwhAP.jpg
  [4]: http://i.stack.imgur.com/sW207.jpg
  [5]: http://i.stack.imgur.com/15V2C.jpg

Understanding Keras LSTMs

I have a functional model in Keras (Resnet50 from repo examples). I trained it with `ImageDataGenerator` and `flow_from_directory` data and saved model to `.h5` file. When I call `model.predict` I get an array of class probabilities. But I want to associate them with class labels (in my case - folder names). How can I get them? I found that I could use `model.predict_classes` and `model.predict_proba`, but I don&#39;t have these functions in Functional model, only in Sequential.

Content Type	Original Author	Original Content on Stackoverflow
Question	malioboro	View Question on Stackoverflow
Solution 1 - Neural Network	Dr. Snoopy	View Answer on Stackoverflow
Solution 2 - Neural Network	Argho Chatterjee	View Answer on Stackoverflow

Why does prediction needs batch size in Keras?

Neural Network Problem Overview

Neural Network Solutions

Solution 1 - Neural Network

Solution 2 - Neural Network

seaborn color_palette as matplotlib colormap

Where to put specs for Clojure.Spec?

Attributions