Deep Neural Networks with R, TensorFlow and Keras

I am a big proponent of using the R-Studio for Machine Learning, as it offers the superb environment where you are in a driving seat and can do even most complicated tasks with a relative ease. Now with the packages for TensorFlow it has become even more powerful.

This post aim at providing an introductory information on what the deep neural networks are good for, and how the R and TensorFlow can be used in this respect. Since I am a firm believer in providing just enough theory so that the concepts can be understood and then moving on with a real-world scenario, this is the way we will do it here as well. The focus of this post is to provide you with a basic info on how to use RStudio, R, TensorFlow, deep neural networks and Keras for a simple machine learning project.

It would be good if you were somewhat familiar with neural networks and basic machine learning concepts, such as using regression, classification and clustering at a basic level. Also, understanding Algebra is a must for any work in artificial intelligence, namely linear equations, functions and graphs, vectors and matrices and basic probability and statistical functions. Knowing Calculus topics, such as Limits, Derivates and Integrals would be a huge plus. Tooling today is so advanced and libraries that exist are so powerful, that you really do not need to be proficient in Math to run machine learning experiments, but if you really want to understand what is going on and not just run automated tasks with 99% of the work being done by some packages, then you should really have a good understanding of the underlying mathematical theory.

The Artificial Intelligence is a big topic these days, and more so with several dystopian scenarios where machines take over everything and we are erased as living beings, or in the best scenario they decide to keep us as pets. Anybody who understands today’s technological level of AI will find this amusing, since AI is nowhere near that level that it can connect to human brain, exchange the information with it directly without any interface, or download the whole brain to a cloud, as it was suggested. I guess this is all entertaining for many people, so they keep on coming up with these grim forecasts, like Elon Musk of Tesla and SpaceX ( https://cnb.cx/359XwX0 ), Ray Kurzweil, Google’s director of Engineering ( https://bit.ly/3bJ6ME9 ) or David Icke ( https://bit.ly/2W2jkjm ).

Apart from being amusing, I do not find too much merit in these views, since the “intelligent” bit in deep learning networks is really down to mathematical functions that “learn” by using some kind of a loss function that automatically optimizes the parameters, and then runs the transformation functions over and over again until the model is trained sufficiently. There is really nothing akin human-like “thinking” there, and we should not even start with things like feelings, deep intuition, inspiration or such. So it is all very technical and firmly rooted in mathematics, and reminds me a little bit of a period in the dawn of the industrial revolution when people would smash the machines as they would perceive them as being a threat to their jobs and lives. This is now a similar situation, just the machine learning algorithms are getting better and better, with the big data in clouds so that we have the data samples that were unthinkable of before and that we can train the models on, but again it just boils down to sophisticated mathematical operations that can solve complex logical and other more fuzzy-like challenges that may appear to be produced by a thinking machine, which is really not the case. To conclude, I see no reason for all this fuss and bad publicity, but we should instead focus on all the good it brings. For that matter, let us leave the philosophical aspects of AI to the philosophers, and all the skewed reality to the imagination of film makers to create something fun to watch, even though not rooted in any real science. Later in this post we’ll get into the mechanics on how these networks and layers work in order to at least conceptually understand it in a more generalized way, but for now let us first get oriented what role deep learning plays in today’s field of the AI. We can represent this relationship with the figure 1.1. below:

 

In the overall field of AI, the first attempts to solve logical problems was made with symbolic programming, then it improved and grew into the machine learning in order to solve other not so well-defined logical challenges. Now we have a further specialization of machine learning in Deep Learning networks, which achieve their result by implementing many layers of data transformation in order to train the models well. Deep learning heavily relies on the existence of big data and employing it on relatively small data sample is likely not to achieve better results than the standard machine learning algorithms.

To understand what it does and how it defers from the machine learning, we first need to understand how the machine learning is different than regular programming. The simplest way to put it is to understand that machine learning is an attempt to not use the regular programming where we have certain rules coded in some programming language and then they operate on some input data and provide the output, but the machine learning instead is trying to analyze data and sort of understand the connections and relations in the data, and comes up with the set of rules itself. This may seem simple enough, and it is, but applied correctly it can solve many intellectual challenges in real life.

Now deep learning is an additional improvement on machine learning algorithms where it used the models that are usually represented as neural networks and then applies transformations with many layer, possibly hundreds of them, in order to make better and better representations that all improve the model in each successive run. Not going too deep down the rabbit hole, it is still beneficial that we understand what is under the hood, at least conceptually. Let us consider the following model:

 

If the deep neural network in Figure 1.1. reminds you of a feed-forward neural network, that is because it is a feed-forward network with many hidden layers.

As shown in figure 1.1, the inputs are fed into the neural network which consists of many layers, often several hundreds, and the output is shown on the right-hand side. So the processing flows between multiple layers, and the actual learning occurs in the loss function that is used to calculate the output that is used to improve the learning, and then the optimizer is deciding on how it will incorporate that result in the proceeding. Going forwarder between these layers is called forwardpropagation, and going back is called backpropagation.

We are going to use one or more tensors as input into the network (more on tensors later when we introduce the TensorFlow, for now just think of tensors as various shaped arrays and matrices), and they will be processed by neural networks from Keras library (more on that later as well). Suffice to say that Keras library helps with implementing various, often very complicated neural networks, as it offers more than 60 typical neural networks, plus others that you can define yourself. In general, we create deep neural networks by combining the compatible neural networks into sort of a pipeline for processing and transforming the input data.

When I say compatible, it really means that not every neural network can accept any type of data. For instance, if we have a standard two-dimensional data in the form of a matrix:

then we are creating a two-dimensional matrix with 3 rows, 3 columns, and filling it in with number 11, so something like this:

Now, we process this kind of data with dense or fully connected layers from Keras. On the other hand, if we have image data, stored in 4D tensors, then we use convolutional layers for processing, and so on, you get the idea. It is important to just understand that certain shape of data used as inputs should be processed by certain types of layers, and those layers then output the data in a specific shape again. As we shall see, Keras is a great library that helps us a lot to provide the proper shape of data to the next layer by being very smart in inferring the data shapes, but more on that later.

Back to the Figure 1.1., the key part of the deep neural networks are loss functions 

and optimizers used to determine how the success determined in the loss function should be used to update the network. It usually uses stochastic gradient descent algorithm to do the work. Loss function and optimizers are the heart of the process, so we should examine them more carefully. The whole process looks like a pipeline that runs something like this:

 

Loss Function – as seen in Figure 1.2 above, the input is fed to the first layer, and then the epoch starts. Loss function is used to compare the predicted output with the actual output, after each pass of the back-propagating algorithm. This loss is fed to the loss function, which then determines how the weights are to be updated. We can use various loss functions, but it is also possible to write our own loss functions, in situations when we have complicated cases or want to achieve high level of predictions. Loss function is run only once after all layers, but if there are multiple loss functions then they still get combined into one, and then call the optimizer to update the weights. After that, the backward propagation is completed, and the next epoch starts.

 

Please note the function that is plotted here:

which is just a 2-D gradient of:

This approach is used when we have an objective function and this approach is using an iterative optimization method to find more optimized gradient. We are going to use a specific implementation of this algorithm called ADAM (Adaptive Moment Estimation) in our example later.

We can think of the gradient descent as a general problem of being in the mountain, and then trying to find the most optimized way to get down to the valley (Figure 1.3. above). The x and y values are to be considered as two values of weights, and z value is the value of the loss function. The goal is to find the value for loss function that will be as close to 0 as possible, and this represent the minima of the function. This is at the core of how the neural networks work, because calculating the loss and from there adjusting the weights is the most important part of the process.

 

If we walk down the hill toward the valley, we can define a plane that is tangential to any point at any moment, and in that exact point there is only one direction when the function would have the smallest loss, meaning the gradient is the steepest.

General approach to calculating the gradient descent can be represented like this:

where 

stands for overwriting the value on the left 

is the gradient of the function, representing all partial derivatives into one vector>

 is the learning rate

So that is just about enough of theory, now we should understand how deep learning neural networks work at least at a conceptual level.

Let us now go ahead and start with a simple project, so that we can understand how all different pieces work together. You will need this:

• R Studio with R
• Anaconda (if on Windows)
• TensorFlow
• Keras

Let us install all these so that we can have our environment ready.

Installing RStudio and R

R is a programming language created for science, with thousands of libraries for almost any work you can imagine. We are here interested in machine learning packages, but first we need to install the RStudio. So, go ahead to this location and install the latest version of R first (version 4.0 was the latest one at the time of this post):

Then go here and install the free version of RStudio:

Open RStudio, you should see something like this:

As you can see, it displays the R version 4.0.0 in the console. Quick orientation here: in the console you type the commands, but you can also go File -> New File -> R Script, and type in the script, save it, then execute the script. Top right window will show the environment objects and their values, and to the bottom right side you have packages or files. You can install the packages from here, but it is more common you would do that from the Console itself. This is just an absolute minimum, there are of course many more features to RStudio, but this should get you started.

Install Anaconda

Install Anaconda Individual from this location:

It is a toolkit for using Python or R for your data science projects, and you need it if installing TensorFlow through RStudio, as we shall be doing. You can also decide to skip this step, and then you shall be asked to install Miniconda during the TensorFlow installation in RStudio, since TensorFlow is distributed as python packages so there needs to be a python environment installed on your box.

Install TensorFlow 2.0

Open RStudio, and type the following in the console:

If you have not installed Anaconda in the previous step, you will be asked to install Miniconda at this point. Accept it, and then just lean back and wait while all the packages are installed. It can take up to a minute, depending on your network connection.

Now execute the following two lines in the RStudio console:

This will select the tensorflow library (go to the bottom right window and you can see that this package has been selected), and then the second command install the tensorflow. Now if you type the following two command:

…you can assert that the TensorFlow is installed properly (see the third in white color, as above).

It is important to note several things here…first, it is recommended to use UNIX like systems for RStudio, TensorFlow and Keras. Secondly, if you are doing a true demanding data science work, then you need a great graphic card, in order to run these computations on GPU, rather than on CPU.

You can look for Nvidia CUDA enabled graphic cards here: https://developer.nvidia.com/cuda-gpus

CUDA is a toolkit created by Nvidia that increases the computational performance of linear algebra, deep learning, images processing and many other domains, which makes it perfect when running demanding computational training of neural networks. Of course, for a real work you should go for something like Nvidia Titan RTX which is based on Turing architecture, and uses the Tensor Cores to immensely improve the artificial intelligence computational speed, but cards like this usually go around $2,500 or so, so it is a pretty pricy toy.

The other choice would be to run the computations on Google cloud or Microsoft Azure. For instance, in Microsoft Azure you can shoot for machine learning processing on dedicated machine which use Nvidia Tesla cards, but this option also does not get cheat if you are doing serious work with deep learning networks and images processing for instance, or such.

When I say UNIX based system, that basically means either Mac or some Linux distro. Again, Mac is a worst choice, since those machines usually do not use Nvidia graphic cards, which leaves you with the Linux then. You should use either Ubuntu, or some distro that was forked out of Ubuntu, like Linux Mint or such. Those work best in my personal experience.

In any case, please bear in mind that you can install all these running on CPU, which would suffice for playing and testing, as in this post, but the processing of even medium-size deep neural networks is likely to take 3-5 hours on CPU, rather than less than 30 minutes if ran on GPU with CUDA.

Keras

Same as TensorFlow, we execute the following command in RStudio console:

Before this, you should create a new R session since it may complain about some dlls being in use or something. Then you go:

This will install the CPU version of Keras. If you have Nvidia graphic card as noted above in the previous section, then you should use this command instead:

Checking the current version is also easy, for instance:

Now this completes the setup of the environment, we now should have RStudio installed, R, TensorFlow and Keras library.

Let us now briefly make an introduction on what the TensorFlow and Keras are, and then we would go on creating the test project to put everything together.

KERAS

EXAMPLE PROJECT