What is deep learning of neural networks. The future of deep learning. What is a neural network

The Coming Revolution smart robots has been predicted every ten years since the 1950s. However, it never happened. Progress in the field artificial intelligence took place uncertainly, sometimes boringly, bringing disappointment to many enthusiasts. Visible successes - the Deep Blue computer, created in the mid-1990s by IBM and which beat Garry Kasparov in chess in 1997, or appearance in the late 1990s electronic translator- were more likely the result of “rough” calculations than the transfer of the mechanisms of human perception to the processes of computer calculations.

However, the story of disappointment and failure is now changing dramatically. Just ten years ago, computer vision and object recognition algorithms could identify a sphere or a parallelepiped on simple background. They can now distinguish human faces as well as humans can, even against complex, natural backgrounds. Six months ago, Google released an application for smartphones that can translate text from more than 20 foreign languages by reading words from photographs, road signs or handwritten text!

All this became possible after it became clear that some old ideas in the field of neural networks, if slightly modified by adding “life”, i.e. projecting the details of human and animal perception can give a stunning result that no one expected. This time, the artificial intelligence revolution seems truly real.

Neural network research in the field machine learning in most cases were always devoted to the search for new recognition techniques various types data. Thus, a computer connected to the camera must, using an image recognition algorithm, be able to distinguish Bad quality a human face, a cup of tea or a dog. Historically, however, the use of neural networks for these purposes has been accompanied by significant difficulties. Even minor success required human intervention - people helped the program determine important features images such as image borders or simple geometric figures. Existing Algorithms They couldn’t learn to do it themselves.

The situation has changed dramatically thanks to the creation of the so-called neural networks with deep learning , which can now analyze an image almost as efficiently as a human. Such neural networks use a poor quality image as input to the first level “neurons”, which then transmits the “picture” through non-linear connections to the next level neurons. After a certain training, the “neurons” are more high levels can use more abstract aspects of an image for recognition. For example, they can use details such as the borders of the image or features of its location in space. Amazingly, such networks can learn to estimate the most important features of an image without human help!

A great example of the use of neural networks with deep learning is the recognition of identical objects photographed from different angles or in different poses (if we are talking about a person or an animal). Algorithms that use pixel-by-pixel scanning “think” that they are looking at two different images, while “smart” neural networks “understand” that they are looking at the same object. And vice versa - images of two dogs of different breeds, photographed in the same pose, could be perceived by previous algorithms as photographs of the same dog. Neural networks with deep learning can identify details in images that will help them distinguish between animals.

Combining deep learning techniques, advanced neuroscience knowledge and power modern computers opens for artificial intelligence prospects that we cannot even assess yet. The truth is already obvious that the mind can have not only a biological nature.

There is a lot of talk and writing about artificial neural networks today, both in the context of big data and machine learning and outside it. In this article, we will recall the meaning of this concept, once again outline the scope of its application, and also talk about an important approach that is associated with neural networks - deep learning, we will describe its concept, as well as the advantages and disadvantages in specific use cases.

What is a neural network?

As you know, the concept of a neural network (NN) comes from biology and is a somewhat simplified model of the structure human brain. But let’s not delve into the wilds of natural science - the easiest way is to imagine a neuron (including an artificial one) as a kind of black box with many input holes and one output.

Mathematically, artificial neuron performs vector transformation input signals(impacts) X into the output vector Y using a function called the activation function. Within the connection (artificial neural network - ANN), three types of neurons function: input (receiving information from the outside world - the values ​​of the variables we are interested in), output (returning the desired variables - for example, forecasts, or control signals), as well as intermediate neurons , performing certain internal (“hidden”) functions. A classical ANN thus consists of three or more layers of neurons, and in the second and subsequent layers (“hidden” and output), each element is connected to all elements of the previous layer.

It is important to remember the concept feedback, which determines the type of ANN structure: direct signal transmission (signals go sequentially from the input layer through the hidden layer and enter the output layer) and recurrent structure, when the network contains connections going back, from more distant to nearer neurons). All these concepts make up minimum required information to go to next level understanding ANN - training a neural network, classifying its methods and understanding the principles of operation of each of them.

Neural network training

We should not forget why such categories are used in general - otherwise there is a risk of getting bogged down in abstract mathematics. In fact, artificial neural networks mean a class of methods for solving certain practical problems, among which the main ones are the problems of pattern recognition, decision making, approximation and data compression, as well as the most interesting for us problems of cluster analysis and forecasting.

Without going to the other extreme and without going into details of the operation of ANN methods in each specific case, let us remind ourselves that under any circumstances it is the ability of a neural network to learn (with a teacher or “on its own”) that is key point using it to solve practical problems.

IN general case, ANN training is as follows:

1. input neurons receive variables (“stimuli”) from the external environment;
2. in accordance with the information received, the free parameters of the neural network change (intermediate layers of neurons work);
3. As a result of changes in the structure of the neural network, the network “reacts” to information in a different way.

That's how general algorithm training a neural network (let’s remember Pavlov’s dog - yes, that’s exactly what the internal mechanism for the formation of a conditioned reflex is - and let’s immediately forget: after all, our context involves operating with technical concepts and examples).

It is clear that a universal learning algorithm does not exist and, most likely, cannot exist; Conceptually, approaches to learning are divided into supervised learning and unsupervised learning. The first algorithm assumes that for each input (“learning”) vector there is a required value of the output (“target”) vector - thus, these two values ​​form a training pair, and the entire set of such pairs is the training set. In the case of unsupervised learning, the training set consists only of input vectors - and this situation is more plausible from the point of view of real life.

Deep learning

The concept of deep learning refers to a different classification and denotes an approach to training so-called deep structures, which include multi-level neural networks. A simple example from the field of image recognition: it is necessary to teach a machine to identify increasingly abstract features in terms of other abstract features, that is, to determine the relationship between the expression of the entire face, eyes and mouth, and, ultimately, clusters of colored pixels mathematically. Thus, in a deep neural network, each level of features has its own layer; it is clear that to train such a “colossus”, the appropriate experience of researchers and the level of hardware. Conditions developed in favor of deep learning of neural networks only by 2006 - and eight years later we can talk about the revolution that this approach produced in machine learning.

So, first of all, in the context of our article it is worth noting the following: deep learning in most cases it is not controlled by humans. That is, this approach involves training a neural network without a teacher. This is the main advantage of the “deep” approach: supervised machine learning, especially in the case of deep structures, requires enormous time – and labor – costs. Deep learning, on the other hand, is an approach that models (or at least attempts to approximate) human abstract thinking, rather than using it.

The idea, as usual, is wonderful, but quite natural problems arise in the way of the approach - first of all, rooted in its claims to universality. In fact, while deep learning approaches have achieved significant success in the field of image recognition, the same natural language processing still raises many more questions than answers. It is obvious that in the next n years it is unlikely that it will be possible to create an “artificial Leonardo Da Vinci” or even - at least! - “artificial homo sapiens”.

However, artificial intelligence researchers are already faced with the question of ethics: the fears expressed in every self-respecting science fiction film, from “Terminator” to “Transformers”, no longer seem funny (modern sophisticated neural networks can already be considered a plausible model the work of the insect's brain!), but are clearly unnecessary for now.

The ideal technological future appears to us as an era when a person will be able to delegate most of his powers to a machine - or at least be able to allow it to facilitate a significant part of his intellectual work. The concept of deep learning is one step towards this dream. The road ahead is long, but it is already clear that neural networks and all the developing approaches associated with them are capable of realizing the aspirations of science fiction writers over time.

From the article you will learn what deep learning is. The article also contains many resources that you can use to master this area.

IN modern world From healthcare to manufacturing, deep learning is being used everywhere. Companies are turning to this technology to solve complex problems, such as speech and object recognition, machine translation and so on.

One of the most impressive achievements this year was AlphaGo beating the world's best Go player. In addition to Go, machines have beaten people in other games: checkers, chess, reversi, and Jeopardy.

Possibly victory in board game seems inapplicable to the solution real problems, however, this is not at all the case. Go was designed to be unbeatable by artificial intelligence. To do this, he would need to learn one important thing for this game - human intuition. Now, with the help of this development, it is possible to solve many problems that were inaccessible to a computer before.

Obviously, deep learning is still far from perfect, but it is already close to being commercially useful. For example, these self-driving cars. Well-known companies like Google, Tesla and Uber are already trying to introduce autonomous cars on city streets.

Ford predicts significant increase share of unmanned Vehicle by 2021. The US government also managed to develop a set of safety rules for them.

What is deep learning?

To answer this question, you need to understand how it interacts with machine learning, neural networks and artificial intelligence. To do this, we use the visualization method using concentric circles:

The outer circle is artificial intelligence in general (e.g. computers). A little further is machine learning, and right in the center are deep learning and artificial neural networks.

Roughly speaking, deep learning is simply a more convenient name for artificial neural networks. “Deep” in this phrase refers to the degree of complexity (depth) of the neural network, which can often be quite superficial.

The creators of the first neural network were inspired by the structure of the cerebral cortex. The network's base layer, the perceptron, is essentially the mathematical analogue of a biological neuron. And, as in the brain, perceptrons intersecting with each other can appear in a neural network.

The first layer of the neural network is called the input layer. Each node in this layer receives some information as input and transmits it to subsequent nodes in other layers. Most often, there are no connections between the nodes of one layer, and the last node of the chain outputs the result of the neural network.

The nodes in the middle are called hidden nodes because they have no connections to outside world, as output and input nodes. They are called only when previous layers are activated.

Deep learning is essentially a neural network training technique that uses many layers to solve complex problems (like speech recognition) using patterns. In the eighties, most neural networks were single-layer due to high cost and limited data capabilities.

If we consider machine learning as a branch or variant of artificial intelligence, then deep learning is specialized type such a branch.

Machine learning uses computer intelligence that does not immediately provide the answer. Instead, the code will run on test data and, based on the correctness of its results, adjust its progress. For the success of this process, a variety of techniques, special software and computer science are usually used to describe static methods and linear algebra.

Deep learning methods

Deep learning methods are divided into two main types:

• Tutored training
• Unsupervised learning

The first method uses specially selected data to achieve the desired result. It requires quite a lot of human intervention, because the data has to be selected manually. However, it is useful for classification and regression.

Imagine that you are the owner of a company and want to determine the impact of bonuses on the length of contracts of your subordinates. With pre-collected data, a supervised learning method would be indispensable and very effective.

The second method does not imply pre-prepared answers and work algorithms. It aims to identify hidden patterns in data. It is typically used for clustering and association tasks, such as grouping customers by behavior. “They also choose with this” on Amazon is a variant of the association task.

While supervised learning is often quite convenient, it is more difficult option still better. Deep learning has proven itself to be a neural network that does not require human supervision.

The Importance of Deep Learning

Computers have long used technology to recognize certain features in an image. However, the results were far from successful. Computer vision has had an incredible impact on deep learning. It is these two techniques that this moment solve all recognition problems.

In particular, Facebook has succeeded in recognizing faces in photographs using deep learning. This is not a simple improvement in technology, but a turning point that changes all earlier beliefs: “A person can determine with 97.53% probability whether the same person is represented on two different photos. The program developed by the Facebook team can do this with a 97.25% probability, regardless of the lighting or whether the person is looking directly at the camera or turned sideways towards it.”

Speech recognition has also undergone significant changes. The team at Baidu, one of the leading search engines in China, has developed a speech recognition system that has managed to outpace humans in the speed and accuracy of writing text in English. mobile devices. In English and Mandarin.

What’s especially interesting is that writing a common neural network for two completely different languages ​​did not require much work: “Historically, people saw Chinese and English as two completely different languages, so each of them required a different approach,” says Andrew Ng, head of the Baidu research center. “Learning algorithms are now so generalized that you can Just learn."

Google uses deep learning to manage energy in the company's data centers. They were able to reduce cooling resource costs by 40%. That's about a 15% improvement in energy efficiency and millions of dollars in savings.

Deep learning microservices

Here short review services related to deep learning.

Illustration Tagger. Enhanced by Illustration2Vec, this service allows you to mark images with a rating of “protected”, “questionable”, “dangerous”, “copyright” or “general” in order to understand the content of the image in advance.

• Google's Theano Add-on
• Editable in Python and Numpy
• Often used to solve a specific range of problems

• Not general purpose. Focus on machine vision
• Edited in C++
• There is an interface in Python

Online courses on deep learning

Google and Udacity have teamed up to create a free course on deep learning, part of the Udacity Machine Learning Course. This program is led by experienced developers who want to develop the field of machine learning and, in particular, deep learning.

Another popular option is the machine learning course from Andrew Ng, supported by Coursera and Stanford.

1. Machine Learning - Stanford by Andrew Ng on Coursera (2010-2014)
2. Machine Learning - Caltech by Yaser Abu-Mostafa (2012-2014)
3. Machine Learning - Carnegie Mellon by Tom Mitchell (Spring 2011)
4. Neural networks for machine learning – Geoffrey Hinton on Coursera (2012)
5. Neural networks class– Hugo Larochelle from Université de Sherbrooke (2013)

Books on deep learning

While the resources in the previous section draw on a fairly extensive knowledge base, Grokking Deep Learning, on the contrary, is aimed at beginners. As the authors say: “If you have completed 11th grade and have a rough understanding of how to write Python, we will teach you deep learning.”

A popular alternative to this book is a book with the self-explanatory title Deep Learning Book. It's especially good because it covers all the math you'll need to get into this area.

1. 1. "Deep Learning" by Yoshua Bengio, Ian Goodfellow and Aaron Courville (2015)

1. “Neural networks and deep learning” by Michael Nielsen (2014)
2. "Deep Learning" from Microsoft Research (2013)
3. “Deep Learning Tutorials” from LISA Laboratory, University of Montreal (2015)
4. “neuraltalk” by Andrej Karpathy
5. "Introduction to Genetic Algorithms"
6. "Modern approach to artificial intelligence"
7. "Overview of deep learning and neural networks"

Videos and lectures

Deep Learning Simplified is a wonderful YouTube channel. Here's their first video:

— The laboratory is young: there are only five people in our team so far, the work is an unplowed field, but we are serious. The main focus was the development and research of dialogue systems - online consultants and assistants who competently answer all user questions. Many companies still have such services, but either they work poorly, constantly producing errors, or there is a living person on the other side of the monitor who cannot be online 24/7, and besides, he has to be paid. We want to develop an algorithm that will allow us to create robots capable of full-fledged conversation. Such a robot will be able to buy you a plane ticket in a matter of minutes or advise you on any pressing issue. Currently, this level of systems does not exist.

Neural networks and artificial intelligence

The idea of ​​neural networks was born in the middle of the 20th century in the USA along with the advent of the first computers. Neuroscientists who have studied theoretical aspects the work of the brain, they believed that organizing the work of a computer in the image and likeness of the work of the human brain would make it possible in the near future to create the first artificial intelligence.

The difference between artificial intelligence and all algorithms of the previous generation is that the trained neural network does not act according to given path, and independently looks for ways to most effectively achieve the goal. The operation of a single computer “neuron” looks like this: for training, the program input is supplied with objects belonging to two types - A and B - and carrying some kind of numerical value. The program, based on the data in the training set, understands which ranges of this value correspond to objects A and which to B, and can subsequently distinguish them independently. In real problems, the system must distinguish between many types, each of which, in turn, can have dozens of properties. To solve them, a more complex structure of layers of neurons is needed, serious computing power And a large number of training tests. The 21st century has marked the beginning of an era in which these technologies can already be used to solve everyday problems.

Mikhail Burtsev, head of the laboratory:

— The concept of how neural networks work is quite simple: we give the machine a large amount of text, and it remembers how the words fit together. Based on this information, it can reproduce similar texts - the machine does not need to know the rules of syntax, declension and conjugation. There are already neural networks that, having learned from Pushkin’s works, are trying to write in his style. This is another feature of neural networks: they learn the “style” that they are given for learning. If you give Wikipedia as material, the program will sprinkle terms and use predominantly journalistic style. Since our laboratory is working on creating question-answering systems, we use ready-made dialogues to train the network. In one of the experiments, they used subtitles from films and let our network study a whole saga about vampires. Having analyzed this array of data, the neural network can already support the conversation.

Dialogues between laboratory staff and a neural network

Team: today and tomorrow

The laboratory cooperates with large research centers based at the National Research Nuclear University MEPhI and the Kurchatov Institute. Foreign experts in the field of machine learning and neuroinformatics also take part in its activities, for example Sergey Plis from The Mind Research Network. In addition, events are regularly held to popularize the laboratory’s activities and search for young talents. Winning a hackathon or successfully completing a course gives you a good chance of getting into the laboratory.

Valentin Malykh, laboratory employee:

“My path to the laboratory was very difficult. Just four years ago, I practically did not touch the topic of machine learning. Then I took up computer linguistics, and off we go... I changed jobs several times: I tried my hand at robotics, I worked in development software associated with computer vision, it was there that I became acquainted with machine learning, and I wanted to do serious research.
During all my work, I managed to go to several hackathons organized by the laboratory - perhaps the most interesting thing that happened to me during that period. Then I came to the guys and said that I wanted to work for them. They took me.

Philosophy of DeepHack

Hackathons, despite their name, have nothing to do with hacking software ( English hack - to hack). These are team programming competitions in which participants spend several days, and sometimes weeks, struggling to solve one specific problem. The theme of the hackathon is announced in advance, and usually several hundred people participate. Such events are organized not only by institutes, but also large companies who are looking for talented specialists. At the Phystech Institute, the Laboratory of Neural Networks and Deep Learning has already organized two hackathons - participants spent a week listening to lectures on question-answering and dialogue systems and writing code.

Vladislav Belyaev, laboratory employee:

— This year and last year we organized hackathons on machine learning. There were a lot of applications, not only from Russia and the CIS, but also from Europe and the States. During the hackathon, lectures were given by scientists from Oxford and Stanford, Google DeepMind and OpenAI, and Russian colleagues, Certainly. Now we are preparing a course on neural networks, we will tell you everything from the very beginning to the end: from the biological concept and basic models in programming to the actual application and specific implementation.

Free time

There are still few employees in the laboratory, so each person has a large amount of work of a different nature: they need to study algorithms, write code, and prepare scientific publications.

Mikhail Burtsev, head of the laboratory:

“You have to work a lot—I don’t think I remember what it’s like anymore.” free time. No joke, there is practically no time to relax: over the past six months we have been able to go out to barbecue once in a group. Although in a sense, work can be relaxation. Hackathons and seminars provide an opportunity to communicate in a less formal setting with colleagues and make new acquaintances. We haven’t yet managed to start traditions of spending time together after work - we’re too young. In the summer we plan to get out into nature with the whole laboratory, rent a cottage and solve the most difficult and interesting problems together for two weeks - we will organize our own mini-hackathon. Let's see how effective this approach can be. Perhaps this will become our first good tradition.

Employment

The laboratory will expand and is already looking for new employees. The easiest way to get a place is to complete a two-month internship, for which you are selected based on an interview. A necessary condition Passing the interview is to complete part of the tasks of the Deep Learning course. During the internship, you have the opportunity to participate in paid custom projects. Funding for the laboratory has not yet been arranged, however, according to laboratory staff, this problem will be solved in the near future. “To come to us now means to get a chance to become the “founding father” of the laboratory in the most promising direction information technologies“says Mikhail Burtsev.

Images and photographs were provided by the MIPT Laboratory of Neural Networks and Deep Learning. Photographer: Evgeny Pelevin.

What is deep learning ( deep learning) ? March 3rd, 2016

Nowadays they talk about fashionable deep learning technologies as if it were manna from heaven. But do the speakers understand what it really is? But this concept has no formal definition, and it combines a whole stack of technologies. In this post I want to explain as popularly as possible and essentially what is behind this term, why it is so popular and what these technologies give us.

In short, this newfangled term (deep learning) is about how to assemble a more complex and deeper abstraction (representation) from some simple abstractions. despite the fact that even the simplest abstractions must be assembled by the computer itself, and not by a person. Those. It’s no longer just about learning, but about meta-learning. Figuratively speaking, the computer itself must learn how best to learn. And, in fact, this is exactly what the term “deep” implies. Almost always, this term is applied to artificial neural networks that use more than one hidden layer, so formally “deep” also means a deeper neural network architecture.

Here in the development slide you can clearly see how deep learning differs from ordinary learning. I repeat, What's unique about deep learning is that the machine finds the features itself(the key features of something by which it is easiest to separate one class of objects from another) and structures these signs hierarchically: simpler ones are combined into more complex ones. Below we will look at this with an example.

Let's look at an example of an image recognition task: before, they stuffed a huge one into a regular neural network with one layer (1024×768 - about 800,000 numerical values) picture and watched the computer slowly die, suffocating from lack of memory and the inability to understand which pixels are important for recognition and which are not. Not to mention the effectiveness of this method. Here is the architecture of such a regular (shallow) neural network.

Then they listened to how the brain distinguishes features, and it does this in a strictly hierarchical manner, and they also decided to extract a hierarchical structure from the pictures. To do this, it was necessary to add more hidden layers (layers that are between the input and output; roughly speaking, information transformation stages) to the neural network. Although they decided to do this almost immediately when neurons were invented, then networks with only one hidden layer were successfully trained. Those. basically deep networks have been around for about as long as regular ones, we just couldn’t train them. What has changed?

In 2006, several independent researchers solved this problem at once (besides, hardware capabilities had already developed enough, quite powerful video cards appeared). These researchers are: Geoffrey Hinton (and his colleague Ruslan Salakhutidinov) with the technique of pre-training each layer of a neural network with a constrained Boltzmann machine (forgive me for these terms...), Yann LeCun with convolutional neural networks, and Yoshuay Bengio with cascaded autoencoders. The first two were immediately recruited by Google and Facebook, respectively. Here are two lectures: one - Hinton, the other - Lyakuna, in which they tell what deep learning is. No one can tell you about this better than them. Another cool one lecture Schmidhuber about the development of deep learning, also one of the pillars of this science. And Hinton also has an excellent course on neurons.

What can deep neural networks do now? They are able to recognize and describe objects; one might say they “understand” what it is. It's about about recognizing meanings.

Just watch this video of real-time recognition of what the camera sees.

As I already said, deep learning technologies are a whole group of technologies and solutions. I have already listed several of them in the paragraph above, another example is recurrent networks, which are used in the video above to describe what the network sees. But the most popular representative of this class of technologies is still LyaKun’s convolutional neural networks. They are built by analogy with the principles of operation of the visual cortex of the cat’s brain, in which the so-called simple cells were discovered, reacting to straight lines at different angles, and complex cells - the reaction of which is associated with activation a certain set simple cells. Although, to be honest, LaCun himself was not focused on biology, he decided specific task(watch his lectures), and then it coincided.

To put it very simply, convolutional networks are networks where the main structural element of learning is a group (combination) of neurons (usually a 3x3, 10x10 square, etc.), and not just one. And at each level of the network, dozens of such groups are trained. The network finds combinations of neurons that maximize information about the image. At the first level, the network extracts the most basic, structural simple elements pictures are, one might say, building units: boundaries, strokes, segments, contrasts. Higher up are already stable combinations of elements of the first level, and so on up the chain. I would like to emphasize once again main feature deep learning: the networks themselves form these elements and decide which of them are more important and which are not. This is important because in the field of machine learning, the creation of features is key and now we are moving to the stage when the computer itself learns to create and select features. The machine itself identifies a hierarchy of informative features.

So, during the learning process (viewing hundreds of pictures), the convolutional network forms a hierarchy of features various levels depths. At the first level, they can highlight, for example, such elements (reflecting contrast, angle, border, etc.).


At the second level, this will already be an element from the elements of the first level. On the third - from the second. We must understand that this picture- just a demonstration. Now in industrial use, such networks have from 10 to 30 layers (levels).

After such a network has trained, we can use it for classification. Having given some image as input, groups of neurons in the first layer run across the image, activating in those places where there is an element of the picture corresponding to a specific element. Those. this network parses the picture into parts - first into lines, strokes, angles of inclination, then more complex parts and in the end it comes to the conclusion that the picture is from this kind of combination basic elements- this face.

More about convolutional networks -