Нейросетевая модель распознавания человека по походке в видеоданных различной природы тема диссертации и автореферата по ВАК РФ 05.13.18, кандидат наук Соколова Анна Ильинична

• Соколова Анна Ильинична
• кандидат наук
• 2020

Оглавление диссертации кандидат наук Соколова Анна Ильинична

Contents

Introduction

1. Background

1.1 Influential factors

1.2 Gait recognition datasets

1.3 Literature review

1.3.1 Manual feature construction

1.3.2 Automatic feature training

1.3.3 Event-based methods

2. Baseline model

2.1 Motivation

2.2 Proposed algorithm

2.2.1 Data preprocessing

2.2.2 Neural network backbone

2.2.3 Feature postprocessing and classification

2.3 Experimental evaluation

2.3.1 Evaluation metrics

2.3.2 Experiments and results

2.4 Conclusions

3. Pose-based gait recognition

3.1 Motivation

3.2 Proposed algorithm

3.2.1 Body part selection and data preprocessing

3.2.2 Neural network pipeline

3.2.3 Feature aggregation and classification

3.3 Experimental evaluation

3.3.1 Performance evaluation

3.3.2 Experiments and results

3.4 Conclusions

4. View resistant gait recognition

4.1 Motivation

4.2 View Loss

4.3 Cross-view triplet probability embedding

4.4 Experimental evaluation

4.5 Conclusions

5. Event-based gait recognition

5.1 Dynamic Vision Sensors

5.2 Data visualization

5.3 Recognition algorithm

5.3.1 Human figure detection

5.3.2 Pose estimation

5.3.3 Optical flow estimation

5.4 Event-based data

5.5 Experimental evaluation

5.6 Conclusions

Conclusion

References

List of Figures

List of Tables

Введение диссертации (часть автореферата) на тему «Нейросетевая модель распознавания человека по походке в видеоданных различной природы»

Introduction

According to Maslow studies [1], safety need is one of the basic and fundamental human needs. People tend to protect themselves, preserve their housing from illegal invasion and save property from stealing.

With the development of modern video surveillance systems, it becomes possible to capture everything that happens in a certain area and then analyze the obtained data. Using video recordings, one can track the movements of people, determine illegal entry into private territory, identify criminals who get captured by the cameras, control access to restricted objects. For example, video surveillance systems help to catch burglars, robbers or arsonists, automatically count the number of people in a line or in crowd, and analyze the character of their movements reducing the amount of subjective human intervention and decreasing the time required for data processing. Besides this, being embedded in the currently widely used home assistance systems ("smart home"), they can distinguish family members and change behavior depending on the personality. For example, it can be configured to conduct different actions if it captures a child or elderly person.

Dissertation topic and its relevance

Recently, the problem of recognizing a person in a video (see Fig. 0.1) has become particularly urgent. A human's personality is identifiable in a video based on several criteria, and the most accurate one is facial features. However, the current recognition quality allows to entrust decision-making to a machine only in a cooperative mode, when person's face is compared with a high quality photograph in a passport. In real life (especially when committing crimes), a person's face may be hidden or poorly visible due to bad view, insufficient lighting, or the presence of a headgear, mask, makeup, etc. In this case, another characteristic is required to make the recognition, and gait is the possible one. According to biometric and physiological studies [2; 3], the manner the person walks is individual and can not be falsified, which makes gait a unique identifier comparable to fingerprints or the iris of the eyes. In addition, unlike these "classic" methods of identification, gait can be observed at a great distance, it does not require perfect resolution of the video and, most importantly, no direct cooperation with a human is needed, thus, a human

may not know that he is being captured and analyzed. So, in some cases gait serves as the only possible sign for determining a person in the video surveillance data.

Figure 0.1 — Informal problem statement: having a video with a walking person one needs to determine this person's identity from the database

The gait recognition problem is very specific due to the presence of many factors that change the gait visually (different shoes; carried objects; clothes that hide parts of the body or constrain movements) or affect the internal representation of the gait model (angle, various camera settings). In this regard, the quality and reliability of identification by gait is much lower than by face, and, despite the success of modern methods of computer vision, this problem has not been solved yet. Many methods are applicable solely to the conditions present in the databases on which they are trained, which limits their usability in real life.

In addition to the classic surveillance cameras that store everything that happens in the frame 25 — 30 times per second, other types of sensors, in particular, dynamic vision sensors (Dynamic Vision Sensors, DVS [4—6]), are gaining popularity in recent years. Unlike conventional video cameras, the sensor, like the retina, captures changes in intensity in each pixel, ignoring points with constant brightness. Under the conditions of a static sensor, events at background points are very rarely generated, preventing the storage of redundant information. At the same time, the intensity at each point is measured several thousand times per second, which leads to the asynchronous capture of all important changes. As a result, such a stream of events turns out to be very informative and suitable for obtaining the data necessary for solving many video analysis tasks that require the extraction of dynamic characteristics, including gait recognition.

Dynamic vision sensors are now a promising rapidly developing technology, which leads to the need to solve video analysis tasks for the received data. Despite the constant development of computer vision methods, no approaches to solving the

gait recognition problem according to the data of dynamic vision sensors have yet been proposed, and it represents a vast field for research.

The methods of deep learning based on the training of neural networks have become the most successful in solving computer vision problems in recent years. Attributes taught by neural networks often have a higher level of abstraction, which is necessary for high-quality recognition. This allows to achieve outstanding results in solving such problems as the classification of video and images, image segmentation, object detection, visual tracking, etc. However, despite the success of deep learning methods, classical computer vision methods are still ahead of neural networks in some gait recognition benchmarks and both approaches have not achieved acceptable accuracy for full integration yet.

Goals and objectives of the research

This research aims to develop and implement the neural network algorithm for human identification in video based on the motion of the points of human figure that is stable to viewing angle changes, different clothing and carrying conditions. To achieve this goal the following tasks are set:

1. Develop and implement the algorithm for human identification in video analysing the optical flow.

2. Develop an implement the multiview algorithm for gait recogntion based on the analysis of the motion of different human body parts.

3. Develop the algorithm for human recognition in the event-based data from Dynamic Vission Sensors.

Formal problem statement

The formal research objects are video surveillance frame sequences v and event streams e from the dynamic vision sensors where the moving person is captured. Having the labelled gallery D given one needs to deternime, which person from the gallery appears in video, i.e. the identity of the person in video is to be defined. Let the gallery be defined as

D = e P,

where N is the number of sequences and P is the set of subjects. The label of the subject x e P is to be found for the video v under investigation. The goal is to construct some similarity measure S according to which the closest object will be

searched for in the gallery.

S(v,vi) ^ min

Vi: 3xi. (vi,Xi)eD

The problem for the event streams is stated similarly. A set of restrictions is imposed on all the sequences:

- each video in the gallery contains exactly one person;

- each person is captured full length;

- no occlusions;

- static camera;

- the set of posible camera settings (its height and tilt) is limited.

The described conditions are introduced due to the limitations of the existing datasets and benchmarks.

Novelty and summary of the authors main results

In this thesis, the author introduces the original method for human recognition by gait stable to view changes, reducing the length of the video sequences and dataset transfer. The following is the list of the main research results. The list of the corresponding publications can be found in section Publications at the page 9.

1. Side-view gait recognition method which analyses the points translations between consecutive video frames is proposed and implemented. The method shows state-of-the-art quality on the side-view gait recognition benchmark.

2. Multi-view gait recognition method based on the consideration of movements of the points in different areas of human body is proposed and implemented. The state-of-the-art recognition accuracy is achieved for cerrain viewing angles and the best at the investigation time approaches are outperformed in verification mode.

3. The influence of the point movements in different body parts on the recognition quality is revealed.

4. Gait recongition method stable to dataset transfer is proposed and implemented.

5. Two approaches for view resistance improvement are proposed and implemented. Both methods increase the cross-view recognition quality and complement each other being applied simultaneously. The model

obtained using these approaches outperforms the state-of-the-art methods on multi-view gait recogntion benchmark.

6. The method for human recognition by motion in the event-based data from dynamic vision sensor is proposed and implemented. The quality close to conventional video recognition is obtained.

The described results are original and obtained for the first time. Below, the author's contributions are summarized in four main points.

1. The first gait recognition method based on the investigation of the point movements in different parts of the body is proposed and implemented.

2. Two original methods of view resistance improvements are proposed. In these approaches the auxiliary model regularization is made and the descriptors are projected into the special feature space decreasing the view dependency.

3. The original research of the gait recognition algorithm transfer between different data collections is made.

4. The first method for human recognition by motion in the event-based data from dynamic vision sensor is propose and implemented.

Practical significance

Gait recognition is an applied problem of computer vision. Being proposed according to natural and mathemathical reasons, all the suggested methods and approaches aim to be applicable. Thus, being implemented, the proposed human identification methods can be intergrated to different automation systems. For example, the developed approach can be used in the home assistance systems ("smart home") which recognize the family members and change the behaviour depending on the captured person. Being united with the alarm, the system can respond to the appearance of the people not included to the family, and track the illegal entrance into the private houses.

Besides this, the gait identification algorithm can be used in crowded places, such as train stations and airports, where it is not possible to take close-up shots, but there is an obvious need to track and control access.

Publications and approbation of the research

Main results of this thesis are published in the following papers. The PhD candidate is the main author in all of these articles. First-tier publications:

- A. Sokolova, A. Konushin, Pose-based deep gait recognition // IET Biometrics, 2019, (Scopus, Q2).

Second-tier publications:

- A. Sokolova, A. Konushin, Gait recognition based on convolutional neural networks // International Archives of the Photogrammetry, Remote Sensing & Spatial Information Sciences, 2017 (Scopus).

- A. Sokolova, A. Konushin, Methods of gait recognition in video // Programming and Computer Software, 2019 (WoS, Scopus, Q3).

- A. Sokolova, A. Konushin, View Resistant Gait Recognition // ACM International Conference Proceeding Series, 2019 (Scopus).

The results of this thesis have been reported at the following conferences and workshops:

- ISPRS International workshop "Photogrammetric and computer vision techniques for video surveillance, biometrics and biomedicine" - PSBB17, Moscow, Russia, May 15 - 17, 2017. Talk: Gait recognition based on convolutional neural networks.

- Computer Vision and Deep Learning summit "Machines Can See", Moscow, Russia, June 9, 2017. Poster: Gait recognition based on convolutional neural networks.

- 28th International Conference on Computer Graphics and Vision "GraphiCon 2018", Tomsk, Russia, September 24 - 27, 2018. Talk: Review of video gait recognition methods.

- Samsung Biometric Workshop, Moscow, Russia, April 11, 2019. Talk: Human identification by gait in RGB and event-based data.

- 16th International Conference on Machine Vision Applications (MVA), Tokyo, Japan, May 27 - 31, 2019. Poster: Human identification by gait from event-based camera.

- 3rd International Conference on Video and Image Processing (ICVIP), Shanghai, China, December 20 - 23, 2019. Talk: View Resistant Gait Recognition (best presentation award).

Thesis outline. The thesis consists of the introduction, background, four chapters corresponding to developed approaches and the conclusion.

Заключение диссертации по теме «Математическое моделирование, численные методы и комплексы программ», Соколова Анна Ильинична

Conclusion

In this thesis, the novel gait recognition approach is presented. Due to variability of influential factors and absense of one general database covering all possible conditions, the Author has selected some of the factors to overcome and proposed a method stable to their changes.

The main contributions of this thesis are as follows.

1. Side-view gait recognition method which analyses the points translations between consecutive video frames is proposed and implemented. The method shows state-of-the-art quality on the side-view gait recognition benchmark.

2. The first multi-view gait recognition method based on the consideration of movements of the points in different areas of human body is proposed and implemented. The state-of-the-art recognition accuracy is achieved for certain viewing angles and the best at the investigation time approaches are outperformed in verification mode.

3. The influence of the point movements in different body parts on the recognition quality is revealed.

4. The original research of the gait recognition algorithm transfer between different data collections is made.

5. Two original approaches for view resistance improvement are proposed and implemented. Both methods increase the cross-view recognition quality and complement each other being applied simultaneously. The model obtained using these approaches outperforms the state-of-the-art methods on multiview gait recogntion benchmark.

6. The first method for human recognition by motion in the event-based data from dynamic vision sensor is proposed and implemented. The quality close to conventional video recognition is obtained.

The further developement of the research is possible in the following areas:

1. Developement and implementation of view estimation methods by gait videos;

2. Integration of view information into a recognition model for identification quality improvement and increasing the view resistance;

3. Developement and implementation of multi-view RGB and event-based gait video synthesis by applying the motion capture methods and generation of 3D data to enlarge the existing training and testing datasets.

4. Synthetic data application for multi-view recognition quality increase in different types of data (RGB, event streams).

Список литературы диссертационного исследования кандидат наук Соколова Анна Ильинична, 2020 год

References

1. Maslow, A. Motivation and Personality / A. Maslow. — Oxford, 1954. — 411 p.

2. Cutting, J. E. Recognizing friends by their walk: Gait perception without familiarity cues / J. E. Cutting, L. T. Kozlowski // Bulletin of the Psychonomic Society. - 1977. - Vol. 9, no. 5. - P. 353-356.

3. Murray, M. P. GAIT AS A TOTAL PATTERN OF MOVEMENT / M. P. Murray // American Journal of Physical Medicine & Rehabilitation. — 1967. - Vol. 46.

4. Lichtsteiner, P. A 128 x 128 120 dB 15 |as Latency Asynchronous Temporal Contrast Vision Sensor / P. Lichtsteiner, C. Posch, T. Delbruck // IEEE Journal of Solid-State Circuits. - 2008. - Vol. 43, no. 2. - P. 566-576.

5. 4.1 A 640x480 dynamic vision sensor with a 9ym pixel and 300Meps address-event representation / B. Son [et al.] // 2017 IEEE International Solid-State Circuits Conference (ISSCC). - 02/2017. - P. 66-67.

6. A 240 x 180 130 dB 3 ys Latency Global Shutter Spatiotemporal Vision Sensor / C. Brandli [et al.] // IEEE Journal of Solid-State Circuits. — 2014. — Oct. - Vol. 49, no. 10. - P. 2333-2341.

7. The TUM Gait from Audio, Image and Depth (GAID) database: Multimodal recognition of subjects and traits. / M. Hofmann [et al.] //J. of Visual Com. and Image Repres. - 2014. - Vol. 25(1). - P. 195-206.

8. The OU-ISIR Gait Database Comprising the Large Population Dataset and Performance Evaluation of Gait Recognition / H. Iwama [et al.] // IEEE Trans. on Information Forensics and Security. — 2012. — Oct. — Vol. 7, Issue 5. - P. 1511-1521.

9. Yu, S. A Framework for Evaluating the Effect of View Angle, Clothing and Carrying Condition on Gait Recognition. / S. Yu, D. Tan, T. Tan // Proc. of the 18'th ICPR. Vol. 4. - 2006. - P. 441-444.

10. Hofmann, M. Gait Recognition in the Presence of Occlusion: A New Dataset and Baseline Algorithms / M. Hofmann, S. Sural, G. Rigoll // 19th International Conferences on Computer Graphics, Visualization and Computer Vision (WSCG). - Plzen, Czech Republic, 01/2011.

11. Efficient Night Gait Recognition Based on Template Matching / Daoliang Tan [et al.] // 18th International Conference on Pattern Recognition (ICPR'06). Vol. 3. - 08/2006. - P. 1000-1003.

12. Gross, R. The CMU Motion of Body (MoBo) Database : tech. rep. / R. Gross, J. Shi ; Carnegie Mellon University. — Pittsburgh, PA, 06/2001.

13. Gait Recognition under Speed Transition / A. Mansur [et al.] //. — 06/2014.

14. The OU-ISIR Gait Database Comprising the Treadmill Dataset / Y. Makihara [et al.] // IPSJ Trans. on Computer Vision and Applications. — 2012. — Apr. - Vol. 4. - P. 53-62.

15. The OU-ISIR Gait Database Comprising the Large Population Dataset with Age and Performance Evaluation of Age Estimation / C. Xu [et al.] // IPSJ Trans. on Computer Vision and Applications. — 2017. — Vol. 9, no. 24. — P. 1-14.

16. Lee, L. Gait Analysis for Recognition and Classification / L. Lee, W. E. L. Grimson // Proc IEEE Int Conf Face Gesture Recognit. — 06/2002. - P. 148-155.

17. The HumanID gait challenge problem: data sets, performance, and analysis / S. Sarkar [et al.] // IEEE Transactions on Pattern Analysis and Machine Intelligence. - 2005. - Feb. - Vol. 27, no. 2. - P. 162-177.

18. Shutler, J. On a Large Sequence-Based Human Gait Database / J. Shut-ler, M. Grant, J. Carter // Applications and Science in Soft Computing. — 2004. - Jan.

19. Silhouette Analysis-Based Gait Recognition for Human Identification / L. Wang [et al.] // Pattern Analysis and Machine Intelligence, IEEE Transactions on. - 2004. - Jan. - Vol. 25. - P. 1505-1518.

20. The OU-ISIR Large Population Gait Database with real-life carried object and its performance evaluation / M. Z. Uddin [et al.] // IPSJ Transactions on Computer Vision and Applications. — 2018. — May. — Vol. 10, no. 1. — P. 5.

21. Nixon, M. S. Human identification based on gait / M. S. Nixon, T. Tan, R. Chellappa //. Vol. 4. — Springer Science & Business Media, 2010.

22. The effect of time on gait recognition performance / D. Matovski [et al.] // International Conference on Pattern Recognition (ICPR). Vol. 7. — 04/2012.

23. The AVA Multi-View Dataset for Gait Recognition / D. Lopez-Fernandez [et al.] // Activity Monitoring by Multiple Distributed Sensing. — Springer International Publishing, 2014. — P. 26—39. — (Lecture Notes in Computer Science).

24. Multi-view large population gait dataset and its performance evaluation for cross-view gait recognition / N. Takemura [et al.] // IPSJ Trans. on Computer Vision and Applications. — 2018. — Vol. 10, no. 4. — P. 1—14.

25. Han, J. Individual Recognition Using Gait Energy Image / J. Han,

B. Bhanu // IEEE Trans. Pattern Anal. Mach. Intell. — Washington, DC, USA, 2006. - Vol. 28, no. 2. - P. 316-322.

26. Bashir, K. Gait recognition using gait entropy image / K. Bashir, T. Xiang, G. S // Proceedings of 3rd international conference on crime detection and prevention. — 2009. — P. 1—6.

27. Chen, J. Average Gait Differential Image Based Human Recognition. / J. Chen, J. Liu // Scientific World Journal. — 2014.

28. Chrono-Gait Image: A Novel Temporal Template for Gait Recognition /

C. Wang [et al.] // Computer Vision - ECCV 2010. — Berlin, Heidelberg : Springer Berlin Heidelberg, 2010. - P. 257-270.

29. Lam, T. H. Gait flow image: A silhouette-based gait representation for human identification / T. H. Lam, K. Cheung, J. N. Liu // Pattern Recognition. — 2011. - Vol. 44, no. 4. - P. 973-987.

30. Effective Part-Based Gait Identification using Frequency-Domain Gait Entropy Features / M. Rokanujjaman [et al.] // Multimedia Tools and Applications. Vol. 74. - 11/2013.

31. Gait Recognition Using a View Transformation Model in the Frequency Domain / Y. Makihara [et al.] // Computer Vision - ECCV 2006 / ed. by A. Leonardis, H. Bischof, A. Pinz. — Berlin, Heidelberg : Springer Berlin Heidelberg, 2006. - P. 151-163.

32. Dalal, N. Histograms of Oriented Gradients for Human Detection / N. Dalal, B. Triggs // Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 1 - Volume 01. - Washington, DC, USA : IEEE Computer Society, 2005. -P. 886-893. - (CVPR '05).

33. Multiple HOG templates for gait recognition. / Y. Liu [et al.] // Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012). — 2012. - P. 2930-2933.

34. Learning Realistic Human Actions from Movies / I. Laptev [et al.] // CVPR 2008 - IEEE Conference on Computer Vision & Pattern Recognition. — Anchorage, United States : IEEE Computer Society, 06/2008. - P. 1-8.

35. Yang, Y. Gait Recognition Using Flow Histogram Energy Image / Y. Yang, D. Tu, G. Li // 2014 22nd International Conference on Pattern Recognition. - 2014. - P. 444-449.

36. Cross-view gait recognition using joint Bayesian / C. Li [et al.] // Proc. SPIE 10420, Ninth International Conference on Digital Image Processing (ICDIP 2017). - 2017.

37. Complete canonical correlation analysis with application to multi-view gait recognition / X. Xing [et al.] // Pattern Recognition. — 2016. — Vol. 50. — P. 107-117.

38. Belhumeur, P. N. Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection / P. N. Belhumeur, J. P. Hespanha, D. J. Kriegman // IEEE Trans. Pattern Anal. Mach. Intell. - 1997. - July. - Vol. 19, no. 7. - P. 711-720.

39. Bashir, K. Gait recognition without subject cooperation / K. Bashir, T. Xi-ang, S. Gong // Pattern Recognition Letters. — 2010. — Vol. 31. — P. 2052-2060.

40. Cross-view gait recognition using view-dependent discriminative analysis / A. Mansur [et al.] // IEEE International Joint Conference on Biometrics. — 2014. - P. 1-8.

41. Joint Intensity and Spatial Metric Learning for Robust Gait Recognition / Y. Makihara [et al.] // 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). - 07/2017. - P. 6786-6796.

42. View-Invariant Discriminative Projection for Multi-View Gait-Based Human Identification / M. Hu [et al.] // Information Forensics and Security, IEEE Transactions on. - 2013. - Dec. - Vol. 8. - P. 2034-2045.

43. Muramatsu, D. View Transformation Model Incorporating Quality Measures for Cross-View Gait Recognition / D. Muramatsu, Y. Makihara, Y. Yagi // IEEE transactions on cybernetics. — 2015. — July. — Vol. 46.

44. Muramatsu, D. Cross-view gait recognition by fusion of multiple transformation consistency measures / D. Muramatsu, Y. Makihara, Y. Yagi // IET Biometrics. — 2015. — June. — Vol. 4.

45. Arseev, S. Human Recognition by Appearance and Gait / S. Arseev, A. Konushin // Programming and Computer Software. — 2018. — P. 258-265.

46. Yoo, J.-H. Automated Markerless Analysis of Human Gait Motion for Recognition and Classification / J.-H. Yoo // Etri Journal. — 2011. — Apr. — Vol. 33. - P. 259-266.

47. Whytock, T. Dynamic Distance-Based Shape Features for Gait Recognition / T. Whytock, A. Belyaev, N. M. Robertson // Journal of Mathematical Imaging and Vision. - 2014. - Nov. - Vol. 50, no. 3. - P. 314-326.

48. Fusion of spatial-temporal and kinematic features for gait recognition with deterministic learning / M. Deng [et al.] // Pattern Recognition. — 2017. — Vol. 67. - P. 186-200.

49. Castro, F. Pyramidal Fisher Motion for multiview gait recognition. / F. Castro, M. Marín-Jiménez, R. Medina-Carnicer // 22nd International Conference on Pattern Recognition. — 2014. — P. 1692—1697.

50. Simonyan, K. Two-stream convolutional networks for action recognition in videos. / K. Simonyan, A. Zisserman // NIPS. - 2014. - P. 568-576.

51. Beyond short snippets: Deep networks for video classification. / J. Ng [et al.] // CVPR. - 2015.

52. Feichtenhofer, C. Convolutional two-stream network fusion for video action recognition. / C. Feichtenhofer, A. Pinz, A. Zisserman // CVPR. — 2016.

53. Varol, G. Long-term Temporal Convolutions for Action Recognition / G. Varol, I. Laptev, C. Schmid // IEEE Transactions on Pattern Analysis and Machine Intelligence. — 2017.

54. Wang, L. Action recognition with trajectory-pooled deep-convolutional descriptors. / L. Wang, Y. Qiao, X. Tang // CVPR. - 2015. - P. 4305-4314.

55. DeepGait: A Learning Deep Convolutional Representation for Gait Recognition / X. Zhang [et al.] // Biometric Recognition. — Cham : Springer International Publishing, 2017. — P. 447—456.

56. DeepGait: A Learning Deep Convolutional Representation for View-Invariant Gait Recognition Using Joint Bayesian / C. Li [et al.] // Applied Sciences. — 2017. - Feb. - Vol. 7. - P. 15.

57. VGR-Net: A View Invariant Gait Recognition Network / D. Thapar [et al.] // IEEE 4th International Conference on Identity, Security, and Behavior Analysis (ISBA). - 10/2017.

58. Wolf, T. Multi-view gait recognition using 3D convolutional neural networks / T. Wolf, M. Babaee, G. Rigoll // 2016 IEEE International Conference on Image Processing (ICIP). - 09/2016. - P. 4165-4169.

59. GEINet: View-invariant gait recognition using a convolutional neural network / K. Shiraga [et al.] // 2016 International Conference on Biometrics (ICB). - 2016. - P. 1-8.

60. A Comprehensive Study on Cross-View Gait Based Human Identification with Deep CNNs / Z. Wu [et al.] // IEEE Transactions on Pattern Analysis and Machine Intelligence. — 2016. — Mar. — Vol. 39.

61. On Input/Output Architectures for Convolutional Neural Network-Based Cross-View Gait Recognition / N. Takemura [et al.] // IEEE Transactions on Circuits and Systems for Video Technology. — 2019. — Sept. — Vol. 29, no. 9. - P. 2708-2719.

62. Siamese neural network based gait recognition for human identification / C. Zhang [et al.] // IEEE International Conference on Acoustics, Speech and Signal Processing. - 2016. - P. 2832-2836.

63. GaitSet: Regarding Gait as a Set for Cross-View Gait Recognition / H. Chao [et al.] // AAAI. - 2019.

64. Invariant feature extraction for gait recognition using only one uniform model / S. Yu [et al.] // Neurocomputing. - 2017. - Vol. 239. - P. 81-93.

65. GaitGAN: Invariant Gait Feature Extraction Using Generative Adversarial Networks / S. Yu [et al.] // 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). - 2017. - P. 532-539.

66. Learning view invariant gait features with Two-Stream GAN / Y. Wang [et al.] // Neurocomputing. - 2019. - Vol. 339. - P. 245-254.

67. Hochreiter, S. Long Short-Term Memory / S. Hochreiter, J. Schmidhuber // Neural Comput. - 1997. - Nov. - Vol. 9, no. 8. - P. 1735-1780.

68. Gait Identification by Joint Spatial-Temporal Feature / S. Tong [et al.] // Biometric Recognition. — 2017. — P. 457—465.

69. Feng, Y. Learning effective gait features using LSTM / Y. Feng, Y. Li, J. Luo // International Conference on Pattern Recognition. — 2016. — P. 325-330.

70. Memory-based Gait Recognition / D. Liu [et al.] // Proceedings of the British Machine Vision Conference (BMVC). - BMVA Press, 09/2016. -P. 82.1-82.12.

71. Pose-Based Temporal-Spatial Network (PTSN) for Gait Recognition with Carrying and Clothing Variations / R. Liao [et al.] // Biometric Recognition. — Cham : Springer International Publishing, 2017. — P. 474—483.

72. Battistone, F. TGLSTM: A time based graph deep learning approach to gait recognition / F. Battistone, A. Petrosino // Pattern Recognition Letters. — 2019. — Vol. 126. — P. 132—138. — Robustness, Security and Regulation Aspects in Current Biometric Systems.

73. Event-Based Moving Object Detection and Tracking / A. Mitrokhin [et al.] // 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). - 10/2018. - P. 1-9.

74. Spatiotemporal multiple persons tracking using Dynamic Vision Sensor / E. Pi^tkowska [et al.] // 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. — 06/2012. — P. 35—40.

75. Yuan, W. Fast localization and tracking using event sensors / W. Yuan, S. Ra-malingam // IEEE International Conference on Robotics and Automation (ICRA). - 2016. - P. 4564-4571.

76. Neuromorphic Vision Sensing for CNN-based Action Recognition / A. Chadha [et al.] // ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). - 05/2019. - P. 7968-7972.

77. Dynamic Vision Sensors for Human Activity Recognition / S. A. Baby [et al.] // 2017 4th IAPR Asian Conference on Pattern Recognition (ACPR). - 11/2017. - P. 316-321.

78. Bay, H. SURF: Speeded Up Robust Features / H. Bay, T. Tuytelaars, L. Van Gool // Computer Vision - ECCV 2006. - 2006. - P. 404-417.

79. Event-driven body motion analysis for real-time gesture recognition / B. Kohn [et al.] // 2012 IEEE International Symposium on Circuits and Systems (IS-CAS). - 05/2012. - P. 703-706.

80. A Low Power, Fully Event-Based Gesture Recognition System / A. Amir [et al.] // 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). - 07/2017. - P. 7388-7397.

81. Asynchronous frameless event-based optical flow / R. Benosman [et al.] // Neural Networks. - 2012. - Vol. 27. - P. 32-37.

82. Zhu, A. EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras / A. Zhu [et al.] // Proceedings of Robotics: Science and Systems. — Pittsburgh, Pennsylvania, 06/2018.

83. Wang, X. Intelligent multi-camera video surveillance: A review / X. Wang // Pattern Recognition Letters. — 2013. — Vol. 34, no. 1. — P. 3—19. — Extracting Semantics from Multi-Spectrum Video.

84. Large-scale video classification with convolutional neural networks. / A. Karpathy [et al.] // CVPR. - 2014.

85. Burton, A. Thinking in Perspective: Critical Essays in the Study of Thought Processes / A. Burton, J. Radford. — Methuen, 1978. — 232 p.

86. Warren, D. Electronic Spatial Sensing for the Blind: Contributions from Perception, Rehabilitation, and Computer Vision / D. Warren, E. R. Strelow. — Netherlands, 1985. - 521 p.

87. Farneback, G. Two-frame motion estimation based on polynomial expansion. / G. Farneback // Proc. of Scandinavian Conf. on Image Analysis. Vol. 2749. - 2003. - P. 363-370.

88. OpenCV (Open Source Computer Vision Library). — URL: https://opencv. org.

89. Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields / Z. Cao [et al.] // CVPR. - 2017.

90. Automatic Learning of Gait Signatures for People Identification / F. M. Castro [et al.] // Advances in Computational Intelligence. — Cham : Springer International Publishing, 2017. — P. 257—270.

91. Return of the devil in the details: Delving deep into convolutional nets. / K. Chatfield [et al.] // Proc. BMVC. - 2014.

92. Ioffe, S. Batch normalization: Accelerating deep network training by reducing internal covariate shift. / S. Ioffe, C. Szegedy // ICML. — 2015.

93. Krizhevsky, A. ImageNet Classification with Deep Convolutional Neural Networks / A. Krizhevsky, I. Sutskever, G. Hinton // Neural Information Processing Systems. — 2012. — Jan. — Vol. 25.

94. Dropout: A Simple Way to Prevent Neural Networks from Overfitting / N. Srivastava [et al.] // Journal of Machine Learning Research. — 2014. — Vol. 15. - P. 1929-1958.

95. Chen, T. Net2net: Accelerating learning via knowledge transfer. In International Conference on Learning Representation. / T. Chen, I. Goodfellow, J. Shlens // ICLR. - 2016.

96. Glorot, X. Understanding the difficulty of training deep feedforward neural networks / X. Glorot, Y. Bengio // Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. Vol. 9 / ed. by Y. W. Teh, M. Titterington. — Chia Laguna Resort, Sardinia, Italy : PMLR, 05/2010. — P. 249—256. — (Proceedings of Machine Learning Research).

97. Simonyan, K. Very deep convolutional networks for large-scale image recognition. / K. Simonyan, A. Zisserman // ICLR. — 2015.

98. Triplet probabilistic embedding for face verification and clustering / S. Sankaranarayanan [et al.] // 2016 IEEE 8th International Conference on Biometrics Theory, Applications and Systems (BTAS). — 2016. — P. 1-8.

99. Small medical encyclopedia //. Vol. 4 / ed. by V. Pokrovskiy. — 1996. — P. 570.

100. Convolutional pose machines / S.-E. Wei [et al.] // CVPR. - 2016.

101. Deep Residual Learning for Image Recognition / K. He [et al.] // 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). —

2015. - P. 770-778.

102. Zagoruyko, S. Wide Residual Networks / S. Zagoruyko, N. Komodakis // Proceedings of the British Machine Vision Conference (BMVC) / ed. by E. R. H. Richard C. Wilson, W. A. P. Smith. - BMVA Press, 09/2016.

103. Lasagne: First release. / S. Dieleman [et al.]. — 08/2015. — URL: http: //dx.doi.org/10.5281/zenodo.27878.

104. Castro, F. M. Multimodal Features Fusion for Gait, Gender and Shoes Recognition / F. M. Castro, M. J. Marín-Jiménez, N. Guil // Mach. Vision Appl. —

2016. - Nov. - Vol. 27, no. 8. - P. 1213-1228.

105. Deep multi-task learning for gait-based biometrics / M. Marín-Jiménez [et al.] // IEEE International Conference on Image Processing (ICIP). — 09/2017.

106. Guan, Y. A robust speed-invariant gait recognition system for walker and runner identification / Y. Guan, C.-T. Li. — 2013. — Jan.

107. Cross-view gait recognition using view-dependent discriminative analysis / A. Mansur [et al.] // IJCB 2014 - 2014 IEEE/IAPR International Joint Conference on Biometrics. — 12/2014.

108. PyTorch: An Imperative Style, High-Performance Deep Learning Library / A. Paszke [et al.] // Advances in Neural Information Processing Systems 32. — Curran Associates, Inc., 2019. - P. 8024-8035.

109. Lichtsteiner, P. A 128x128 120 dB 15^s Latency Asynchronous Temporal Contrast Vision Sensor / P. Lichtsteiner, C. Posch, T. Delbruck // IEEE Journal of Solid-State Circuits. - 2008. - Vol. 43, no. 2. - P. 566-576.

110. Newell, A. Stacked Hourglass Networks for Human Pose Estimation / A. Newell, K. Yang, J. Deng // ECCV. - 2016.

111. 2D Human Pose Estimation: New Benchmark and State of the Art Analysis / M. Andriluka [et al.] // CVPR. - 2014. - P. 3686-3693.

112. The event-camera dataset and simulator: Event-based data for pose estimation, visual odometry, and SLAM / E. Mueggler [et al.] // The International Journal of Robotics Research. - 2017. - Vol. 36, no. 2. - P. 142-149.