Применение нескольких МЭМС инерциальных измерительных блоков низкого качества в алгоритмах навигации и синхронизации времени / Navigation and Time Synchronization Algorithms Using Multiple Low Grade MEMS IMUs тема диссертации и автореферата по ВАК РФ 05.13.18, кандидат наук Файзуллин Марсель Фарисович

• Файзуллин Марсель Фарисович
• кандидат наук
• 2022

Оглавление диссертации кандидат наук Файзуллин Марсель Фарисович

Contents

Page

Acknowledgments

Introduction

Chapter 1. Best Axes Composition: Multiple IMUs Sensor Fusion

to Reduce Systematic Error

1.1 Introduction

1.2 Related Work

1.3 Background

1.3.1 State Variables and Frames of Reference

1.3.2 IMU Model

1.3.3 Kinematic Model

1.3.4 A competitor: Averaged Virtual Estimator

1.4 Best Axes Composition for Multiple IMU Sensor Fusion in Open Loop

1.4.1 Stage I. Stationary IMU Parameters Estimation

1.4.2 Stage II. Time-dependent IMU Parameters Estimation

1.4.3 Stage III. Best Axes Composition

1.5 Implementation

1.6 Experiments

1.7 Conclusions

Chapter 2. Twist-n-Sync: Software Time Synchronization with Microseconds Accuracy and Precision Using

MEMS-Gyroscopes

2.1 Introduction

2.2 Related Work

2.3 Background

2.3.1 Time synchronization problem statement

2.3.2 Rigid Body Rotations and Angular Velocities

2.3.3 MEMS Gyroscope Model

2.4 Twist-n-Sync Time Synchronization Algorithm

Page

2.4.1 Coarse Time Offset Calculation

2.4.2 Online Calibration

2.4.3 Time Offset Refinement

2.4.4 Algorithm Implementation

2.5 Empirical validation

2.5.1 Calibration Influence

2.5.2 Downsampling and Offset Influence

2.5.3 Cut-Off Frequency Influence

2.5.4 Trial Duration Influence

2.5.5 Summary

2.6 Conclusion

Chapter 3. Practical Applications of Twist-n-Sync for

Synchronization of Homogeneous and Heterogeneous

Sensor Systems

3.1 Introduction

3.2 Related work

3.3 Twist-n-Sync for Synchronized Capturing of Color Images by Two Smartphones

3.3.1 Mobile Application for Synchronized Image Capture by Twist-n-Sync

3.3.2 Evaluation Methodology and Tool

3.3.3 Results

3.4 Twist-n-Sync for Synchronized Capturing of Smartphone Camera

Color and Depth Camera Range Images

3.4.1 System Overview

3.4.2 Synchronization

3.4.3 Evaluation Methodology and Results

3.5 Conclusion

Conclusion

List of Figures

List of Tables

References

89

Введение диссертации (часть автореферата) на тему «Применение нескольких МЭМС инерциальных измерительных блоков низкого качества в алгоритмах навигации и синхронизации времени / Navigation and Time Synchronization Algorithms Using Multiple Low Grade MEMS IMUs»

Introduction

Relevance and Degree of Elaboration of the research topic. This work covers the problems of (1) navigation and (2) time synchronization, therefore the relevance and the degree of elaboration of the research topic consist of two parts below.

1. State estimation by Multiple consumer-grade MEMS IMUs. State (orientation and/or position) estimation is a basic problem in navigation. Among the tasks that continuously increase the demand for solving the navigation problem, one can list localization of a self-driving car on a road, wheeled rescue robot navigation under building debris, unmanned aerial robot navigation in the air, handheld 3D-scanner pose estimation for precise 3D-model construction of an object, and other cases. In order to perceive information about rigid body state, data from different sensors may be gathered and analyzed. The sensors may include lidars; depth, thermal, or visual cameras; radars; sonars; inertial measurement units (IMUs). An IMU placed on a moving body is a key sensor that can provide information about angular velocity (by its internal gyroscope) and acceleration (by its internal accelerometer) of this body. Integration of IMU measurements provides orientation and position estimates of the body. For state estimation, IMU data can be used solely (so-called dead-reckoning problem) or can also be fused with other types of data (e.g. visual images or navigation satellite data).

The most accurate and precise state estimation results using an IMU can be reached if the IMU, in turn, provides accurate and precise measurements. The most important characteristics that describe the accuracy and precision of the IMU measurements are noise strength and bias diffusion [61]. According to [97], it is generally accepted to use these measures among others in order to put IMU sensor performance into four categories, ranging the sensors from high- to low-performance: navigation-grade IMUs, tactical-grade IMUs, industrial-grade IMUs, and low grade or consumer-grade IMUs. While the three first categories usually propose massive and expensive sensors [97], they appear to be not available for widespread usage in mobile robotics and mass consumption. In contrast, consumer-grade IMUs are ubiquitous, cheap ($100 and less), tiny (less than several millimeters for every dimension), these properties make this category of IMUs occupy the majority of devices where they are presented and tasks being solved [12]. Consumer-grade IMUs are based on Micro-electromechanical systems (MEMS) technology that allows the de-

sign of tiny (less than 1 mm) integrated devices or systems that combine mechanical and electrical components [23].

Despite the mentioned advantages of the consumer-grade IMUs, they have the main drawback, in addition to high values of noise strength and bias diffusion, they suffer from systematic errors including non-unit scaling of measurements, axes misalignment (non-orthogonality), non-linear distortion, the dependence of measurements on temperature and other factors [69, 91, 87]. Some of the errors can be minimized by well-defined calibration techniques (non-unit scaling, axes misalignment) [69], while the other imperfections cannot be obtained by classical calibration techniques and produce a problem for accurate state estimation. Therefore, a single consumer-grade IMU sensor is not usually used for pose estimation tasks solely but applied as an aid to other types of sensors.

Nowadays, to improve state estimation performance when utilizing consumer-grade MEMS IMUs, multiple IMU (MIMU) data fusion became popular in the state estimation topic; a recent review [63] states that more than three hundred research papers were published in recent decades on MIMU data fusion techniques and applications. The classical probabilistic model of MIMU data fusion that takes into account only stochastic errors, shows noticeably better estimation performance in theory [96]. However, the classical model does not take into account the sources of systematic errors mentioned above, such that the application of the probabilistic model for the fusion of real multiple MEMS IMU data cannot reach theoretical gain in performance.

For better accuracy, systematic errors in consumer-grade MEMS IMU measurements should be considered either directly - by defining their models explicitly or indirectly - by assuming their existence and possible behaviour. In this work, we propose a multiple consumer-grade MEMS IMU data fusion algorithm that indirectly handles the systematic errors.

Thus, the first aim of this work is to propose, develop, examine, and justify a method of multiple consumer-grade MEMS IMU data fusion for state estimation that indirectly handles the systematic errors.

2. Time synchronization using Multiple consumer-grade MEMS gyroscopes. Time synchronization of data obtained from sensors in a sensor network is a fundamental component of the data gathering process that provides correct information about data temporal relation. For data fusion algorithms, misalignment of data timestamps leads to data stream mismatch and may become a source of criti-

cal errors in target applications. Some clear examples of synchronization importance may come from a wide range of areas in research and engineering, e.g., in computer vision (CV). Thanks to the great advances in microelectronics, computer science, and CV algorithms, visual data represent a great value for world economics and society in common due to the massive application of object detection, recognition, and segmentation algorithms, or robot localization and environment mapping algorithms in such areas as transportation, manufacturing, agriculture, healthcare, retail, and others. This type of data stream vitally needs time synchronization with visual or other types (e.g., inertial) of data streams when data fusion is applied: in case of fusing data from multiple cameras placed on a moving platform, unsynchronized capture of images negatively affects the projection error when the platform moves, especially during rotations [64]; adverse artifacts appear during panorama stitching by images obtained from stationary multi-view camera systems, which capture high-dynamic scenes [28]; out of synchronization can severely weaken the performance of IMU-based image stabilization of photo or video shooting systems [21]; Visual-Inertial Odometry (VIO) methods lose the ability to correctly track image keypoints if IMU measurements are not correctly synchronized with the images [89]. Analogous problems can be found in other areas and for other types of data.

There exist different solutions for time synchronization. Among them is hardware (HW) time synchronization, which is the most accurate and precise solution for time synchronization (nanoseconds order and better). HW time synchronization in its core may include dedicated electronic circuits that provide triggering signals or reference clock pulses. Considering huge diversity of modern sensors and sensor networks, it can be complicated to provide HW synchronization in every case. The impossibility of HW synchronization may be explained by geographical, technological, economic, compatibility, and other reasons. Accordingly, to overcome these issues, software (SW) synchronization methods that employ mathematical models and analysis of physical phenomena, become an alternative solution. Among them, network-based and data-driven synchronization methods stand at the forefront.

The widely applied network-based time synchronization protocols such as Network Time Protocol (NTP) and others [55, 56, 34, 33, 30] may provide several millisecond or better synchronization performance; however, the performance is highly dependent on the current network connection latency and symmetry that is not guaranteed in general [36, 67]. Moreover, network cards are not always available on sensor platforms.

If network-based synchronization is not proper or not available, data-driven synchronization may be applied. It employs the analysis of properties of data streams themselves to capture mutual temporal relation between the streams. For instance, this concept was applied to the synchronization of seismic sensors during the MesoAmerica Subduction Experiment [49] or to the synchronization of image streams from independent cameras during hockey match [84]. The accuracy and precision of such a type of synchronization is commonly have the same order as data stream sampling periods. For synchronization of image streams, this value shows ten milliseconds order [79, 11] that is unacceptably high for CV task examples mentioned in a paragraph above, where sub-millisecond synchronization order should be preserved for the robustness of working pipelines. This means that for such examples, there is need to provide data streams with higher data rates, which is not technically possible.

Consumer-grade MEMS IMUs that are widely used for solving state estimation problems, can generate high data rate measurements (1-10 kHz order) and resolve even modest dynamics of movements (10-3 rad/s of angular velocity and 5-10-3 m/s2 of linear acceleration) [58]. One can make a finding that multiple consumer-grade MEMS IMUs can also be used as a source of data for data-driven time synchronization if they experience dependent motions. In turn, the main case when motions are dependent is motions of rigidly connected IMUs, among others. Considering this hypothesis, we propose to examine the application of multiple consumer-grade MEMS IMUs in time synchronization tasks of rigidly connected but independent systems.

Thus, the second aim of this research is to propose, develop, examine, justify, and apply a data-driven time synchronization algorithm that utilizes multiple consumer-grade MEMS gyroscopes.

From now, we will refer to consumer-grade MEMS IMUs as just IMUs and use MIMU abbreviation for multiple of them.

To achieve two stated aims, it was needed to solve the following main tasks:

1. Investigate state-of-the-art (SOTA) MIMU data fusion methods to assess directions of improvement of the existing navigation algorithms.

2. Examine SOTA software-based techniques for synchronization of sensors in sensor networks.

3. Provide a navigation algorithm that utilizes data from multiple IMUs and outperforms SOTA algorithms of MIMU data fusion.

4. Provide a time synchronization algorithm that utilizes data from multiple IMUs and outperforms SOTA software-based time synchronization algorithms.

5. Provide an empirical justification of the performance of the proposed navigation algorithm:

• Provide an implementation of the algorithm.

• Develop hardware-software data gathering system for obtaining data for performance estimation of the algorithm.

• Gather MIMU data set.

• Employ the MIMU data fusion algorithm to the gathered inertial data.

• Carry out the analysis of the performance.

6. Provide an empirical justification of the performance of the proposed synchronization algorithm:

• Provide an implementation of the algorithm.

• Develop hardware-software inertial data gathering system with known and controlled time offsets for obtaining data for performance estimation of our synchronization algorithm.

• Gather data from multiple (two) IMU sensors by the system.

• Apply the synchronization algorithm to the gathered inertial data.

• Perform the analysis of the performance of the algorithm under influence of different factors.

• Perform the analysis of the robustness of the algorithm on real applications including (i) synchronized data gathering from multiple modern rigidly attached smartphones and (ii) a smartphone rigidly attached to an external depth camera.

Scientific novelty of the results obtained in the thesis:

1. An essentially new algorithm of consumer-grade MEMS MIMU data fusion is presented that outperforms classical probabilistic Multiple IMU data fusion technique due to handling the assumption that systematical errors have considerable contribution to state estimation error.

2. A new software-based time synchronization algorithm using consumer-grade MEMS gyroscope data is proposed that outperforms any known network-based and data-driven time synchronization algorithms for sensors in sensor networks.

3. A deep analysis of gyroscope data and configurations that affect the accuracy and precision of the synchronization method has been carried out in order to provide a reference for the optimal choice of parameters.

4. A novel customized evaluation methodologies of the frame synchronization accuracy and precision between (i) multiple visual rolling-shutter cameras and (ii) a visual rolling shutter camera and depth camera utilizing the rolling-shutter effect have been proposed.

Scientific and practical significance. The work covers both theoretical and practical aspects of the stated problems due to the examination of theoretical models of physical sensors and their accordance with practical tasks. Theoretical significance of the work consists of the justification of the hypotheses that (i) it is possible to handle systematic errors in multiple consumer-grade MEMS IMU measurements in order to provide improvement of the state estimation accuracy; (ii) application of consumer-grade MEMS gyroscope data is reasonable not only in navigation but also in time synchronization algorithms, and constitutes an effectively working solution. Practical significance consists of the practical application of results obtained during solving the proposed theoretical tasks. The practical significance of the time synchronization algorithm is confirmed by the application of the results obtained in the dissertation by two industrial projects with Samsung AI Center at Moscow on creating a dataset of an interior with a person in the foreground using a smart-phone, depth camera, and IMUs (2021) and a dataset of surrounding environment using multiple depth cameras and IMUs (2022, in progress) for benchmarking and versatile analysis of SOTA 3D-reconstruction, localization, and mapping algorithms. The practical significance of the navigation algorithm consists of the justified superiority over SOTA MIMU data fusion methods and the possibility of its integration into VIO and other data fusion algorithms for state estimation tasks.

Methodology includes application of the following methods of the research: theory of probability, theory of random processes, theory of optimization, theory of Lie algebra and groups (groups of rotation matrices), kinematics, correlation analysis, digital signal processing, mathematical modeling, numerical linear algebra, interpolation, approximation, hypothesis testing, empirical justification, data collection, data verification, software and hardware design.

Propositions for the defense are the following:

1. Best Axes Composition (BAC) navigation algorithm that is based on multiple consumer-grade MEMS IMU data fusion has been proposed that

outperforms a classical probabilistic multiple consumer-grade MEMS IMU data fusion technique for up to 20% in state estimation accuracy improvement.

2. Twist-n-Sync consumer-grade MEMS gyroscope-based time synchronization algorithm that provides repeatable microseconds accuracy and precision of synchronization has been proposed. It outperforms other software time synchronization methods to the best of our knowledge.

3. An analysis of factors and parameters that affect consumer-grade MEMS gyroscope-based time synchronization has been presented.

4. Twist-n-Sync performance has been justified in two practical applications.

5. Twist-n-Sync algorithm implementation have been made publicly available for society (open-source materials on the Internet) that can be applied, modified, and extended with no restrictions:

https://github.com/MobileRoboticsSkoltech/twistnsync-python, https://github.com/achains/Twist-n-Sync-CPP-Module.

Obtained propositions are in accordance with the passport of the specialty 05.13.18 - Mathematical Modeling, Numerical Methods and Software, in particular, to the points:

• p.1. Development of new mathematical methods for modeling objects and phenomena.

• p.2. Development of qualitative and approximate analytical methods for the study of mathematical models.

• p.4. Implementation of effective numerical methods and algorithms in the form of complexes of problem-oriented programs for conducting a computational experiment.

Veracity and approbation of thesis research. The reliability of the obtained results is ensured by the reports and discussions during leading conferences on robotics and computer vision: main results of the work were reported at the 10th European Conference on Mobile Robots (ECMR 2021) and accepted to and will be reported at the A* CORE rating Conference on Computer Vision and Pattern Recognition (CVPR 2022). The ECMR 2021 paper is chosen as one of the best papers at the conference and invited for paper submission to a special issue of Robotics and Autonomous Systems journal that belongs to Q1 quartile; the paper has been submitted to the journal. Also, the results were presented during semi-

nars at the Autonomous Non-profit Organization for Higher Education "Skolkovo Institute of Science and Technology".

Personal contribution. All the original results presented in this thesis were obtained by the author personally or with his direct involvement under the supervision of the scientific supervisor.

Publications. The main results of the thesis topic are described in 3 publications, all of them are indexed by Web of Science and/or Scopus databases; 2 of the publications are published in periodical scientific journals that belong to Q1 ranking quartile [II, III]; 1 another publication is a full-text conference proceeding [I].

[I] Marsel Faizullin and Gonzalo Ferrer. "Best Axes Composition: Multiple Gyroscopes IMU Sensor Fusion to Reduce Systematic Error". In: 2021 European Conference on Mobile Robots (ECMR). IEEE. 2021, pp. 1-7. DOI: 10.1109/ECMR50962.2021.9568800.

[II] Marsel Faizullin, Anastasiia Kornilova, Azat Akhmetyanov, and Gonzalo Ferrer. "Twist-n-sync: Software clock synchronization with microseconds accuracy using MEMS-gyroscopes". In: Sensors 21.1 (2021), p. 68. DOI: 10.3390/ s21010068.

[III] Marsel Faizullin, Anastasiia Kornilova, Azat Akhmetyanov, Konstantin Pakulev, Andrey Sadkov, and Gonzalo Ferrer. "SmartDepthSync: Open Source Synchronized Video Recording System of Smartphone RGB and Depth Camera Range Image Frames with Sub-millisecond Precision". In: IEEE Sensors Journal 22.7 (2022), pp. 7043-7052. DOI: 10.1109/JSEN.2022.3150973.

Personal contribution of the author in the works with co-authors is as follows:

[I] - The Best Axes Composition algorithm of multiple low-grade MEMS IMU data fusion for state estimation is proposed, developed, examined, and justified; empirical justification is provided by data from a specially developed multiple IMU handheld system.

[II] - The Twist-n-Sync time synchronization algorithm using low grade MEMS gyroscopes is proposed, developed, examined, justified and applied; empirical justification is provided by data from a specially developed multiple IMU handheld

system; deep analysis of factors that affect the performance of the algorithm is provided; application of the algorithm for synchronized photo capturing from by two smartphones is provided.

[III] - The Twist-n-Sync time synchronization algorithm is applied to a hardware-software handheld data gathering system that includes a depth camera, a smartphone camera and IMU; two levels of synchronization are provided, namely, time synchronization of the smartphone and depth camera clocks and frame synchronization of images capture.

Thesis structure and volume. The thesis consists of Introduction, 3 Chapters, and Conclusion. Chapter 1 is entirely devoted to the proposed consumer-grade MEMS IMU data fusion algorithm including theory and evaluation on real data. Chapter 2 explains theoretical aspects of the proposed time synchronization algorithm using data from consumer-grade MEMS gyroscopes and empirical justification of its performance. Chapter 3 describes the application and performance evaluation of the proposed time synchronization algorithm for synchronized hardware-software data gathering systems: (i) two smartphones and (ii) a smartphone with an external depth camera. The entire volume of the thesis occupies 99 pages, including 29 figures and 3 tables. References contain 98 items.

Список литературы диссертационного исследования кандидат наук Файзуллин Марсель Фарисович, 2022 год

References

[1] Tousif Ahmed, Mohsin Y Ahmed, Md Mahbubur Rahman, Ebrahim Nemati, Bashima Islam, Korosh Vatanparvar, Viswam Nathan, Daniel McCaffrey, Ji-long Kuang, and Jun Alex Gao. "Automated Time Synchronization of Cough Events from Multimodal Sensors in Mobile Devices". In: Proceedings of the 2020 International Conference on Multimodal Interaction. 2020, pp. 614-619.

[2] Sameer Ansari, Neal Wadhwa, Rahul Garg, and Jiawen Chen. "Wireless software synchronization of multiple distributed cameras". In: 2019 IEEE International Conference on Computational Photography (ICCP). IEEE. 2019, pp. 1-9.

[3] Jared B Bancroft and Gerard Lachapelle. "Data fusion algorithms for multiple inertial measurement units". In: Sensors 11.7 (2011), pp. 6771-6798.

[4] Timothy D Barfoot. State Estimation for Robotics. Cambridge University Press, 2017.

[5] Maik Benndorf, Marika Kaden, Eric Zuchantke, Frederic Ringsleben, Thomas Haenselmann, and Martin Gaedke. "MobileTimeSync - An Android App for Time Synchronization for Mobile Construction Assessment". In: International journal on advances in systems and measurements 11 (2018), pp. 13-21.

[6] HG Berns and RJ Wilkes. "GPS time synchronization system for K2K". In: 1999 IEEE Conference on Real-Time Computer Applications in Nuclear Particle and Plasma Physics. 11th IEEE NPSS Real Time Conference. Conference Record (Cat. No. 99EX295). IEEE. 1999, pp. 480-483.

[7] Per Brekgaard, Michael Kai Petersen, and Jakob Eg Larsen. "In the twinkling of an eye: Synchronization of EEG and eye tracking based on blink signatures". In: 2014 4th International Workshop on Cognitive Information Processing (CIP). IEEE. 2014, pp. 1-6.

[8] Derek Bradley, Bradley Atcheson, Ivo Ihrke, and Wolfgang Heidrich. "Synchronization and rolling shutter compensation for consumer video camera arrays". In: 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. IEEE. 2009, pp. 1-8.

[9] Gilles Callebaut, Geoffrey Ottoy, and Lieven De Strycker. "Bring your own Sensor: Use your Android Smartphone as a Sensing Platform". In: 2019 IEEE Sensors Applications Symposium (SAS). IEEE. 2019, pp. 1-5.

[10] Carlos Campos, Richard Elvira, Juan J Gomez Rodriguez, Jose MM Mon-tiel, and Juan D Tardos. "ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial, and Multimap SLAM". In: IEEE Transactions on Robotics (2021).

[11] Yaron Caspi, Denis Simakov, and Michal Irani. "Feature-based sequence-to-sequence matching". In: International Journal of Computer Vision 68.1 (2006), pp. 53-64.

[12] Haiyang Chao, Calvin Coopmans, Long Di, and YangQuan Chen. "A comparative evaluation of low-cost IMUs for unmanned autonomous systems". In: 2010 IEEE Conference on Multisensor Fusion and Integration. IEEE. 2010, pp. 211-216.

[13] Santiago Cortes, Arno Solin, Esa Rahtu, and Juho Kannala. "ADVIO: An authentic dataset for visual-inertial odometry". In: Proceedings of the European Conference on Computer Vision (ECCV). 2018, pp. 419-434.

[14] John L Crassidis. "Sigma-point Kalman filtering for integrated GPS and iner-tial navigation". In: IEEE Transactions on Aerospace and Electronic Systems

42.2 (2006), pp. 750-756.

[15] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. "Scannet: Richly-annotated 3d reconstructions of indoor scenes". In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2017, pp. 5828-5839.

[16] Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Christian Theobalt. "Bundlefusion: Real-time globally consistent 3d reconstruction using on-the-fly surface reintegration". In: ACM Transactions on Graphics (ToG) 36.4 (2017), p. 1.

[17] Carl De Boor. A practical guide to splines. Vol. 27. Springer-Verlag, 1978.

[18] Olaf Deppe, Georg Dorner, Stefan König, Tim Martin, Sven Voigt, and Steffen Zimmermann. "MEMS and FOG technologies for tactical and navigation grade inertial sensors—Recent improvements and comparison". In: Sensors

17.3 (2017), p. 567.

[19] Kevin Eckenhoff, Patrick Geneva, and Guoquan Huang. "Sensor-failure-resilient multi-imu visual-inertial navigation". In: International Conference on Robotics and Automation (ICRA). IEEE. 2019, pp. 3542-3548.

[20] Ezzaldeen Edwan, Stefan Knedlik, and Otmar Loffeld. "Constrained angular motion estimation in a gyro-free IMU". In: IEEE Transactions on Aerospace and Electronic Systems 47.1 (2011), pp. 596-610.

[21] Andrew English, Patrick Ross, David Ball, Ben Upcroft, and Peter Corke. "TriggerSync: A time synchronisation tool". In: 2015 IEEE International Conference on Robotics and Automation (ICRA). IEEE. 2015, pp. 6220-6226.

[22] Marsel Faizullin, Anastasiia Kornilova, and Gonzalo Ferrer. Open-Source Li-DAR Time Synchronization System by Mimicking GPS-clock. 2021. arXiv: 2107.02625 [cs.RO].

[23] Andreas C Fischer, Fredrik Forsberg, Martin Lapisa, Simon J Bleiker, Göran Stemme, Niclas Roxhed, and Frank Niklaus. "Integrating mems and ics". In: Microsystems & Nanoengineering 1.1 (2015), pp. 1-16.

[24] Martin A Fischler and Robert C Bolles. "Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography". In: Communications of the ACM 24.6 (1981), pp. 381-395.

[25] Christian Forster, Luca Carlone, Frank Dellaert, and Davide Scaramuzza. "IMU preintegration on manifold for efficient visual-inertial maximum-a-poste-riori estimation". In: Robotics: Science and Systems (RSS). Georgia Institute of Technology. 2015.

[26] Christian Forster, Luca Carlone, Frank Dellaert, and Davide Scaramuzza. Supplementary material to: IMU preintegration on manifold for efficient visu-al-inertial maximum-a-posteriori estimation. Tech. rep. Georgia Institute of Technology, 2015.

[27] Paul Furgale, Timothy D Barfoot, and Gabe Sibley. "Continuous-time batch estimation using temporal basis functions". In: IEEE International Conference on Robotics and Automation. IEEE. 2012, pp. 2088-2095.

[28] Vamsidhar R Gaddam, Ragnar Langseth, Hakon K Stensland, Carsten Gri-wodz, Michael Riegler, Tomas Kupka, Havard Espeland, Dag Johansen, Havard D Johansen, and Pal Halvorsen. "Camera Synchronization for Panoramic Videos". In: MediaSync. Springer, 2018, pp. 565-592.

[29] Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow. "Unsupervised monocular depth estimation with left-right consistency". In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2017, pp. 270279.

[30] Jana van Greunen and Jan Rabaey. "Lightweight Time Synchronization for Sensor Networks". In: Proceedings of the 2nd ACM International Conference on Wireless Sensor Networks and Applications. WSNA '03. San Diego, CA, USA: Association for Computing Machinery, 2003, pp. 11-19. ISBN: 1581137648. DOI: 10.1145/941350.941353. URL: https://doi.org/10. 1145/941350.941353.

[31] Marco Grossi. "A sensor-centric survey on the development of smartphone measurement and sensing systems". In: Measurement 135 (2019), pp. 572592.

[32] Junjie Hu, Mete Ozay, Yan Zhang, and Takayuki Okatani. "Revisiting single image depth estimation: Toward higher resolution maps with accurate object boundaries". In: 2019 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE. 2019, pp. 1043-1051.

[33] Hyojung Lee, Wonpil Yu, and Youngmi Kwon. "Efficient RBS in Sensor Networks". In: Third International Conference on Information Technology: New Generations (ITNG'06). 2006, pp. 279-284. DOI: 10.1109/ITNG.2006.61.

[34] "IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems". In: IEEE Std 1588-2008 (Revision of IEEE Std 1588-2002) (2008), pp. 1-300. DOI: 10 . 1109 / IEEESTD . 2008 . 4579760.

[35] Andrey Ignatov, Grigory Malivenko, David Plowman, Samarth Shukla, and Radu Timofte. "Fast and Accurate Single-Image Depth Estimation on Mobile Devices, Mobile AI 2021 Challenge: Report". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021, pp. 25452557.

[36] Wang Jingchao, Zhao Ruohan, and Gu Weiwen. "Time synchronization in networks: A survey". In: Proceedings of the 2nd International Conference on Control and Computer Vision. 2019, pp. 121-126.

[37] Steven G Johnson and Matteo Frigo. "A modified split-radix FFT with fewer arithmetic operations". In: IEEE Transactions on Signal Processing 55.1 (2006), pp. 111-119.

[38] R Kim, T Nagayama, Hongki Jo, and BF Spencer Jr. "Preliminary study of low-cost GPS receivers for time synchronization of wireless sensors". In: Sensors and Smart Structures Technologies for Civil, Mechanical, and Aerospace Systems 2012. Vol. 8345. International Society for Optics and Photonics. 2012, 83451A.

[39] Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. "Tanks and temples: Benchmarking large-scale scene reconstruction". In: ACM Transactions on Graphics (ToG) 36.4 (2017), pp. 1-13.

[40] Qingkai Kong, Asaf Inbal, Richard M Allen, Qin Lv, and Arno Puder. "Machine learning aspects of the MyShake global smartphone seismic network". In: Seismological Research Letters 90.2A (2019), pp. 546-552.

[41] Anastasiia Kornilova, Marsel Faizullin, Konstantin Pakulev, Andrey Sad-kov, Denis Kukushkin, Azat Akhmetyanov, Timur Akhtyamov, Hekmat Taherinejad, and Gonzalo Ferrer. "SmartPortraits: Depth Powered Handheld Smartphone Dataset of Human Portraits for State Estimation, Reconstruction and Synthesis". In: arXiv, 2022. DOI: 10 . 48550/ARXIV. 2204 . 10211. URL: https://arxiv.org/abs/2204.10211.

[42] Mathieu Labbe and François Michaud. "RTAB-Map as an open-source lidar and visual simultaneous localization and mapping library for large-scale and long-term online operation". In: Journal of Field Robotics 36.2 (2019), pp. 416446.

[43] Patrick Lazik, Niranjini Rajagopal, Bruno Sinopoli, and Anthony Rowe. "Ultrasonic time synchronization and ranging on smartphones". In: 21st IEEE Real-Time and Embedded Technology and Applications Symposium. IEEE. 2015, pp. 108-118.

[44] Wlodzimierz Lewandowski, Jacques Azoubib, and William J Klepczynski. "GPS: Primary tool for time transfer". In: Proceedings of the IEEE 87.1 (1999), pp. 163-172.

[45] Mingyang Li and Anastasios I Mourikis. "High-precision, consistent EK-F-based visual-inertial odometry". In: The International Journal of Robotics Research 32.6 (2013), pp. 690-711.

[46] JoonHoo Lim, Won Jae Yoo, La Woo Kim, You Dam Lee, Hyung Keun Lee, et al. "Augmentation of GNSS by low-cost MEMS IMU, OBD-II, and digital altimeter for improved positioning in urban area". In: Sensors 18.11 (2018), p. 3830.

[47] Chaojun Liu, Shengzhi Zhang, Shuai Yu, Xuebing Yuan, and Sheng Liu. "Design and analysis of gyro-free inertial measurement units with different configurations". In: Sensors and Actuators A: Physical 214 (2014), pp. 175186.

[48] Shaoshan Liu, Bo Yu, Yahui Liu, Kunai Zhang, Yisong Qiao, Thomas Yuang Li, Jie Tang, and Yuhao Zhu. "The Matter of Time-A General and Efficient System for Precise Sensor Synchronization in Robotic Computing". In: arXiv preprint arXiv:2103.16045 (2021).

[49] Martin Lukac, Paul Davis, Robert Clayton, and Deborah Estrin. "Recovering temporal integrity with data driven time synchronization". In: 2009 International Conference on Information Processing in Sensor Networks. IEEE. 2009, pp. 61-72.

[50] Kevin M Lynch and Frank C Park. Modern Robotics. Cambridge University Press, 2017.

[51] Kevin M.. Lynch and Frank Chongwoo Park. Modern Robotics: Mechanics, Planning, and Control. Cambridge University Press, 2017.

[52] Hicham Marouani and Michel R Dagenais. "Internal clock drift estimation in computer clusters". In: Journal of Computer Systems, Networks, and Communications 2008 (2008).

[53] Microsoft. Azure Kinect DK, 2015. URL: https://azure.microsoft.com/ en-us/services/kinect-dk/.

[54] D. Mills. Network Time Protocol (Version 1). Specification and Implementation. RFC 1059. RFC Editor, July 1988. URL: https://tools.ietf.org/ html/rfc1059.

[55] David Mills. Network Time Protocol Version 4'- Protocol and Algorithms Specification. RFC 5905. RFC Editor, June 2010. URL: https://www.rfc-editor. org/info/rfc5905.

[56] David Mills. Simple Network Time Protocol (SNTP) Version 4 for IPv4, IPv6 and OSI. RFC 4330. RFC Editor, Jan. 2006. URL: https://www.rfc-editor. org/info/rfc4330.

[57] Aurelien Monot, Nicolas Navet, and Bernard Bavoux. "Impact of clock drifts on CAN frame response time distributions". In: ETFA2011. IEEE. 2011, pp. 14.

[58] MPU-9150 Product Specification. Revision 4.3. InvenSense Inc., Sept. 2013.

[59] Raul Mur-Artal and Juan D. Tardos. "ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras". In: IEEE Transactions on Robotics 33.5 (2017), pp. 1255-1262. DOI: 10.1109/TRÛ.2017.2705103.

[60] Vinayak Narasimhan, Holden Li, and M Jianmin. "Micromachined high-g ac-celerometers: a review". In: Journal of Micromechanics and Microengineering 25.3 (2015), p. 033001.

[61] Janosch Nikolic. "Characterisation, Calibration, and Design of Visual-Inertial Sensor Systems for Robot Navigation". PhD thesis. ETH Zurich, 2016.

[62] Janosch Nikolic, Joern Rehder, Michael Burri, Pascal Gohl, Stefan Leuteneg-ger, Paul T Furgale, and Roland Siegwart. "A synchronized visual-inertial sensor system with FPGA pre-processing for accurate real-time SLAM". In: 2014 IEEE international conference on robotics and automation (ICRA). IEEE. 2014, pp. 431-437.

[63] John-Olof Nilsson and Isaac Skog. "Inertial sensor arrays — A literature review". In: 2016 European Navigation Conference (ENC). IEEE. 2016, pp. 110.

[64] Edwin Olson. "A passive solution to the sensor synchronization problem". In: 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE. 2010, pp. 1059-1064.

[65] Edwin Olson. "AprilTag: A robust and flexible visual fiducial system". In: IEEE International Conference on Robotics and Automation. 2011, pp. 34003407.

[66] OpenCamera Sensors - an Android application for synchronized recording of video and IMU data. Accessed Jul. 13, 2021. Mobile Robotics Lab, Skoltech. URL: https : / / github . com / MobileRoboticsSkoltech / OpenCamera -Sensors.

[67] Linh-An Phan, Taejoon Kim, Taehong Kim, JaeSeang Lee, and Jae-Hyun Ham. "Performance analysis of time synchronization protocols in wireless sensor networks". In: Sensors 19.13 (2019), p. 3020.

[68] Michael James David Powell et al. Approximation theory and methods. Cambridge university press, 1981, p. 37.

[69] Joern Rehder, Janosch Nikolic, Thomas Schneider, Timo Hinzmann, and Roland Siegwart. "Extending kalibr: Calibrating the extrinsics of multiple IMUs and of individual axes". In: 2016 IEEE International Conference on Robotics and Automation (ICRA). IEEE. 2016, pp. 4304-4311.

[70] Matthew Rhudy, Jason Gross, Yu Gu, and Marcello Napolitano. "Fusion of GPS and redundant IMU data for attitude estimation". In: AIAA Guidance, Navigation, and Control Conference. 2012, p. 5030.

[71] Kaylin Rochford, Jennifer A Strauss, Qingkai Kong, and Richard M Allen. "MyShake: Using human-centered design methods to promote engagement in a smartphone-based global seismic network". In: Frontiers in Earth Science (2018), p. 237.

[72] Sandeep Singh Sandha, Joseph Noor, Fatima M Anwar, and Mani Srivastava. "Exploiting smartphone peripherals for precise time synchronization". In: 2019 IEEE International Symposium on Precision Clock Synchronization for Measurement, Control, and Communication (ISPCS). IEEE. 2019, pp. 1-6.

[73] Thomas Schmid, Zainul Charbiwala, Jonathan Friedman, Young H Cho, and Mani B Srivastava. "Exploiting manufacturing variations for compensating environment-induced clock drift in time synchronization". In: ACM SIGMET-RICS Performance Evaluation Review 36.1 (2008), pp. 97-108.

[74] Thomas Schops, Torsten Sattler, and Marc Pollefeys. "Bad slam: Bundle adjusted direct rgb-d slam". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019, pp. 134-144.

[75] D. Schubert, T. Goll, N. Demmel, V. Usenko, J. Stueckler, and D. Cremers. "The TUM VI Benchmark for Evaluating Visual-Inertial Odometry". In: International Conference on Intelligent Robots and Systems (IROS). Oct. 2018.

[76] Tixiao Shan, Brendan Englot, Drew Meyers, Wei Wang, Carlo Ratti, and Daniela Rus. "Lio-sam: Tightly-coupled lidar inertial odometry via smoothing and mapping". In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE. 2020, pp. 5135-5142.

[77] Claude Elwood Shannon. "Communication in the presence of noise". In: Proceedings of the IRE 37.1 (1949), pp. 10-21.

[78] Weizhao Shao, Srinivasan Vijayarangan, Cong Li, and George Kantor. "Stereo visual inertial lidar simultaneous localization and mapping". In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE. 2019, pp. 370-377.

[79] Prarthana Shrestha, Hans Weda, Mauro Barbieri, and Dragan Sekulovski. "Synchronization of multiple video recordings based on still camera flashes". In: Proceedings of the 14th ACM international conference on Multimedia. 2006, pp. 137-140.

[80] Prarthana Shrstha, Mauro Barbieri, and Hans Weda. "Synchronization of multi-camera video recordings based on audio". In: Proceedings of the 15th ACM international conference on Multimedia. 2007, pp. 545-548.

[81] Isaac Skog and Peter Handel. "Calibration of a MEMS inertial measurement unit". In: XVIIIMEKO world congress. 2006, pp. 1-6.

[82] Isaac Skog, John-Olof Nilsson, and Peter Handel. "An open-source multi inertial measurement unit (MIMU) platform". In: International Symposium on Inertial Sensors and Systems (ISISS). IEEE. 2014, pp. 1-4.

[83] Isaac Skog, John-Olof Nilsson, Peter Handel, and Arye Nehorai. "Inertial sensor arrays, maximum likelihood, and cramer-rao bound". In: IEEE Transactions on Signal Processing 64.16 (2016), pp. 4218-4227.

[84] Matej Smid and Jiri Matas. "Rolling shutter camera synchronization with sub-millisecond accuracy". In: International Conference on Computer Vision Theory and Applications. 2017, p. 8.

[85] Joan Sola, Jeremie Deray, and Dinesh Atchuthan. "A micro Lie theory for state estimation in robotics". In: arXiv preprint arXiv:1812.01537 (2018).

[86] Sanjib Sur, Teng Wei, and Xinyu Zhang. "Autodirective audio capturing through a synchronized smartphone array". In: Proceedings of the 12th annual international conference on Mobile systems, applications, and services. 2014, pp. 28-41.

[87] David Titterton, John L Weston, and John Weston. Strapdown inertial navigation technology. Vol. 17. IET, 2004.

[88] Charles Toth, Sung Woong Shin, Dorota A Grejner-Brzezinska, and Jay Hyoun Kwon. "On accurate time synchronization of multi-sensor mobile mapping systems". In: Journal of Applied Geodesy 2.3 (2008), pp. 159-166.

[89] Florian Tschopp, Michael Riner, Marius Fehr, Lukas Bernreiter, Fadri Furrer, Tonci Novkovic, Andreas Pfrunder, Cesar Cadena, Roland Siegwart, and Juan Nieto. "VersaVIS—An open versatile multi-camera visual-inertial sensor suite". In: Sensors 20.5 (2020), p. 1439.

[90] J. Wang and E. Olson. "High-performance inertial measurements using a redundant array of inexpensive gyroscopes (RAIG)". In: IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI). IEEE. 2015, pp. 71-76.

[91] Zihui Wang, Xianghong Cheng, and Jinbo Fu. "Optimized Multi-Position Calibration Method with Nonlinear Scale Factor for Inertial Measurement Units". In: Sensors 19.16 (2019), p. 3568.

[92] Junyuan Xie, Ross Girshick, and Ali Farhadi. "Deep3d: Fully automatic 2d-to-3d video conversion with deep convolutional neural networks". In: European conference on computer vision. Springer. 2016, pp. 842-857.

[93] Liang Xue, Cheng-Yu Jiang, Hong-Long Chang, Yong Yang, Wei Qin, and Wei-Zheng Yuan. "A novel Kalman filter for combining outputs of MEMS gyroscope array". In: Measurement 45.4 (2012), pp. 745-754.

[94] Haoyang Ye, Yuying Chen, and Ming Liu. "Tightly coupled 3d lidar inertial odometry and mapping". In: 2019 International Conference on Robotics and Automation (ICRA). IEEE. 2019, pp. 3144-3150.

[95] Ji Zhang and Sanjiv Singh. "Visual-lidar odometry and mapping: Low-drift, robust, and fast". In: 2015 IEEE International Conference on Robotics and Automation (ICRA). IEEE. 2015, pp. 2174-2181.

[96] Ming Zhang, Yiming Chen, Xiangyu Xu, and Mingyang Li. "A Lightweight and Accurate Localization Algorithm Using Multiple Inertial Measurement Units". In: arXiv preprint arXiv:1909.04869 (2019).

[97] Lin Zhou, Eric Fischer, Can Tunca, Clemens Markus Brahms, Cem Ersoy, Urs Granacher, and Bert Arnrich. "How we found our imu: Guidelines to IMU selection and a comparison of seven IMUs for pervasive healthcare applications". In: Sensors 20.15 (2020), p. 4090.

[98] Michael Zollhofer, Patrick Stotko, Andreas Gorlitz, Christian Theobalt, Matthias NieBner, Reinhard Klein, and Andreas Kolb. "State of the Art on 3D Reconstruction with RGB-D Cameras". In: Computer graphics forum. Vol. 37. 2. Wiley Online Library. 2018, pp. 625-652.