004.6

В статье рассматриваются текущие тенденции и перспективные направления в изучении коммуникации, примеры исследований, основанных на мультимодальных данных, реализованные решения для сбора, хранения и обработки мультимодальной информации. Авторы делают заключение о том, какие ключевые функции должна выполнять платформа для исследований, основанных на мультимодальных данных и предлагают интерфейсы для работы с такими данными и архитектуру репозитория, которая позволит автоматизировать большинство задач по управлению данными в исследовательской работе. Платформа предназначена для совместной обработки данных в междисциплинарных коллективных исследованиях, в том числе с использованием удаленных рабочих мест. Реализация данной платформы будет иметь важное значение для исследовательских групп, преподавателей и предпринимателей широкого круга дисциплин и сфер деятельности. Удобный интерфейс для работы с мультимодальными данными и интеграция инструментов автоматической обработки в совокупности с преимуществами совместной работы над проектом и простого управления доступом к данным позволят исследователям существенно упростить рутинные манипуляции с данными и сосредоточиться на научных задачах. Архитектура платформы построена на современных решениях, является открытой, масштабируемой и устойчивой.

The paper discusses current trends and promising areas in the study of communication. Examples of studies based on multimodal data are provided as well as implemented solutions for the collecting, storage and processing of multimodal information. The authors conclude what key functions a research platform based on multimodal data should perform, and offer interfaces for working with such data and a repository architecture that will automate most data management tasks in research work. The platform is designed for joint data processing in interdisciplinary collective research, including remote workstations. The implementation of this platform will be important for research groups, lecturers in a wide range of disciplines and businessmen. The platform makes it possible to download data manually or configure the collection from applications. Datasets through metadata are associated with experiments, raw data collections, markup files, researcher profiles, and data processing tools. Using annotation, classification, information, types of data used or tools used, researchers can find experiments in their area of interest, get acquainted with the research plan and, if necessary, reproduce the results on their own data or enter into a dialogue with the authors of the experiments. A user-friendly interface for handling with multimodal data and the integration of automatic processing tools, together with the advantages of teamwork on a project and simple access control to data, will allow simplifying significantly routine data manipulations and focusing on scientific tasks. The architecture of the platform is built on modern solutions; it is open, scalable and sustainable.

заместитель руководителя, ООО «ЮНИМЕД»

кандидат филологических наук, маркетолог, ОЧУ ДПО «Московская высшая школа инжиниринга»

кандидат технических наук, доцент кафедры ПиИТ, ФКН, ФБГОУ ВПО Воронежский государственный университет

Halliday M.A.K. Language as social semiotic: The social interpretation of language and meaning. Hodder Arnold. 1978.

Flewitt R., Price S., Korkiakangas T. Multimodality: Methodological explorations. 2019.

Басов О.О. Модели и метод синтеза полимодальных инфокоммуникационных систем: дис. док. техн. наук. Орел. 2016. 292 с.

Karpov A.A., Yusupov R.M. Multimodal interfaces of human-computer interaction // Herald of the Russian Academy of Sciences. 2018. V. 88. No. 1. Pp. 67-74.

Lahat D., Adali T., Jutten C. Multimodal data fusion: an overview of methods, challenges, and prospects // Proceedings of the IEEE. 2015. V. 103. No. 9. С. 1449-1477.

Bhandari D., Paul S., Narayan A. Multimodal Data Fusion and Prediction of Emotional Dimensions Using Deep Neural Network // Computational Intelligence: Theories, Applications and Future Directions-Volume II. Springer, Singapore. 2019. Pp. 215-228.

Zheng W.L. et al. Emotionmeter: A multimodal framework for recognizing human emotions // IEEE transactions on cybernetics. 2018. V. 49. No. 3. Pp. 1110-1122.

Page Fortin M., Chaib-draa B. Multimodal Multitask Emotion Recognition using Images, Texts and Tags // Proceedings of the ACM Workshop on Crossmodal Learning and Application. ACM. 2019. Pp. 3-10.

Jansen M.P. Communicative Signals and Social Contextual Factors in Multimodal Affect Recognition // 2019 International Conference on Multimodal Interaction. ACM. 2019. Pp. 468-472.

Casanovas A.L. et al. Blind audiovisual source separation based on sparse redundant representations // IEEE Transactions on Multimedia. 2010. V. 12. No. 5. Pp. 358-371.

Kendon A. Gesture: Visible action as utterance. Cambridge University Press. 2004.

Loehr D. Aspects of rhythm in gesture and speech // Gesture. 2007. V. 7. No. 2. Pp. 179-214.

Esposito A., Esposito A.M. On speech and gestures synchrony // Analysis of Verbal and Nonverbal Communication and Enactment. The Processing Issues. Springer, Berlin, Heidelberg. 2011. Pp. 252-272.

KraMer N. et al. Smile and the world will smile with you. The effects of a virtual agent's smile on users' evaluation and behavior //International Journal of Human-Computer Studies. 2013. V. 71. No. 3. Pp. 335-349.

Zhu Y. et al. Automated depression diagnosis based on deep networks to encode facial appearance and dynamics // IEEE Transactions on Affective Computing. 2017. V. 9. No. 4. Pp. 578-584.

Desmet B., Hoste V. R. Emotion detection in suicide notes // Expert Systems with Applications. 2013. V. 40. No. 16. Pp. 6351-6358.

Priyanka C., Gupta D. Identifying the best feature combination for sentiment analysis of customer reviews // 2013 International Conference on Advances in Computing, Communications and Informatics (ICACCI). IEEE. 2013. Pp. 102-108.

Bharti S.K. et al. Sarcastic sentiment detection in tweets streamed in real time: a big data approach // Digital Communications and Networks. 2016. V. 2. No. 3. Pp. 108-121.

Gitari N.D. et al. A lexicon-based approach for hate speech detection // International Journal of Multimedia and Ubiquitous Engineering. 2015. V. 10. No. 4. Pp. 215-230.

Mihalcea R., Strapparava C. The lie detector: Explorations in the automatic recognition of deceptive language // Proceedings of the ACL-IJCNLP 2009 Conference Short Papers. Association for Computational Linguistics. 2009. Pp. 309-312.

Liao C.Y., Chen R.C., Tai S.K. Emotion stress detection using EEG signal and deep learning technologies // 2018 IEEE International Conference on Applied System Invention (ICASI). IEEE. 2018. Pp. 90-93.

Costa A. et al. Emotions detection on an ambient intelligent system using wearable devices // Future Generation Computer Systems. 2019. V. 92. Pp. 479-489.

Cabestrero R. et al. Some insights into the impact of affective information when delivering feedback to students // Behaviour & Information Technology. 2018. V. 37. No. 12. Pp. 1252-1263.

Shoumy N.J. et al. Multimodal big data affective analytics: A comprehensive survey using text, audio, visual and physiological signals // Journal of Network and Computer Applications. 2019. Pp. 102-447.

Liu H. et al. Deep learning-based multimodal control interface for human-robot collaboration // Procedia CIRP. 2018. V. 72. Pp. 3-8.

Rau P.L.P., Zheng J. Modality Capacity and Appropriateness in Multimodal Display of Complex Non-Semitic Information Stream // International Journal of Human-Computer Studies. 2019.

Mannheimer S. et al. Qualitative data sharing: Data repositories and academic libraries as key partners in addressing challenges //American Behavioral Scientist. 2019. V. 63. No. 5. Pp. 643-664.

Laessig M., Jacob B., AbouZahr C. Opening Data for Global Health // The Palgrave Handbook of Global Health Data Methods for Policy and Practice. Palgrave Macmillan. London. 2019. Pp. 451-468.

DataCite [электронный ресурс]. URL: www.re3data.org (дата обращения: 28.11.2019).

DuBois J.M., Strait M., Walsh H. Is it time to share qualitative research data? // Qualitative Psychology. 2018. V. 5. No. 3. Pp. 380.

Perazzo J. et al. Creation of Data Repositories to Advance Nursing Science // Western journal of nursing research. 2019. V. 41. No. 1. Pp. 78-95.

The health research data repository. University of Alberta. (2019). [электронныйресурс] URL: https://www.ualberta.ca/nursing/research/supports-and-services/hrdrhttps://www.ualberta.ca/nursing/research/supports-and-services/hrdr (дата обращения: 28.11.2019).

Wilkinson M.D. et al. The FAIR Guiding Principles for scientific data management and stewardship // Scientific data. 2016. V. 3.

Kamaruddin N., Wahab A. Interlaboratorydata fusion repository system (InDFuRS) for tocotrienols-based treatment // Indones. J. Electr. Eng. Comput. Sci. 2019. V. 13. No. 3. Pp. 1130-1135.

Guishard M.A. Now's not the time! Qualitative data repositories on tricky ground: Comment on Dubois et al.(2018). 2018.

Jones K. et al. Qualitative data sharing and re-use for socio-environmental systems research: A synthesis of opportunities, challenges, resources and approaches. 2018.

McCurdy S.A., Ross M.W. Qualitative data are not just quantitative data with text but data with context: On the dangers of sharing some qualitative data: Comment on Dubois et al.(2018). 2018.

Checklist for a data management plan. v.4.0. Digital Curation Centre. (2013). Retrieved from http://www.dcc.ac.uk/resources/data-management-planshttp://www.dcc.ac.uk/resources/data-management-plans (дата обращения: 28.11.2019).

Michener W.K. Ten simple rules for creating a good data management plan // PLoS computational biology. 2015. V. 11. No. 10. Pp. e1004525.

DMP tool: Build your data management plan. University of California Curation Center. (2019). [электронный ресурс] URL: https://dmptool.org/plans/new.

Williams M., Bagwell J., Zozus M.N. Data management plans: the missing perspective // Journal of biomedical informatics. 2017. V. 71. Pp. 130-142.

Encode Genome Browser and the National Center for Biotechnology Information. [электронный ресурс] URL: www.ncbi.nlm.nih.gov (дата обращения: 28.11.2019).

Catlin A.C. et al. Fully integrating data with compute workflows: a platform to better serve scientific research // Gateways: the 13th gateway computing environments conference. 2018.

Parashar M. et al. The Virtual Data Collaboratory // Computing in Science & Engineering. 2019.

Antonio M.G. et al. Qualitative Data Management and Analysis within a Data Repository // Western journal of nursing research. 2019. Pp. 0193945919881706.