Augmented Behavioral Annotation Tools, with Application to Multimodal Datasets and Models: A Systematic Review

Watson, Eleanor, Viana, Thiago ORCID: 0000-0001-9380-4611 and Zhang, Shujun ORCID: 0000-0001-5699-2676 (2023) Augmented Behavioral Annotation Tools, with Application to Multimodal Datasets and Models: A Systematic Review. AI, 4 (1). pp. 128-171. doi:10.3390/ai4010007

Text (© 2023 by the authors)
ai-04-00007-v2.pdf - Published Version
Available under License Creative Commons Attribution 4.0.

Download (4MB) | Preview


Annotation tools are an essential component in the creation of datasets for machine learning purposes. Annotation tools have evolved greatly since the turn of the century, and now commonly include collaborative features to divide labor efficiently, as well as automation employed to amplify human efforts. Recent developments in machine learning models, such as Transformers, allow for training upon very large and sophisticated multimodal datasets and enable generalization across domains of knowledge. These models also herald an increasing emphasis on prompt engineering to provide qualitative fine-tuning upon the model itself, adding a novel emerging layer of direct machine learning annotation. These capabilities enable machine intelligence to recognize, predict, and emulate human behavior with much greater accuracy and nuance, a noted shortfall of which have contributed to algorithmic injustice in previous techniques. However, the scale and complexity of training data required for multimodal models presents engineering challenges. Best practices for conducting annotation for large multimodal models in the most safe and ethical, yet efficient, manner have not been established. This paper presents a systematic literature review of crowd and machine learning augmented behavioral annotation methods to distill practices that may have value in multimodal implementations, cross-correlated across disciplines. Research questions were defined to provide an overview of the evolution of augmented behavioral annotation tools in the past, in relation to the present state of the art. (Contains five figures and four tables).

Item Type: Article
Article Type: Article
Additional Information: This article belongs to the Special Issue Feature Papers for AI
Uncontrolled Keywords: Machine learning; Annotation; Behavior; Foundation models
Subjects: Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Divisions: Schools and Research Institutes > School of Business, Computing and Social Sciences
Research Priority Areas: Applied Business & Technology
Depositing User: Thiago Viana
Date Deposited: 06 Feb 2023 11:51
Last Modified: 31 Oct 2023 12:43

University Staff: Request a correction | Repository Editors: Update this record

University Of Gloucestershire

Bookmark and Share

Find Us On Social Media:

Social Media Icons Facebook Twitter Google+ YouTube Pinterest Linkedin

Other University Web Sites

University of Gloucestershire, The Park, Cheltenham, Gloucestershire, GL50 2RH. Telephone +44 (0)844 8010001.