Meta-Transformer is an extreme multi-modal Transformer that utilizes the same frozen ViT backbone to encode 12 modalities: images, natural language, point cloud, audio spectrogram, video, infrared, hyperspectral, X-Ray, IMU, tabular, graph, and time-series data. It has three main components: a unified data tokenizer, a modality-shared encoder, and task-specific heads for downstream tasks, Meta-Transformer is the first framework to perform unified learning across 12 modalities with unpaired data. Its overall performance is not as impressive as state of the art models. But it is a single model for understanding several modalities together. Also, it has very small number of trainable parameters. Lastly, it establishes SOTA on some tasks like 3D Point Cloud Understanding.

In this video, I will talk about the following:
00:00:00 What is Meta-Transformer?
00:03:32 3 components in Meta-Transformer
00:04:56 How is Data-to-Sequence Tokenization done in Meta-Transformer?
00:09:25 How does the encoding in Meta Transformers work?
00:12:19 How does Meta Transformer perform?

For more details, please look at [ Ссылка ] and [ Ссылка ]

Zhang, Yiyuan, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xiangyu Yue. "Meta-Transformer: A Unified Framework for Multimodal Learning." arXiv preprint arXiv:2307.10802 (2023).

свернуть