Multimodal Document Representation Learning

Learning representations (embeddings) of documents from their constituent elements, such as images, text, and even the layout of the document i.e., the positions of various document components such as text, paragraphs, tables, etc. These embeddings can be used to search for similar documents, which in turn can be used for document classification based on similarity to existing entries in the database, which is something useful for routing documents to different processes.

Concrete implementation suggestions

1. Use CLIP [1] with an image encoder and a text encoder, aligning the representations of both. With two distinct encoders sharing an embedding space, it would in principle be possible to search for documents using text and also using images of other documents.
2. Follow approach 1 but use a text and layout encoder instead of text alone, or use 3 encoders: one for text, one for layout, and one for images. Using the layout, it would be possible to also search by elements at certain positions in the document.
3. Instead of using CLIP, simply use the encoder of a multimodal model already pre-trained on documents (UDOP [2], LayoutLMv3 [3], etc.) with the various modalities described above and train the embeddings to bring similar documents closer together. This approach has the added difficulty of needing to aggregate similar documents and define what makes a document similar, making it not 100% self-supervised.

Cooperation with external entity: Fidelidade S.A.