Data61 Scholarship Agreement Zeyu Zhang

The Unification of Structured Multi-Modal Knowledge Base and Multi-Modal Foundation Models The progress of large multi-modal models has opened unprecedented opportunities to revolutionize many fields of science. However, these black-box models have been criticized for lacking factual knowledge that users often experience hallucinations. On the other hand, structured knowledge bases (KBs, including knowledge graphs and databases) not only offer facts but also can provide interpretability which is critical for scientific domains. However, KBs are difficult to construct to handle dynamic nature of real-world facts and complex relations. Furthermore, most of the existing knowledge graphs are represented with pure symbols denoted in the form of text, which weakens the capability of machines to describe and understand the real world.