ELDER: Enhancing Lifelong Model Editing with Mixture-of-LoRA

Jiaang Li, Quan Wang, Zhongnan Wang, Yongdong Zhang, Zhendong Mao

Code

• Current Lifelong Model Editing approaches manage sequential edits through discrete data-adapter mappings. They assign a unique adapter for each new edit via key-value mapping, making them sensitive to minor data changes, causing inconsistent outputs and poor generalization.
• In contrast, our method maps each edit to a mixed combination of preset adapters, ensuring minor data changes don’t entirely alter the adapter parameters, enhancing generalization. Experiments on Llama2-7B and GPT2-XL have proven the effectiveness of our method.
• Additionally, our method is scalable by sharing preset adapters across increasing edits, while previous discrete methods require a separate adapter for each edit data.

Random Entity Quantization for Parameter-Efficient Compositional Knowledge Graph Representation

Jiaang Li, Quan Wang, Yi Liu, Licheng Zhang, Zhendong Mao

Code Slides

• Knowledge Graph (KG) Embedding represents entities with independent vectors and faces the scalability challenge. Recent studies propose compositional KG representation for parameter efficiency. We abstract the existing works, and define the process of obtaining corresponding codewords for each entity as entity quantization, for which previous works have designed complicated strategies.
• Reveal that simple random entity quantization can achieve similar results to current strategies. Further analyses reveal that entity codes have higher entropy at the code level and Jaccard distance at the codeword level under random entity quantization, which make different entities more distinguishable.
• We prove that the random quantization strategy is already good enough for KG representation. Future work could focusing on improving entity distinguishability from other aspects.

Inductive Relation Prediction from Relational Paths and Context with Hierarchical Transformers

Jiaang Li, Quan Wang, Zhendong Mao

Code Slides Video

• Propose a method that performs inductive reasoning on Knowledge Graphs and can naturally generalize to the fully-inductive setting, where KGs for training and inference have no common entities.
• Capture both connections between entities and the intrinsic nature of each single entity, by simultaneously aggregating relational paths and context. Use the aggregated upper-level semantics for reasoning to improve model prediction performance.
• Design a Hierarchical Transformer model for path-neighborhood encoding and information aggregation. Our proposed model performs consistently better than all baselines on almost all the eight version subsets of two fully-inductive datasets. Moreover. we design a method to interpret our method by providing each element’s contribution to the prediction results.