Learning Hypernetwork-Based LoRA Composition in Low-Rank Subspace for Personalized Image Generation

Low-Rank Adaptation (LoRA) has become one of the most widely used techniques for personalizing text-to-image diffusion models. While LoRAs can effectively represent different subjects or artistic styles, combining them into a single model remains challenging, since both subject identity and artistic style must be preserved without introducing semantic interference. Existing methods either rely on pair-specific optimization or learn merging strategies in the output space of the LoRA updates. This work proposes an alternative approach that performs subject-style LoRA composition directly in the LoRA rank space. A lightweight hypernetwork is trained to predict merging coefficients for the rank components of two LoRAs and generalize across different subject-style combinations. The proposed method operates on top of Stable Diffusion XL without modifying the pretrained diffusion model. The framework is evaluated on a benchmark of personalized subject and style LoRAs using both conventional CLIP-DINO metrics and a MLLM-based evaluation protocol. Different rank-space representations, training configurations, and architectural variants are also investigated to better understand the impact of different design choices. Experimental results show that learning LoRA composition directly in the rank space is an effective and competitive alternative to existing merging methods. Both quantitative and qualitative evaluations demonstrate that the generated compositions successfully preserve subject identity while transferring the desired artistic style in most cases, confirming that the LoRA rank space provides a suitable representation for learning automatic LoRA composition.