A wide range of encoders have been developed for 3D scene understanding, but it remains unclear how their representations transfer to different tasks. Lexicon3D addresses this gap by introducing a unified probing framework to compare 3D encoders on a variety of different 3D scene understanding tasks. However, Lexicon3D only evaluates a small portion of the encoders available today, leaving newer encoder architectures and encoder families unexplored. This thesis extends the Lexicon3D framework by comparing four additional encoders on 3D semantic segmentation and 3D visual grounding: DINOv3(2D to 3D lifted), Point Transformer v3 with Point Prompt Training (3D native), Concerto (3D Self-supervised) and Uniscene3D(CLIP-based). These models were evaluated on ScanNet for 3D semantic segmentation and ScanRefer for 3D visual grounding using the same metrics as Lexicon3D. This thesis also analyzes the learned representations by using PCA visualizations in addition to linear pairwise CKA. The results show that there is no single encoder that is best on both tasks with the relative performance of the encoders changing depending on the task. The best overall encoder is PTv3+PPT placing second on both tasks. In contrast, Uniscene3D and Concerto each excel at one task (Uniscene3D in 3D visual grounding and Concerto in 3D semantic segmentation) but perform poorly on the other. The representational analysis further indicates that models that perform well on semantic segmentation tend to learn more consistent features while models that perform well on visual grounding are more likely to preserve more fine-grained details. Overall this thesis demonstrates the importance of considering both the representation learned by the encoder and the requirements of the specific task when selecting a 3D encoder.

Probing diverse 3D encoder representations for scene understanding

FERRABEE, FRANCES ELIZABETH REDICAN
2025/2026

Abstract

A wide range of encoders have been developed for 3D scene understanding, but it remains unclear how their representations transfer to different tasks. Lexicon3D addresses this gap by introducing a unified probing framework to compare 3D encoders on a variety of different 3D scene understanding tasks. However, Lexicon3D only evaluates a small portion of the encoders available today, leaving newer encoder architectures and encoder families unexplored. This thesis extends the Lexicon3D framework by comparing four additional encoders on 3D semantic segmentation and 3D visual grounding: DINOv3(2D to 3D lifted), Point Transformer v3 with Point Prompt Training (3D native), Concerto (3D Self-supervised) and Uniscene3D(CLIP-based). These models were evaluated on ScanNet for 3D semantic segmentation and ScanRefer for 3D visual grounding using the same metrics as Lexicon3D. This thesis also analyzes the learned representations by using PCA visualizations in addition to linear pairwise CKA. The results show that there is no single encoder that is best on both tasks with the relative performance of the encoders changing depending on the task. The best overall encoder is PTv3+PPT placing second on both tasks. In contrast, Uniscene3D and Concerto each excel at one task (Uniscene3D in 3D visual grounding and Concerto in 3D semantic segmentation) but perform poorly on the other. The representational analysis further indicates that models that perform well on semantic segmentation tend to learn more consistent features while models that perform well on visual grounding are more likely to preserve more fine-grained details. Overall this thesis demonstrates the importance of considering both the representation learned by the encoder and the requirements of the specific task when selecting a 3D encoder.
2025
Probing diverse 3D encoder representations for scene understanding
segmentation
grounding
3D
File in questo prodotto:
File Dimensione Formato  
MSc_Thesis_Frances_Ferrabee (2).pdf

accesso aperto

Dimensione 6.94 MB
Formato Adobe PDF
6.94 MB Adobe PDF Visualizza/Apri

The text of this website © Università degli studi di Padova. Full Text are published under a non-exclusive license. Metadata are under a CC0 License

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/20.500.12608/115836