Post-doctoral research visit f - m vision language models towards interpretable deep learning for earth observation h/f

A propos d'Inria

Inria est l'institut national de recherche dédié aux sciences et technologies du numérique. Il emploie 2600 personnes. Ses 215 équipes-projets agiles, en général communes avec des partenaires académiques, impliquent plus de 3900 scientifiques pour relever les défis du numérique, souvent à l'interface d'autres disciplines. L'institut fait appel à de nombreux talents dans plus d'une quarantaine de métiers différents. 900 personnels d'appui à la recherche et à l'innovation contribuent à faire émerger et grandir des projets scientifiques ou entrepreneuriaux qui impactent le monde. Inria travaille avec de nombreuses entreprises et a accompagné la création de plus de 200 start-up. L'institut s'eorce ainsi de répondre aux enjeux de la transformation numérique de la science, de la société et de l'économie. Post-Doctoral Research Visit F/M Vision language models towards interpretable deep learning for Earth observation
Le descriptif de l'offre ci-dessous est en Anglais
Type de contrat : CDD

Niveau de diplôme exigé : Thèse ou équivalent

Fonction : Post-Doctorant

Niveau d'expérience souhaité : De 3 à 5 ans

A propos du centre ou de la direction fonctionnelle

The Inria centre at Université Côte d'Azur includes 42 research teams and 9 support services. The centre's staff (about 500 people) is made up of scientists of dierent nationalities, engineers, technicians and administrative staff. The teams are mainly located on the university campuses of Sophia Antipolis and Nice as well as Montpellier, in close collaboration with research and higher education laboratories and establishments (Université Côte d'Azur, CNRS, INRAE, INSERM), but also with the regiona economic players.

With a presence in the fields of computational neuroscience and biology, data science and modeling, software engineering and certification, as well as collaborative robotics, the Inria Centre at Université Côte d'Azur is a major player in terms of scientific excellence through its results and collaborations at both European and international levels.

Contexte et atouts du poste

We are seeking a highly motivated postdoctoral researcher to work on the interpretability and explainability of Vision-Language Models (VLMs) for remote sensing applications, with a specific focus on Sentinel-2 data. This research aims to enhance our understanding of how VLMs process satellite imagery and textual descriptions to improve geospatial analysis and decision-making.

IT will build on recent work () developed in the EVERGREEN team on aligning remote sensing and textual representations.

Mission confiée

Cross-Modal Alignment of Ground-Level and Satellite Imagery :
- Develop or leverage existing remote sensing-based VLMs that align ground-level images with satellite imagery, integrating time-series and multi-spectral data for a more comprehensive geospatial understanding.
- Improve the model's capacity to interpret textual prompts by incorporating ground-level perspectives, strengthening the connection between aerial and terrestrial observations.

Interpretability and Explainability of VLMs :
- Employ interpretability and explainability techniques to analyze the textual component of the model, identifying how prompts influence predictions and determining the most influential features.
- Examine how different textual descriptions of locations impact the model's decision-making process when comparing ground-level image alignments VLMs with other remote sensing VLMs trained on captioned satellite imagery and text pairs.
- Develop visualization and analytical methods to explore the interaction between textual and visual data within the model.
- Investigate attention mechanisms, activation maps, and latent space representations to enhance model transparency.
- Implement feature attribution techniques, such as SHAP and Grad-CAM, to provide insights into model decision-making.
- Design explainability-driven refinements to improve model interpretability and trustworthiness for geospatial applications.

Evaluation and Application to Earth Observation Tasks :
- Assess model performance on key remote sensing applications, including land cover-land use classification and map generation based on textual prompts with reasoning.
- Develop strategies to mitigate biases and improve the robustness of VLMs for Earth observation.
- Collaborate with interdisciplinary teams to apply findings to real-world geospatial challenges.

Principales activités

The main objective of this postdoctoral research is to develop methodologies for improving the transparency and interpretability of VLMs applied to remote sensing. This includes :
- Investigating the latent representations of Sentinel-2 imagery within VLM architectures.
- Identifying biases and limitations in how VLMs process textual and visual geospatial data.
- Evaluating model trustworthiness for Earth observation tasks through user studies.
- Writing a journal publication sharing the findings from the project.

Compétences
- Python programming.
- Deep Learning with Python (preferably with Pytorch).
- Experience with GIS.
- Experience with NLP would BE a plus.

Avantages
- Subsidized meals
- Partial reimbursement of public transport costs
- Leave : 7 weeks of annual leave + 10 extra days off due to RTT (statutory reduction in working hours) + possibility of exceptional leave (sick children, moving home, etc.)
- Possibility of teleworking and flexible organization of working hours
- Social, cultural and sports events and activities
- Contribution to mutual insurance (subject to conditions)

Rémunération

2788 € per month