Project Outline

Satellite imaging obtained from sensors of Sentinel (a few generations of satellites) provide optical and radar data every 5 to 10 days to track changes in soil moisture [1]. Drones and other remote sensing systems provide complementary information of different granularity at different intervals. While water management, crop health and yield can be assessed by segmentation using remote sensing data, predictions become heavily affected when cloud coverage obscures fields of crops [2]. With recent advances in masked image modeling [3], this PhD project will seek to design a multi-spectral multi-modal masked auto-encoder based on transformers which can take multiple inputs, simulate cloud coverage on clean input data and force the machine learning model to recover information lost by the input coverage. Such models are known currently to provide high quality self-supervised embeddings [3] for downstream tasks such as segmentation and detections in images while benefiting form simulated noise. However, the multi-spectral data contains multiple bands, and other sources operate with different granularity and periodicity. Thus, establishing an ML statistical model which can use these complementary sources with the goal of cloud coverage removal will be the key research question. We expect publications in top-tier ML venues (CVPR, ICCV, ECCV, Neurips, ICML, ICLR, AAAI) and a possibility of domain specific publications (Remote Sensing, Nature etc.) should the model lead to practical high-level accuracy.

[1] Satellite Data and Remote Sensing for Agricultural Operations, InfoPulse 2024.
[2] Cloud Cover Assessment for Operational Crop Monitoring Systems in Tropical Areas, Remote sensing 2016
[3] Pre-training with Random Orthogonal Projection Image Modeling, ICLR 2024

The student will:
- get familiar with SOTA masked modeling models and multi-modal self supervised learning
- get familiar with multi-modal data sources in agriculture and multiple sources of remote sensing 
- get familiar with domain specific problems (water management, crop health, crop yield segmentation and prediction)
- design next generation of masked modeling in multi-spectral multi-modal setting
- become an expert on remote sensing with agricultural application
- learn PyTorch and other key deep learning packages

To register an expression of interest, click here. You will need to outline why you have selected the research project and how your skills, experience and/or knowledge meet the project requirements.