4 PhD positions on Foundation Models for Oncology

Foundation Models hold significant potential in the diagnosis and treatment of cancer. At our new fomo.fo (FOundation MOdels For Oncology) lab, we intend to be at the forefront of these advancements. This lab is a collaborative effort between the Netherlands Cancer Institute (NKI), the University of Amsterdam (UvA), and Kaiko.ai. We are looking for researchers to join us in developing this transformative technology and, using AI, change how we treat and diagnose cancer. In particular, we are looking for PhD candidates on the following projects:

Project 1: 3D Foundation Models
This candidate will train 3D Foundation Models by (i) adopting 3D data-aware tokenisation; (ii) inserting relative 3D positional encodings; and (iii) applying memory‑efficient attention so 3D patches of scans fit on GPUs. To train a well-performing Foundation Model, they curate a 100k – 1M CT scan multi‑centre dataset with automatic quality checks. They will run hyperparameter sweeps and ablation studies to find the optimal Foundation Model architecture, and benchmark these on clinically relevant downstream tasks. The PhD will build on already collected data to convert prior projects to downstream tasks, on which the Foundation Model can be validated.

Project 2: Multimodal Models for Radiology
This project will learn to combine vision (like 3D Foundation Model embeddings) and language (LLM with radiology reports) to create grounded findings in the CT (textual description + localization/segmentation). The model will produce a structured draft in Dutch or English while highlighting every referenced lesion in the imaging scan. The research will then extend to longitudinal tracking through a temporal decoder architecture.

Project 3: Multimodal Fusion of Unimodal Embeddings
The focus of this PhD is on developing time-aware fusion strategies combining foundation model embeddings from multiple oncological modalities - including, but not limited to, radiology, histopathology, and genomics - into a unified patient-state representation. The aim is to capture complementary information across domains while preserving the strengths of each unimodal encoder.

Project 4: Test-Time Generative Oncology Decoders
This PhD will explore test-time scaling of oncology native decoders, that is, having the ability to operate adaptively with any subset of input modalities at inference. The work will focus on generative applications where shared multimodal embeddings - produced by the other PhDs or state-of-the-art public Foundation Models - are provided to an instruction-tuned large language model (LLM). The LLM will generate structured oncology reports that integrate available modalities including but not limited to histopathology and radiology reports.

What are you going to do?

• Develop new foundation model methods within the context of one of the four research projects;

• Perform novel academic research on the crossroads of deep machine learning and medical physics. The research will be published in the top conferences, including ICLR, NeurIPS, ICML, and CVPR;

• Actively collaborate with other researchers within the lab, NKI, UvA, and Kaiko.ai;

• Develop high-quality code for the foundational models in a fast-paced, highly-skilled team;

• Regularly report and present internally on your progress;

• Regularly present intermediate research results at international conferences and workshops and publish them in proceedings and journals;

• Assist in relevant teaching activities;

• Complete and defend a PhD thesis within the official appointment of four years.

• A Master’s degree in Artificial Intelligence, Computer Science, or a related field;

• A strong background in machine learning and computer vision;

• Strong analytical skills and technical skills;

• Excellent programming skills, preferably in Python; experience with full-stack dev ops is a bonus;

• Excellent mathematics foundations, especially statistics and probability theory, calculus and linear algebra;

• You are highly motivated and creative, with a proactive and independent mindset;

• An interdisciplinary mindset and an open and proactive personality in interacting with researchers from different disciplines;

• Strong communication, presentation, and writing skills and excellent command of English;

• Prior publications in relevant vision and machine learning venues will benefit your application.

Your development opportunities and employment conditions

The basis for your employment conditions is in accordance with the CLA Hospitals. You will receive from us:

• A temporary contract for a period of four years, for 36 hours per week;

• A gross monthly salary between €3.665,- and €4.450,-, based on a 36 hours working week, in line with the OIO scale and depending on your experience;

• 144 holiday hours and 57 hours of Personal Life Budget, with full-time employment;

• Annual holiday pay of 8,33% and a fixed end-of-year bonus of 8,33%;

• A travel allowance of €0,23 per km;

• Free parking at the Netherlands Cancer Institute. And with our bicycle plan and discount on public transport we also make it attractive for you to leave the car at home;

• A great opportunity in a specialized hospital where you can also continue to learn and grow yourself of you wish: the AVL Academy offers innovative and inspiring education in the field of oncology professional knowledge, skills and personal development;

• Opportunity to exercise during working hours at our Fit Boutiqs;

• An active staff association.

Interested?

Please send your application via the application (‘apply’) button at the top of the page; applications sent directly by e-mail will not be processed. Incomplete applications will be desk rejected without feedback.

We want you to submit separately your curriculum vitae, including your list of publications if applicable, and 1 single PDF file with all the documents listed below:

• A motivation letter that explains why you are interested in the PhD vacancy and joining the team. State clearly at the beginning of the motivation letter your preferred projects (max 3);

• A separate research statement where you explain your preferences on the project and your initial ideas, e.g., how you would propose to start the research. We are not looking at this stage for working solutions; creativity is appreciated. Do not discuss more than three projects;

• A link to your Master’s thesis;

• A link to your GitHub account;

• A complete record of Bachelor and Master courses (including grades and explanation of the grading system);

• A list of projects and publications you have worked on (with brief descriptions of your contributions, max 2 pages);

• The names and contact addresses of at least two academic references (please do not include any recommendation letters).

Would you like to know more about this vacancy or our organization? Please contact dr. Jonas Teuwen, j.teuwen@nki.nl (NKI, group leader AI for oncology) or prof. dr. Cees Snoek, c.g.m.snoek@uva.nl  (UvA, head of the Video & Image Sense lab, at the Faculty of Science).

This vacancy is open for applications until October 19th.

About us
You will conduct AI research in one of the four projects mentioned above as part of the Fomo.fo lab at both the Netherlands Cancer Institute and the University of Amsterdam (UvA). The University of Amsterdam is the Netherlands' largest university, offering the widest range of academic programmes. At the UvA, 42,000 students, 6,000 staff members and 3,000 PhD candidates study and work in a diverse range of fields, connected by a culture of curiosity. The position is with Prof. dr. Cees Snoek, Professor, head of the Video & Image Sense lab (VIS lab), at the Faculty of Science of the University of Amsterdam. VIS lab is a world-leading lab on Computer Vision and Machine Learning, and has over 30 PhD students, postdoctoral researchers and faculty members working on a broad variety of deep learning, computer vision, and foundation model subjects. The position is also embedded in the ELLIS Network of Excellence in AI. The principal investigators in the Netherlands Cancer Institute lab are Prof. dr Lodewyk Wessels (director digital oncology program), dr. Jonas Teuwen (group leader AI for oncology) and dr. Kevin Groot Lipman (clinical implementation lead AI). There will be a close collaboration with the researchers and developers at Kaiko.ai.