MICCAI Hackathon

Task 1 - Deploy your MICCAI method to the clinics

Deploy your MICCAI method as a "shadow mode" application in the clinics by integrating it with the Picture Archiving and Communication System (PACS). Your method will run at your radiology department on the Kaapana platform for evaluating its clinical applicability.

• Get familiar with Kaapana and apply the nnU-Net on exemplary data.
• Containerize your MICCAI method for inference such that it can be used within Kaapana.
• Write a directed acyclic graph (DAG) to integrate your container for inference into Kaapana.
• Write a directed acyclic graph (DAG) for training your model in Kaapana.
• Draft a tutorial/manual on the usage of Kaapana.
• Investigate the anonymization of DICOM images.

Task 2 - Towards clinically-oriented evaluation

MICCAI methods are typically evaluated on specific metrics for the task at hand (e.g., Dice coefficient for segmentation) rather than actual clinical endpoints. Such evaluation might not suffice to show, or even overestimate, the benefit for the clinical problem at hand, as for instance, the Dice coefficient is rarely a clinically relevant outcome. Bridging the gap also means adapting the way we report results.

• How can we show that a proposed method not only achieves SOTA with respect to the standard metric but also works as intended in a clinical setting?
• How can we set up proper double-blind randomized control trials where various unwanted human-AI factors (lacking trust, etc.) are not changing the results.
• Propose ways to make the evaluation scenario more realistic.
• Demonstrate possible limitations/shortcomings (or even threats) of evaluations typically performed for MICCAI methods.
• How would you describe your method if you would not “sell” it for publication but describe it for use in clinics, and you would be responsible for its successful application?

Task 3 - Assess brittleness of MICCAI papers in the clinical setting

Analyze how a MICCAI method behaves in a clinical-like setting with realistic out-of-distribution cases and/or clinically realistic adversarial attacks.

• Identify possible sources of error when using a MICCAI model in the clinics (e.g., different scanner types, different target populations).
• Select a problem and model you will work on (e.g., from one of the MICCAI challenges).
• Investigate the generation of clinically realistic out-of-distribution cases (e.g., by trying to mimic a new scanner type or by generating extreme cases, different image resolutions). Can you implement a library with methods for generating out-of-distribution cases?
• What are the conclusions of your experiments?

Task 4 - Identify safety threats in medical decision support systems

How can we guarantee the safety of medical decision support systems? Are such systems robust to adversarial attacks, could we deploy them to the cloud? What could be issues of federated and continuous learning scenarios?

• Identify potential vulnerabilities for adversarial attacks when a MICCAI model is deployed to the clinics (e.g., cloud PACS, online learning).
• Exploit vulnerabilities in the pipeline by simple examples (e.g., noise or renaming of files in open-source DICOM library). You could also mimic a federated learning scenario and introduce simple perturbations to show how easy it is to attack the systems.
• Report results and conclusions.

Task 5 - Increase the consistency and quality of annotations

Annotations in medical imaging are extremely difficult to get and also very challenging to keep consistent across multiple annotators. How could we increase the consistency and quality of annotations?

• Discuss and summarize the challenges of annotation policy standardization across multiple annotators in the context of images of cells.
• Discuss how to increase the annotation quality without extensive training and years of experience.
• Run an experiment and produce a proof-of-concept (on the provided cell data).
• Annotation fatigue might influence the annotation quality. Can we detect the fatigue by mouse, eye tracking (cf. fatigue detection in car drivers), or something else? Implement a simple fatigue detection system on an annotation tool (e.g., ITK-SNAP).

Task 6 - Educate along the gap

People on the other side of the gap might have different perceptions about the capabilities of MICCAI methods and how to bridge the gap. Identify and compile educational material for the people on the other side of the gap (clinicians or MICCAI researchers) to prevent misunderstandings and false expectations when translating MICCAI methods to a clinical (or clinical-like) setting.

For MICCAI researchers:

• Try to identify the needs of clinicians with respect to MICCAI methods. How would you convince the physician at your local hospital to collaborate with you?
• Compile instruction material for clinicians that explains the principles of AI methods such that it could be used during a discussion with clinicians. For instance, the material should answer questions such as: What is the general principle? What is required? What are the limitations? What are threats?
• Imagine you are responsible for a method that will be translated to the clinics and you need to write a description about it. Identify and discuss the differences between such a description and a typical MICCAI paper.

For clinicians:

• List the expectations you have in MICCAI methods (or automated/AI methods in general).
• Identify the potential of AI and automated methods in the clinics.
• Identify the gaps between MICCAI methods and the clinical needs.
• Propose approaches to bridge the gap.

Task 7 - Analyze the clinical problems addressed by MICCAI papers

Based on the data from previous MICCAI editions, extract as much information as possible on the type of clinical problems addressed and how this evolved over time.

• Collect longitudinal statistics of MICCAI papers from the proceedings.
• Analyze and illustrate the statistics. For instance try to answer questions such as: what is the variety of tasks and clinical problems addressed by MICCAI papers? Can you observe a change over time? How diverse are the contributions?
• Collect further statistics on a certain task you identified (how many sites, size of cohorts, ...).
• Discuss your findings and propose measures where necessary.

Bonus task - Do It Yourself

Do you have your own task in mind? Then do it yourself (DIY)! Simply work on your own data/problem or use one of our suggested datasets. Team up with others who might be interested in exploring a dataset and some hacking. Note that you will not have dedicated mentors for support available.

You could work on one of the datasets freely available at The Cancer Imaging Archive (TCIA). Some TCIA datasets that come with data beyond medical images, and, therefore, might be closer to a clinical setting, can be found in the resources.

Participation

• Who can participate? Everyone. The more diverse the better. So, it does not matter whether you are a clinician, a researcher, an undergraduate student, a postdoc, or a professor.
• What do I do as participant? You will work on a task you choose from a set of tasks during the hackathon weekend. Mentors will provide feedback and guidance to you in dedicated mentoring sessions. You will shortly present your contribution at the end of the MICCAI Hackathon.
• Can I participate individually? Yes, working individually on a task is possible, although you would miss the chance to meet new people.
• Can we participate as a preformed team? Yes. To do so, each member needs to fill in the same team name in the registration form.

Registration

• Why do I need to register for the hackathon? A registration (here) is required to facilitate the communication between the organizers and the participants.
• Is a hackathon registration binding? No, you can unregister until the start of the hackathon weekend on October 23.
• Until when can I register for the hackathon? Registration for the MICCAI Hackathon will be open until the hackathon weekend on the October 23 at 09:30 UTC.
• Can I update my registration? Yes, you can always change your registration such as joining a preformed team. Your registration can be updated through the link in the welcome e-mail. Keep in mind that this might affect other people (e.g., switching a team)!
• Why do I need to register for the kick-off at the satellite event day? Attending the kick-off at the satellite event on September 27 requires a registration for the satellite event day of the MICCAI conference as the MICCAI Hackathon is an official satellite event.
• Is it mandatory to attend the kick-off at the satellite event to participate? No, you can participate at the hackathon weekend without attending the kick-off at the satellite event day. All relevant material will be made publicly available shortly after the September 27.
• Does the hackaton weekend imply additional fees? No, the participation in the MICCAI Hackathon is free of charge. However, as the MICCAI Hackathon is an official satellite event at the MICCAI conference, attending the kick-off at the half-day satellite event on September 27 underlies the registration fees of the satellite event day.

Organization

• Can I work on my own task? Yes, as long as it is related to the topic of the hackathon. If you want, you can propose a task.
• Can I work on multiple task? Yes, you can combine the ideas of several tasks to form a new task.
• Can I start working on the task before the hackathon weekend? Sure, as soon as you are matched to a team (if you desire this) you can start hacking.
• How are the teams formed/matched? Participants that select to be matched to a team will be brought together according to their task preferences, i.e., a matching based on common interest in a task.
• When are the teams formed/matched? We will start forming the teams approximately one week after the kick-off at the satellite event day and end, at last, on October 23 at 09:30 UTC (for those who register late).
• When will the data be available? All relevant material, including a recording of the session, will be made publicly available shortly after the September 27.
• Do I need to choose a mentor? No, mentors get assigned to teams based on the task.

Format

• What platform is used for the hackathon? The kick-off at the satellite event on September 27 will happen on the official platform of the MICCAI conference. The platform for the virtual hackathon weekend will be Discord with Zoom for presentations.
• What does the award consist of? The award consists of Amazon vouchers sponsered by the Center for Artificial Intelligence in Medicine (CAIM), University of Bern. The first prize contribution will receive a voucher valued 500 USD. In case of eight or more contributions, a second (300 USD) and third (200 USD) place prize will also be awarded.
• Who gets awarded? Four independent judges will grade the contributions during the presentations of the contributions. The grading involves three aspects: creativity, applicability, and estimated effort; each graded from 1 (worst) to 5 (best). The best contribution(s) will win an award.
• How do the presentations look like? The presentations consist of a 5 minute recording of your outcome in which you should promote your idea/finding/solution.
• Will everything be publicly available? We encorage participants to open source their outcomes. Final presentations, keynotes, and all provided materials will be available. Mentoring sessions will not be recorded.
• The timezone is suboptimal for me, what can I do? You are free to work on your task whenever you like, but try to benefit from the interactions with your mentor and your teammates.

General

• What happens with all the contributions at the end of the hackathon? We might summarize the outcome in a white paper as in the last edition of the MICCAI Hackathon.
• Who has the rights on the outcomes? The outcome is free to use (open source) and participants as well as mentors might decide to follow up on their ideas.
• Is there a template for the presentation? The Google Slides template of the MICCAI Hackathon is available here.
• Is the MICCAI Hackathon a regularly occuring event? Yes. It's already the second edition and we aim at having it recurring. Contact us if you have ideas for the next edition or are interested to help organizing.