With an ever-growing number of research papers, writing high quality reviews becomes an increasingly time consuming and tedious job: there is just too much new material. And with junior researchers often skipping the related and previous work diligence, the wheel is quite often reinvented. So what if we could disrupt the reviewing process so that instead of writing in-depth reviews and meta-reviewers balancing those subjective views, LLM-based reviewers write reviews with published system prompts and on a defined base of existing work (think: arxiv). Consequently, reviewing would become meta-reviewing. This would ideally result in better reviews, and at the same time educate particularly junior reviewers.

This thesis focuses on a first case study, comparing systematic vs. agentic RAG-based scientific paper review. We will focus on an established venue (ISCA Interspeech) which has been publicly recorded for many years (isca-archive). A first set of research questions could be:

  • How should existing work be indexed?
  • Do citation networks benefit or hurt review quality?
  • Do existing open source LLMs exhibit bias in the way they generate the reviews?
  • Is there a benefit of an agentic workflow over a systematic one?

If interested, please email Korbinian Riedhammer