FB 6 Mathematik/Informatik/Physik

Institut für Mathematik


Navigation und Suche der Universität Osnabrück


Hauptinhalt

Topinformationen

Grounding LLMs into the physical world

8.3600

Dozenten

Beschreibung

In this block course, we explore the intriguing debate about whether Large Language Models (LLMS) , known for learning through next-word prediction, can develop an understanding of the physical world. The course highlights the recent emergence of multimodal LLMs, like GPT-4, which learn from both text and visual inputs, and examines the extent to which this multimodality might enhance language grounding and physical understanding.

A core part of the course involves analyzing the traditional research approach in this domain, which focuses on LLMs performing next-word prediction tasks. We critically assess whether this method truly tests a model's understanding of physical concepts,. The course also proposes a novel evaluation framework for assessing 'genuine' physical understanding in LLMs. This framework involves using realistic physical simulators as proxies for the real world, requiring LLMs to solve tasks through sensorimotor interaction with these simulators. This approach aims to test the models' ability to bridge natural language with simulated environments and their grasp of intuitive physics, offering a more comprehensive evaluation of their cognitive abilities.

Learning objectives:

- Understand the current state and potential of LLMs in simulating human-like cognition.
- Evaluate the effectiveness of multimodal learning in LLMs for physical understanding.
- Critically analyze traditional next-word prediction tasks in assessing physical comprehension.

Weitere Angaben

Ort: nicht angegeben
Zeiten: Termine am Montag, 16.09.2024 12:00 - 18:00, Dienstag, 17.09.2024 09:00 - 12:00, Dienstag, 17.09.2024 13:00 - 16:00, Mittwoch, 18.09.2024 09:00 - 12:00, Mittwoch, 18.09.2024 13:00 - 16:00, Donnerstag, 19.09.2024 09:00 - 12:00, Donnerstag, 19.09.2024 13:00 - 16:00, Freitag, 20.09.2024 09:00 - 12:00, Freitag, 20.09.2024 13:00 - 16:00, Montag, 23.09.2024 12:00 - 18:00, Dienstag, 24.09.2024 09:00 - 12:00, Dienstag, 24.09.2024 13:00 - 16:00, Mittwoch, 25.09.2024 09:00 - 12:00, Mittwoch, 25.09.2024 13:00 - 16:00, Donnerstag, 26.09.2024 09:00 - 12:00, Donnerstag, 26.09.2024 13:00 - 16:00, Freitag, 27.09.2024 09:00 - 12:00, Freitag, 27.09.2024 13:00 - 16:00
Erster Termin: Montag, 16.09.2024 12:00 - 18:00
Veranstaltungsart: Blockseminar (Offizielle Lehrveranstaltungen)

Studienbereiche

  • Cognitive Science > Bachelor-Programm
  • Cognitive Science > Master-Programm
  • Human Sciences (e.g. Cognitive Science, Psychology)

Past and Forthcoming Events

Publications

  • Asymptotics of a time-bounded cylinder model, with N. Aschenbruck and S. Bussmann, Probability in the Engineering and Informational Sciences, https://doi.org/10.1017/S0269964822000420
  • The method of cumulants for the normal approximation, with S. Jansen and K. Schubert, Probability Surveys 2022, Vol. 19, 185-270, https://doi.org/10.1214/22-PS7
  • Sedentary Random Waypoint, with C. Betken, arXiv:2009.02941
  • The Impact of Bit Errors on Intra-Session Network Coding with Heterogeneous Packet Lengths, with B. Schütz, N. Aschenbruck, S. Bussmann and M. Juhnke-Kubitzke, Proc. of the 45th IEEE LCN Symposium on Emerging Topics in Networking LCN, virtually hosted in Sydney, Australia, Nov. 16–19, 2020.
  • Stationarity for the Small World in Motion Mobility Model, with Nils Aschenbruck, Christian Heiden und Matthias Schwamborn, MSWIM '19: Proceedings of the 22nd International ACM Conference on Modeling, Analysis and Simulation of Wireless and Mobile Systems, Nov 25-29, 2019, https://doi.org/10.1145/3345768.3355935
  • Crossing Numbers and Stress of Random Graphs, with Markus Chimani and Matthias Reitzner, In Proceedings 26th International Symposium, GD 2018, Barcelona, Spain, 255--268, 2018 available here and for an extended journal version here: https://arxiv.org/abs/1808.07558
  • Fluctuations in a general preferential attachment model via Stein's method, with Carina Betken and Marcel Ortgiese, Random Structures & algorithms, vol.55, no.4, 2019 available here
  • Connection times in large ad-hoc mobile networks, Bernoulli, vol.22, no.4, 2143--2176, 2016 available here
    with Gabriel Faraud, Wolfgang König
  • The random disc thrower problem, Proceedings of the 90th European Study Group Mathematics with Industry, 59-78, 2013  available here with T. van der Aalst, D. Denteneer, M. Hong Duong, R. J. Kang, M. Keane, J. Kool, I. Kryven, T. Meyfroyt, T. Müller, G. Regts, J. Tomczyk
  • Edge fluctuations of eigenvalues of Wigner matrices, High Dimensional Probability VI: the Banff volume, Progress in Probability, vol.66, 261-275, Springer, Basel, 2013 available here
    with Peter Eichelsbacher
  • Moderate deviations for the determinant of Wigner matrices, Dedicated to Friedrich Götze on the occasion of his sixtieth birthday, Limit Theorems in Probability, Statistics and Number Theory, Springer Proceedings in Mathematics & Statistics, vol.42, 253-275, 2013, available here
    with Peter Eichelsbacher
  • Moderate deviations for the eigenvalue counting function of Wigner matrices, ALEA, Lat. Am. J. Probab. Math. Stat. 10 (1), 27-44, 2013, available here
    with Peter Eichelsbacher
  • Moderate deviations via cumulants, Journal of Theor. Probability, 2012, available here
    with Peter Eichelsbacher
  • Moments of recurrence times for Markov chains, Electronic Comm. Probab., 16(28), 296-303, 2011, available here
    with Frank Aurzada, Marcel Ortgiese, Michael Scheutzow
  • Moderate deviations in a random graph and for the spectrum of Bernoulli random matrices, Electronic Journal of Probability, Vol. 14, Paper no. 92, 2636-2656, 2009, available here
    with Peter Eichelsbacher
  • Perpendicular transport of charged particles in slab turbulence: recovery of diffusion for realistic wave-spectra?, Journal of Physics G: Nuclear and Particle Physics, 35, 025202, 2008
    with Andreas Shalchi
  • Velocity correlation functions of charged test particles, Journal of Physics G: Nuclear and Particle Physics, 34, 859, 2007
    with Andreas Shalchi