# SBI2021 Day 3
(for Day 1, see https://notes.desy.de/FUwV3KniQUiQrRWdH9sb1w?view)
(for Day 2, see https://notes.desy.de/O4OwuQMIQ0CJDDeXEn013w?view)
Please use this note pad to outline questions, add notes auxiliary to the material the instructors deliver or share other resources that you might useful.
## House Keeping
The workshop website documents that schedule: https://events.hifis.net/event/167/timetable/#20210920
(Note that times might shift depending on the delivery by the instructors.)
## To get a jupyter notebook
login to https://jupyter-jsc.fz-juelich.de/hub/login
- note you must register (again) to get access to the jupyter notebook
- in the notebook spawn website:
- version: JupyterLab
- System: JUWELS
- project: hai_sbi2021
- partition: booster
- reservation: haicore_workshop_day2
- runtime (in min): 4*60 = 240 min
- if you need to download something from the web, please spawn a notebook in partition "LoginNode" (the booster nodes do not have access to the WWW)
# Notes, Questions
**Note that we will store this note pad as markdown file and share it publicly after the event!**
## Jan-Matthis Lückmann: Benchmarking SBI
- paper this talk is based on: https://arxiv.org/abs/2101.04653
- what is c2st?
- see https://github.com/mackelab/sbi/blob/fb587e970031c5fba23d0d3ff436f169b1431797/sbi/utils/metrics.py#L15
- you consider 2 bags of samples (of any dimensionality):
- ground truth (theta)
- predicted (theta) from SBI
- consider these as labels for each sample
- you then ask the question: were all of these samples generated from the same PDF (because if that is the case, SBI did its job very well)
- in the hold-out set: if the classifier is presented a sample, it should emit a soft-max probability of 0.5 if ground truth and predicted data are the same (because the classifyer is incapable to predice which-is-which)
- in the hold-out set: if the classifier predicts `0.9-1.` for a given sample, the classifier was able differentiate and hence SBI was unable to predict samples correctly
- 2 papers on this (references taken from Jan-Matthis paper):
- Friedman 2004 http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.411.6104&rep=rep1&type=pdf
- Lopaz-Paz 2017 https://arxiv.org/abs/1610.06545
## Jan Bölts: The Bayesian workflow
- notebooks:
- https://github.com/mlcolab/sbi-workshop/blob/main/slides/3_2_troubleshooting_sbi_in_practice.ipynb
- https://github.com/mlcolab/sbi-workshop/blob/main/slides/3_3_bayesian_workflow.ipynb
- https://jmlr.csail.mit.edu/papers/v17/15-017.html
- Are there any paper or reference for this leakage issue and corrections with MCMC?
- The leakage issue is discussed in the SNPE-C publication in the Appendix [1] and [2] touches upon it.
- [1] https://arxiv.org/pdf/1905.07488.pdf
- [2] https://arxiv.org/pdf/2002.03712.pdf
- on flows:
- MAF = Masked Autoregressive Flow https://arxiv.org/abs/1705.07057
- NSF = Neural Spline Flow https://arxiv.org/abs/1906.04032
- review: https://arxiv.org/abs/1912.02762
- bayesian workflow: https://arxiv.org/abs/2011.01808 (Gelman et al)
- model comparison discussed in the SNRE paper: http://proceedings.mlr.press/v119/hermans20a/hermans20a.pdf
- simulation based calibration:
- The appendix of https://arxiv.org/abs/1804.06788 has the proof
- https://arxiv.org/abs/2103.10522 is also useful for the intuition (and some nice visualizations)
- another intuition:
![](https://s3.desy.de/hackmd/uploads/upload_1d1a1aae1cad7ed7cd86d928c86779db.png)
## Jan Bölts: Bayesian model comparison
https://github.com/mlcolab/sbi-workshop/blob/main/slides/3_4_bayesian-model-comparison-in-sbi.pdf
# Day 3 Feedback
## Share something that you liked about today or something that you learned :+1:
- Really helpful having instructors available to help talk through and clarify various concepts. +1
- There are a lot of information covered today, good for looking up later even if I can't fully understand everything. +1
- I very much liked the troubleshooting and bayesian workflow part.
- Really appreciated the flexibility and availability o
## Share something that you didn't like about today or something that you want us to improve :-1:
- I had a difficult day, the topics were more advanced. I need to digest the materials. I think the practical parts was not clear and strightforward to follow.
- More tutorials/practical sessions would be helpful. For example, I think some of the morning sessions could be a bit more practical, while the others could be more like a lecture. The tutorials/practices earlier in the day really helped me catching up with things, as I haven't heard many of the concepts before.
- Refers also to yesterday. I had to add plt.show() to many of the cells