Background Reading

Probabilistic programming covers a broad collection of topics in programming languages, probability, and artificial intelligence. Here are some general resources (mostly textbooks) for filling in background in these areas. Most of these are books on my bookshelf.

There are a few textbooks on probabilistic programming that I recommend:

• Barthe, G., Katoen, J.-P., & Silva, A. (2020). Foundations of probabilistic programming. Cambridge University Press.
  • This book is open-access and available here
  • It is very programming-languages oriented – it has nice chapters on semantics, program logics for couplings, etc.
• van de Meent, J.-W., Paige, B., Yang, H., & Wood, F. (2018). An introduction to probabilistic programming. ArXiv Preprint ArXiv:1809.10756.
  • This book is very statistical machine-learing oriented, and focuses heavily on inference.

Programming Languages

• Good introductory texts:
  • Pierce, B. C. (2002). Types and programming languages. MIT press.
  • Harper, R. (2016). Practical foundations for programming languages. Cambridge University Press.
  • Pierce, B. C. (2004). Advanced topics in types and programming languages. MIT press.
• Denotational semantics
  • Gunter, C. A. (1992). Semantics of programming languages: structures and techniques. MIT press.

Probability

• Probability introductions
  • Chapter 2 of (Murphy, 2012)
  • Chapter 8 of (Graham et al., 1989)
• Measure-theoretic probability
  • Good introductions: (Axler, 2020)
  • General references: (Tao, 2011) (Kallenberg & Kallenberg, 1997)
  • Advanced and comprehensive references: (Fremlin, 2000)
• Inference
  • Excellent starting points: (Murphy, 2012) (Murphy, 2022) (Murphy, 2022)
  • Probabilistic graphical models: (Darwiche, 2009) (Koller & Friedman, 2009)

Similar Courses

Sometimes it is useful to see how similar things are taught in other contexts or at other universities. Here are some links:

• A prior offering at Northeastern: https://www.khoury.northeastern.edu/home/sholtzen/CS7480Fall21/
• https://web.cs.ucla.edu/~guyvdb/teaching-service/cs267a/2018f/
• http://danroy.org/teaching/2015/STA4516/
• http://probcomp.csail.mit.edu/9.S915/
• https://github.com/hongseok-yang/probprog17
• https://www.cs.ubc.ca/~fwood/CS532W-539W/

Bibliography

1. Murphy, K. P. (2012). Machine learning: a probabilistic perspective. MIT press.
2. Graham, R. L., Knuth, D. E., Patashnik, O., & Liu, S. (1989). Concrete mathematics: a foundation for computer science.
3. Axler, S. (2020). Measure, integration & real analysis. Springer Nature.
4. Tao, T. (2011). An introduction to measure theory (Vol. 126). American Mathematical Soc.
5. Kallenberg, O., & Kallenberg, O. (1997). Foundations of modern probability (Vol. 2). Springer.
6. Fremlin, D. H. (2000). Measure theory (Vol. 4). Torres Fremlin.
7. Murphy, K. P. (2022). Probabilistic Machine Learning: An introduction. MIT Press. probml.ai
8. Murphy, K. P. (2022). Probabilistic machine learning: Advanced topics. MIT press.
9. Darwiche, A. (2009). Modeling and reasoning with Bayesian networks. Cambridge university press.
10. Koller, D., & Friedman, N. (2009). Probabilistic graphical models: principles and techniques. MIT press.