Probably the most valuable tip when learning something new, is “what to read?” Here is a (partial) list of the books I found most valuable in my research.

  1. Cox, D. R. Principles of Statistical Inference. A high level, almost philosophical text, on the basic ideas in statistics. Possibly the most important, in this list, but make sure you read it last.

  2. Wasserman, Larry. All of Statistics: A Concise Course in Statistical Inference. A one stop shop for almost all of the fundamental ideas in statistics.

  3. van der Vaart, A. W. Asymptotic Statistics. Much, if not most, of classical statistics is asymptotic. This book can serve both as an introduction, and as a technical reference for the vast realm of asymptotic statistics.

  4. Hastie, T, R Tibshirani, and JH Friedman. The Elements of Statistical Learning. A one stop shop for machine learning, cast in statistical terminology.

  5. Tsybakov, A. B. Introduction to Nonparametric Estimation. My favorite reference on non parametric statistics.

  6. Greene, William H. Econometric Analysis. My favorite reference on the analysis of linear models. The economic orientation adds very useful applied motivation, and an emphasis on causal inference rarely seen in the statistical literature.

  7. Grimmett, Geoffrey R., and David R. Stirzaker. Probability and Random Processes. An excellent introductory textbook to probability and stochastic processes.

  8. Cover, T. M., and Joy A. Thomas. Elements of Information Theory. As the name suggests, your one stop shop for information theory.

  9. Wilcox, Rand R. Introduction to Robust Estimation and Hypothesis Testing. A very well written exposition of the vast world of robust statistics. A must read with whoever works with actual data, and not only theory.

  10. Robert E. Weiss. Modelling Longitudinal Data. My favorite reference on the topic.

  11. James E. Gentle. Numerical Linear Algebra for Applications in Statistics. Because we may write , but a computer would never invert a matrix to solve this problem.

  12. Venables, W.N. and Ripley, B.D. Modern Applied Statistics with S. Everything you need to actually analyze data in R.

  13. Rasmussen, C.E and Williams C.K.I. Gaussian Processes for Machine Learning. The first time I felt I understand RKHS.

Feel free to use this list, and recommend more entries.

Some other book recommendations worth following: