Supplemental reading: AIMA 17.1.0-17.1.2. (AIMA uses slightly different notation than Sutton/Barto. I'm going with Sutton/Barto notation, but the concepts in both are the same.)