Should I be obsessing about load times?

Ryan in Dearborn, MI asks: “Many SEO’s are obsessing about your speed comments, but I get the feeling speed will be a very minor component in ranking. Am I right, or should I be obsessing about load times as well?”