Our jenny is equally shy about hands, and fast movements as well.
She will only eat Dumor Horse Cookies (the Oatmeal Raisin & Honey ones.) She likes that kind so much, that she'll wrap her lips around your entire wrist, because she's so excited about the cookie you're trying to feed her.
Her daughter, on the other hand, who was born here less than a year ago, will take anything from you (even sour green blackberries), and isn't shy in the slightest about hands. (Correction - she won't take her momma's cookies, she doesn't like those for some reason.)
I have the feeling that somewhere in the past of your boy, and my jenny, someone used hands in a not-so-nice way. They are nervous about hands and what they might do, so they stop what they are doing, out of preparation for what they think might be coming.
It takes time, and you still might never be able to overcome the issue. My suggestion is to hunt for treats that he likes and work them to your advantage.