I think there are some women who don't like babies. I'm not one of them. I think a lot of men are intimidated by how small they are and find themselves afraid of handling them.
I can understand that. I have no kids, but when my nieces and nephews came along, I was very involved with them. I gave each of the couples a week's babysitting so they could go on vacation. I had the kids do sleepovers for Thanksgiving, Christmas, and other times. I cleaned up snotty noses, changed diapers gave them baths, etc. Nos they've had their own kids, and I haven't done any of that. They now have their own aunts and I get great-aunt status :)
I get called Grumpy, even though I rarely am with them. My granddaughter in particular is the apple of her grandfather's eye - I had to scold her once for scribbling on the wall and she was scandalized! "Grumpy doesn't get angry with me, that's not how the world works!"
My grandson is harder to reach, due to his autism. I love him to bits, but he's in a different world much of the time.
I'm sure you're doing exactly as you should with your beautiful grandson. You'll continue to love him and keep working on the bridge that will connect your world with his. I'll bet you both have lessons to share with one another.
He's four and currently non-verbal. His sister is two and talks enough for both of them and then some, but given her age, quite a lot of what she says is intelligible to her and nobody else.
They are when they're behaving. Zachary has autism-related "meltdowns" from time to time, which are never pleasant. Zayianna is a red-headed diva with a temper to match - luckily I never have to be the "bad guy", discipline is Daddy's problem.