When I rewatched the episode I couldn’t help but look at Jackson’s face during this scene. He just looks so… surprised? It’s like he expected Lydia to have a petty reason for keeping the key. He expected her to fight or call him names or cause a big scene.

He didn’t expect her to pull the key from around her neck. He didn’t expect her to keep it so close to her heart. I think, most of all, he didn’t expect her to care that much. Jackson strikes me as the kind of person who expects everyone to leave, so he pushes them away before they get a chance. At least then it’s his choice. At least then, when they’re gone, he could say that he saw it coming. 

And if he’s alone? Well, that’s his choice, too. Who needs them, right?  
So when he’s confronted with genuine affection or caring, when he thinks about Lydia carrying his key around all the time, I don’t think he knows how to handle that. And to see it in the wake of him being such a colossal ass? For a second he’s completely thrown by the idea that she could still love him after all of that. 

I know he has his moments of being an unbearable asshole, but right here? He kinda broke my heart.