I’ll confess to getting annoyed when someone pulls on my seat back, particularly on an overnight flight. Some flight etiquette sources call it out.
19. Don't be a wrecking ball when walking down the aisle
Your fellow passengers don’t deserve to be yanked around every time you rise, so do not under any circumstances short of complete collapse grab the back seats when you walk down the aisle: A fellow passenger's headrest is not there to help you gain momentum on your long crawl to the bathroom. And if you're wearing a backpack while boarding, take it off and carry it by your side or snugly in front of you, lest both your butt and your backpack smack into someone's head the moment you turn around.
There is an option, the “secret” handrail - https://flytrippers.com/pro-tip-the-secret-handrail-to-help-you-walk-down-the-plane-aisle