The first, obvious, problem is your statement that you're "grabbing" him, which NO bird is going to like, ever!
The second problem you mentioned is that 4 people are interacting with him and that leaves so many options open - like.......... are any, or one, of these other people taunting, teasing, or also trying to "grab" at the bird when you are there? or even when you are not there?
If so, then you are going to have a very hard time training this bird, until you teach the bird that you are his "friend" and not an enemy. Building trust with this bird is your FIRST and MOST IMPORTANT priority right now and not teaching him tricks until that is established.
I really hope you love and care enough about this bird to keep asking questions and trying to do your best for him.