I would absolutely have his wings clipped by either an avian vet or a breeder immediately, it makes taming and training so much easier. First of all they need to rely on you to get up off the floor, back in their cage, etc. And soon they realize that they can't fly off, so they will stop trying and just stay on your finger. Second, if you towel him many more times you're going to have a HUGE problem ever earning his trust. My Quaker was hand raised and I could hold her from day 1, and I clipped her wings the day after she got home, simply in anticipation of needing to in order to train her as she was only 9 weeks old. Plus she was flying into things all over my house because it was new, so it was also a safety thing. I didn't even need to towel her, I just clipped her and she was fine. Now she's full flighted. However, I had a budgie that was a parent raised baby and was scared to death of me, and once I clipped her I had her tame in a couple of weeks, to the point I could hold her and give her scritches.
That being said, never towel your bird unless you've built a strong bond with it. At that point the bird will get mad/scared at the towel, not your hands. I always let the vet towel my birds so there is no problem with me doing it.
Also, don't try to clip it's wings yourself unless you know what you're doing, they can bleed to death from a blood feather quickly.
I would make an appointment ASAP with a certified avian vet that is local to you, and preferably the avian vet you'll be using in the future. You should have a well-birdy checkup done since I don't think you've done it yet. You should always do this as soon as you get a new pet bird, whether the breeder/pet store says they did it already or not. They will do a full exam, weight check, fecal smears for parasites (bring a fresh dropping from right before your appointment), and some like to do blood work. They can then clip your Quaker's wings and toenails, if the toenails need it. They can also talk to you about proper diet and nutrition, common issues to Quakers, talk about behavioral problems, and answer any other questions you may have.
I would do this like YESTERDAY. You'll be amazed how much more quickly taming your bird will go once it's wings are clipped. And who knows, your bird may have a parasite, mites, wing lice, etc. that could be effecting it's behavior as well. If a bird is in pain, is sick, or is itchy and constantly uncomfortable it's impossible to build trust with him as he can think about nothing else but his health problems, and he probably blames you for the problems.