I don't think you are overeacting. I think if you continue on the same path it's very likely you are heading into some really rough times with your fid. If you have the commitment & willingness to correct the problems now, you'll need the dedication & to put in time not only to correct the problems now but also to maintain it so he has a happy, fulfilling life .. if you do then get started. If not well why do you have a parrot then?
Someone with more knowledge and experience can help address the biting/aggression but personally I tend to believe it's more frustration / anger / boredom, not being shown boundaries, lack of attention etc. Learning to read his body language, keeping him busy, bite pressure training, spending time with him etc. I'm also not sure if it could be an age/hormonal thing.
Some more information may help
How big is his cage?
How many/what type of toys are available to him?
How much out of cage time does he have daily?
How much hands on/ hanging out with you time daily?
What's his diet like?
Where are you located? Are the temps/weather ideal to set up space for outside time as enrichment?
That's great that you took him for his annual checkup.. was it an avian vet & were there any health issues?
Parrots are a commitment. You have to make time for their care & well being. If it's not possible than you're probably going to have a lot of issues on your hands. Would you want to chill out in a cage alone all the time, watching your busy human?
How do you keep your parrot happy if you are so busy?
The short answer, I made a commitment when I brought a parrot into my life, I can't be so busy that I'm not fulfilling her needs and giving her the best quality of life I can that especially & including my time. If that means I miss out on other things.. so be it. If that means I get up earlier for morning play time on a day I know I can't be with her for a few hours.. so be it. Those times she is left to her own devices I ensure she has plenty of choices in toys etc. that she isn't stressing out or becoming bored.
Time is subjective ... just because you may not have time to stop & play with him hands on for long duration several times thru out the day.. he could be sitting with you while you're on the computer, watching TV, singing with him while you're getting a shower, dressing, cooking etc. there are a lot of busy times that can be used to socialize and give companionship during those days/times when life is hectic.
They can be busy with us occupying the same area so they feel a part of what is going on even if it's just hanging out doing their own thing. This is still quality time spent to them.
When you aren't able to be home it really is imperative that lots of play room within the cage, varying types of toys including foraging, shredables, destructible whatever fun toys can be found along with various types of perches. This helps keep them active, entertained & not just turning into perch potato's waiting for the next time you walk by or come home.
I can't ignore
Unfortunately, I panicked a bit, so I hit him so he can get off me (not hard, but enough to feel really badly about it)
I'm sure you feel badly but it's obviously concerning that your reaction was to hit him. That can't happen... you're talking about a very small being .. not hard? To a parrot that weights what 165 grams.. any hit is hard. I'd highly recommend that when you attempt to handle him that you have a toy, treat, something as a distraction to avoid the bite if you can't handle getting bitten. I implore you to keep control of your emotions .. the damage his beak can do is nothing compared to what you can do to him in retaliation. Please research bite pressure training, work with him to understand "easy" "no bite" "gentle" etc. I usually say handle away from the cage but in this case if you are prone to panicking and/or reacting physically then I'd say work with him at a stand or near the cage so you can quickly get him down before you react.
If you haven't read it yet I really like this thread
Big Beak O Phobes Guide to Understanding Macaw Beaks
Of course there is a ton of other information under the Macaw forum section.
I am sure the more experienced parronts will be along to help.