Welcome TKS; what's your birds name?
I have worked on "rehabilitating" many phobic animals, but most of them have been dogs and cats. I can give some advice that applies to most animals but there may be parrot/amazon experts out there who can give more species specific info - the first such person to contact would be a good avian vet if you have not already had your bird in for a thorough check up.
Patience is key in working with a phobic animal, his reactions will partially reflect your reactions, so controlling your own emotions/behavior is essential. If he panics make sure you remain calm, let him see that whatever "spooked" him does not bother the rest of his flock in the slightest. Of course don't scold him for panicking; but also be careful of trying to calm him down by talking nicely to him - when a phobic animal panics many owners naturally want to calm them down so they talk to them in a soft comforting voice but what they end up doing is rewarding them for being in a agitated state. Behavioral contingencies do not effect only consciously controlled or 'willful' behaviors but they also effect physiological reactions of fear. So in other words if he panics and runs back to his cage just let him go, let him see that NOTHING happens: nothing bad, nothing good. Then when he calms down a bit you can go talk to him a bit.
Thats the general 'attitude' to approach a phobic animal with, but now a couple of specifics that should help. First I'd recommend that you don't close his cage door when he comes out. He should have a safe-zone that he can always escape to. It's much harder for an animal (or human) to feel safe exploring a new environment unless they know they can still get back to their safe familiar place.
With the cage door open he can come and go as he pleases, if he panics he can run home and calm down. With the cage door closed (when he is out) if he panics he has no where to run which most likely will increase his anxiety or panic, but more importantly it will force him to remain in the out-of-cage environment (living room or wherever) in a panicked anxious state. You don't want him to associate your home with that anxiety. So leave the door open.
But of course you do want him to come out for longer periods, but you want that to be his choice. To get him to Choose to come out you can use a shaping technique:
It is very good that he will take peanuts from you and this will be important. If a phobic animal will accept rewards it is MUCH easier to help them overcome their fears. Do explore other treats to in order to see what really motivates him.
Leave the cage door open and sit close to the cage with some treats. Hold one out on the floor where he can come get it. If/when he approaches avoid any kind of reaction or eye contact. He will likely accept the treat then go back to the cage; once he does this much then repeat the same thing while calmly talking to him. Move incrementally farther and farther away from the cage (small increments - baby steps) for future sessions/repetitions. Mix in going farther from the cage with gradually adding more activity - perhaps another person in the room, or you move around a little, or have a houseplant or any other potential frightening object with you.
Continue this until he will come out of his cage and across the room to take the treat (obviously he will be more motivated if you do this before he is fed.) Eventually when he can come a good distance out of his cage keep offering him treats to keep him with you: ie give him the first treat when he gets to you, then offer another, and another so he doesn't flea as quickly. Finally extend the time between these treats so that he will sit next to you out of his cage waiting for the next treat.
By the time you have gone through all the above he should be quite comfortable staying out of his cage for extended periods of time. There will likely be things he still needs to habituate to: perhaps a phone ring will startle him, or the clanking of pots and pans. When this happens remember where we started - do not react to his reaction. Let him have his panic attack but ignore it, and when he starts to calm down talk to him and entice him back out of his cage.
Rehabilitating a phobic animal can take quite a bit of time working a little each day over weeks or sometimes months. But I suspect it should go well for you, if he already accepts treats from you that is a very big initial step.
Keep us updated as the above overview was just that - an overview. It will not likely work as smoothly as I have described it, but for every 'snafu' there is a solution.
Lastly to reiterate if he has not been to a good avian vet yet that would be the absolute first step: animals can appear phobic if they are sick or in pain, so those should be ruled out before much training is attempted.