I'm pretty sure that James was not hand-raised by a breeder but rather parent-raised, the pet shop most likely got him from a mass vendor that breeds birds in a large room or warehouse, and they have little to do with the babies. So this is why James is scared of you, he was not abused and did not have a traumatic thing happen to him, he's just a normal, parent-raised baby bird that still has to be hand-tamed before he will like people! So that's the way you need to approach this, it's not a matter of giving him time to adjust to you, it's a matter of him not being a hand-raised bird and not being exposed to people much at all. In fact, most of the baby birds that come from vendors that mass breed and parent-raise the babies that they sell don't even see people on a regular basis until they get to the pet store. The "breeders" in these bird mills only have contact with their birds when they put food and water in their cages or aviaries. You have to realize that baby birds are typically inside their nest boxes and fed by their parents until they are 6+ weeks old, so James probably never even saw a human being until he was fully weaned and came out of his nest box to eat. Then he and his siblings were gathered up and shipped to the pet shops. So you have to start working with him every day, multiple training sessions a day to start to hand tame him. There are a lot of different methods to doing this, but start off by making sure that his cage is located in the room of your home that you're in the most and that has the most traffic. This is usually the living room. Every time you or a visitor even walks past his cage this will help to get him used to people. Talk to him every time you walk past his cage. He needs to be exposed to as many people as possible.
The method of hand-taming that works the best in my experience is very boring, but very effective. It involves putting a chair next to his cage and just sitting there reading to him (a book, magazine, etc.) or continually talking to him while resting your hand just inside his cage door. You need to do this for 20-30 minutes at a time, a few times a day. Each week you should be able to move your hand a little bit closer to him. Only move your hand towards him to the point where he just sits there and stops backing away from it. Then start reading or talking to him for 20-30 minutes. After each training session place his favorite treat either in his cage or if he'll take it from your hand or through his cage bars, however. Then a bit later do another session. The point is to be able to rest your hand a little bit closer each week or so. You do it at his pace, he dictates how fast it goes, not you. If you try to move your hand closer and he backs away or freaks out then pull it back again. Some people actually have to start by putting just the chair as close to their bird's cage as they can without their bird freaking out, and then every week or so they move the chair a little bit closer to the cage, until they get the chair right next to the cage, then they don't sit and read or talk with just the door open for a week. Then they test their hand on the outside of the open door. It's wherever he's comfortable that dictates it. When your hand finally gets close enough to him inside his cage that he could reach it on his own, then you need to put his favorite treat in your hand at that distance. Miller sprays are actually really good for this because they can stick out of your hand towards him at a distance. Once he's comfortable eating from the millet spray in your hand, then put the millet spray in the middle of your hand while extending your index finger out towards him, and freeze. You do this until he steps-up onto your finger and eats the millet spray. Once he's VERY comfortable doing this, the next step is moving him on your finger as close to the cage door as you can, little by little, until one day you can move him on your finger outside of the cage.
This process can takes weeks to months to many months to years. It's at the bird's pace. But it works very well. Just letting the bird outside of his cage and trying to get him to step-up isn't going to do it. He's the safest inside his cage, that's his space. They are very territorial of their space. So the idea is that if you can get him to trust you enough so that he'll step-up onto your hand inside his space, eat from your hand inside his space, and allow you to move him outside of his space, he'll then trust you anywhere and all the time.
"Dance like nobody's watching..."