I'm not sure there is a silver bullet for this stuff. I think for the most part you just have to chip away at getting him comfortable with you and building trust. If he's food motivated, then you can make some inroads through treats. I've developed a routine with mine where they get a meat treat out of the fridge every morning, and they look forward to it, and it's a good bonding thing. If he's very play motivated, then that can also be a big positive for you to be a favorite playmate for him. But these things take time.
Some don'ts though...
1. Don't fall into the trap of forcing affection or physical contact on him when he's not receptive towards it. It doesn't "get him used to it", it just makes him resent it, and you.
2. Unless it's a life-or-death type of deal, try to avoid having to use excessive physical force to get your cat into the carrier, and then once at the vet, getting him to get shots and so forth. That leads to sort of a death-spiral of lack of trust and increased stress. I fell into that trap myself with one of my cats, Serafina, and it's taken a lot of years to rectify that, and I probably never really will completely rectify that. I think if I were to do it over again, I would:
a. See if I can get a vet to come to my home to do the exam if my cat was just too stressed out.
b. Using a sedative is another option. At times I've been able to slip that into Serafina's food prior to an exam, which makes that go better. Talk to your vet about this option.
c. Make sure that the vet is cat-saavy enough to know not to use a bunch of physical force get give shots and so forth. Now that I've seen how both good and bad vets approach that stuff, I know better than to settle for such a vet.
As far as getting him back into the house, again, I think you just need to work on bonding with him. The fact that he's a scardy-cat will mostly work in your favor in terms of him being a flight risk, as the outside will probably scare him too, and once you've bonded with him, and he's gotten real comfortable with the house, he'll be a very low-flight-risk cat I'm guessing. Make sure he gets a chip though, and a break-away collar (preferably with a bell on it) is always an option too.
Scardy cats can be kind of awesome because once you do bond with them, then you have a pretty special relationship with them, so keep the faith.
The little monsters 3