IMHO a shelter trip results in a quick death for most animals, esp. now with the economy, in calif they are euthanizing something like 1700 cats a month now, more then triple from last year.
No one should ever, EVER, think of turning ANY animal loose out in the country. If you think it would be a death sentence to take it to the SPCA, chances are it is also a death sentence to release a domesticated cat in the country.
I'm sure there must be no-kill shelters in your area. We have two no-kill shelters for cats here in my area.
If its a no kill shelter, and the owners do not pick him up, they will vaccinate him, and get him ready to be fixed. If the owners want him back after that, they'll have to pay.
Maybe you should just post a letter in their male box first and tell them that if you catch him on your property, you will take him to a shelter, no questions asked. Tell them he is aggressive, and that you aren't going to deal with it.
If they persist, call the cops. If it was a dog on your property, terrorizing your animals, they would do something. If you know for sure its the cat, maybe they can pick it up, or deliver a notice to have your neighbors contain it properly.
You could get him neutered then turn him loose again. The owners probably wouldn't even notice. Maybe the other neighbors that are having trouble with him two will pitch in for the neutering cost!
Hope your male kitty gets well soon. I'm sure he feels lots better after going to the vet, your a great cat owner.