Hmm I would say I have a decently strong army

and I do not raise my port above 3. Granted I stir up more trouble than most, but I am not really worried about someone breaking through my defenses. More of a peace of mind thing I suppose. Besides I don't do that much shipping. But for you I would not go above 3, 4 if you are feeling risky or have the army 100k+ to back it up.
Things that make you a target Low military, 4 or higher level port, lots of gold stock and then pissing someone off.