No need to feel remorse for being a bully. To state the obvious, there are only two ways to play this game:

Sim-Colony Enthusiast
You like seeing your colony grow and flourish through trade, resource production, and keen management style. You have a low wall level, low barracks, high warehouse, and an even higher port to load those goods immediately. To avoid the costs of maintaining a war machine you prefer very few troops for general defense. Gold is an important commodity because it's gold (buy low sell high). AP has very little impact. VERDICT: Will eventually lose.

Conquerer
Sim-Colony types are your bread and butter and it doesn't matter if they're equal contributors to island resources, part of a large guild, or someone who has supplied you goods via trade: they're all fair game. You swoon when your spy reports a target with a loading dock greater than 3, and gold is only as useful as the distance your armada must travel before bankrupting and disbanding. AP is the tool of the devil. VERDICT: Guaranteed to win.

I don't want to pillage someone into folding, but it's not that difficult to stymie an attacker and make a pillage as unprofitable (or more costly) as possible. I'd say that 4 out of 5 my farms all have a General score under 200 and a port at least level 2. Simple solution: raze the port to level 1, crank out a handful of phals, and for the love of all that is holy build a ship - any ship - to repel casual attacks.