Nobody bothers you personally, but eventually the two factions clash in Moonbrook. Quite unintentionally, the collateral damage reaches you. You're forced to flee Westfall to get far enough from Moonbrook to evade the damage.
Go back to when you could have returned to Westfall.