Rank is bestowed for completing main quests (not repeatable ones).
Taking the Vertibird to Mass Fusion during Spoils of War turns the Institute hostile.Completing End of the Line requires wiping out the Railroad.Teleporting to Mass Fusion during Mass Fusion turns the Brotherhood hostile.However, certain quests will turn factions permanently hostile: All four major factions can be joined in the course of the game.
