Unlike higher gas prices, which hurt almost everyone, higher borrowing costs have an uneven impact.