theres nothing you can do about this except lowering settings to the absolute minimum which may or may not give you a slight boost
i have a gtx 1080 in my system and my frames still dropped to 30-45 from around 100 when looking at certain areas or faction bases, it's just the way clockwork/garrys mod fucks you