There are a crap ton of little mice running around our compound out here in Afghanistan its gotten stupid. Yes they are cute little critters but they can tear up land and property like crazy. Live traps like everyone says and relocate them. If you off them I suggest you use a subsonic .22. All you literally hear is the firing pin strike the end of the round.
I've caught 2 mice by using the good old shoe box, prop and string method lol. Other guys i work with got so bored that they made a wooden trap with string, sat off in the distance with night vision googles and caught 15 in one night.