4 nights is probably good enough. We did a whale tour in September one of our days there, but not sure if you’ll get that chance in June.
Location looks fine. We had lunch on that side of the sound one day and enjoyed lobster sandwiches. Otherwise we didn’t do much on that side
Is most of the hiking and sights over by bar harbor or is it pretty evenly spread around? We aren't really looking at spending much time in town.
Like shaftnuts said, Acadia is rather small so it’s not a big deal to go from your side of the sound over to bar harbor and back.
Granted we went in 2014 so my memory may be fuzzy, but the only really “crowded” attraction was Cadillac mountain sunrise. If you plan to do that, get up early enough or I think you could bring a tent and camp out if you have the people willing to do it.