Here are two images taken at what must have been period nudist colonies:
The whole "shirts-as-underwear" thing is an exaggerated overstatement; an etiquette book may generally refer to proper company or more formal situations. Just because construction workers today will take their shirts off while working doesn't mean that's how they show up for dinner at the Waldorf. Just as then, there were casual situations where people did without jackets/vests, and didn't send everyone running for their nearest fainting couch.
Marc A. Hermann.
The Daybreak B'hoys.
Liberty Rifles - Hardtack Society.
Oliver Tilden Camp No. 26, SUVCW - Co. I, 83rd NYV, SVR.
Descendant of Pvt. E. Hermann, 45th PA Militia - Capt. Wm. K. and Lt. Geo. W. Hopkins, 7th PA Reserves - Pvt. Jos. A. Weckerly, 72nd PA Infantry - Pvt. Thos. Will, 21st PA Cavalry.