Matsue has two areas famous for its natural hot spring water. These hot springs are rated highly in Japan for health and beautifying of skin and can be enjoyed in free natural foot spas, day trip hot springs (where you can bathe without staying for as little as 400 yen), and many traditional Japanese inns (ryokan) which include hot springs as part of their traditional Japanese experience. Some even buy the water to bring home with them and use in their daily hygiene routines.
The more famous of the two areas is the neighborhood Tamatsukuri Onsen, which was built around its hot springs. The name literally means "egg-making hot springs" - hot springs baths being a popular method of soft boiling eggs in Japan. The main street has a fantastic atmosphere created by the rows of cherry blossom trees, (yes, it's a wonderful hanami spot) which stretches for around 2 km, and a natural stream between the roads. At several points natural hot spring water flows into sectioned-off parts of the river so you can enjoy a comfortable foot spa at the riverbank. In warmer months you will find most people walking through the street in yukata, light Japanese clothing. The ryokan come in at various price points and all have unique charms. For example, Kasui-en Minami is renowned for its beautiful Japanese garden you can enjoy from their open-air hot spring, and Chorakuen boasts the largest open-air hot spring in Japan. I didn't check with Guinness but it is very large and I couldn't find anyone challenging them for the title.
The “Tamatsukuri line” bus goes directly to the onsen street from Matsue Station (25 mins) and Matsue Shinjiko Onsen Station (40 mins).
Tamatsukuri Onsen is definitely worth the excursion, but if you wanted to stay closer to the center of Matsue there are plenty of traditional inns where you can enjoy hot springs. In fact, even every western-style hotel I looked at offered natural hot spring water in each room's private bath, piped in from the hot spring area "Shinjiko Onsen." Located near the castle, Shinjiko Onsen features many ryokan inns, day trip hot springs, and free foot spas, just without all the scenery of Tamatsukuri. It's a convenient way to enjoy natural hot springs while staying closer to the action, and to the transport hub for visiting other spots.
Among central inns, Minamikan is the most famous. The inn is famous for its gardens which envelope the restaurant (which can be enjoyed without staying overnight), the quality of its private bath hot springs, and the view of the river and lakes from the rooms. The building was originally opened as a vinegar distillery in the early Meiji Period (they don't say the year, but I would assume early 1870's) before becoming an inn in 1888.
Minamikan is located between Matsue Station (15 minutes by foot) and Matsue Castle (10 minutes by foot), and is served by many bus routes.