I’ve been to Nikko many times but somehow never got around to visiting the Tamozawa Imperial Villa before. It turned out to be well worth a visit. The place has a quiet atmosphere and nice garden, and the residence itself is impressive. In fact, it's one of the largest of its kind in Japan.
This time I drove, but I think it’s walkable in about twenty minutes from the Tōshō-gū area. There's also a bus stop nearby, but if you're visiting in high season and want to continue afterwards to Lake Chūzenji, you're risking not being able to get on that same bus again (it will be packed with people, especially on a weekend). So, I'd recommend to either have your own wheels or make it just a trip to central Nikko without going up the Irohazaka road to the mountains. There are plenty of things to see and do around the Tamozawa Villa to occupy you for a full day.