The shitamachi still has several sento, or public baths. This one is in Ueno. What’s the difference between an onsen and a sento? The former has natural hot water; the latter runs hot water from a tap. Read more here.
PS: The sign for hot water is ゆ (yu).