This is true in a sense, but also a bit misleading. The Speaker is essentially acting in the name of a House majority, and they can be removed by that majority at any time; it just provides them a degree of separation from accountability. If the Speaker is doing something that a majority of the House doesn’t want, they can always remove him or use tools like discharge petitions.

That said, there are very complicated power dynamics at play there, and you’re right that leadership does have too much power. But it does need to be reminded that this is only done with the consent of the majority.