This is probably a can of worms, but the "get metadata at every single login" approach doesn't seem very reasonable to me.
We probably need to broaden the considerations here to the outbound OIDC support and how they're dealing with RP metadata, so we might park this for now or make a small improvement to it (maybe using a caching HttpClient is good enough?) but seems like a metadata resolver service for this kind of metadata is probably needed, either by expanding the SAML version or making a second one.
I would imagine the need to handle constant key rotation probably makes a straight addition of SAML metadata as an option here not a great fit for most OPs even if it's more secure, but it's not a hard thing to fit in if it's useful.
One thing a service abstraction around this would give is a way to encapsulate the key lookup steps. A model like SAML's would work easily since the keys are just part of the initial document, but the methods to access the keys would take the "kid" and be able to work in cached lookups of the keys from separate documents.