The --pkcs11Config option acquires a configured PKCS#11 provider by:
- Using reflection to get the 1-string-arg constructor for the keystore provider class,
- Calling that constructor with the pkcs11Config value to generate the Provider to use.
As far as I'm aware, that was the only way to do this when we pulled this code together originally; it may or may not be the only way to do it in Java 8 but as described below I don't know of one.
This doesn't work in Java 9; that constructor has been removed. Instead, you must acquire an unconfigured provider using Security.getProvider("SunPKCS11") and calling #configure to generate a new, configured, provider.
The existence by default of the unconfigured provider, and the presence of the #configure API, starts with Java 9, the same release in which I think reflective access stopped working.
As a result, I don't think there's a way to write code that will work on a Java 8 baseline and still work under Java 11, or (by definition) a way to write code under Java 11 that will still run under Java 8. So, I think it's true to say that we can only fix this by using a completely new implementation in a V3 of xmlsectool. At least this means we can throw the reflective code away, so that's something.
In terms of workrounds, you can still use xmlsectool 2.x with the alternate mechanisms but that involves user modification to the JRE's java.security file and I am sure people will be very wary of that.