Ory Hydra cannot parse the base64 encoded code_verifier parameter on token request

[jfyne]

My Problem
I am trying to implement Code Flow + PKCE, using Ory Hydra as my identity provider. I am unable to retrieve a token. I believe the problem originates with this library and the way it generates the code_verifier parameter for token request (although I am not certain).

My config:

export const authConfig: AuthConfig = {
  issuer: 'http://localhost:4444/',
  redirectUri: window.location.origin,
  clientId: 'test-client',
  responseType: 'code',
  scope: 'openid profile',
  requireHttps: false,
  clearHashAfterLogin: true,
  showDebugInformation: true,
};

What works so far is the redirect to login with initCodeFlow(), I am redirected back to my angular app and then angular-oauth2-oidc attempts to request the token. This is where it starts to go wrong. The request that is made (I formatted it as CURL because its easy to read):

curl 'http://localhost:4444/oauth2/token' \
  -H 'Accept: application/json, text/plain, */*' \
  -H 'Referer: http://localhost:4200/' \
  -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.75 Safari/537.36' \
  -H 'Sec-Fetch-Mode: cors' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  --data 'grant_type=authorization_code&code=Sb3VNw4DoshWvNKwcik0c0RnOjULsxNHJdPtIPZ73fc.x1L4xBijtwk9zC_6EPGsPrWplXoNf-tzN6Plr-Fg2-Y&redirect_uri=http://localhost:4200&code_verifier=WIMRUzBojQj8-MYoyvM3IWek99R3OfhDeBYnvFX6witWu&client_id=test-client' \
  --compressed

The bit that is apparently not working is this:

code_verifier=WIMRUzBojQj8-MYoyvM3IWek99R3OfhDeBYnvFX6witWu

The response I get from Hydra is this

{
  "error":"invalid_grant",
  "error_description":"The provided authorization grant (e.g., authorization code, resource owner credentials) or refresh token is invalid, expired, revoked, does not match the redirection URI used in the authorization request, or was issued to another client",
  "error_hint":"Unable to decode code_verifier using base64 url decoding without padding.",
  "status_code":400
}

Now tracking this down to their code base the error_hint field is useful we see this code https://github.com/ory/fosite/blob/master/handler/pkce/handler.go#L176-L203. I've also used the Go playground to replicate it: https://play.golang.net/p/vVz7UOVd7-y. I have also noticed if I change the length of the verifier string by 1 character it parses successfully (its 33 characters at the moment).

Have I misconfigured the library somehow? Is this a bug?

Expected behavior
A valid token exchange.

[jeroenheijmans]

Interesting. I've seen this library work with at least IdentityServer4, and I believe others used other IDSes successfully with code/pkce.

You mention that "[the code is] 33 characters at the mooment" but I count 46 in the one you posted, which is more than the required minimum of 43.

Is it possible that this is a problem with your IDS implementation or configuration?

On the other hand, looking at your configuration for the JS library, it's significantly different from my attempt at using Code/PKCE, specifically the redirect uri (which is mentioned in your error message):

export const authConfig: AuthConfig = {
  issuer: 'https://demo.identityserver.io',
  clientId: 'spa', // The "Auth Code + PKCE" client
  responseType: 'code',
  redirectUri: window.location.origin + '/index.html',
  silentRefreshRedirectUri: window.location.origin + '/silent-refresh.html',
  scope: 'openid profile email offline_access api',
  silentRefreshTimeout: 5000, // For faster testing
  timeoutFactor: 0.25, // For faster testing
  sessionChecksEnabled: true,
  showDebugInformation: true, // Also requires enabling "Verbose" level in devtools
  clearHashAfterLogin: false, // https://github.com/manfredsteyer/angular-oauth2-oidc/issues/457#issuecomment-431807040
};

Hope that helps.

[jfyne]

Hi @jeroenheijmans thanks for the help. After I wrote this I actually tried switching out my Hydra endpoint and client ID with the one you have (I think I found your example repo), and it worked completely correctly.

You mention that "[the code is] 33 characters at the moment" but I count 46 in the one you posted, which is more than the required minimum of 43.

Sorry for the confusion, the encoded string is indeed not 33 but actually 45 long which I think comes from here https://github.com/manfredsteyer/angular-oauth2-oidc/blob/master/projects/lib/src/oauth-service.ts#L2091. What is 33 long is the number of decoded bytes that that maps to.

I have now tried changing the generated nonce length to 46 rather then 45 and now it works with Ory Hydra and IdentityServer4. Now the reason behind this escapes me at the moment, I may open an issue up in the Hydra repo to see if anyone there can explain.

[jeroenheijmans]

Okay let us know what you find, and whether we might close this issue for now until we know the culprit is on the angular library side?

[jfyne]

The more I look at this, the createNonce function is both generating a random number and then base64URL encoding it but not using the built in btoa. Could it be that this homegrown base64 encoding is not producing a valid base64URL encoded string that the Go standard library cannot parse?

[jeroenheijmans]

I think this is on purpose. Base64 is just that, and base64 url-encoded is something else. The latter is required by the spec. In fact, part 4.2 of the spec adds even more 'magic' to this, for fully compliant implementations.

It might be that your server only supports plain instead of S256 for PKCE? I'm not sure if our library supports both of them...

[jfyne]

Hi @jeroenheijmans after reading the spec, I am close to convinced the createNonce function is the culprit here. I created a PR #629 to change how it is implemented to follow what is mentioned in the spec. Ory Hydra does support S256, you can see the implementation here.

The modified implementation is like this, I quoted the parts of the spec so you can see my reasoning.

1. Generate a high entropy cryptographic string, the current implementation looks like it is using just the base64url characters and is not using "." and "~".

code_verifier = high-entropy cryptographic random STRING using the
unreserved characters [A-Z] / [a-z] / [0-9] / "-" / "." / "_" / "~"
from Section 2.3 of [RFC3986], with a minimum length of 43 characters
and a maximum length of 128 characters.

2. Then base64url encode the high entropy string. So once we have generated our string, base64url encode it to be transmitted. For that I just used the base64UrlEncode function in base64-helper.ts which does use btoa.

The octet sequence is then base64url-encoded to produce a
43-octet URL safe string to use as the code verifier.

[tech-sam]

Hi @jeroenheijmans ,I am also facing the same issue posted by @jfyne regarding base64 encoding using with using Ory Hydra as my identity provider

{
    "error": "invalid_grant",
    "error_description": "The provided authorization grant (e.g., authorization code, resource owner credentials) or refresh token is invalid, expired, revoked, does not match the redirection URI used in the authorization request, or was issued to another client",
    "error_hint": "Unable to decode code_verifier using base64 url decoding without padding.",
    "status_code": 400,
    "error_debug": "illegal base64 data at input byte 44"
}

can we have a solution for this as PR already raised by @jfyne
@jfyne thanks for the PR and mentioning the details

[jeroenheijmans]

Branch got merged, I presume this is now fixed in version 9. Closing the issue, let us know if it persisted nonetheless.

[jfyne]

@jeroenheijmans I have tested with the new release, everything works as expected. Thanks!