cannot add a 3048 character long environment variable

[12rambau]

Details

• Read the Docs project URL: https://sepal-ui.readthedocs.io/en/latest/
• Read the Docs username (if applicable): borntobealive

description

In my repository one of the dependancy is Google Earthe Engine API. this API require a manual authentification or a private key. In order to use the private key I've been to my admin panel and added it to environment variable (EE_PRIVATE_KEY).

The build unfortunately failed.

Looking closely at my env variable, I realize that the final character is not the appropriate one.

the Value field of the website can only handle 2048 characters and my key is 3044 character long. is there a way to change that ?

[stsewd]

Is that key provided by google or a manually generated one?

[12rambau]

google provide a json file,

I used the following script to change it into a string :

import json
import base64

with open('sepal-key.json') as jsonfile:
    data = json.load(jsonfile)
    datastr = json.dumps(data)
    encoded = base64.b64encode(datastr.encode())

print(encoded)

Looking at my code I assume that you would prefer me to use base16 or base32 instead of base64 right ?

I'm using the same key in travisCI and there were no problem so is there any technical reason why you don't handle it ?

[stsewd]

@12rambau I'd recommend to not store the full file in an env var, but encrypt the json file in the repo and store the key in an env var, then you can decrypt it in the script you have. Not sure about increasing the size of the env vars would help to completely support this case, the size could increase over time, and we would need to increase the size again.

[12rambau]

Your solution seems totaly legit.
I will wait before implementing because it will require me to change the decoding procedure of my API key in my lib and change it in all the othoer services that already use this key :

• TravisCI
• codeClimate
• CodeCover
• GithubCI (when I'll manage to make it work).

My point is the following : even though my strategy wasn't the most efficient, wouldn't it be normal to align the RDT requirements for environment variable with the other comonly used services ?

[humitos]

wouldn't it be normal to align the RDT requirements for environment variable with the other comonly used services ?

What's the max length of each of the other services? I'm fine to have a similar value here.

[12rambau]

No idea I just know that mine was entering. I've asked on SO let's see if someone knows

[12rambau]

It seems TravisCI is not a concern anymore and my SO question was long deleted. nonetheless, I found the information I was looking for in the Github Documentation:

https://docs.github.com/en/actions/learn-github-actions/variables#limits-for-configuration-variables

max size = 64 Kb

Do you think it would be doable to set the same limitation to your service ?

[humitos]

This looks like a pretty edge case to me. However, I'm not opposed to use 64Kb as max size if it that's what other services are doing.

[12rambau]

I faced it again in 5 different documentations. In fact anything that should rely on a Google API token.
the initial workaround is forcing me to use set a encrypted private key in my repository which is really failing the whole purpose of the environment variable.

Any news from your side ?

[humitos]

Hi @12rambau. No news here. Unfortunately, we weren't able to prioritize this work yet.

[12rambau]

@stsewd I tried as much as I can to use the method you advertised. It works ok from Readthedocs but then the authentication mechanism needs to be tuned again for public contribution which vastly limit the impacts of my projects as the users face lots of difficulties to locally build the docs. Everyone is saving the json key forwarded by google services because it's the expected way to use service account.

I see it's low priority for you but I'm now facing the same problem with https://github.com/google/Xee, which requires the same mechanism and a wider audience than my usual projects.

You remain the only service where I cannot use this key directly (but you are also the only one that manages sphinx docs
in a satisfying way so here I am crying 😢)

I usually try to contribute but unfortunately here I cannot do anything else than require your support.

[humitos]

docs.github.com/en/actions/learn-github-actions/variables#limits-for-configuration-variables

I read this, and I think we could probably do the following:

• Increase the individual environment variable size to 48KB (using CharField's max_length property)
• Limit the size of all the variables together to 256KB (using a validation form that sums up all the variables for the project)

you are also the only one that manages sphinx docs in a satisfying way so here I am crying 😢

Thanks 🙏🏼

I usually try to contribute but unfortunately here I cannot do anything else than require your support.

I put this issue in our list to be discussed in our next meeting. I'll propose the solution from above, which seems doable without too much effort. Let's see if all agree on that and we can prioritize it.

[ericholscher]

We discussed this today, and assuming we don't hit a technical issue, we're 👍 on moving forward here.

[12rambau]

Thanks a lot for accepting the request, that will help a lot people using Google tools in their documentation!