Use HEX_CACERTS_PATH environment variable to allow custom override of system defined CA certificates

[madlep]

Allow specifying a custom cacertfile for mix operations that need to make HTTPS requests. This allows mix operations to run in corporate environments that have a managed network with an SSL proxy that requires use of a custom cert file instead of the system default.

On managed network with SSL proxy

Without environment variable on work laptop:

> mix local.hex
** (Mix) httpc request failed with: {:failed_connect, [{:to_address, {~c"builds.hex.pm", 443}}, {:inet, [:inet], {:tls_alert, {:unknown_ca, ~c"TLS client: In state wait_cert_cr at ssl_handshake.erl:2125 generated CLIENT ALERT: Fatal - Unknown CA\n"}}}]}
 
Could not install Hex because Mix could not download metadata at https://builds.hex.pm/installs/hex-1.x.csv.
 
Alternatively, you can compile and install Hex directly with this command:
 
    $ mix archive.install github hexpm/hex branch latest

With environment variable enabled on work laptop:

> HEX_CACERTS_PATH=~/.local/share/my_custom_cacert.crt mix local.hex
Found existing entry: /Users/jdoherty/src/opensource/elixir/.mix/archives/hex-2.0.6
Are you sure you want to replace it with https://builds.hex.pm/installs/1.16.0/hex-2.0.6.ez? [Yn] y
* creating /Users/jdoherty/src/opensource/elixir/.mix/archives/hex-2.0.6

On regular connection

On personal laptop, without need for custom CA cert file

> mix local.hex
Are you sure you want to install "https://builds.hex.pm/installs/1.16.0/hex-2.0.6.ez"? [Yn] y
* creating .mix/archives/hex-2.0.6

[josevalim]

Maybe we should use HEX_CACERTS_PATH, since that’s what Hex uses anyway so it is fewer stuff to set? cc @ericmj

[madlep]

@josevalim that would make sense. From what I can tell mix only does direct HTTP (without delegating to hex) for mix local.hex and mix local.rebar.

Rebar also has it's own separate mechanism for setting this when it's installed via it's config file.

[josevalim]

Ok, once it is pushed we will merge. :)

[madlep]

@josevalim have updated to use HEX_CACERTS_PATH, and added new section to docs on environment variables to separate it from the MIX_... defined variables.

Have manually tested on work laptop with managed SSL proxy (Zscaler) , and personal laptop without, and both work as expected 👍

[josevalim]

💚 💙 💜 💛 ❤️

[ericmj]

We have been considering deprecating HEX_CACERTS_PATH in Hex, or at least suggest not using it, in favor of configuration to use the system certs instead of Hex's built-in certs. This would be in line with Elixir that from 1.17.0 will use the system certs by default.

If you are in environment that requires custom certs presumably they are already installed with the system certs so I don't think any extra configuration is needed.

@madlep Is there any reason why your custom certs are not installed in the system store?

[madlep]

@ericmj

@madlep Is there any reason why your custom certs are not installed in the system store?

With the way IT/security has got our laptops setup, they're installed in the System keychain on macos, but NOT in the System Roots keychain - which is where :public.cacerts_get() gets it's config from.

Regular web access and desktop apps are ok with that. For CLI apps - it's hit and miss. Some work fine, others need some environment flag or config set for them to be able to make HTTPS calls. So any HTTPS access via Elixir/Erlang is blocked unless the cert file is explicitly set when setting up the request in code.

If I run the following, I can see 149 certs configured in System Roots keychain, but NONE configured in System keychain (ie not the custom cert for the corporate SSL proxy)

Wondering if that's something that could be looked at upstream in the Erlang public_key module... 🤔

:public_key.cacerts_get()
|> Enum.map(fn {:cert, _der, otp} -> otp end)
|> Enum.map(fn {:OTPCertificate, tbs_cert, _sig_alg, _sig} -> tbs_cert end)
|> Enum.map(fn {:OTPTBSCertificate, _v, _sn, _sig, issuer, _val, _subj, _subjpki, _issid, _subid, _exts} -> issuer end)
|> Enum.map(fn {:rdnSequence, attrs} ->
  attrs
  |> Enum.sort()
  |> Enum.find_value(fn
    [{:AttributeTypeAndValue, {2,5,4,3}, value}] -> value
    [{:AttributeTypeAndValue, {2,5,4,11}, value}] -> value
    _ -> nil
  end) || attrs
end)
|> Enum.map(fn {_, val} -> to_string(val) end)
|> Enum.sort_by(&String.upcase/1)
|> Enum.each(&IO.inspect/1)

[madlep]

@ericmj Yup. Looks like it's hardcoded to just check the System Root keychain at https://github.com/erlang/otp/blob/master/lib/public_key/src/pubkey_os_cacerts.erl#L172

Will have a look at if it makes sense to check other keychains there too. That might be a future option if it can be changed in the Erlang stdlib.