Fix escript archive layout and support priv

[jonatanklosko]

Escript layout

I noticed that Elixir builds escript archive with all modules, app files and priv files flattened out. Contrarily rebar builds escript with directory layout batching _build.

The :code documentation states:

When an `escript` file contains an archive, there are no restrictions on the
name of the `escript` and no restrictions on how many applications that can be
stored in the embedded archive. Single Beam files can also reside on the top
level in the archive. At startup, the top directory in the embedded archive and
all (second level) `ebin` directories in the embedded archive are added to the
code path.

From the above it is clear that modules are loaded correctly with a flat structure, however when we deal with multiple applications, this may break other things, for example:

:code.lib_dir(:my_escript) #=> {:ok, ~c"/path/to/my_escript"}
:code.lib_dir(:phoenix) #=> {:error, :bad_name}

Given this, I would consider the current layout to be a bug, and this PR switches to a _build-like layout.

With the new layout, the above works as expected:

:code.lib_dir(:my_escript) #=> {:ok, ~c"/path/to/my_escript/my_escript"}
:code.lib_dir(:phoenix) #=> {:ok, ~c"/path/to/my_escript/phoenix"}

priv dirs

Currently, all application priv dirs are included in the archive, however reading priv from the archive requires special care (regular file access doesn't work), so in most cases this just unnecessarily inflates the escript size. For example, Phoenix priv dir includes all generator templates, it doesn't make sense to include these in the escript.

I propose a new option :include_priv_for to opt-in which privs should be included.

Note that this is not a breaking change, because currently escript.build docs actually say:

escripts do not support projects and dependencies that need to store or read artifacts from the priv directory.

Sidenote: due to flattening, :code.priv_dir/1 does not point to a valid location in the archive. Also, the priv layout is lost, and files could even be lost if there are multiple with the same name. This is to say, even without the above restrictions, I does not really work as expected.

For the reference, rebar does not include priv dir (other than for the rebar escript itself, via a private escript_incl_priv option).

[jonatanklosko]

We could also be more strict and say that they should not rely on the exact archive layout, instead they should do something like this:

priv_path = :code.priv_dir(:app_name)
escript_path = Path.expand(:escript.script_name())
priv_in_archive = Path.relative_to(priv_path, escript_path)

(Ideally they would use :erl_prim_loader, but using it with escript is apparently deprecated, because it will change in the future)

[jonatanklosko]

FTR the deprecation note for escript + :erl_prim_loader was added in erlang/otp#8091.

[josevalim]

It is fine to leave this as is, but I think these tests could be written without defining modules. We have a way of pushing updated configuration when we do Mix.Project.push, which simplify not having to create a bunch of modules.

We should probably refactor this whole test case to use said style, so it could be done later anyway.

[jonatanklosko]

Do you mean Mix.ProjectStack.post_config/1? I can refactor in a separate PR :)

[josevalim]

Yes!!

[josevalim]

💚 💙 💜 💛 ❤️