Epic: Re-use existing project scripts (Makefile, CI, Dockerfile)

[loujaybee]

For most projects, they already have existing scripts for building/starting their projects [1]. Gitpod should be smart about detecting and re-using these scripts without forcing developers to re-write a new gitpod.yml file. We already have auto-detection for things like NPM commands, but we also should look at other configuration set up's like:

• CI configurations (GitHub actions workflows)
• Makefiles
• devcontainer.json [1]
• Existing Docker configurations
• Run/Debug configurations

There are two broad options (for the different configuration files):

1. Directly integrate with the configurations, like the devcontainer standard [1]
2. Map from these configurations to Gitpod configurations (either implicitly behind the scenes, or via explicit suggestion)

Related [1][2][3]

[csweichel]

Sounds a lot like the project configuration inference @gitpod-io/engineering-meta just shipped, no?

[JanKoehnlein]

See also #7392

[jankeromnes]

Sounds a lot like the project configuration inference [...] just shipped, no?

True. 🎯 I read this like a collection of feature requests for that feature (the 4 bullet points) & a discussion about approach (1. vs 2.).

CI configurations (GitHub actions workflows)

This sounds like a good idea in principle, but I wonder:

• how easy it is to parse a GitHub Action / other CI config file (they generally come with complicated multi-step workflows, dependencies, extra features like between-step caches, or third-party workflows, etc)
• how relevant a CI job is to produce an actual dev environment

For example, if we look at two of our repositories (one using GitHub Actions, another using CircleCI), I don't see much that we can use to infer a relevant dev environment:

• https://github.com/open-vsx/publish-extensions/tree/master/.github/workflows
• https://github.com/gitpod-io/workspace-images/blob/master/.circleci/config.yml

Makefiles

The inferrer actually does something super basic already (i.e. "Is there a Makefile? If yes --> add task init: make)

gitpod/components/server/src/config/config-inferrer.ts

Lines 105 to 115 in 94f3b38

	protected async isMake(ctx: Context) {
	return await ctx.exists('Makefile') || await ctx.exists('makefile');
	}
	protected async checkMake(ctx: Context) {
	if (await ctx.exists('CMakeLists.txt')) {
	this.addCommand(ctx.config, 'cmake .', 'init');
	} else if (await this.isMake(ctx)) {
	this.addCommand(ctx.config, 'make', 'init');
	}
	}

We could probably be smarter here, e.g. if we know common Makefile rule names used for development (e.g. ./configure scripts come to mind, which isn't in Makefiles, but is commonly found next to one).

devcontainer.json [1]

(C.f. linked issue)

Existing Docker configurations

Interesting. I guess we could detect files called Dockerfile (and possibly other common, more dev-oriented names, e.g. dev.Dockerfile if that's a common thing), then try to use that as a base for the dev environment.

Does this always make sense? Would be cool to research this, e.g. anecdotally with a few example repositories.

There are two broad options (for the different configuration files)

1. Directly integrate with the configurations, like the devcontainer standard [1]
2. Map from these configurations to Gitpod configurations (either implicitly behind the scenes, or via explicit suggestion)

My 2 cents are that 2. makes the most sense currently (i.e. generate an explicit .gitpod.yml based on what we know, then let users take it from there -- doing implicit things sounds more dangerous, especially since users have no way to fix our inevitable mistakes).

Longer-term, if there is a standard that emerges, I agree that we should play nice and participate in the standardization effort + natively support the new standard early.

[loujaybee]

Sounds a lot like the project configuration inference @gitpod-io/engineering-meta just shipped, no?

Yep—it definitely relates! The issue originates from some usability conversations, and conversations with/in the community.

As @jankeromnes mentioned, this issue is more on a higher-level than inference for specific config (unless our inference "engine" is somewhat generic / almost plugin-oriented. This issue is more about Gitpod's philosophy on integrating existing scripts. If/when we come to a more concrete decision we can update the issue/community on our stance here 🙏 🚀

Also, all great points/thoughts @jankeromnes ! 🙏

[maphew]

Yes please! I searched for and found this issue because my naive attempt to use "Existing Docker configurations" by simply referencing the project Dockerfile in .gitpod.yml did not fare well, and I lack the knowledge so far to rebuild it for Gitpod land. With that in mind I present the Dockerfiles under https://github.com/OSGeo/gdal/tree/master/docker as fodder for you to consider while working on this feature.

[axonasif]

Hey @maphew, I think you were trying to use the alpine dockerfiles that you have on that repo, please keep in mind that we don't support alpine images as the base for now. You should be good to go with ubuntu ones.

[maphew]

@axonasif I was using ubuntu. I've since figured how to use the published image and then modify it post boot with RUN commands, so I'm not blocked anymore (tho' I still haven't cracked the nut of getting sudo to work passwordless):

• maphew/gdal@4eb513d
• https://github.com/maphew/gdal/blob/4eb513dde5c5c0348b260f632bc6957dbc81bb00/.gitpod.yml

Here's my failed attempt to use the dockerfile in place: maphew/gdal@bcf888d

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.