Proposal: integrating efforts

[dsebastien]

Hello,

I've seen @IgorMinar 's presentation of angular-cli & @pkozlowski-opensource talk about ES2015, modules, System, JSPM etc while at AngularConnect in London and found the idea of Angular CLI to be pretty similar to 2 small projects I have created recently:

• https://github.com/dsebastien/modernWebDevBuild
• https://github.com/dsebastien/modernWebDevGenerator

Since @IgorMinar mentioned that community participation was welcomed, here I go! :)

I don't know how far you guys are willing to go with Angular CLI and how prescriptive you want to be, but maybe some ideas below would be worth discussing.

Here's an overview of my projects:

ModernWebDevBuild is based on Google's Web Starter Kit (heavily tweaked/enhanced) and can be used to:

• transpile ES2015 to ES5 w/ sourcemaps
• transpile TypeScript to ES5 w/ sourcemaps
• transpile SASS to CSS w/ sourcemaps
• check JS/TS code style & code quality (requires JSCS/JSHint/TSHint config files to be present)
• serve the app with support for multi-device sync (using BrowserSync) and watch tasks that constantly do all the above without stopping when an error occurs (continuous flow for the developer with useful feedback)
• generate production bundles for CSS/JS/HTML/images
• serve the production build (without watch)

ModernWebDevBuild has minimal requirements regarding the project structure and the presence of some files, but I think this is positive as it provides a good starting point for larger projects.

There are some major features still missing such as testing integration, configurability & extensibility, but I think that it is already useful as is.

ModernWebDevGenerator is a Yeoman generator that integrates the use of ModernWebDevBuild (which is just one npm dep) and generates a working project starting point. It integrates some code organization/design guidelines for structuring the application components, pages, etc in a logical manner while considering componentization as first-class concept by regrouping all pieces of a given component together (e.g., component class, component styles, component tests, ...). Given that the build is flexible enough, it doesn't prevent users from reorganizing as they wish.

Both projects are pretty opinionated thus I suppose that there are choices in there that you wouldn't impose on your users (e.g., Gulp, SystemJS, JSPM, SASS, ...), but maybe the feature set and/or necessary extensibility could be integrated within Angular CLI and/or Angular Toolkit?

I think that:

• having a recommended project structure & organization guidelines is beneficial at many levels
• having live reloading with multi-device synchronization support is great
• having continuous transpilation, code compilation/quality/style feedback (without breaking) is awesome for developer productivity
• having a "standard" way of building projects (from dev to prod) is tremendously useful for larger projects and integration with CI & CD solutions
• having an extension system to let the community cater for its own needs is a must (e.g., CSS copy by default and a SASS plugin for those who want that)

Most importantly, I think that having a Google-backed basis for all this, with enough anchor points for extensions would be awesome for larger projects requiring more than Plunker :)

By having this one central piece in the Angular "build puzzle", it might be easier to make related projects evolve in the good direction; for example for now TypeScript can look up modules in node_modules but is totally incapable of doing the same with jspm_packages, although that could be really nice.

[dancancro]

Hi @dsebastien

I collect information about projects like this to help people better understand the differences between them. If you or anyone else is interested in shedding light on what distinguishes your project or angular-cli, please visit this site to do a questionnaire about it. I refer to things like your project and angular-cli as "Application Starters" for lack of a better word. "Stack" and "generator" are probably more familiar. The questionnaires are really long so don't feel compelled to finish them.

From data collected through questionnaires and other sources, the system generates head-to-head trade-off reports between pairs of technologies. You can view those here. While viewing a report if you see any errors, you can submit corrections using forms in the report. So that's another way to share what you know.

[IgorMinar]

@dsebastien thanks for reaching out and sorry about the delay.

My plan is take some time after Angular 2 beta is out to write down all my thoughts on all things cli.

In the meantime just briefly: There are reasons why we decided to try the ember-cli path rather than yeoman path. The main one is that yeoman is a scaffolding tool that can be customized via generators, while ember-cli is a platform/ecosystem of addons that complement each other and can work in unison. Achieving the same with yeoman is pretty tricky because it was never designed for this.

[dsebastien]

Hello @IgorMinar. Indeed, ember-cli seems a much better fit for extensibility than Yeoman; I'm currently only using it for the initial project creation. My current plan is to create a separate (global) package to act as CLI, which could also help me avoid the need for systematically installing 100+ dependencies in the project (which takes forever with npm).

I'll try and help when there are more concrete plans for angular-cli. Ultimately I prefer to have my needs covered either through angular-cli or extensions thereof than maintain my own isolated island :)

[IgorMinar]

Awesome. I'll write up my thoughts as a design doc or a blog post and will
link it here. After Angular 2 beta.
On Sun, Nov 22, 2015 at 11:55 PM Sebastien [email protected] wrote:

Hello @IgorMinar https://github.com/IgorMinar. Indeed, ember-cli seems
a much better fit for extensibility than Yeoman; I'm currently only using
it for the initial project creation. My current plan is to create a
separate (global) package to act as CLI, which could also help me avoid the
need for systematically installing 100+ dependencies in the project (which
takes forever with npm).

I'll try and help when there are more concrete plans for angular-cli.
Ultimately I prefer to have my needs covered either through angular-cli or
extensions thereof than maintain my own isolated island :)

—
Reply to this email directly or view it on GitHub
#49 (comment).

[SBoudrias]

Hey, a note about Yeoman.

Even though you want to go the angular-cli way rather than combining tools through a generator (and that's fine), Yeoman is an easily embeddable tool.

Your community would really benefit from still having the scaffolding parts of your cli tools done by Yeoman. If published both as the cli tool and a generator, you'll allow other author to extend and compose the angular2 generator on top of different backend system (for example generator-angular-fullstack) or with differents styling frameworks, etc etc.

Integrating a yeoman generator in your own cli is actually a very easy thing to do: http://yeoman.io/authoring/integrating-yeoman.html - I'm sure @eddiemonge, @Swiip, @DaftMonk and @kingcody would be happy to give you a hand to get an angular2 generator bootstrapped.

[Swiip]

Yeoman community is definitely already working on generators for Angular 2. @zckrs and I already have a walking skeletton of an Angular 2 generator working both with Webpack of SystemJS / JSPM: generator-fountain-angular2. It still miss the switch between Babel or TypeScript but it’s on what I’m working right now.

I think Yeoman is the best platform to address seed projects and component generation, it’s designed for that and offers lots of built in tools to do it efficiently like inquirer or mem-fs-editor.

Yeoman is perfectly capable to handle a coherent ecosystem of generators. I think we prove it through the very high number of options in generator-gulp-angular. But there is more, there is dedicated features for that in Yeoman like generator inheritance and composition. In fact FoutainJS is exactly based on this concept and is already capable to compose 12 generators to generate very differents apps. (README, DESIGN)

Finally, I must admit I’m not a fan of the "framwork-cli" approach. It implies too much that you have to use the framework tools to use the framework and doesn't promote community. The best way to start an Angular 1 project as always been to use Yeoman and that’s great. The Yeoman community created a lots of great project (like mine I hope but also :) angular-fullstack and jhipster.

[dsebastien]

The scaffolding part of the story is nice and necessary to quickly get started but it isn't enough.

I think that front-end apps need generic & pluggable build systems that cover the whole spectrum from transpilation to quality/style checking, through testing, code coverage checks etc.

Having a project generator drop a kilometer-long gulp file that won't be kept up to date/versioned just means that each project will get a different build evolving differently in each project.. On the other end it just sounds just plain stupid to go about and create project-specific gulp tasks that more or less do the same, but it feels like it's where the front-end world is still at nowadays.

IMHO, the build itself should be kept separate from the actual projects and be versioned like any other dependency; this is why I've created the "modernWebDevBuild" project. The build should be flexible enough to support different project structures and to allow for different tools to be used (e.g., sass vs less, angular vs react, ...).

I dream of a tool for the front-end world that would be a mix between a scaffolding tool and a build tool (let's call that a front-end CLI). Such a tool should be powerful & pluggable enough to be both generic (i.e., support a common set of tasks) & support framework-specific tasks (e.g., like angular-cli which provides an easy way to scaffold components, filters).

My feeling is that angular-cli is nice for the Angular world and will be cool, but that in the long run, the whole community would benefit much more from a more general solution.

Of course it's easier to say than do ;-)

[Swiip]

I work with seed projects for a while now and I'm still convinced that the build process is THE thing in web app dev where people needs to get started with a good configuration but can't be abstracted because people has too many different needs.

Grunt tried and results in programming with JSON, Webpack, I fear is doing the same today (even if I use it because it's very powerful). Before that Maven tried and failed too (IMHO). It's why I'm so much tied to Gulp because you don't configure your build, you develop it with code.

It's also why someone like me who really believes in abstractions in development is working on a code generation tool. Because I'm still convinced people should start with a good tooling environment but has to understand it and modify it when the project grows.

If you want to hide everything in a lib or a generic cli, just take Webpack or Brunch but I think you'll end up with a JSON nightmare like many others.

[SBoudrias]

@dsebastien my point really was that Yeoman is pluggable. Angular-cli will need a scaffolding tool, and you shouldn't create your own; instead just wrap Yeoman.

Then I was saying the scaffolding part should be published as it's own composable generators so it can benefit the whole community if they wish to add and bind it on top of specific backends systems.

I totally get the point of going with a tool specific cli tool - but I just want to make sure you're not losing what the Yeoman community already provide for angular users. IMO that's a mistake the ember team did.

[Swiip]

@SBoudrias just helped me to focus on the point. I lost myself in philosophical thought :)

Yep, as a user of Yeoman API: it's a great scaffolding tool and you'll benefit both of it's API and community.

[dancancro]

@SBoudrias I believe the plan is to adopt the ember blueprints approach to scaffolding. Also for what it's worth, here are some notes from discussions and Google hangouts that were had around the goal of uniting the different projects and among other things producing a CLI.

[NathanWalker]

@Swiip Due to what I encountered here:
#222

Curious if yeoman has or is planning on providing similar ability to generate files with more flexibility around the location. The benefit to the file generation is you get (.css, .html, .ts, and .spec) scaffolds which is very helpful. In addition, I think it would be nice to be able to point a generator at custom templates so if your application used custom decorators for components lets say, the component generate would use the component template you specified for your application 👍

[Swiip]

In generator-gulp-angular we propose to configure the main directories (https://github.com/Swiip/generator-gulp-angular/blob/master/docs/usage.md#yo-options) but still, the template app inside is quite fixed. But most of the users are looking for a directory structure.

In FountainJS, we want to propose at least 2 options: a minimal hello world where there is no structure just the minimal configuration and a more complete example which can help juniors to find their start.

[NathanWalker]

Thanks for the info @Swiip I'll look forward to that and continue further discussion over on the Fountain repo.

[gruppjo]

Hi everybody, I read this conversation with great interest and I usually just lurk on topics like these, but with this one I really want to get in on it and share my experiences with you, because I feel it might be very beneficial for the discussion.

Background

I'm the initiator and maintainer of Generator-M-Ionic. It's the result of more than 2.5 years of experience of me and my colleagues with building enterprise-scale apps with Angular 1 and Ionic/Cordova for the company I work at.

Requirements for front-end tools

As I said, I carefully read this discussion and from what @dsebastien, @SBoudrias, @NathanWalker and @Swiip wrote, I assembled this list of requirements for front-end tools and augmented it with my own. There may be others, but I'm quite sure that this is what covers about 90% of community use-cases. Sometimes I give examples of which features these requirements correspond to features in our generator.

1. easy project generation for new developers/employees so you can start coding right away after answering a minimal set of questions
2. recommended project structure, technology stack & guidelines so you don't have to figure out everything on your own at the beginning
3. having a "standard" way of building/delivering projects (from dev to prod)
   • including code minification/sourcemaps, quality mechanisms (i.e testing using Karma/Jasmine and Protractor, style-checks using ESLint etc...) and easy integration into common CI/CD (i.e. Travis/Jenkins) environments or other distribution platforms (i.e Ionic Platform/CLI, ...)
4. create new components throughout the development process that are then automatically integrated and ready to use
5. Be extensible, having technology recommendation rather than restrictions and allow the community to cater for their own needs
   • using/setting up 3rd party libraries, plugins (i.e other angular modules, ...)
   • make technology choices (i.e. CSS/SCSS, different templating like jade, ...)
   • modifying out-of-the-box behaviour (i.e. file structure, build process, coding-guidelines like ESLint, ...)
6. individual components can be independently updated while keeping potential custom-extensions (e.g. custom coding guidelines with ESLint)

Our story

We built all of that (well almost). Using yeoman and gulp.

Why? Because the Ionic CLI did not at all cater our needs and there was no way of extending it. It completely lacked requirements 2 through 6. I even wrote a blog article on our company blog about it (in case you are interested in some of the details). I don't want to brag here, that's not my intention. I just want make a point on what's possible. We built a powerful tool, so for the sake of not making this longer than it already is, I assembled a more extensive feature-gist for anyone interested.

I said built almost all of it. Here are the things that are tricky and we're still working on:

• modifying out-of-the-box behaviour
  • modifying core features like the build process is itself easy. You just augment the gulpfiles. However in conjunction with requirement (6) there is no automatic way of doing so without loosing updateability of the project.
• file-structure adaption and integration with the Ionic Platform is on the way
  • side note: we built a generator because the cli doesn't serve our purpose and now we're integrating it back in because aparat from the insufficient scaffolding they're now offering quite handy tools for working with their delivery platform. See how weird this gets, when there's no proper separation of concerns?
• updateability of individual parts (like gulp tasks for building) or the project as a whole

Some more stuff we're working on:

• Typescript/ES2015 compilation as an upgrade/transition path to an angular2/ionic2 stack
• device live-reloading

To be fair: downsides I see with Yeoman/Gulp or any other task runner:

• gulp hard to test workflows
• hard to test generated files
  => increased maintenance (how do you do that?)

Why no CLI?

@IgorMinar and everyone else in the angular-cli team, please don't make the same mistakes: I already explained, from our point of view the Ionic CLI is missing essential features to meet our standards for professional software engineering. I absolutely don't doubt that you guys will be able to do an enormously good job on requirements 1 through 4. However just like the Ionic CLI I see great downsides of your approach in requirements 5 and 6:

• there is no way of extending without directly changing the core.
• the CLI wraps everything and as a result the users stay uneducated. It's a black-box.
  • They don't know what's behind a build, behind running tests. They might get totally dependent on your tool unless they explicitly want to get involved.

I think this in this areas Yeoman just trumps, because:

• if there had been a good, adaptable ionic generator developed by the core team, we could have just built on-top of that using Yeoman's compability feature and add our stuff
• everything is in your project once you set it up, all the files, all the tasks (whether gulp or anything else). No magic. You can simply modify, extend, adapt them.
• when you make changes, it's super easy to get a discussion started and a PR issued since you know the code and already have a working implementation
• you just wire the tools together that you need and it's transparent that you are using those rather than hiding them behind a rainbowy magic cloud. For instance Gulp for live-reloading and building or the Cordova CLI to deploy to the device. Because you effectively type gulp build and not cli build. Just an example, this is true on many more levels. With the Ionic CLI for instance many users don't even know what Cordova is.

I think this can have a great impact on the whole community, how educated they are and how quickly they reach professional development skills and how quickly they may start contributing.

angular-cli

So to close everything off, since we're trying to make our switch to angular2/ionic2 soon, I have some questions:

• Do you plan to provide hooks for developers to extend upon your cli or will we just be stuck with the decisions you make?
  • For instance, can I modify how a build is performed? For instance maybe I want to inject build parameters during a build into the app, how do I do that?
• How do you plan to cope with the black box CLI phenomenon?
  • I'll predict a lot of users will ask you to implement features for them rather than being able to contribute

Conclusion

I heavily agree with @Swiip and @SBoudrias on many things they said, and I do so from my own extensive experience and the one of my colleagues that I really hope I achieved to explain in a comprehensive fashion. I'm not trying to discredit any work of anybody here, if somebody feels I did so, I apologise. I wrote this with the sole purpose of contributing valuable experience to the topic.

[filipesilva]

Hey @gruppjo!

First off, I want to apologize for not replying to this earlier. It's been pinned on my inbox for ages now, but I've delayed putting up a response because I wanted to be sure I could address all of your excellent points. At the time of your post some of the intended design was still being fleshed out.

I do reply as my own voice though, rather than an official response of the CLI team.

So, out of your list of requirements, I feel pretty confident that 1 to 4 are well on their way to being ready. You're absolutely right that 5 and 6 are both the hard and important, and they've been consistently part of every conversation the team has had. We know the community needs it.

We're gunning to have 5 and 6 in the CLI but as far as the beta is concerned, they aren't the highest priority. We're still trying to get what we can in though... for instance @jkuri just put in #388 that bakes in CSS preprocessors. It's not the same as extending, but in the meanwhile people can still use preprocessors.

We're also looking at adding a config that allows users to customize file structure, because that'll be very relevant when the style guide comes out. Linting and formatting are currently completely customizable - ng lint simply runs npm run lint, so you can change that script.

Other things aren't extensible at all currently, like the build process. Unit testing is also tightly coupled with the build process, because of the TS compilation step and watch modes.

So currently we do have to make the trade off, and the priority is points 1 to 4 as fast as possible. But let me be clear on this: extensibility and configurability are well on the radar after those. There hasn't been a single week were we haven't talked about it.

The black box phenomenon..... not as easy to avoid, at least at the moment. Part of it is the tight coupling between some commands, part of it is the ember-cli architecture in which logic is part of commands and tasks at the addon level.

The addon architecture is also part of the solution though: users could create addons (like https://github.com/IgorMinar/angular-cli-github-pages) and add them to projects. But even that functionality is currently being rolled into the base CLI (#366) at the moment, mostly for more rapid iteration and not having to manage a second repo. Stuff like this is meant to be factored out into extensible bits in the future.

In this scenario users would still need to get involved in the CLI to be able to alter it's behavior... it's hard to balance between putting config logic on the repo and still have deep integration between all the moving parts.

So yeah, that's what I have to say. I hope I was able to address your points.

[NullVoxPopuli]

They don't know what's behind a build, behind running tests. They might get totally dependent on your tool unless they explicitly want to get involved.

• @gruppjo

There is nothing wrong with that. Imo, even caring about the details of all that just get in the way of development... I want to be productive, not configure.

[filipesilva]

Closing as obsolete.

[gruppjo]

Hi @filipesilva!

I really hope I'm not getting on your nerves. If so, I'm really sorry! I still think it's an important and constructive topic, so why do you consider this as obsolete?

I really like how far the CLI has come. Kudos! I happily tried out the component based CSS/SASS, AOT with lazy loading and e2e and unit tests and was pleasantly surprised. The tasks run quickly, the build size is really good with AOT, tree-shaking, minification and gzipping, there's a task for copying static assets and a basic environment system.

BUT I won't be able to use it for our project. Again. Just today I found myself ditching the angular-cli in favor of a different system. Why? Because there are crucial requirements for our project that we cannot fulfill using the CLI and there doesn't seem to be a way of extending it's behavior!

Just to give you two feasible examples:

• Dynamically import a stylesheet depending on the environment #4685
• [Feature Request] Change place where css bundles are injected in index.html or let us specify a media on the angular cli json configuration file #4220

I feel like having a déja-vu: As I suggested almost a year ago, we could easily take care of this ourselves if the whole webpack task system of the CLI was part of the project and not black-boxed and hidden away in the CLI itself.

Really looking forward to hearing your thoughts.

Sincerely and respectfully,
Jon

[filipesilva]

Heya @gruppjo, of course you're not getting on my nerves, these topics are here to be discussed :D

This specific issue is closed obsolete because the CLI architecture and feature set are now frozen for 1.0, and an addon architecture has already been announced as being a major 2.0 feature.

There isn't an issue open to currently track the addon functionality though, one will be open when work starts on it.

There's also ongoing work for an "eject" functionality for 1.0 in #4680.

[angular-automatic-lock-bot]

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}