Release Plan 0.10.x, migration instructions and the future roadmap

[wzrdtales]

Hey guys,

First of all, as of today I will release the first beta version of the 0.10.x version. This will follow some releases, described like in the following plan:

• beta A generally stable version, some tests are missing. documentation is lacking details.
• rc1 Providing tests to cover new programable API functionality.
• final release Polishing everything to release the final version.

As you might notice I have moved some features into later version instead and removed some release requirements. Those are not lost, but rescheduled to later releases. The next steps are to provide the missing tests and to enrich the documentation. To note all PRs that are accepted within the beta and rc state are going to be released together in the final 0.10.x version.

The master branch is now pretty ahead of the 0.9.x track, while many things have changed since then, it has kept basically completely backwards compatible. There are some things you need to know about this release, as there are some breaking changes. For example the drivers are not a part of this repository anymore. DB-Migrate now allows everyone to create their own drivers, without actually contributing directly to the project, this is actually not the only benefit of moving them out, but that by side. So now that the changes at the structure have been made I'm going to release the first beta release.

Breaking Changes

• You need to migrate old migrations under certain conditions

As of #400 we're removing the globals async and dbm. This has the effect that your migrations are broken. We will deliver a toolset to automatically fix your existing migrations. Otherwise you could also install async and db-migrate local to your package and your old migrations start working again.

• You need to install the drivers before you can use them

Beside all that you may notice that there is actually a kind of a breaking change. Now that the drivers have moved out of db-migrate itself, you need to install them first to be able to use them.

This can be a global installation if db-migrate is installed and used globally or project wise. For example, if you have installed db-migrate globally and you want to use the mysql driver, you install the following package:

npm install -g db-migrate-mysql

• DBMigrate prefers now your local version over the global one

Another important note is that db-migrate now always prefers your local version of db-migrate. This means if you have installed db-migrate within your project and you call db-migrate within this project, it will recognize this version and execute this one instead. This makes it easier if you want to stick to a specific version on a specific project.

• Old schemas are broken under certain conditions, db-migrate now self injects into migrations

If you're using an old schema that does a require('db-migrate') this still works, but only if you've have installed db-migrate locally or set the NODE_PATH environment variable to the npm global installation path, which I do not suggest to do so. If you have this old migrations you can replace the require of db-migrate. The new migration schema looks like the following, this should be easily adaptable, you basically need to insert the setup function and remove the require, but keep the definition to have the variable available in the up and down function.

'use strict';

var dbm;
var type;
var seed;

/**
  * We receive the dbmigrate dependency from dbmigrate initially.
  * This enables us to not have to rely on NODE_PATH.
  */
exports.setup = function(options, seedLink) {
  dbm = options.dbmigrate;
  type = dbm.dataType;
  seed = seedLink;
};

exports.up = function(db) {
  return null;
};

exports.down = function(db) {
  return null;
};

Future Releases

Future Releases are going to be much smaller packages, this is now much easier that the drivers are separated. The future releases have as of now the following plans:

• Make migrations and seeders zero downtime friendly
• Further working on the seeder suite and enhancing it step by step

The future releases of one drivers also already has plans, I'm going to introduce a mariadb specific driver (maybe using mariasql instead), this will include some of the new features and some of the old but not yet supported such as:

• dynamic columns
• virtual columns
• specifying the algorithm for indexes (hash, btree, rtree)
• ...

--- Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/27492175-release-plan-0-10-x-migration-instructions-and-the-future-roadmap?utm_campaign=plugin&utm_content=tracker%2F73887&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F73887&utm_medium=issues&utm_source=github).

[edorivai]

Hey man. Great work so far! One question:

I'm getting This function "insert" is deprecated and is going to be completely removed, use version controlled seeds instead.. But I can't seem to find any docs on seeder usage. Am I missing something?

[wzrdtales]

You're missing the missing docs about those new things. I'm working on that right now :)

[wzrdtales]

@edorivai To give a quick shot of information here:
insert is "deprecated" if used in a migration, it remains basically the same, but should be called in the future from a vc or static seeder instead.

[edorivai]

Haha, I got that much. Just have no idea how to implement a seeder.

[wzrdtales]

Just use the insert in a migration for now, it is quite easy to transfer those to seeders later, I will come up with some docs and some additions to all this stuff over the next days (lets say monday as I'm quite busy this weekend). I'm currently working on the docs and some enhancements and going to push them the next days.

I'm also working on strategies for users that used db-migrate for a long time and would be in the need to migrate their migrations, this could be a lot of work and this is not what I want.

[edorivai]

Awesome man, keep up the good work!

[RhinoBleu]

Hi, guys !
@wzrdtales did you finish your doc on seeders ?
Thanks for your work man !

[wzrdtales]

Hey @RhinoBleu, actually I have also redesigned them a bit and thought about them again, of what functionality they will provide at start.

The current plan is to deliver first the very basic and essential functionality:

• insert
• remove
• update
• lookup

My current plans are to finish on the tests for them and then push everything up. I also want to rewrite all existing tests, but I think I will move that a bit to the back to get things faster into release.

So the current tasks are:

• finish all drivers to match the rework
• write missing tests for seeders
• create example projects
• bring together docs and example projects (I'm currently also looking in replacing in readthedocs again, as I really hate it that the search function seems to be never fixed for mkdocs, looking into introducing raneto instead)
• write missing docs for programable api
• write missing tests for programable api

Also to give some details about the currently allocated days to db-migrate from my side those will be:
29th January, 30th January, 3rd February, 4th February

[RhinoBleu]

Great ! and thanks for your quick reply.
I'm impatient too see your work =)

[prateekagr98]

Hey!
@wzrdtales did you finish your doc on seeders ?
I'm getting this message
"This function "insert" is deprecated and is going to be completely removed, use version controlled seeds instead."
Thanks for the work dude !

[akshayp]

Thanks for the detailed changelog! I see that you were hoping to get this out of Beta a few weeks ago. Are there any issues you need help with to get the 0.10 milestone out?

[wzrdtales]

@akshayp Sry for the late response, actually the most work left are tests and docs. The greater problem for me is that I have been sick for some weeks which throwed me back and currently also have less time, which is a temporary problem.

If you want to help out on this I would appreciate that. I'm currently not at home and where I am is an incredible bad internet thus I wont give any examples of tasks that are open, I will put this on my list and will look on sunday evening again.

[fcarreiro]

Hi @wzrdtales, I got here by searching the This function "insert" is deprecated and is going to be completely removed, use version controlled seeds instead. message. Is there any plan on updating the documentation? I'm trying to use this package and I'm finding it quite difficult and confusing because of the documentation. For example, some parts use callbacks, others don't. The insert function is used. The update function is used but it doesn't exist anymore. I found out (by looking at the code) that I can use promises instead of callbacks but that is not documented. I couldn't find any information about seeders, etc.

Don't get me wrong, I value your contribution, and this package is very useful! I felt like contributing to the docs myself but I feel that I would have to read all the code and understand it to be able to fill in the blanks. The gap is just too big.

[wzrdtales]

@fcarreiro No worries, I don't get you wrong on that :)

And yes, there is a plan to extend and rewrite a whole bunch of the docs. I would highly appreciate if you want to contribute to the docs though, but I understand that this would be hard without understanding many things. But just feel free to throw questions at me and I will respond as soon as I get a free minute,, which is usually at the same day.

[fcarreiro]

Great :) Could you maybe give a small description of what a seeder is and a small example on how to use define/use it? I've seen the interface defined in the code but I don't know what it is supposed to mean, conceptually, in this framework... let alone how to use it ;)

[wzrdtales]

how to use define/use it?

There are some docs that theoretically describe the usage, but for now you won't really be able to use them. I have also postponed them to a later version as I have decided to change the release strategy of db-migrate. I will go for a release cycle of 1 month instead of currently feature oriented.

The last thing I now needed to do was completely define the API for the state of now, this is done though and rewrite all tests to a new testing framework. This is also done apart from the driver tests. So next step is moving the driver tests out to their drivers instead, after that I would place that version as the first release candidate. Next topic for the road would be the docs and test coverage for the new API endpoints.

I will come back later with the conceptual point of your question, so don't you wonder that I left that out, but don't have the time right now :)

[wzrdtales]

#400 Is an additional breaking change, but with a migration plan

[DjOli-zz]

Hey @wzrdtales, has there been any updates on this? Have been ignoring this problem since starting to use db-migrate late last year in the hope that some documentation or instructions would show up to aid with the "This function "insert" is deprecated and is going to be completely removed, use version controlled seeds instead." error. I'm happy to wait for your rework as it sounds like it's bigger than Ben Hur, just thought it was worth chasing up. Thanks :)

[jonkofee]

@wzrdtales, I look forward to

[fcarreiro]

Since I don't use mongodb, I ended up using knex which has nice migrations.

[wzrdtales]

fcarreiro Since I don't use mongodb, I ended up using knex which has nice migrations.

As your comment does not make any sense at all, as db-migrate is not at all about mongodb it just happens to support it, it is clear that this is just a very bad way of advertising though. Just to let you know, this type and way of advertising is generally viewed as a very ugly move. It is like moving into the google headquarters and telling everyone how awesome yahoo is b/c you don't use google, b/c you do not search for google on google. So you're harming the knex project and yourself with this.
As I am strictly against censorship, I won't delete this, but keep it here like it is.

To make a statement here, that actually makes sense, unlike yours:
To everyone reading this: Knex is a completely different thing, compared to db-migrate. Db-migrate is a migration library, while Knex is a query builder at first that happen to include migrations. Knex is not a bad product though, if you want to try it, give it a shot though.

So to come back to the important stuff here:

@djoli Sry that I came to you this late, I wanted to answer a bit earlier but this got lost somehow. Basically the things I want to finish are not that much, actually if I would have just a full day of time allocated to db-migrate I would be able to push everything out. So why am I moving so slow again here. The reason is quite simple, I am involved in multiple other projects that also demand my time and I have to take time for every of this projects. Some time ago, I would say a good year ago, I have had way more time in general, the truth currently is, I thought I would have more time right now, but I don't. Under the week my day to day work often result in days with up to 16 hours of work. There is not that much space left, so what is left, is actually the weekend. My spot to invest in all the projects currently running, which I do, but unfortunately I can't work on all of them at once. This is also the reason I'm always more than happy to see people contributing to db-migrate, as this is the thing that makes open source actually awesome! For this reason I try to always take time to read PRs and issues and answer on them as fast as it is possible for me. I used to have an alert on my mobile phone for issues, but I have resetup my phone a while ago, so my usually near to instant answers are not that likely anymore. Putting that on my notes to resetup this setup again :)
And just a side note, do not fear to ping me again, if you don't receive an answer, it is not intentional and might be possible that this got lost in space though...

So far, so good,
Have a great weekend!

[fcarreiro]

Hi @wzrdtales, I see that I made you angry. That was not the plan, but I understand why. Let me explain a bit more why I did what I did.

First of all, I want to explain that I am not related to knex. I'm not a dev, I just happen to be using it in some projects.

Second, my point about "not using mongodb" is that knex does NOT support mongodb, and db-migrate does. Therefore, as I'm not using mongodb, knex is an option and I ended up using it.

Why did I even mention knex? Which, as you say, it serves a different purpose, but happens to have a migration system. The reason is: I tried to use db-migrate and I partially succeeded. There were some warnings and the "real" way to use it was undocumented. The documentation is not very complete. The roadmap is unclear. I was not the only one here asking for clarifications. And that was a year ago. As far as I could see, this project is not being actively developed nor supported.

I understand that you do it as a side project, and that you actually have no obligation whatsoever to provide support or to even code a single line of code. My comment was supposed to be a warning for people that come to this thread, opened in 2015, saying "Look, things are going slowly here, you might want to consider this other project as well."

Does it make more sense now?

I appreciate your non-censorship approach.

[wzrdtales]

@fcarreiro It did not make me angry that you posted about knex here. The awesome thing about open source is the diversity, ultimately driving software into the right direction, b/c there actually exist diversity.In fact, the way you did it, wasn't the nice one. If you would have told the people, maybe you can look into TJs migration framework or into knex if it can cover your use case as of now, this would have been totally OK.

But to get to your points, yes I agree:

As far as I could see, this project is not being actively developed nor supported.

Actively developed, well not as much as I wish. So absolutely, you're right on this. While it is still supported though :)

And yes it makes more sense now, as stated above, I had a problem with the way, not that you mention something else, although I would rather emphasize contribution though. Another thing, I would naturally be interested what does partially succeeded mean for you? In fact the warnings are all about new stuff. If you go prior the v0.10.x version you won't have any warnings though.

[fcarreiro]

You are right there. My way of stating it was not nice. It was a mix of (1) a bit of frustration, your project was the first migration tool I used and I had hoped to see it moving (2) as there hadn't been much discussion in this thread I honestly thought you had "dropped" the project and therefore wouldn't care too much (sorry!).

In any case, that's not an excuse ;)

I really don't remember very well which problems I had. From the top of my head I remember something about the global variables (dbm, seeds) not working/being undefined (or maybe it was actually the local variables in the up/down functions?). I also got a bit frustrated using the functions in the docs and getting a warning but then I couldn't find out how which was the right way to do things. There might also have been something about seeds, which I couldn't do. But honestly I don't remember right now.

[wzrdtales]

mh, I guess seeds could be. The global variables are being removed in v0.10.x, but this shouldn't be a problem since those are delivered via the setup method.
About the deprecation warnings. Yes, from a view of today it was maybe not the best idea to place them there at that time. But at the time I have placed them, I had planned to push out the seed feature just in a few weeks and shortly after that many things changed for me rapidly (which is not a bad thing, but my time schedule drastically changed). So it would actually make sense for the v0.10.x version to remove the deprecations again and place them as soon as the real deprecation cycle starts. If you remember anything, I would be happy to know.

In the end I am very happy that this ended up in an exchange of opinions :)

[wzrdtales]

They are being removed by the way b/c db-migrate aims to be used although programatically, where global variables are a no go.