Fixes #92 - Allow case insensitive like and not like statements.

[jasonaden]

This adds "ilike" and "nilike" so we can do case insensitive searches with Postgres. Fixes #92

[slnode]

Can one of the admins verify this patch? To accept patch and trigger a build add comment ".ok\W+to\W+test."

[raymondfeng]

@jasonaden Thank you for the patch. I would like to generalize the operator to be regexp and nregexp. It should also be implemented by other connectors too. Do you want to give a try?

[jasonaden]

@raymondfeng I've seen the comments floating around for a long time about using regexp operator to generalize. But I'm not sure why we would have to do it with a regexp. It would seem to me that by the same logic we shouldn't have "like" or "nlike" since they are basically a shortcut regexp.

Also, if we use a regexp, doesn't that mean you need to change the regexp based on what connector you're using? Or are we saying that the input would be a JavaScript regexp and we would translate that into something that works across all connectors? Using ILIKE and NILIKE would be a very easy implementation across all connectors (maybe using regex in some, or DB features in others).

I guess my suggestion is to go with something that is useful immediately for a very specific purpose of doing LIKE in a case insensitive way. Yes, regexp is needed as well, but that is much more powerful and actually a different feature.

What do you think? If you don't agree, can you tell me what you think the regexp/nregexp signature would look like for doing a case insensitive text search?

[raymondfeng]

I'm trying to find a good balance between the number of operators, the functional needs, and the connector implementations. If an operator is introduced to the LoopBack query language, we need to make sure all connectors either support it or report a meaning error.

There are a few options:

1. Introduce the ilike/nilike operators as you proposed
2. Extend the like/nlike operators with a casing option
3. Introduce the regexp/nregexp operators. The ilike and nilike is the regexp with /i option.

[jasonaden]

@raymondfeng Okay. My point on regex is that "/i" is not always how you would lowercase a regex. I don't know all the connectors out there, but there are different implementations depending on the database being connected to. I doubt we want to deconstruct JavaScript regexes and convert them to the regex for the database being connected to (just seems to be way outside the realm of what LB should be doing).

Having an operator (or an option) seems like the most straightforward way since it can easily be implemented in any connector. I don't mind making it an option though. Are there other operators that take options like that right now? If so I can easily change the PR to match existing patterns. If not, my vote would be to move this one forward so people can search without being locked into case sensitivity.

Thoughts?

[nicolasembleton]

In my opinion using a regex that is cross-connectors (and actually currently supported by connectors like Mongo) is the way to go. ilike is nice, but it's just a small part of what someone would want to do to trigger simple searches. And PGSQL supports Regex pattern searches. (but to begin with, either one is fine).

[jasonaden]

@nicolasembleton While I would agree that a cross-connector regex option would be great, I think it should be a completely different feature.

@raymondfeng @bajtos Any update on this one? It would be nice to get this moved forward. And to re-iterate the point:

Case insensitive searching is easy to implement across any connector (e.g. determine the most optimal way to do case insensitive searching against a given backend and implement). The code is simple, tests are simple, documentation is simple.

Now take the case of a regex. The current supported connectors are: memory, mongo, mysql, oracle, postgresql, redis, and sql server. What regex are we putting in our code that will work against each of these databases/sources? There isn't one!

The code that developers write to utilize a regex filter will always be database specific. It needs to be a separate feature.

[bajtos]

FWIW, I am personally fine with introducing case-insensitive searching as proposed here. I think it makes more sense to implement the new operator in the base SQL connector though, so that all SQL-like connectors get it for free.

[jasonaden]

@bajtos Can you point me in the direction of that base SQL connector? I can put in a PR there if wanted.

I also have a PR on the juggler: loopbackio/loopback-datasource-juggler#634

[bajtos]

@jasonaden see https://github.com/strongloop/loopback-connector/blob/98eeab1044cd49f67d90d756a49e92d27d4a28b5/lib/sql.js#L709-L714. Please wait with changes until @raymondfeng approves this direction.

[raymondfeng]

@bajtos That's probably not practical as different DBs have different ways to implement ILIKE/NILIKE.

[jasonaden]

@raymondfeng Makes sense.

Thoughts though on moving forward with this rather than going down the regex path?

[raymondfeng]

@jasonaden I just realized that we already support like with i option for mongodb. See https://github.com/strongloop/loopback-connector-mongodb/blob/master/test/mongodb.test.js#L1220. Can you update the patch to align with that?

[jasonaden]

@raymondfeng Great! Yeah, I'll update the PR.

So that probably means we don't need the Juggler change, right (basically, the "options" key will be passed through)? loopbackio/loopback-datasource-juggler#634

[raymondfeng]

No change is required for juggler. The options will be passed to the connector.

[Thr1ve]

Any update on this by chance ?

[raymondfeng]

While we're waiting for @jasonaden to update the PR, @superkhau has started to add the regexp operator for all connectors.

[pandino]

I created a pull request (loopbackio/loopback-connector#29) to allow the loopback-connector module to pass the options to the postgreSQL connector. If it will be accepted, I can implement ILIKE and NOT ILIKE operators that can be called like:

{where: {name: {like: 'Bob', options: 'i'}}}

modifying the buildExpression() function.

The next part may be of interest to #59 :
In our app we also make use of JSONB columns and if it is OK I would like to implement some special operators like ?| and ?&. Because they operate on arrays, I thought to implement them as INQ options.
Any suggestion about the names? Should I use directly the operators like {inq: [], options: '?|'} ?

Other operators are more tricky to implement due to the limited amount of supported operators in the datasourcejuggler dao. The ? operator would be nice but I can't imagine where it can fit with the available operators, except to modify the generic buildWhere() implementation of the sql connector to allow the single db connector implementations to handle the equality operator. So a query like {where: {country: 'Italy', options: '?'}} can generate WHERE "country" ? $1 (This implementation should be backward compatible with the current connectors).

[raymondfeng]

We already have the regexp operator (https://docs.strongloop.com/display/LB/Where+filter#Wherefilter-Regularexpressions). Do we still need this?

[pandino]

I think it would be a nice feature to be able to use advanced features of a database without mining the compatibility of the framework. The regex operator is for sure less performing than the ilike one (and it may be confusing to use to some clients). Moreover, special queries on jsonb cannot be replaced by regular expressions.

[jorgeramos]

+1 on allowing the use of advanced features to pass through

[kronnakrit]

This is my solution at the moment using regexp

var pattern = new RegExp('.*'+ name +'.*', 'i');
Model.find({where: {name: {regexp: pattern}}})
.then(function(){..});

[timgit]

Maybe I'm misunderstanding the proposal, but I think it's as follows.

1. I need to do a like: use the like operator
2. I need to do a not like: use the not like operator
3. I need to do a case insensitive like: use the regex operator with /i flag
4. I need to do a case insensitive not like: use the regex operator with /i flag.

I know this issue has been idle for a while, but I just ran into a wall with option 4 because of case insensitivity. It turns out that it's extremely difficult to implement NOT LIKE in regex for case insensitive exclusions. For example, /^((?!foo).)*$/i finds all items without foo or Foo. I could hide this behind my own abstractions for my users, but it is an inefficient way of going about it because of how regex works.

While I'm most interested in Postgres because it's my data source, I quickly searched Mongo and MySQL to see how they handle this use case. The results are 3 for 3. Each platform implements a negation operator.

1. Postgres: !~ (case sens) and !~* (case insens)
2. Mongo: $not with a pattern (not the same as $not with regexp operator, btw)
3. MySQL: NOT REGEXP

Based on this quick unofficial survey, it would seem valuable to implement some sort of negation operator so it can be implemented across multiple data sources.

Looking over the where filter docs, it seems most natural to implement this as a new operator called nregexp. Thoughts?

[slnode]

Can one of the admins verify this patch? To accept patch and trigger a build add comment ".ok\W+to\W+test."

[duffn]

What is the status on allowing ilike and nilike for this connector? I have a PR working in juggler to add these operators that has been approved by @superkhau, however, the downstream connectors all fail tests because they do not implement these operators.

loopbackio/loopback-datasource-juggler#1091

[slnode]

Can one of the admins verify this patch?

[slnode]

Can one of the admins verify this patch?

[superkhau]

@duffn It looks like from the discussions with @raymondfeng above that they want to pass these capabilities to the connectors. #93 (comment)

I'm now debating what to do with the other PR. Does the regex solution work for you ATM?

[duffn]

Regex is fine for this connector. I have some comments about ilike in general that I think are more appropriate for the loopbackio/loopback-datasource-juggler#1091 PR.