Caching Question

[jasonmacdonald]

Perhaps I'm misunderstanding how caching works, but clearing the cache using maxAge doesn't seem to work. I expected a defineResource like below to completely clear the cache and require a service call after 1 second.

DS.defineResource({
            name: 'media',
            idAttribute: "id",
            endPoint: "media",
            maxAge: 1000,
            deleteOnExpire: "aggressive",
            cacheFlushInterval: 100,
            recycleFreq: 100,

            // This on expire never seems to get called
            onExpire: function(key, value){
               console.log("Removing id " + key  + " from cache", value);
            }, 
        });

When I test this the data still seems to come from the data store and the onExpire method never gets called. Am I missing something or misunderstand how this is supposed to work? The docs aren't very clear about caching.

The only thing that seems to clear the cache is EJECT.

[jmdobry]

See this line, it explains why your onExpire isn't called (not documented very well, sorry). Try verifying with something like the following:

``js
DS.inject('media, { id: 1, test: 'test' });
DS.get('media', 1); // { id: 1, test: 'test' }

$timeout(function () {
DS.get('media', 1); // what do you see here?
}, 2000);

[jasonmacdonald]

I see, so it's not a callback, or rather, you can't override what's there as you can when using angular-cache directly. I misunderstood the documentation I guess.

Aside from that though, using the timer it still appears that we are grabbing stuff from cache and not refetching from the server. I should note that I'm doing this using "find" not "get". Perhaps that's the difference? I'm still a bit confused why there's "find" AND "get". I'm using find because I don't want to always have to append "loadFromServer" to every call. Again, I'm sure I'm doing stuff wrong here. What are the use cases for Find vs Get?

Here's the test I'm running using my local Node server as a REST service for media.

DS.defineResource({
            name: 'media',
            idAttribute: "id",
            maxAge: 1000,
            deleteOnExpire: "aggressive",
            cacheFlushInterval: 100,
            recycleFreq: 100,
            endPoint: "media"
        });

        var promise = DS.find("media", 1).then(function(e){
            console.log("Gets media from server");
            setTimeout(function () {
                //DS.eject("media", 1);
                console.log("should get from server");
                DS.find("media", 1).then(function(){
                    console.log("Nope, got it from storage");
                });
            }, 10000);
        });

My expectation is that after 10 seconds, the internal call to find inside the promise callback should refetch from the server, but it does not. However, if I uncomment that DS.eject, it does.

Tracing the onExpire function you pointed out earlier it seems it is never hit. :(

[jmdobry]

Change your test to:

DS.defineResource({
    name: 'media',
    idAttribute: "id",
    maxAge: 1000,
    deleteOnExpire: "aggressive",
    cacheFlushInterval: 100,
    recycleFreq: 100,
    endPoint: "media"
});

var promise = DS.find("media", 1).then(function(e){
    console.log("Gets media from server");
    console.log(DS.get("meda", 1)); // should print your media object
    setTimeout(function () {
        console.log(DS.get("media", 1)); // should be undefined
        //DS.eject("media", 1);
        console.log("should get from server");
        DS.find("media", 1).then(function(){ // should make another GET request, check the network tab
            console.log("Nope, got it from storage");
        });
    }, 10000);
});

And tell me what you get.

[jasonmacdonald]

THanks for the quick reply. Get is returning undefined, but there's still no rerequest to the server.

Here's the console log

GET request:http://localhost:3000/media/1 Time taken: 46ms  angular.js:10318
Gets media from server mediaModel.js:19
{object}
undefined mediaModel.js:22
should get from server mediaModel.js:24
Nope, got it from storage

This screenshot may help http://imgur.com/VpQv57g

[jmdobry]

What version of angular-data are you on?

[jmdobry]

Explanation of sync vs async methods:

async	sync
These methods operate through an adapter (http, localstorage, etc.), and persist the result of their actions to the in-memory datastore	These method operate on the in-memory datastore only
find	get
findAll	filter
create	inject
destroy	eject
destroyAll	ejectAll
save	n/a
update	n/a
updateAll	n/a

Example:

DS.get('user', 5); // undefined
var p = DS.find('user', 5);
DS.get('user', 5); // undefined

// async operation completes after some latency
// If you're using the http adapter (default), then
// a GET request is performed
// the result of the request is injected into the datastore
// and the user with id of 5 is available synchronously now
p.then(function (user) {
  user; // { id:5, ... }
  DS.get('user', 5); { id: 5, ... }
  user === DS.get('user', 5); // true
});

And here's another test to try:

DS.defineResource({
    name: 'media',
    idAttribute: "id",
    maxAge: 1000,
    deleteOnExpire: "aggressive",
    cacheFlushInterval: 100,
    recycleFreq: 100,
    endPoint: "media"
});

var promise = DS.find("media", 1).then(function(e){
    console.log("Gets media from server");
    console.log("item in store:", DS.get("meda", 1));

    // internal stuff, you wouldn't normally access this
    console.log(JSON.stringify(DS.store.media, null, 2));
    console.log("item in cache:", DS.store.media.index.get(1));

    // Change this to $timeout
    $timeout(function () {
        console.log("item in store (should be undefined):", DS.get("media", 1)); // should be undefined

        // internal stuff, you wouldn't normally access this
        console.log(JSON.stringify(DS.store.media, null, 2));
        console.log("item in cache (should be undefined):", DS.store.media.index.get(5));

        DS.find("media", 1).then(function() {
            console.log("Nope, got it from storage");
        });
    }, 10000);
});

And tell me what you get.

[jasonmacdonald]

What version of angular-data are you on?

Using bower, version is 0.10.5

GET request:http://localhost:3000/media/1 Time taken: 7ms 
[Object]
 angular.js:10318
Gets media from server mediaModel.js:19
item in store: 
Object {id: "1", guid: "880ebac8-eb2e-4319-bfd3-64caf6fb765e", isActive: true, picture: "http://placehold.it/120x120", latitude: 54.215578…}
 mediaModel.js:20
{
  "collection": [
    {
      "id": "1",
      "guid": "880ebac8-eb2e-4319-bfd3-64caf6fb765e",
      "isActive": true,
      "picture": "http://placehold.it/120x120",
      "latitude": 54.215578,
      "longitude": -33.097056,
      "tags": [
        "incididunt",
        "et",
        "ullamco",
        "deserunt",
        "ea",
        "labore",
        "et"
      ],
      "encodings": [
        {
          "id": 0,
          "type": "occaecat"
        },
        {
          "id": 1,
          "type": "consectetur"
        },
        {
          "id": 2,
          "type": "dolor"
        }
      ]
    }
  ],
  "completedQueries": {
    "1": 1408290840241
  },
  "pendingQueries": {},
  "index": {
    "$$data": {
      "1": {
        "key": "1",
        "value": {
          "id": "1",
          "guid": "880ebac8-eb2e-4319-bfd3-64caf6fb765e",
          "isActive": true,
          "picture": "http://placehold.it/120x120",
          "latitude": 54.215578,
          "longitude": -33.097056,
          "tags": [
            "incididunt",
            "et",
            "ullamco",
            "deserunt",
            "ea",
            "labore",
            "et"
          ],
          "encodings": [
            {
              "id": 0,
              "type": "occaecat"
            },
            {
              "id": 1,
              "type": "consectetur"
            },
            {
              "id": 2,
              "type": "dolor"
            }
          ]
        },
        "created": 1408290840241,
        "accessed": 1408290840241,
        "expires": 1408290841241
      }
    },
    "$$id": "DS.media",
    "$$storage": null,
    "$$expiresHeap": {
      "heap": [
        {
          "key": "1",
          "value": {
            "id": "1",
            "guid": "880ebac8-eb2e-4319-bfd3-64caf6fb765e",
            "isActive": true,
            "picture": "http://placehold.it/120x120",
            "latitude": 54.215578,
            "longitude": -33.097056,
            "tags": [
              "incididunt",
              "et",
              "ullamco",
              "deserunt",
              "ea",
              "labore",
              "et"
            ],
            "encodings": [
              {
                "id": 0,
                "type": "occaecat"
              },
              {
                "id": 1,
                "type": "consectetur"
              },
              {
                "id": 2,
                "type": "dolor"
              }
            ]
          },
          "created": 1408290840241,
          "accessed": 1408290840241,
          "expires": 1408290841241
        }
      ]
    },
    "$$lruHeap": {
      "heap": [
        {
          "key": "1",
          "value": {
            "id": "1",
            "guid": "880ebac8-eb2e-4319-bfd3-64caf6fb765e",
            "isActive": true,
            "picture": "http://placehold.it/120x120",
            "latitude": 54.215578,
            "longitude": -33.097056,
            "tags": [
              "incididunt",
              "et",
              "ullamco",
              "deserunt",
              "ea",
              "labore",
              "et"
            ],
            "encodings": [
              {
                "id": 0,
                "type": "occaecat"
              },
              {
                "id": 1,
                "type": "consectetur"
              },
              {
                "id": 2,
                "type": "dolor"
              }
            ]
          },
          "created": 1408290840241,
          "accessed": 1408290840241,
          "expires": 1408290841241
        }
      ]
    },
    "$$storageMode": "memory",
    "$$storagePrefix": "DS.media",
    "$$prefix": "DS.mediaDS.media",
    "$$disabled": false,
    "$$capacity": 1.7976931348623157e+308,
    "$$deleteOnExpire": "aggressive",
    "$$recycleFreq": 100,
    "$$recycleFreqId": 14,
    "$$maxAge": 1000,
    "$$cacheFlushInterval": 100,
    "$$cacheFlushIntervalId": 15
  },
  "modified": {
    "1": 1408290840241
  },
  "saved": {
    "1": 1408290840241
  },
  "previousAttributes": {
    "1": {
      "id": "1",
      "guid": "880ebac8-eb2e-4319-bfd3-64caf6fb765e",
      "isActive": true,
      "picture": "http://placehold.it/120x120",
      "latitude": 54.215578,
      "longitude": -33.097056,
      "tags": [
        "incididunt",
        "et",
        "ullamco",
        "deserunt",
        "ea",
        "labore",
        "et"
      ],
      "encodings": [
        {
          "id": 0,
          "type": "occaecat"
        },
        {
          "id": 1,
          "type": "consectetur"
        },
        {
          "id": 2,
          "type": "dolor"
        }
      ]
    }
  },
  "observers": {
    "1": {
      "state_": 1,
      "target_": {
        "id": "1",
        "guid": "880ebac8-eb2e-4319-bfd3-64caf6fb765e",
        "isActive": true,
        "picture": "http://placehold.it/120x120",
        "latitude": 54.215578,
        "longitude": -33.097056,
        "tags": [
          "incididunt",
          "et",
          "ullamco",
          "deserunt",
          "ea",
          "labore",
          "et"
        ],
        "encodings": [
          {
            "id": 0,
            "type": "occaecat"
          },
          {
            "id": 1,
            "type": "consectetur"
          },
          {
            "id": 2,
            "type": "dolor"
          }
        ]
      },
      "directObserver_": {},
      "value_": {
        "id": "1",
        "guid": "880ebac8-eb2e-4319-bfd3-64caf6fb765e",
        "isActive": true,
        "picture": "http://placehold.it/120x120",
        "latitude": 54.215578,
        "longitude": -33.097056,
        "tags": [
          "incididunt",
          "et",
          "ullamco",
          "deserunt",
          "ea",
          "labore",
          "et"
        ],
        "encodings": [
          {
            "id": 0,
            "type": "occaecat"
          },
          {
            "id": 1,
            "type": "consectetur"
          },
          {
            "id": 2,
            "type": "dolor"
          }
        ]
      },
      "id_": 1
    }
  },
  "collectionModified": 1408290840241
} mediaModel.js:23
item in cache: 
Object {id: "1", guid: "880ebac8-eb2e-4319-bfd3-64caf6fb765e", isActive: true, picture: "http://placehold.it/120x120", latitude: 54.215578…}
 mediaModel.js:24
item in store (should be undefined): undefined mediaModel.js:28
{
  "collection": [
    {
      "id": "1",
      "guid": "880ebac8-eb2e-4319-bfd3-64caf6fb765e",
      "isActive": true,
      "picture": "http://placehold.it/120x120",
      "latitude": 54.215578,
      "longitude": -33.097056,
      "tags": [
        "incididunt",
        "et",
        "ullamco",
        "deserunt",
        "ea",
        "labore",
        "et"
      ],
      "encodings": [
        {
          "id": 0,
          "type": "occaecat"
        },
        {
          "id": 1,
          "type": "consectetur"
        },
        {
          "id": 2,
          "type": "dolor"
        }
      ]
    }
  ],
  "completedQueries": {
    "1": 1408290840241
  },
  "pendingQueries": {},
  "index": {
    "$$data": {},
    "$$id": "DS.media",
    "$$storage": null,
    "$$expiresHeap": {
      "heap": []
    },
    "$$lruHeap": {
      "heap": []
    },
    "$$storageMode": "memory",
    "$$storagePrefix": "DS.media",
    "$$prefix": "DS.mediaDS.media",
    "$$disabled": false,
    "$$capacity": 1.7976931348623157e+308,
    "$$deleteOnExpire": "aggressive",
    "$$recycleFreq": 100,
    "$$recycleFreqId": 14,
    "$$maxAge": 1000,
    "$$cacheFlushInterval": 100,
    "$$cacheFlushIntervalId": 15
  },
  "modified": {
    "1": 1408290840241
  },
  "saved": {
    "1": 1408290840241
  },
  "previousAttributes": {
    "1": {
      "id": "1",
      "guid": "880ebac8-eb2e-4319-bfd3-64caf6fb765e",
      "isActive": true,
      "picture": "http://placehold.it/120x120",
      "latitude": 54.215578,
      "longitude": -33.097056,
      "tags": [
        "incididunt",
        "et",
        "ullamco",
        "deserunt",
        "ea",
        "labore",
        "et"
      ],
      "encodings": [
        {
          "id": 0,
          "type": "occaecat"
        },
        {
          "id": 1,
          "type": "consectetur"
        },
        {
          "id": 2,
          "type": "dolor"
        }
      ]
    }
  },
  "observers": {
    "1": {
      "state_": 1,
      "target_": {
        "id": "1",
        "guid": "880ebac8-eb2e-4319-bfd3-64caf6fb765e",
        "isActive": true,
        "picture": "http://placehold.it/120x120",
        "latitude": 54.215578,
        "longitude": -33.097056,
        "tags": [
          "incididunt",
          "et",
          "ullamco",
          "deserunt",
          "ea",
          "labore",
          "et"
        ],
        "encodings": [
          {
            "id": 0,
            "type": "occaecat"
          },
          {
            "id": 1,
            "type": "consectetur"
          },
          {
            "id": 2,
            "type": "dolor"
          }
        ]
      },
      "directObserver_": {},
      "value_": {
        "id": "1",
        "guid": "880ebac8-eb2e-4319-bfd3-64caf6fb765e",
        "isActive": true,
        "picture": "http://placehold.it/120x120",
        "latitude": 54.215578,
        "longitude": -33.097056,
        "tags": [
          "incididunt",
          "et",
          "ullamco",
          "deserunt",
          "ea",
          "labore",
          "et"
        ],
        "encodings": [
          {
            "id": 0,
            "type": "occaecat"
          },
          {
            "id": 1,
            "type": "consectetur"
          },
          {
            "id": 2,
            "type": "dolor"
          }
        ]
      },
      "id_": 1
    }
  },
  "collectionModified": 1408290840241
} mediaModel.js:31
item in cache (should be undefined): undefined mediaModel.js:32
Nope, got it from storage 

Here's the full model file I'm using

define(function(require) {

    var app = require("shell/app");

    app.factory("mediaModel", ["DS", "$timeout", function(DS, $timeout) {

        DS.defineResource({
            name: 'media',
            idAttribute: "id",
            maxAge: 1000,
            deleteOnExpire: "aggressive",
            cacheFlushInterval: 100,
            recycleFreq: 100,
            endPoint: "media"
        });

        var promise = DS.find("media", 1).then(function(e){
            console.log("Gets media from server");
            console.log("item in store:", DS.get("media", 1));

            // internal stuff, you wouldn't normally access this
            console.log(JSON.stringify(DS.store.media, null, 2));
            console.log("item in cache:", DS.store.media.index.get(1));

            // Change this to $timeout
            $timeout(function () {
                console.log("item in store (should be undefined):", DS.get("media", 1)); // should be undefined

                // internal stuff, you wouldn't normally access this
                console.log(JSON.stringify(DS.store.media, null, 2));
                console.log("item in cache (should be undefined):", DS.store.media.index.get(5));

                DS.find("media", 1).then(function() {
                    console.log("Nope, got it from storage");
                });
            }, 10000);
        });


         return DS;

    }]);


});

[jmdobry]

I'm baffled. As you can see in this test is works the way it should. I don't yet know why it's not working for you.

And just a point on usage:

angular.module('myApp').factory('myModel', function (DS) {
  var myModel = DS.defineResource({
    name: 'myModel'
  });

  // decorate myModel with extra static methods

  return myModel;
});

Now you can inject myModel into controllers, directives, etc. Technically, you only have to inject DS itself when you want to define a resource.

[jmdobry]

Ah! I just found it. It has to do with your use of cacheFlushInterval. Remove it from the options and it should work. The problem is that angular-data relies on onExpire being called to eject the item from the same. When cacheFlushInterval is set, angular-cache removes all items without calling onExpire, so they don't get ejected.

[jasonmacdonald]

Yup, that did the trick. So, I shouldn't be using cachFlushInterval. Thanks for all the help. I knew I must have been doing something wrong :)

[jmdobry]

You weren't necessarily doing something wrong, it's just a quirk of angular-cache. From a feature standpoint, a cache flush should also trigger some sort of onExpire, so items can be ejected properly (or at the very least developers could be notified of it). From a performance standpoint it could very inefficient if implemented poorly, which is probably why I haven't addressed it yet.

I need to document the integration of angular-cache and angular-data better.

[jmdobry]

At the least, this issue resulted in this enhancement.

[jasonmacdonald]

Well, I'm glad something good came of it :) Cheers!