Indexing speedup #5939
Indexing speedup #5939
Conversation
|
|
||
| try: | ||
| for _ in range(chunk_size): | ||
| objects_id.append(qs_iterator.__next__().pk) |
There was a problem hiding this comment.
You can just call next(qs_iterator).pk
|
|
||
| try: | ||
| for _ in range(chunk_size): | ||
| objects_id.append(qs_iterator.__next__().pk) |
There was a problem hiding this comment.
This section could probably be simplified by using next() with a default value.
There was a problem hiding this comment.
What should be the default?
There was a problem hiding this comment.
I think default is not required, current logic required to catch the exception.
| 'index_name': index_name, | ||
| } | ||
|
|
||
| while not is_iterator_empty: |
There was a problem hiding this comment.
Actually you can just use islice from itertools
from itertools import islice
objects_id = list(islice(qs_iterator, chunk_size))There was a problem hiding this comment.
And you can use https://docs.djangoproject.com/en/2.2/ref/models/querysets/#values-list to only get an iterator with only the ids
There was a problem hiding this comment.
With this approach we will have a list of all the ids at once.
I think it is better to iterate over on generator object?
There was a problem hiding this comment.
It returns a QuerySet of values, it's lazy https://stackoverflow.com/questions/37140426/does-django-queryset-values-list-return-a-list-object
There was a problem hiding this comment.
Ohhh...
But we also want to logging based on the pk.
Sorry... But I'm not able to think on how to use this method and to log also.
Do you have something on your mind?
There was a problem hiding this comment.
you can get the last pk from the list last_pk = objects_id[-1] if objects_id else 0
There was a problem hiding this comment.
Not so important anyway, current logic works too
There was a problem hiding this comment.
I prefer the current implementation. It feels more explicit to me.
|
|
||
| try: | ||
| for _ in range(chunk_size): | ||
| objects_id.append(next(qs_iterator).pk) |
There was a problem hiding this comment.
I'd also like some logging here to be able to track progress. Something like if pk % 5000 = 0: log.info('Total: pk')
No description provided.