Try reverting prefetch changes for project/version listing views

[agjohnson]

• Reverts Add support for successful build prefetch #11613
• Reverts Prefetch build and project on version list #11616
• Disables previously used prefetch on the project dashboard view
• Refs Try to tune project listing query again #11620

[agjohnson]

I'm testing this and it's actually not solving the problem with query time. I'm still seeing 6s response time. This is still because of the prefetch query that we were previously using. Something seems like it regressed here and I'm guessing it probably solves the underlying problem with these changes above too.

[agjohnson]

Noted in chat, but it seems there is a missing foreign key on builds_build for project. I have this locally, but it is not in production:

    "builds_build_project_id_5c35b3fd_fk_projects_project_id" FOREIGN KEY (project_id) REFERENCES projects_project(id) DEFERRABLE INITIALLY DEFERRED

It's definitely not clear if this helps the planner or not though, this might be another dead end.

I would sooner merge in #11620 than this PR as it at least dropped the view time to 3s for my user.

• Try to tune project listing query again #11620

[agjohnson]

With the revert of the additional prefetch not fixing the response time, I've disabled all prefetching on the dashboard view. If there are no other ideas on resolving the response timing issues here, we can revisit prefetching to reduce query load later.

Undoing the prefetch we've had in place for a while now, the response time is <1s and the query response time is instant:

In [1]: %time a = list(Project.objects.dashboard(User.objects.filter(is_staff=True).first()))
CPU times: user 11 ms, sys: 3.92 ms, total: 14.9 ms
Wall time: 66.3 ms

There are still 120 queries for each dashboard load however.

[ericholscher]

Yea, I'm 👍 on rolling this back and thinking about some other approaches here, like perhaps the normal Django cache across processes or similar.

[agjohnson]

To be clear, this isn't just rolling back my changes. This wasn't enough to reduce response time (it's still 6s response time). This revert is removing all prefetching from the project dashboard view, added years ago here #5637. If there were performance benefits to these, maybe they got lost over time?

Caching is maybe on the table, I might also lean towards making this an explicit model field or relationship.

[ericholscher]

Yea, denormalizing the latest build ID or similar onto the model is another really obvious answer 👍

[humitos]

I might also lean towards making this an explicit model field or relationship.
denormalizing the latest build ID or similar onto the model is another really obvious answer

Is there a simple way to do this at DB level or we need to implement this in Python to update that field every time we trigger a new build for the project (latest_build) and once the build finishes successfully (latest_successful_build)?

[agjohnson]

Materialized views can do this at the database level, but this approach almost always feels too heavy. These views are harder to manage and use with Django migrations/ORM, or at least have been in the past.

So yeah, it might be more of a model operation at the end of builds. I'd like to continue discussing this, I opened an issue so we can merge this:

• Build: reducing query count, prefetching, or intermediate tables #11631