Skip to content

Performance issue with PostgresqlRow.getColumn(String name) when select many columns #636

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
yuki-teraoka opened this issue Feb 19, 2024 · 1 comment
Labels
type: enhancement A general enhancement
Milestone

Comments

@yuki-teraoka
Copy link

When selecting many fields, a lot of CPU is consumed because the fields are looped.

In one case I encountered, with a select of millions of records with about 700 fields, I forced the query to stop because it was consuming 100% CPU for several hours.

At this time, I checked Java's stack trace many times, and most of them were processing the following for statement.

for (int i = 0; i < this.fields.size(); i++) {
RowDescription.Field field = this.fields.get(i);
if (field.getName().equalsIgnoreCase(name)) {
return i;
}
}

Would it be possible to prepare field name and order mappings in advance?

@mp911de
Copy link
Collaborator

mp911de commented Feb 19, 2024

This is possible. PGJDBC uses a lazy approach to create the column name to index map, see https://github.com/pgjdbc/pgjdbc/blob/master/pgjdbc/src/main/java/org/postgresql/jdbc/PgResultSet.java#L3108-L3136

Happy to merge a pull request.

@mp911de mp911de added type: enhancement A general enhancement status: ideal-for-contribution An issue that a contributor can help us with labels Feb 19, 2024
cty123 added a commit to cty123/r2dbc-postgresql that referenced this issue Feb 24, 2024
…ooping through the fields unnecessarily.

[resolves pgjdbc#636]
@mp911de mp911de added this to the 1.0.5.RELEASE milestone Feb 28, 2024
@mp911de mp911de removed the status: ideal-for-contribution An issue that a contributor can help us with label Feb 28, 2024
mp911de added a commit that referenced this issue Feb 28, 2024
Refine hash map initial sizing. Use ROOT locale for field names.

[#636][closes #640]
mp911de pushed a commit that referenced this issue Feb 28, 2024
Avoid looping through the fields unnecessarily.

[resolves #636][#640]
mp911de added a commit that referenced this issue Feb 28, 2024
Refine hash map initial sizing. Use ROOT locale for field names.

[#636][closes #640]
cty123 added a commit to cty123/r2dbc-postgresql that referenced this issue Mar 3, 2024
cty123 added a commit to cty123/r2dbc-postgresql that referenced this issue Mar 3, 2024
mp911de added a commit that referenced this issue Mar 4, 2024
Simplify code. Use putIfAbsent instead of custom check.

[#636][closes #641]
mp911de added a commit that referenced this issue Mar 4, 2024
Simplify code. Use putIfAbsent instead of custom check.

[#636][closes #641]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: enhancement A general enhancement
Projects
None yet
2 participants