Subproject search sharing sibling search index #8678
Comments
|
I'm not aware that this feature existed. We have always returned results from subprojects when searching on the main project.
I think that refers to returning search results of subprojects from the main project. And that's done at the query level, not at the index level. |
|
Yeah, I think we might have implemented it in a proof of concept state for a customer or two, I don't think we ever made it an actual feature of our search.
It seems our docs do pretty clearly describe sibling subprojects sharing an index:
"Sharing index" here maybe isn't technically correct, but yeah the effect to the user is a shared index even if we're just altering the queries used. |
|
Just noting: I think this would have been implemented in the old search/lib.py file, but I am not finding anything obvious in the history. Here are the commits before the file was refactored: |
|
BTW, there are some ideas about how to make this configurable in It's about subprojects, but could be make more general, like: Don't include results from subprojects version: 2
search:
projects: []Include results from some sub/projects version: 2
search:
projects:
- one
- two |
|
Yeah, that's a great example. So, to make this configuration option make sense as a Per-version configuration of the search index could be the most explicit version of this configuration. For example, the ability to target a specific version of a project: search:
projects:
- project: one
version: latest
- project: two
version: 1.0Perhaps, to step a bit back, another option that we haven't explored much yet is a separate configuration file for project level settings. This file is only valid on the main branch, so there don't need to be conflicts with readthedocs.yaml on each branch or historical branch. |
Codecov has this option https://docs.codecov.com/docs/codecov-yaml#team-yaml We can use something like that, this isn't a default setting for all projects, but for all versions (this is also helpful for projects migrating already released versions to rtd I guess). This will require to rebuild the versions to apply some settings. |
This is another good point yeah. There are some options, like maybe search configuration, that seem fine to apply on build like that. Redirects seem like another case for a project-level configuration file.
I like how they merge the files, that could be something else to explore. Options like redirects would benefit from both project and version level configuration files. |
|
Had another user request this. It would be great if there was a simple way to optionally do this, but I agree that #7217 is the better & more flexible solution. |
|
Another option that we could explore is to do this at the API level, this is pass an array or projects as an option instead of just one, that won't need users to re-build a project, we could make it as an option in our search as you type extension (also I think we could do the default search override as an extension). I'm thinking something like:
The only thing missing would be the version, so what about using We already check for permissions on each version at the api level readthedocs.org/readthedocs/search/api.py Line 237 in ee18557 this could allow to search across multiple versions as well! |
I'm not sure to understand this. Won't this require knowing all the subprojects of a project at build time? How the hardcoded URL for search will know the exact list of projects it has to pass? What happens if a new subproject is added after the build of the other subprojects is done? |
This is a more general solution, to allow to share results across any set of given projects. But to include all sibling projects, we could do something similar at the API level instead of the build level, like Of course, those options will probably be hardcoded at build time to generate the final URL in our extension, but it leaves the door open for adding more options to the search UI, like a checkbox to change those options at runtime. |
|
I definitely think the API should be more flexible. If we allow people to pass multiple projects, users could also build some kind of collection search concept completely without us having to be involved. I'm definitely 👍 on expanding the API as a first step, and then adding new features on our first-party implementations that have a nice UX around the API extensions. |
|
So, completing my idea, here is how it will look like:
The response from our current API already supports including results from multiple projects, so the only new field that will be added is the field that says what were the projects/versions that we used in the final query. https://docs.readthedocs.io/en/stable/server-side-search.html#api Some examples of a request would be:
To decide
I'm putting this in a comment, but let me know if you prefer a design doc. |
|
I don't have a great understanding of all the usage of our search API, but this description looks great to me 💯 . I see it flexible and promising about the UX we can build on top of it. I'm 👍🏼 on it.
Is this
We could consider returning 400 here as well and explaining the error: "When using
I think this parameter would require more thinking. Where the API would perform the search if we pass I'd start implementing what you have described: "search on all the projects received by parameters; no matter if they are subprojects or they are not related at all, just search on them all". This looks simpler and more explicit for API users to me. Also, combined with an API query of
I assume this would be something like |
Not sure, since this is kind of specific to our platform, the most similar comparison could be a platform where there are namespaces/hierarchies, like GitHub/GitLab I think you mean the
I was thinking this will look for subprojects on all given projects, yes. |
|
@agjohnson no configuration is needed, users just need to search with the |
|
It's possible to do that, but I think what we're talking about with this feature is automatic sibling project search. This was the feature that we lost from our in-doc search. Right now, readers won't know how to configure our search at all, the advanced queries are more a dashboard search UX. Configuring the query to append search terms seems like a neat idea. This would need to happen outside our Sphinx search extension too, in the standard search override. |
|
IMHO, documentation authors should not define whether or not the reader's query is performed in subprojects/siblings. This should be a reader's decision. The UI has to be clear about "where your query is going to be performed" and also "results should be clear about where they come from". Imagine that GH search always perform the search over the whole organization because the owner of that organization decided that in advance. I'd say that's bad UX. Clicking in our own in-doc search should show the modal with something similar to this:
Even if the user are currently able to search on siblings by themselves, they have to know the project slugs from the other projects as well, which is almost impossible, and even if it's possible, pretty hard to type 😄
This reduces the "platform integrations" we have been talking about and requires the author to update this setting each time they add a new subproject. Besides, it still considers being a Sphinx extension instead of tool agnostic. |
|
We have the |
|
Right, but query strings are undiscoverable for normal users. We need a much nicer UX to make those concepts discoverable, which I think the GH example @humitos showed is 💯 |
This is more specific to GitHub, where the users of GitHub are probably familiar with GitHub's concept of repository/organization/etc. For our use case, most reader users are not familiar with RTD at all, and probably won't know whether they mean to search subprojects/organization projects/etc. Also, it's hard/impossible for a reader user to guess where content might live in a superproject/subproject relationship.
Yeah true, it could go either way though. The documentation author probably knows more that the reader about project topology though, so can most likely decide this for the user with mostly positive outcomes.
This is a great idea 👍 Perhaps we should apply more UI like this, but also make this UI default configurable. That is, a project author can decide the default mode of search, but users can always override it. I'm thinking mostly of commercial projects that use nesting. Community likely would indeed benefit from consistency in the search method, as the reader users are generally more technical already. |
Well, I think communicating "whether they mean to search subprojects/organization projects/etc" is part of the work the UI has to do. Showing these options to the user will make them to ask themselves this question and think about it. Definitely, not giving them these options is a worse UX.
👍🏼 on giving authors the ability to override defaults. This is part of the "platform integrations" we are talking about:
My point here is that I don't want to leave the readers outside the equation. "Authors know more about their project than readers" is not 100% accurate. It may be true, but not accurate in multiple contexts. The experience from the author's perspective is completely different than from the reader's perspective. |
Yeah a bit, though I'm describing a problem deeper than this too. How is the reader supposed to know that the documentation they are looking for is in a subproject, or why they even need to search in subprojects or sibling projects at all? The reader won't know this, except through trial and error, which doesn't fair well with less/non technical readers. I'm not sure how to solve this with UI, other than complicating the UX a good deal. But for now, giving authors control of the project search defaults will be the most we can do to help reader users. The documentation authors know more about how to find content in their documentation than we could, and definitely know more than readers do.
To clarify, I'm not advocating for this option. Surfacing UI to override search defaults is still a good option.
Yeah, still agreed here 💯 And to keep this UI approachable, I'd probably say that we shouldn't be talking about subprojects/sibling projects at all. These are maintainers terms that readers will be confused by. What about reducing this to |
I like this, or perhaps |
|
Making sure we are all on the same page, we want to implement this in our search extension, right? Not in our indoc-search override or dashboard (this could be done with the new templates) |
If I understand correctly what we've discussed, we want one and only one search interface. This means that the UX for indoc-search and dashboard will be exactly the same, with small differences in the UI items presented due to the context (ie. "search related projects" may not appear in dashboard search). There is a long conversation at https://github.com/readthedocs/reports/pull/20#discussion_r1046872899 with more context about why I'm saying this. That said, I'd this work should be done in a Javascript library that we can embed in all the places where we need search 😄 |
|
Yeah, I would probably implement the scoping options in a larger overhaul of the front end pieces. I'm not yet certain how this will look, given some technical hurdles with joining documentation and dashboard UI and/or reusing the code for this UI. What exactly would the end solution look like, technically speaking? Are we going to automate appending all of the If so, and if we want to expose this configuration to users in the final solution, perhaps our first step is to add these configuration options and automation on our side. If our end goal is to eventually have a search dropdown like Just a thought, I haven't thought too deeply here. |
I was thinking of having users define a default query, but this query would be shown in the text box, so it's explicit to users what they are searching for, and they can change that default query if needed. For example, something like this, the
And we could have "shortcuts" below the saerch box, like And even we could expose a
Are we thinking on an option in our js extension/library, or something more high level like in our config file? |
+1 on this eventually, but what I'm trying to describe would be a step in the middle. We wouldn't add this UI yet, but we would start by adding the default query configuration option at least. The goal would be to give RTD users, especially users not using our search extension, the ability to search subprojects by default. The UX on the search scope options is one I'd like to put off, because we are talking about changing the search UI for all projects, not just projects using the search extension. I'd rather visit this particular change when we're rethinking the flyout search UI instead of changing this in multiple places. The query addition as text that you're describing really goes a long way here though, so I think the UI additions will just be later polish.
I think I agree, though might extend this further. The end goal for configuration options is probably something like: So, if we add a default query option, we should make sure it aligns with this plan, or we're building in backwards incompatibilities already. I think that
What does this look like for users not using our extension? I'm mostly concerned about that use case right now as we have a few customers waiting on this change. Prepending the search works less well in that UI element (we don't want to pollute the default search input box), but we could still sneak the term into the search query on the fly, and show it on the search results page. That is, the in-doc search input does not have
That's a neat addition 👍 This could come later of course though, I sort of like this pattern from GitHub though. We probably have some more too, like
No idea, I was going to ask you this 😆 This needs to be usable from our JS, so it needs to be in the JSON context var block (or API return, but I don't think that use aligns well). So it would either need to be a sphinx configuration (for the short term), or would need to be a Long term, |
I think this may be related to our decision about how we want to integrate our features going forward. I think there is a conversation going on about this in some place... I was thinking making this change into our extension/js library, but not sure what we decided about keep expanding our custom injected search. Maybe your suggestion about using a sphinx configuration could work, that setting could be used for both, our custom injected search and the sphinx extension, but then we would need to maintain that forever p: I like the idea of allowing users to define their custom scopes! |
|
I'd like to see a quick way to add the ability to automatically pull in subprojects in the search without user interaction, a feature flag perhaps? |
|
Discussion action items:
|
I thought we landed on working on the extension first? The dashboard UI is the view I think we should drop, as we are only talking about maintaining a singular search UI. We'd reuse the in-doc search UI in the dashboard in some fashion however. But getting users and customers to use dashboard search over in-doc search (which matches their doc styling/branding/etc) is a big ask. |
|
I thought David made some good points around this:
So instead of doing work on a Sphinx extension nobody is using, we do the simple thing and make our dashboard search a better experience, and it's somewhere that we have more expeirence doing template work with minimal JS? That lets us do very simple UX iteration without complexity of trying to have JS & frontend assets that works across arbitrary websites. |
We keep coming back to this, but we also seem to all share fairly strong reservations about maintaining two search UIs. Our overall goal doesn't feel changed by only maintaining an in-doc search UI, as our search is primarily meant for readers, not dashboard users/maintainers. We've noted this in our meeting and issues where we've discussed this, but the dashboard search doesn't feel like where we want to invest if we're focusing efforts on one search UI. We know usage of the dashboard search is already very low traffic, and that customers/users want a native, in-doc search experience. To me all signs still point towards polishing the in-doc search experience instead. We should retain a design that allows for search UI reuse in the dashboard though, so we can give maintainers a shortcut to in-doc search too.
The dashboard search is subject to the same though. Project dashboard search queries in 2022 totaled 65k unique pageviews for the year, and many of the queries seem empty or for spam projects. Frankly, I wouldn't be surprised if our search extension invokes more usage. But either way, the extension (namely it's UI/UX) has already been planned as the basis for the in-doc search going forward. We'll generalize it as a library, but the UI/UX is already closest to what we know users want to use. I did share the hesitation about starting implementation here, but it seemed like we were happy starting here as long as there is an upgrade path for extension users to use a general search UI plugin for in-doc search later. I'm +1 on this plan now.
Frankly, I don't think we'll ever have this, especially for search. Our users have wanted control over our 404 pages, login pages, and even our flyout styling. Even pointing to our dashboard URL will be a no-go for users. I'm not very worried about retaining control of the UI. The approach EthicalAds uses seems like a pattern to duplicate for our search and our flyout -- base themes with the option to manually override styling.
Well, we can iterate, but it's leading towards a development dead end essentially -- the in-doc search can't reuse Django specific code. This is headed towards maintaining two search UI implementations at very least, and I'm guessing we'll continue to have UI/UX disparity or not maintaining one of the implementations well. As I understood things from our discussions, next steps were to start with a configuration value on the extension and to recommend users that want sibling/subproject in-doc search to use the extension instead. This is minimal, and solves my original push, which was to expose sub/sibling project search to readers for customers/users that have asked. Future steps would be making the search extension a drop-in library instead, and then start bringing more of our search features to readers through this library. The end goal is a library that is injected, and activated through our flyout, and likely opt-in for in-doc search input fields too. |
|
Notes from the call: High-level status✔️ We agree on having a single UI/UX for search (what our search extension currently has) Next steps
|
|
We've done a lot of progress on this issue and it seems we are already working on the v3 of the proposed plan. I don't know if it makes sense to close it or not, but I've opened readthedocs/addons#29 with the idea of "moving the checkbox filters inside the search suggestions", which I think it will gives a great UI/UX |
|
All the backend work is done already. I'll close this issue for now. We can continue the conversation on the frontend side around beta addons and the linked issues. Feel free to reopen if you consider there is anything missing on the backend side and comment on the addons related issues if there are ideas from here that should be exposed there. |
It's not quite clear where we are with this feature. At some point this was a feature of our subproject search, though I recall it originally requiring some hacks to make this work. We discussed this internally and it seems this is indeed not a feature of our search anymore. However, currently our docs do mention that sibling projects share an index, and we have issues like #4623 that found sibling sharing is possible.
Did we refactor this feature out and need to add it back?
We should:
cc Wizard of Search, @stsewd
The text was updated successfully, but these errors were encountered: