[BUG]: Workspace Documents popup Becomes Unusable with Large Number of Documents

[blazeyo]

How are you running AnythingLLM?

Docker (local)

What happened?

When working with a large number of documents (around 6.5k in my case after importing a repository), the workspace UI becomes unresponsive. Rendering all items at once causes the browser to freeze for several minutes, making it nearly impossible to use.

Proposed Solution(s)

Implement Lazy Loading

To improve performance, implement lazy loading for the document list. Instead of rendering all documents upfront, only load a subset initially and load more as the user scrolls.

Load a small batch of items initially (e.g., 20 documents).
When scrolling is detected in the container, load the next batch of items.
This would significantly reduce the initial rendering time and prevent the UI from freezing.

Add Fuzzy Search for Document Lists

In addition to lazy loading, improve document navigation by implementing fuzzy search functionality. This would help users quickly find the documents they are looking for without needing to scroll through thousands of files.

Initially load a small subset of items (e.g., 20 documents).
Add input fields for each document container (e.g., left and right panels) to allow users to search.
When the search input changes, display only the subset of files that match the search string using fuzzy search techniques.
Both features would improve performance and usability when dealing with large document sets.

Impact

By implementing either lazy loading, fuzzy search, or both, the usability of the workspace with large document sets would improve dramatically. These changes would reduce the load time, prevent browser freezes, and make it easier for users to locate documents quickly.

What do you think?

Are there known steps to reproduce?

Add 6k files to the database and open the workspace documents popup.