Hi everyone,
I am the original author of Searx. I started Hister with a similar motivation: reducing our dependence on external search engines while keeping searches and personal data under our control.
Searx is a metasearch engine that forwards queries to other search providers. Hister takes a different approach. It builds a private full text index from content you choose, then searches that index entirely on your own infrastructure.
Hister can automatically index pages through its Firefox and Chrome extensions. It can also watch local directories, import browser history and bookmarks, index individual URLs, and crawl complete documentation sites.
The feature I find most useful is offline previews. Hister stores the readable content and HTML of indexed pages locally. You can open a result in a clean and sanitized preview beside the search results without visiting the original website again.
Some other features:
- Full text search across web pages, PDFs, docx files, Markdown, OrgMode and text files
- Phrase searches, field filters, date filters, wildcards, negation, aliases, labels, facets, and result priorities
- Optional semantic search using an embeddings endpoint you configure
- Persistent website crawls
- Imports from browser history, Linkwarden, Karakeep, Shaarli, Wallabag, and Linkding
- Web, terminal, command line, HTTP API, and MCP interfaces
- SQLite and PostgreSQL support, plus optional multiple user hosting
Hister cannot replace a global search engine (yet) for subjects you have never encountered because it only searches what you have indexed. My workflow is to search Hister first, then use its shortcut to fall back to traditional search when I need broader web results.
The project is free software under the AGPLv3+ license. It can be installed as a standalone binary or with Docker.
Project: https://github.com/asciimoo/hister
Website and documentation: https://hister.org/
Small read-only demo: https://demo.hister.org/
I’d appreciate feedback, questions, and suggestions as well as joining our growing community.
AI disclosure: AI assisted contributions are not strictly prohibited, but all contributions should be made by humans. More details: https://github.com/asciimoo/hister/blob/master/CONTRIBUTING.md#ai-policy
You must log in or register to comment.
@asciimoo@lemmy.ml please update the title with the appropriate tag (rule 7) and if AI was involved in development, add the disclosure per rule 8 (sample disclosures can be found here)
Thanks!
Thanks for letting me know. I’ve updated the post.
SearXNG user here. Thanks for your work!
I’m looking forward to checking this out.
Like others have mentioned, a federated version of this seems like it would be really, really powerful.
Any interested in making this a federated search?
I’ve been toying with the idea of a web-ring style search engine where it uses the fediverse’s post and sub system so that you can subscribe to “indexers” that are made by other users (or even mastodon users themselves) so that you can have highly targeted search engine systems. So, for example, you could make multiple search “scopes” and each of those scopes would follow specific other “sources” and search from them. Likewise, users could provide links and descriptions to add new entries to each scope which would, in essense, work like a fediverse post and then be distributed to all other interested parties.
I haven’t done a lick of implementation yet, but thought I’d share the idea here as I consider doing this more and more every day. The only thing I haven’t figured out yet is images, because ideally that would be a special type of scope view.
Federated search is the direction I’d like to go when the core is mature enough.
I’m still trying to figure out the best approach to make the federation secure (no accidental private/confidential data leak) and easy to use. Related conversations: https://github.com/asciimoo/hister/discussions/432 & https://github.com/asciimoo/hister/issues/387 . I’d appreciate help to figure out an optimal solution.
I’d made a separate comment asking the same question, but glad to know you’re already working on this. Great work btw!
Awesomely cool and federated search would be a boon later! Hope you keep it going!
Asciimoo always with the cool stuff!
Think I’ll try getting it setup on Kubernetes and using it for searching documentation when programming.
Would you be interested in getting a Kubernetes example in the docs too then? If you have any requirements for it let me know.
I’d appreciate it, thanks. No special requirements. Providing sensible defaults and explaining usage/potential customization options would be great.
I’ve been running this for a while since I first heard of it, it’s already saved me a few times trying to recall a page for something.
I treat it like a more advanced history and it’s great!
Already installed and loving this idea.
Can i use this in my self hosted environment? the docs talk mainly about running it in the terminal
does it have a docker install? found it https://hister.org/docs/dockercan i connect it to other services like paperless, or would i need to manually import files?
I have tried it a bit today and I like it, but I have a few questions
- Is it better to run it in docker or e.g. a LXC in proxmox? If i want to index files, it seems it needs the config.yml file
- After just playing with it for a day, the disk usage is 150 MB. I figure I should use postgres instead of sqlite. Do you have any examples on how to set that up, preferably with docker? I guess I can figure it out, but it might take me some time. I am not familiar with pgvector
can i connect it to other services like paperless, or would i need to manually import files?
Hister supports importing data from a few services, but paperless isn’t supported yet. More details: https://hister.org/docs/import
Is it better to run it in docker or e.g. a LXC in proxmox? If i want to index files, it seems it needs the config.yml file
Docker is perfectly fine. Every settings option from the config file can specified using environment variables. The syntax is
HISTER__[SECTION]__[OPTION]=[VALUE].After just playing with it for a day, the disk usage is 150 MB
Probably most of the disk space is occupied by the Hister binary which contains all the N-grams required to identify ~30 languages. The index should be much smaller.
I figure I should use postgres instead of sqlite.
SQLite is more than enough for personal use, but if you prefer to use postgres, just specify the standard DSN formatted connection data to the
server.databaseconfig option: https://hister.org/docs/configuration#database-backendshow to set that up, preferably with docker
Use the
HISTER__SERVER__DATABASE="host=localhost user=hister password=hister dbname=hister port=5432"environment variable.I am not familiar with pgvector
Hister automatically creates the database model and handles the migrations if required.
Thank you for your response 🙂 I will stick to sqlite
How do i use environment variables to make a indexer? I couldn’t get it to work
i.e. the part with directories https://hister.org/posts/indexing-local-files-with-hister
indexer: directories: - path: ~/Documents filetypes: ['md', 'txt'] - path: ~/code/projects filetypes: ['txt'] excludes: [, 'secret/*', '*key'] - path: ~/notes patterns: ['*.org', 'doc_*', 'README']How do i use environment variables to make a indexer? I couldn’t get it to work
Repeatable config options are not supported from env vars. In this case you have to mount a config file to your container. But, if you want to track local files from docker, you have to mount those files/directories anyway. I suggest to use the standalone binary release if you’d like to track local files, it is way more convenient.
This seems super cool, you can curate your own search index :O
Being able to preview your search results offline is quite neat too!
One question though, how will you solve the issue of “echo chambers”, because from my understanding, it looks like it will only present results that you search for / have searched for, meaning for some people, you could be stuck with results that only support your view of the world.
I mean it kinda looks like that’s by design currently. It’s an indexer for your corners of the web, not a search aggregator to find new ones.
Exactly, the echo chamber phenomenon is mostly problematic for “discovery type” searches while Hister is mainly for “recall type” search.
Implementing federated search/index sharing could be a partial solution to this issue in the long run.
Is there a future where Hister takes on the functions of SearXNG? I.E. new content is discovered through APIs but local or previous content is prioritized?
Currently it is not planned. Hister guarantees that non of your data/query/metadata leaves the service if you use it. As I see, this is a more valuable and unique feature than having an integrated metasearch. There are already great metasearch solutions and Hister provides an easy fallback to search providers, so in my opinion this direction would be more of a sacrifice than an improvement.
Oh I didnt realize there was a fallback option, that’s great.
Oh damn this could replace the bookmakers. I have Linkding with the Linkding Injector extension, but this would be next level.
You can view the saved text too, nice. Would it be possible to attach an HTML file from like single file for sites that have heavy image content as part of the “view” button? That’d completely replace Linkding/Karakeep/Linkwarden use cases for me
Understandable if not, that’s ancillary to the text search focus
Oh damn this could replace the bookmakers.
The inspiration for Hister was a bookmarking app, but I realized that I always forget to manually trigger the bookmarking and I miss so many great resources.
Would it be possible to attach an HTML file from like single file
Currently you can import SingleFile HTMLs using the
hister import filecommand, but no further integrations are implemented yet. Although, rendering the exact SinglePage file as a preview can be added relatively quickly. It would be also nice to accept files directly from the SingleFile extension.Thanks for the good suggestion, I’ve added it to my TODO! =]
I have tons of webpages saved using the Firefox built-in page saver, which saves and html file and a corresponding folder for the other resources (images, javascript). Would be cool if these could be imported as well. Though maybe the resource folder can be ignored and the html file can already be imported?
Though maybe the resource folder can be ignored and the html file can already be imported?
It depends. Hister always requires a unique URL for each document. SingleFile snapshots include the original URL of the document as a meta HTML element. I’m not sure if the built-in page saver provides URL information.
No unfortunately it does not. Sometimes the webpage itself contains a meta tag with the original URL but not always
The name reminds me of this crazy nostradamus “documentary “ that used to play on HBO when I was a kid.
deleted by creator
If just it was called hipster
and utilized a rapper with backwards facing basecap and bling bling as logoLets see how it stacks up against the big ones
