🔎 Mi 搜 - a personal search engine

Misou is a personal search engine very much inspired by monocle that looks through my knowledge sources. It is written in Go and React and uses wails to build a native desktop app. The app is mainly build for my usage but I tried to make it generally usable. It's usable at this point but still early-stage.

Features

• search and configure different sources
• file links are opened in Obsidian
• browser links are opened in Browser
• shortcuts

Installation

Words of caution: The installation is suited to my setup and has not been tested on other machines. If you encounter problems, please contact me :)

0. Install the dependencies:

   brew install ripgrep-all jq
   

1. Create appconfig.json and fill fields that apply:

   {
     "repoPath": "",
     "readwiseCsvPath": "",
     "instapaperPath": "",
     "peoplePath": "",
     "peopleUrl": "",
     "twitterPath": "",
     "fileScrapeRgaPath": "/Users/name/homebrew/bin/rga",
     "fileScrapePath": "",
     "secretsSubPath": "/secrets",
     "historyBashSubPath": "/source/history.sh",
     "bookmarkBashSubPath": "/source/bookmarks.sh",
     "sources": {
       "bookmarks": true,
       "history": true,
       "localfiles": true,
       "gdrive": false,
       "instapaper": true,
       "readwise": true,
       "people": false,
       "twitter": true
     }
   }

   gdrive and people are not supported by me :)

2. Move appconfig.json to desired directory

3. Export config path:

   export misouCfg=/Users/adria/Programming/misou/appconfig.json
   

4. Build app:

   make build
   

   Note that the config location is injected during build time and cannot be changed afterwards!

5. Generate search index (if local sources are used, see Sources)

   make index
   

Development

Use make dev to build the backend and exposes it through a websocket and make dev-front for the development frontend server. Note that for make dev it is necessary to set cfgPath inside the source file integration.go, since it is not possible to inject the path as in build!

Sources

using external API (slow ~500ms):

• local file folder
• bookmarks and history (tested with Chrome)

locally indexed (much faster ~30ms):

• readwise (provided with exported csv)
• instapaper
• twitter
• personal CRM (not public but based on noahm's fork of mira)

The data of indexed sources need to be manually updated in the folder. Afterwards, the index can be rebuilt using make index

Readwise

If you use Obsidian, it's better to use the Readwise-sync plugin and combine it with local file search. Otherwise export the csv manually from the UI.

Instapaper

If you use Readwise, highlights are synced, but this plugin allows for full-text search in archive by locally downloading the file content.

Twitter

To get the twitter archive check here. The update of the twitter archive requires a conversion of the tweet.js to a json file. This converter can be used, saving the file in tweet.json to format the file in vscode. The latter ensures proper escaping of " in field text.

Future features

• support exact search (indicated with surrounding ") not only for indexed sources
• integrate Memex
• Alfred workflow

Bugs

I use fzf and the bookmarks and history function and found that after a system restart, I need to run these commands once before running my scripts for the corresponding sources.

In sources, there is a Gdrive client which I don't support anymore because I don't need it. I remember that after a few days the token had to be deleted manually.

Dependencies

• wails: for native app build
• RGA: for local file search
• Blevesearch: used as main indexer (alternative implementation based on Apollo can also be used)

Attributions

Appicon (Sou): Freepik from www.flaticon.com

This project started on a hackathon with the help of my friends from university.