A recommender system service based on collaborative filtering written in Go

Overview

Language: English | 中文

gorse: Go Recommender System Engine

Build Coverage Report GoDoc RTD Demo
build codecov Go Report Card GoDoc Documentation Status Website

gorse is an offline recommender system backend based on collaborative filtering written in Go.

This project is aim to provide a high performance, easy-to-use, programming language irrelevant recommender micro-service based on collaborative filtering. We could build a simple recommender system on it, or set up a more sophisticated recommender system using candidates generated by it. It features:

  • Implements 7 rating based recommenders and 4 ranking based recommenders.
  • Supports data loading, data splitting, model training, model evaluation and model selection.
  • Provides the data import/export tool, model evaluation tool and RESTful recomender server.
  • Accelerates computations by SIMD instructions and multi-threading.

For more information:

  • Visit GoDoc for detailed documentation of codes.
  • Visit ReadTheDocs for tutorials, examples and usages.
  • Visit SteamLens for a Steam games recommender system based on gorse.

Install

  • Download from release.
  • Build from source:

Install Golang and run go get:

$ go get github.com/zhenghaoz/gorse/...

It will download all packages and build the gorse command line into your $GOBIN path.

If your CPU supports AVX2 and FMA3 instructions, use the avx2 build tag to enable AVX2 and FMA3 instructions.

$ go get -tags='avx2' github.com/zhenghaoz/gorse/...

Usage

gorse is an offline recommender system backend based on collaborative filtering written in Go.

Usage:
  gorse [flags]
  gorse [command]

Available Commands:
  export-feedback Export feedback to CSV
  export-items    Export items to CSV
  help            Help about any command
  import-feedback Import feedback from CSV
  import-items    Import items from CSV
  serve           Start a recommender sever
  test            Test a model by cross validation
  version         Check the version

Flags:
  -h, --help   help for gorse

Use "gorse [command] --help" for more information about a command.

Evaluate a Recommendation Model

gorse provides the tool to evaluate models. We can run gorse test -h or check online documents to learn its usage. For example:

$ gorse test bpr --load-csv u.data --csv-sep $'\t' --eval-precision --eval-recall --eval-ndcg --eval-map --eval-mrr
...
+--------------+----------+----------+----------+----------+----------+----------------------+
|              |  FOLD 1  |  FOLD 2  |  FOLD 3  |  FOLD 4  |  FOLD 5  |         MEAN         |
+--------------+----------+----------+----------+----------+----------+----------------------+
| [email protected] | 0.321041 | 0.327128 | 0.321951 | 0.318664 | 0.317197 | 0.321196(±0.005931)  |
| [email protected]    | 0.212509 | 0.213825 | 0.213336 | 0.206255 | 0.210764 | 0.211338(±0.005083)  |
| [email protected]      | 0.380665 | 0.385125 | 0.380003 | 0.369115 | 0.375538 | 0.378089(±0.008974)  |
| [email protected]       | 0.122098 | 0.123345 | 0.119723 | 0.116305 | 0.119468 | 0.120188(±0.003883)  |
| [email protected]       | 0.605354 | 0.601110 | 0.600359 | 0.577333 | 0.599930 | 0.596817(±0.019484)  |
+--------------+----------+----------+----------+----------+----------+----------------------+

u.data is the CSV file of ratings in MovieLens 100K dataset and u.item is the CSV file of items in MovieLens 100K dataset. All CLI tools are listed in the CLI-Tools section of Wiki.

Setup a Recommender Server

It's easy to setup a recomendation service with gorse.

  • Step 1: Import feedback and items.
$ gorse import-feedback ~/.gorse/gorse.db u.data --sep $'\t' --timestamp 2
$ gorse import-items ~/.gorse/gorse.db u.item --sep '|'

It imports feedback and items from CSV files into the database file ~/.gorse/gorse.db. The low level storage engine is implemented by BoltDB.

  • Step 2: Start a server.
$ gorse serve -c config.toml

It loads configurations from config.toml and start a recommendation server. It may take a while to generate all recommendations. Detailed information about configuration is in the Configuration section of Wiki. Before set hyper-parameters for the model, it is useful to test the performance of chosen hyper-parameters by the model evaluation tool.

  • Step 3: Get recommendations.
$ curl 127.0.0.1:8080/recommends/1?number=5

It requests 5 recommended items for the 1-th user. The response might be:

[
    {
        "ItemId": "919",
        "Popularity": 96,
        "Timestamp": "1995-01-01T00:00:00Z",
        "Score": 1
    },
    {
        "ItemId": "474",
        "Popularity": 194,
        "Timestamp": "1963-01-01T00:00:00Z",
        "Score": 0.9486470268850127
    },
    ...
]

"ItemId" is the ID of the item and "Score" is the score generated by the recommendation model used to rank. See RESTful APIs in Wiki for more information about RESTful APIs.

Use gorse in Go

Also, gorse could be imported and used in Go application. There is an example that fits a recommender and generate recommended items:

package main

import (
	"fmt"
	"github.com/zhenghaoz/gorse/base"
	"github.com/zhenghaoz/gorse/core"
	"github.com/zhenghaoz/gorse/model"
)

func main() {
	// Load dataset
	data := core.LoadDataFromBuiltIn("ml-100k")
	// Split dataset
	train, test := core.Split(data, 0.2)
	// Create model
	bpr := model.NewBPR(base.Params{
		base.NFactors:   10,
		base.Reg:        0.01,
		base.Lr:         0.05,
		base.NEpochs:    100,
		base.InitMean:   0,
		base.InitStdDev: 0.001,
	})
	// Fit model
	bpr.Fit(train, nil)
	// Evaluate model
	scores := core.EvaluateRank(bpr, test, train, 10, core.Precision, core.Recall, core.NDCG)
	fmt.Printf("[email protected] = %.5f\n", scores[0])
	fmt.Printf("[email protected] = %.5f\n", scores[1])
	fmt.Printf("[email protected] = %.5f\n", scores[1])
	// Generate recommendations for user(4):
	// Get all items in the full dataset
	items := core.Items(data)
	// Get user(4)'s ratings in the training dataset
	excludeItems := train.User("4")
	// Get top 10 recommended items (excluding rated items) for user(4) using BPR
	recommendItems, _ := core.Top(items, "4", 10, excludeItems, bpr)
	fmt.Printf("Recommend for user(4) = %v\n", recommendItems)
}

The output should be:

2019/11/14 08:07:45 Fit BPR with hyper-parameters: n_factors = 10, n_epochs = 100, lr = 0.05, reg = 0.01, init_mean = 0, init_stddev = 0.001
2019/11/14 08:07:45 epoch = 1/100, loss = 55451.70899118173
...
2019/11/14 08:07:49 epoch = 100/100, loss = 10093.29427682404
[email protected] = 0.31699
[email protected] = 0.20516
[email protected] = 0.20516
Recommend for 4-th user = [288 313 245 307 328 332 327 682 346 879]

Recommenders

There are 11 recommendation models implemented by gorse.

Model Data Task Multi-threading Fit
explicit implicit weight rating ranking
BaseLine ✔️ ✔️ ✔️
NMF ✔️ ✔️ ✔️
SVD ✔️ ✔️ ✔️
SVD++ ✔️ ✔️ ✔️ ✔️
KNN ✔️ ✔️ ✔️ ✔️
CoClustering ✔️ ✔️ ✔️ ✔️
SlopeOne ✔️ ✔️ ✔️ ✔️
ItemPop ✔️ ✔️ ✔️
KNN (Implicit) ✔️ ✔️ ✔️ ✔️ ✔️
WRMF ✔️ ✔️ ✔️ ✔️
BPR ✔️ ✔️ ✔️
  • Cross-validation of rating models on MovieLens 1M [Source].
Model RMSE MAE Time (AVX2)
SlopeOne 0.90683 0.71541 0:00:26
CoClustering 0.90701 0.71212 0:00:08
KNN 0.86462 0.67663 0:02:07
SVD 0.84252 0.66189 0:02:21 0:01:48
SVD++ 0.84194 0.66156 0:03:39 0:02:47
  • Cross-validation of ranking models on MovieLens 100K [Source].
Model [email protected] [email protected] [email protected] [email protected] [email protected] Time
ItemPop 0.19081 0.11584 0.05364 0.21785 0.40991 0:00:03
KNN 0.28584 0.19328 0.11358 0.34746 0.57766 0:00:41
BPR 0.32083 0.20906 0.11848 0.37643 0.59818 0:00:13
WRMF 0.34727 0.23665 0.14550 0.41614 0.65439 0:00:14

Performance

gorse is much faster than Surprise, and comparable to librec while using less memory space than both of them. The memory efficiency is achieved by sophisticated data structures.

  • Cross-validation of SVD on MovieLens 100K [Source]:

  • Cross-validation of SVD on MovieLens 1M [Source]:

Contributors

Any kind of contribution is expected: report a bug, give a advice or even create a pull request.

Acknowledgments

gorse is inspired by following projects:

Limitations

gorse has limitations and might not be applicable to some scenarios:

  • No Scalability: gorse is a recommendation service on a single host, so it's unable to handle large data.
  • No Features: gorse exploits interactions between items and users while features of items and users are ignored.
Comments
  • Prevent recommending an item for an user who provided feedback

    Prevent recommending an item for an user who provided feedback

    I have been trying the gorse with bpr model to recommend content to users, whom have option to give feedback which are fed into gorse.

    I'm using gorse as standalone web server and accessing with it's own apis.

    What I'm trying to accomplish is preventing the engine from recommending content the user already provided feedback. Bu I have looked both into the docs and the code but nothing really looked like what I wanted. There is somewhat related #33 but it recommends an entry only once and never again.

    Is there any way to accomplish what I want.

    opened by abdullahcanakci 8
  • cdn好像不太稳定

    cdn好像不太稳定

    https://cdn.gorse.io/example/github.sql 经常访问不了,但有时候又可以

    会造成docker安装的时候错误

    [[email protected] ~]# docker run -p 8088:8088 zhenghaoz/gorse-in-one --playground Welcome to Gorse v0.4.8 Playground {"level":"fatal","ts":1666740804.338797,"caller":"gorse-in-one/main.go:89","msg":"failed to initialize database","error":"Get \"https://cdn.gorse.io/example/github.sql\": dial tcp 185.183.84.75:443: i/o timeout","errorVerbose":"Get \"https://cdn.gorse.io/example/github.sql\": dial tcp 185.183.84.75:443: i/o timeout\nmain.initializeDatabase:177: ","stacktrace":"main.glob..func1\n\t/go/gorse/cmd/gorse-in-one/main.go:89\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:876\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:990\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:918\nmain.main\n\t/go/gorse/cmd/gorse-in-one/main.go:144\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}

    bug 
    opened by warrana2321 6
  • gorse-in-one启动时候还有个panic

    gorse-in-one启动时候还有个panic

    只在启动时候出现, 不过捕获到了... 就是看着难受 -_-! { "level":"error", "ts":1656313095.142161, "caller":"base/util.go:61", "msg":"panic recovered", "panic":"runtime error: index out of range [-1]", "stacktrace":"github.com/zhenghaoz/gorse/base.CheckPanic /Users/ak47/dec/gorse-master/base/util.go:61 runtime.gopanic /usr/local/go/src/runtime/panic.go:838 runtime.goPanicIndex /usr/local/go/src/runtime/panic.go:89 github.com/zhenghaoz/gorse/base/search.(*IVF).Build.func1 /Users/ak47/dec/gorse-master/base/search/ivf.go:205 github.com/zhenghaoz/gorse/base/parallel.Parallel /Users/ak47/dec/gorse-master/base/parallel/parallel.go:31 github.com/zhenghaoz/gorse/base/search.(*IVF).Build /Users/ak47/dec/gorse-master/base/search/ivf.go:192 github.com/zhenghaoz/gorse/base/search.(*IVFBuilder).Build /Users/ak47/dec/gorse-master/base/search/ivf.go:274 github.com/zhenghaoz/gorse/master.(*Master).findItemNeighborsIVF /Users/ak47/dec/gorse-master/master/tasks.go:386 github.com/zhenghaoz/gorse/master.(*Master).runFindItemNeighborsTask /Users/ak47/dec/gorse-master/master/tasks.go:254 github.com/zhenghaoz/gorse/master.(*Master).runRankingRelatedTasks /Users/ak47/dec/gorse-master/master/tasks.go:857 github.com/zhenghaoz/gorse/master.(*Master).RunPrivilegedTasksLoop /Users/ak47/dec/gorse-master/master/master.go:277" }

    bug irreproducible 
    opened by ppyupp 6
  • Migration error with PostgresSQL database

    Migration error with PostgresSQL database

    I want to use Gorse with PostgreSQL (latest alpine image). The second and re-launch of containers are performed with the following error in the log.

    master_1    | 2022/05/23 14:58:11 /go/pkg/mod/gorm.io/driver/[email protected]/migrator.go:270 pq: syntax error at or near "not"
    master_1    | [0.374ms] [rows:0] ALTER TABLE "users" ALTER COLUMN "labels" TYPE json not null default '[]'
    postgres_1  | 2022-05-23 14:58:11.740 UTC [33] ERROR:  syntax error at or near "not" at character 53
    postgres_1  | 2022-05-23 14:58:11.740 UTC [33] STATEMENT:  ALTER TABLE "users" ALTER COLUMN "labels" TYPE json not null default '[]'
    master_1    | {"level":"fatal","ts":1653317891.7413962,"caller":"master/master.go:190","msg":"failed to init database","error":"pq: syntax error at or near \"not\"","errorVerbose":"pq: syntax error at or near \"not\"\n/go/gorse/storage/data/sql.go:134: ","stacktrace":"github.com/zhenghaoz/gorse/master.(*Master).Serve\n\t/go/gorse/master/master.go:190\nmain.glob..func1\n\t/go/gorse/cmd/gorse-master/main.go:57\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:842\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:943\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:883\nmain.main\n\t/go/gorse/cmd/gorse-master/main.go:70\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}
    

    Gorse version I'm tried both nightly and latest tag of docker images.

    My docker-compose.yml file:
    version: "3"
    services:
      redis:
        image: redis
        restart: unless-stopped
        ports:
          - 6379:6379
    
      postgres:
        image: postgres:alpine
        restart: unless-stopped
        ports:
          - 5432:5432
        environment:
          POSTGRES_DB: "gorse"
          POSTGRES_USER: "gorse"
          POSTGRES_PASSWORD: "gorse_pass"
          PGDATA: /data/postgres
        volumes:
          - pg_data:/data/postgres
    
      worker:
        image: zhenghaoz/gorse-worker:nightly
        restart: unless-stopped
        ports:
          - 8089:8089
        command: >
          --master-host master --master-port 8086 
          --http-host 0.0.0.0 --http-port 8089
          --log-path /var/log/gorse/worker.log 
          --cache-path /var/lib/gorse/worker_cache.data
        volumes:
          - gorse_log:/var/log/gorse
          - worker_data:/var/lib/gorse
        depends_on:
          - master
    
      server:
        image: zhenghaoz/gorse-server:nightly
        restart: unless-stopped
        ports:
          - 8087:8087
        environment:
          GORSE_CACHE_STORE: redis://redis:6379
          GORSE_DATA_STORE: postgres://gorse:[email protected]:5432/gorse?sslmode=disable
        command: >
          --master-host master --master-port 8086 
          --http-host 0.0.0.0 --http-port 8087
          --log-path /var/log/gorse/server.log 
          --cache-path /var/lib/gorse/server_cache.data
        volumes:
          - gorse_log:/var/log/gorse
          - server_data:/var/lib/gorse
        depends_on:
          - master
    
      master:
        image: zhenghaoz/gorse-master:nightly
        restart: unless-stopped
        ports:
          - 8086:8086
          - 8088:8088
        environment:
          GORSE_CACHE_STORE: redis://redis:6379
          GORSE_DATA_STORE: postgres://gorse:[email protected]:5432/gorse?sslmode=disable
        command: >
          -c /etc/gorse/config.toml 
          --log-path /var/log/gorse/master.log 
          --cache-path /var/lib/gorse/master_cache.data
        volumes:
          - ./config.toml:/etc/gorse/config.toml
          - gorse_log:/var/log/gorse
          - master_data:/var/lib/gorse
        depends_on:
          - redis
          - postgres
          
     
    volumes:
      pg_data:
      worker_data:
      server_data:
      master_data:
      gorse_log:
    

    What's wrong?

    bug 
    opened by ben73 6
  • Inserting feedback fails on DB constraint

    Inserting feedback fails on DB constraint

    Gorse version latest

    Describe the bug Whenever I try to insert feedback through API or dashboard, it fails with the message

    Error 4025: CONSTRAINT `users.labels` failed for `gorse`.`users`","errorVerbose":"Error 4025: CONSTRAINT `users.labels` failed for `gorse`.`users`
    

    To Reproduce Startup Gorse on clean DB -> Import feedback through dashboard -> On confirm import it fails

    Expected behavior The insert should pass for feedback.

    Additional context I have no problem with inserting users and items. I had the same problem even with items and users imported.

    Stack trace:

    {"level":"error","ts":1634292942.5676055,"caller":"server/rest.go:1323","msg":"internal server error","error":"Error 4025: CONSTRAINT `users.labels` failed for `gorse`.`users`","errorVerbose":"Error 4025: CONSTRAINT `users.labels` failed for `gorse`.`users`\n/go/gorse/storage/data/sql.go:805: ","stacktrace":"github.com/zhenghaoz/gorse/server.InternalServerError\n\t/go/gorse/server/rest.go:1323\ngithub.com/zhenghaoz/gorse/server.(*RestServer).insertFeedback.func1\n\t/go/gorse/server/rest.go:1153\ngithub.com/emicklei/go-restful/v3.(*FilterChain).ProcessFilter\n\t/go/pkg/mod/github.com/emicklei/go-restful/[email protected]/filter.go:21\ngithub.com/zhenghaoz/gorse/server.LogFilter\n\t/go/gorse/server/rest.go:69\ngithub.com/emicklei/go-restful/v3.(*FilterChain).ProcessFilter\n\t/go/pkg/mod/github.com/emicklei/go-restful/[email protected]/filter.go:19\ngithub.com/emicklei/go-restful/v3.(*Container).dispatch\n\t/go/pkg/mod/github.com/emicklei/go-restful/[email protected]/container.go:291\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2046\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2424\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2878\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1929"}
    

    Screenshot from 2021-10-15 12-53-25 Screenshot from 2021-10-15 12-53-33

    Gorse config:

    # This section declares settings for the database.
    [database]
    
    cache_store = "redis://localhost:6379"
    
    data_store = "<connection string>"
    
    cache_size = 50
    
    auto_insert_user = true
    
    auto_insert_item = false
    
    positive_feedback_types = ["watch_full"]
    
    click_feedback_types = []
    
    read_feedback_type = "view"
    
    positive_feedback_ttl = 0
    
    item_ttl = 0
    
    [master]
    port = 8086                     # master port
    host = "0.0.0.0"                # master host
    http_port = 8088                # HTTP API port
    http_host = "0.0.0.0"           # HTTP API host
    n_jobs = 4                      # number of working jobs
    meta_timeout = 10               # cluster meta timeout (second)
    
    [server]
    default_n = 20                  # default number of returned items
    api_key = "<API-key>"  # secret key for RESTful APIs (SSL required)
    
    [recommend]
    
    popular_window = 7
    
    fit_period = 10
    
    search_period = 60
    
    search_epoch = 100
    
    search_trials = 10
    
    refresh_recommend_period = 1
    
    fallback_recommend = ["item_based", "popular"]
    explore_latest_num = 20
    
    item_neighbor_type = "auto"
    
    user_neighbor_type = "auto"
    
    enable_latest_recommend = true
    
    enable_popular_recommend = true
    
    enable_user_based_recommend = true
    
    enable_item_based_recommend = true
    
    enable_collaborative_recommend = true
    
    bug help wanted 
    opened by PetrBezousek 6
  • Feedbacks import problem

    Feedbacks import problem

    Gorse version last version

    Describe the bug I imported my feedbacks in the Import feedbacks window on Gorse. However, in the Gorse Dashboard Overview I do not have any feedback displayed. Moreover, then, I exported the feedbacks from Gorse and there is literally nothing in there. It is as if Gorse fakes to import the feedbacks, but in reality it does not. I don't think it's a TimeStamp Type issue (but maybe it is), as I tried with many different types and it still doesn't work.

    Even when I try with a csv file of 5 lines like this one it does not work : favori,UserId,ItemId,Timestamp favori,3423,186758,2022-06-24 21:20:11 favori,3423,186758,2022-06-24 21:20:13 favori,2823,186085,2022-06-24 21:20:19 favori,2140,185965,2022-06-24 21:20:20

    feedback_jeudi.csv

    Can you help me please ?

    bug 
    opened by VictoriaSaden 5
  • [solved] Extend the api to get last viewed items

    [solved] Extend the api to get last viewed items

    I know the recommender is responsible for making recommends, but I think it would be useful to get access to the last viewed items over the api. This should be easily made.

    All the needed data is already collected for gorse. So it would be nice to get access to the last 10 viewed items, instead of collecting the same data again.

    I know this is not the primary goal of you recommender, but I guess this is also a typical use case for others.

    feature/minor 
    opened by almare 5
  • Memory build-up on Redis

    Memory build-up on Redis

    Gorse version 0.3.4

    Describe the bug I've been noticing a memory build-up (probably a leakage) on the Gorse Database used on Redis. Not sure why but the memory has been slowly building until it reaches 100%.

    To Reproduce Let gorse running for a few days and memory should hit 100%.

    bug 
    opened by guiscaranse 5
  • valid positive 0

    valid positive 0

    0 valid positive

    Gorse version

    zhenghaoz/gorse-master:latest

    Describe the bug

    Status from the overview:

    USERS 201,393 ITEMS 402 TOTAL POSITIVE 58,548 VALID POSITIVE 0 VALID NEGATIVE 0

    Search click-through rate prediction model | Failed | 0000/12/31 20:06 |   | No feedback found. -- | -- | -- | -- | -- Fit collaborative filtering model | Failed | 0000/12/31 20:06 |   | No feedback found. Fit click-through rate prediction model | Failed | 0000/12/31 20:06 |   | No feedback found. Search collaborative filtering model | Failed | 0000/12/31 20:06 |   | No feedback found.

    positive feedback in config.toml: positive_feedback_types = ["contact"] I can see just item-related recommendations

    To Reproduce

    config.yml:

    # This section declares settings for the database.
    [database]
    # The database for caching, support Redis only:
    #   redis://<user>:<password>@<host>:<port>/<db_number>
    cache_store = "redis://redis:6380"
    # The database for persist data, support MySQL, Postgres, ClickHouse and MongoDB:
    #   mysql://[username[:password]@][protocol[(address)]]/dbname[?param1=value1&...&paramN=valueN]
    #   postgres://bob:[email protected]:5432/mydb?sslmode=verify-full
    #   clickhouse://user:[email protected][:port]/database?param1=value1&...&paramN=valueN
    #   mongodb://[username:[email protected]]host1[:port1][,...hostN[:portN]][/[defaultauthdb][?options]]
    data_store = "mysql://gorse:[email protected](mysql:3306)/gorse?parseTime=true"
    # The cache size for recommended/popular/latest items. The default value is 100.
    cache_size = 200
    # Insert new users while inserting feedback. The default value is true.
    auto_insert_user = true
    # Insert new items while inserting feedback. The default value is true.
    auto_insert_item = true
    # The feedback types for positive events.
    positive_feedback_types = ["contact"]
    # The feedback types for read events.
    read_feedback_types = ["read"]
    # The time-to-live (days) of positive feedback, 0 means disabled. The default value is 0.
    positive_feedback_ttl = 0
    # The time-to-live (days) of items, 0 means disabled. The default value is 0.
    item_ttl = 0
    # This section declares settings for the master node.
    [master]
    port = 8086                     # master port
    host = "0.0.0.0"                # master host
    http_port = 8088                # HTTP API port
    http_host = "0.0.0.0"           # HTTP API host
    n_jobs = 4                      # number of working jobs
    meta_timeout = 10               # cluster meta timeout (second)
    # This section declares settings for the server node.
    [server]
    default_n = 20                  # default number of returned items
    api_key = ""                    # secret key for RESTful APIs (SSL required)
    # This section declares settings for recommendation.
    [recommend]
    # The time window of popular items (days). The default values is 180.
    popular_window = 30
    # The time period for model fitting (minutes). The default values is 60.
    fit_period = 10
    # The time period for model searching (minutes). The default values is 100.
    search_period = 60
    # The number of epochs for model searching. The default values is 100.
    search_epoch = 100
    # The number of trials for model searching. The default values is 10.
    search_trials = 10
    # The time period to refresh recommendation for inactive users (days). The default values is 5.
    refresh_recommend_period = 1
    # The fallback recommendation method is used when cached recommendation drained out:
    #   item_based: Recommend similar items to cold-start users.
    #   popular: Recommend popular items to cold-start users.
    #   latest: Recommend latest items to cold-start users.
    # Recommenders are used in order. The default values is ["latest"].
    fallback_recommend = ["item_based", "latest"]
    # The type of neighbors for items. There are three types:
    #   similar: Neighbors are found by number of common labels.
    #   related: Neighbors are found by number of common users.
    #   auto: If a item have labels, neighbors are found by number of common labels.
    #         If this item have no labels, neighbors are found by number of common users.
    # The default values is "auto".
    item_neighbor_type = "similar"
    # The type of neighbors for users. There are three types:
    #   similar: Neighbors are found by number of common labels.
    #   related: Neighbors are found by number of common liked items.
    #   auto: If a user have labels, neighbors are found by number of common labels.
    #         If this user have no labels, neighbors are found by number of common liked items.
    # The default values is "auto".
    user_neighbor_type = "similar"
    # Enable latest recommendation during offline recommendation. The default values is false.
    enable_latest_recommend = true
    # Enable popular recommendation during offline recommendation. The default values is false.
    enable_popular_recommend = true
    # Enable user-based similarity recommendation during offline recommendation. The default values is false.
    enable_user_based_recommend = true
    # Enable item-based similarity recommendation during offline recommendation. The default values is false.
    enable_item_based_recommend = true
    # Enable collaborative filtering recommendation during offline recommendation. The default values is true.
    enable_collaborative_recommend = true
    # Enable click-though rate prediction during offline recommendation. Otherwise, results from multi-way recommendation
    # would be merged randomly. The default values is true.
    enable_click_through_prediction = true
    # The explore recommendation method is used to inject popular items or latest items into recommended result:
    #   popular: Recommend popular items to cold-start users.
    #   latest: Recommend latest items to cold-start users.
    # Recommenders are used in order. The default values is { popular = 0.0, latest = 0.0 }.
    explore_recommend = { popular = 0.1, latest = 0.2 }
    

    docker-compose.yaml

    version: "3"
    services:
      redis:
        image: redis
        restart: unless-stopped
        ports:
          - 6380:6380
      mysql:
        image: mysql/mysql-server
        restart: unless-stopped
        ports:
          - 3306:3306
        environment:
          MYSQL_ROOT_PASSWORD: root_pass
          MYSQL_DATABASE: gorse
          MYSQL_USER: gorse
          MYSQL_PASSWORD: gorse_pass
        volumes:
          - ./var/lib/mysql:/var/lib/mysql
      worker:
        image: zhenghaoz/gorse-worker
        restart: unless-stopped
        ports:
          - 8089:8089
        command: >
          --master-host master --master-port 8086 --http-host 0.0.0.0 --http-port 8089
          --log-path /var/log/gorse/worker.log --cache-path /var/lib/gorse/worker_cache.data
        volumes:
          - ./var/log/gorse:/var/log/gorse
          - ./var/lib/gorse:/var/lib/gorse
      server:
        image: zhenghaoz/gorse-server
        restart: unless-stopped
        ports:
          - 8087:8087
        command: >
          --master-host master --master-port 8086 --http-host 0.0.0.0 --http-port 8087
          --log-path /var/log/gorse/server.log --cache-path /var/lib/gorse/server_cache.data
        volumes:
          - ./var/log/gorse:/var/log/gorse
          - ./var/lib/gorse:/var/lib/gorse
      master:
        image: zhenghaoz/gorse-master
        restart: unless-stopped
        ports:
          - 8086:8086
          - 8088:8088
        command: -c /etc/gorse/config.toml --log-path /var/log/gorse/master.log --cache-path /var/lib/gorse/master_cache.data
        volumes:
          - ./config.toml:/etc/gorse/config.toml
          - ./var/log/gorse:/var/log/gorse
          - ./var/lib/gorse:/var/lib/gorse
    

    (simplified) data

    feedback.sql

    DROP TABLE IF EXISTS `feedback`;
    CREATE TABLE `feedback` (
      `feedback_type` varchar(256) NOT NULL,
      `user_id` integer NOT NULL,
      `item_id` integer NOT NULL,
      `time_stamp` timestamp NOT NULL,
      `comment` text NOT NULL,
      PRIMARY KEY (`feedback_type`,`user_id`,`item_id`),
      KEY `user_id` (`user_id`)
    ); 
    LOCK TABLES `feedback` WRITE;
    INSERT INTO `feedback` VALUES
    ('contact', 9285, 2, '2013-01-21 18:47:08', ''),('contact', 9521, 2, '2013-04-22 14:21:10', ''),('contac
    t', 12614, 2, '2013-04-22 14:21:10', ''),('contact', 39577, 2, '2013-04-01 18:47:47', ''),('contact', 42
    083, 2, '2013-04-08 17:28:57', '');UNLOCK TABLES;
    

    users.sql

    DROP TABLE IF EXISTS `users`;
    
    CREATE TABLE `users` (
      `user_id` integer NOT NULL,
      `labels` json NOT NULL,
      `comment` text NOT NULL,
      `subscribe` json NOT NULL,
      PRIMARY KEY (`user_id`)
    ) ;
    LOCK TABLES `users` WRITE;
    INSERT INTO `users` VALUES
    (111830, '["Uruguay", "Montevideo", "Pocitos"]', '', '[]'),(113915, '["Uruguay", "Montevideo", "Pocitos"
    ]', '', '[]'),(119975, '["Uruguay", "Montevideo", "Pocitos"]', '', '[]'),(121938, '["Uruguay", "Montevid
    eo", "Pocitos"]', '', '[]');UNLOCK TABLES;
    

    items.sql

    DROP TABLE IF EXISTS `items`;
    CREATE TABLE `items` (
      `item_id` integer NOT NULL,
      `time_stamp` timestamp NOT NULL,
      `labels` json NOT NULL,
      `comment` text NOT NULL,
      `is_hidden` tinyint(1) NOT NULL DEFAULT '0',
      `categories` json NOT NULL,
      PRIMARY KEY (`item_id`)
    ) ;
    
    LOCK TABLES `items` WRITE;
    INSERT INTO `items` VALUES 
    (1257, '2013-09-27 02:10:06', '["Uruguay", "Montevideo", "Pocitos"]', '', 0, '[]'),(2814, '2014-10-09 13
    :26:37', '["amoblado", "amenities", "encargado", "portero", "vigilancia", "seguridad", "Argentina", "Bs.
    As. G.B.A. Zona Norte", "Pilar", "La Lonja"]', '', 0, '[]');UNLOCK TABLES;
    

    Expected behavior

    recommendations

    Additional context

    logs

    master.log

    {"level":"info","ts":1638877759.9023194,"caller":"master/tasks.go:60","msg":"load dataset","positive_feedback_types":["contact"],"read_feedback_types":["read"],"item_ttl":0,"feedback_ttl":0,
    "positive_feedback_types":["contact"]}
    
    
    bug 
    opened by brivadeneira 5
  • pq: sorry, too many clients already

    pq: sorry, too many clients already

    Hi, I have updated my gorse version to latest v0.2.5, along with the latest config.toml . Now we are getting below error after the application runs for few time:-

    {"level":"error","ts":1631013939.8973596,"caller":"server/rest.go:1252","msg":"internal server error","error":"pq: sorry, too many clients already","errorVerbose":"pq: sorry, too many clients already\n/go/gorse/storage/data/sql.go:226: ","stacktrace":"github.com/zhenghaoz/gorse/server.internalservererror\n\t/go/gorse/server/rest.go:1252\ngithub.com/zhenghaoz/gorse/server.(*restserver).getmeasurements\n\t/go/gorse/server/rest.go:1234\ngithub.com/emicklei/go-restful/v3.(*filterchain).processfilter\n\t/go/pkg/mod/github.com/emicklei/go-restful/[email protected]/filter.go:21\ngithub.com/zhenghaoz/gorse/server.logfilter\n\t/go/gorse/server/rest.go:71\ngithub.com/emicklei/go-restful/v3.(*filterchain).processfilter\n\t/go/pkg/mod/github.com/emicklei/go-restful/v3:1932"}
    {"level":"info","ts":1631013939.897418,"caller":"server/rest.go:74","msg":"GET /api/measurements/ClickThroughRate?n=30","status_code":500}
    

    I have used the docker implementation with Postgres as backend database. Thanks.

    help wanted 
    opened by rinshadka 5
  • Information regarding valid time formats.

    Information regarding valid time formats.

    Hi, I recieve errors in the form of 'failed to parse datetime `` at line x' in the console while loading csv's. Could you point me to the valid datetime formats acceptable for timestamps in the csv file? Any help would be appreciated.

    Thanks, Avinash.

    question 
    opened by avinashkurup 5
  • More label types

    More label types

    Currently, only string labels are supported. This makes it impossible to have numeric relations like a price. ("User 1 likes Items that cost below 300 $")

    It would be helpful to have more types of labels like numeric (i.e. prices) and dates (i.e. event dates).

    I helped myself by having price ranges as string labels (i.e. "200-300$). This is a good workaround but does not fully solve the problem.

    feature/future 
    opened by phil-break 1
  • Filter possible items for recommendation

    Filter possible items for recommendation

    Currently, the only possibility to filter items for a recommendation is the use of categories. Only one category can be used as a filter. I.e. this scenario in an e-com shoe store is not possible: "Recommend shoes from category sneaker with colors black, white or gray."

    I think it would be helpful to have the opportunity to filter by multiple categories, as well as by certain labels.

    feature/future 
    opened by phil-break 1
  • implement queue in data storage

    implement queue in data storage

    Background

    In Gorse v0.4.x, neighbors, and models are not updated until the next round of full dataset loading. In practice, it would be better to update neighbors or models once a new user, item, or feedback is inserted. A queue is required to achieve this goal.

    Design

    type Database interface {
    
           // Enqueue put the value to the back of a named queue.
           Enqueue(name, value string) error
    
           // Dequeue remove the front value of a named queue and return this value. 
           // If no value in the queue, return io.EOF.
           Dequeue(name string) (value string, err error) 
    }
    
    opened by zhenghaoz 0
  • support more APIs in Go client

    support more APIs in Go client

    opened by zhenghaoz 0
  • No feedback found.

    No feedback found.

    Fit click-through rate prediction model | Failed | 0001/01/01 03:41 |   | No feedback found. Search click-through rate prediction model | Failed | 0001/01/01 03:41 |   | No feedback found. Search collaborative filtering model | Failed | 0001/01/01 03:41 |   | No feedback found.

    docker-compose.yml

    rec_db: image: postgres:14.5 environment: - POSTGRES_USER=user - POSTGRES_DB=recm - POSTGRES_PASSWORD=password - POSTGRES_PORT=5432 volumes: - ./data/rec_db:/var/lib/recm/data/pgdata ports: - "5479:5432" redis: image: redis expose: - "6379" ports: - "6379:6379"

    worker: image: zhenghaoz/gorse-worker restart: unless-stopped ports:

    • 8089:8089 command: > --master-host master --master-port 8086 --http-host 0.0.0.0 --http-port 8089 --log-path /var/log/gorse/worker.log --cache-path /var/lib/gorse/worker_cache.data volumes:
    • gorse_log:/var/log/gorse
    • worker_data:/var/lib/gorse depends_on:
    • master links:
    • master

    server: image: zhenghaoz/gorse-server restart: unless-stopped ports:

    • 8087:8087 command: > --master-host master --master-port 8086 --http-host 0.0.0.0 --http-port 8087 --log-path /var/log/gorse/server.log --cache-path /var/lib/gorse/server_cache.data volumes:
    • gorse_log:/var/log/gorse
    • server_data:/var/lib/gorse depends_on:
    • master links:
    • master

    master: image: zhenghaoz/gorse-master restart: unless-stopped ports:

    • 8086:8086
    • 8088:8088 environment: GORSE_CACHE_STORE: redis://redis:6379 GORSE_DATA_STORE: postgres://user:[email protected]_db:5432/recm?sslmode=disable command: > -c /etc/gorse/config.toml --log-path /var/log/gorse/master.log --cache-path /var/lib/gorse/master_cache.data volumes:
    • master_data:/var/lib/gorse
    • ./config.toml.template:/etc/gorse/config.toml
    • gorse_log:/var/log/gorse depends_on:
    • redis
    • rec_db links:
    • redis
    • rec_db

    volumes: worker_data: server_data: master_data: gorse_log:

    opened by timurchikk 3
Releases(v0.4.10)
Owner
Zhenghao Zhang
Knowledge is power, France is bacon.
Zhenghao Zhang
A Kubernetes Native Batch System (Project under CNCF)

Volcano is a batch system built on Kubernetes. It provides a suite of mechanisms that are commonly required by many classes of batch & elastic workloa

Volcano 2.7k Dec 2, 2022
Learn how to design large-scale systems. Prep for the system design interview. Includes Anki flashcards.

English ∙ 日本語 ∙ 简体中文 ∙ 繁體中文 | العَرَبِيَّة‎ ∙ বাংলা ∙ Português do Brasil ∙ Deutsch ∙ ελληνικά ∙ עברית ∙ Italiano ∙ 한국어 ∙ فارسی ∙ Polski ∙ русский язы

Donne Martin 204.7k Dec 4, 2022
Go implementation of the yolo v3 object detection system

Go YOLO V3 This repository provides a plug and play implementation of the Yolo V3 object detection system in Go, leveraging gocv. Prerequisites Since

Wim Spaargaren 56 Sep 23, 2022
PaddleDTX is a solution that focused on distributed machine learning technology based on decentralized storage.

中文 | English PaddleDTX PaddleDTX is a solution that focused on distributed machine learning technology based on decentralized storage. It solves the d

null 82 Nov 18, 2022
Derivative microservices based in Go.

Derivative Microservices Essentially this repository contains a re-write of the Islandora microserives: houdini, homarus, hypercube, and FITS (a TODO)

Islandora Digital Collections - JHU Sheridan Libraries 0 Dec 17, 2021
small go-based app for note taking

Description A small note taking web-app which I am building for myself just for fun and learning purposes. My goal is too use as much of the standard

Ekin Tiras 2 Oct 8, 2022
Genetic Algorithms library written in Go / golang

Description Genetic Algorithms for Go/Golang Install $ go install git://github.com/thoj/go-galib.git Compiling examples: $ git clone git://github.com

Thomas Jager 193 Sep 27, 2022
Neural Networks written in go

gobrain Neural Networks written in go Getting Started The version 1.0.0 includes just basic Neural Network functions such as Feed Forward and Elman Re

Go Machine Learning 530 Nov 27, 2022
A Naive Bayes SMS spam classifier written in Go.

Ham (SMS spam classifier) Summary The purpose of this project is to demonstrate a simple probabilistic SMS spam classifier in Go. This supervised lear

Dan Wolf 13 Sep 9, 2022
k-means clustering algorithm implementation written in Go

kmeans k-means clustering algorithm implementation written in Go What It Does k-means clustering partitions a multi-dimensional data set into k cluste

Christian Muehlhaeuser 412 Nov 10, 2022
a* pathfinding algorithm written in go

astar a* (a-star) pathfinding algorithm written in go Wikipedia: EN: A* search algorithm DE: A*-Algorithmus Install go get github.com/jpierer/[email protected]

Julian Pierer 26 Mar 21, 2022
A simple utility, written in Go, for interacting with Salesforce.

Salesforce CLI A simple utility, written in Go, for interacting with Salesforce. Currently only specific functionality is implemented, and the output

Darren Parkinson 0 Dec 14, 2021
A simple yet customisable program written in go to make hackerman-like terminal effects.

stuntman a simple program written in go to make you look like a hackerman Demo stuntman -binar -width 90 -color cyan stuntman -text -width 90 -vertgap

Solaris 10 Aug 4, 2022
Suricate-bank - API to transfer money between accounts at Suricate Bank,written in Go

⚠️ WORK IN PROGRESS ⚠️ Suricate Bank is an api that creates accounts and transfe

João Saraceni 12 Oct 8, 2022
ncurses matrix/log app written in go to visualize chess problems.

dorrella/matrix-curses Matrix using ncurses and gbin/goncurses. Visual matrix based puzzles Install need libncurses-dev. Probably hard to run on windo

null 0 Jan 12, 2022
An open source recommender system service written in Go

gorse: Go Recommender System Engine Gorse is an open-source recommendation system written in Go. Gorse aims to be a universal open-source recommender

Gorse 6.4k Dec 5, 2022
Collaborative Filtering (CF) Algorithms in Go!

Go Recommend Recommendation algorithms (Collaborative Filtering) in Go! Background Collaborative Filtering (CF) is oftentimes used for item recommenda

Tim Kaye 193 Oct 19, 2022
Stackoverflow-Tag-Recommender - Calculate similarity of different tags based on the data of stackoverflow

Recommender System with Stackoverflow's Data This project is written to recommen

Roozbeh Sayadi 8 Apr 4, 2022
Moby Project - a collaborative project for the container ecosystem to assemble container-based systems

The Moby Project Moby is an open-source project created by Docker to enable and accelerate software containerization. It provides a "Lego set" of tool

Moby 64.6k Dec 4, 2022
Moby Project - a collaborative project for the container ecosystem to assemble container-based systems

The Moby Project Moby is an open-source project created by Docker to enable and accelerate software containerization. It provides a "Lego set" of tool

Moby 64.6k Nov 30, 2022