Skip to content

Commit 5dbf36f

Browse files
lunnylunny
authored
Issue search support elasticsearch (#9428)
* Issue search support elasticsearch * Fix lint * Add indexer name on app.ini * add a warnning on SearchIssuesByKeyword * improve code
1 parent 1765602 commit 5dbf36f

File tree

286 files changed

+57032
-25
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

286 files changed

+57032
-25
lines changed

‎.drone.yml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,12 @@ services:
8686
pull: default
8787
image: gitea/test-openldap:latest
8888

89+
- name: elasticsearch
90+
pull: default
91+
environment:
92+
discovery.type: single-node
93+
image: elasticsearch:7.5.0
94+
8995
steps:
9096
- name: fetch-tags
9197
pull: default

‎custom/conf/app.ini.sample

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -368,8 +368,12 @@ CONN_MAX_LIFETIME = 3s
368368
MAX_OPEN_CONNS = 0
369369

370370
[indexer]
371-
; Issue indexer type, currently support: bleve or db, default is bleve
371+
; Issue indexer type, currently support: bleve, db or elasticsearch, default is bleve
372372
ISSUE_INDEXER_TYPE = bleve
373+
; Issue indexer connection string, available when ISSUE_INDEXER_TYPE is elasticsearch
374+
ISSUE_INDEXER_CONN_STR = http://elastic:changeme@localhost:9200
375+
; Issue indexer name, available when ISSUE_INDEXER_TYPE is elasticsearch
376+
ISSUE_INDEXER_NAME = gitea_issues
373377
; Issue indexer storage path, available when ISSUE_INDEXER_TYPE is bleve
374378
ISSUE_INDEXER_PATH = indexers/issues.bleve
375379
; Issue indexer queue, currently support: channel, levelqueue or redis, default is levelqueue

‎docs/content/doc/advanced/config-cheat-sheet.en-us.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -228,8 +228,10 @@ relation to port exhaustion.
228228

229229
## Indexer (`indexer`)
230230

231-
- `ISSUE_INDEXER_TYPE`: **bleve**: Issue indexer type, currently support: bleve or db, if it's db, below issue indexer item will be invalid.
232-
- `ISSUE_INDEXER_PATH`: **indexers/issues.bleve**: Index file used for issue search.
231+
- `ISSUE_INDEXER_TYPE`: **bleve**: Issue indexer type, currently supported: `bleve`, `db` or `elasticsearch`.
232+
- `ISSUE_INDEXER_CONN_STR`: ****: Issue indexer connection string, available when ISSUE_INDEXER_TYPE is elasticsearch. i.e. http://elastic:changeme@localhost:9200
233+
- `ISSUE_INDEXER_NAME`: **gitea_issues**: Issue indexer name, available when ISSUE_INDEXER_TYPE is elasticsearch
234+
- `ISSUE_INDEXER_PATH`: **indexers/issues.bleve**: Index file used for issue search; available when ISSUE_INDEXER_TYPE is bleve and elasticsearch.
233235
- The next 4 configuration values are deprecated and should be set in `queue.issue_indexer` however are kept for backwards compatibility:
234236
- `ISSUE_INDEXER_QUEUE_TYPE`: **levelqueue**: Issue indexer queue, currently supports:`channel`, `levelqueue`, `redis`.
235237
- `ISSUE_INDEXER_QUEUE_DIR`: **indexers/issues.queue**: When `ISSUE_INDEXER_QUEUE_TYPE` is `levelqueue`, this will be the queue will be saved path.

‎docs/content/doc/advanced/config-cheat-sheet.zh-cn.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -89,7 +89,9 @@ menu:
8989

9090
## Indexer (`indexer`)
9191

92-
- `ISSUE_INDEXER_TYPE`: **bleve**: 工单索引类型，当前支持 `bleve` 或 `db`，当为 `db` 时其它工单索引项可不用设置。
92+
- `ISSUE_INDEXER_TYPE`: **bleve**: 工单索引类型，当前支持 `bleve`, `db` 和 `elasticsearch`，当为 `db` 时其它工单索引项可不用设置。
93+
- `ISSUE_INDEXER_CONN_STR`: ****: 工单索引连接字符串，仅当 ISSUE_INDEXER_TYPE 为 `elasticsearch` 时有效。例如: http://elastic:changeme@localhost:9200
94+
- `ISSUE_INDEXER_NAME`: **gitea_issues**: 工单索引名称，仅当 ISSUE_INDEXER_TYPE 为 `elasticsearch` 时有效。
9395
- `ISSUE_INDEXER_PATH`: **indexers/issues.bleve**: 工单索引文件存放路径，当索引类型为 `bleve` 时有效。
9496
- `ISSUE_INDEXER_QUEUE_TYPE`: **levelqueue**: 工单索引队列类型，当前支持 `channel`， `levelqueue` 或 `redis`。
9597
- `ISSUE_INDEXER_QUEUE_DIR`: **indexers/issues.queue**: 当 `ISSUE_INDEXER_QUEUE_TYPE` 为 `levelqueue` 时，保存索引队列的磁盘路径。

‎go.mod

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,7 @@ require (
7474
github.com/nfnt/resize v0.0.0-20160724205520-891127d8d1b5
7575
github.com/niklasfasching/go-org v0.1.9
7676
github.com/oliamb/cutter v0.2.2
77+
github.com/olivere/elastic/v7 v7.0.9
7778
github.com/pkg/errors v0.8.1
7879
github.com/pquerna/otp v0.0.0-20160912161815-54653902c20e
7980
github.com/prometheus/client_golang v1.1.0

‎go.sum

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,7 @@ github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5/go.mod h1:wHh0iHkY
6868
github.com/asaskevich/govalidator v0.0.0-20180720115003-f9ffefc3facf/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY=
6969
github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a h1:idn718Q4B6AGu/h5Sxe66HYVdqdGu2l9Iebqhi/AEoA=
7070
github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY=
71+
github.com/aws/aws-sdk-go v1.25.25/go.mod h1:KmX6BPdI08NWTb3/sm4ZGu5ShLoqVDhKgpiN924inxo=
7172
github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
7273
github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8=
7374
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
@@ -154,6 +155,8 @@ github.com/facebookgo/subset v0.0.0-20150612182917-8dac2c3c4870 h1:E2s37DuLxFhQD
154155
github.com/facebookgo/subset v0.0.0-20150612182917-8dac2c3c4870/go.mod h1:5tD+neXqOorC30/tWg0LCSkrqj/AR6gu8yY8/fpw1q0=
155156
github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568 h1:BHsljHzVlRcyQhjrss6TZTdY2VfCqZPbv5k3iBFa2ZQ=
156157
github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568/go.mod h1:xEzjJPgXI435gkrCt3MPfRiAkVrwSbHsst4LCFVfpJc=
158+
github.com/fortytw2/leaktest v1.3.0 h1:u8491cBMTQ8ft8aeV+adlcytMZylmA5nnwwkRZjI8vw=
159+
github.com/fortytw2/leaktest v1.3.0/go.mod h1:jDsjWgpAGjm2CA7WthBh/CdZYEPF31XHquHwclZch5g=
157160
github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I=
158161
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
159162
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
@@ -248,6 +251,7 @@ github.com/golang-sql/civil v0.0.0-20190719163853-cb61b32ac6fe h1:lXe2qZdvpiX5WZ
248251
github.com/golang-sql/civil v0.0.0-20190719163853-cb61b32ac6fe/go.mod h1:8vg3r2VgvsThLBIFL93Qb5yWzgyZWhEmBwUJWevAkK0=
249252
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
250253
github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
254+
github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
251255
github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
252256
github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
253257
github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFUx0Y=
@@ -322,6 +326,7 @@ github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 h1:BQSFePA1RWJOl
322326
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99/go.mod h1:1lJo3i6rXxKeerYnT8Nvf0QmHCRC1n8sfWVwXF2Frvo=
323327
github.com/jessevdk/go-flags v1.4.0 h1:4IU2WS7AumrZ/40jfhf4QVDMsQwqA7VEHozFRrGARJA=
324328
github.com/jessevdk/go-flags v1.4.0/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI=
329+
github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k=
325330
github.com/jmhodges/levigo v1.0.0 h1:q5EC36kV79HWeTBWsod3mG11EgStG3qArTKcvlksN1U=
326331
github.com/jmhodges/levigo v1.0.0/go.mod h1:Q6Qx+uH3RAqyK4rFQroq9RL7mdkABMcfhEI+nNuzMJQ=
327332
github.com/joho/godotenv v1.3.0 h1:Zjp+RcGpHhGlrMbJzXTrZZPrWj+1vfm90La1wgB6Bhc=
@@ -411,13 +416,16 @@ github.com/niklasfasching/go-org v0.1.9/go.mod h1:AsLD6X7djzRIz4/RFZu8vwRL0VGjUv
411416
github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U=
412417
github.com/oliamb/cutter v0.2.2 h1:Lfwkya0HHNU1YLnGv2hTkzHfasrSMkgv4Dn+5rmlk3k=
413418
github.com/oliamb/cutter v0.2.2/go.mod h1:4BenG2/4GuRBDbVm/OPahDVqbrOemzpPiG5mi1iryBU=
419+
github.com/olivere/elastic/v7 v7.0.9 h1:+bTR1xJbfLYD8WnTBt9672mFlKxjfWRJpEQ1y8BMS3g=
420+
github.com/olivere/elastic/v7 v7.0.9/go.mod h1:2TeRd0vhLRTK9zqm5xP0uLiVeZ5yUoL7kZ+8SZA9r9Y=
414421
github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
415422
github.com/onsi/ginkgo v1.7.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
416423
github.com/onsi/ginkgo v1.8.0 h1:VkHVNpR4iVnU8XQR6DBm8BqYjN7CRzw+xKUbVVbbW9w=
417424
github.com/onsi/ginkgo v1.8.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
418425
github.com/onsi/gomega v1.4.3/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
419426
github.com/onsi/gomega v1.5.0 h1:izbySO9zDPmjJ8rDjLvkA2zJHIo+HkYXHnf7eN7SSyo=
420427
github.com/onsi/gomega v1.5.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
428+
github.com/opentracing/opentracing-go v1.1.0/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o=
421429
github.com/openzipkin/zipkin-go v0.1.6/go.mod h1:QgAqvLzwWbR/WpD4A3cGpPtJrZXNIiJc5AZX7/PBEpw=
422430
github.com/pborman/uuid v1.2.0/go.mod h1:X/NO0urCmaxf9VXbdlT7C2Yzkj2IKimNn4k+gtPdI/k=
423431
github.com/pelletier/go-buffruneio v0.2.0/go.mod h1:JkE26KsDizTr40EUHkXVtNPvgGtbSNq5BcowyYOWdKo=
@@ -486,6 +494,7 @@ github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1
486494
github.com/smartystreets/assertions v0.0.0-20190116191733-b6c0e53d7304/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
487495
github.com/smartystreets/assertions v1.0.1 h1:voD4ITNjPL5jjBfgR/r8fPIIBrliWrWHeiJApdr3r4w=
488496
github.com/smartystreets/assertions v1.0.1/go.mod h1:kHHU4qYBaI3q23Pp3VPrmWhuIUrLW/7eUrw0BU5VaoM=
497+
github.com/smartystreets/go-aws-auth v0.0.0-20180515143844-0c1422d1fdb9/go.mod h1:SnhjPscd9TpLiy1LpzGSKh3bXCfxxXuqd9xmQJy3slM=
489498
github.com/smartystreets/goconvey v0.0.0-20181108003508-044398e4856c/go.mod h1:XDJAKZRPZ1CvBcN2aX5YOUTYGHki24fSF0Iv48Ibg0s=
490499
github.com/smartystreets/goconvey v0.0.0-20190330032615-68dc04aab96a/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA=
491500
github.com/smartystreets/goconvey v0.0.0-20190731233626-505e41936337 h1:WN9BUFbdyOsSH/XohnWpXOlq9NBD5sGAB2FciQMUEe8=
@@ -573,6 +582,7 @@ go.mongodb.org/mongo-driver v1.1.1/go.mod h1:u7ryQJ+DOzQmeO7zB6MHyr8jkEQvC8vH7qL
573582
go.opencensus.io v0.20.1/go.mod h1:6WKK9ahsWS3RSO+PY9ZHZUfv2irvY6gN279GOPZjmmk=
574583
go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
575584
go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8=
585+
go.opencensus.io v0.22.1/go.mod h1:Ap50jQcDJrx6rB6VgeeFPtuPIf3wMRvRfrfYDO6+BmA=
576586
go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
577587
go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0=
578588
go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=

‎integrations/mysql.ini.tmpl

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@ PASSWD = {{TEST_MYSQL_PASSWORD}}
1010
SSL_MODE = disable
1111

1212
[indexer]
13+
ISSUE_INDEXER_TYPE = elasticsearch
14+
ISSUE_INDEXER_CONN_STR = http://elastic:changeme@elasticsearch:9200
1315
ISSUE_INDEXER_PATH = integrations/indexers-mysql/issues.bleve
1416
REPO_INDEXER_ENABLED = true
1517
REPO_INDEXER_PATH = integrations/indexers-mysql/repos.bleve

‎modules/indexer/issues/bleve.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -170,7 +170,7 @@ func NewBleveIndexer(indexDir string) *BleveIndexer {
170170
}
171171
}
172172

173-
// Init will initial the indexer
173+
// Init will initialize the indexer
174174
func (b *BleveIndexer) Init() (bool, error) {
175175
var err error
176176
b.indexer, err = openIndexer(b.indexDir, issueIndexerLatestVersion)

‎modules/indexer/issues/elastic_search.go

Lines changed: 230 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,230 @@
1+
// Copyright 2019 The Gitea Authors. All rights reserved.
2+
// Use of this source code is governed by a MIT-style
3+
// license that can be found in the LICENSE file.
4+
5+
package issues
6+
7+
import (
8+
"context"
9+
"errors"
10+
"fmt"
11+
"strconv"
12+
"time"
13+
14+
"code.gitea.io/gitea/modules/log"
15+
16+
"github.com/olivere/elastic/v7"
17+
)
18+
19+
var (
20+
_ Indexer = &ElasticSearchIndexer{}
21+
)
22+
23+
// ElasticSearchIndexer implements Indexer interface
24+
type ElasticSearchIndexer struct {
25+
client *elastic.Client
26+
indexerName string
27+
}
28+
29+
type elasticLogger struct {
30+
*log.Logger
31+
}
32+
33+
func (l elasticLogger) Printf(format string, args ...interface{}) {
34+
_ = l.Logger.Log(2, l.Logger.GetLevel(), format, args...)
35+
}
36+
37+
// NewElasticSearchIndexer creates a new elasticsearch indexer
38+
func NewElasticSearchIndexer(url, indexerName string) (*ElasticSearchIndexer, error) {
39+
opts := []elastic.ClientOptionFunc{
40+
elastic.SetURL(url),
41+
elastic.SetSniff(false),
42+
elastic.SetHealthcheckInterval(10 * time.Second),
43+
elastic.SetGzip(false),
44+
}
45+
46+
logger := elasticLogger{log.GetLogger(log.DEFAULT)}
47+
48+
if logger.GetLevel() == log.TRACE || logger.GetLevel() == log.DEBUG {
49+
opts = append(opts, elastic.SetTraceLog(logger))
50+
} else if logger.GetLevel() == log.ERROR || logger.GetLevel() == log.CRITICAL || logger.GetLevel() == log.FATAL {
51+
opts = append(opts, elastic.SetErrorLog(logger))
52+
} else if logger.GetLevel() == log.INFO || logger.GetLevel() == log.WARN {
53+
opts = append(opts, elastic.SetInfoLog(logger))
54+
}
55+
56+
client, err := elastic.NewClient(opts...)
57+
if err != nil {
58+
return nil, err
59+
}
60+
61+
return &ElasticSearchIndexer{
62+
client: client,
63+
indexerName: indexerName,
64+
}, nil
65+
}
66+
67+
const (
68+
defaultMapping = `{
69+
"mappings": {
70+
"properties": {
71+
"id": {
72+
"type": "integer",
73+
"index": true
74+
},
75+
"repo_id": {
76+
"type": "integer",
77+
"index": true
78+
},
79+
"title": {
80+
"type": "text",
81+
"index": true
82+
},
83+
"content": {
84+
"type": "text",
85+
"index": true
86+
},
87+
"comments": {
88+
"type" : "text",
89+
"index": true
90+
}
91+
}
92+
}
93+
}`
94+
)
95+
96+
// Init will initialize the indexer
97+
func (b *ElasticSearchIndexer) Init() (bool, error) {
98+
ctx := context.Background()
99+
exists, err := b.client.IndexExists(b.indexerName).Do(ctx)
100+
if err != nil {
101+
return false, err
102+
}
103+
104+
if !exists {
105+
var mapping = defaultMapping
106+
107+
createIndex, err := b.client.CreateIndex(b.indexerName).BodyString(mapping).Do(ctx)
108+
if err != nil {
109+
return false, err
110+
}
111+
if !createIndex.Acknowledged {
112+
return false, errors.New("init failed")
113+
}
114+
115+
return false, nil
116+
}
117+
return true, nil
118+
}
119+
120+
// Index will save the index data
121+
func (b *ElasticSearchIndexer) Index(issues []*IndexerData) error {
122+
if len(issues) == 0 {
123+
return nil
124+
} else if len(issues) == 1 {
125+
issue := issues[0]
126+
_, err := b.client.Index().
127+
Index(b.indexerName).
128+
Id(fmt.Sprintf("%d", issue.ID)).
129+
BodyJson(map[string]interface{}{
130+
"id": issue.ID,
131+
"repo_id": issue.RepoID,
132+
"title": issue.Title,
133+
"content": issue.Content,
134+
"comments": issue.Comments,
135+
}).
136+
Do(context.Background())
137+
return err
138+
}
139+
140+
reqs := make([]elastic.BulkableRequest, 0)
141+
for _, issue := range issues {
142+
reqs = append(reqs,
143+
elastic.NewBulkIndexRequest().
144+
Index(b.indexerName).
145+
Id(fmt.Sprintf("%d", issue.ID)).
146+
Doc(map[string]interface{}{
147+
"id": issue.ID,
148+
"repo_id": issue.RepoID,
149+
"title": issue.Title,
150+
"content": issue.Content,
151+
"comments": issue.Comments,
152+
}),
153+
)
154+
}
155+
156+
_, err := b.client.Bulk().
157+
Index(b.indexerName).
158+
Add(reqs...).
159+
Do(context.Background())
160+
return err
161+
}
162+
163+
// Delete deletes indexes by ids
164+
func (b *ElasticSearchIndexer) Delete(ids ...int64) error {
165+
if len(ids) == 0 {
166+
return nil
167+
} else if len(ids) == 1 {
168+
_, err := b.client.Delete().
169+
Index(b.indexerName).
170+
Id(fmt.Sprintf("%d", ids[0])).
171+
Do(context.Background())
172+
return err
173+
}
174+
175+
reqs := make([]elastic.BulkableRequest, 0)
176+
for _, id := range ids {
177+
reqs = append(reqs,
178+
elastic.NewBulkDeleteRequest().
179+
Index(b.indexerName).
180+
Id(fmt.Sprintf("%d", id)),
181+
)
182+
}
183+
184+
_, err := b.client.Bulk().
185+
Index(b.indexerName).
186+
Add(reqs...).
187+
Do(context.Background())
188+
return err
189+
}
190+
191+
// Search searches for issues by given conditions.
192+
// Returns the matching issue IDs
193+
func (b *ElasticSearchIndexer) Search(keyword string, repoIDs []int64, limit, start int) (*SearchResult, error) {
194+
kwQuery := elastic.NewMultiMatchQuery(keyword, "title", "content", "comments")
195+
query := elastic.NewBoolQuery()
196+
query = query.Must(kwQuery)
197+
if len(repoIDs) > 0 {
198+
var repoStrs = make([]interface{}, 0, len(repoIDs))
199+
for _, repoID := range repoIDs {
200+
repoStrs = append(repoStrs, repoID)
201+
}
202+
repoQuery := elastic.NewTermsQuery("repo_id", repoStrs...)
203+
query = query.Must(repoQuery)
204+
}
205+
searchResult, err := b.client.Search().
206+
Index(b.indexerName).
207+
Query(query).
208+
Sort("id", true).
209+
From(start).Size(limit).
210+
Do(context.Background())
211+
if err != nil {
212+
return nil, err
213+
}
214+
215+
hits := make([]Match, 0, limit)
216+
for _, hit := range searchResult.Hits.Hits {
217+
id, _ := strconv.ParseInt(hit.Id, 10, 64)
218+
hits = append(hits, Match{
219+
ID: id,
220+
})
221+
}
222+
223+
return &SearchResult{
224+
Total: searchResult.TotalHits(),
225+
Hits: hits,
226+
}, nil
227+
}
228+
229+
// Close implements indexer
230+
func (b *ElasticSearchIndexer) Close() {}

0 commit comments

Comments
 (0)