Skip to content

http reader support multiple prefix - #678

Merged
wonderflow merged 1 commit into
qiniu:masterfrom
mockerzzz:prefix_socket_reader
Aug 8, 2018
Merged

wonderflow merged 1 commit into
qiniu:masterfrom
mockerzzz:prefix_socket_reader

Conversation

@mockerzzz

@mockerzzz mockerzzz commented Jul 31, 2018 •

Copy link
Copy Markdown
Contributor

@mockerzzz
mockerzzz force-pushed the prefix_socket_reader branch 5 times, most recently from 531a518 to d05b58f Compare August 6, 2018 05:13
Comment thread mgr/dataflow_test.go
"senders": senders,
}

go func() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

下面没等这个进程跑起来就会结束

Comment thread reader/http/http_test.go
wg.Add(1)
go func() {
got, err := httpReader.ReadLine()
assert.NoError(t, err)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

goroutine里面用到的变量,如果没有当作参数传进去,就会出现并发错误,此处要把httpReader和t当作参数传入函数。

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

下同

Comment thread reader/http/http.go Outdated
}
log.Debugf("Runner[%v] %q daemon is stopping", r.meta.RunnerName, r.Name())

close(r.readChan)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里在http 仍然在serve的时候就close channel,可能会导致panic, send on closed channel

@mockerzzz
mockerzzz force-pushed the prefix_socket_reader branch 3 times, most recently from 88df7d1 to 150500c Compare August 6, 2018 06:45
Comment thread reader/http/http.go

r.server.Shutdown(context.Background())
err := r.bufQueue.Close()
close(r.readChan)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里要确认一下,如果http server在shutdown的之前,postData函数被阻塞了,比如数据来不及读导致channel传不进去数据,那么这里close会导致panic吗

@mockerzzz
mockerzzz force-pushed the prefix_socket_reader branch from 150500c to 3768624 Compare August 6, 2018 09:19
Comment thread reader/http/http.go
Content: line,
Path: path,
}
}

This comment was marked as resolved.

@wonderflow

Copy link
Copy Markdown
Contributor

尝试实际运行起来,然后用大量的请求去请求这个http reader,然后请求不要停,紧接着logkit关掉,看看现象

Comment thread reader/http/http.go Outdated
"github.com/qiniu/logkit/queue"
"github.com/qiniu/logkit/reader"
. "github.com/qiniu/logkit/utils/models"
"sync"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

移到最上面的分组

Comment thread reader/http/http.go
func NewReader(meta *reader.Meta, conf conf.MapConf) (reader.Reader, error) {
address, _ := conf.GetStringOr(reader.KeyHTTPServiceAddress, reader.DefaultHTTPServiceAddress)
path, _ := conf.GetStringOr(reader.KeyHTTPServicePath, reader.DefaultHTTPServicePath)
paths := strings.Split(path, ",")

This comment was marked as resolved.

@mockerzzz
mockerzzz force-pushed the prefix_socket_reader branch from 3768624 to 6336754 Compare August 8, 2018 01:29
Comment thread reader/http/http.go Outdated
if val[0] == '/' {
validPaths = append(validPaths, val)
} else {
log.Warnf("path[%v] not begin with '/',this path ignored", val)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里直接错误返回吧,不用Warn,反正是初始配置的时候

@mockerzzz
mockerzzz force-pushed the prefix_socket_reader branch from 6336754 to 87c42d4 Compare August 8, 2018 02:46
Comment thread reader/http/http.go Outdated
path, _ := conf.GetStringOr(reader.KeyHTTPServicePath, reader.DefaultHTTPServicePath)
paths := strings.Split(path, ",")
for _, val := range paths {
if val[0] != '/' {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

你这样写如果val是空就panic了,先trimeSpace,空的continue,然后检查 strings.HasPrefix()

@mockerzzz
mockerzzz force-pushed the prefix_socket_reader branch from 87c42d4 to 2a94417 Compare August 8, 2018 03:15
Comment thread reader/http/http.go
path, _ := conf.GetStringOr(reader.KeyHTTPServicePath, reader.DefaultHTTPServicePath)
paths := strings.Split(path, ",")
for _, val := range paths {
if !strings.HasPrefix(val, "/") {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

空的过滤一下

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

考虑: /api,/ap2,,

@wonderflow

Copy link
Copy Markdown
Contributor

文档也对应更新一下 mars上

@mockerzzz
mockerzzz force-pushed the prefix_socket_reader branch from 2a94417 to 488b480 Compare August 8, 2018 04:00
Comment thread reader/http/http.go Outdated

r.server.Shutdown(context.Background())
err := r.bufQueue.Close()
//Note:确保不会往已关闭的channel中写入message

@unknwon unknwon Aug 8, 2018 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里描述不对吧,应该是和 wg.Done 一个作用,因为是 wait 完才 close 的 channel

@wonderflow

Copy link
Copy Markdown
Contributor

LGTM

@mockerzzz
mockerzzz force-pushed the prefix_socket_reader branch from 488b480 to 2462fc8 Compare August 8, 2018 04:15
@unknwon

unknwon commented Aug 8, 2018

Copy link
Copy Markdown
Contributor

LGTM

@wonderflow
wonderflow merged commit c7f9ff7 into qiniu:master Aug 8, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

⌂ 换网址