Skip to content

优化 性能 - #626

Merged
wonderflow merged 3 commits into
qiniu:masterfrom
wonderflow:opitimizex
Jul 23, 2018
Merged

wonderflow merged 3 commits into
qiniu:masterfrom
wonderflow:opitimizex

Conversation

@wonderflow

@wonderflow wonderflow commented Jul 22, 2018 •

Copy link
Copy Markdown
Contributor

Fixes [issue number]

Changes

  • ip transformer (prefix key先算好)
  • Pandora Key优化,加上cachemap
  • 其他transformer中getkeys能先算好的用Init计算

Reviewers

  • @[someone] please review
  • @[someotherone] please review

Wiki Changes

  • options1...
  • options2...

Checklist

  • Rebased/mergeable
  • Tests pass
  • Wiki updated

Comment thread transforms/ip/ip.go Outdated
keysLatitude []string
keysLongitude []string
keysDistrictCode []string
stats StatsInfo

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这部分为了性能的字段建议单独放一起(空行分隔),然后加个注释简单说一下方便以后维护的人

Comment thread transforms/ip/ip_test.go Outdated
Key: "ip",
DataPath: "./test_data/17monipdb.dat",
}
ipt.Init()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

assert.Nil(t, ipt.Init())

下同

Comment thread transforms/ip/ip_test.go Outdated
//},
//{
// "ip2": "183.251.28.250",
//},

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里注释掉的如果是不用的话,直接删了吧,后来的维护会很迷惑又不敢乱动

Comment thread utils/models/utils.go
// 2. 首字符为非法字符时,去掉首字符(例如,如果字符串全为非法字符,则转换后为空)
// 3. 非首字符并且为非法字符时,使用 "_" 替代非法字符
func PandoraKey(key string) (string, bool) {
func CheckPandoraKey(key string) bool {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个函数补一下是做什么用的说明吧

Comment thread utils/models/utils.go Outdated
}

/*
//1000000 1584 ns/op 32 B/op 2 allocs/op

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

留个注释,就说如果确认新版有性能和内存上的优化的话,就可以删掉了。或者 git 本来就有历史,直接删了也能找回的

Comment thread utils/models/utils_test.go Outdated
func BenchmarkCheckPandoraKey(b *testing.B) {
b.ReportAllocs()
testKeys := []string{"@timestamp", ".dot", "percent%100", "^^^^^^^^^^", "timestamp", "aaa"}
//testKeys := []string{"timestamp", "aaa"}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

同上

Comment thread utils/models/utils_test.go Outdated
},
{
"timestamp": "2018-07-18T10:17:36.549054846+08:00",
//"timestamp": "2018-07-19T10:17:36.549054846+08:00",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

同上

},
{
"timestamp": "2018-07-18T10:17:36.549054846+08:00",
//"timestamp": "2018-07-19T10:17:36.549054846+08:00",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

同上

Comment thread utils/models/utils_test.go Outdated
},
{
"timestamp": "2018-07-18T10:17:36.549054846+08:00",
//"timestamp": "2018-07-19T10:17:36.549054846+08:00",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

同上

Comment thread utils/models/utils.go Outdated
}
if valid {
return key, true
return []byte(key), true

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

来回转换 bytes 到 string 从逻辑上来讲其实反而多分配了一次 []byte 吧?上层也是仍旧要转回 string 类型而不能直接使用 []byte

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

我觉得返回 *string 是可以考虑的?

@wonderflow wonderflow changed the title 优化ip transformer 性能 优化 性能 Jul 22, 2018
Comment thread sender/fault_tolerant.go Outdated
writeLimit, _ := conf.GetIntOr(KeyFtWriteLimit, defaultWriteLimit)
strategy, _ := conf.GetStringOr(KeyFtStrategy, KeyFtStrategyBackupOnly)
longDataDiscard, _ := conf.GetBoolOr(KeyFtLongDataDiscard, false)
sendType, _ := conf.GetStringOr(KeySenderType, "") //此处不会没有SenderType,在调用NewFtSender时已经检查

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/sendType/senderType/

Comment thread sender/fault_tolerant.go Outdated
memoryChannel: memoryChannel,
memoryChannelSize: memoryChannelSize,
longDataDiscard: longDataDiscard,
innerSendType: sendType,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/innerSendType/innerSenderType/

@unknwon

unknwon commented Jul 23, 2018

Copy link
Copy Markdown
Contributor

LGTM

@wonderflow
wonderflow merged commit 8646fcf into qiniu:master Jul 23, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

⌂ 换网址