Performance/Meetings/2008 07
Performance/Meetings/2008/07/31
Meeting Minutes
IRC Meeting of Sun Microsystems (StarOffice) with RedFlag2000 and the Community ??
Performance Project
Date: 2008/07/31
Time: 15:38– 17:12
Meeting No.:
Agenda:
(15:58:35) liheng: hi xiuzhi
(15:59:11) xiuzhi: liheng:hi afternoon
(16:00:16) xiuzhi: I suggested that we can have bi-weekly meeting if we have no more issues to discuss.
(16:02:01) liheng: xiuzhi:OK, i am agree :)
(16:03:04) mhu: Hi all
(16:03:08) xiuzhi: liheng::)
(16:03:22) xiuzhi: mhu: moin
(16:03:24) liheng: hi: matthias
(16:03:28) liangjun: mhu: hi:)
(16:03:34) kuangliang: hi,mhu
(16:03:44) mhu: moin :-)
(16:05:20) yugq: hi
(16:05:20) liheng: mhu:xiuzhi suggested that we can have bi-weekly meeting in the future , what is your opinion?
(16:05:21) mhu: Okay, I have read the agenda; you have a new team member? will he join our meeting also?
(16:06:19) liheng: Guo yongshi, jion this meeting next time, today, he have another meeting of our company
(16:06:19) liheng: :)
(16:06:20) mhu: liheng: I think, having a meeting every two weeks only is fine when we are all busy working.
(16:07:25) mhu: (also: I will be out on vacation for the next two weeks; so next meeting would be in 3 weeks)
(16:07:51) liheng: I can have a meeting every week, yugq , kuangliang, liangjun also, but xiuzhi maybe so busy :)
(16:08:23) xiuzhi: liheng: not true. I means if we have not too many issues to discuss
(16:08:36) mhu: When we still have so much to discuss, as we did in the past, I think it is okay to have a meeting every week.
(16:09:00) xiuzhi: liheng: weekly meeting is ok for me
(16:09:09) mhu: xiuzhi: yes, that is what I also mean.
(16:09:53) liheng: Okay, we keep weekly meeting
(16:10:43) liheng: xiuzhi: Okay, and the proposal of paper "Improving and Maintaining Performance of OO in Long-term Development" has been passed selection,
(16:11:14) xiuzhi: xiuzhi: great
(16:11:29) xiuzhi: liheng: great
(16:11:36) mhu: yes, I think that is great also
(16:11:54) xiuzhi: liheng: and OO -> OOo
(16:12:00) liheng: we need more discussion for tools and analysis of performance of OOo
(16:12:05) liheng: Yes, OOo:)
(16:12:27) liheng: Peter has mentioned to me:)
(16:13:59) mhu: okay, discusssion of benchmarking tools... do have any ideas?
(16:14:13) mhu: s/do have/do you have/
(16:15:00) liheng: mhu:In your opinion, what are urgent improvement of Tuning and Benchmark tools?
(16:16:33) mhu: well, I think we once discussed that first of all we need a set of "standard and reproduceable" measurements / tests.
(16:17:15) mhu: ...say: how long it takes to load (and display?) a particular reference document.
(16:18:30) mhu: ...this would be the macroscopic / user relevant measurement.
(16:18:50) liheng: Yes, for a long time most raw-data will be comparability and explicit
(16:20:07) liheng: please explain "how long it takes to load (and display?) a particular reference document." :(
(16:20:29) mhu: what I am thinking about is something like: Y=f(x1, x2, ...,xN); where Y is the user visible / macroscopic value, and the x's are the variables that cause it
(16:20:58) mhu: liheng: okay, I will explain...
(16:22:31) mhu: I said: "...set of reproduceable measurements"; one measurement out of that set of measurements could be a measurement of the time it takes to load one of the (yet to be defined) reference documents.
(16:22:42) mhu: does this make it clearer?
(16:24:10) liheng: you mean the individual measurements in specified parts?
(16:24:36) liheng: for load a reference documents?
(16:25:01) mhu: not yet, measurements of specified parts may be measurements of those x1, ...,xN's
(16:25:46) mhu: in this example, I meant a simple number, say: 5 seconds
(16:25:51) yugq: mhu: about Y=f(x1, x2, ...,xN), may I understand that like: Y=speed, x1=startup speed, x2=load/save document speed, x3=menu shown speed ... ?
(16:25:59) liheng: If we have enough manpower we can not only define the f(x1.......xN) but also paint a big picture for key paths and hotspots
(16:27:48) mhu: yugq: well, almost; Y would the total load time, say 5 seconds, and x1 may be xml parsing speed, x2 memory allocation speed, x3 locking speed, and so on; f() would basically tell how these factors contribute to the end result Y.
(16:28:09) mhu: liheng: yes, that is the direction that I am thinking :-)
(16:30:23) liheng: after them, we must define effectual and comparability benchmarks for all node (x1.....xN)
(16:30:28) liheng: right?
(16:31:37) mhu: yes, probably; but hopefully not for so many, but only the most important ones; that is, for those that contribute to many Y's (like xml parsing, memory alloc, ...).
(16:34:18) liheng: For a long time, I think for create a set of tools to automatically analyze data of benchmark and compare them between several versions
(16:35:12) liheng: mhu:Network seems slowly:(
(16:35:31) mhu: so, yugq already mentioned: startup speed, load/save speed, menu open speed; that would be the different macroscopically observable Y's in my world; they may all depend on different x's, but may also depend on a set of same x's.
(16:36:18) mhu: liheng: I can wait, dont worry too much about your network
(16:37:26) liheng: mhu:For a long time, I think of create a set of tools to automatically analyze data of benchmark and compare them between several versions
(16:37:35) mhu: ...that set of same x's (contributing to many Y's) would be the ones that I would create individual benchmarks for, to start with.
(16:38:07) mhu: oh yes, I forgot to answer that one, stay tuned...
(16:38:25) mhu: ..set of tools to automatically analyse...
(16:39:35) mhu: ...I think of that can be done with a "T-Test" called statistical method (testing statistically difference between two datasets), do you mean someting like this?
(16:41:37) mhu: sorry for all this math; I have had a course in "six sigma methodology", and in my previous life I have been a physicist :-)
(16:41:44) liheng: Yes, most of, and all data can be traced in difference versions
(16:42:02) mhu: yes
(16:44:14) liheng: Yes, now we need a start point to try which tools can do that
(16:44:14) liheng: :)
(16:45:24) mhu: well, tooling would be two parts: one part to collect the data (maybe that is what you have from Malte?), the other part to analyze it.
(16:46:02) liheng: Three parts in my mind, testcase, benchmarks, method of report
(16:46:52) mhu: if you can produce a set of measurements, say: measure a documents load time 10 times in a row, for 2 versions of OOo, than I can provide you with the code to do a T-Test of that datasets.
(16:47:58) mhu: yes, you right; but at this high level, I think report can initially be "the diff is 5% and it is statistically significant" or so.
(16:48:17) liheng: i am not clear of "than I can provide you with the code to do a T-Test of that datasets."
(16:48:20) liheng: :)
(16:48:52) mhu: I have written a small set of Java methods some years ago; I can give you the code.
(16:49:27) mhu: meaning you dont need to invent the math yourself
(16:50:05) liheng: oh, yes thank you,
(16:50:22) mhu: you are welcome
(16:50:46) liheng: i have a mistake for 'code' and 'datasets'
(16:50:47) liheng: :)
(16:51:37) mhu: TTest.java = code, {5,4,4,5,...7,5} = dataset
(16:51:42) mhu: :-) :-)
(16:52:39) liheng: i mean the report must can lead more developers to notice some problem of performance
(16:53:51) liheng: one target is tuning for every version ,and another is refactoring in the future.
(16:54:01) mhu: well, yes. on the other hand, the numbers of course must be able to speak for them selves; than we can think of how to present them; but yes, presentation is important also.
(16:55:14) mhu: yes, we agreed on that already: tuning for a version is short-term only; the real goal is indeed long-term (refactoring) improvements.
(16:57:37) liheng: will we discuss a example in OOo performance (maybe common hotspot), to define a set of benchmark
(16:58:51) mhu: back to tooling: maybe, for reporting we can make a small extension / addin for Calc to do the T-Test between two datasets; then we would have an initial presentation also ? just an idea.
(16:59:28) mhu: sorry, I needed to finish that idea first; what was your question wrt "example..."
(17:00:17) liheng: a sample for a hotspot or a key path
(17:01:17) mhu: and you want to discuss what about that sample? sorry, I have not yet understood
(17:01:47) xiuzhi: mhu: study a case in formance
(17:02:02) xiuzhi: liheng: does that your meaning?
(17:02:24) mhu: oh yes, we should study a case to learn what is going on and how to analyze and present it, yes.
(17:03:05) liheng: xiuzhi:thank you, i just try to find clear words:)
(17:03:33) mhu: maybe, for practical purposes, we start with a configmgr example?
(17:04:31) liheng: mhu:Of course!!!;) back to tooling: I want to setup a server for datasets:)
(17:04:34) mhu: that could be: Y=configmgr contribution to startup time, and we could try to find the major x's (hotspots) contributing to that Y by some f().
(17:05:23) mhu: yes, that server to store the measurement results was a good idea.
(17:06:48) mhu: ...we could also so a "synthetic" configmgr benchmark, not use startup time of entire office; I think yugq already wrote some "unit test" or "benchmark" ?
(17:07:26) yugq: mhu: yes
(17:07:46) liheng: so, it's good for persisting all data of teat
(17:07:57) liheng: sorry teat/test
(17:08:37) mhu: yes, if we want to learn something from the data, and other developers shall learn something of it, then it is good to store that data.
(17:09:07) liheng: okay, :)
(17:09:30) mhu: okay
(17:10:08) liheng: mhu:that all for me.
(17:10:27) mhu: yes, that's all from me also
(17:10:33) liheng: I wish you have a nice holiday
(17:10:41) mhu: thank you
(17:11:10) liheng: you are welcome, and see you at Aug 21th
(17:11:36) mhu: see you in three weeks, august 21.
(17:11:45) mhu: bye all
(17:11:47) xiuzhi: see you
(17:11:51) liheng: bye:)
(17:12:04) kuangliang: bye
(17:12:27) liangjun: bye
