A series of MF CT pub­lic­a­tion was sim­u­lated in sim 3.0 from to with step in­ter­val us­ing best over­all strategy (MFdCoast Depth: 0 : xxx, MFd2­Coast Depth: 0 : xxx and MFd3­Coast Depth: 0 :xxx) in depth 0 (No brute­force). Due to the pur­pose of this in­vest­ig­a­tion and my tech­nical dif­fi­culties, in­vest­ig­a­tion in­volving a higher depth was not used. Then, data set entries con­sist of the de­tail of re­set po­s­i­tions, vari­able pur­chase po­s­i­tions, gain of of each of the 601 pub­lic­a­tions are ob­tained.

Data sets re­lated to re­sets are isol­ated from the crude list to in­vest­ig­ate the pat­tern of re­set in each of the pub­lic­a­tion. The gain between re­set is cal­cu­lated and the re­set data sets are ex­cluded if the gain yields a res­ult of or above, in which most of the data set are caused by rapid changes of and the re­set pat­tern can­not be fol­lowed strictly by the sim­u­lator due to its ma­nip­u­la­tion lim­it­a­tion at the very start of a pub­lic­a­tion.

Sub­sequently, data sets re­lated to re­sets, as well as the data entry which con­sist of a “Be­fore”, a re­set, and a “After” is isol­ated for de­term­in­a­tion of a cri­terion to com­pare the cost of levels and the cost of re­set us­ing ra­tio (i.e., “re­l­at­ive cost ra­tio” between , and re­set cost be­fore a re­set.). The set of entry is in­cluded only if all three above-men­tioned ele­ments is in­cluded (i.e. a “Be­fore”, a re­set, and a “After”). For ex­ample, with the ref­er­ence of Table 1 where the “Be­fore”, a re­set, and the “After” had been isol­ated. The re­l­at­ive cost ra­tio of “Be­fore” com­pared to re­set cost can be cal­cu­lated by , while the re­l­at­ive cost ra­tio of “After” com­pared to re­set cost can be cal­cu­lated by .

To de­term­ine the strength of a cost cri­terion as a cutoff, a Re­ceiver-op­er­at­ing Char­ac­ter­istic Curve (ROC) is plot­ted us­ing a series of cutoffs and the Area Un­der Curve (AUC) of the ROC is cal­cu­lated. The ex­act cutoff po­s­i­tion will be then de­term­ined by ex­plor­ing the list of “Be­fore” and “After” data with the aid of a box and whisker dia­gram. The ap­proach is re­peated for , , , and . More de­tailed ex­plan­a­tions on the nature and in­ter­pret­a­tion of graphs will be ex­plained sub­sequently.

Lastly, the re­set pat­tern and the gain of each of the 601 pub­lic­a­tions will be re­trieved for ana­lysis of pub­lic­a­tion be­ha­vior and ex­plore the re­la­tion­ship between of pub­lic­a­tion and the re­set pat­tern used, which will be use­ful for pre­dict­ing the pat­tern used in a pub­lic­a­tion without re­fer­ring to the sim­u­lator.

A Re­ceiver-op­er­at­ing Char­ac­ter­istic (ROC) curve is a com­mon graph­ical plot visu­al­iz­ing the per­form­ance of a bin­ary clas­si­fier by plot­ting True Pos­it­ive Rate (Sens­it­iv­ity; ) against False Pos­it­ive Rate (1 – Spe­cificity; ) across all clas­si­fic­a­tion thresholds, mainly in ma­chine learn­ing and the med­ical field. It demon­strates the trade-off between sens­it­iv­ity and spe­cificity, with higher curves in­dic­at­ing bet­ter clas­si­fic­a­tion ac­cur­acy.

With the aid of the above concept, we can design a threshold which is to ef­fect­ively and cor­rectly sep­ar­ate the list of vari­ables pur­chased be­fore and after a re­set by their re­l­at­ive cost ra­tio (e.g., cost di­vided by re­set cost). Take an ex­ample of as the vari­able and as the re­l­at­ive cost ra­tio threshold, the in­ter­pret­a­tion of res­ult can be ref­er­enced by Table 2 and a set of data re­gard­ing “True Pos­it­ive Rate” and “False Pos­it­ive Rate” , or a point of ROC Curve, can be ob­tained.

Next, re­peat the cal­cu­la­tion with a vari­ety of re­l­at­ive cost ra­tio threshold and ob­tain a set of data points re­gard­ing “True Pos­it­ive Rate” and “False Pos­it­ive Rate”. Plot­ting the dots in a graph will yield a ROC Curve. The Area Un­der the Curve (AUC) of a ROC curve is cal­cu­lated to meas­ure over­all per­form­ance. An AUC of a ROC of in­dic­ates a per­fect iden­ti­fic­a­tion of “Pos­it­ive” and “Neg­at­ive” ex­ists, while the AUC of a ROC of is no bet­ter than ran­dom guess­ing. The lar­ger the AUC of a ROC, the bet­ter the per­form­ance of a clas­si­fic­a­tion threshold, if ap­pro­pri­ately set. In sum­mary, one can in­ter­pret a ROC Curve as in­dic­ated in Graph 1.

There are three ap­proaches to de­term­ine the pre­ferred threshold when no def­in­ite threshold can be es­tab­lished (i.e. AUC of ROC Curve less than ), namely Youden’s In­dex ap­proach, min­imal dis­tance ap­proach (MDA), and weighted ap­proach. The Youden’s In­dex is cal­cu­lated by Sens­it­iv­ity Spe­cificity (i.e., Sens­it­iv­ity “False Pos­it­ive rate”). Then the threshold will be de­term­ined when Youden’s In­dex reached max­imum.

The min­imal dis­tance ap­proach cal­cu­lates the dis­tance between dot plots of ROC Curve and an ima­gin­ary ideal situ­ation (i.e., on a ROC plot) via. the for­mula , The threshold will be de­term­ined when the dis­tance between the two men­tioned point is at min­imum. The fi­nal ap­proach takes ac­count of weighted factors in each situ­ation, such as the time loss due to “in­cor­rect” vari­able pur­chases. Since the factors and the weight­ing of each factor is sub­ject­ive and can dif­fer from user to user, this method will be omit­ted in sub­sequent con­sid­er­a­tions of re­l­at­ive cost ra­tio threshold.

A Box and whisker dia­gram is an­other com­mon graphic plot visu­al­iz­ing the dis­per­sion of a sets of dis­crete data, as well as the range, Lower Quart­ile (Q1), Me­dian (Q2), Up­per Quart­ile (Q3) and out­liers, if any, of a data set in stat­ist­ical ana­lysis. In this in­vest­ig­a­tion, a box and whisker dia­gram will il­lus­trate the spread of data set, as well as giv­ing an ap­prox­im­ate pic­ture of how a re­l­at­ive cost ra­tio threshold per­form when com­pared to sim 3.0. With a box and whisker dia­gram, one can in­ter­pret as be­low in Graph 2.

A total of data sets re­lated to re­sets are isol­ated from the crude list cor­res­pond­ing to the of pub­lic­a­tion to in­vest­ig­ate the pat­tern of re­set in each of the pub­lic­a­tion. There are mainly three types of re­set pat­terns used in dif­fer­ent MF CT pub­lic­a­tion , they are re­set , re­set , and / re­set . The use of three types of re­set pat­terns in their re­spect­ive of pub­lic­a­tion is presen­ted be­low (Graph 3), ex­clud­ing out­liers, one can sum­mar­ize that re­set are the main pat­tern used be­fore , then the pat­tern gradu­ally shifts to re­set and be­come the main­stream pat­tern after . The pat­tern con­tin­ues un­til , when re­set star­ted to be in­creas­ingly used un­til . The trend of us­age of re­set pat­tern is il­lus­trated in Graph 4.

A total of sets of data entries con­sist­ing of a “Be­fore”, a re­set, and a “After” were iden­ti­fied from the crude data set and the re­l­at­ive cost ra­tio of “Be­fore” and “After” were cal­cu­lated and com­pared. A ROC Curve were plot­ted (Graph 5) and AUC were cal­cu­lated to be in­dic­at­ing a mod­er­ate-strength threshold, if ap­pro­pri­ately set, ex­ists for cost vs. re­set cost with a con­sid­er­able num­ber of in­con­sist­en­cies. The ac­cur­acy of identi­fy­ing “Be­fore” and “After” us­ing a series of re­l­at­ive cost ra­tio thresholds ran­ging from to with in­ter­val of and the cor­res­pond­ing Youden’s In­dex and MDA are dis­played in Graph 6 and Graph 7 re­spect­ively.

While Youden’s In­dex achieves max­imum when re­l­at­ive cost ra­tio threshold is set at (Sim­ilar value in­ter­val ), dis­tance from ideal via. MDA reaches min­imum at re­l­at­ive cost ra­tio threshold of (Sim­ilar value in­ter­val ). Since Youden’s In­dex meas­ures the greatest ver­tical dis­tance between ROC curve and ran­dom guess­ing line (i.e., straight line con­nect­ing and of a ROC curve), us­ing the lower re­l­at­ive cost ra­tio threshold cal­cu­lated with Youden’s In­dex Ap­proach en­sures more “After” to be iden­ti­fied while the ac­cur­acy of identi­fy­ing “Be­fore” is sig­ni­fic­antly lower . On the other hand, us­ing threshold from MDA usu­ally en­sures a bal­anced res­ult, es­pe­cially for skewed data. In this case, both identi­fy­ing “Be­fore” and “After” are con­sid­er­ably ap­pro­pri­ate with the threshold of cal­cu­lated above.

A not­able men­tion is the min­imum dis­tance cal­cu­lated in MDA re­mains at a low plat­eau from re­l­at­ive cost ra­tio to , in re­sponse to this, two ad­di­tional re­l­at­ive cost ra­tios of and are eval­u­ated. For re­l­at­ive cost ra­tio of , ac­curacies of identi­fy­ing “Be­fore” and “After” be­ing and , while those for re­l­at­ive cost ra­tio of are and re­spect­ively. A box and whisker dia­gram com­par­ing “Be­fore” and “After” is il­lus­trated be­low as sup­ple­ment­ary in­form­a­tion (Graph 8).

A total of sets of data entries con­sist­ing of a “Be­fore”, a re­set, and a “After” were iden­ti­fied from the crude data set and the re­l­at­ive cost ra­tio of “Be­fore” and “After” were cal­cu­lated and com­pared. A ROC Curve was plot­ted (Graph 9) and AUC were cal­cu­lated to be in­dic­at­ing a per­fect threshold ex­ists for cost vs. re­set cost. With ref­er­ence to a box and whisker dia­gram com­par­ing “Be­fore” and “After” (Graph 10), a sub­sequent threshold of re­l­at­ive cost ra­tio was set to be and the ac­cur­acy of identi­fy­ing “Be­fore” and “After” were and re­spect­ively. This res­ult can be in­ter­preted as a threshold of re­l­at­ive cost ra­tio is def­in­ite for cost vs re­set cost.

A total of sets of data entries con­sist­ing of a “Be­fore”, a re­set, and a “After” were iden­ti­fied from the crude data set and the re­l­at­ive cost ra­tio of “Be­fore” and “After” were cal­cu­lated and com­pared. A ROC Curve were plot­ted (Graph 11) and AUC were cal­cu­lated to be in­dic­at­ing a mod­er­ate-to-good-strength threshold, if ap­pro­pri­ately set, ex­ists for cost vs. re­set cost with a con­sid­er­able num­ber of in­con­sist­en­cies. Since the re­l­at­ive cost ra­tio of spans across a con­sid­er­able range of mag­nitudes, the re­l­at­ive cost ra­tio has been con­ver­ted to to fa­cil­it­ate easier com­par­is­ons. The ac­cur­acy of identi­fy­ing “Be­fore” and “After” us­ing a series of re­l­at­ive cost ra­tio thresholds ran­ging from to with in­ter­val of and the cor­res­pond­ing Youden’s In­dex and MDA are dis­played in Graph 12 and Graph 13 re­spect­ively. A sim­ilar meth­od­o­logy de­scribed in “ cost vs re­set cost” sec­tion was ad­op­ted. Both Youden’s In­dex Ap­proach and MDA gives a con­sensus of an re­l­at­ive cost ra­tio threshold at , which in­dic­ates for the ac­tual re­l­at­ive cost ra­tio threshold. In this case, both ac­curacies in identi­fy­ing “Be­fore” and “After” are re­l­at­ively good with the threshold of cal­cu­lated above. With a box and whisker dia­gram com­par­ing “Be­fore” and “After” il­lus­trated be­low (Graph 14), it may be more sens­ible to set a threshold of , in which achieve an ac­cur­acy of for identi­fy­ing “Be­fore” and for identi­fy­ing “After”.

A total of sets of data entries con­sist­ing of a “Be­fore”, a re­set, and a “After” were iden­ti­fied from the crude data set and the re­l­at­ive cost ra­tio of “Be­fore” and “After” were cal­cu­lated and com­pared. A ROC Curve was plot­ted (Graph 15) and AUC were cal­cu­lated to be in­dic­at­ing a per­fect threshold ex­ists for cost vs. re­set cost. With ref­er­ence to a box and whisker dia­gram com­par­ing “Be­fore” and “After” (Graph 16), a sub­sequent threshold of re­l­at­ive cost ra­tio was set to be and the ac­cur­acy of identi­fy­ing “Be­fore” and “After” were and re­spect­ively. This res­ult can be in­ter­preted as a threshold of re­l­at­ive cost ra­tio is def­in­ite cost vs. re­set cost.

A total of sets of data entries con­sist­ing of a “Be­fore”, a re­set, and a “After” were iden­ti­fied from the crude data set and the re­l­at­ive cost ra­tio of “Be­fore” and “After” were cal­cu­lated and com­pared. A ROC Curve was plot­ted (Graph 17) and AUC were cal­cu­lated to be in­dic­at­ing an ex­cel­lent threshold, if ap­pro­pri­ately set, ex­ists for cost vs. re­set cost with oc­ca­sional in­con­sist­en­cies. With ref­er­ence to a box and whisker dia­gram com­par­ing “Be­fore” and “After” (Graph 18), a sub­sequent threshold of re­l­at­ive cost ra­tio was set to be and the ac­cur­acy of identi­fy­ing “Be­fore” and “After” were and re­spect­ively. This res­ult can be in­ter­preted as a threshold of re­l­at­ive cost ra­tio is ex­cel­lent for cost vs. re­set cost. The ac­cur­acy of identi­fy­ing “Be­fore” and “After” us­ing other re­l­at­ive cost ra­tio thresholds and the cor­res­pond­ing Youden’s In­dex and MDA are dis­played in Table 3. Both Youden’s In­dex and MDA are co­her­ent in set­ting the re­l­at­ive cost ra­tio threshold as is more ideal than any other thresholds. The in­con­sist­ency of fail­ure to identi­fy­ing “After” was ret­ro­spect­ively found to be in some re­sets between to and pub­lic­a­tions. Us­ing the re­l­at­ive cost ra­tio threshold as for may res­ult in time dif­fer­ence com­pared to the sim­u­la­tion res­ult.

Throughout the 9 months of in­vest­ig­a­tions, sev­eral hy­po­theses con­cern­ing the re­la­tion­ship between gain, of pub­lic­a­tions, and re­set pat­terns have been pro­posed. First part of this in­vest­ig­a­tion was to eval­u­ate the ef­fect of re­set pat­tern on the gain of the pub­lic­a­tion, it was hy­po­thes­ized that the rho dif­fer­ence between re­sets has some in­flu­ence on the gain of the pub­lic­a­tion, which was dis­proved by a graph plot­ting against (Graph 19). The graph re­veals no vis­ible re­la­tion­ship between re­set pat­tern and of the pub­lic­a­tion, des­pite sev­eral at­tempts us­ing arith­metic mean, geo­met­ric mean, and root-mean-square (RMS) of data sets.

The second part of this sec­tion in­vest­ig­ates the ef­fect of of pub­lic­a­tion on gain in a pub­lic­a­tion and, hence, in­vest­ig­ates the pos­sib­il­ity of a pub­lic­a­tion table where one can com­plete MF CT at a shorter time with a series of set pub­lic­a­tion rhos. To verify the above in­vest­ig­a­tion, the gain was ob­tained and plot­ted against of pub­lic­a­tion for MF CT and a sub­sequent mov­ing-av­er­age of gain of the pre­vi­ous 20 was also plot­ted (Graph 20). Sim­ilar graphs were also plot­ted for Basal Prob­lem (BaP CT; Graph 21) where a pub­lic­a­tion table is veri­fied to be ef­fect­ive in short­en­ing the com­ple­tion time of BaP CT, and for Wei­er­strass Sine Product (WSP CT; Graph 22) where there is no pub­lic­a­tion table for com­par­ison.

One can ap­pre­ci­ate the mov­ing-av­er­age plot for BaP CT is re­l­at­ively spiky, while that for WSP CT is re­l­at­ively a straight line with slight up-and-down des­pite highly fluc­tu­at­ing gain val­ues. It is also no­tice­able that gain fluc­tu­ates more for pub­lic­a­tion that goes across a mile­stone when com­pared to other of pub­lic­a­tion when the pub­lic­a­tion does not go across a mile­stone.

With the graph plot­ted for MF CT com­par­ing to the two con­trolled graphs, one can ap­pre­ci­ates the mov­ing-av­er­age plot for MF CT re­mains re­l­at­ively straight but in­cludes some spikes that has lar­ger amp­litudes than those in WSP CT but smal­ler amp­litudes than those in BaP CT, with large fluc­tu­ation for pub­lic­a­tion that goes across a mile­stone. With the ob­ser­va­tion, one can con­clude a pub­lic­a­tion table, like BaP CT, is less likely but still pos­sible for MF CT. Find­ing the des­ig­nated pub­lic­a­tion po­s­i­tion was out of the scope of this in­vest­ig­a­tion, sub­sequent in­vest­ig­a­tion would be needed if any other evid­ence sug­gests a pub­lic­a­tion table.

The above in­vest­ig­a­tions il­lus­trate the fact that a gen­er­al­ized re­set pat­tern across dif­fer­ent of pub­lic­a­tion can be found in MF CT (with a con­sid­er­able por­tion of un­ex­plained ex­cep­tional cases). By in­vest­ig­at­ing the re­la­tion­ship of re­l­at­ive cost ra­tio of , , , , and “Be­fore” and “After” to re­set cost, us­ing a re­l­at­ive cost ra­tio threshold of on , , and is ex­cel­lent in dif­fer­en­ti­at­ing “Be­fore” and “After” for a re­set. Mean­while, the strength of a re­l­at­ive cost ra­tio threshold of and is lim­ited, and a range of the re­l­at­ive cost ra­tio of and was sug­ges­ted with vari­able ac­curacies which need fur­ther evid­ence-based in­vest­ig­a­tions to dis­cover un­der­ly­ing cri­teria. The gain of the pub­lic­a­tion has no re­la­tion­ship with re­set pat­terns, and a pub­lic­a­tion table is less likely but still pos­sible for MF CT.

By re­call­ing the for­mula of MF CT, one can sim­plify the for­mula for rho dot as fol­low,

Where , , and are ex­po­nents that only changes when cor­res­pond­ing mile­stones are used.

The un­der­ly­ing reas­ons for the trans­ition of re­set to re­set from to are prob­ably due to the in­crease dom­in­ance of (via. ) in growth of in re­spond to lengthened time between re­sets and mile­stone 5 and 6 (at and re­spect­ively) which in­creases the ex­po­nent of , hence length­en­ing the re­set time to be­ne­fit ad­di­tional due to by “com­bin­ing” two re­sets into one single re­set. As MF CT pro­gresses, value of gradu­ally in­creases and sig­ni­fic­antly ac­count for the growth of with the ef­fect of mile­stone 7 and 8 (at and re­spect­ively), such that the ef­fect starts to over­come that of (See Graph 23), the re­set pat­tern gradu­ally trans­its again from re­set to re­set. However, the above hy­po­thesis has not been veri­fied and re­quire fur­ther veri­fic­a­tion with ac­count for the ef­fect of on . The ex­ist­ence of out­lier has also not been ex­plained in this in­vest­ig­a­tion; fur­ther eval­u­ation is needed and un­der­ly­ing cri­teria may be yet to be dis­covered.