1. Introduction
Recently diversity of research field is being emphasized all over the world. Nations with advanced programs in research field keep developing new R&D programs for interdisciplinary studies. These new research atmospheres need objective validity in evaluation process.
Peer review is recognized as the best evaluation method for national research and development programs over the decades. However, peer review system has restrictive boundary because of its subjectivity produced by the reviewers. Bibliometrics has been developed as a supplementary and alternative system to peer review. But the effectiveness of quantitative indicators which also has qualitative aspect is rarely analyzed in the real R&D program selection evaluation. And even the same peer review system, has not been studied on similarities and differences between the results of other nations. Therefore these two purposes have been selected for this study. First purpose is finding the validity of quantitative indicators which was newly designed at World Class University(WCU) Program in terms of bibliometrics. Second is finding the validity of international peer review, especially comparing the results of Korean panel and U.S. panel.
For reaching these purposes, the data of WCU program was chosen because this program adopted 5 different quantitative indicators below.
- the number of articles published by the research team in the world’s top(10%) journals
- the total number of citations for all articles published by the team
- the average number of citations per each article published by the applicants
- the participants’ experience as editorial board members of high- quality international journals
- the number of domestic and international patents achieved
Also WCU program adopted international peer review which was conducted by overseas peer faculties as a second phase of evaluations.
2. Analysis Framework
In order to analyze 5 bibliometric criteria and overseas evaluation data with Korean peer review results, this analysing structure has been designed.
For analysing quantitative variables, basic statistics, T-test, and correlation analysis were conducted. Also for analysing performer variables, basic statistics, T-test, correlation analysis, comments analysis, and words analysis of frequency were conducted.
3. Results and Findings
The following are the results of various analyses on quantitative variables.
First, in terms of discrimination, it was found that bibliometric criteria has stronger assessment function than peer review. The standard deviations of quantitative variables were much bigger than the one for the peer review.
Second, quantitative variables were effective in the field of natural science, bio science, and engineering. Through the result of correlation analysis, these three areas have strong correlation between quantitative criteria and peer review. However, I could not find any correlation in the field of humanities, and social science. Also the number of papers in the top journals criteria was fit enough as appropriate criteria. But two criteria about citation showed similar impact. Editor and patent criteria needed some other application method.
Third, through the result of T-test and correlation analysis, I could find that bibliometics had to be used as a supplementary method and not as an alternative one.
The following are the results of various analyses on performer variables.
First, in terms of discrimination, No attentive difference between the results of Korean panel and US panel was founded. It is because standard deviation and odd ratio from logistics regression analysis of two variables were very similar in which no one could state the differences.
Second, through the result of T-test and correlation analysis, both Korean and US panel drew similar average score and two results had a positive correlation which means even if evaluator’s nationality is different from Korean and US, the review results would have the same trend.
Third, through the comment analysis and words analysis of frequency, I could find that even the results of evaluation had similar trend in Korea and US, but the point of view in which the reviewer had priority was little different. Korean reviewer seemed more focus on the participant’s career and capacity of research more. But US reviewer seemed more focus on the proposal’s detail composition.
4. Implication
There has been much theoretical exploration about the bibliometrics. On this study, through the real case analysis, the effectiveness of these theoretical quantitative criteria were proven. And I also proved the validity of the U.S. panel review. This can be developed to apply participation of U.S. reviewers in the national R&D program evaluation in Korea. These two main results could be expansively used as the institutional framework to national research evaluation policy.