Journal of Strategic Human Resource Management

Open Access Open Access  Restricted Access Subscription Access
Open Access Open Access Open Access  Restricted Access Restricted Access Subscription Access

The Effect of Multi-Rater Consensus on Performance Rating Accuracy


Affiliations
1 Ernest C. Trefz School of Business, University of Bridgeport, United States
     

   Subscribe/Renew Journal


This study examined the extent that consensus affects performance rating accuracy. Participants (n=96) viewed a video depicting teams working on a problem-solving exercise. The ratees were evaluated on behaviours within three performance dimensions: verbal communication, collaboration, and decision making. Rating accuracy across three conditions (consensus, discussion without consensus, control) was calculated using Cronbach's (1955) accuracy indexes: elevation, differential elevation, stereotype accuracy, and differential accuracy. It was hypothesized that ratings provided by participants in the consensus condition would demonstrate the highest degree of accuracy. Findings in support of this hypothesis provided justification for use of consensus-driven multirater teams for performance evaluation. Moreover, organisations that incorporate a multi-rater strategy in the performance management process may consider a consensus approach rather than individual ratings submission to enhance accuracy.

Keywords

Employee, Performance, Evaluation, Appraisal, Rating, Accuracy.
Subscription Login to verify subscription
User
Notifications
Font Size


  • Balzer, W. K., & Sulsky, L. M. (1990). Performance appraisal effectiveness. Psychology in Organizations: Integrating Science and Practice, 133-156.

  • Barnes-Farrell, J. L., & Lynch, A. M. (2003). Performance appraisal and feedback programs. In J.E. Edwards, J.C. Scott, & N.S. Raju (Eds.), The Human Resources Program-Evaluation Handbook. Thousand Oaks, CA: Sage Publications, 155-176.

  • Borman, W. C. (1977). Consistency of rating accuracy and rating errors in the judgment of human performance. Organizational Behavior and Human Performance, 20, 238-252.

  • Borman, W. C. (1978). Exploring the upper limits of reliability and validity in job performance ratings. Journal of Applied Psychology, 63, 135-144.

  • Cronbach, L. J. (1955). Processes affecting scores on understanding of others and assumed similarity. Psychological Bulletin, 52, 177-193.

  • DeNisi, A. S., & Peters, L. H. (1996). Organization of information in memory and the performance appraisal process: Evidence from the fi eld. Journal of Applied Psychology, 81(6), 717-737.

  • Guion, R. M. (1965). Personnel Testing. New York: McGraw-Hill.

  • Ilgen, D. R., & Favero, J. L. (1985). Limits in generalization from psychological research to performance appraisal processes. Academy of Management Review, 10, 311-321.

  • Landy, F. J., & Farr, J. L. (1980). Performance rating. Psychological Bulletin, 87, 72-107.

  • Landy, F. J., & Farr, J. L. (1983). The Measurement of Work Performance. New York: Academic Press.

  • London, M. (2007). Performance appraisal for groups: Models and methods for assessing group processes and outcomes for development and evaluation. Consulting Psychology Journal: Practice and Research, 59(3), 175-188.

  • London, M., Mone, E. M., & Scott, J. C. (Winter 2004). Performance management and assessment: Methods for improved rater accuracy and employee goal setting. Human Resource Management, 43(4), 319-336.

  • Lord, R. G. (1985). Accuracy in behavioral measurement: An alternative defi nition based on raters’ cognitive schema and signal detection theory. Journal of Applied Psychology, 70, 66-71.

  • McGourty, J. (2001). The Team Developer: An Assessment and Skill Building Program Instructors Resource Guide. Hoboken: John Wiley & Sons, Inc.

  • Martell, R. F., & Borg, M. R. (1993). A comparison of the behavioral rating accuracy of groups and individuals. Journal of Applied Psychology, 78(1), 43-50.

  • Martell, R. F., & Guzzo, R. A. (1991). The dynamics of implicit theories of work group performance: When and how do they operate? Organizational Behavior and Human Decision Processes, 50, 51-74.

  • Mero, N. P., & Motowidlo, S. J. (1995). Effects of rater accountability on the accuracy and the favorability of performance ratings. Journal of Applied Psychology, 80(4), 517-524.

  • Mitchell, T. R., & Silver, W. S. (1990). Individual and group goals when workers are interdependent: Effects on task strategies and performance. Journal of Applied Psychology, 75(2), 185-193.

  • Murphy, K. R., & Balzer, W. K. (1989). Rater errors and rating accuracy. Journal of Applied Psychology, 74, 619-624.

  • Murphy, K. R., Balzer, W. K., Kellam, K. L., & Armstrong, J. G. (1984). Effects of the purpose of rating on accuracy in observing teacher behavior and evaluating teaching performance. Journal of Educational Psychology, 76, 45-54.

  • Murphy, K. R., Garcia, M., Kerkar, S., Martin, C., & Balzer, W. K. (1982). Relationship between observational accuracy and accuracy in evaluating performance. Journal of Applied Psychology, 67, 320-325.

  • Nieman-Gonder, J. (2006). The effect of feedback and development planning on subsequent behavioral ratings and objective group performance. Unpublished doctoral dissertation, Hofstra University, Hempstead, NY.

  • Roach, D. W., & Gupta, N. (1992). A realistic simulation for assessing the relationships among components of rating accuracy. Journal of Applied Psychology, 77(2), 196-200.

  • Roch, S. G. (2006). Discussion and consensus in rater groups: Implications for behavioral and rating accuracy. Human Performance, 19(2), 91-115.

  • Roch, S. G. (2006). Benefi ts of rater teams: Role of consensus and rater motivation. Poster presented at the 2006 annual meeting of the Society for Industrial and Organizational Psychology, Dallas, TX.

  • Roch, S. G., & Woehr, D. J. (1997, August). The effect of rater motivation on the accuracy of performance evaluation: An NPI approach. Paper presented at the annual meeting of the American Psychological Association, Chicago, IL.

  • Salvemini, N. J., Reilly, R. R., & Smither, J. W. (1993). The influence of rater motivation on assimilation effects and accuracy in performance ratings. Organizational Behavior and Human Decision Processes, 55, 41-60.

  • Smither, J. W., Barry, S. R., & Reilly, R. R. (1989). An investigation of the validity of expert true score estimates in appraisal research. Journal of Applied Psychology, 74(1), 143-151.

  • Smither, J. W., London, M., & Reilly, R. R. (2005). Does performance improve following multisource feedback? A theoretical model, meta-analysis, and review of empirical fi ndings. Personnel Psychology, 58, 33-66.

  • Smither, J. W., & Reilly, R. R. (1987). True intercorrelation among job components, time delay in rating, and rater intelligence as determinants of accuracy in performance ratings. Organizational Behavior and Human Decision Processes, 40, 369-391.

  • Sulsky, L. M., & Balzer, W. K. (1986, May). The behavioral diary format: Increasing rating accuracy through consideration of rater cognitive processes. Paper presented at the annual meeting of the Midwestern Psychological Association, Chicago, IL.

  • Sulsky, L. M., & Balzer, W. K. (1988). Meaning and measurement of performance rating accuracy: Some methodological and theoretical concerns. Journal of Applied Psychology, 73(3), 497-506.

  • Tetlock, P. E. (1985). Accountability: The neglected social context of judgment and choice. In B.M. Staw & L. Cummings (Eds.), Research in Organizational Behavior (Vol. 7, 297-332). Greenwich, CT: JAI Press.

  • Woehr, D. J. (1994). Understanding frame-of-reference training: The impact of training on the recall of performance information. Journal of Applied Psychology, 79(4), 525-534.


Abstract Views: 512

PDF Views: 0




  • The Effect of Multi-Rater Consensus on Performance Rating Accuracy

Abstract Views: 512  |  PDF Views: 0

Authors

Carrie Ann Picardi
Ernest C. Trefz School of Business, University of Bridgeport, United States

Abstract


This study examined the extent that consensus affects performance rating accuracy. Participants (n=96) viewed a video depicting teams working on a problem-solving exercise. The ratees were evaluated on behaviours within three performance dimensions: verbal communication, collaboration, and decision making. Rating accuracy across three conditions (consensus, discussion without consensus, control) was calculated using Cronbach's (1955) accuracy indexes: elevation, differential elevation, stereotype accuracy, and differential accuracy. It was hypothesized that ratings provided by participants in the consensus condition would demonstrate the highest degree of accuracy. Findings in support of this hypothesis provided justification for use of consensus-driven multirater teams for performance evaluation. Moreover, organisations that incorporate a multi-rater strategy in the performance management process may consider a consensus approach rather than individual ratings submission to enhance accuracy.

Keywords


Employee, Performance, Evaluation, Appraisal, Rating, Accuracy.

References