Download Hawthorne Effect in Educational Research: Duration & Control Procedures and more Study notes Literature in PDF only on Docsity!
DOCUMENT RESUME
ED021308 24
By-Cook,DesmondL. THEIMPACTOFTHEHAWTHORNEEFFECTINEXPERIMENTALDESIGNS INEDUCATIONALRESEARCK
FINAL
REPORT. OhioStateUniv.,Columbus. SponsAgency-OfficeofEducation(DHEW),Washington,D.C.BureauofResearch. ReportNo-P- BureauNo-BR-5-
PubDateJun
Contract-OEC-3-10-
Note-213p
EDRSPriceMF-$1.00HC-$8. Descriptors-*BEHAVIORALSCIENCERESEARCH, EDUCATIONALEXPERIMENTS *EDUCATIONALRESEARCH, FIELD STUDIES^ LITERATURE REVIEWS^ REACTIVE^ BEHAVIOR,^ *RESEARCH^
DESIGN, RESEARCH luZTHODOLOGY,TABLE'S(DATA),VALIDITY
Identifiers-*HawthorneEffect
Projectobjectivesincluded(1) establishingabodyof knowledgeconcerningthe roleoftheHawthorneeffectin experimentaldesignsineducationalresearch,(2) assessing the influenceoftheHawthorne effectoneducationalexperiments conductedundervarying conditionsofcontrol,(3) identifyingthemajorcomponents comprisingtheeffect and(4)suggesting methodologicalapproachesto controlthe
possibleinfluenceonthe effect in educationalexperiments. Methodologyand proceduresinvolved(1)establishinga fieldresearchsituationin whichselected variableswereintroducedand manipulatedaspossiblecuesto generatetheeffect, and(2)analyzingcompletedpublishedstudiestodetermine ifresultsofsuchstudies couldbeexplainedonthebasis ofthepossiblepresenceoftheHawthorne effect.
Fieldresearchfindingsshowedno systematicrelationtothe achievementgainsto cuingvariablesortoawareness ofexperimentation.Analysis oftheliteraturerevealed nosystematic relationshipbetweenstudyresults andcontrolprocedures
employed.
(Author/TT)
EA
EA
Z
ED
U.S.DEPARTMENTOFHEALTH,EDUCATION
&WELFARE
OFFICEOFEDUCATION
THISDOCUMENTHASBEENREPRODUCED
EXACTLYASRECEIVEDFROMTHE
PERSONORORGANIZATIONORIGINATINGIT.
POINTSOFVIEWOROPINIONS
STAIDDONOTNECESSARILYREPRESENTOFFICIAL
OFFICEOFEDUCATION
POSITIONORPOLICY.
=
-I
rn
=
(-)
z=
n
r-
°.i
rt
=a
-Ila
o
cu^
-s
rn
n0m
C-h
ri
r
CI:
M
a
C-,C
(1)Z
rtt.....
0.)
al C
il
DIWR --f-V
M^
0rt.
r°
-iZ
l^
=0>
0-h
r--T
00rn
--h
0-pa
Lo
rn
m.
g171,
zpa
rn
=-I
ON
I^
%,
....
rn-I
t.A.)
xi
alC
n^
En
I
-.
-I
th
n^
>z
vs
70
cn
(1)^
ED
0-I
o
I
n-*
Pa
rn
00
-P-
>z
a'=
rn
2>CI
"^
'V
11
MI
70
73
rn
3 rnz-I>r-
'tfti6Vesii1,
rn C,
tas
Ar%
Contents
- Acknowledgments0 0.0 0 0 0^0^00000000000 0
- Chapter I - Introduction 0 0 0 0^0 0 0^0 0 0^0 0 0 0 0 - The Problem0 - Relevant Literature0 - Objectives of the Study0 - Phenomenon 0 0 00.0 o , 0 .0 A Conceptualization of the Hawthorne effect - Organization of the Report
- Chapter II - Methodology and Procedures0 - Field Research Phase0 - Experimental Design. - Selection of Experimental Sample - Orientation of Participating Teachers. - Instrumentation and Data Collection. - Literature Analysis Methodology 0 0 0 0 0 0 0 0 0 0 - Research Areas investigated. 0 0 0 0 0 0 0 0 * - Abstracting Procedure0
- Chapter IlB - Field Research Results 0 .0 .0 - introduction - Cues, Awareness, and Achievement - Duration of Study0 - Teacher interview Results.
- Chapter IV - Literature Analysis Results - Control Procedures for the Hawthorne Effect - Definitions and Suggested Controls - Results.° Control Procedures as Related to Experimental - Study Duration and Results. - Mechanical Changes and Study Results
- Chapter V - Discussion, Conclusions, and Implications. - Introduction - Discussion - Conclusions. 0 0 0 0 0 0 0 0 0 0 0 0 0 : : - Implications 0 0 0 0 0 0 0 0 0^0 0 0 0^0 0 G
Chapter Vi - Summary .0..0.0.0^ 0.0^ 0..
The Problem0^0
.
0
125
125 Procedure. 0 ..........^00000000 Results00000^0000000000 0 0 0000 Conclusions. 0 0^ 0000000n^0 o^0 o o^ o^ o^ o^129 1mplications.. 000000000000000 0 129
References 0. 0000000000000000006000 131
Bibliography. 0 0 00000050000000000000 e^139
Appendix A - Instructions to Teachers.^000000000.^ A-
Appendix B - Development of the Awareness inventory. 0.^.^ B-
Appendix C - Literature Analysis Materials 0 o^000 o^ C-
Part 1 - Major Areas of^ Research..^.^0000000 o^ C- Part 2 - Abstract Summary Form..^000000009 o^ C- Part 3 - Abstracting Procedure - Experimental Teaching Methods Studies. 0 o 0 0 o 0 0 0^ C-
This report has been prepared in compliance with the publication instructions for Dmaring Substantive 22.22LIA Based on Research aojects, Supported la the^ Office^ of^ Education,^ Bureau^ of Research, issued August 18, 1966.
A special debt of gratitude is owed to the Columbus Public School System, Dr. Harold Eibling, Superintendent, for granting the project staff permission to use selected classrooms for a two year period of time with a minimum ofadministrative difficulties. Concurrent^ with^ this^ acknowledgment^ is^ a^ special note of thanks for the contributions of the participating teachers, many of whom were required to learn the new mathematics to fulfill their role in the classroom.
Such a project as this would not have moved far without the talents and skills of the project secretaries. Mrs.^ Kay^ Taylor helped to initiate the project and was followed by Miss Barbara Kinney who served most of the total project time, Mrs.^ Marilyn Mamrick typed most of the final draft material as well as the final manuscript, Their^ contributions^ are^ deeply^ appreciated.
Appreciation is also expressed to Dr, Egon Guba, formerly Director, Bureau of Educational Research and Service, The Ohio State University, both for providing sufficient released time to conduct the project and for his many valuable comments and suggestions.
Despite ail of this help, the principal investigator must, and does, accept full responsibility for the final product.
ERRATA
page Line^ Correction
5 10 reaction's^ should^ be^ reactions
17 1 results^ should^ be^ result
17 23 insert^ after^ the^ word^ definition "are referred to Kaplan's statement that strict definition..."
32 9 fro^ should^ be^ for
42 12 second^ from^ should^ be^ as
48 footnote^3 insert^ certain^ between^ with^ and other
49 Table^7 Heading^ "n"^ should^ read "final^ n"
50 4th^ line^ change^ the^ to^ no from bottom
94 8 change^ prime^ ob'ective^ to^ general question
96 27 delete^ first^ sentence^ "Each^ r.. in^ a study"
115 10 approved^ should^ be^ appeared
121 5th^ line^ should^ read^ would^ not^ be from bottom
123 8 delete^ in
130 13 .logical^ biases^ should^ be^ logic
157 Kranfer^ should^ be^ Kanfer
B-I footnote^1 5tructive^ should^ be^ interview
B-I3 2nd^ line^ Varimox^ should^ be^ Varimax from bottom
5-21 8 should^ read^ "mean^ summated and^ scale scores"
-c
The lack of informationabout^ the^ Hawthorne effect^ places
a large number^ of^ educational^ research^ studies, particularly methods studies conducted in^ classroom^ settings, into^ the following uncomfortable ca-.egories:
- Significant^ differences^ in favor^ of^ experimental groups in situations where^ the^ Hawthorne effect was not controlled^ may^ be^ disregarded^ or dismissed on this basis alone.
- No^ significantdifference^ studies,^ where both^ the control and experimental groups were cued^ to^ the experimental nature of their situation^ or^ provided with unusual opportunities for^ obtaining^ attention, may be invalidated^ on^ the^ assumption that^ the effect of the treatment wasobliterated^ by^ the more powerful effect^ of^ the^ experimental setting.
- Studies^ that^ did^ attempt^ to control^ for^ the^ reactive effect may be questioned on the^ basis^ that^ the^ wrong component of the effect^ was^ controlled^ or that^ the effect is not important enough^ to^ justify^ elaborate control.
14-.*
The situation into which^ many^ educationalstudies^ would^ be placed because of thereactive^ effect^ occurring between experimental arrangementsand^ the^ subject's subsequent participation was stated well by^ Clark^ (11)^ in^ hisdiscussion of the costs associated with^ bringing^ about qualitative^ gains in education.
It is probably fair to state that^ essentially^ all educational experiments of the^ past^ seventy^ years have not been able todisentangle^ the^ effect of
enthusiasm from the effects ofthe^ experimental
procedure and content. If^ during^ the^ next^ seventy years this is accomplished,^ it^ may^ then^ be possible ,to assess with confidence^ the^ factors^ that will improve the quality of education. Until^ such procedures are adopted, however,it^ is^ going^ to^ be extraordinarily difficult todetermine^ the^ impact of specific content or method on quality,^ 1,^ c^ 49)
If the above statement has validity,^ it^ would^ seem imperative that some systematic studybe^ made^ regarding^ the Hawthorne^ effect and its role in educational research.
2
The primary goal of project^ described^ in^ this^ report^ was to attempt to build up abody^ of^ information^ aboutthe^ Hawthorne effect which couldprovide^ a^ basis^ for^ the subsequent development of a theoryadequate^ to^ explain^ the effect.^ It was further hoped^ that^ the^ investigations made^ would^ provide information which would allow educational^ researchers^ to arrive at more informed decisions^ with^ regard^ to handling^ or controlling the effect in their^ experimental^ designs.^ The specific objectives andquestions^ to^ be^ answeredduring^ the course ofthe^ project^ appear^ in^ a subsequent^ section^ of^ this chapter
in order to provide asetting^ for^ the^ actual^ research activities undertaken duringthe^ course^ of^ the^ project,^ some attention should be given tothe^ role^ that^ the^ Hawthorne effect is said to play in educational^ research,^ the original development of the concept, and^ previous^ investigations which were influential^ in^ the^ formulation of^ the^ general^ topic under investigation as the project moved^ along.
Relevant Literature
A systematic review ofdiscussions^ about^ the^ Hawthorne effect is rather difficult to present^ because^ the phenomenon is not consistently referred^ to^ in^ educational^ and other literature by its famWar name. A^ complete^ review^ of relevant available literature atthis^ point^ is^ premature because^ a substantial portion of thetotal^ project^ time^ wasdevoted^ to an examinationof^ published^ research studies^ including^ direct reference to the use of the concept by^ persons^ engaged^ in research.
The research and otherrelevant^ literature^ on^ the Hawthorne effect concept which was examined^ did^ tend^ to^ fall into^ several general categories. Within^ each^ of^ the^ categories,^ a large number of position papers and^ research^ reports^ were found^ to exist. A^ detailed^ summary of^ each^ category^ would constitute a substantially large portion of^ this^ report.^ Consequently,^ an attempt has been madconly^ to^ highlight^ the^ areas by^ referring to selected publications^ which^ give orientation^ and^ focus^ to the categories. Bibliographies^ relatir.1^ to^ each of^ the^ categories have been prepared and areincluded^ as^ the^ Bibliography^ to this report. Readers^ interested^ in^ any^ one^ orall^ of^ the^ categories are directed tothe^ Bibliography^ forfurther^ information regarding the category of concern.
Numerous reports and interpretations of the Hawthorne findings have been prepared over the succeeding years. Among the more outstanding are those by Mayo (51)2 Pennock (59) Homans (34)2 Landesberger (42), and Blum (4 )
Social flycholon of Exeerimentation. While^ the^ original Hawthorne researchers focused their attention on the human relations program, researchers in the social science areas began to develop interest in the nature of a subject's reaction to experimental activities and procedures and the relation^ of these reaction's to the subsequent behavior or performance in the experimental situation. It^ should^ not^ be^ assumed^ that^ the report of the Hawthorne studies prompted this concern directly. Rather its appearance during the emergence of the areas of industrial and social psychology no doubt influenced some of the work on research methodology being developed in these areas.
Numerous examples of this concern over.subject-situation interaction are evident in many fields. Reports^ by^ Crespi^ (18) and Star and Hughes (77) in the area of public opinion polling demonstrate the positive effect pre-interviewing can have upon follow-up responses. Lana^ (41)^ on^ the^ other^ hand,^ reported that pretesting for attitudes did not sensitize the individual with regard to subsequent attitudes. Studies^ by^ Hovland^ and others (36) revealed that in some situations the subject- situation interaction may be inhibiting in that the subject becomes nonresponsive or responds in an unanticipated manner to the treatment condition. Herzog^ (32)^ discussed,^ in^ a^ monograph on guides for evaluation of research in psychotherapy, additional instances of treatment contamination due to possible influence of sensitization on the part of subjects.
Of more relevance than the above research to the general concern of this project is the research relatingdirectly^ to the social psychology of the experimental situation. This -topfc has-been-deemed of sufficient importance that symposia have been held on this topic at annual meetings of the American Psychological Association and the American Educational Research Association.
The respective works of Rosenthal on experimenter-bias and Orne in the area of subject's demands to have their needs met in the experimental situation are worthy of note.
Rosenthai's efforts have concentrated on the influence that the experimenter's wishes and desires have on the outcome of the experimental situation. He^ hasP^ demonstrated^ in^ a^ variety of situations that the experimenter can have his hypothesis
5
.=..72,71,111:Z.nz.Vz
validated because of his biases and their interaction with participating subjects, even with animals. His work included the finding of similar results when surrogates (e.g., graduate assistants) were used. A summary of the major findings of several years of research by Rosenthal appears in The American Scientist for June 1963 (69).
The work of Orne has dealt largely with the phenomenon of subject's needs being met in the experimental situation. (^) Orne presents evidence that subjects tend to demand a purpose or reason from the experimental situation, even if a situation is created which has no purpose. Orne indicates that various kinds of cues exist which direct the subject to operate in a certain manner in a specific situation. These various cues are referred to as demand characteristics. Probably the best account of Orne's work appears in the November 1962 issue of The American Psychologist (55)
The combined works of Rosenthal and Orne strongly influenced the thinking of the project staff during the course of the project, (^) While each research effort focused on a different aspect of the experimental situation, the experimenter in one case and the subject in the other, the findings are highly relevant to the general problem of studying the roles of the student and the researcher in a classroom experimental situation.
While not directly involved in actual research, Reicken (63) has stated very well the need to investigate the subject experiment interaction problem if research is to be conducted successfully in the social and behavioral sciences. His general proposition is that the typical experimental situation is a somewhat one-sided operation. The experimenter alone knows what is to be done and what information is to be presented to the subject at what time. _The subject, on the other hand, initially possesses none of this information but as he proceeds through the experimental situation he tries to secure more and more meaning. As he does so, his cooperation and responses are modified. (^) Thus, Reicken proposes that a type of .pbject-- experimenter ntgoliation takes place during the course of an experiment. The subject is also faced with what Reicken refers to as a deutero-ualbLtm. On the one hand, he is faced with ascertaining the task to be done and thus meeting the experimenter's approval yet at the same time possessing certain personal needs which call for him to do well (e.g., to protect himself) in the experimental situation.
6
edlication. Since the study was conducted for only a very short period of time (two weeks) and as an adjunct to a regular classroom situation, the findings were not conclusive although there was some evidence obtained that students could become aware of their participation in an experimental situation, albeit working under the handicaps noted.
The general conclusion to be drawn from the above research done largely with adults in various situations is that in some cases subjects could indicate their awareness of special experimental conditions but at the same time there were difficulties encountered in developing adequate means to assess this awareness. (^) No evidence was encountered that children of an elementary school age could ascertain their participation in an experimental situation. (^) Such evidence would be necessary if one is going to suggest that many of the studies conducted at the elementary school level, where many outstanding results are obtained, could be attributed to the student'sawareness of being the object of special attention.
The Hawthorne effect in Education. Although there is recognition in the literature of educational research as to the existence and possible confounding involvement of the Hawthorne effect in experimental designs, the most frequent reference is usually in terms of a post-experimental apology relative to the possible role of the phenomenon in explaining the success of an experimental group or the appearance of no difference existing between "methods." (^) This point was well made by Smith (75) in his review of why many projects were not approved under the Cooperative Research Program of the U. S. Office of Education. Further, the employment of the concept does not meet with any degree of consistency having been variously referred to as the novelty effect, the halo effect, attention, motivation, and similar synonyms (15).
The principal investigator of this project summarized the early work on the origin of the concept and some of its intervetations plus suggestions for control as they have appeared in the educational research literature in the December 1962 issue of the Phi Delta Kappan (12) (^) I Although referred to in a Phi Delta Kappa symposium on educational research design (67), the concept was not defined. Campbell and Stanley (10) in
A detailed review of the content pres.nted in this article has been purposefully omitted in this review since the material is available in published form.
8
the Handbook of Research on Teaching chose to include the phenomenon under a more general term of "interaction" when discussing factors affecting the external validity of experimental designs. An^ examination^ of^ standard^ introductory texts on research methodology for education for direct reference to the phenomenon, its impact on experimental designs or suggestions for its control, reveals only occasional reference to the concept, often without elaboration.
The infrequent reference to the concept in educational research literature directly does not mean that educators are not unaware of its existence. Of anything, they perhaps may be only too well aware of its role in the conduct and results of research. Some^ examples^ of^ this^ concern^ are^ presented^ below
in the excellent study on the relation of cost to quality of education noted above, Clark (11) refers generally to the concept in terms of teacher enthusiasm for the new method resulting in a transference over to the students causing them to accomplish more than the traditional or control groups. After summarizing selected research studies in several areas of education, Clark is prompted to indicate the significance of this teacher enthusiasm factor.
The wide range of experimental results in the schools would seem to confirm much the same conclusion. Experiments that have emphasized phonics have brought improvements in reading; experiments that have emphasized the meaning of words but with less emphasis on phonics have also brought increases in reading ability. Comparable^ experiments^ have^ been^ conducted in all major subject matter fields. Again, the most reasonable interpretation is that there is strong experimental interest-enthusiasm factor at work (ll,p, 48).
A somewhat similar interpretation is given bo,Cronbach^ (19) in his discussion of problems encountered in curriculum evaluation in order to bring about needed course improvements,
In an educational experiment, ft is difficult to keep pupils unaware they are an experimental group. And it is quite impossible to neutralize the biases of the teacher as those of the doctor are neutralized in the double-blind design. (^) It is thus never certain whether any observed advantage is attributable to the educational innovation as such or to the greater
9
This emphasis on the role enthusiasm plays is given additional emphasis by Scriven in a paper dealing with the methodology of evaluation (72)2 Scriven places the concept under a more generic concept labeled "disturbance effects" as may be noted in the following statement:
The enthusiasm difficulty here is simply an example of what we might call disturbance effects (italics mine), of which the placebo effect in medicine and the Hawthorne effect in industrial and social psychology are well-known instances0 ln^ each^ case we are interested in finding^ out^ the^ effects^ of^ a certain factor, but we cannot introduce the factor into the experimental situation without producing a disturbance which itself may be responsible for the observed changes. ln^ the^ drug^ field^ the^ disturbance consists in the act of giving the patient something which he considers to be a drug, something which does not ordinarily happen to him, and consequently may produce effects of its own, quite apart from the effects of the drug. On^ the^ Hawthorne^ effect,^ the disturbance is the disruption of e.g0 conditions which may suggest to the worker that he is the subject of special study and interest, and this may lead to improved output not the physical changes in the environment that are the intended parameters under study, (7T,
The above quotations serve to point up the general importance of the phenomenon investigated in the present study, whatever its fundamental nature might be, to both^ research methodologists and persons actually conducting research. While recognized as an important variable in educational research activities, there appears to have been little direct investigation of the phenomenon which would be of assistance to persons desiring to control for the phenomenon in research°
Not all persons sFare the concern of the research methodologist and the researcher with regard to the contaminating influence of the Hawthorne effect in research. As^ noted^ by Miles, there are those persons who take the general position that if the phenomenon does, in fact and irregardless of^ its nature, produce better learning or achievement^ on^ the^ part^ of^ the student, then we ought to incorporate the Hawthorne effect^ into all educational methods and/or curriculum study in a deliberate manner. Change,^ novelty,^ attentkm)^ and^ similar^ variables would then be considered to be a part of every modification in
education. In^ one^ sense,^ it^ would^ allow^ educators^ to^ take^ the position that the current emphasis on educational change is good if for no other reason than the sake of change. An educator could find it difficult to argue against such a position if there is true and valid interest in improving the educational situation for students to maximize the learning. In^ short,^ the^ ends^ justify^ the^ means.
The principal investigator of this project has taken the position that while we want the best education possible for students, it is equally necessary that we have an understanding of the means by which we achieve the ends. Since^ the^ Hawthorne effect possibly operates to contaminate both the means and ends, it is necessary to study the phenomenon to see exactly, or at least with more precision than now exists, what role the phenomenon plays in the research and development activities education undertakes to produce better means designed to achieve the desired ends.
Ob'ectives of the Study.
The overall objective of the study was an attempt to build up a body of information and knowledge about the Hawthorne effect in educational research which, in turn, might provide a basis for the later development of a theory adequate to explain the phenomenon. It should be understood that this objective did not include the development of a theory about the phenomenon as a direct product of the investigations contained in this report. This^ position^ was^ taken^ on^ the^ basis^ of^ a^ general principle derived from the history of science that the first task in establishing a new area of scientific inquiry is an organization of what is already known, it^ was^ hoped^ that^ the activities of the project might provide further information which would permit educational researchers to make better decisions with regard to the handling of the phenomenon in experimental designs.
General Objectives. Within the overall objective stated above, three general objectives guided the work of the project.
A. To assess the relative influence of the Hawthorne effect on educational experiments conducted under varying conditions of control.
B. (^) To identify the major components which comprise the Hawthorne effect.
C. (^) To suggest effective ,N-ad eWcient methodological approaches to control or account for any influence which the Hawthorne effect might have on educationa/ experiments.
12
