Research COMP Cheat Sheet (DRAFT) by Moomooe

Ways of Knowing: Tradition or History, Authority, Personal Observ­ation, Rational Analysis, Scientific Method

Common errors resear­chers make: Inaccurate observ­ation, Overge­ner­ali­zation, Selective observ­ation, Illogical reasoning

Types of Knowledge: Research Approaches: Descri­ption Consists of attempts to describe natural, social, or psycho­logical events • Focus on assess­ment, which allows people to describe identified events. Prediction Involves developing ways to predict identified outcomes Improv­ement Involves developing inform­ation designed to determine the effect­iveness of interv­entions Explan­ation Resear­chers frame questions and problems in terms of theories or explan­ations of phenomena

Ethical Theories Utilit­ari­anism: the end justifies the means Deonto­logical: the outcome is less important than following a rule or principle (More concerned with principles of right or wrong)

Ethical Guidelines: APA, ACA, Laws (state & fedral)

Profes­sional Codes of Ethics of ACA Code Research respon­sib­ilities of the counselor, Rights of research partic­ipants (the importance of informed consent), Reporting of results, Issues related to public­ation

Questions to ask to determine the ethical quality of research What possible negative implic­ations or harm does this study have on the popula­tion? What are the possible benefits of the research results for the popula­tion? Are the sample and population studied fairly repres­ent­ative of the general popula­tion?

Identi­fying Research Topics: Research ideas can extend previous studies or invest­igate areas that have not been researched before. (Goal is to contribute to knowledge in a meaningful way and motivate others to learn more about the topic) Strategies for Identi­fying a Topic: Personal: think about experi­ences that have raised your curiosity Interp­ersonal: brainstorm with others (class­mates, profes­sors, superv­isors) Printed Sources: what have you read but would like to learn more about? Computer Strategies: do PsycInfo, and other online searches.

Operat­ional Defini­tions & Research Variables: Variable any charac­ter­istic, behavior, event, or other phenomenon that is capable of varying on at least two different levels or conditions Indepe­ndent Variable variable that is believed to affect or change the status of another variable (the DV) Dependent Variable the variable whose status seems to “depend on” the status of another variable (the IV)

Through program evalua­tion, we can gather inform­ation about our programs, determine whether programs are effective, and improve our programs

Profes­sional Standards and Guidelines for Program Evaluation Joint Committee on Standards for Educat­ional Evaluation (1994) 30 standards in four catego­ries: Utility, Feasib­ility, Propriety, Accuracy The Guiding Principles of Evaluators (American Evaluation Associ­ation, 2004) Five basic princi­ples: Systematic inquiry, Compet­ence, Integrity and honesty, Respect for people, Respon­sib­ilities for general and public welfare

Needs Assessment: Typically conducted before designing programs to determine what is needed Formative Evaluation (aka:I­mpl­eme­nta­tio­n/P­rocess) Focus on amount and quality of effort needed to implement program. Results typically provided to those implem­enting program Summative Evaluation (aka: Effect­ive­nes­s/O­utcome) Results typically provided to policy makers and/or funding sources

Steps in Evaluation Step 1: Formation of the evaluation team (Indiv­idual v. team & Insiders v. outsiders) Step 2: Identi­fic­ation of relevant stakeh­olders Step 3: Determ­ination of a focus for the evaluation Step 4: Identi­fic­ation of evaluation model and methods Step 5: Selection of evaluation methods and designs Step 6: Selection of measures for the evaluation Step 7: Collection of the data Step 8: Analysis of the data Step 9: Reporting of the results to relevant stakeh­olders

Identify the stakeh­old­er(s) who were contacted about the PE.Sum­marize questi­ons­/goals stated by referral source­(s)­/st­ake­hol­ders.

Describe the agency­/or­gan­ization where the program is taking place (Location, mission, types of services provided, clientele served, etc.)

Review articles on topic in an integr­ative way, summar­izing what we know about each topic (comparing and contra­sting)

Highlight strengths / limita­tions / gaps in previous litera­ture.

Introd­uct­ion­/Li­ter­ature Review Introd­uction to the problem, Develo­pment of the framework of the study, Statement of purpose, research questions, hypoth­eses, provide an argument for conducting the research. Should include relevant defini­tions and descri­ptions of unknown concepts

Results Summarizes data and results of statis­tical analyses

What to Include in Annotation: Summary: main topic/­ideas discussed, purpose and justif­ica­tion, method­ology and major findings Critiq­ue/­Eva­luation: goal and credib­ility of the source, major strengths and weakne­sses, value/use of the work Reflection how this was useful, how it fits with other works you reviewed, what is relevant to your study and how you plan to integrate the inform­ation into your work.

Purpose Statement: Should provide inform­ation about the design of the study, Should be testable or resear­chable (variables should be defined), Population being studied should be identified

Types of Hypothesis: Null hypothesis: Assumes that there is no difference between groups or no relati­onship between variables. Altern­ative hypothesis (two types): Non-di­rec­tional: posits that there is a difference between groups or a relati­onship between variables, but does not indicate which group will be more or less or the direction of the relati­onship, Direct­ional: posits that one group will be more or less than at least one other group or there is a positive or negative relati­onship between two or more variables

Broadly, validity is about how well-f­ounded the conclu­sions are that we can make about a research study

Threats to Internal Validity:

Diffusion or Imitation of Treatment The interv­ention given to one group is uninte­nti­onally provided to another group (tell partic­ipants not to talk to eachother)

Maturation Changes over time that result from physic­al/­psy­cho­logical processes within partic­ipants (no-tr­eatment group, random assign­ment),

Selection Biases Systematic differ­ences between groups before any experi­mental manipu­lations or interv­entions (random assign­ment),

Attrition Loss of partic­ipants, problem of longit­udinal studies (Parti­cipants more likely to remain in study if they are doing something intere­sting, something that has little or no cost or adverse side effects, seems plausible, and is effective)

External Validity: The extent to which the results can be genera­lized to circum­stances other than those in the particular experi­ment. Relates to how true to life (gener­ali­zable) the study is.

Histor­y/T­rea­tment Intera­ction Events occurring at time of treatm­ent­/in­ter­vention may affect outcome (Hard to control this, but can address by extending study beyond impact of uncont­rolled event)

Reactivity of Assessment If awareness of assessment leads people to respond differ­ently (Include some unobtr­usive measures)

Stimulus Charac­ter­istics and Settings Features of the study with which the interv­ention or condition may be associated (Use multiple experi­men­ters, settings, stimuli)

Reactivity of Experi­mental Arrang­ements The influence of the partic­ipants’ awareness that they are partic­ipating in an experiment (Archival data, Observ­ational data)

Test Sensit­ization The effect of a previous test on subsequent perfor­mance (Various group designs (post-test only, Solomon four-g­roup)

Timing of Measur­ement The results of an experiment may depend on the point in time that assessment devices are admini­stered

Threats to Construct Validity

Attention and Contact with the Clients Differ­ential attention across groups may be the basis for group differ­ences ( To control for this, you would need to include a placebo group and ensure that experi­menters are blind to the conditions to which partic­ipants are assigned)

Cues of the Experi­mental Situation May include inform­ation conveyed to prospe­ctive partic­ipants prior to their arrival to the experi­ment, instru­ctions, proced­ures, and any other features of the experiment

Other

Types of Popula­tions: Target population: ll indivi­duals or objects the researcher is interested in and to which the study results will be applied Accessible population: the segment of the population that is accessible to the researcher

Sample Size: Several factors influence the sample size: Design of the study, Whether the researcher is using qualit­ative or quanti­tative methods. For quanti­tative methods, generally, more is better: Rule of thumb: Catego­rical: 15 partic­ipants per group. Continuous (descr­ipt­ive): 10-15 partic­ipants per variable

Clarifying Methods When the purpose statement denotes an attempt to identify phenomena, events, or exercises

Simple random sampling Every single individual in the population has an equal chance of being chosen. Advantages: If completed adequa­tely, the results can be genera­lized readily back to the popula­tion. Disadv­antages: It is difficult to ensure that every individual in the population has an equal chance of being chosen.

Stratified sampling (propo­rtional and nonpro­por­tional) Selection of indivi­duals from the population who represent subgroups. Advantages: Ensures adequate repres­ent­ation on relevant variables (e.g., ethnicity) Disadv­antages: May focus on a variable that is not as important as the others.

Conven­ience sampling Section of a population that is convenient and accessible to the resear­cher. Advantages Conven­ient; reduces costs and amount of effort in conducting a study. Disadv­antages Severely limits potential for genera­lizing the results back to the population

Some methods are the same as those used in quanti­tative research: Simple random sampling, Systematic random sampling, Stratified random sampling, Conven­ience sampling

Maximal variation sampling: a form of purposive sampling; resear­chers select partic­ipants who differ on some charac­ter­istic or trait to obtain a sample with maximum variation

Snowball sampling: resear­chers identify people who have relevant charac­ter­ist­ics­/traits and then ask those people to identify other people with the same charac­ter­ist­ics­/traits

Issues

Sampling bias: occurs when the researcher actively selects a sample that differs from the target population (can be contro­lled)

There are generally two types of approaches to scoring a formal measure

Norm referenced: scores are interp­reted based on the comparison of one group’s perfor­mance with other groups’ perfor­mance, all of whom represent a clearly defined population

Reliab­ility of Instru­ments

Test scores are determined by two factors: True score, Error

Reliab­ility is reported as a correl­ation ranging from -1 to +1

Ways to Determine Reliab­ility

Inter-­rater: Measures the consis­tency of ratings across different raters. Measures whether a scoring key/manual is developed and used consis­tently

Internal consis­tency: Measures whether items in a measure are correl­ated. Measures how consistent responses are within a measure

Validity is how well an instrument measures what it purports to measure

Conten­t-r­elated sources: concerns the extent to which responses of test items represent a particular content

Criter­ion­-re­lated sources: concerns how well a test predicts outcomes based on a particular behavior or skill

Quanti­tative: Direct observ­ation and behavioral measures, Self-r­eport and invent­ories, Ratings of others’ behavior, Physio­logical approa­ches, Interviews

Qualit­ative: Direct observ­ation, Interview, Triang­ulation methods, Documents, Audiov­isual materials

Self-r­eport invent­ories Rate the extent to which an identified behavior, attitude, or feeling is present. Strengths: Ease of time and admini­str­ation, Do not require extensive training, Achieve access to non-ob­ser­vable events Limita­tions: Possib­ility of distortion bias by the responder attempting to achieve socially desirable responses, Responder may not be aware of certain feelings, attitudes, or behaviors

Physio­logical methods Assessment of responses such as heart rate, ECG, blood pressure, GSR, EEG, and immune system response Strengths: Measur­ement is objective and accurate Limita­tions: Method is generally time consuming, Measuring the variable of interest is expensive

Procedures

Resear­cher(s) should describe (in chrono­logical order) every step of the study: Recrui­tment of research assist­ants, Recrui­tment of partic­ipants, What the partic­ipants experi­enced (Informed consent, Assignment to groups­/co­ndi­tions, Admini­str­ation of study materials, Any interv­ent­ion­s/t­rea­tments, Debrie­fing)

Statis­tical Conclusion Validity: The extent to which a relation is shown and the extent to which the experiment detects effects if they exist. Basically, how “correct” or “reaso­nable” are the conclu­sions we are drawing about the data

Alpha (α): the probab­ility of rejecting the null hypothesis when that hypothesis is true (risk of a type 1 error)

Beta (β): The probab­ility of accepting the null hypothesis when it is false (risk of a type 2 error)

Effect Size: The size/m­agn­itude of the difference or relati­onship

Validity Threats

Subject Hetero­geneity: Variation among partic­ipants. The more diverse your sample is, the less likely it is that you will find a difference between groups. (Address this by choosing hetero­geneous samples, but ensure that the effect of selected partic­ipant charac­ter­istics can be evaluated in the design)

Multiple Compar­isons and Error Rates: more tests = more chance of a Type I error

Results are presented mathem­ati­cally. Generally both descri­ptive and infere­ntial statistics are used.

Terms: Normality: data are normally distri­buted; data fit a normal bell curve Homoge­neity of variance: whether the differ­ences between scores are similar or different between the comparison group Indepe­ndence: there should be no relati­onship between the data points (betwe­en-­sub­jects data is indepe­ndent; repeat­ed-­mea­sur­es/­wit­hin­-su­bjects data is not indepe­ndent) Linearity: when graphed, the data (on average) forms a line instead of a curve

The results section of a qualit­ative study is very different from the results section of a quanti­tative study. The results section in a qualit­ative study is charac­terized by descri­ptions of categories and themes, with quotat­ions.

The Discussion Section

Quanti­tative research = research that is based on measur­ement and quanti­fic­ation of data (i.e., all data turned into numbers)

Indepe­ndent variable: An event, condition, or measured attribute or charac­ter­istic that the researcher manipu­lates

Dependent variable: Changes as a result of changes in indepe­ndent variables

Three methods for contro­lling for extraneous variables: Build the variable into the design and control for its effects (make it an indepe­ndent variable), Remove possible effects of the extraneous variable (sample from one level of the variable), Control through statis­tical methods after the study has been conducted

Two charac­ter­istics: Random assignment to groups, Manipu­lation of the indepe­ndent variable

Types: Pretes­t-p­osttest equivalent group, Postte­st-only group, Solomon Four-Group Design

Quasi-­Exp­eri­mental Designs

Types: Pretes­t-p­osttest design, Time series design, Multiple time series design

have no random assign­ment, but have an interv­ention

Descri­ptive Designs

Four approaches: Survey, Observ­ati­onal, Correl­ati­onal, Causal compar­ative

Best suited to applied research. Types: AB design. ABA design. ABAB design. Multiple baseline design. Altern­ating treatment design (AB1AB2 or ABAC) A=base­line, B=inte­rve­ntion

Five qualities: A natura­listic approach: Involves the collection of data in the natural enviro­nment The use of descri­ptive data: Data is collected and presented through language and pictures An emphasis on process: Focus on the way things are done rather than the outcom­es/­acc­omp­lis­hments An inductive approach: Resear­chers explore what comes up, as opposed to the deductive approach, in which resear­chers hypoth­esize what is there and look for support for the hypothesis A focus on meaning: Unders­tanding the meaning that certain things in the enviro­nment have for the people in that enviro­nment

Six steps in using a case-study approach: Establ­ishing the boundaries of the case, Identi­fying themes of emphasis, Focusing on specific patterns of data, The use of triang­ulation in data interp­ret­ation, Consid­ering altern­ative views, Determ­ining the approp­riate genera­liz­ations from the case

Grounded Theory: The primary approach is the dynamic intera­ction of identi­fying catego­ries, which are analyzed and recons­tituted into more complex ones with each continuous level of analysis

Historical: The purpose is to system­ati­cally understand past events and phenomena to obtain a clearer unders­tanding of current issues. May involve the use of systematic methods (e.g., diaries, oral records, relics). Four Steps: Define the problem or develop a hypoth­esis, Identify potential sources of historical data, Evaluate the historical sources, Report and summarize the results.

Partic­ipants: Any/all descri­ptors or charac­ter­istics required of the partic­ipants goes here.

Procedure: Give the step-b­y-step process of doing the study, starting with partic­ipant recrui­tment. Like it’s a recipe. If there is a treatm­ent­/in­ter­ven­tion, be sure to describe it in detail.

Antici­pated Findings: Write a few sentences describing what the findings would be if you were to actually conduct this study.

Rationale For Selected Design: Explain how/why the design used in the Method was selected (e.g., how it fits with previous research, RQ/hyp­oth­eses, referral question).

Implic­ations of Antici­pated Findings: Say what the implic­ations of your antici­pated findings would be for the agency­/st­ake­hol­ders. Make sugges­tions for future research.

Backgr­oun­d/H­istory: In a 1952 study, Hans Eysenck concluded that talk therapy had no effect.30 years later Smith, Glass & Miller (1980) found that at the end of treatment, the average treated person is better off than 80% of the untreated sample. Since the mid-80s, the number of therapists has increased 275%. The number of DSM diagnoses has gone from 66 to 286 in the DSM-IV. There are over 200 therapy models (a 600% increase from the 1960’s!) and over 400 therapy techni­ques. Compar­ative studies of treatment type routinely show none to be signif­icantly superior to the others. (Dodo Bird Hypoth­esis- they all can be succes­sful). So, what makes therapy work? If they all work equally well, there must be things common to all of them that is making them work. Enter Common Factors...

Ingred­ients of a Healing Relati­onship An emotio­nally charged, confiding relati­onship between healer and client. A healing setting. A rationale, conceptual scheme, or myth that provides a plausible explan­ation for the client’s sympto­ms/­dis­tress and prescribes a ritual or procedure for resolving them. Active partic­ipation by both client and healer in that ritual­/pr­ocedure that both believe to be the means of restoring the client’s health.

Client Variab­les­/Ex­tra­the­rap­eutic factors (account for 40% of variance in outcome) Client partic­ipation Goals, Motiva­tion, Expect­ation that it will help Experience of the client Time/place to focus on self, Person­ality of the therapist, Having someone who cares and listens, Having someone who encourages and gives advice Make use of natural healing process Intera­ctive, Think together

Placebo, hope, expect­ancies (accounts for 15% of variance in outcome) Pathways thinking hought about ability to produce one or more workable route. A therap­eutic ritual­/pr­oce­dure; therap­ist’s confidence in method enhances client belief in potential healing. Agency thinking houghts about ability to begin and continue movement toward goals. Emotional, confiding relati­onship with therapist who is hopeful and determined to help. A therap­eutic setting that reinforces perception of therapist as helper who is effective. Way to increase hope is to help client find a new goal, pathway, or sense of agency

Blending Common Factors and Empiri­cal­ly-­Sup­ported Treatment (EST)

Local Clinical Scientist (Stricker & Trierw­eiler, 1995) Bring attitudes and knowledge base of a scientist to clients’ problems. (Observe, test hypoth­eses, reflect, conclude repeat­edly.) Unders­tanding the “local situation” is at least as important as knowing something about clients or techniques in general.