COMM1110LectureSlidesWeek10_T3_2022.pdf
COMM1110 Evidence-Based Problem Solving
Week 10: Evaluation and communication
Lecturer: Kevin Liu
General housekeeping:
• Please switch your microphone to mute to avoid disruption to the class
• Use the chat channel to ask questions or make a comment, or raise your 'virtual' hand
• If you have poor internet, turn off your video
• Wait for your lecturer to start
1 2 3 4 5 6 7 8 9 10
LEC
TUR
EST
UT
OR
IALS
10Problem
articulation and
disaggregation
Frameworks for ethical decision-making
Understanding problems –
fact Gathering
Patterns, biases,
hypothesising effects
Flexibility Week
Analysing the issues –
Identifying the causes
Analysing the issues – With
limited evidence
Problem solving & making
decisions
Evaluation and communicate
Week 1Toolkit
application
Week 2 Toolkit
application
Week 3 Toolkit
application
Week 4 Toolkit
application
Week 5 Toolkit
application
Week 7 Toolkit
application
Week 8 Toolkit
application
Week 9 Toolkit
application
11
Problems & evidence-
based problem solving
11
ASS
ESSM
ENTS
COMM1110 EVIDENCE-BASED PROBLEM SOLVING
Assessments
Week 10 Toolkit
application
Briefing Pack: 25%
Associate level: 10%
Expert level: 10%
Business Report:
40%
Please see COMM1110 Assessment Guide for details including exact deadlines
Excel Training Program: 10% + 10%
Case: 25% + 40%
Online Discussion Questions: 15%
This week:
Bullet Proof Problem Solving
Framework
Problem Solving Tools
Case studies and examples
• Evaluating Evidence
• Boundaries of Problem-
solving
• Synthesis and
Communication
• Furniture store
This week will focus on:
Problem-solving step
1. Scoping (define the problem,
disaggregate)
2. Analyse (prioritize, workplan,
analyse)
3. Decision (synthesize,
communicate)
This week’s to-do list:
• Attend: Synchronous (live) lecture online. You can access the zoom link underneath the Lecture and Tutorial Links Moodle section.
• Watch: Recording of lecture, if you missed the lecture or need to do revision.
• Attend: Synchronous (live) tutorial. You can access the zoom link underneath the Lecture and Tutorial Links Moodle section.
• Provide feedback: Tell us how Week 10 went for you.
• Assessments: continue completing the Case Report (Assessment 2b)
• Reliability and validity
Evaluating Evidence
Reliability and validity
Reliability – Consistency Validity – Accuracy
Reliability ≠ Validity (i.e. getting consistent results doesn’t mean that your results are correct!)
Reliability
Reliability refers to how consistently a method measures something, there are 3 types of consistency:
Type of reliability Measures the consistency of… Examples
Test-Retest Over time
A measure of stability
Multiple blood tests for
diabetes show same results
Internal Consistency Across items
A measure of how consistently
each item measures the same
underlying construct
Customer satisfaction
survey: multiple questions
on the same issue are
answered with same
responses by the
respondent
Inter-Rater Across different observers
A measure of agreement
Multiple judges in a sporting
competition that arrives at
same or very similar score
Validity: internal vs. external validity
Validity: relates to how well the results of a study accurately reflect the “truth”
Internal validity: relates to how well a study is conducted – are we sure that we know what caused the results of the study?
• e.g. Are we sure that the low customer satisfaction is caused by long waiting time?
External validity: relates to how applicable/generalisable the findings are in the real world
• e.g. This study shows that the low customer satisfaction is caused by long waiting time in our Kensington store. Is this finding also applicable to our Town Hall store?
Internal Validity
The extent to which a piece of evidence supports a claim about a cause and effect, within the context of a particular study
It can also be determined by how well alternative explanations for the findings can be ruled out
Cause Effect
Training staff on how to deliver
company specific customer service
procedures e.g. greeting, follow up,
after sales service
High standardisation of customer
service delivery across different
customer service representatives
Offering a bonus on the sale of
specific furniture items
Increase in the sale of specific
furniture items that are tied to the
bonus
Brewer, M.2000. Research Design and Issues of Validity. In Reis, H. and Judd, C. (eds.) Handbook of Research Methods in Social and Personality Psychology. Cambridge: Cambridge University Press.
Internal Validity
• Have just stressed the role of sampling error
− This is uncertainty that is unavoidable because we are dealing with a sample and not the entire population
− This by itself does not invalidate the inferences we make
− Other types of error exist and may be more problematic
• Recall our discussion of confoundment
− Is the evidence that is being considered free of this (and other) problems?
− This is the issue of internal validity
• A statistical analysis is said to have internal validity if inferences are valid for the population being studied
− Confoundment leads to biased estimates of the population parameters of interest
Internal Validity
Sample selection as a threat to internal validity – e.g. survivorship bias (non-representative sample)
Internal Validity
Sample selection as a threat to internal validity – e.g. “self-selection” (non-representative sample)
• Another common sampling problem occurs where the sampling units are “self-selected”
• Examples occur where the sampling unit “chooses” to be part of the sample
• In such cases the reason for self-selecting may be related to the outcome being measured and so bias may again occur
Internal Validity
Internal Validity
Internal Validity
External Validity
The extent to which the findings can be generalised to other situations, people, stimuli and times.
Australian Housing Market Example
Sydney to Brisbane property prices Low external validity since Sydney and
Brisbane contrast greatly with population
size and demand
Aronson, E., Wilson, T. D., Akert, R. M., & Fehr, B. 2007. Social psychology. (4 ed.). Toronto, ON: Pearson Education
Sydney eastern suburbs to western
suburbs
Some external validity but still large
differences in demand and income
Sydney eastern suburbs (e.g. Bondi
Junction) to eastern suburbs (e.g.
Maroubra)
High external validity since suburb
demographics, demand and income very
similar
External Validity
• A statistical analysis is said to have external validity if inferences and conclusions can be generalized from the population studied to other populations and settings
• Evaluating external validity will often be very subjective but will typically require detailed knowledge of the context and subject matter
• Inferences from well-designed and conducted experiments provide evidence with good internal validity
External Validity
• It makes sense to analyse house prices separately for capital cities
• The differences in Sydney and Brisbane prices are large and likely reflect population differences in price distributions
• It would be inappropriate to infer much about Brisbane prices on the basis of a sample of Sydney prices
External Validity
• If similar findings emerge from studies on different but related populations then this supports external validity
• Recall our discussion of accumulating evidence
• A formal comparison of related studies in order to detect commonalities is called meta-analysis
• An important topic but beyond the scope of the course
• Cognitive biases continued
• (Some) solutions to biases involve gathering feedback• Individual feedback to build self-awareness
• Team reviews to improve collaborative processes and tasks
Boundaries of problem-solving
Recall bounded rationality (from Week 1)
• Our ability to be rational problem solvers is limited by numerous constraints
• Common constraints are:
ComplexityTime & money
constraints
Cognitive capacity,
values, skills habits and
unconscious reflexes
Imperfect information
Information overload
Different priorities
Problem is complex and hard to understand
Not enough time and/or money to gather information
People have varying degrees of abilities, biases and heuristics
Available information is fragmented
Too much information to process
Some data perceived to be more important, so certain facts ignored
Cognitive biases: Extending Week 1
• See asynchronous video on Moodle for more details and examples:
Sunk costs bias Escalation of commitment
Over-confidence biases
Hindsight bias Groupthink
The tendency of decision makers to be influenced by
the way a situation or problem
is presented to them.
Where decision makers increase
their commitment to a project despite
negative information about
it.
The bias in which people’s subjective confidence in their decision making is greater than their
objective accuracy.
The tendency of people to view events as being
more predictable than they really
are.
When group members strive to
agree for the sake of unanimity
and thus avoid accurately
assessing the decision situation.
Overcome biases
First step: gather feedback to build self-awareness
• Feedback is a powerful tool to facilitate thinking and changes in behaviour that produce improved outcomes.
• Johari window highlights different ways self-awareness can grow
• In COMM1110, a peer feedback tool was used in week 9 tutorial
• Synthesis of findings
• Telling compelling stories – communication• Using logic tree structure pyramid to organise a compelling
story
• Your one-day answer structure of situation-observation-resolution is the starting point for the governing thought of your narrative
• Try several storyline structures to see which are most clear and compelling
• Influence tactics – the power of persuasion
Synthesis and Communication
Synthesis and Communication
Final stage of good problem-solving
• Synthesising your findings (in a way that highlights your insights) and telling a compelling story (i.e. answer your decision maker’s question, “What should I do?” in compelling way that motivates action)
• Done right, your conclusions are an engaging story, supported with facts, analyses, and arguments that convince your audience of the merits of your recommended path
Synthesis of findings
• The quality of your synthesis will determine the quality of your decision making
• First step toward compelling story telling: synthesise the findings from the relevant literature, your data gathering, interviews, analysis, and modelling (i.e. all the evidence you collected)
• Synthesis brings together all the separate pieces of evidence (e.g. literature or your analytic work) and often yields new insights you didn't notice when you were in the weeds of analysis
Synthesis of findings
Start at the end
In consulting firms, consultants typically:
• Start writing your final presentation before you even meet the client!
• Have early-stage hypotheses ready (even though some of them might be wrong), so you know what questions to ask the client when you meet them
• Imagining a set of hypotheses allowed the team to begin focused data gathering from the first meeting
Before moving to final synthesis
• Iterative problem-solving process: interaction of the hypotheses with the analyses
• Alternative hypotheses should already have been tested, and embraced or rejected (and dead-end analyses pruned off)
• Link analysis and emerging findings in a convincing way
Synthesis of findings
Final synthesis
• Draw together the individual findings of the work on each branch and twig of your logic tree into an overall picture
• Represent each of your findings in the form of graphics that highlight the insights that emerged from your work
• Where possible, the most powerful visualisation is to show each graphic as branches on your revised tree structure
– This kind of visualisation of findings illuminates the overall problem.
– It allows for insights to emerge that cut across and combine the different branches of analysis → the overall story typically begin to emerge from individual branches
Conn, C. & McLean, R. 2019. Bulletproof Problem Solving The One Skill that Changes Everything, John Wiley & Sons Inc (US)
Synthesis of findings
Final synthesis
The synthesis in pictures (previous slide) begins to tell a visual narrative (which helps you to draft your final report)
Draw graphs from the analysis that synthesise your findings also allow you to pressure test your solutions with your evidence and analyses
• Do these graphs support what you’ve concluded? If not, ditch them.
• Is there a really interesting point you made about a factor, but you don’t have a chart on that? Get that in the pack to support your findings.
• Go through this pressure test yourself and/or bring the whole team (if you solve a problem in a group) together for synthesis sessions to synthesise findings until you have a more complete and supported argument (from the simple situation-observation-resolution story summary)
Telling a compelling story
Once findings have been synthesised, structuring a compelling story is key in communicating to your audience
Move from simply stating the implications of analyses (early stages) toward how those results can motivate a plan for change (final stage) – i.e. What should we do and how should we do it?
Conn, C. & McLean, R. 2019. Bulletproof Problem Solving The One Skill that Changes Everything, John Wiley & Sons Inc (US)
Telling a compelling story (e.g. written report)
From one-day answers to pyramid structure
• Once you have synthesised your findings → move to the final step of
structuring a compelling story for your audience
Start from a story summary:
• Start from a simple situation-observation-resolution story summary
• Situation: summarising the situation allows us to update our best
understanding of the problem
• Observation (or complication): provide the tension in the problem,
what isn't working, and our best insights into ways to unpick it;
• Resolution is our best understanding of the solution path that moves
us toward the answer
Telling a compelling story (e.g. written report)
e.g. situation-observation-resolution story summary
Conn, C. & McLean, R. 2019. Bulletproof Problem Solving The One Skill that Changes Everything, John Wiley & Sons Inc (US)
Telling a compelling story (e.g. written report)
Structures for organising the storyline
Conn, C. & McLean, R. 2019. Bulletproof Problem Solving The One Skill that
Changes Everything, John Wiley & Sons Inc (US)
• The standard visual structure we use to represent our story structure is a pyramid structure (which helps to structure arguments and support into a powerful story)
• The basic pyramid structure:
– is just the logic tree on its side
– enables an organised structure that links arguments, data, analyses to overarching argument at the top
– helps show clearly how each element of our argument is supported by evidence
Telling a compelling story (e.g. written report)
Structuring your arguments
• Choose grouping structure or argument structure
• There is no general preference for these structures (your decision)
• Try several storyline structures to see which are most clear and compelling
Conn, C. & McLean, R. 2019. Bulletproof Problem Solving The One Skill that Changes Everything, John Wiley & Sons Inc (US)
or
Telling a compelling story (e.g. written report)
Structuring your arguments – e.g.
Conn, C. & McLean, R. 2019. Bulletproof Problem Solving The One Skill that Changes Everything, John Wiley & Sons Inc (US)
Recommendation: Hechinger needs to change its business model quickly to address the competitive threat of Home Depot.
Telling a compelling story (e.g. written report)
Structuring your arguments – e.g.
Recommendation: Oilco (the refinery business) needs to cut costs substantially and become a modest growth, niche operator
Conn, C. & McLean, R. 2019. Bulletproof Problem Solving The One Skill that Changes Everything, John Wiley & Sons Inc (US)
Persuasion in written communication
Logic
• Persuasive report writing requires the presentation of ideas through reason and logic, in order to influence the audience
• Persuasion may use an argument to convey an idea or to convince the audience to perform a certain action
• It is an art of using logic, demonstration of credibility and invoking emotions to effectively communicate the argument
Storytelling
• Once you get your logic right, you also need to work on telling a compelling story – i.e. answer your decision maker’s question, “What should I do?” in compelling way that motivates action
• Humans are storytelling creatures, not logical robots (remember this as you craft your messages!)
Problem-solving step
(bullet-proof problem solving in
brackets)
Critical thinking tools
to analyse different forms of evidence
Information tools Ethics tools Statistical tools
1. Scoping (define the problem,
disaggregate)
• 5Ws
• Cognition & problem-solving styles
• Library skills to review literature
• Assessing the situation with moral
imagination (step 1)
• Assumptions and worldviews (step
2)
• Descriptive statistics
• Graphs: frequency distributions,
bar charts, pie charts & histograms
2. Analyse (prioritize, workplan,
analyse)
• Logic trees – reiterative process
• Heuristics
• Pattern recognition:
− Novice vs expert
− Inductive, deductive, abductive
• Thematic analysis
• Fishbone tool
• Causality with quantitative data:
research design (including
experiments, confoundment &
reverse causality)
• Principles, duties and care needs
(step 3)
• Process, outcomes and
consequences (step 4)
• Character factors (step 5)
• Comprehensive assessment (step
6)
• Scatterplots and correlations
• Regression
• Probability
• Normal distribution
• Sampling distribution – Central
Limit Theorem
• Inferential statistics
• T-test, regression
3. Decision (synthesise,
communicate)
• Collaborative decision-making
techniques
• Evaluating evidence: reliability and
validity
• Boundaries of decisions: cognitive
biases
• Using logic tree structure pyramid
to organise a compelling story
• Justify your decision (step 7) • Hypothesis testing
• Confidence intervals, statistical
significance, effect sizes
• Boundaries of decisions: statistical
biases
COMM1110 Summary
Thank you
If you have any questions about the
course, please email:
comm1110@unsw.edu.au
The lecture recording will be made
available in your Moodle course site.
The Survey is now open!
Tell us about your experience and shape the future of education at UNSW.
Log-in on your laptop, tablet or phone:• Link on your Moodle course page; or• URL http://myexperience.unsw.edu.au use (zID@ad.unsw.edu.au).
The myExperience survey is anonymous and brief. It will only take about 2 minutes to complete.
Each session feedback is sought from students about the course and continual improvements aremade based on your feedback. We take your feedback very seriously and respond to your commentsand suggestions in future course design. Your feedback is very important to us. We look forward tohearing your feedback.