Final Year Computing Project Surveys For Projects Using Surveys • The survey is a (potentially simple) research method which can add value to a project • A survey used early in the project process can help to clarify ideas, identify features most important to users or to consider different interfaces • A survey late in the project can be used to assess whether people like a product (it is not used to test the product) Delivering Surveys • Within the fields of Computing and Business, surveys are almost exclusively delivered online using specialist software • Using paper based surveys, or mailing Word documents to people is usually a sign of a very poor project • Never be tempted to make up survey results – the whole purpose of a survey is to useful data which will improve your project (and this is an academic integrity issue) • Being able to design and deliver surveys is a useful skill to place on your CV Survey Dangers • The people surveyed are not a good sample of the users of a finished product • The survey designer biases the survey questions based around the answers that they want to receive • Many people wish to fill in surveys as quickly as possible and so do not choose realistic answers • When survey participants feel that the choices they make reflect on them, they will choose answers that they think the survey designer wants to receive • Surveys often give opinions rather than facts (other definitive quantitative data sources may be available) What Do Real People Think About Completing Surveys? • See infographics: http://thumbnails.visually.netdna-cdn.com/surveys-in-the-social-world_50290d088c60a.png http://www.businessinsider.com/infographic-are-customers-getting-sick-of-surveys-2012-11 Delivering Surveys (options) • Paper (last resort!) • Online survey software: – Self installed (e.g. LimeSurvey) – Free/commercial software (e.g. Survey Monkey - note, there are cheaper/better alternatives – and it may be worth "clubbing together" with other students for a single account • Google Docs for surveys: http://google.about.com/od/toolsfortheoffice/ss/forms_googledoc.htm (Google Forms is also an option) • Survey software integrated within social networks Example of Moodle Survey Questions Need To Be Piloted • Show people the proposed questions: – Are there any ambiguities? – Are any questions impossible? – Are any possible answers missing? – With a scaled response, it is clear which end is "good" and which end is "bad"? Bad Survey Questions – Why? • What is wrong with the following three bad survey questions? – Often, there are multiple problems • How can they be improved? • Examples from: http://www.wku.edu/~holli.drummond/first%20page/classes/strategies%20of%20social%20research/example%20of%20bad%20survey%20questions.pdf Question 1 More people have attended the movie, Gone with the Wind, than any other motion picture produced this century. Have you seen this movie? Yes No Question 1 – Possible Observations • This is based on outdated information (last century) • This is a leading question Question 2 People grow up in all different types of families. What type of family did you grow up in? ______ Mom as single parent ______ Dad as a single parent ______ Both Mom and Dad Question 2 – Possible Observations • The spellings are Americanised • There are many combinations of family members that are not included • Some people's family situation will have changed during their childhood – at which point should they answer this? • Some people will be unwilling to disclose this information Question 3 In your opinion, how would you rate the speed and accuracy of your work? Excellent Good Fair Poor Question 3 – Possible Observations • The question needs to be asked with regards to a specific subject or scenario • Speed and Accuracy are two different factors – this could be expressed as two questions • The choices are not symmetrical – there is no "middle answer" • People's opinions of what constitutes each band will be different – could ask for specific percentage values – could test users instead The Survey As A Whole Needs To Be Piloted • Are the questions presented in a sensible order? • Are any questions repeated? • Are any obvious questions missing? • Is the question "skip logic" sensible? • Remember that the final survey will be completed without the benefit of you being there to answer concerns that participants have – Most surveys will need two rounds of piloting before being ready for live use – Other project students can be useful to pilot surveys – Keep a record of changes to demonstrate critical evaluation within your project report Processing Survey Results • Recommended software = Microsoft Excel – One row of data per participant • Don't use the graphs that come straight from online surveys • Always work from a copy of the data – never the original • Use formulas (e.g. COUNTIF) to process the data • Don't sort the data. The formulas should recalculate if changes are needed (e.g. additional survey responses are obtained) Comparability Of Data • Raw numbers are generally a poor way to compare data. • This should be presented in the form of percentages (1 decimal place is sufficient for most cases). • It is useful to carry out restrictive analysis of data with different sample groups, e.g.: – males compared with females – different age ranges of respondents – different physical location of respondents – comparing educational background of respondents – comparing "inexperienced" and "experienced" users Presentation of Survey Results (Text) • Focus these according to the audience and to present the results in an efficient manner • Many times, a textual representation is best: "Invitations to complete the survey were sent to a cross-section of 250 people. 88 responses were received, which is a response rate of 35.2%." "A demographical analysis of the people who responded to the survey revealed that 48 out of 88 (54.5%) were male. 40 out of 88 were female (45.4%)." Presentation Of Survey Results (Graphics) • Graphics (visualisations) work well when there are multiple variables to compare • They also break up the text in a project report and focus the attention of the user • Graphics need to be given a Figure number (e.g. the first graphic in Chapter 3 would be labelled Figure 3.1) • "Figure 3.1 shows the numbers of hours of TV watched per week broken down by age group" Choice Of Graphics • Academic documents use simple clear 2D graphics • More sophisticated graphics can be used to enhance posters and presentations – Review online infographics for ideas Example Graphical Representation Presentation Of Survey Results (Tables) • Tables are an efficient mechanism with which to compare multiple results against one another • Tables can often be extracted straight from Excel, which saves design time Example Table Deenah, N., Lancaster, T. (2012), Measuring The Extent Of Placement Resources To Support Computing Students. In Preparation Academic Papers • These can be useful in the survey process in multiple ways: (1) the presentation of survey results within an academic paper will indicate how you can present your results (2) surveys in academic papers can inform how you should design your own (they have likely been carefully piloted and tested) (3) a good survey used as part of a project can also be used as the basis for an academic paper, written with your project supervisor Know Everything About Surveys? Image from http://www.slideshare.net/jtneill/survey-design-introduction-overview Underlying Survey Principle (Conclusion) • Every survey designed should have a clear purpose, that can be expressed in one sentence • Share with the project group - What is the purpose of your survey?