Psychology Research Data Management: The Results

On 7 June the TraD team carried out a feedback session on the training we’ve carried out for the professional doctorate students in educational psychology. The training consisted of introductory presentations and offering the MANTRA course material via the newly launched Moodle virtual learning environment at UEL. We wanted to find out their views on how well it went but also on the future design of the course: we’re building for the long term.

Valley Life Sciences Building, University of California, Berkeley, CA Image by Adam Fagan http://www.flickr.com/photos/afagen/2730921504

Valley Life Sciences Building, University of California, Berkeley, CA
Image by Adam Fagan http://www.flickr.com/photos/afagen/2730921504

Read More

Psychology RDM Workshop

On Wednesday 13th March Stephen and I organised a workshop for academics in the School of Psychology.

Our aims were the following:

  1. To inform participants of funder expectations for RDM
  2. To share good practice in data management planning
  3. To gather RDM requirements from psychology researchers
  4. To offer tailored support via data clinic sessions

Also helping us on the day was Dr Richard Plant, who formerly worked on two JISC MRD Projects (DMTpsych Project at the University of York, and Data Management Planning and Storage for Psychology (DMSPpsych) at the University of Sheffield.

Read More

Teaching RDM to Psychology students

School of Psychology at UEL

School of Psychology at UEL

We’ve recently kick-started our RDM training for post-graduate students in the School of Psychology. In addition to the introduction presentations on RDM we are offering five online modules using the MANTRA course material via the newly launched Moodle virtual learning environment at UEL.

Our course will cover the following MANTRA modules:

1. Research Data Explained
2. Organising Data 
3. Documentation and Metadata
4. Storage and Security
5. Data protection, rights & access

Our first Introductory class included 27 professional doctoral students in clinical psychology and our second 12 professional doctoral students and 1 PhD student in educational psychology.

Read More

Lords of the Data: psychology and research data

One of the work outputs on the TraD project is is to deliver a course in research data management for Psychology postgradate students at University of East London. We’ve therefore been keenly following an academic scandal directly affecting the world of social psychology involving the Dutch psychologists Diederik Stapel (falsifying data) and Dirk Smeesters (massaging data).  Similar scandals and retractions in the field have also involved Lawrence Sanna (2012)  Marc Hauser (fabricating data 2010) and Karen Ruggiero (2001).

Brain storm: Social Psychology Theory / Distributed Cognition / CSCW by Rob Enslin

Brain storm: Social Psychology Theory / Distributed Cognition / CSCW by Rob Enslin

Dutch investigators have released their final report into the case of Stapel from Tilburg University, entitled: Flawed science: The fraudulent research practices of social psychologist Diederik Stapel.

Its findings reveal that Stapel fabricated data in 55 articles and book chapters. So far, 31 of those published papers have been retracted — three others have expressions of concern — although more might follow. In addition, 10 dissertations by students Stapel supervised were found to contain fraudulent data and this is what should be brought to the attention of our psychology Postgrad students here at UEL.

The final report makes for sober reading (Stapel personally taught his department’s scientific ethics course, for example) and a damning assessment of the discipline itself referring to “a failure to meet normal standards of methodology. [bringing] into the spotlight a research culture in which… sloppy science, alongside out-and-out fraud, was able to remain undetected for so long.” P. 5

The report has highlighted a number of issues with regards to research data handling, standards and attitudes and which we are likely to cover in our course.  More points from the report:

  • “The Stapel group had no protocols for, for example, the collection of data (including standards for questionnaires) or research reports. The PhD students in Mr Stapel’s group were not familiarized with fixed and clear standards. (Stapel) underscored the lack of (fixed and clear) standards; but that too was far from a local issue, according to Mr Stapel.” P. 42

We cannot be certain if as Stapel says this is not a local issue but one that crosses social psychology as a whole but how research data is collected and standards applied to that data (as well as taught) should be something that is paramount in the teaching of postgraduates. The report again:

  • “The doctoral examination board must form a clear impression of the way in which research data has been collected.” P. 57

The importance of research data verification and replication of findings from research data will be something we shall likely emphasize to  students studying for doctorates in our course.

  • “Research data that underlie psychology publications must remain archived and be made available on request to other scientific practitioners. This not only applies to the dataset ultimately used for the analysis, but also the raw laboratory data and all the relevant research material, including completed questionnaires, audio and video recordings, etc. It is recommended that a system be applied whereby on completion of the experiment, the protocols and data used are stored in such a way that they can no longer be modified. It must be clear who is responsible for the storage of and access to the data. The publications must indicate where the raw data is located and how it has been made permanently accessible. It must always remain possible for the conclusions to be traced back to the original data. Journals should only accept articles if the data concerned has been made accessible in this way.” P. 58

There is not much we can add to this set of recommendations; whether they can or will be implemented is as yet unclear particularly as as far back as 2002 similar recommendations were made following the Ruggiero case. But there seems to be a desire to change and the way data is managed in this field is something that unbeknownst to us is something we are now a part of.

As Uri Simonsohn, the researcher who flagged up questionable data in studies by social psychologist Dirk Smeesters, has said: “We in psychology are actually trying to fix things… It would be ironic if that led to the perception that we are less credible than other sciences are. My hope is that five years from now, other sciences will look to psychology as an example of proper reporting of scientific research.”

We hope that the psychology course in research data management we will be running at UEL will be a part of this hoped-for progress and will be seen as an example of good practice for future psychologists.

John

Review of existing RDM training materials

Two workpackages (2 and 3) in TraD will produce an adapted data management course for professional doctorate/PhD students in Psychology, and new data management materials for Computer Science postgraduate students.

Magnifying glass

Our first task was made a lot easier by two factors. One, the expectation from JISC that funded projects make their outputs available for re-use and adaptation and secondly, there were already existing projects with excellent training materials for RDM in psychology (MANTRA and DMTPsych).

We reviewed both these training packages and have outlined our top three most valuable elements for reuse in the training of psychology research students in RDM at UEL.

Top 3 for Psychology

1.    Mantra

An excellent online course that makes full use of multimedia materials such as video, images, flowcharts as well as learning technology methodologies in the form of quizzes within modules, drag and drop (words into phrases), questions from scenarios, drag and drop of words to fit tables (for example, a data management plan). The modules also have an excellent “further reading” element at the end of each module.

2.    MANTRA – Data Handling

A useful teaching module on Data Handling using the statistical analysis software SPSS 18 (aka PASW 18) which is commonly used in Psychology and Social Sciences.

3.    DMTpsych

A six-part lecture series on Data Management Planning in Psychology with copious slides full of excellent material and images. This certainly provides us with an option to pick and mix from each lecture to fit our own training materials. Accompanying the Powerpoint presentations are slide-by-slide feedback from students on the course which means we have an excellent opportunity for tweaking and working on ideas for training at UEL.

We will also be looking into using the comprehensive guidance booklet (136 pages!) entitled: Guidance Notes for Completing a “Checklist for a Data Management Plan v3.0” For Researchers in the Psychological Sciences”.

Our second task as part of workpackage 3 was to review all other existing RDM training materials for ideas related to creating a training course for Computer Science students. In total we reviewed 19 JISC-funded projects that contained elements of RDM training and to which we could use for our own training modules. Read on for our Top 5 finds. Read More