Galaxy Zoo Starburst Talk

"Ideas for Citizen Science in Astronomy" (Marshall+ 2014) - section on Quench in need of serious editing?

  • JeanTate by JeanTate

    Thanks to zutopian for posting about this (here):

    Ideas for Citizen Science in Astronomy

    Philip J. Marshall, Chris J. Lintott, Leigh N. Fletcher

    (Submitted on 15 Sep 2014)

    Comments: In progress. The most up to date PDF file should be downloaded from this http URL .

    We invite feedback via github issues at this http URL, and aim to submit to ARAA on September 26

    http://arxiv.org/abs/1409.4291

    The feedback URL is https://github.com/drphilmarshall/Ideas-for-Citizen-Science-in-Astronomy/issues

    Here's the abstract:

    We review the relatively new, internet-enabled, and rapidly-evolving field of citizen science, focusing on research projects in stellar, extragalactic and solar system astronomy that have benefited from the participation of members of the public, often in large numbers. We find these volunteers making contributions to astronomy in a variety of ways: making and analyzing new observations, visually classifying features in images and light curves, exploring models constrained by astronomical datasets, and initiating new scientific enquiries. The most productive citizen astronomy projects involve close collaboration between the professionals and amateurs involved, and occupy scientific niches not easily filled by great observatories or machine learning methods: citizen astronomers are most strongly motivated by being of service to science. In the coming years we expect participation and productivity in citizen astronomy to increase, as survey datasets get larger and citizen science platforms become more efficient. Opportunities include engaging the public in ever more advanced analyses, and facilitating citizen-led enquiry by designing professional user interfaces and analysis tools with citizens in mind.

    No surprise that GZ features prominently, as does GZ forum mediated serendipitous discoveries! 😄

    However, as zutopian notes, there are some, um, oddities; for example, Letters is not mentioned in the body, but one of the references is a Letter! Also, the section on Quench is full of errors (yes, I plan on giving feedback).


    The above is a copy/paste of most of the thread I recently started in GZ Talk.

    Maybe we can use this thread to discuss what the preprint says about Quench?

    Posted

  • JeanTate by JeanTate in response to JeanTate's comment.

    ... to discuss what the preprint says about Quench

    The preprint has a subsection on Quench, in section 5 "CITIZEN-LED ENQUIRY". Here's the intro to that section:

    The previous sections have focused on specific, and somewhat isolated activities
    in which citizens have participated. In most cases, the community's involvement
    has been a contribution to a scientific investigation defined by professionals. The
    most important part of any scientific investigation is the question at its heart:
    what is it we are trying to find out about the universe? In this section we look
    at some cases where the process of enquiry, the science itself, has been instigated
    or led by citizens.

    In principle, this is an area of great potential. The constraints of funding
    proposals and management of research groups can often mean that professional
    scientists focus very narrowly on particular topics of research, specializing in particular
    techniques or datasets. Steering away from this course implies taking risks
    with time management, and allocation of resources to an ultimately fruitless research
    area can be detrimental to careers. Citizen scientists are largely free of
    these managerial and budgetary constraints, and are able to devote their attentions
    to whatever topics interest them. Moreover, we might expect outsiders to
    ask some unusual questions, and make connections and suggestions that highly
    focused professionals may not have thought of. What are some enquiries that
    citizens have led in astronomy to date, and how have they been enabled and
    supported?

    Here's the entire, current, subsection:

    Galaxy Zoo: Quench. Examples such as those above show that advanced
    work is possible within distributed citizen science projects, but that this requires
    volunteers to take on such tasks themselves. In order to increase the number,
    and perhaps the diversity, of volunteers moving beyond simple classification, experiments
    have been conducted to provide more scaffolded experiences. One of
    the most ambitious was the Galaxy Zoo: Quench project (Trouille et al. in prep)
    which offered volunteers the opportunity to \experience science from beginning
    to end."

    In this project, classification of a sample of potential post-merger galaxies
    selected from the main Galaxy Zoo sample was followed by open exploration of both the classification data and the metadata for these galaxies (available from
    the Sloan Digital Sky Survey) by the volunteers, enabled by a "dashboard37."
    Thousands of users (around 20% of those who participated in the classification
    stage and discussion) led to the formulation of a set of astrophysical interesting
    conclusions; a small number of participants ( 10) collaborated on writing a paper
    (in preparation). These later stages required intensive support from professional
    scientists (and in fact it was constraints on their time that prevented earlier
    submission of the paper).

    Quench demonstrated that a hierarchical approach, with simple tasks leading
    to more advanced analysis, can be successful in encouraging large numbers of
    volunteers to move beyond simple classification; the number participating in exploring
    the data was much higher as a percentage of participants than in Planet
    Hunters. However, once engagement with the literature (either by reading or
    writing) is required there remains no substitute for significant involvement by
    professionals.

    37 http://tools.zooniverse.org/#/dashboards/galaxy_zoo

    Posted

  • JeanTate by JeanTate in response to JeanTate's comment.

    Fellow zooite mlpeck has already posted some comments on the Quench section, in GZ Talk (link later):

    I'd say the "information" about Quench isn't especially accurate. At the time it went on hiatus there were 3 active "citizen scientist" participants by my count. There might have been as many as 10 involved early in the analysis phase but most of them faded away over time. As far as I know there were never thousands participating in Quench Talk, not even during the classification stage. And also as far as I know there is no scientific paper underway yet. There was a paper presented by Trouille at an AAS meeting in Chicago this August, but that was about the project more than it was results from the project and (again, AFAIK) no citizen scientist directly participated in its preparation.

    Finally, we had largely abandoned "Tools" (referred to as "dashboard" in Marshall+) as a data sharing and analysis tool well before the project went on hiatus. Even the last data updates from the science team were shared via a dropbox account.

    Posted

  • JeanTate by JeanTate

    To add to what mlpeck wrote:

    • perhaps the first 'setback' (if it can be called that) was the discovery (by an ordinary zooite - me - not a professional) that the dataset included duplicate objects (same object included more than once, with different IDs)
    • a much later setback was the discovery (triggered by me, confirmed by Kyle Willett) that the classifications database contained a surprisingly large number of 'repeats', two or more classifications of the same object by a zooite (this was explicitly supposed to not happen)
    • the datasets which the 'dashboard' (Zoo Tools) used, in the Quench project, were changed, several times, in the course the project; this rendered all previous (provisional) results unreliable
    • we never got to 'collaborating on writing a paper', unless you count discussions like 'what charts do you think we should include?' Even now the Authorea webpage for this paper (where we can go to collaborate on writing) remains inaccessible
    • so, where does this come from? "These later stages required intensive support from professional scientists (and in fact it was constraints on their time that prevented earlier submission of the paper)."
    • The first sentence in the last para may well be true, but the second is, um, completely without basis in concrete facts (IMHO).

    Posted

  • zutopian by zutopian in response to JeanTate's comment.

    Fellow zooite mlpeck has already posted some comments on the Quench section, in GZ Talk (link later):

    (..) As far as I know there were never thousands participating in Quench Talk, not even during the classification stage.(...)

    According to below GZ blog article over 1600 people had participated during the classification stage!

    GZ Quench: Classification Complete – Now the Real Fun Begins!
    http://blog.galaxyzoo.org/2013/08/02/gz-quench-classification-complete-now-the-real-fun-begins/

    Posted

  • mlpeck by mlpeck in response to zutopian's comment.

    According to below GZ blog article over 1600 people had participated
    during the classification stage!

    I can believe ~1600 people contributed clicks. What I don't believe is that "thousands" ever went beyond that by participating in Talk, let alone making a substantive contribution to the science discussion.

    Posted

  • JeanTate by JeanTate

    From the github "Closed Issues", #55 seems to suggest that zkChris wrote the Quench project section. It begins with Dr Phil, on 3 July, kicking the issue off:

    Citizen-led Enquiry: write GZ Quench case study

    Emphasis on:

    • experimental nature
    • attempt to enable and guide citizen enquiry.

      What worked well? What didn't?

    The history is interesting:

    • "drphilmarshall added the Urgent label 3 days ago"
    • "chrislintott closed this issue from a commit 2 days ago"

    Perhaps this section of the preprint is so poor because the author had next to zero involvement with Quench?

    Posted

  • drphilmarshall by drphilmarshall

    Thanks for reading the draft carefully! I'm afraid the numbers error is my fault: the original text was oddly punctuated, and I edited it as best I could so that it made sense, but did not check back with Chris that my version was correct. Apologies for this error - but thanks for providing some justification for our post-to-arxiv-and-invite-feedback approach! I will start an issue to correct the numbers, and then fix this with the other issues when I get back from my current trip.

    Posted

  • mlpeck by mlpeck in response to drphilmarshall's comment.

    Thanks for stopping by.

    The PI did a blog post at the end of round 1 of classifications saying around 1600 people took part, which seems reasonable -- KWillett is the keeper of the clicks database and he could certainly provide a definitive number. Anyway, thousands of participants isn't wrong. Probably also 10 or so people have contributed to the science discussion at one time or another.

    When doing classifications the last question for each object is something like "do you want to discuss this object" and if yes the classifier is given the opportunity to make a tweet length comment. I'd guess that's where the 20% figure came from, but that makes ~300 who took part in GZ Quench Talk at any level.

    Posted

  • JeanTate by JeanTate in response to mlpeck's comment.

    As noted by you, mlpeck, over in GZ Talk:

    For anyone not following along on github, the submitted version of Marshall, Lintott & Fletcher 2015: "Ideas for Citizen Science in Astronomy" was posted earlier this week. I haven't done a detailed comparison with version 1, but the authors clearly responded to feedback left on the "issues" section of github and the end result looks like a considerable improvement to me.

    Here's the whole submitted-for-review text/section, on Quench (I hope I've copied it faithfully):

    Galaxy Zoo: Quench. Examples such as those above show that advanced
    work is possible within distributed citizen science projects, but that this requires
    volunteers to take on such tasks themselves. In order to increase the number, and
    perhaps the diversity, of volunteers moving beyond simple classification, experiments
    have been conducted to provide more scaffolded experiences. One of the
    most ambitious was the Galaxy Zoo: Quench project45 (Trouille et al. in prep.) which offered volunteers the opportunity to "experience science from beginning to end."

    In this project, classification of a sample of potential post-merger galaxies selected
    from the main Galaxy Zoo sample was followed by open exploration of both
    the classification data and the metadata for these galaxies (available from the
    Sloan Digital Sky Survey) by the volunteers, enabled by a "dashboard.46" 3298
    users participated in the classification stage, and around 25% of those Zooniverse registered
    users who did so took part in data analysis. These results contributed
    to a discussion from which a set of astrophysically interesting conclusions were
    formulated by a small number of participants ( 10), with support from the project
    science team.

    Galaxy Zoo: Quench demonstrated that a hierarchical approach, with simple
    tasks leading to more advanced analysis, can be successful in encouraging
    large numbers of volunteers to move beyond simple classification; the number
    working with the data was much higher as a percentage of participants than in
    Planet Hunters, a project with success in volunteer user engagement. However,
    engagement with the literature (either by reading or writing) required close collaboration
    with the professionals involved. One interesting feature of the Quench
    project was its teething problems: issues with the data were discovered by the
    citizens, and needed to be fixed. (Similar problems have been encountered in
    Kaggle challenges.) While this caused the project to slow down and lose engagement
    somewhat, it does illustrate a key feature of citizen-led enquiry, namely
    that the same book-keeping, cleaning and calibration problems will arise in these
    projects just as they do in professional ones, and the limiting factor may well be
    the amount of professional effort available. The challenge is to enable the crowd
    to solve them quickly and keep investigating.

    The two footnotes are:

    45http://quench.galaxyzoo.org

    46http://tools.zooniverse.org/#/dashboards/galaxy zoo

    It was surely not the intention of the authors - Phil Marshall, Chris Lintott, and Leigh Fletcher - to provide any Quench zooite with the sort of experience envisioned by the Quench Science Team for Stage 3 ("Writing the Professional Journal Article"), but having been heavily involved in the Quench project throughout, I found it very informative (even illuminating) to see how the draft paper evolved from the v1 posted on arXiv to the version sent off for review. Many thanks to Phil for using GitHub, so enabling at least this zooite with a window into the nitty-gritty of at least part of the process.

    Posted