A Web 2.0 Peer-Reviewed Science Journal

Who Killed Amateur Science?

As I have discussed in earlier posts, amateur science is in a very bad way. It's not that there aren't people doing amateur science, but their efforts have become largely irrelevant to the advance of mainstream science. There are two major causes for this decline in amateur science:

• The extreme professionalisation of mainstream science
• Copyright

Professionalisation

Science can be defined in very abstract terms, such as "learning about the nature of reality by developing falsifiable hypotheses and then testing them", but in practice modern science is actually "research carried out by PhD-qualified scientists working in academic or other research institutions who publish in peer-reviewed scientific journals".

These criteria of qualification and publication are the result of innocent intentions: to maximise the quality of what counts as "science". But they have an unintended consequence, which is the almost complete exclusion of amateurs from the scientific process.

In other words, even if an amateur scientist develops some theory which solves some well-known scientific problem, the theory will not become part of the accepted body of scientific knowledge until a professional PhD-qualified scientist rediscovers the theory and publishes it in a peer-reviewed scientific journal.

Copyright

Copyright adds to the woes of the amateur scientist. For example, to write a paper suitable for publication in a peer-reviewed journal, you must have access to all significant papers published in the subject area, where these papers may have been published in a variety of different and often very expensive journals. Access to all relevant past papers is required because the author of a paper is expected to acknowledge all existing knowledge and theory in the subject area which has any bearing on the content of the paper.

Unfortunately, although science is a nominally "human" enterprise – advancing the knowledge and understanding of the whole human race – the business model promoted by the copyright system is that journals publishing scientific papers seek to prevent the dissemination of information to the human race (i.e. to the general public).

Normally journals are only purchased by academic and research institutions. These institutions can justify the expense because the journals they purchase can be made available to everyone who belongs to the institution. And the journals have to charge a high price, because academic and research institutions are the only customers buying them. The amateur scientist will usually not be able to afford to purchase all the relevant journals, and they almost certainly won't be able to join any of those institutions that can afford them. The copyright-based business model denies the amateur scientist access to the very resources which are required in order to participate in the scientific enterprise.

(There has been some mitigation of this problem in recent times, with authors placing draft copies of papers in pre-print archives, some journals making all papers available on the web, and most journals putting at least titles and abstracts on the web. There are also initiatives like PLOS which seek to challenge the validity of the copyright-based business model as applied to scientific journals. However, there continues to be a substantial body of both existing and new journal content which is not made freely available on the web.)

Options for the Amateur Scientist

The amateur scientist with work to publish has limited options:

• Accept the terms dictated by the professional science establishment, and attempt to write up their results in a form which will be acceptable for publication in a peer-reviewed journal, which, despite their best efforts, is unlikely to occur.
• Publish their work in some other outlet, where it will simply be ignored.

A New Web-Based Publishing Model?

Is it possible for amateur scientists to publish theories and research results on their own terms?

The web is the most attractive option for an amateur scientist wishing to publish. But, by itself, the web does not provide enough structure to select and then attract attention to quality amateur-published material.

Is it possible for the web to provide an effective replacement for formalised peer review? Although various quality-selection systems exist which help web users to find quality new content – like Google, and vote-driven bookmarking systems – none of these systems is specifically designed to favour peer review of new scientific theories.

The Importance of Stated Intentions

It would seem that the web provides many options for posting and reviewing content. There are forums, where people comment on each other's postings, and wikis, where people directly edit each other's content. Voting sites implicitly review content, in that if you vote for something then you are implying that it is good. Slashdot allows users to vote on comments made by other users. Even plain web pages can work as a review mechanism: you post your web page, then I review your web page on my web page, linking to your web page so that my readers know what it is that I am talking about.

Unfortunately, none of these systems of review quite works for reviewing the results of original scientific research. New scientific theories are necessarily speculative, so they cannot be dealt with by editing systems like that of, for example, Wikipedia, which is designed to document accepted knowledge. Scientific theories also require considerable effort to evaluate properly; most web-based comment and review systems are designed to support least-effort comments, and do not readily distinguish between formal and casual reviews of other people's ideas.

To deal with these problems, a web-based peer review system needs to have a formal statement of intent. A good example is Wikipedia, which is a wiki which is intended to produce an encyclopedia. There is no major difference between the editing mechanisms provided by Wikipedia and those provided by c2.com (the original "wikiwikiweb"). But c2.com is not an encyclopedia, because there is no intention that it should be an encyclopedia. A formal statement of intent affects how the users of a website respond to the behaviour of other users, and in particular it determines what is considered to be inappropriate editing or posting of content. A statement of intent does not guarantee that a website or web system will work how it's meant to work, but it can make a significant difference.

So what kind of statement of intent should a web-based peer-review system have? The intention should be that the system provides for reviewers to seriously and carefully review papers, on the assumption that authors of papers will take into consideration the results of reviews, updating their papers in response to valid issues raised by reviewers. For each paper there will be a cycle of review and revision, until either the reviewers have nothing more to say, or until the author decides that they are not going to make any further changes in response to reviewer comments.

Reviewing versus Arguing

Reviewing is not the same as arguing, where the intention is to help a reader select between different points of view. A reviewer may mention arguments against the content of a paper, but only for the purpose of ensuring that the paper deals with those arguments if doing so is essential to the point that the paper is trying to make. The result of a review should be information that the author can use to improve their paper. However, in some cases a reviewer may highlight problems which are so serious that the author cannot deal with them, which may indicate that the content of the paper is of no value at all to the advancement of science.

This distinction between reviewing and arguing is another reason why forums and commenting systems do not perform an effective reviewing function. Typically people respond to a posted message by posting a reply message which presents their own ideas and theories. This is not quite the same as helping the author of the original message improve the presentation of ideas in that message. (Furthermore, usually a forum won't let you edit a message once it has been publicly posted, so there is no way to fold required improvements back into the same message.)

The Webified Peer-Reviewed Journal

Web 1.0 Version

One way to do peer-reviewed journals on the web is to take the traditional process of peer review, and translate it as is onto the web. The statement of intent of the "webified" journal will be almost identical to the non-web journal's policies and/or mission statement. The web version of the journal will continue to have primary ownership of the process, i.e. it will control and mediate all communication between author and reviewer, and at the conclusion of the review process the journal will publish the final accepted paper on its website, or, alternatively, it will publish nothing at all if the paper is deemed unacceptable by the reviewer.

Because a traditional journal owns the publication process and the results of the publication process, the journal is necessarily responsible for selecting its reviewers. It must also be responsible for determining the required level of quality for any paper that is to be published. And any problems with a paper must be fixed before official publication, because journals only ever publish one official version of a paper, which cannot be changed after publication. (A journal can publish errata for a paper, or even retract a paper, if it really needs to, but this is not an option to be used routinely.)

Web 2.0 Version

Simply translating existing journals onto the web is a very Web 1.0 way of defining the publication and review process for a webified journal. In the new, exciting and somewhat hyped world of Web 2.0, process and ownership can be freely separated, and the process can be decentralised. Translated into specifics, this implies that we don't necessarily need a journal. The primary content is the papers, which should be owned by their authors, and the reviews, which should be owned by the reviewers. If there is a requirement to consolidate the output of authors and/or reviewers, this can be done afterwards. And it can be done more than once, and in many different ways. In other words, a Web 2.0 peer-reviewed journal can be no journal at all, or it can be as many journals as there are readers.

Because the authors and reviewers in a Web 2.0 peer review system would each own and control their own part of the process, each would be free to set their own quality levels.

Authors would be free to follow whatever standards they felt were sufficient for ensuring that the papers they wrote contributed to the advance of scientific knowledge.

Reviewers would be free to put whatever level of effort they felt was appropriate into the review of any particular paper. And then authors would be free to respond to reviewers, by making alterations, or not, in each case according to whether the author felt that the reviewer had made a valid point.

And afterwards, online journals could consolidate the papers and reviews, based on their own criteria for selecting reviews and reviewed papers that they wished to publish.

In this proposed Web 2.0 system of peer review, everyone is free to do whatever they like, and it might seem that there is no difference between "Web 2.0 peer-review", and the informal discussions found in forums, wikis and the comment sections of weblogs. But the statement of intent is the critical component of the system. By posting a paper, and asking for reviews, an author is implying a commitment of effort on their part to produce quality output, and to respond to valid comments, if necessary, by updating or improving their paper accordingly. By advertising their willingness to review, a reviewer is indicating a commitment to make an effort to review content submitted. (A reviewer may of course decide that some submitted papers merit more reviewing effort than others, but it would be reasonable to expect that every review written should give some indication of how much effort had been put into it.)

Practical Design

So how would this work in practice? Firstly, authors should publish their papers as web pages (on their own websites), and declare their interest in having them reviewed. Each paper should have the following:

• A permanent URL
• A title
• An author (or authors)
• A date (so that reviewers can specify which version of a paper a particular review is referring to)
• Optionally there may be a history of changes made in response to reviews together with any comments that the author might have made about those changes.

A reviewer's website should have the following features:

• An invitation to authors to submit papers for open peer review. The invitation should indicate what type of papers the reviewer is prepared to review. Submission could be by web form, or by email.
• A page which contains an index of all reviewed papers. This index might be divided into particular categories as a function of the reviewer's opinion of the papers (for an example of this, see my list of specific categories below).
• Each review should contain the following items:
  • URL of paper being reviewed
  • Title and author of paper
  • Date of the version of the paper that the current review applies to
  • Description and review of the paper

My New Web 2.0 Peer-Reviewed Science Journal

To give substance to this design, I am offering myself as a reviewer for scientific papers. My official review page is at http://www.1729.com/peerreviews. This page contains a formal invitation to authors of scientific papers to submit their papers for review. It also contains a list of categories, which will be used to indicate my opinion (as a reviewer) of the quality of each submitted paper. These categories are:

• Interesting: the content of the paper may represent a useful advance in the state of science.
• Uncertain: I am too ignorant of the paper's subject matter to be able to form a useful opinion.
• Dubious: I consider the science in the paper unlikely to be of any value.
• Dubious for a well-known reason: same as the previous item, with the added criticism that the author's errors lack originality – their "science" belongs to a well-known category of bad science, such as parapsychology, creationism or "free energy" science.

Papers will also be ranked within each category according to my opinion of their quality.

Including very negative categories has an upside and a downside. The upside is that I can guarantee to authors that almost every paper submitted will be published and reviewed, at least to some degree, without necessarily compromising my own standards (whatever those might be). The downside is that authors might still be discouraged and depressed to see their paper go into one of the negative categories.

However, receiving a poor review is not as bad as not being published at all. Also, in a Web 2.0 peer review system, authors can submit their papers to as many reviewers as possible. There will be less incentive for any one reviewer to thoroughly review any particular paper, but this may not necessarily be a problem – an author can accept comments about a paper from all the reviewers who put even moderate effort into reviewing the paper. After several cycles of revision and reviewing, there is no reason why a paper which has been published and reviewed on the web should not be as high quality as a paper published in an academic journal.

What we may discover is that a web review process highlights different aspects of quality to those that matter in academic journals. Web reviewers may emphasise the clarity of papers so that they are accessible to a wider audience (including the reviewers). This may lead to a more informal approach to writing papers (something that the Internet seems to encourage in general). Reviewers may also be willing to complain if papers do not provide educational references, such as pointers to online tutorials which explain the concepts that a paper is based on. (Whereas authors of papers to be published in academic journals are expected to provide references to "seminal" papers, which may not be at all suitable for quickly learning a new topic; secondary material which paraphrases existing knowledge to make it more readable is not generally deemed worthy of formal citation.)

The Consolidation Application

If this concept of online peer-reviewed journals is to be truly Web 2.0, we will need to write software applications that consolidate papers and reviews according to various criteria (the simplest being that a "journal" consists of all papers reviewed by a selected set of reviewers). These applications haven't been written yet, however it should not be too difficult to write a simple "mashup" application which allows the selection of reviewers by the user, so that the application can then download the corresponding review sites and the papers reviewed by the review sites, and consolidate all the downloaded information into a database.

One thing that is needed to facilitate writing such an application is the development of relevant HTML microformats. I have defined an implicit HTML microformat for reviews in my own reviews site mentioned above. I have included some example reviews (one in each category), and these example reviews link to example papers which define an implicit HTML microformat for papers. (When I say "implicit HTML microformat", I am referring to the HTML class tags that I have used to identify those HTML elements which represent the required items for reviews and papers as listed above.)