You are here

Evaluating scientific journals

General

Scientific journals constitute the main publication channel in many domains, and the international scientific community has through peer review developed a quality control system for the content of these journals. Although not always water-tight, the fact that their articles have been screened and approved by (in general) two experts in the field constitutes an external positive evaluation of the described research results.

The international scientific community is also paying much attention to quantitative data that are supposed to measure the "impact factor" of the journal, based on the number of citations the journal receives in other publications worldwide. We refer to the annex about bibliometrics for a more detailed description and critical analysis of these methods and their resulting ranking lists of journals, but it goes without saying that this citation-linked impact factor will most often be of little relevance for the impact of a publication on the developing world: the majority of citations that result in the impact factor are coming from research institutes in the North.

In any case, evaluators should be aware of the necessity to try to obtain themselves some expert evaluation of the quality of the journal content; we give some hints for doing this below.

Objectives of evaluating scientific journals

The possible objectives of an evaluation of scientific journals may be seen from the viewpoint of various stakeholders:

  • Journals, particularly journals in the South, and/or lesser known journals, and/or journals not in English.
    • to achieve higher quality
    • to attract more or better authors
    • to get recognition
    • to widen their dissemination or to increase their number of subscriptions
  • Researchers
    • to decide where to submit their manuscripts
    • to improve scientific quality
  • Universities and research institutes.
    • to strengthen their research capacity, in the North as well as in the South
    • to coaching their researchers
    • to evaluate their researchers
    • to choose subscriptions for the library

Authors of a specific guide will choose objectives according to their particular needs. It is obvious that the evaluation criteria may strongly differ according to the objective at hand. Nevertheless, the intrinsic quality of a journal will always be a primordial criterion, and we will discuss this first in more detail. Afterwards, we will look at other criteria, depending on the chosen objective.

The intrinsic quality of a journal

The full evaluation of the quality of a journal can, of course, only be performed by an expert in the field. Nevertheless, we list below the most important dimensions to be considered during an independent evaluation of a journal’s quality:

Scientific quality

The scientific quality of a journal depends, of course, on the quality of the individual articles, depending in their turn on:

  • the intrinsic quality of the research that is reported and the writing ability of the articles’ authors,
  • he applied peer review: the notoriety of the Editor; the composition of the editorial committee; the choice of reviewers; the selectivity of the reviewing process,
  • the quality of editorials and of letters to the Editor.

A very selective editorial policy, leading to a high rejection rate, will in general result in a high quality journal. Important scientific papers are cited more frequently in other papers than less important papers, and this has lead to the quantitative assessment of a journal’s quality through the calculation of the average number of citations received by its articles during a given period of time (e.g. 2 years), which is called the journal’s "impact factor" (IF). The Journal of Citation Reports publishes yearly ranking lists of journals ordered by subject categories, with their IF's and other quantitative measures. These ranking lists, however, strongly discriminate in favour of journals published in the North (see the page on bibliometrics for a discussion on this aspects.)

Relevance

The relevance of a journal is the extent to which it is appropriate for its intended use. Therefore, we will come back to this criterion below, when discussing in more detail the various objectives for the selection of a journal. The most important aspects of a journal to consider when evaluating its relevance are:

  • is it relevant for development?
  • is its content scope relevant for the subject at hand?
  • is it appropriate for its intended audience (with respect to its language, its level of complexity,...)?

A journal may be considered to be relevant for development when the research results described in its articles are often relevant for the developing world. This may be a consequence of the subjects treated in the journal, but it may also result from the way in which articles are solicited from possible authors. Since many developing countries are suffering from the same problems, a journal with a large number of publications by researchers from one such country will often be very relevant for other developing countries.

Availability and visibility

Since by definition the intention of a publication in a scientific journal is to 'make public' the results of some research, it would be a waste of effort to publish in a journal that nobody can read because it is not available in important libraries or on the Internet. Furthermore, in view of the large number of scientific journals, the chance that a potential reader finds an article through accidental browsing is almost zero (except maybe for the papers in the most prestigious journals). Most articles are discovered through a search operation in one of the bibliographic (or indexing) journals or databases, or by being cited by another paper. The interested reader is at that moment confronted with the problem to get hold of the full text of the article that he wants to consult. For these reasons, the availability and visibility of a journal are determined by the following aspects:

  • Is the journal published by one of the large international publishers or by a local enterprise (with limited distribution channels)? Does it receive institutional support (from an institute, a university, a scientific or professional society, a foundation, others)?
  • Geographical coverage: is it a really international journal or only a local journal with mostly local readership?
  • Language of the journal: English is now widely accepted as the universal language of science, and means that publications in other languages (even important ones such as French or Spanish) will have a more limited readership, as long as good quality automatic translation software does not become available.
  • Regularity (periodicity) of publication: a journal with a very irregular publication schedule is easily neglected by potential readers.
  • Is the journal covered by major international bibliographic databases, so that readers can easily be alerted about its contents?
  • Is it published as an Open Access journal, with global availability without delay or financial barriers (the "golden road")? Such open access journals are available free of charge from the Internet. More and more articles published in the commercial journals are now simultaneously made available through the freely accessible institutional or subject repositories that follow the OAIMHP (Open Archives Initiative Metadata Harvesting Protocol), so that they will turn up in searches by the major international open access repository harvesters (e.g. BASE, openAIRE, OAIster) and Google Scholar? (This is called the "green road"). Some publishers allow this open access archiving, but not all of them. Is the journal ad hand allowing this alternative publication channel? (An extensive list of the policies applied by various publishers is available from Sherpa/Romeo). 
    While many researchers worldwide still use the selective ISI group of journals as both their main source of scientific information and as their favoured outlet for publishing research results, there is an increasing amount of evidence that open access publications are generally cited more often than papers available via a paid subscription only, thus delivering enhanced visibility, use and scientific impact.
  • Does the publisher specify under which copyright provisions his journal is published? (If nothing is specified, one should in principle ask the editor’s permission before applying or further elaborating on his results, which may be an impediment to useful applications in the developing world. Many of the Open Access journals and repositories automatically include a license that allows using the results – under the condition of proper citation of the original work – as specified by one of the Creative Commons licenses.)

All these aspects are not always independent: they are often linked to each other, or influencing each other.

Journal quality for specific stakeholders

According to the objective of specific stakeholders, the elements to take into account may be different. We can, e.g., discern the following cases:

  • The journals themselves:
    • The journal editors should be aware of the criteria that determine journal quality. An essential element is an international editorial board with reviewers, who – independent from the authors – can follow a  given set of selection criteria. Some journals apply here the so-called "double blind" procedure, where not only the authors do not know who reviewed their paper, but where also the reviewers do not know the name of the author (although reviewers can sometimes guess who is the author of a paper, especially in small scientific communities...). The most important rule, however, remains that the reviewers should be able to form their judgement independent from personal relations with the author.
    • Regarding relevance, they should determine for which kind of articles there is a real need, and describe the scope of their journal accordingly.
    • The broadest availability can be reached through an Open Access policy. As a consequence, there will be no income from subscriptions, so that the journal has to look for a different stable financial model. For a discussion on such models, see the Open Access Scholarly Information Sourcebook.
  • The authors:
    • The scientific quality of their paper depends on the research they performed and on their writing skills. Through publication in a high-quality journal the authors often wish to obtain international recognition and to improve their c.v., but this may come at a price since many of these journals apply rather high publication charges.
    • Publishing in a journal that is relevant for the subject of the submitted paper certainly increases the chance that the paper will reach its intended audience.
    • In order to make a paper visible and available to a broad readership, nothing goes above Open Access publishing, either in an Open Access journal or in a regular journal, accompanied by the archiving in an institutional repository. Authors should insist that their university installs such a repository, for which several Open Source solutions are available.
  • Universities and research institutes:
    • Achieving an optimal collection development has always been an important task for all research departments worldwide, but it can be especially problematic for libraries in developing countries where budgets are scarce and high quality journals often prohibitively expensive. The collaboration with a research group in the North can offer the best means of keeping informed about scientific progress.
    • Whatever budget available, it will be a delicate task to spend the money wisely such that with the available means the optimal collection of journals can be acquired that are relevant for the local students and researchers. This can only be done through a close collaboration between the university administrators, the researchers and the librarians.
    • From the above given discussion about the Open Access publications follows that priority should be given to making as much as possible publicity for all qualitative sources that are available for free on the Internet. There are many ways in which this can be done, e.g., through a library website with lists of Open Access harvesters and journals, preferably arranged by subject.

Building the specific guide

The steps to be followed by the authors of a specific guide for evaluating scientific journals are very similar to those taken for the evaluation of other aspects: see the page on writing the specific guide.

References