SIGCRAP

For the last year I’ve been working on a new platform for publishing an open access academic journal. This is specifically motivated by the journal Communications in Cryptology, but ideally I’d like it to be useful for any journal that specifically accepts LaTeX. I should say up front that much of this work has been joint with Joppe Bos who has been wonderful to work with.

In the process of doing this project, I’ve learned a lot about the publishing process. It also caused me to understand why commercial publishers have managed to preserve their lock-in on academic publishing. Part of it is due to brand and the importance of reputation. Something like the Journal of Number Theory has been in existence since 1969, and the reputation of the journal has been built up over decades of performance. Once a journal is owned by an organization, it’s not easy to wrestle it away. There have been examples of this (e.g., the Journal of Algorithms, in which I had a paper long ago).

Software for publishing an open access journal

When we set out to start a new open access journal, I looked around for what software we could use. I spent a considerable amount of time evaluating Open Journal System (OJS). In my opinion this is by far the best option in 2023, but I still came to the conclusion that it really didn’t make sense for us to use it. There were several reasons:

1. OJS incorporates a traditional model of reviewing, whereas we wanted to use a new reviewing model that is more common in computer science where papers are reviewed by a committee (and their identities are public).
2. The software for OJS is written in PHP, and is about 26,500 lines of code at last count. The schema looks way too complicated for our needs, but still doesn’t support features that we want. Much of the important functionality is through third-party plugins, but support for these breaks frequently.
3. OJS has only crude support for copy editing, and no native support for LaTeX.
4. OJS is lagging somewhat on the capture of metadata. In particular they have no support for multiple affiliations after I reported the bug 30 months ago. This is partly due to the fact that it depends on a plugin and the schema is hard to evolve.

I think OJS deserves a lot of credit for changing the open access publishing landscape, and IACR uses it for two existing journals (TCHES and ToSC). They deserve a lot of credit for having advanced the open access landscape. I still think there are opportunities to improve in the future, but maybe not on that codebase. It has a lot of technical debt and is not adapting to the changing ways that people publish (for example, why are papers in PDF?).

A lot has been written about the cost structure for publishing a journal. Many journals from commercial publishers charge thousands of dollars for making an article open access. There are real costs associated with running an open access journal, but they don’t arise from IT costs of running servers. Most academic papers get very few readers, so a tiny $20/month website can hold hundreds of thousands of papers. The thing that costs real money is either the software development or the cost for any human labor. For more on this subject of the cost structure of a journal, I recommend reading this article.

Since I have donated my time for the last year to write software, it now appears that the cost structure for publishing a paper on cic.iacr.org will be about $5 plus the cost of copy editing (more on this later). One thing I noticed about OJS is that it is somewhat labor-intensive. Everything comes into the system through a form, and they assume that a human is entering the information. You might think authors do their job to enter correct information, but in my experience they don’t understand the importance of metadata to the reputation of their publications. Moreover, anything done by a human other than the author either has to be done as a volunteer or through paid labor. Since our goal is to drive the cost of publishing as low as possible, we sought ways to eliminate the need for human labor.

Architecture of our system

I wrote a paper with Joppe Bos earlier this year about how to incorporate metadata capture into LaTeX. We ended up publishing a shorter version of this paper in the TeX User’s Group journal (which is ironically only open access after an embargo). One goal of this work is to devise ways of using LaTeX so that authors can write their metadata directly into the paper, and have the system extract it in the process of publishing. This metadata goes straight into the indexing agencies and the website for hosting the papers. We believe that more publishers using LaTeX should adopt this strategy.

Our system is now about to launch for the journal “Communications in Cryptology“. The workflow for a journal has many steps, but roughly breaks down into three major pieces

Within these three clouds, there are multiple steps in the workflow, but they form the major parts and most importantly there is a clean transfer from one step to the next. Because of this, our system consists of three parts:

1. A reviewing system (we currently use a modified version of hotcrp).
2. A production and copy editing system that accepts LaTeX from authors and compiles it in the cloud. In the course of doing this it extracts all metadata (e.g., title, authors, affiliations, bibliography, abstract, funding, etc). This builds upon the work in our paper and our LaTeX style file is crucial to the process.
3. A web hosting system that is relatively simple. We need to import papers and assign them DOIs, and we need to export metadata in various formats like OAI-PMH, RSS, and others.

The copy editing and production server is fairly complicated, since it has to run LaTeX in the cloud and keep track of feedback between copy editors and authors. The authors get a chance to view their LaTeX errors and make changes, and then it gets sent to the copy editor. We use a LaTeX log parser to extract the most important warnings and errors from the LaTeX log, and we inspect the references to make sure they have DOIs and other required elements. The author gets a set of items to respond to, and they can upload their final version.

Once they upload their final revised version, the copy editor is presented with a view where they can view the changes to the PDF, the changes to the LaTeX sources, and the list of items the author was asked to change.

Once the author ↔ copy editor interaction is completed, the paper is finalized for publication. When an issue of final version papers has been collected, it gets forwarded to the hosting system where it is actually published. This transfer also constitutes the unit of archiving for an archiving service like LOCKSS.

The system we have built is not open source at this point, in part because I don’t want it to be used to simply improve profitability of commercial publishing. I undertook this venture to allow IACR to publish papers as cheaply as possible without access restrictions. Unlike some other societies that maintain a staff of people that have to be supported by selling publications from members, IACR has for years had volunteers who maintain the scientific programs though their volunteer effort. I’d like to see more scientific societies adopt open access policies and streamline their cost structure in the process.

Now that almost all academic publishing has moved online, there are a lot of assumptions that start to look silly with time. The whole notion of a “Periodical” doesn’t make much sense. Traditionally it meant articles that were bound together and published on a regular interval (e.g., daily, weekly, monthly, or quarterly). The reason they were published on a schedule is that they were bound together as a block of paper, and then those packages were physically shipped to readers. We still cling to this publishing culture in many ways.

These bound issues of a journal were a very strong part of my academic career. When I was in graduate school I would stop by the mathematics library every day and visit the rack against the wall to see what issues had just arrived. That was how you found out about most new mathematical research (that photo below is the room in Altgeld Hall at the University of Illinois).

Now that we no longer bind things into issues, we still cling to the periodic publication of “issues” of journals. This is baked into the review process for the new journal Communications in Cryptology, but it’s also baked into the culture of IACR Transactions on Symmetric Cryptology (ToSC) and IACR Transactions on Cryptographic Hardware and Embedded Systems (TCHES). Papers are submitted on a deadline for an issue, and then the papers for that issue are reviewed by a committee, producing a list of accepted papers for the issue as a synchronous action.

Because we are using this method of reviewing, it makes some sense for us to talk about “issues” of a journal. The fact remains that once a paper is accepted, the author(s) still have to produce their final version and go through a copy editing phase. There’s no particular reason for this to proceed on a synchronous schedule for the issue, so it’s possible that one paper in the issue might be published before another. Journal issues need not operate like trains, with all of the cars showing up at the same time. For some reason we persist with this.

Then there is the issue of why we have “volumes”. Why would we collect several issues together into a volume? Traditionally the notion of a “volume” corresponds to the year that it is published ToSC and TCHES even number their volumes as the year, but oddly some papers in volume 2023 were published in 2022 and Volume 2019, Issue 4 of ToSC was published in 2020. I guess since volumes don’t align exactly to years, we shouldn’t number them by the year. In fact, I would argue that we don’t need volumes at all since volume # and year # are redundant in a bibliographic reference.

Then there is the issue of why we have “Issue numbers”. ToSC had Volume 2020, Issue 1 but then also Volume 2020 “Special Issue 1”. I guess that means Issue 1 was “not special”, and Issue 2 was actually Issue 3. 😕

Then there is the issue of why we have page numbers. In the old days it was important to be able to find the paper in a huge bound volume. You want to immediately turn to the paper in that big clunky book.

Some journals have innovated in this space to have a “paper number” in the issue instead of page numbers, but that doesn’t make a hell of a lot of sense to me. Are the papers supposed to be read sequentially or individually? They aren’t chapters in a narrative book – they are independent authored works. Many authors have been arguing that we should list authors in random order on a paper rather than alphabetically, because Anthony Aardvark would always be listed as “first author”. In some fields joint authors are listed alphabetically unless there is a difference in their levels of contribution. Following the same line of reasoning as random author order, papers could be listed in the table of contents or the web page in random order each time.

I would argue that all of these sequential identifiers are unnecessary in today’s world. What you need to identify a journal article is the URL for the article. Unfortunately a lot of web culture doesn’t really believe in persistent URLs, so the notion of a DOI was invented as a persistent unique identifier for the paper. The DOI can always be tacked onto the end of https://doi.org/ to get a redirection to the current URL for the paper. This DOI is really the only thing you need to identify and locate a paper. That’s why

You should always include a DOI in a bibliographic reference!

A friend of mine recently reviewed a number theory paper that is over 100 pages long. He found a critical error on page 52 that may invalidate the main result of the paper.

This raises the question of what the hell is reasonable to expect from a peer review system. It’s a herculean effort to produce a research paper in number theory that is over 100 pages long. It’s also a herculean effort to do a referee’s report on such a paper. The author may have spent years on producing the result – why would a referee spend this much time when it does essentially nothing to advance the career of the referee? Such observations are not unusual these days – Steven Galbraith brought it up in an interview with Ben Green, and much has been written in recent years about the changing nature of peer review.

Some people rely upon peer review to confirm that a result is accurate. That’s pure folly considering how complicated some published results have become. It’s even more doubtful for data-driven machine learning research, which often isn’t even repeatable. I think at best we can count on the peer review process to convey a sense of plausibility to the results, but ultimately it is the responsibility of scientists to study the result and examine it from every point of view, in careful and deliberate pursuit of knowledge.

Some people rely on peer review to tell them what is important to read. There is far too much research being produced these days for a researcher to read and understand everything, and our reliance upon screening tools have proven to be very important for having a productive research career. Unfortunately this has a potential downside as well, since it may steer a research community toward the “safe” side of science.

The recent proposal for IACR to start a new open-access journal got stuck on this issue (among others). Some people are completely reliant upon the reputation of their publishing venue to bolster their research reputation. They see it as a threat to their reputation if their research is published alongside “less interesting” research, and they need to maintain this selectivity to prove to their peers that they are among the best researchers. I suspect that the underlying problem here is that a lot of research has very narrow appeal, and people are grasping at whatever they can to claim relevance for their work. OK maybe that is too harsh, but it’s a lingering doubt of mine.

The fact of the matter is that we live in a world of competitiveness. We compete for jobs; we compete for awards, and we compete for attention. For any person is driven in their career, they may be encouraged to use whatever means possible to eke out an advantage in a very competitive landscape of academic research.

Personally I look forward to a more open discussion of research. We used to need peer review to limit the number of papers because publishing on paper is expensive. In a world where all research is downloadable and hosted for almost zero cost on the world wide web, peer review has instead been propped up as a mechanism for selectivity and filtering. Personally I think scientists should be more open to new ideas, and less dependent on what conventional wisdom tells them to read. Do your own homework.

One thing that I think could improve the peer review process is to publish more than a boolean to say that “this is acceptable research”. We should be asking reviewers to rate papers for their scientific contribution, their plausibility of correctness, their novelty, their honesty in citing previous work, etc. There is a good collection of recommendations on this for the Eurocrypt 2022 program committee. The change I might make is that instead of focusing on which papers to include, we focus on only eliminating the really bad papers, and publishing scores for the factors that we typically rank things on. This is in conflict with the tradition of computer science where a publication is essentially the same as a conference talk, because we don’t have enough speaking slots to accommodate all of the research being produced. I still think we need to adapt.