The Elephant in the Room

I have a couple of posts on the back burner, waiting for clearance from my internal censor.  But rather than rush those out, I think it’s important to address the most important argument for staying with ACM.

I believe—and I think the computational geometry community believes—that the computational geometry community can comfortably run our flagship annual conference without ACM.  We have a standing offer from LIPIcs to publish our proceedings, at considerably lower cost and 100% open-access. While there may be some concern in the short run about financial backing, we can easily draw on the experience of other events with overlapping scope and similar scale—like EuroCG, CCCG, ESA, STACS, SWAT, WADS, and Graph Drawing—which have operated for decades without a sponsoring organization, and in many cases without even a persistent bank account.  In terms of the raw logistics of running a conference and publishing the proceedings, association with ACM is simply unnecessary.

But our current arrangement with ACM does have one significant advantage: brand recognition.

Hiring committees, tenure and promotions committees, funding panels, and even university accreditors base decisions in part on the reputations of the venues in which researchers publish. To first approximation, papers in “good” conferences (like STOC) and journals (like J. ACM) “count”, and papers in other venues (insert your own examples) don’t “count”, regardless of the quality of the actual work.

In the absence of expert opinion (or actual experts), evaluators turn to various kinds of secondary evidence to decide whether a conference is “good”:

Let’s consider these one by one.


Society sponsorship

The sponsorship argument seems the easiest to knock down.

First, many highly prestigious computer science conferences are not associated with ACM (or IEEE or SIAM, the other two big players in theoretical computer science). There are smaller professional societies like USENIX for systems and security, or ACL for computational linguistics, who publish open-access proceedings. A more inspiring example is machine learning, where several major conferences—including ICML, AISTATS, and COLT—publish open-access proceedings through JMLR, which (as a publisher) is really just a few guys with a web page.

Second, any real prestige attached to society-backed proceedings flows from authors to conferences and from conferences to their sponsoring societies, not the other way around.  ACM plays absolutely no role in the selection of papers at SOCG. Although its official policies suggest otherwise, the SOCG community neither seeks nor expects SIGACT’s or SIGGRAPH’s approval in electing our steering committee, or in the steering committee’s selection of PC chairs, or in the PC chairs’ selection of program committee members. (In principle, the SIGs could refuse to sponsor the conference because they don’t like the committees, but I don’t think anyone has ever seriously considered that option.)  As far as the SOCG technical program is concerned, ACM enters the picture only after all acceptance decisions have been made, and then only in its role as our publisher. The flow of prestige from author to venue to publisher is precisely the reason JMLR is so successful, and (I suspect) one of the main reasons SIGACT and SIGGRAPH are working hard to keep SOCG.

ACM does deserve credit for avoiding the worst failings of other professional publishers, like publishing fake journals and computer-generated garbage.  But as I’ve said elsewhere, “better than IEEE” is a pretty low bar.  The unprofessional behavior of other professional societies undercuts any claim that society sponsorship is a priori a mark of quality.

My OCD compels me to make one final observation: SOCG stands for “the [nth] Annual Symposium on Computational Geometry”.  The acronym “ACM” does not appear in the name of the conference or in the title of its proceedings.

However, regardless of whether professional society sponsorship should be a requirement for being considered a “real” conference, it is in fact a requirement at some institutions.  In particular, the bid for SOCG 2014 to be held in Kyoto required association with ACM; without the professional society backing, the host universities would not have been willing to sponsor the conference.


Indexing

Christian Duncan tells me that the Graph Drawing community has been considering whether to move their proceedings away from Springer’s venerable Lecture Notes in Computer Science to the open-access Leibniz International Proceedings in Informatics (LIPIcs) series, as both STACS and FSTTCS did in 2008.

The biggest concern we are getting from some researchers is that LIPIcs is not “indexed” in Scopus (or a similar database). Apparently, in some countries like Italy, the evaluation of their scholarly work, which affects their grant funding, promotions, and raises, is based on having their work published in journals and proceedings that are indexed by Scopus or Web of Science.

But in fact, the LIPIcs proceedings for STACS and FSTTCS are indexed in SCOPUS (with the exception of FSTTCS 2013, which was about six months ago).  There is every reason to believe that the-conference-formerly-known-as-SOCG (and, for that matter, Graph Drawing) would also be indexed.

LIPIcs claims that they also submit their proceedings for indexing in Thomson-Reuters’ Conference Proceedings Citation Index, but either  their Web of Science interface is too utterly broken for me to figure out how to access that index (which says something about how useful it is) or my university doesn’t subscribe to it (which says something about how important it is). Searching for a few key people reveals that Web of Science indexes only four years of SOCG proceedings, from 2008 to 2011.  So they’re irrelevant.

All LIPIcs proceedings are also indexed on DBLP, along with every other series produced by Dagstuhl.  And of course they’re also indexed by Google Scholar and Microsoft Academic Search, like every other web page on the planet.

It is hard to imagine that ACM would agree to include the-conference-formerly-known-as-SOCG in its Digital Library if we vote to go independent, at least in the short run. In the long run, however, ACM bills its Digital Library as “The ACM Guide to Computing Literature”.  If it wants to live up to this motto, the Digital Library must index high-quality proceedings (and journals) no matter who publishes them. Otherwise, it will continue to be “The ACM Guide to ACM’s Computing Literature” and people will continue to rely on other indices for complete records.


Rankings

Perhaps most importantly, because ACM has declined to offer us the same courtesy as it offered COLT and ICML when they became independent, an independent computational geometry conference would need a different name.  That new name necessarily will not appear in any existing conference ranking, and it would require a few years to build up enough evidence (typically in the form of citations) to be included in any new rankings.  In the short run, changing the name of the conference—even if we stay with ACM—would negatively impact the conference’s visibility and reputation.

(“Even if we stay with ACM” is not just irrelevant speculation.  Herbert Edelsbrunner has been advocating appending “…and Topology” to the conference name for years.)

But again, I believe this would be a short-term effect.  The computational geometry community is thriving.  As long as we maintain our current level of quality and impact, an independent computational geometry conference will eventually regain the stellar reputation and high rankings that SOCG now enjoys.

Of course, I might be wrong.  Maybe the computational geometry community doesn’t have the research momentum to regain the brand recognition of our flagship conference across a name change.  But in that case, we have a much more serious problem than our frustration with ACM!


So now what?

To sum up, moving SOCG away from ACM and changing its name would have some painful side effects.  Some researchers would have a significant incentive not to submit to the independent conference—because a paper at SOCG++ wouldn’t “count” toward their students’ job searches, or their tenure case, or their department’s allocation of faculty slots—which would be a loss both for those researchers and for the community.

Speaking only for myself, I believe these side effects would be temporary, that the computational geometry community is strong enough to rebuild any lost reputation, and that independence would improve the health and stature of the community in the long run. But inevitably, the side effects would be more painful and long-lasting in some departments than others.  This is not a decision to be taken lightly.

Advertisements

About Jeff Erickson

I'm a professor of computer science at the University of Illinois, Urbana-Champaign and the chair of the steering committee for the International Symposium on Computational Geometry.
This entry was posted in Uncategorized. Bookmark the permalink.

23 Responses to The Elephant in the Room

  1. Hi Jeff
    is there any evidence to suggest that leaving ACM will harm the CG community when it comes to getting awards (Turing/Knuth/Kanellakis) – I’m excluding Godel since that’s for a JOURNAL paper. The problem is that these are all long-term prizes so there’s no real way to tell if subtle biases (how the committees get formed etc) might make a difference.

    Also, I think you’re possibly underrating the damaging effects of losing ACM affiliation for institutes and countries that use this as a metric for professional evaluation. What might seem like a “temporary” blip could encompass a few generations of careers (measured in terms ot time-to-tenure/time-to-hire)

    • Perhaps. But then how do machine learning people, or computational linguists, or security researchers, or (now) complexity theorists, in those places get tenure? (Perhaps by sending more of their papers to journals?)

      • The machine learning folks have a conference that was attended by Mark Zuckerberg. I think it’s safe to say that they don’t have a problem signalling what venues are important. Also, they’re a much larger community (as is security and NLP) and so have many people placed at different institutions who can write letters, make statements about the conference quality, and so on. Complexity theory isn’t a good example because it’s just happening right now, and I suspect they’ll face many of the problems we’re discussing here. Note though that they ditched IEEE, not ACM: in CS, arguably ACM is a far more important label.

    • Looking at the Turing award list, it is US-centric, just as ACM.
      Not a problem for us (us=the non US computational geometers).

  2. Continuing the point about awards (which was brought up by one of my colleagues), consider for example the Athena Lecturer award: the winner is nominated by a SIG officer. Now why on earth would SIGACT choose (say) a deserving geometer when (a) there are many candidates and (b) CG is not part of the SIGACT community ?

    Even if your view on awards is like that of Oded Goldreich, awards are an important signalling mechanism both for individual quality as well as the quality of the community.

    • Organizing SOCG independently from SIGACT absoluty would not mean that computational geometers would not be part of the SIGACT community. After all, many computational geometers are dues-paying members of SIGACT, and mam cmputational feometers publsh in other SIGACT-sponsored conferences (like STOC and SODA). Computational geometers would still be eligible to run for positions as SIGACT officers, along with the complexity theortists, game theorists, learning theorists, and data structuralists.

      Perhaps I’m being unreasonably optimistic about SIGACT fairly representing the entire theoretical computer science research community, and not just the conferences that make it money.

    • I completely agree with you about awards as an important signaling mechanism. We should have more.

  3. Otfried Cheong says:

    Is “SIGRAPH working hard to keep SoCG”? I had the feeling they couldn’t care less. All the positive vibes coming from ACM seem to be coming straight from SIGACT.

    When I applied for “in-cooperation” status for SoCG 2007, ACM asked me to get consent from the SIG chairs, so I wrote them an email. The SIGACT chair responded immediately (and positively). I NEVER got a reply from the SIGGRAPH chair. In the end, ACM seems to have gotten consent from SIGGRAPH – they never lowered themselves to communicating directly with me, a mere organizer of a SIGGRAPH-sponsored conference.

    If we decide to stay with ACM, it may make sense to revisit the sponsorship by SIGGRAPH. While it has grown historically, it doesn’t make much sense at this point. Certainly there is a lot of graphics-related research in CG, but by the same argument we also do databases, robotics, big data, etc. And the culture of our field is very much the culture of theoretical computer science, and very different from graphics’.

    • Is “SIGGRAPH working hard to keep SoCG”?

      Yup. I’ve had some discussion in the background with Brian Wyvill and other members of the SIGGRAPH Small Conferences Committee. They definitely would like us to stay, but having faced the same dilemma themselves 20 years ago with their flagship conference, they’re also being incredibly understanding of our frustration with ACM HQ.

  4. Otfried Cheong says:

    Interesting – I had no idea that there were “Small Conferences” in graphics. If we stay with ACM, we should definitely talk to other “small conferences” and see if we cannot get ACM to adopt more flexible rules for small events, where financial risk is very limited (especially if its held on a campus). It’s counter-productive that ACM applies the same rules for events with 100 and with 1000 participants.

  5. INRIA researchers seem to be lucky on at least one point: As a member of the INRIA evaluation committee, I participated in many hiring and promotion committees in the past 9 years. I don’t remember ever hearing any reviewer mention stamps (ACM, IEEE, or whatever) on publications as a measure of quality of a candidate’s file.
    Committees are large enough to cover a large number of fields (only computer science, applied maths, and applications). Still, there is not always an expert on all fields, but when we are reviewing a file in a field that we don’t know well enough, we ask experts about the conference or journal selectivity in their field, we do not rely on stamps.

    Apart from this ‘detail’, I completely share Jeff’s point of view.

    • Jeff Erickson says:

      My experience at Illinois is the same. Independent of whether a conference is sponsored by ACM or IEEE or whatever, hiring and promotions committee members always ask “How good is that conference?”

      I personally find even that question frustrating; being presented at a good conference is neither necessary nor sufficient for being a good paper. The correct question should be “How good is that paper?” or better yet, “How good is this person’s research?” But at least “How good is that conference?” is asking about second-order data, instead of third-order data.

  6. Sure, of course I agree. Reviewers read (or at least they have a look at) papers when they know the field well enough to have a relevant opinion on the strength of the results. Otherwise, the quality of the conference/journal is at least an indication. Especially when there are several papers in high level conferences or journals.

    In any case, there are other data and aspects in a candidate file, it is definitely not just a list of publications.

    Anyway, going further on this topic would make the discussion diverge from its main topic.

  7. Coming from a smaller school, I actually feel like the ACM affiliation could be very important. I’m glad to hear that in many places, the paper and its venue are more important than any ACM label. (Hopefully, the letters help place this in context and are taken seriously by the committee!) But on my end, not too long ago even getting conferences publications recognized as being just as important as journals was difficult. The fact that some international colleagues would be hurt concerns me also; they can indeed publish more in journals to make up for it, but I’d bet people in those areas would tell you that that it makes things more difficult. I also think mathematicians who publish in our field might also find the ACM label helpful in their cases, since it provides a sort of brand-name recognition to people outside of our area.

    As long as ACM is willing to let international locations organize more independently, which sounds like the case, I’m not sure I see much of a gain from making a break. I’m sure we’d be fine in the long run, but it does seem like a risk, especially for the next few generations of more junior people.

    • I don’t think that conference papers are as important as journal papers. It depends on the conference and on the journal.
      Not all proofs are given in a conference paper, then it requires more work to complete the paper for a journal, in which proofs will be convincing (the word “proof” can be replaced by “benchmarks” for an experimental paper).

      That said, some conferences are more selective than some journals, a conference paper in maths is not the same as a conference paper in CS, etc
      Committees should take into account publication “habits” of each field.

      • Jeff Erickson says:

        I absolutely agree that committees should take field-specific publication habits into account. Unfortunately, the real issue is not how researchers should be evaluated, but how researchers actually are evaluated.

        If we do vote to go independent, it will be crucial for us to educate our colleagues—in person, in applications, in recommendation letters, and even just in informal conversations—that SOCG++ is still the flagship conference in a thriving and important field of research, despite the missing stamp, because some evaluators will look for the stamp first.

      • (Jeff) “the real issue is not how researchers should be evaluated, but how researchers actually are evaluated.”

        Sure – but we both mentioned places where they actually are evaluated in some reasonable way.
        Shall we structure our community to obey the bad practices of some evaluators, or shall we structure it to obey the reasonable practices of other evaluators?

        As you say, evaluators can be educated/informed… and it is our responsibility to work on this.

  8. Otfried Cheong says:

    UIUC and INRIA are big institutions in big countries – it’s easy for them to have an informed opinion. Unfortunately, in many smaller countries, evaluation is done entirely based on “indicators”. On my yearly evaluation I essentially only get full points for a paper in an ACM or IEEE-sponsored conference. In many Korean universities, the first round of candidate evaluation is done entirely by adding up impact factors of their papers. Theoretical computer science is so small here that the chance that the panel evaluating your research proposal includes one is close to zero. So it helps to have ACM conferences that are easily recognized by other computer scientists. And I don’t think I would have gotten $10,000 sponsoring for SoCG 2007 from KAIST without the ACM brand.

    Korea shouldn’t be a reason for a decision for or against ACM – there are so few papers accepted to SoCG from Korea that it wouldn’t make a difference. But I suspect it may be very similar in many smaller countries that rely entirely on “indicators” to evaluate scholarship.

    • frightening… these indicators have really been doing harm to research…

    • I think perhaps what I didn’t say well above is that I don’t believe this is an issue only in countries such as Korea, but also in smaller schools here in the US. The “indicators” used are not good, I agree, but it’s difficult to change that when you are the one being evaluated or the only one in a particular discipline at your school.

      • Jeff Erickson says:

        or the only one in a particular discipline at your school

        I don’t entirely buy this last point. The objection isn’t that computer science or computational geometry is exceptional, but rather that relying on secondary (or tertiary) indicators to evaluate researchers is poor practice in any field. It’s just as stupid/frustrating that some evaluators automatically consider papers in Science or Nature or Annals of Mathematics or Econometrica “better” than papers in other venues, or that some evaluators automatically consider papers in open-access journals less valuable than papers in traditional subscription journals.

  9. As Erin said, letters count in hiring committees.

    So, we will have to write in our letters (in a footnote) a statement about publication policy in the domain:
    That we left ACM (if it happens) for open access reasons, and that we consider SoCG++ as the best venue in computational geometry (and that we consider JoCG as a very good journal).

    People in hiring committees are researchers as us, facing the same problem with open access,
    increase ofprices from editors… They should understand the problem and trust our statement on the level of the conference/journal. Especially if the letter is a review letter of a candidate who is not your former student.

  10. Pingback: For Me, It’s About Open Access | Making SOCG

Comments are closed.