Sue B. Moon's Blog

어제 미국 NRC에서 미국내 대학원 과정을 랭킹한 결과가 발표되었다.  그런데 전산학과 랭킹 계산에 있어서 큰 오류가 지적되어 전산과 랭킹은 내년 가을에 수정해서 공지하게 되었다.  가장 큰 문제가 저널 논문만 연구 실적에 계산했다는 점이다.  수정된 방법론에서 제일 큰 변화는 Thomson-Reuters Web of Science의 자료를 사용하지 않고 "based on CVs submitted by faculty members"을 쓰기로 한 것이다.  이게 무슨 말이냐 하면 저널 자체가 전산학 분야에는 좋은 논문이 발표되는 곳이 아니기 때문에 저널 피인용횟수에 근거하지 않고, 대신 미국 대학 교수들의 이력서에 있는 학술대회들을 고려하겠다는 뜻이다.  이렇게 정리된 학술대회가 265개.  어느 나라에서나 새로운 분야, 특히 기존 학문과 많이 다른 경우 평가가 쉽지 않음을 또 다시 느꼈다.

http://www.cra.org/ Computing Research Association은 미국내 200여개 전산학과를 대변하는 단체로 이번 조사의 문제점을 지적하고 NRC와 같이 새로운 평가 기준을 만드는 노력을 하고 있다.  지금 홈피에 가보면 "CRA Statement on NRC Ranking of Graduate Program"으로 이번 NRC 랭킹에 대한 입장이 정리되어 있다.

http://www.cra.org/resources/crn-online-view/reinvigorating_the_field/
Reinvigorating the Field
Musings from the chair by Eric Grimson, CRA Board Chair

http://www.cra.org/resources/conferencelist/
3DIM International Conference on 3D Digital Imaging and Modeling
AAAI AAAI Conference on Artificial Intelligence
AAMAS International Conference on Autonomous Agents and Multi-agent Systems
ACL Association for Computational Linguistics
ACSAC Asia-Pacific Computer Systems Architecture Conference
AICCSA ACS/IEEE International Conference on Computer Systems and Applications
AINA International Conference on Advanced Information Networking and Applications
AIPS Conference on AI Planning and Schedulig
AISTATS International Conference on Artificial Intelligence and Statistics
ANCS ACM/IEEE Symposium on Architectures for Networking and Communications Systems
APSCC Asia-Pacific Services Computing Conference
ARITH IEEE Symposium on Computer Arithmetic
ASAP International Conference on Apps for Specific Array Processors
ASE Automated Software Engineering Conference
ASIACRYPT International Conference on the Theory and Application of Cryptology and Information Security
ASPLOS Architectural Support for Programming Languages and Operating Systems
ATS Asian Test Symposium
AVSS Advanced Video and Signal Based Surveillance
BIBE IEEE Bioinformatics and Bioengineering
BPM International Conference in Business Process Management
CASES International Conference on Compilers, Architecture, and Synthesis for Embedded Systems
CAV Computer Aided Verification
CC International Conference on Compiler Construction
CCC IEEE Symposium on Computational Complexity
CCCG Canadian Conference on Computational Geometry
CCGRID IEEE International Symposium on Cluster, Cloud, and Grid Computing
CCS ACM conference on Computer and communications security
CGO International Symposium on Code Generation and Optimization
CHI International Conference on Human Factors in Computing Systems
CIKM ACM international conference on Information and knowledge management
CIVR International Conference on image and video retrieval
CLUSTER IEEE International Conference on Cluster Computing
CODES+ISSS International conference on Hardware/software codesign and system synthesis
COGSCI Annual Meeting of the Cognitive Science Society
COLING International Conference on Computational Linguistics
COLT Annual conference on Computational learning theory
COMM Conference on Applications, technologies, architectures, and protocols for computer communication
COMPSAC International Computer Software and Applications Conference
CONEXT ACM International Conference on Emerging Networking Experiments and Technologies
CP International Conference on Principles and Practice of Constraint Programming
CRYPTO Advances in Cryptology
CSB IEEE Computational Systems Bioinformatics Conference
CSCW Conference on Computer supported cooperative work
CSEET Conference on Software Engineering Education and Training
CSF IEEE Computer Security Foundation Symposium
CVPR IEEE Conference on Computer Vision and Pattern Recognition
DAC Digital Arts and Culture
DATE Design Automation and Test in Europe Conference
DCOSS IEEE International Conference on Distributed Computing in Sensor Systems
DDECS IEEE International Symposium on Design and Diagnostics of Electronic Circuits and Systems
DIGITEL IEEE International Conference on Digital Game and Intelligent Toy Enhanced Learning
DISC International Symposium on Distributed Computing
DS-RT IEEE International Symposium on Distributed Simulation and Real Time Applications
DSN Workshop on Assurance Cases for Security
EC ACM conference on Electronic commerce
ECAI Europeon Conference on Artificial Intelligence
ECCB European Conference on Computational Biology
ECCV European Conference on Computer Vision
ECOOP European Conference on Object Oriented Programming
EDBT Extending Database Technology
EDCC European Dependable Computing Conference
EG Eurographics
EMNLP Empirical Methods in Natural Language Processing
EMSOFT ACM Conference on Embedded Software
EPEP Conference on Electrical Performance of Electronic Packaging
ER International Conference on Conceptual Modelling
ESA European Symposium on Algorithms
ESEC European Software Engineering Conference
ESOP European Symposium on Programming
ESORICS European Symposium on Research in Computer Security
EUROCRYPT International Conference on the Theory and Applications of Cryptographic Techniques
Eurosys Eurosys Conference
FAST Conference on File and Storage Technologies
FCCM IEEE Symposium on Field-Programmable Custom Computing Machines
FMCAD Formal Methods in Computer-Aided Design
FOCS IEEE Symposium on Foundations of Computer Science
FSE International Workshop on Fast Software Encryption
GECCO Genetic and Evolutionary Computation
GRID IEEE/ACM International Conference on Grid Computing
HC Hot Chips
HiPC Int'l. Conf. High Performance Computing
HiPEAC International Conference on High Performance and Embedded Architectures and Compilers
HOTI IEEE Symposium on High Performance Interconnects
HotOS USENIX Workshop on Hot Topics in Operating Systems
HPCA International Symposium on High Performance Computer Architecture
HPCC IEEE International Conference on High Performance Computing & Communications Conference
HPDC ACM International Symposium on High Performance Distributed Computing
Hypertext ACM conference on Hypertext and hypermedia
i3D ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games
IAAI Innovative Applications in AI
IACR International Association for Cryptologic Research
ICAC IEEE International Conference on Autonomic Computing
ICALP International Colloquium on Automata, Languages and Programming
ICAPS International Conference on Automated Planning and Scheduling
ICASSP International Conference on Acoustics, Speech and Signal Processing
ICC IEEE International Conference on Communications
ICCAD IEEE/ACM International Conference on Computer-Aided Design
ICCCN International Conference on Computer Communications and Networks
ICCI IEEE International Conference on Cognitive Informatics
ICCP IEEE International Conference on Intelligent Computer Communication and Processing
ICCS International Conference on Computational Science
ICCV IEEE International Conference on Computer Vision
ICDCS International Conference on Distributed Computing Systems
ICDE International Conference on Data Engineering
ICDM IEEE International Conference on Data Mining
ICDT International Conference on Database Theory
ICEBE IEEE International Conference on e-Business Engineering
ICFP Intl Conf on Functional Programming
ICGSE IEEE International Conference on Global Software Engineering
ICICIC International Conference on Innovative Computing, Information and Control
ICIS International Conference on Information Systems
ICLP International Conference on Logic Programming
ICMAS International Conference on Multi-Agent Systems
ICML International Conference on Machine Learning
ICNP International Conference on Network Protocols
ICPP Int'l. Conf. Parallel Processing
ICPR International Conference on Pattern Recognition
ICRA International Conference on Robotics and Automation
ICS ACM International Conference on Supercomputing
ICSE  International Conference on Software Engineering
ICSLP International Conference on Spoken Language Processing
ICSM IEEE International Conference on Software Maintenance
ICST International Conference on Software Testing Verification and Validation
ICWS IEEE International Conference on Web Services
ICWSM International Conference on Weblogs and Social Media
IFIP-TM IFIP International Conference on Trust Management
IISWC International Symposium on Workload Characterization
IJCAI International Joint Conference on Artificial Intelligence
IJCAR International Joint Conference on Automated Reasoning
IMC Internet Measurement Conference
INFOCOM IEEE Conference on Computer Communications
INFOVIS  IEEE Information Visualization Conference
INTERACT IFIP Conference on Human-Computer Interaction
INTERSPEECH Annual Conference of the International Speech Communication Association
IOLTS IEEE International On-Line Testing Symposium
IPCCC IEEE International Performance Computing and Communications Conference
IPDPS IEEE International Parallel and Distributed Processing Symposium
IPMI Information Processing in Medical Imaging
IPSN Information Processing in Sensor Networks
IROS International Conference on Intelligent Robots and Systems
ISC Information Security Conference
ISCA ACM International Symposium on Computer Architecture
ISLPED International Symposium on Low-Power Electronics and Design
ISMAR IEEE/ACM International Symposium on Mixed and Augmented Reality
ISMB Intelligent Systems for Molecular Biology
ISMM International Symposium on Memory Management
ISMVL International Symposium on Multiple-Valued Logic
ISORC IEEE International Symposium on Object-Oriented Real-time Distributed Computing
ISPA IEEE International Symposium on Parallel and Distributed Processing with Applications
ISPASS IEEE International Symposium on Performance Analysis of Systems and Software
ISSAC Intl. Symposium on Symbolic and Algebraic Computation
ISSCC Int'l Solid State Circuits Conf
ISSRE International Symposium on Software Reliability Engineering
ISSTA International Symposium on Software Testing and Analysis
ISVLSI IEEE Computer Society Annual Symposium on VLSI
ISWC Intl Semantic Web Conference
ITC IEEE International Test Conference
ITNG International Conference on Information Technology: New Generations
IUI International Conference on Intelligent User Interfaces
JCDL/DL International conference on Digital libraries
KDD Conference on Knowledge Discovery and Data Mining
KR International Conference on Principles of Knowledge Representation and Reasoning
LCN IEEE Conference on Local Computer Networks
LCTES ACM SIGPLAN Conference on Languages, Compilers and Tools for Embedded Systems
LICS IEEE Symposium on Logic in Computer Science
MASCOTS Symposium Model Analysis and Simulation of Computer and Telecommunications Systems
MASS IEEE Internatonal Conference on Mobile Adhoc and Sensor Systems
MDM IEEE International Conference on Mobile Data Management
MICCAI Medical Image Computing and Computer Assisted Interventions
MICRO International Symposium on Microarchitecture
MIDDLEWARE  ACM/IFIP/USENIX International Middleware Conference
MM ACM Multimedia Conference
MMCN ACM/SPIE Multimedia Computing and Networking
MobiCom ACM International Conference on Mobile Computing and Networking
MobiHoc ACM Symposium of Mobile and Ad Hoc Computing
Mobisec International ICST Conference on Security and Privacy in Mobile Information and Communication Systems
MobiSys ACM SIGMOBILE International Conference on Mobile Systems, Applications and Services
NAACL/HLT Human Language Technologies: Conference of the North American Chapter of the Association for Computational Linguistics
NANOARCH IEEE/ACM International Symposium on Nanoscale Architectures
NAS International Conference on Networking, Architecture and Storage
NDSS Usenix Network and Distributed System Security Symposium
NIPS Advances in Neural Information Processing Systems
NOMS Network Operations and Management Symposium
NSDI Symposium on Networked Systems Design and Implementation
OOPSLA ACM Conference on Object-Oriented Programming, Systems, Languages, and Applications
OSDI USENIX Symposium on Operating Systems Design and Implementation
PACT International Conference on Parallel Architectures and Compilation Techniques
PADS ACM/IEEE/SCS Workshop on Parallel and Distributed Simulation
PERCOM IEEE International Conference on Pervasive Computing and Communications
PERFORMANCE International Symposium on Computer Performance, Modeling, Measurements and Evaluation
PLDI ACM SIGPLAN Conference on Programming Language Design and Implementation
PODC ACM Symposium on Principles of Distributed Computing
PODS ACM SIGMOD-SIGACT-SIGART Conference on Principles of Database Systems
POLICY IEEE International Symposium on Policies for Distributed Systems and Networks
POPL ACM-SIGACT Symposium on Principles of Programming Languages
PP SIAM Conference on Parallel Processing for Scientific Computing
PPoPP Principles and Practice of Parallel Programming
PSB Pacific Symposium on Biocomputing
PVLDB Proceedings of the Very Large Databases Endowment
RAID International Symposium on Recent Advances in Intrusion Detection
RE IEEE International Requirements Engineering Conference
RECOMB International conference on Computational molecular biology
RSS Robotics: Science and Systems Conference
RTAS IEEE Real-Time and Embedded Technology and Applications Symposium
RTSS IEEE Real-Time Systems Symposium
S&P IEEE Symposium on Security & Privacy
SAC ACM symposium on Applied Computing
SACMAT Symposium on Access Control Models and Technologies, ACM (was RBAC)
SAINT International Symposium on Applications and the Internet
SBAC-PAD International Symposium on Computer Architecture and High Performance Compuing
SC ACM/IEEE Supercomputing Conference
SCG ACM Symposium on Computational Geometry
SDM SIAM International Conference on Data Mining
SEA International Symposium on Experimental Algorithms
SEC IFIP International Information Security Conference
Securecomm International ICST Conference on Security and Privacy in Communication Networks
SenSys ACM Conference on Embedded Networked Sensor Systems
SGP Eurographics symposium on Geometry Processing
SIGCOMM ACM Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication
SIGGRAPH ACM SIG International Conference on Computer Graphics and Interactive Techniques
SIGIR ACM International Conference on Research and Development in Information Retrieval
SIGMETRICS ACM SIG on Computer and Communications Metrics and Performance
SIGMOD ACM Special Interest Group on Management of Data Conference
SIGSOFT International symposium on Foundations of software engineering
SoCG Symposium on Computational Geometry
SODA ACM/SIAM Symposium on Discrete Algorithms
SOSE IEEE International Conference on Sytem of Systems Engineering
SOSP ACM SIGOPS Symposium on Operating Systems Principles
SPAA Symposium on Parallelism in Algorithms and Architectures
SPLASH Systems, Programming, Languages, and Applications: Software for Humanity
SRDS Symposium on Reliable Distributed Systems
SSDBM International Conference on Scientific and Statistical Data Base Management
STACS International Symposium on Theoretical Aspects of Computer Science
STOC ACM Symposium on Theory of Computing
Supercomputing The International Conference for High Performance Computing and Networking, Storage and Analysis
TCC Theory of Cryptography Conference
TOPLAS ACM Transactions on Programming Languages and Systems
TPHOLs International Conference on Theorem Proving in Higher Order Logics (will be ITP from 2010)
UAI Conference on Uncertainty in Artificial Intelligence
UBICOMP Ubiquitous Computing
UIST ACM Symposium on User Interface Software and Technology
USENIX ATC USENIX Annual Technical Conference
USENIX FAST Conference on File and Storage Technologies
USENIX NSDI Symposium on Networked Systems Design and Implementation
USENIX Security USENIX Security Symposium
VEE International Conference on Virtual Execution Environments
VIS IEEE Visualization Conference
VL/HCC IEEE Symposium on Visual Languages and Human-Centric Computing
VLDB International conference on Very large data bases
VLSI Symposia on VLSI Technology and Circuits
WADS Algorithms and Data Structures Symposium (was Workshop on Algorithms and Data Structures)
WCCI World Congress on Computational Intelligence
WiMOB IEEE International Conference on Wireless and Mobile Computing, Networking and Communications
WoWMoM IEEE International Symposium on a World of Wireless, Mobile and Multimedia Networks
WSDM ACM International Conference on Web Search and Data Mining
WWW International World Wide Web Conference

지난 주초였나보다.  예전에 영국 캠브리지의 인텔 연구소에서 만났던 친구가 이멜을 보내왔다.  작년 서울에서 열렸던  Conference for Future Internet에서 발표했던 4장짜리 extended abstract를 어떻게 뒤져냈는지 읽고는 자기도 비슷한 거 한다며 같이 해보자고 연락을 한 것이다.  우리야 계속 그 작업을 하고 있었는데, 손이 딸려서 그 친구 밑으로 학생들이 여럿 있다기에 같이 해볼까 궁리하던 중에 한국학술진흥재단(KRF)의 글로벌 네트워크 연구사업 공지를 보게 되었다.  살펴보니 우리가 하려는 협력 형태에 잘 맞는 것 같아서 신청을 해보려고 하니 하나 걸리는게 있었다.  국내 연구자 연구 업적.  2006년 1월 1일 이후 SCI급 학술지 5편.

전산 분야에서는 SCI급 학술지는 외국 대학에서는 전혀 신경쓰지 않는다.  지난 달에 우리 학과를 다녀간 Microsoft Research India의 Ramachandran Ramjee 박사는 "irrelevant"라고 얘기했고, CMU 전산학과 학과장이신 Peter Lee 교수는 딱 잘라서 영년직 교수 심사에서 저널은 전혀 고려 대상이 되지 않는다고 못을 박았다.  독일 Max Planck Institute for Software Systems의 Paul Francis 박사는 자기는 저널이 1편밖에 없다면서 전혀 중요하지 않다고 얘기해주었다.  내가 SCI 저널 논문이 없어서 국내 과제 신청을 못할지도 모른다고 했더니 이해를 못하겠다는 표정을 지으며 돌아갔다.

나는 학술지 논문을 못쓰는게 아니다.  외국의 유명하고 일 잘하고 열심히 하는 연구자들이 저널 논문 안쓰고 연구에 매진하는데, 나도 그만큼 하려면 1초도 낭비할 수가 없기 때문에 일단은 학술대회에 신경을 쓰는 것이다.  우리 분야에서 저널은 학술대회 쓰고 거기에 미처 다 싣지 못한 내용을 좀더 보강해서 내거나 하는데 연구 내용상 더 붙일게 없으면 저널에 안 내기도 한다.  더 보탤 내용이 있는 연구 내용이면 작업 좀 더 해서 좋은 저널에 내고, 그렇지 않으면 학술대회 논문으로 일을 마무리짓는다.

이런 전산학 분야의 현실을 학술진흥재단에서는 아직 반영을 못하고 있는게 우리나라의 현실이다.  우리 학과에서는 지난 수년간 많은 신진 연구 인력을 보충했는데, 곧 나와 똑같은 벽에 부딪힐텐데, 어떡하나 걱정이 앞선다.  근데 이 SCI 문제가 단순히 과제 신청만의 문제가 아니다.  교수 승진 심사에서도 적용된다는게 문제다.  승진 심사 내용이야 대외비이기 때문에 누가 무슨 근거로 어떻게 판단을 했는지야 알 수 없지만, 국가 과학 정책의 선봉에 있는 학술진흥재단 및 과학재단에서 속수무책인데 하물며 다른 곳은 어떠랴.

그나마 BK21에서는 전산학의 경우 IEEE, ACM, USENIX 학회가 주관하고, 논문 게재율이 30% 이하이며, 50편 이상 게재되는 경우는 학술지급으로 인정해주는 것으로 바뀌었다.  근데 이것 또한 실정을 모르는 얘기다.  ACM 및 USENIX 주관 학회들은 대개 single track으로 3일.  해서 발표 논문 수가 30~35편을 넘지 않는다.

우리는 왜 이렇게 숫자에 급급해졌을까?  누가 어떤 연구를 했는데, 그 연구가 좋더라 나쁘더라를 이렇게 숫자로밖에 판단하지 못하는 불신의 사회가 됐을까?  내력이나 이유는 잘 모르겠지만, 바꿔나가야하는데 어떻게 해야할지 좋은 아이디어가 딱히 떠오르지 않는다.

5년 전에 귀국할 때 지도교수님를 찾아가 SCI 얘기를 하고 걱정을 했더니, 지도교수님께서 쓰신 추천서의 일부를 보여주셨다.  미국에서도 예전에는 학술지를 중요하게 생각하는 다른 분야의 사람들 때문에 단대나 대학 차원의 인사심의위원회에서 문제가 된 적이 있어서 National Research Council 등에서 만든 자료를 아래처럼 꼭 붙여서 낸다고.

[1] "If I had to find any potential issue with Professor XXX's tenure case, journal publications would be the only possible issue.  However, I would note that the conferences in which xxx has published are as
selective, and in some cases more so, than most journals.  Thus, many of her top conference papers are undoubtedly of the caliber of journal papers.  I am sure you are also aware that the best conference
proceedings in Computer Science are often more heavily read and cited than journals." [A,B]

[A] A 1994 report from the Nation Research Council notes, "A substantial majority of respondents to the CRA-CSTB survey of ECSE faculty preferred conferences as the means of dissemination by which to achieve maximum intellecutal impact; many fewer preferred journals," from Academic Careers for Experimental Computer Scientists and Engineers.  National Research Council, National Academy Press, Washington, D.C., 1994.

[B] In its study of publication and publication dissemination in computer science, the Research Libraries Group notes, "In computer science, conferences are the venue for presenting important new research, and competition for the opportunity to do so intense.  In fact, presenting a paper at the more prestigious conferences is preferred to publication in a leading journal" [from C. Gould, K. Pearce, "Information Needs in the Sciences: An Assessment, report prepared for the Program for Research Information Management, Resaerch Libraries Group, Inc. Mountain View, CA, pp.71]. Cited in footnote [A] above.

작년에 안식년을 가 있을 때도 이 얘기가 나왔었는데 거기 교수들은 Computing Research Association에서 발간하는 Computing Research Review에 1999년 David Patterson (U. of California Berkeley), Lawrence Snyder (U. of Washington), Jeffrey Ullman (Stanford U.)가 작성한 "Best Practices Memo: Evaluating Computer Scientists and Engineers for Promotion and Tenure"에서 "publish in archival journals .... Obligating faculty to be evaluated by this traditional standard handicaps their careers and indirectly harms the field."라고 천명해놓은 이후로는 저널은 신경쓰지 않는다고 한다.  저널에 논문내는게 handicap 정도가 아니라 심지어 해가 된다고 얘기해놨으니, 당연히 국가 과제 심사도 같은 기준으로 peer review되고.  이걸 읽고서 나는 그래도 국내 과제비를 따기 위해 저널 논문을 써야하는지, 외국 연구자들과 경쟁하기 위해서 학술대회 논문을 써야하는지.  결정이야 이미 내렸지만.  KRF, KOSEF 및 다른 정부 과제 심사 기준이 현실을 반영하는 방향으로 바뀌어나가고는 있지만, 아직도 갈 길이 멀게 느껴진 하루다.