NSF Review Panel

I had an opportunity to participate in a National Science Foundation (NSF) review panel some time ago. The experience was rather different from the project reviews I had in Korea, and I write about it in this posting.

NSF is the core funding agency to researchers in science and engineering in the US. We have a similar
foundation, KOSEF (KOrea Sience and Engineering Foundatin), modeled after NSF. NSF is renowned for its competitive and fair review process and effective project management system. One of their grants, the NSF career award, is a much coveted prestige to assistant professors in the US.

I was asked to review about 10 proposals in my area of expertise. A proposal has a bio of the principal investigator (PI), a budget plan, a list of granted and pending NSF grants, and a 15-page or shorter technical description. The page limit of 15 keeps the proposal concise and to the point.

The budget plan was quite interesting. Unlike Korean professors who are paid full 12 months a year, professors at US universities are paid only 9 months a year and have to bring in funds to cover the remaining 3 months. From one NSF grant or multiple grants combined, a professor could cover only up to 2 months a year. That is, a professor cannot depend solely on NSF grants, and should have other funding sources (e.g. Dept. of Energy, Dept. of Homeland Security, grants from industry) for full 12-month support. The institutional overhead is markedly higher than what we pay in Korea: 20 to 40%. Then to pay for grad students, travel, and equipments. The budget plan should have exact amounts for professors and grad students, but for travel and equipments, only a ballpark figure would suffice. No detailed itemization is requested. In comparison, we have to specify detailed items, e.g., $200 for printer toner, $500 for conference registration, etc., for all government-funded project proposals in Korea.

The duration of a project was quite reasonable. Most projects are 3 years long and a small percentage up to 4 years.  Centers, like ERC and SRC of KOSEF, are for longer terms.  This long-term, flexible programming is attractive to researchers, as very little can be done in a year on any work of research.  All reviewers were asked to write about intellectual merits and broader impacts of proposals.  Those proposals that stated their intellectual merits and broader impacts up front clearly were easier to review.  Though served in many conference program committees before, I was unaccustomed to evaluating unfinished work based on potentials.  Is a new idea good enough?  Or a well-formulated problem with partial solutions? I heard from friends in the US that you write an NSF proposal when the work is half done and an DARPA proposal when you have built a decent career track record.  Reading the proposals, I could see why.  A proposal with a publication or two on the proposed idea stands on a firm ground in comparison to other papers. Still, there is room for proposals without publications, but very well thought-out, intriguing problems and well-written formulations.

For a big project proposal, I heard that people bring in a "red team" to tear down the proposal in preparation to pieces. That is, the red team looks at every angle and points out weaknesses from a reviewer's perspective.  Those preparing the proposal take all the feedback from the red team and rewrite the proposal before the final submission to the funding agency.  I can see the benefits of having a red team. Some proposals read well, but some didn't make the main problem and contributions up front and clear.

At the end of the review, we got together to write feedback on the proposals. We wrote carefully and encouragingly at every possible opportunity in light of acknowledging the potentials of the work.

The whole review proecess was grueling (10 reviews in a week and the meeting from 8:30am to 7pm non-stop for two consecutive days), but rewarding. At the end, we were asked to behave as if we were flashed by the MIB (Men in Black) thing -- we remember no name, no idea, no nothing. This is just a record of how I did, not what work I did.