Artifact evaluation is only open to accepted papers. Authors should be notified, ideally starting with the Call for Papers, of the AEC process, so they can prepare their artifact for it. Typically, artifacts are due a week after paper acceptance notifications go out, giving authors time to perform final packaging. Typical AEC review times have been about one month.

During initial paper submission time, authors should be asked both whether they intend to submit an artifact and the skill set their artifact will require to evaluate. This gives the AEC chairs the opportunity to pick or strengthen their membership to accommodate the required skills. (Of course, not all skills will eventually be necessary, since this is the list of skills required for submitted papers, not accepted ones.)

Every artifact is reviewed by two committee members. They are given the accepted version of the paper to read before examining the artifacts. They are then asked to individually submit a detailed review along these lines:

• Summary and contributions of the paper.
• Artifact packaging and reproducibility.
• Artifact implementation and usability.
• Overall assessment.
• On what platform/how was the artifact evaluated.

Some AECs have used a five-point scale for assessment:

1. Significantly exceeded expectations
2. Exceeded expectations
3. Met expectations
4. Fell below expectations
5. Significantly fell below expectations

In practice, however, even a two-point scale (“met or exceeded expectations” vs. “failed expectations”) has sufficed.

If there is a difference in overall assessment between the two reviewers, they are asked to try to reconcile it between themselves. Chairs break ties.

The final decision needs to be communicated to authors about a week before the camera-ready deadline, so authors can include the decision in their paper. The AECs have stipulated nothing about how (or even whether) to report it.

One complication with the current AEC structure, coming after the papers have been accepted, is that revewiers are required to base their views on the paper as is. Sometimes reviewers are unhappy with the paper itself. They needed to be reminded that they cannot allow this to bias their views too much: the AEC is only charged with deciding whether the artifact meets the expectations set by the paper, no matter how low or incorrect those might be! This is of course a difficult emotional barrier to ignore.

Conflicts of Interest

In general, we treat conflicts of interest precisely as we would for regular conference papers.

One issue to consider is who can submit artifacts. If artifacts become part of the evaluation process of papers, the status of AEC members becomes akin to that of PC members. Since we have a two-phase approach where paper decisions are made independently of and before artifact evaluation, only those with direct conflicts with the chairs need to be barred from submitting artifacts. This means, however, that papers by those authors would not be able indicate their artifact quality, which might raise questions with some readers. Such authors are therefore welcome to mention in their paper that they are prohibited from submitting an artifact due to the conflict of interest.