computer "science"  predictive theory  antagonistic experiments  datasets  about

Computer "science"

Critical look at state of research and a path forward.

Computer science deals with machines, employs mathematics and has perfectly reproducible experiments. Science! Right? What if the mathematics do not predict real-world behaviour? What if the experimental design does not allow to generalise beyond a handful problem instances? Should we just call it informatics instead, or should we seek some way to do better? A longer discussion of different philosophies and computer science traditions can be found on the about page.
The science paradigm has not been part of the mainstream perception of computer science. But soon it will be. — Former ACM President Peter J. Denning [source]

Why computer science should try to live up to its name

Does computer science already live up to its name?

It could be argued that computer science leaves itself open to the following criticisms: Asymptotic computational complexity has been a staple of computer science for many decades and while it is a very useful tool, it is perhaps not surprising that simple formulas can only provide some overly simplistic view of computational realities which are often more complex [more details]. There have been some efforts to address cherrypicking and replication problems:

How could computer science try to live up to its name?

While previous efforts have been a step in the right direction, a general culture change is needed, specifically: Clearly, a lot of it comes down to peer-review of publication manuscripts and applications for grants or positions. If reviewers set unrealistic expectations that all new approaches should be better in every regard, then authors are forced to run a tailored benchmark that shows the new approach in the best light. Furthermore, if old approaches only need to be better in one way, it would almost guarantee getting stuck with bad approaches. If reviewers only consider experimental or purely theoretical results, then there is no incentive for predictive theory although it has larger scientific value.

How does computer science match up with social sciences?

One way to reflect on computer science, is to compare it to other sciences. While social/medical sciences investigate a human population, computer science investigates a population of problem instances:
Computer ScienceMedical/Social Sciences
PopulationProblem instancesHumans
SampleProblem instances used in experimentsHumans participating in experiments
Sample SizeE.g., handful of datasets (N < 10) Typically, N > 100
Sample TypeTypically a non-random, "cherrypicked" sampleTypically a randomised convenience sample
Sample publicly availableTypically yesTypically no
Theorye.g., asymptotic complexitypredictive/descriptive theory
Independent Variableold approaches vs new approachcontrol group vs intervention
Dependent Variableperformance measuresvarious measures
Study Designwithin-subject/repeated-measures experiment (without order effects)various
ReproducibilityLimited to "sample"See replication crisis.
AnalysisDescription of effect sizesStatistical/effect size analysis
GeneralisabilityCompletely subjectiveCritically discussed and investigated
In medical/social sciences most of the statistical analysis is aimed to rule out that the observed effects are merely sampling artifacts. For non-random samples as used in computer science this is not possible and potentially all reported improvements generalise very poorly to other problem instances. It is therefore important to interpret such results very carefully and try to avoid overreaching generalisations. Furthermore, antagonistic sampling can help a bit (that we can think of as the opposite of cherrypicking), i.e., looking for problem instances that are likely harder than a random sample would be.

This is all old news!

It would then appear the content is non-controversial and there might be a desire to make sure that it is old news for everyone. One simple way of doing this is to share this website, or create similar content that perhaps could be linked here as well.