The paper “Experiments or Simulation? A Characterization of Evaluation Methods for In-Memory Databases” by Karsten Molka and Giuliano Casale has been accepted for full paper presentation at IFIP/IEEE CNSM 2015.
The paper compares the applicability of response surface methods and queueing network simulation in performance assessment of in-memory databases, based on an extensive experimental study on SAP HANA. Simulation is shown to be surprisingly accurate against several performance metrics such as response times, server utilization, energy consumption and memory usage.