Don’t Forget To Consider The Hardware When Optimizing Systems To Handle Big Data
In this episode of the TPM podcast show Steven Garcia, Head of Engineering and Operations for Cloud Solution Services at SAP talks about his experience in building high performance Big Data processing systems that handle millions of transactions a second. It’s not enough to just tune your app, you need to profile your data to understand usage patterns and optimize your system to accommodate the strengths and weakness of the available hardware. Some servers have different memory handling characteristics and the read/write speeds can be different for various types of storage. Just one of Steven’s ideas is that the 20% of your data that handles 80% of the use cases could get stored in memory or SSDs.
Steven also explains the value in storing ALL your data to allow for greater depth in analyses rather than keeping only truncated datasets that limit analyses to little more than bean counting.