One aspect of Big Data is the arrival pattern of the data being stored/analyzed.
Yesterdays Rangers win in the ALCS is a excellent example of this. At justSignal we monitor all of the MLB teams and when the Rangers won we experienced a phenomenal spike in data arrival. Right now – which is fairly representative of normal volumes – we are receiving 7.87 Tweets per Second from Twitter. Last night we peaked out at 43.5 Tweets per Second.
Here is what that looks like graphically:
Managing that kind of data arrival while continuing to generate meta data dimensions (transformations) and deliver data to clients in real time is no small task.