Synchronicity Risk
I'm in Dallas Texas this weekend getting ready to experience the 2024 solar eclipse (as long as the clouds clear up). August 21, 2017 was the last time a solar eclipse blanketed the United States and this will be the last US eclipse until 2045. We decided to use the opportunity to visit some good friends in that part of the country and explore Dallas which is in the path of totality.
Celebrations interrupted
August 21, 2017 was a Monday. Around 2pm as the eclipse started to hit the west coast, we started receiving alerts and alarms from our production systems as they received large spikes of traffic. Being in the business of social media archiving, we were used to viral events but typically those viral events were focused on a single city. The eclipse was happening to many cities as it moved across the country causing major spikes in traffic as videos and comments were shared.
As soon as we realized these alarms were being caused by a celestial event we were able to scale up systems and adjust to get the systems working again. In another weird coincidence, this major event also lined up with my 4th work anniversary. The team came together to both solve the problem and also enjoy ice cream in the Durham plaza. I can definitely say I haven't had such a unique anniversary celebration before or since.
Sympathy of two clocks
Synchronicity is an impressive phenomenon that occurs when two events that have no discernible causal connection, but appear to be related. A great example of this comes from a 1664 experiment by Christiaan Huygens trying to solve one of the biggest challenges of his day, oceanic navigation. In order to determine longitude you needed to know the time where you were using the sun and the time at your port city. He wanted to show that pendulum clocks could be used to accurately carry the port time. In order to provide redundancy, two clocks were placed on a shared beam of the ship so if one stopped due to rough seas, the other could provide a reset.
In doing this, he noticed a very strange phenomenon in his experiment:, the pendulum of both clocks would become synchronized after about a half an hour. He referred to this observation as "the sympathy of two clocks" which would take another 300 years to be fully understood. Due to challenges keeping pendulum clocks running on ships during heavy storms, Huygens abandoned his effort to solve the problem with pendulum clocks in 1683. The solution to longitude accuracy wasn't achieved for another 50 years until the Marine chronometer was invented by John Harrison.
Synchronization Spikes
Over the next two centuries, different people revisited the “sympathy of two clocks” problem. They determined that energy was being transferred through the wooden structure connecting the two pendulum clocks. Years later, scientists would learn the exact ratio of mass between pendulum and connecting bar that leads to two pendulums becoming synchronized. We can see this same pattern repeated when we look up to the moon and always see the same side facing us. In that case, the gravity interaction between the two bodies over billions of years has resulted in the same synchronicity that Huygens saw with his pendulum clocks.
This type of indirect synchronization can become a problem when we expect randomness but find unexpected order. When we had built our systems, we had unknowingly assumed the different populations would operate independently with individual spikes in traffic being averaged out across the larger group. When a large event triggered synchronization across these groups, it resulted in a much larger spike in traffic than we normally were prepared to handle.
When planning, it's important to not assume systems are going to operate independently all the time. There are often subtle forces at work that lead to synchronization even when it is not obvious. This synchronization introduces risk and if we're not prepared it can be surprising. Synchronization can also produce great beauty like solar eclipses or friends coming together. If you have the opportunity and the weather cooperates, enjoy the eclipse tomorrow!
