The NYSE Outage and the Need for Automation & Change Management
Learning from the NYSE and United Airlines outages: Why IT automation and change management are critical for minimizing technology disruptions.
Between the hours of 11 a.m. and 3 p.m. last Wednesday, July 18th, a rare event unfolded at the New York Stock Exchange (NYSE) – the largest stock exchange in the world shut down, bringing all trading to a stymied halt and leaving traders on the floor helpless until the NYSE reopened just an hour before the closing bell sounded. On the same day, the Wall Street Journal’s website went down due to technical difficulties and United Airlines grounded all flights across the U.S. for nearly two hours, resulting in travel delays, cancellations, and even hand-writing tickets for passengers at multiple airports.
The inconspicuous culprit behind these costly halts: computer glitches.
introduction of electronic readouts and computer screens.
Major IT Outages
As cyber-security threats and hackers take center stage in the media and in the minds of CIOs, people are turning a blind eye to the sleeping dragon of good ol’ fashioned tech glitches and poor infrastructure.
In the NYSE’s case, the NYSE issued a statement Thursday, July 19th saying that on Tuesday evening the NYSE began the rollout of a software release on one trading unit in preparation for an upcoming SIP timestamp requirement.
As customers began connecting after 7 a.m. on Wednesday morning, communication issues between the customer gateways and the trading unit cropped up and it was determined the customer gateways were not loaded with a compatible configuration with the new release. Prior to the market opening, the correct version was loaded, but the update continued to cause communication issues, forcing the NYSE to halt operations at 11:32 a.m.
In United’s case, the details are less clear, with United issuing a statement last week citing a “network router problem” that hampered network connectivity for United’s software applications. According to the FAA, the problem affected software that automates United’s operations.
As businesses increasingly rely on technology to drive innovation and as we as a society depend more and more on technology for our day to day needs, the consequences of ignoring IT infrastructure and the processes that underlie key operations become costlier.
According to Gartner study, “Through 2015, 80% of outages impacting mission-critical services will be caused by people and process issues, and more than 50% of those outages will be caused by change/configuration/release integration and hand-off issues”.
The dynamic needs of today’s businesses are driving organizations to increasingly adopt new technologies and applications that oftentimes just don’t play nice with the existing set of infrastructure and legacy systems in place. This is especially true of longstanding companies like the more than 200 year-old New York Stock Exchange as well as United (who has also been through several mergers, most likely creating additional complexity and mismatches in IT infrastructure). As a result, light and agile IT computer and networking environments turn unwieldy, becoming a brittle, cobbled-together arrangement of old and new that struggles with each new update or change.
We often speak with customers who have dozens of different applications, both legacy and cutting-edge, that need to communicate with each other and work in sync in order to perform day to day business operations. Relying on custom scripting or manual updates exposes organizations to a high amount of risk as a single error in the script or a delay in manual alerting can grind operations to a halt and take hours or even days to troubleshoot.
Modern IT Automation
Modern automation and change management are essential to reliable and effective IT operations. Automation and solid change management allow IT Operations to flexibly add on new technologies with automated processes for building, testing, upgrading, and configuring IT. Additionally, automation solutions with pre-built integrations make it easier for disparate applications to communicate amongst each other, creating an end-to-end approach to workflow creation and maintenance. As a result, IT organizations can minimize the risk of business disruptions and achieve a more fluid technology delivery to the business.
The problem for today’s businesses is that the frequency of these events is on the rise. In the airline industry alone, the glitches are all too familiar these days with United grounding 150 flights last month due to an issue with pilots not having access to their digital flight plans and American Airlines issuing flight delays in April because of a glitch in the airline’s iPad software, resulting in pilots not having access to accurate airport maps.
There is no question that technology will play a pivotal role for businesses and individuals moving forward. The question now is how organizations will build the necessary processes and communication to support our modern technological infrastructure.
Read more about why IT automation solutions need to be designed for change in this White Paper.
