IT Automation – Overcoming Internal Barriers

IT automation offers an increase in productivity and a reduction in risk. Unfortunately, there are often internal barriers to implementation.  

Benefits of IT Automation 

Increased Productivity 

Computers and software systems have automated business workflows and provided massive economic improvements through productivity. Automating routine tasks gives employees more time for non-routine and revenue-generating activities. These improvements have been so significant that without computers, there are not enough people worldwide to perform all the tasks now handled by software. 

Reduced Risk 

Automation also reduces risk. Today, basic functions for most businesses depend on IT systems, which require ongoing maintenance to stay healthy. However, when maintenance depends on tasks people perform, there is always an opportunity for error. Automation reduces the risk of human error and increases consistency, leading to better outcomes. 

Barriers to IT Automation 

Despite the productivity improvements and the reduced human error risks that automation provides, those managing IT systems and especially IT infrastructures have been less receptive to automation in their own world. Why?  

A healthy and stable IT infrastructure is vital, so much so that the need to reduce risk in IT operations has led to the creation of frameworks (such as ITIL) of repeatable processes and structures to ensure proper maintenance.  

IT operations teams have grown and specialized to support these policies and procedures, creating expanded teams and separation between architecture, engineering, and operations departments. While software has been developed to help automate ITIL change management (e.g., Service Now), the work has remained largely manual. 

In addition, the importance of system uptime has resulted in it often being a key performance indicator (KPI) for IT leaders. This KPI drives the IT leader to build a solid team to handle risk and lessens his or her appetite to change those processes that safeguard this critical system.  

Automation is often seen as a force reduction tool rather than an opportunity to redeploy team members to higher-value tasks, making it unpopular with both IT staff and leaders who may see a smaller team as a reduction of influence (and compensation).  

Finally, functions within IT departments are often in siloes leading to local optimization based on functional KPIs, which can be at odds with improvements in overall productivity. 

The Benefits of Automation: A Real-World Example 

To illustrate the benefits of automation, let us look at a BlueAlly case study. While troubleshooting a major issue with a financial services firm, our team found a problem with the configuration of a routing protocol that was causing episodic and devastating outages for the firm. The problematic issue was that the work needed to fix the problem involved configuration changes on over 700 routers. 

The client’s change management board only permitted a small number of weekly changes. With other infrastructure changes already in the queue, the implementation time was projected to be over 9 months! Given the outages, this seemed outrageous, but everyone felt their hands were tied by the change management processes that had been established to reduce risk. 

In addition, the client’s managed services provider (MSP) had stated that even if the change management process were suspended, it would still take weeks to make the changes. Why? The changes were being implemented manually. The MSP was paid by the number of changes and the manpower needed to make them. Their contract was a fixed price to maintain the systems as-is, but each configuration change was a separate billable item. 

BlueAlly demonstrated to the business leadership team that doing nothing also involved risk since they were experiencing episodic but significant outages. 

We recreated the issue in our lab and created scripts to test for the configuration error, change and apply the configuration, test that the change was applied, and back out the change if needed. We were approved to proceed after demonstrating the technology to the client management team. 

Through a discovery script running multiple times over a week to ensure we had a proper census, we found 712 devices needing the change. In a four-hour window, we ran our change scripts. 704 were successful and finished in under 30 minutes. Eight failed and were processed manually within 20 minutes. All were tested and retested.   

Using automation, BlueAlly fixed the problem in one week, which otherwise would have taken nine months to address. 

Overcoming Internal Barriers 

So, what can executive leaders do to promote automation? Change the incentives.  

• For IT Management – Define the bonus and pay structure around outcomes. For many large organizations, bonuses are tied to uptime and the scale of the organization. These organizations live and die by ITIL processes to achieve uptime, but they also incentivize manual work – the more bodies on your team, the better your bonus. Infrastructure as Code directly threatens these types of existing compensation models. 

• For Infrastructure Operations and Change Management – Getting engaged in automation reviews and demanding complete script sets that implement, test, and even back out change requests is key to improving success rates and permitting the organization to adopt DevOps speed while maintaining strict controls. 

• For Engineers – Care must be taken to define their development plans around value rather than volume. A script set that can implement firewall rules and test the changes is a far more fulfilling task than manually entering firewall rules on dozens of firewalls. Once implementation tasks are automated, engineers have more time for their jobs’ interesting and productive parts. 

• For MSPs (Managed Service Providers) – Define your contracts based on outcomes and value – not on low cost, the number of change requests, or the unit labor used to implement them. 

Finally, though individual engineers or projects taking on automation can drive improvement, there will be a broader impact if IT management picks a direction, process, and mandates using those automation tools. Utilizing a common toolkit simplifies training and provides cross-organizational improvements. The most successful automation implementations we have seen are made by companies that pick a direction and go with it.  

When you are ready, BlueAlly Consultants are available to discuss automation and systems to improve productivity and reduce risk to your business. To learn more, contact us about the assessments and professional services we can provide.