Shadow IT and shadow IoT

Can an IoT coffee maker leak company secrets? Where do you put the ‘S’ in 'IoT’? Join Alison Naylor, Senior Manager for Information Security at Red Hat, in this episode of Security Detail as she discusses the importance of securing IoT devices and how to proceed with caution.

The calls were coming from inside the house. When one of our associates reported their laptop was being attacked by what appeared to be another Red Hatter’s IP address, we knew we had an incident on our hands. More than casual SSH scanning and brute-forcing attempts, this persistent attack was eventually traced back to a storage appliance with outdated firmware at an associate’s home.

The appliance had been directly connected to the Internet, which allowed attackers to take advantage of a well-known security flaw. The result was an appliance riddled with malware and coin mining software, some of which searched for more vulnerable devices to compromise.

This associate was working from home during the pandemic, and was using the storage appliance alongside their work-issued equipment. We discovered that careless configurations had inadvertently created a path from the public Internet, through their home network, to a corporate network, ultimately leading to another associate’s device becoming compromised.

Many of us have Internet-connected devices in our homes, like our laptops, tablets and phones. Some of us might even have home labs, with a few servers and connected storage appliances. Increasingly, we’re also likely to have devices that are part of the Internet of Things (IoT).

What is the Internet of Things?

The Internet of Things is made up of devices like smart TVs, security cameras, thermostats, smart bulbs, gaming consoles, DVRs and more. Maybe you have home appliances that are connected: refrigerators, washers and dryers, even kitchen scales, blenders, sous vide machines and outdoor grills.

Hopefully, when we set up these devices, we’re at minimum changing the factory-default credentials and setting strong passphrases. But once our devices are up and running, many of us neglect to stay on top of software updates. We’re busy people, and it’s so easy to click "Remind me later"—if we bother to check at all.

We might not think of these connected devices as targets for cyber attacks, but they can provide points of entry that allow attackers to get into other systems, putting your data and privacy at risk.

Examples of IoT at the office

What about the Internet of Things at the office? Smart TVs may be used as digital signs and there can be connected security cameras, DVRs to store video footage, smart locks and badge-reading systems, environmental controls and fire suppression systems.

Consumer-grade IoT devices can also appear in our offices and on our networks. Pat’s streaming stick was a hit last spring when they brought it in to show the big game on the giant display in the conference room. Steve’s smart kettle and Brenda’s treasured toaster could be in a break room near you right now!

"The Network Is Down — But I’m Feeling Better," said the faded early-aughts cartoon pinned to the cubicle wall. But today there is a network outage at HQ, and a few hundred IT professionals are feeling pretty annoyed.

Network and security engineers scramble to find the root cause, and track the source of the problem to a crowded meeting room filled with interns. Inside, they discover some rogue consumer-grade network gear nestled in a tangle of bright yellow ethernet cables. “We needed more network ports, so we brought in our own gear!” beamed a proud young person, completely unaware of the trouble they’d caused.

This hardworking group of future engineers had only wanted to solve their problem without bothering anyone–but their collection of misconfigured networking devices ended up making a bit more work than simply filing a request would have.

What is shadow IT?

Shadow IT refers to information technology systems deployed by individuals or departments outside of an organization’s central IT department. These unofficial systems are often set up due to perceived or actual shortcomings of centrally-managed information systems.

When we investigate cases involving shadow IT, we don’t typically find shady characters. Instead, we find smart, dedicated people with good intentions who are trying to solve problems and be productive in their work. Sometimes, workers are simply unaware of existing offerings and options that could meet their needs.

Shadow IT examples

Examples of shadow IT can include anything from the proverbial “server under the desk” to cloud provider accounts to productivity tools like chat, file-sharing and note-taking solutions. Workers turn to Shadow IT when time pressures, frustrations and the desire for flexibility collide. The idea of waiting on the IT department to deploy a new application (let alone going through the contract negotiation process) seems completely untenable when deadlines loom.

Let’s consider an example of tech workers who need to collaborate regularly in near-real-time.

Many of them are highly technical, and some are road warriors who frequently travel in order to work with their clients. They want a chat program that works on their phones as well as their computers, doesn’t require a VPN connection, supports long-lived sessions and meets their technical expectations around features. Must-have features include APIs that allow them to develop chatbots that integrate with other tools, their preferred threading experience in group chats and plenty of ways to express themselves beyond just text.

If these tech workers are dissatisfied with their company’s existing chat program, they know they have plenty of options; they could stand up their own chat server, use personal messaging apps or social media accounts, or they could sign up for a free-tier Software as a Service (SaaS) offering online.

Shadow IT consequences

What might the consequences of pursuing shadow IT include? For our tech workers in the example above, their chat solution may not use end-to-end encryption, passing their messages in clear text. It may not have a secure authentication mechanism, and very likely will not be integrated into the company’s SSO.

Who will manage the accounts, adding new employees when they join the company, and more importantly, removing them when they leave? Old accounts left behind with no oversight may still have full access to chat history, and could be compromised and used for corporate espionage or other nefarious purposes.

What if business processes grow to become dependent on the service, such as ChatOps integrations with CI/CD pipelines, and what would be the impact if the chat solution is later abandoned?

What if only part of the organization uses the alternative chat solution? This could lead to isolated islands of different groups of workers, or worse, overwhelmed workers that must follow conversations across multiple platforms.

What would be exposed if the chat solution were compromised and all discussions made public? If the chat solution has insufficient logging capabilities or the logs of a social media app or SaaS offering simply aren’t available to end users, how could a potential compromise be investigated?

If the team paid for a service, did they read the terms and conditions closely enough to understand the contract they’ve signed on behalf of their employer? Might they have agreed to their data being sold, or agreed that the provider isn’t responsible for damages in case of a data breach?

If our tech workers’ discussions about their customers, their customers’ data, and their contract details become public, it could be disastrous for their business. A serious leak of customer data could result in brand-damaging public disclosures and fines associated with the GDPR and other privacy regulations.

Dealing with any of these consequences might prove to be much more of a hassle than simply using the company-provided solution!

Preventing shadow IT

In an ideal world we would be able to prevent shadow IT before it starts by providing IT services that meet our employees’ needs in every way, along with the education that empowers them to make full use of those services. However, as the universe of options continues to expand, and our users’ needs become more specialized, avoiding shadow IT entirely can prove impractical.

As IT professionals, we can make choices that can help mitigate the risk of shadow IT. First, we can try to detect the use of unauthorized systems. This task is less daunting if we begin with an accurate and up-to-date inventory of all of our hardware and software assets.

A complementary approach might be to simply have conversations with our users in order to ask them what needs aren’t being met by IT-offered services, and how they work around those limitations. These conversations could present opportunities to better educate our users about what our current IT offerings can provide in terms of features and capabilities. We may also discover a surprising number of shadow IT systems that are already in use.

Dealing with existing shadow IT

When we do learn about existing shadow IT systems, our first reaction may be a strong impulse to shut it all down. Instead, consider that our users who put in the effort to set up shadow IT systems are trying to tell us something, and we would do well to pause and listen to them.

When we approach shadow IT with curiosity, we can learn about our employees’ pain points and the limitations of our systems, and then make a plan to address them responsibly. If we find our users aren’t taking advantage of existing offerings, that presents us with the opportunity to improve our education efforts. We can partner with our users to propose a plan for migration to official systems.

In some cases, our users’ shadow IT systems address a need that our existing systems cannot, and our plan might involve working with teams to bring their system into compliance with corporate IT and security standards. This type of solution can help mitigate the risk of compromise and data leaks, and set reasonable expectations around support for users of the system.

In cases where a shadow IT system is particularly beloved or entrenched, the best approach may be to help the teams running those systems navigate the process of “going legit.” This can include completing security and data privacy audits, negotiating an enterprise agreement with a service provider and configuring centralized authentication and logging. When we can bring a shadow IT system into the light as an official IT system, our users feel heard, perceive their IT team as responsive to their needs and the end result can be acceptable to all.

Shutting down shadow IT

In cases where a Shadow IT system cannot be brought into compliance, we may need to pursue a shutdown. This can prove frustrating and difficult to accept for some of our users, but clear and open communication helps. Sharing the reasons why a shutdown is unavoidable, setting expectations around sunset timelines and outlining our plans to address unmet needs with an alternative solution can preserve or rebuild goodwill towards the IT department.

When we don’t take steps towards a managed risk approach, the path to legitimacy, or a controlled shutdown, we choose inaction. Ignoring shadow IT not only encourages its proliferation and fails to address its risks, it sends a message about our IT organization. Whether our users interpret our silence as incompetence or indifference, it reflects poorly on us. Conversely, taking positive actions—paired with transparency and two-way communication—lead to the view that IT is a trusted partner, not a barrier.

Whatever it is, it’s acting like part of a botnet. The traffic I was seeing didn’t make sense. Many connection attempts at regular intervals, odd-looking packets, a sketchy-looking hostname and web requests that just didn’t look like normal human browsing activity.

The user-agent strings indicated a system so old, they just had to be faked. It all seemed to trace back to a network port just two floors above me. I ran up the stairs, found the location of the network drop, and to my surprise, found an employee browsing an internal page on a very new-looking laptop. After a short conversation, it became clear that what I was seeing didn’t match up with this employee’s laptop or activity.

Then I noticed the PC in the corner.

How had I missed this beige behemoth? After a few more questions, I had my answer. This brand-new employee had seen an e-waste bin that morning and thought it would be a shame to send this venerable old tower out to pasture permanently. They’d fished it out, taken it to their desk, plugged it in and pushed the power button. They’d planned on wiping it right away, but it was time for their new hire orientation, so, yes, they’d left it there for a while. They’d only just returned, and had been busy getting the hang of their new laptop before trying to resurrect the old PC.

It turned out that another new employee had recently hauled in their parents’ ancient computer and dumped it in the bin. It had been languishing in their garage for years awaiting proper disposal, after having been relegated there because it had "started running slow". They’d decided the e-waste bin at their new job was a more responsible choice than the landfill, and had been glad to finally be rid of it!

Our team discovered that the old PC’s malware was trying to phone home to rejoin an old botnet that had long since been taken down, and it didn’t do any further damage. However, botnets continue to pose a problem across the Internet.

What is a botnet?

A botnet is a collection of computers that have been infected with malware that allows them to be controlled as a group, without the knowledge or permission of the computer owners. Botnet controllers use covert channels to issue commands to their hordes of machines, directing them to send spam, engage in click fraud, steal personal information, run password-cracking tools, or to conduct Distributed Denial of Service (DDoS) attacks.

Botnets may be centralized, where newly-compromised devices reach out to establish a communications channel with a central command and control (C&C) server. The C&C can then use this command channel to send instructions to its bots. Because a centralized C&C server is often a single point of failure for a botnet (that is, taking it down can neutralize the effectiveness of the botnet), other botnet organization schemes have emerged.

Tiered C&Cs may have many layers of different C&C servers. Decentralized or peer-to-peer botnet members act as both clients that receive commands as well as C&C servers, issuing commands to other members of the botnet. Some botnets even try to hide in plain sight, making use of public services like Pastebin for communications.

What is an IoT botnet?

An IoT botnet is simply a botnet made up of compromised IoT devices, and is used in similar ways as traditional botnets. However, many IoT botnets also try to find additional devices to compromise, scanning for open ports and default credentials, spreading their malware to more and more devices. Because many thousands of similarly-configured (and similarly-vulnerable) IoT devices may be deployed at once, modern IoT botnets can grow larger in scale than traditional botnets that relied on infecting vulnerable PCs and servers.

IoT devices are attractive to attackers because they are typically always on and connected (think DVRs and security cameras). The potential scale of IoT botnets makes them particularly well-suited to launching massive DDoS attacks. Some, such as Mirai, have even surpassed 1Tbps of generated traffic.

Many IoT devices run variants of Linux and Unix-based operating systems, and can be targeted with malicious executable and linkable format (ELF) binaries, a file format commonly found in the firmware of embedded systems. When these devices are directly connected to the public Internet, malware may be easily delivered directly via SSH or telnet network protocols, simply by taking advantage of hardcoded default credentials (admin/admin).

Other methods may include exploiting security vulnerabilities present in factory-shipped, unpatched versions of firmware. Once access to an IoT device is established, the malware payload is delivered to complete enrollment of the device into the botnet. Attackers might then change default credentials, preventing access from the device’s owners (or other attackers) in order to maintain control.

Even when actively participating in a DDoS attack, many compromised IoT devices don’t exhibit much degradation in their performance and simply continue to work as intended. Thus, their owners may never know of the compromise, let alone find motivation to address it. Because many IoT devices simply aren’t maintained regularly or monitored closely, attackers are able to make use of them for weeks, months, even years without being detected.

Avoiding IoT malware

When shadow IT and the IoT combine, the results can wreak havoc for IT and security teams, well-meaning workers and even the bottom line. But the good news is that staying protected against IoT malware is relatively simple:

• Always change default credentials on any embedded device, replacing them with a strong passphrase.

• Users should regularly check with device manufacturers for firmware updates that can help protect against potential threats.

• If the best product to meet your needs is a smart device, you can still limit your exposure. Some devices like televisions can have their “smart” features disabled. You can take measures to isolate these devices on your network, using separate VLANs and firewall rules. Or, you can simply choose to not connect the device to the network at all if the features aren’t needed.

Learn more

• Understanding IT security

• Security for IoT devices

• Taking Control of your IoT APIs

What is Security Detail?

Security Detail brings you an insider look into the world of IT security. Join Red Hat security experts to better understand cybersecurity threats and learn strategies to improve security processes and protect your enterprise from data breaches and cyberattacks.