By accident, CalPERS has let us, and perhaps a lot of other people, know it’s not doing a shipshape job of keeping its network secure. Since CalPERS has the names, addresses, and Social Security number of 1.6 million beneficiaries and also executes billions of dollars worth of securities transactions per year, this is not a good look.
CalPERS exposed a mid-level network security vulnerability to us when an employee using CalPERS’ network and equipment left a comment on Naked Capitalism. This security shortcoming in and of itself is unlikely to have exposed any sensitive information. However, it is an indicator of a failure to do adequate firewall testing. That raises the question of what else on CalPERS’ IT security front might be handled in a casual manner.
CalPERS declined to respond to a request for comment.
This post reflects extensive input from four computer professions: our regular bank/payments system expert Clive, two other professionals with considerable experience in running large networks in the banking/securities industry, and an academic who specializes in cybersecurity. One of the three practitioners also conferred with other colleagues to make sure his assessment jibed with current best practice. This was typical of the comments he got:
One response was “It looks like their network security monkey is pretty lazy”. The general perception is that if they are lazy about trivial tasks, how attentive are they to more complicated serious threats?
To be clear: we are not saying that this security shortfall is a big deal taken in isolation. In fact, despite being classified in software standards as a medium-level deficiency, everyone agreed that by itself, it wasn’t consequential. As one IT professional noted:
The cyber security world is all about splitting hairs, so pulling on a thread like this is likely to produce collisions of technical interpretation. Obfuscation of internal IP addresses in external packets have become such a standard practice, that failure to do so has a stigma of laziness in the biz because there are so many competing options for mitigation, but is also not considered to be a great concern per se.
It’s my experience too, that it is these types of “medium” issues that tend to bring out the most ‘how many angels can dance on the head of a pin?’ geek arguments.
The real issue is whether this is just an isolated case of sloppiness or a symptom of slipshod network management. Clive stresses that an institution like CalPERS needs to have internal procedures that are far more rigorous than for most organizatons:
IT security is in the eye of the beholder. What is fine for one kind of enterprise or institution is definitely not okay for another.
We get people from small and even medium sized companies in as new hires and very often they are totally blasé about the need to step up their thinking to match the environment they are now in. Screw up the PayPal interface on an organic cosmetics wholesaler and who cares. Screw up our interbank payments gateway for more than a few hours at my TBTF and a good chunk of the entire UK economy goes tits up.
How Did We Find Out About This Problem?
An accompanying post on CalPERS today describes how a member of CalPERS’ legal department left a comment under an assumed name on a post criticizing the censure of board member JJ Jelincic.
In our backstage, I saw two IP addresses for that comment, which is unusual. One was the IP address associated with the computer’s connection to the internet. The second address was from a range (10.x.x.x) that is reserved for intranets only1. We’ve embedded CalPERS’ response to our related Public Records Act request at the end of this post.
As CalPERS stated, “Christopher Phillips was the user identified with the internal IP address noted in request number three for the time period noted.” Phillips is a senior staff attorney who has worked at CalPERS for four years. As we stated in a companion CalPERS post today, CalPERS confirmed that it was indeed Phillips who left the comment.
What Are the Immediate Implications of Exposing an Internal IP Address?
The fact that Phillips’ internal IP address was exposed externally does not mean his computer or the network files under his account were outside CalPERS’ firewall and therefore readily hackable. However, even a deficiency like this does increase CalPERS’ security risk. And as one professional put it: “This demonstrates basic firewall configuration incompetence.” In addition to the limited information it provides, a deficiency that reflects laziness might encourage hackers to poke around and see what other weaknesses they find.
We’ll rely heavily on Clive’s explanations in this post, since by virtue of having to deal patiently with un-tech savvy and often stubborn senior managers, he gives accessible explanations of fine points.
Publicly available security standards almost universally show exposing internal IP addresses as a medium or Level 2 vulnerability on a typical 3 level scale. For instance, the security standard, issued by the the PCI Security Standards Council shows a worked example of vulnerability testing and analysis on page 26. “Internal IP address disclosure” is cited as a medium vulnerability.
The UK Special Interest Group which created this the data security standard is 20 years old. Internal IP address disclosure has been in it from the start and was, and remains, a severity 2 vulnerability.
Clive explains why medium risks are worth worrying about:
IT security pros usually class security flaws in terms like “critical” or “medium” vulnerabilities. “Critical” vulnerabilities are those which if left unfixed would directly by their presence allow access controls comprise and/or data loss. The large-scale cyber attack on 15/05 was as a result of systems which weren’t patched to prevent a known vulnerability being exploited. Microsoft classed the vulnerability as “critical” – rightly so, because if it wasn’t patched against, your machine was toast.
“Medium” vulnerabilities, by contrast, are those which do not by themselves allow for compromise or data loss but they do potentially help and enable this to happen. Usually it is because they increase what is commonly referred to as the “attack surface” – the size of the target area a malicious actor can aim for. Exposure of an internal IP address falls into this category. Once you know one internal IP address, you can make an educated guess at a network’s entire range of internal addresses.
This is like an art thief wanting to steal a particular masterpiece because they’ve been contracted to heist it. Initially, they don’t even know where it is in the world. But then if they get some inside intelligence that gives them the address, they are hugely further forward in their ability to plan an attack. They may still have to contend with whatever security arrangements have been put into the address where the painting is (alarm systems, guards, locks) but at least now, they can case the joint and find out what these are.
Armed with an exposed IP address, like a thief in possession of a property address, a hacker can now probe all ports looking for known high-vulnerability ones what have been left open. They can test for services which have not been disabled that contain hackable back doors. They can try the same on what they can now guess are other internal IP addresses.
It is also easier to probe which IP addresses belong to workstations and which are servers. Servers are more vulnerable and a potentially richer target to aim for, so if you can get to know the internal IP address of a server, you can concentrate your efforts where they may yield the biggest impacts.
Another IT professional added:
To follow the analogy of an art thief, a more sophisticated thief would want to map out the identities and locations of objects so that a specific plan could be formed to minimize the potential for alert or response while grabbing the maximum value in the briefest amount of time.
What an internal address exposure does is support the ability to map out a target network. If the mapping is thorough enough, nodes, types, and functions can be determined so that when they get in, they can quickly go straight to what they want to take out. They can also use this mapping knowledge to use some weaker nodes as staging points to load exploitation tools and to leverage any trusts that those nodes may have on the network.
And what about a worst-case scenario, that Phillip’s machine was exposed? Generally speaking, the files on that machine aren’t the target. A hacker is interested in any permissions which that machine has, especially if he can access the permission of the logged-on user. Even seemingly pedestrian permissions like being able to parse the print queue can expose all sorts of random but useful nuggets.
But that’s before you get to the fact that Phillips, as a member of the legal department, himself would be on the list of high-priority targets. Clive again:
If someone compromised my PC, then even if they got no further than the files I store locally, they’d be able to tell nothing much more exciting than that my TBTF is completely useless and wastes vast amounts of money in project delivery or that it gets taken for a ride by all and sundry. While potentially embarrassing, it would merely confirm a commonly-held (public domain) belief.
Conversely, if the CalPERS machine was used by someone on the legal team, then by the nature of their role within CalPERS they would reasonably be expected to have for more sensitive and compromising (compromising to CalPERS if it got out, that it) files – files which might be stored right there on their machine. Meaning that a hacker would need to get no further than accessing the machine, not anything else in CalPERS’ IT environment.
Unless the user was incredibly diligent and never held any high value/high confidentiality material on their local machine, they’d be a prime target for a hacker simply by virtue of who they are and what they do. At my TBTF, a lot of intrusion attempts are *specifically targeted* at C-level execs, audit, IT security operations and so on. High-value targets. The reason is obvious – don’t waste your time on mid level apparatchiks like me, go for the big fish who will have more juicy prizes at stake.
What Does This Security Failure Say About CalPERS IT Vigilance?
Turing the mike over to Clive again:
If I had to guess, I think that someone inside CalPERS intended to deliberately create what is known as an Exposed Host. There are a variety of legitimate and even necessary reasons why you might want to do this – running a web server, an email service, an extranet or similar. Videoconferencing is another possible use.
But that is okay for a small business who might run a “disposable” network which they don’t care about and don’t put anything on that anyone would be interested in. It is difficult to envisage why a institution of the scale and sophistication of CalPERS would find itself in this position. Especially with the risk that confidential data (from scheme beneficiaries, for example) could inadvertently end up on a network which it should not ever be put on (you, or rather they, are then depending on operational controls and these are the weakest because they are most failure prone and don’t fail-safe).
An excerpt from a typical firewall’s documentation has this to say on the subject of Exposed Hosts:
When a computer on your LAN is designated as the exposed host, it loses much of the protection of the firewall and is exposed to many exploits from the Internet. If compromised, the computer can be used to attack your network.
Whenever I implement a change at my TBTF which involves an externally facing system (and this interpretation is extremely broad; firewall changes and setting up publicly-facing services would definitely be included) are subject to a sign-off by a “White Hat” security testing specialist – hackers who will try to hack your infrastructure to test for weaknesses. One of the tests is for exposure of any internal IP address. Any internal IP address exposure is an automatic fail and must be fixed before the change can be promoted to a Live environment.
If CalPERS are not undertaking this sort of diligence and QA in managing their systems – and if we did indeed see one of their internal IP addresses it proves they are not – this is a significant governance and management capability shortcoming.
The mere fact that you can see the innermost configuration of CalPERS internal network shows that something is potentially amiss with their security. If I lived in CA and my data was being processed by CalPERS, I’d contact the appropriate state regulator and ask them to make enquiries.
If this had happened at my TBTF, it would be regarded as a serious security incident. There’d have to be an internal review and a post-mortem and a set of Lessens Learned drawn up.
As another IT professional put it:
The hiding of the internal IP addresses, as a matter practice, have become such a commonplace standard (because hiding them is a very trivial process), that security experts view it as sloppy hygiene.
Richard Smith explained why network security is often not given the attention it warrants:
Managing one’s managers is especially tricky in the IT world and it can get very dysfunctional: striking the balance between escalating the stuff that needs to be escalated and not escalating non-urgent problems just gets harder and harder, especially when non-urgent stuff suddenly gets urgent, for instance if the CEO gets a bee in his bonnet about some trendy IT boondoggle he vaguely remembers from some recent conference.
This recent NHS cyberattack is a classic instance of a better-justified sudden change of priorities, and in fact IT security is pretty much always the orphan child that suddenly claims attention. IT pros who are nonchalant about security are cruising for a bruising. But as for when and how that bruising will be delivered, it may be impossible to be specific enough to make a business case if the company culture doesn’t work that way. The perverse end result is that elementary IT security precautions get sidelined as speculative investment. This is precisely why IT security’s orphan child status will continue until it can be packaged up as a service offering, and overall we are a massive, massive distance from that Nirvana.
Maybe this little incident will get CalPERS to revisit its priorities.
1 Technically, it’s called a bogon. For instance, on home computer networks, they are usually of the format 192.168.x.x