When I find or read about software vulnerabilities I often chalk the root cause of the flaw up to human error or ignorance. Occasionally I see something that makes me scratch my head and really wish I knew what stream of logic and events caused something to occur. The topic of this post is one of those.
The TLDR version of the story can be found on the Full Disclosure list.
Cisco Unified Computing System Manager (UCSM) is a Cisco product that manages all aspects of the Unified Computing System (UCS) environment including Fabric Interconnects, B-Series blades servers and the related blade chassis. C-Series (non-blade) servers can also be managed. These solutions are deployed in high performance / high density compute solutions and allow for policy based and rapid deployment of resources. They are are typically found in Data Center class environments with 10 GB network and 8 GB Fibre Channel connectivity. A single UCS environment could contain 20 chassis with each containing 8 servers which, in the case of the B230 M2, can contain 40 cores and 512 GB each. The resulting 6,400 cores and 81,920 GB of memory represent a significant investment and enough horsepower to run a data center for a medium sized enterprise or service provider.
Unfortunately, if you tried to help secure the environment by sending the UCSM logs to a Syslog server you may actually be adding an unexpected risk vector. When configured to use Syslog UCSM versions 1.3 through 2.2 will transmit the usernames and password hashes for all local users every 12 hours. If the environment has two Fabric Interconnects in a cluster then both nodes of the cluster will transmit the data to the Syslog server.
This behavior was experienced with the following Syslog configuration:
- Level: Information
- Facility: Local7
- Faults: Enabled
- Audits: Enabled
- Events: Disabled
Here is an example of what this looks like in the Syslog logs. Portions of the password hash have been replaced with <!snip!>.
Oct 28 23:31:37 xxx.Xxx.xxx.242 : 2014 Oct 28 23:49:15 CDT: %USER-6-SYSTEM_MSG: checking user:User1,$1$e<!snip!>E.,-1.000000,16372.000000 - securityd Oct 28 23:31:37 xxx.Xxx.xxx.242 : 2014 Oct 28 23:49:15 CDT: %USER-6-SYSTEM_MSG: checking user:admin,$1$J<!snip!>71,-1.000000,16372.000000 - securityd Oct 28 23:31:37 xxx.Xxx.xxx.242 : 2014 Oct 28 23:49:15 CDT: %USER-6-SYSTEM_MSG: checking user:samdme,!,-1.000000,16372.000000 - securityd
I have verified that the hashes were valid by cracking them using John the Ripper. They are in a common Linux/Unix format which uses salted MD5.
The flaw is present in multiple versions of code published over at least 4 years. I have verified that it existed in version 1.3(1c) released June 2010 until at least 2.2(1b) released in December 2013. I reported the issue to Cisco at the end of October 2014 ( Bug CSCur54705 ) and the first reported fix was in version 2.2(3e) which was released to the public in early March 2015.
My experience with reporting the issue to Cisco was a bit frustrating. I think this was primarily due to the Cisco PSIRT team handing the vulnerability off to the internal development team [2].
PSIRT Evaluation:
The Cisco PSIRT has evaluated this issue and does not meet the criteria for PSIRT ownership or involvement. This issue will be addressed via normal resolution channels.
This resulted in both the PSIRT and Cisco TAC being unable to provide me with any status updates during the process. About a month into the process Cisco posted an advisory on the Cisco Security Advisories, Responses, and Alerts site [4] but this seems to have disappeared in late January or February. At this time limited information can be found on the public page [3] and more detailed data is available here [2] if you have a Cisco login. The release notes [1] for the fixed version, 2.2(3e) contain this entry:
Cisco UCS Manager Release 1.3 through Release 2.2 no longer sends UCS Manager username and password hashes to the configured SYSLOG server every 12 hours.
As far as I know no CVE was assigned.
From my perspective, the flaw creates the following concerns:
- Individuals who have access to the Syslog logs may not be authorized to have access to the UCSM environment and this information represents an exposure.
- Authorized users with the ‘Operations’ roles can configure Syslog settings, capture hashes, crack them, and elevate access to Administrator within the UCSM.
- Syslog is transmitted in plain text.
To address these concerns I made the a few recommendations to Cisco in my original submission in October 2014:
- Remove the username and password hash data from the Syslog output.
- Allow the configuration of the Syslog destination port to enable easier segmentation of Syslog data on the log aggregation system.
- Add support for TLS wrapped Syslog output.
Removing the username and password hashes from the Syslog data is obviously the best way to handle this and that is what Cisco has done. I had hoped for the other two recommendations to be implemented as it would allow for better segmentation of the data on log servers such as Splunk and better protection of the data in flight. Perhaps these will be implemented in a future version of the code.
As I mentioned at the beginning of this post I am puzzled by what sequence of events resulted in this sensitive data being being written to the logs and then forwarded via Syslog. Also puzzling and disappointing is how long the flaw was present without being corrected. Somehow in this time period either nobody noticed or nobody brought it to the attention of the correct team at Cisco. Optimism says that most who are using Syslog are reviewing their logs but they are using a lower level of detail. Life experience says that few have logging enabled, even fewer are using Syslog, and those that are using it are not reviewing, or don’t have time to review, the logs.
So on that cheery note, upgrade ’em if you got ’em. If you haven’t done it in a year then you are addressing OpenSSL and Bash bugs in the process.
– Tom
References:
1. http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/release/notes/ucs_2_2_rn.html#21634
2. https://tools.cisco.com/bugsearch/bug/CSCur54705 ( Cisco login required )
3. https://tools.cisco.com/quickview/bug/CSCur54705 ( NO login required, limited data )
4. http://tools.cisco.com/security/center/publicationListing.x
Faded Lab 