Some weeks ago I accidentally changed data on senate.gov. At the time I didn't realize it. Only later, as I looked into some strange firewall rules from a government vendor did I realize what happened.

The breach that allowed me to do this involved an open production Elasticsearch database. It belonged to a company called Granicus. This breach had previously been reported on here some weeks ago. Because this database stores and generates the metadata that is used to populate thousands of government video streaming sites, changing data on the database allowed me to change data on www.floor.senate.gov.

At the time I wrote a small collection of posts about what I had found. I wish I had put more effort and thought into them, but I think they are decently well structured and the first can be found here. I want to now look into the specific firewall rules that I mentioned above.  They are of concern to people, and I find myself being able to sleep less and less the more and more I think of them.

Granicus is a Big Government Software Vendor:

Granicus is one of the largest provider of SaaS software to the US and UK governments. It has grown over the years extensively through acquisitions that have put it near the top of a small government software empire. From Drupal, to the Emergency Alert System, to various niche SaaS buys, a lot of government runs on Granicus.

Logos of customers promoted by Granicus on their website.
Granicus has a much larger base of customers and potential to analyze data than people realize.

Granicus recommends some odd firewall rules:

Granicus asks all agencies to whitelist the following IP addresses and bypass authentication. They ask agencies to install a special video streaming server on-premise with their other servers. The streaming server is very similar to what Elemental Technologies offered before the AWS acquisition. There are some nuances to what I describe as far as the network but generally that is the gist of it and what I feel is a fair representation. Roughly 1,700 agencies in the local/state/federal government are asked to create pass through exceptions for the following network IPs:

207.7.154.0/24
209.237.241.0/24
34.208.128.232

What Is On The Servers They Ask To Bypass Firewalls:

If you look at the server 209.237.241.3, which I picked randomly from Shodan simply because it was located within one of the above subnets, you can extract the fingerprint of the SSH Public Key. This specific kind of fingerprint is relatively new and known as the "hassh". It is as follows:

  • 8e:c3:91:a4:f6:c0:b3:66:01:e9:85:a9:da:a6:24:f9

The Shodan scan above can be verified by me manually extracting identical values via nmap. 

Now let's run the hassh fingerprint through Shodan. Since Shodan actively scans all open computers on the internet, it will tell us whether this fingerprint has appeared anywhere else.

A Strange Fleet of Ninety Unaccounted Servers is Revealed:

Ninety servers come back as of March 20th, 2020. They are scattered all across the world, and the majority are in China, across nearly all provinces. Later, I will show that some of the networks in China seem to be more malicious than an average Chinese ASN.

Granicus only does business with governments in the United States, and very recently the UK. These servers really should not exist, and no one can account for them. These servers have a direct link, that can bypass most - and sometimes all - of the layers put up to protect a majority of United States government agencies and departments. Granicus has a broad portfolio across local, state, and federal organizations, so the reach is quite significant.

The result of the Shodan scan that reveals the fleet of unaccounted for servers 

These servers should be investigated, and Granicus should likely seek the help of the FBI within the coming days as a potential victim.

Verifying the Shodan Data via Censys:

Proving the data even more, we can, using similar fingerprinting techniques, produce a similar set of servers via Censys.io, as seen below. The SSH public key being used here is:

AAAAB3NzaC1yc2EAAAABIwAAAQEA1X6uhe/7ZkJvN7P0w1UtgiqE81iyfkO5yt0PejN/kODBWo0x+DzQrnV9CfzJatG7droJ0gilvAxE4A2c0h0qhsiEG+58qEPNKBjZmgREm48S14C/Uwju6tLX2BVWVsbOA/Ud8qwnLEzyk/EuBzvI8H9pv9RbpsJPeyp0q+30S/82ItG0Eol/VrSDMvyd3ZEZSof+nK387AlVMNkm5DyTqUE578fZ1q/riRJCaQ8UBrh1L2Y1H8Un3om+esh7vZ0MO9MKRsBO7rX8iPBN2EbQHdKq2CwffPadC9KZUyr0nfyT1uJTmxkDmPPB58wLS2tJW+83B2lTPvRyNE3x8EBmRQ==
A map of all the servers returned by Censys


Additional metadata of the scan that returned our unaccounted for network of servers



Verifying the Data One Last Time with BinaryEdge:

For a third time we can verify and see the seriousness of the data. If we run the original fingerprint through BinaryEdge, we actually get 103 total distinct servers scattered across the world. The higher number here is due in part to the fact that BinaryEdge scans the IPv6 address space:



The Chinese networks are the most concerning, because that is where the most foreign servers that came up were located. A breakdown of the various networks is below:

The four Autonomous System Networks (ASNs, think of them as giant ISPs) that the mainland Chinese servers were found in

This is quite concerning because if we look up some of those networks such as ASN 4134 or ASN 24138, we find they are labeled as malicious via Greynoise:

Greynoise scan of ASN 4134
Greynoise scan of ASN 24138

This issue would likely take several hours to solve. It would save perhaps 3-4 weeks, or thousands of hours of recovery time, in actual cyber conflict. It would be easy to make the case for longer. Right now though, American government networks look very much like this cartoon in the abstract.  

Very very simplified network diagram of the unaccounted for server fleet (far left), how it connects into Granicus Cloud (center), and then finally the US government (far right).


What Is The Purpose of this Server "Ghost Fleet":

I am not sure. Why do 90 or so random servers, sitting out on the public internet, share the same access key token that only servers allowed to officially access and provide support to US government agencies should have? Who created these servers? What is the plan? Why do they keep funding this infrastructure?