NOTE: This post is highly technical for most of its length and then turns to highly legal for the concluding section on GDPR. The post after this, Essay 4, maybe 5, will go into depth on the questions we have been asked much more frequently and with greater enthusiasm: could you tap into video feeds? could you do it on a large scale? and if some group did do it could such actors be identified and where they went traced? This post took a while because we kept trying to extend it in length from the original as well as release 4 and 5 at the same time, which we will do shortly. We apologize for any grammatical errors, as they are annoying. They will be fixed later.
A few conclusions (but not all we are going to make) about Wyze infrastructure:
- Since March 2019, the main Wyze production MySQL data warehouse in the United States has been left open for anyone to access.
- Over a 7 day period in which server logs were analyzed, 2,000,000 API calls were made from outside of North America. Why these are happening is not clear. The simple answer would be that once bought in the United States Wyze's cameras can run anywhere, however if that is true, the amount of European data exposed would make this an extraordinarily significant GDPR breach.
- The Wyze iOS app quickly begins sending user data to China within 60 seconds after a user registers their email and password. The first item to be sent out is, if a user has uploaded one, their profile picture to an S3 bucket in AWS China owned by Hualai. This adds significant latency and data transfer cost to the Wyze bill and must be seen as a simple but intentional decision.
In regards to Point #1, the below edited image shows the part of the source code where credentials were hard coded in. We can see they are using the Druid library from Alibaba, and there are two databases, one in the United States and one in China, and that they are linked together in this application.
These credentials were committed to the source code 11 months ago in early February 2019. The server https://gitlab.wyzelabs.com.cn, where this code was hosted, began to be picked up by internet scanners in March of 2019, which is when the SSL cert also shows first registration, and from that time on access for anyone became possible. Gitlab, when self-hosted, can be secured with a username and password and even 2FA, however this server allowed users to self-register, or to simply go to the bottom of the page and click "Explore" to avoid even self-registration. It appears the latest version of Gitlab fixes this known flaw.
I again want to stress that this server, once it was online, would have been picked up immediately by any one of several dozen capable groups/actors/individuals.This server was listed with an IPv4 address, so this point has no counterpoint and is a statement of complete universal fact. Software such as ZMap, which is free and open source, can scan the whole internet in 5 minutes, and that is if not operating in parallel. Zmap, combined with software that specifically looks for published passwords in documents on the internet such as leakScraper, would be trivial to set up.
At this point Wyze would have been fully compromised assuming a moderately skilled hacker. Malware could have been left on the database server that would have compromised the next person to log in, likely a sysadmin with broad access, and then the next person, and then the next person. It is truly shocking how quickly an organization can be taken over like this, and the best analogy would be soldiers in enfilade. There is a tragic photo from the Deippe Raid of 1942 with Canadian troops massacred along the beach by a single German gunner. We do not think the time is far off when incidents such as this can cause damage at equivalent levels.
That being said, there was plenty of information within the database to compromise everything else even if no one else ever logged in. Certificates, authorization data, keys, api tokens, etc. It is to what this data exactly is we will turn to next.
This screenshot was posted last time, but here I'd like to call out specific tables. I'm not going to cover everything in red, but from top to bottom:
- GE_Device_Kinesis ... This refers to AWS Kinesis, a collection of services for moving in large amounts of streaming data quickly into AWS
- GE_Device_Info_Authentication ... contains the certificate chain of x509 certs to authenticate with AWS IOT Core
- GE_Device_Reco_Person/Person_Face/Person_Times ... unknown facial recognition data. Likely the coordinate points that make up a facial pattern that can be matched against a person. Again unsure.
- GE_Ift_* ... extensive IFT information in the form of credentials, tokens, user set up, user configuration, etc
Encryption is a term thrown at customers today almost entirely for purposes of marketing instead of mathematics. A couple of points to begin our discussion:
- Most services, apps, software, or companies don't really have encryption in the way we think about it.
- For instance a database that is encrypted is meaningless. Databases are only encrypted when the hard drive isn't mounted and being read by the server...which is pretty much never today. If the application is using something like Oracle's Transparent Data Encryption (TDE), and it has been properly configured, than the encryption we think about is much more closer to reality. We do want to note anonymization, data masking, and limiting access rights / scope are extremely important though.
- For cloud architectures today, the most important thing is to be using SSL/TLS with a strong cipher suite to encrypt the data as it moves. Moving data is dangerous data. When the data is at rest on a server, hardening and defense-in-depth tend to play much more of a role than encryption.
The iOS app testing was done on shows that Wyze does encrypt data leaving from the phone. However, data streaming from the camera, the actual video streams, has significant issues. This was noted a few months ago by Dark Cubed in their report here (the Wyze section begins around Page 140). This is related more to authentication than encryption, although much of the math is similar and relies on at times identical algorithms. The issues they note are dire, and We direct your attention the very last paragraph of this screenshot:
I'm going to lump a couple of technical things together here for sake of brevity, but what they are saying is 1). it is possible to fool the Wyze camera into thinking you are AWS and intercepting all video feeds and 2). the API requests made from the camera to AWS actually somehow stunningly contain credentials for AWS, the permissions of which are unknown (assuming some developer left them their by mistake they would be quite powerful then) and these credentials are still available to anyone willing to sit down and intercept Wyze camera API traffic (although by now Wyze has certainly rotated their internal certification tokens).
As for point #1, this danger is noted itself in the AWS documentation. One could get around that perhaps via Dark Cubed's hypothetical route, but if you remember the GE_Device_Info_Authentication table from above, which had the X509 certificate chain, this is not needed, and you can now impersonate AWS IOT for Wyze cameras. What this would look like on a large scale for global surveillance will be in a following essay.
iOS PHONE APP
As discussed in the introduction, the Wyze app begins sending data to China incredibly early in the process. This was so shocking to see and so sloppy I'm not sure what to make of it. Here is the exact log event from Charles Proxy on iOS. We don't think the United States is any less safer because of what S3 bucket Wyze does or doesn't send the user profile thumbnail to, but this is still incredibly concerning for an AWS architect. The app would have been much faster sending this domestically, why connect abroad? Surely a video streaming company that makes cameras is concerned and familiar with latency?
At the beginning of this post we showed a heat map of countries around the world making API connections to Wyze. To give you a sense of scale, the top country is India with around 180,000 connections, with Great Britain somewhere in the middle at 80,000 or so. In table format the logs, stripped of everything but the connecting IP address, look like below. These are real logs and in case you are curious they belong to Alibaba Cloud:
The next post will go into great detail on exactly how one would have hypothetically, pre December 2019, tapped into Wyze camera feeds, what that would have looked like, and where the videos would have gone. Additionally, because IOT makers now are so similar, a point alluded to in my first post, we will see that about 2/3rds of these techniques still work for cameras being happily sold right now on Amazon and tomorrow morning at Best Buy.
For now we want to address the issue of the logs though. Across just a few days some 350,000 API connections are made from Europe. How is this possible? Do Wyze devices work in Europe? A large portion of Google's hardware portfolio for instance won't even turn on past the initial setup mode if one is in Portugal.
I want to be extremely clear in that not only do Wyze devices work in Europe, European users with emails provided by European ISPs have signed up en-masse, the app can be installed even when one has a SIM card that places the phone in Europe, and for all intents and purposes Wyze is a company open for business and data storage in Europe. These are not just American users traveling abroad.
This brings up extraordinarily concerning and complicated legal questions that no lawyer really will understand, and that even sysadmins will struggle through most of. The law is not just the law but a collection of writings and legal theory on what it means, and that body of work does not exist to date. That being said, for Wyze, a company employing Chinese developers, who live in China, who host a significant amount of their infrastructure in China despite doing no business there on paper...requires some questions be asked by governments, intelligence agencies, people, and the press.
I have the CISSP, CCSP, and CISM, minus any dues we probably haven't paid this year to ISACA and ISC. That makes me, again minus membership dues, somewhat qualified to say the following:
- Wyze is either the 2nd or 3rd largest GDPR incident (really breach) to date.
- Because of the extraordinary sensitive nature of what the data allows you to do, to watch, to see, to observe, the EU could very well say it is the largest GDPR incident to date. Some of these capabilities will be detailed in the next post, as well as connections to Chinese government entities.
- Article 9, Article 25, Article 32, and Article 35 all would be extraordinarily problematic for Wyze and we see literally no way out of them short of ceasing all business operations on the European continent.
- Lastly Article 83 gives guidance on how punishments for breaches should be decided, and subsections a,b,d,e,and g are very much not in Wyze's favor.