How will the cybersecurity landscape change in 2024? Will it become safer or more dangerous? We can guess, make predictions… or just wait a bit and see… However, to move forward and make some assumptions first we need to look back at the past. The following article is the first in a series of this type where we’ve decided to dig into DevOps-related “fackups”.
Let’s today take a look at GitHub-related infamous incidents, including outages, high-severity flaws, and malicious attacks. Do you still consider backing up your GitHub data or not? Let us help you make up your mind.
DECEMBER 2023
GitHub Status info: 10 incidents
Hackers abuse GitHub to avoid detection & control compromised hosts
Researchers from ReversingLabs noticed that threat actors started using GitHub for their malicious purposes and reported that “lately, we have observed the increasing use of the GitHub open-source development platform for hosting malware.” The novel methods that the malicious actors use include leveraging secret Gists and issuing malicious commands via git commit messages. So, the trick is that they can host their malware on a legitimate public service and use it as a dead-drop resolver to retrieve the real command-and-control (C2) address.
These sneaky techniques permit hostile actors to conceal their malicious network traffic inside of legitimate communications on a compromised network. In this case, it’s difficult to identify and address the threats quickly and effectively. Consequently, the infected endpoint corresponding with a GitHub repository may not be reported as a suspicious one, giving a threat actor the green light to create an attack infrastructure that’s reliable and inexpensive, and threaten other users and their data.
15K Go Module Repos on GitHub are vulnerable to RepoJacking attack
GitHub RepoJacking has been threatening GitHub users for a while, however, in 2023 it has always been on the tip of the tongue. Researchers from AquaSec concluded that 9 million repos can be vulnerable to RepoJacking (we will look at this case further), the Checkmarx team discovered that GitHub’s vuln could have exposed over 4K packages to RepoJacking attacks, and recently VulnCheck had been investigating this issue and found out that over 15K Go module repos are vulnerable to this kind of an attack.
It’s worth mentioning that Go programming language modules are particularly vulnerable to RepoJacking as, in contrast to other package manager solutions, including npm packages or PyPI, they are uploaded to version control systems, like GitHub or Bitbucket.
In its research, VulnCheck stated that:
_- “More than 9,000 repositories are vulnerable to RepoJacking due to GitHub username changes.”
- “More than 6,000 repositories were vulnerable to RepoJacking due to account deletion.”_
To address the issue, GitHub implemented a countermeasure known as popular repository namespace retirement. This countermeasure is aimed at blocking the attempts to create repos with the names of retired namespaces that have already been cloned more than 100 times before the owner’s account name has been changed or removed.
Source: VulnCheck
NOVEMBER 2023
GitHub Status info: 9 incidents
OCTOBER 2023
GitHub Status info: 7 incidents
SEPTEMBER 2023
GitHub Status info: 20 incidents
AUGUST 2023:
GitHub Status info: 17 incidents
JULY 2023:
GitHub Status info: 11 incidents
Cyberattack on GitHub customers
On July 18, GitHub published an alert that it tracked and identified a low-volume social engineering campaign that was targeting the personal accounts of technology companies’ employees. Though, the majority of the targeted accounts were related to the blockchain, cryptocurrency, or online gambling sectors, a few accounts associated with the cybersecurity sector fall victim as well.
The hacking group from North Korea, determined by Microsoft as Jade Sleet, impersonated a developer and created a GitHub account and other fake accounts on social media, including LinkedIn, Slack, and Telegram, and after initiating a contact persuaded the victim to collaborate on a GitHub repository. Thus, by cloning and executing the content of the repo (which contained the malicious npm packages!), the threat actors managed to infect the victim’s data with malware.
The Record. Recorded Future News
JUNE 2023
GitHub Status info: 13 incidents
Millions of GitHub repos can be vulnerable to RepoJacking
According to AquaSec’s security team, “Nautilus”, millions of GitHub repos can be potentially vulnerable to RepoJacking, which could enable malicious actors to launch supply chain attacks affecting lots of users. In this case, an attacker takes over a retired organization’s or user’s name and publishes trojanized versions of repos to run the malicious code in it.
After analyzing a sample of more than a million GitHub repositories, the group of researchers found out that about 2.95% of them might be vulnerable to this kind of vulnerability.
As a result, the researchers estimated that RepoJacking can affect appx. 9M projects, which is a lot. To minimize the risks of the repository being exposed to this vulnerability, developers should:
- have a clear understanding of what GitHub open-source software they are using,
- check if the name of the open-source repo they use has ever been changed or moved,
- fork the open-source repository before using it,
- keep up with the software bill of materials (SBOM),
- adopt security practices, like two-factor authentication, firewalls, access restriction controls, regular change of access tokens or SSH keys, and backup of its critical data.
MAY 2023
GitHub Status info: 10 incidents
GitHub addresses several availability incidents in a month
According to GitHub’s report, the service provider experienced some availability issues, both long-running and of shorter duration. For sure, the second week of May was a hard one for GitHub DevOps… They had to handle 3 incidents during 3 days in a row. So, let’s look at the retrospective of the events:
May 9th, 2023 – Git Databases degraded due to configuration change
Triggered by the alteration in the setup of the internal server that serves Git data, GitHub experienced a major outage when 8 of 10 services were degraded. For an hour many services experienced widespread failures as they were unable to read newly-written Git data. “Following this outage, there was an extended timeline for post-incident recovery of some pull request and push data,” writes Mike Hanley, the Chief Security Officer and SVP of Engineering at GitHub.
To remediate the issue the GitHub team decided to revert the config change and tried to revert a rollback yet failed due to an internal infrastructure error. So, they decided to complete a gradual failover, after which they managed to restore write operations. However, “additional time was needed to get Git data, website-visible contents, and pull requests consistent for pushes received during the outage to achieve a full resolution,” states the GitHub report.
May 10th, 2023 – GitHub App authentication token issuance degradation
The next day after the outage, GitHub faced another incident that led to 6 of 10 main services being degraded. As it turned out, “the database cluster serving GitHub App auth tokens saw a 7x increase in write latency for GitHub App permissions”.
The incident also affected GitHub functionalities that depend on tokens for operation. It included the source of each GITHUB_TOKEN in GitHub Actions and the tokens used for giving GitHub Codespaces access to the users’ repos, which serve as a security mechanism for private GitHub Pages. In this case, due to token issuance failure, GitHub Actions and GitHub Codespaces can’t access the data they require to run, failing to launch.
May 11th, 2023 – Git database degraded as a result of loss to read replicas
Yet the troubles weren’t over… On May 11, GitHub experienced a crash in a database cluster serving Git data during which 8 of 10 main services were degraded for a short period. That, in turn, triggered an automated failover during which from 15% to 26% of requests for Git data were failed or slow… let’s not forget that around 100M developers across the globe use GitHub for coding.
APRIL 2023
GitHub Status info: 12 incidents
MARCH 2023
GitHub Status info: 20 incidents
GitHub’s private SSH key exposed in a public GitHub repository
After discovering its RSA SSH private key for GitHub.com being briefly exposed in a public GitHub repository, the version control service provider urgently rotated its secret for GitHub.com. This measure was taken as a proactive one to protect GitHub users “from any chance of an adversary impersonating GitHub or eavesdropping on their Git operations over SSH,” states Mike Hanley in GitHub’s report.
Customers using GitHub’s ECDSA or Ed25519 keys didn’t notice any changes, though those who saw the warning message while connecting to GitHub.com via SSH had to manually renew the RSA SSH public key entry to their files. GitHub has provided detailed instructions on how to do that in its report.
After a nasty incident, GitHub explained that “this issue was not the result of compromise of any GitHub systems or customer information. Instead, the exposure was the result of what we believe to be an inadvertent publishing of private information.”
The only question that stays open is how long the exposed key was available to the public eye.
FEBRUARY 2023
GitHub Status info: 19 incidents
JANUARY 2023
GitHub Status info: 17 incidents
A set of repos connected to GitHub Desktop and Atom accessed by a hacker
This incident tracks its roots in December 2022, when GiHub reported that a hostile actor gained access to a set of repos that the service provider used in the planning and development of the text and source code editor Atom, and GitHub Desktop. Before you think about how that is related to 2023, here is the answer: all the updates that GitHub users needed to perform were during January.
So, let’s look at the summary of the incident. On December 6, 2022, using a compromised PAT (Personal Access Token) linked to a machine account to clone repos from its Atom, desktop, and “other deprecated GitHub-owned organizations”, a malicious actor gained access to the source code repositories and stole a set of encrypted code-signing certificates.
Though GitHub reported that all the certificates were password-protected and no sign of
malicious use was detached, the service provider decided to take preventive measures. It
revoked the exposed certificates that it used for the GitHub Desktop and Atom applications.
Thus, in order to continue using their applications securely, the GitHub users needed to update them by February 2, 2023… as it has already been mentioned during January.
How to ensure the security of your GitHub data in 2024?
As you see it’s important to keep the hygiene of your GitHub data. If you use open-source, always double-check the libraries that you utilize to not fall victim to RepoJacking or other malicious actors’ traps.
Also, no matter what repos you use public or private ones, you should keep up with GitHub security best practices, including restricting and controlling access, rotating personal access tokens and SSH keys, including defense-in-depth principles, zero-trust approach while building your CI/CD.
And, don’t forget about GitHub backup, as the final line of your source code and metadata security. Whether it’s an outage or a ransomware attack, you can simply use Disaster recovery technologies to instantly restore all your GitHub repos and metadata to any location of your choice – the same GitHub account, a new GitHub account, cross-overly to GitLab or Bitbucket, or your local device, guaranteeing uninterrupted workflow. Moreover, you shouldn’t forget that DevOps backup is one of the requirements to meet security certifications, including ISO 27001 and SOC 2.
So, keep your source code secure. Who knows what ingenious malicious ideas and schemes threat actors can imagine this year?
Useful resources:
GitHub backup best practices
GitHub Compliance – All you need to know
Ultimate review of the most infamous GitHub-related security incidents in 2022
✍️ Subscribe to GitProtect DevSecOps X-Ray Newsletter – your guide to the latest DevOps & security insights
🚀 Ensure compliant DevOps backup and recovery with a 14-day free trial
📅 Let’s discuss your needs and see a live product tour
Top comments (0)