GitHub is the preeminent source control tool today and is used by many enterprises without a second thought. But GitHub was built first and foremost for open source collaboration with later work focused on making it viable for enterprises. How successful was that work as applied to security?
Short answer: Mixed results.
Longer answer: GitHub has gotten much better in recent years, perhaps as a side effect of their acquisition by Microsoft, and they still have a long way to go.
The rest of this post will share various sharp edges in GitHub in no particular order.
Status Checks (In)Security
Prior to December 2021 all status checks in GitHub could be overridden by any person with write access to the repo via a simple API call. After that date, status checks may or may not be overridable depending on how the repository’s admin has configured each check. A status check can be bound to a single GitHub App or unbound (allowing any person or integration with write access to update it).
Innumerable repos use GitHub status checks to enforce various quality and approval checks, such as passing unit tests or seeking approval from appropriate engineers. In many cases these approval checks may be tied to compliance requirements and/or security controls, such as when changes to some sensitive payment flow requires additional approvals.
The fix put in place in December 2021 added a new “associate on first use” behavior where any status check created by a GitHub App will only accept updates from that same app. While this change did improve the situation, it is still incomplete where:
- Multiple apps are required for a single status check
- All GitHub apps have rate limits set by GitHub and larger repositories and/or organizations will inevitably exceed these limits
- Once the limits are reached, the best solution is to create multiple apps (because GitHub doesn’t allow rate limit changes)
- The relevant status checks must be unbound, allowing anyone with write access to update the status.
- Apps are not used (e.g. service users)
- Status checks can only be bound to GitHub Apps, so a status check created by a service user can always be overwritten by any other user with write access
- In theory the solution is to move all status checks to Apps, though finding resources to modify load bearing code may be a challenge
- Legacy status checks that haven’t been bound to an App
- Any status check created prior to December 2021 will remain in an unbound state forever, unless a repository admin intervenes
- Repository administrators disable the binding for any reason
- Organization owners have no visibility into repository branch protections without custom API tooling. This makes auditing and remediation challenging
Repositories using a fork-based development pattern are more resistant to this issue if the root repository has fewer people with write access. But forks have their own issues as discussed below, and forks aren’t necessarily used universally.
GitHub extensively encourages the use of repository forking for development. By default though, forks are created in personal accounts, placing them outside nearly all organization security controls. While GitHub has documented this issue vaguely , they do not warn owners when the feature is enabled.
Therefore, once an engineer forks
- The repo is accessible without transiting SSO/SAML (assuming you have that enabled)
- The engineer can grant repo access to people and integrations outside the organization
- Actions taking place in the repo will not be included in organization audit logging
In June 2022, GitHub added support for restricting forks to within an organization (or various other security boundaries), though this feature is only available as an enterprise policy (see below). Restricting forks to within the organization (or enterprise, depending on desired security boundaries) is the safe option here.
In another case of GitHub being an open source collaboration platform first, and enterprise platform second, nearly all users in GitHub are solely and completely owned by the human that created them rather than the enterprise to which they are associated.
For many people this is a great feature, allowing them to maintain a single public identity and history of their work. But from an enterprise perspective, this is a terrible state to be in. This is best explored in a hypothetical scenario:
Sam joins a company and adds their personal user to the GitHub organization. Some time later they go on vacation to an area with no connectivity (e.g. camping).
While Sam is on vacation, their account begins acting suspiciously and the company wants to investigate. The company then finds that no user-level logs are accessible to them, so they ask GitHub for help.
GitHub responds (likely hours or days later because timely support costs extra), and they say user logs can only be provided with explicit permission from the user (which is impossible to get in the short term) or in response to a subpoena.
The company has nearly no ability to investigate until Sam returns from vacation.
In short, since the user belongs to the human rather than the enterprise, the enterprise has no right to access.
GitHub has been slowly rolling out partial mitigations for this issue over time:
- 2021-09: Released Enterprise Managed Users (EMUs)
- 2022-08: Added IP address information to some audit log events
- 2022-10: Added more events to audit logs
- 2023-02: Added more events to audit logs
Of the improvements made, only EMUs address the core issue by moving user ownership into the Enterprise. While this change is welcome, the transition to EMUs is very involved and requires all usernames in the system being remapped or rewritten (or a clean start).
Uselessly Universal Enterprise Policies
GitHub has a useful, though confusingly named, abstraction layer called an “Enterprise” which can contain multiple organizations. This is akin to an AWS Organization containing multiple AWS Accounts.
The GitHub Enterprise acts as a container for all of your organizations, location for new security features (such as IP logging), and control point for policies that are then applied to all constituent organizations. These policies allow an enterprise to enforce certain practices that organization owners cannot disable or change. This is useful, in theory, for situations where you want to enforce a best practice on all organizations, while also allowing some low-security accounts to be managed by less experienced staff.
In practice however, these policies are almost entirely useless. All GitHub Enterprise policies are enforced globally with no exceptions or variation allowed. If you have a set of production organizations and a set of test organizations there are almost no policies that can be applied universally as some test organizations likely need an exception.
Since releasing GitHub Enterprises some years back there has been no change or improvement in this area, though they do continue to add new policies. The obvious solution is some form of organization tree structure, akin to AWS Organization OUs, where policies can be applied either globally, or to a subset of the tree.
Repository Admins or Organization Owner?
GitHub grants incredible powers to repository administrators, including some that owners may not be aware of.
GitHub Owners looking to secure their organization may use the security settings to restrict GitHub App approvals to require an owner. Once enabled, the owner will begin receiving emails requesting app approvals from various organization members. Incredibly though, this restriction doesn’t apply to repository admins approving apps that have repo-only API scopes, allowing a malicious application to have free reign within the scope of a single repository.
As such, any repository administrator can attach nearly any GitHub App to their repositories without any approval or review. That app could then download or modify the code at its own leisure, likely with organization owners none the wiser.
The only known methods to prevent this behavior are:
- Don’t allow anyone to be a repository administrator
- Enable GitHub’s IP Allowlist on the organization
- This doesn’t prevent apps from being added, but they would not be allowed to access your organization or its repositories
- Requires that IP Allowlisting not trust App provided IPs
- Active monitoring of all App approvals in the audit log with immediate remediation
Both solutions are wildly challenging and unpopular, so good luck.
The IP Allowlist Nuclear Option
Enabling the IP Allowlist is the single greatest security improvement possible in GitHub, and I strongly recommend that all organizations enable it.
Enabling the IP Allowlist immediately:
- Blocks unapproved GitHub Apps from accessing any part of the Organization (addressing the previous topic)
- Prevents GitHub credentials from functioning outside your controlled network (e.g. VPN or zero-trust proxy), adding defense-in-depth for organization assets (the user would still be compromised but not their access to the organization)
The disadvantages are not insignificant though:
- Requires the complete deployment of a VPN or Zero Trust access layer for all GitHub users
- Incompatible with GitHub Actions unless you move to self-hosted runners
- Many third party services are unable or unwilling to provide source IP addresses
- In-house systems will need to access GitHub from a relatively stable set of IP addresses, possibly requiring infra changes
- Some process and/or tooling will be necessary to maintain the allowlist
I don’t expect that anyone will read this and immediately look for the exit, but I do hope that it raises awareness of the risky default configuration (and limited options) of GitHub for enterprise users. In talking to other GitHub owners over the years I know some of these issues are relatively unknown while others are practically a shibboleth.
I hope that GitHub continues to advance the state of security in their system, ideally with even more effort than they are already showing. Numerous public security incidents have shown that GitHub is a clear target for attackers who are often more familiar with these issues than organization owners. GitHub has an opportunity to advance the security of all of their customers and save everyone a great deal of trouble.
Please send comments my way by responding to my post on Mastodon.