CVE-2023-23767: Release notes - GitHub Enterprise Server 3.9 Docs

Enterprise Server 3.9.6Download GitHub Enterprise Server 3.9.6

October 24, 2023

📣 This is not the latest release of Enterprise Server. Please use the latest release for the latest security, performance, and bug fixes.

Warning: A change to MySQL in GitHub Enterprise Server 3.9 and later may impact the performance of your instance. Before you upgrade, make sure you’ve read the “Known issues” section of these release notes.

3.9.6: Security fixes

• HIGH: Due to an incorrect permission assignment for some configuration files, an attacker with access to a local operating system user account could read MySQL connection details including the MySQL password. GitHub has requested CVE ID CVE-2023-23767 for this vulnerability.

• Packages have been updated to the latest security versions.

3.9.6: Bug fixes

• The ghe-cluster-repl-status command did not display all replication statuses.

• On an instance in a cluster configuration with high availability enabled, ghe-config-apply timed out while waiting for hookshot-go to start on replica application nodes.

• SpokesRepairRepoReplicaJob and SpokesSyncCacheReplicaJob jobs failed, causing cache server replicas to not update and potentially prolonging replication issues.

• /var/log/lastlog was not copied over as a sparse file during ghe-upgrade, which could cause issues by using additional disk space.

• On an instance in a cluster configuration, when managing maintenance mode using ghe-cluster-maintenance, an erroneous warning appeared that read "Warning: Maintenance mode set on primary, please make sure to set it on any active replica if needed". - | ghe-repl-status did not identify Git replicas in certain incomplete states and incorrectly suggested that a failover could be performed safely. In some cases, this led to data loss during failover.

• Repository exports using ghe-migrator or the REST API’s operation for organization migrations could fail when a large number of commit comments or long commit comments were present.

• On an instance with a GitHub Advanced Security license and secret scanning enabled, secret scanning suggested incorrect filters when viewing both open and closed alerts.

• On instances using the private beta of SCIM provisioning, some users were presented with a “single sign-in” hover card.

• On an instance with multiple nodes, ghe-spokes status did not identify Git replicas in certain incomplete states, causing a false report that replication was in sync and leading to data loss or replication issues during failover.

• On an instance with GitHub Actions enabled, administrators received a 500 error after attempting to force cancel a workflow run via Staff Tools.

• On an instance with a GitHub Advanced Security license, repositories within organizations created using the + dropdown menu did not have GitHub Advanced Security features enabled automatically, even if the features should have been enabled.

• As a security measure, GitHub Pages does not build sites that contain symbolic links except when using custom GitHub Actions workflows. This change strengthens GitHub Pages’s symbolic link detection.

• On an instance with a GitHub Advanced Security license and secret scanning enabled, dry runs sometimes incorrectly reported no results for custom patterns.

3.9.6: Changes

• Instructions in the “Migrations” section of the Management Console clarify that only standard AWS S3 endpoints are supported when configuring AWS S3 as a blob storage provider for migrations.

• When running async repository repairs, the output message about scheduling a repair job is more accurate.

• On an instance in a cluster configuration, administrators can identify the repository networks or gists that are common across a specified set of storage nodes using the spokesctl find-on-replicas command.

3.9.6: Known issues

• Custom firewall rules are removed during the upgrade process.

• The GitHub Packages npm registry no longer returns a time value in metadata responses. This was done to allow for substantial performance improvements. We continue to have all the data necessary to return a time value as part of the metadata response and will resume returning this value in the future once we have solved the existing performance issues.

• During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

• If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see “Troubleshooting access to the Management Console.”

• On an instance in a high-availability configuration, passive replica nodes accept Git client requests and forward the requests to the primary node.

• If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

• When running ghe-config-apply, the process may stall with the message Deployment is running pending automatic promotion.

• The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn’t required, and outputs an error instead of a warning.

• On an instance with subdomain isolation disabled, Mermaid diagrams in the web UI display an “Unable to render rich display” error and fail to render.

• When enabling CodeQL via default setup at scale, some checks related to GitHub Actions are omitted, potentially preventing the process from completing.

• After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see “Known issues with upgrades to your instance.” [Updated: 2023-08-11]

• After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9 or 3.10, I/O utilization will increase, and in some cases the instance’s performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see “Known issues with upgrades to your instance.”

• In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support.

• On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes.

• Jobs in a deprecated queue are not processed and may accumulate over time. These jobs are reflected in the monitor dashboard’s “Aqueduct queue depth” graph as an increase in resource_activity. In some cases, a buildup of unprocessed jobs can result in memory exhaustion. If you observe memory exhaustion on your instance and see a high metric for resource_activity, contact GitHub Support.

• When an administrator uses the -p flag with the ghe-support-bundle utility to collect data for a specific number of hours, the utility erroneously collects more logs than necessary.

Enterprise Server 3.9.5Download GitHub Enterprise Server 3.9.5

September 21, 2023

📣 This is not the latest patch release of this release series, and this is not the latest release of Enterprise Server. Please use the latest release for the latest security, performance, and bug fixes.

Warning: A change to MySQL in GitHub Enterprise Server 3.9 and later may impact the performance of your instance. Before you upgrade, make sure you’ve read the “Known issues” section of these release notes.

3.9.5: Security fixes

• HTTP Strict Transport Security (HSTS) is enabled within the Management Console.

• Packages have been updated to the latest security versions.

3.9.5: Bug fixes

• On an instance with GitHub Actions enabled, scale sets configured at the enterprise level did not appear for use within the instance’s organizations or repositories.

• On an instance with a GitHub Advanced Security license and secret scanning enabled, secret scanning alerts could fail to show an error message in the UI when a failure occurred closing or reopening the alert.

• On an instance with a GitHub Advanced Security license and secret scanning enabled, and when using Safari, changing additional match requirements for a custom pattern did not retrigger custom pattern evaluation against a user submitted test string.

• On an instance with a GitHub Advanced Security license and secret scanning enabled, when token location(s) included a commit that introduced a large change, the page for viewing the alert would load slowly.

• In some cases, users could reopen a pull request that should not have been able to be reopened.

• When running the ghe-saml-mapping-csv CLI command in dry run mode, the operation failed with errors.

• When uploading migration archives to blob storage, the GitHub Enterprise Server instance’s outbound web proxy server was not used.

• On an enterprise with the policy setting that disallows repository admins from enabling/disabling secret scanning, transferring a repository to a new organization that automatically enabled secret scanning wouldn’t result in the transferred repository being automatically enabled for secret scanning.

• When viewing fine-grained personal access tokens, the permissions text for pre-receive hooks was not visible for selection when filtering by permission.

• When migrating a repository from a GitHub Enterprise Server instance to another location, the ghe-migrator target_url command allows you to record the repository’s new location. The new URL is displayed when you visit the main page of the repository in the web interface.

• On an instance with subdomain isolation disabled, a notebook could not be loaded due to incorrect asset paths.

• On an instance with subdomain isolation disabled, a notebook could not be loaded due to an extra / character in the URL path.

• On an instance with a GitHub Advanced Security license and secret scanning enabled, in some cases, custom patterns would erroneously show no results for a dry run.

• On an instance with GitHub Actions enabled, the software on ephemeral runners would not automatically update to the latest version.

• On an instance with subdomain isolation disabled, Mermaid diagrams in the web UI display an “Unable to render rich display” error and fail to render.

3.9.5: Changes

• When listing the node metadata for all nodes using the Manage GitHub Enterprise Server REST API, information about whether a given node is a replica is included.

• When GitHub Enterprise checks for a new upgrade or hotpatch package, if the check fails the failure details are output to the ghe-update-check log, and the Management Console UI provides a “Check Again” button to rerun the check.

• When providing data to GitHub Support, GitHub Enterprise Server displays a notice describing how support data is used before uploading the support files.

• When running async repository repairs, the output message about scheduling a repair job is more accurate.

3.9.5: Known issues

• Custom firewall rules are removed during the upgrade process.

• The GitHub Packages npm registry no longer returns a time value in metadata responses. This was done to allow for substantial performance improvements. We continue to have all the data necessary to return a time value as part of the metadata response and will resume returning this value in the future once we have solved the existing performance issues.

• During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

• If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see “Troubleshooting access to the Management Console.” [Updated: 2023-02-23]

• On an instance in a high-availability configuration, passive replica nodes accept Git client requests and forward the requests to the primary node.

• If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

• When running ghe-config-apply, the process may stall with the message Deployment is running pending automatic promotion.

• The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn’t required, and outputs an error instead of a warning.

• When enabling CodeQL via default setup at scale, some checks related to GitHub Actions are omitted, potentially preventing the process from completing.

• After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see “Known issues with upgrades to your instance.” [Updated: 2023-08-11]

• After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9 or 3.10, I/O utilization will increase, and in some cases the instance’s performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see “Known issues with upgrades to your instance.”

• In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support.

• On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes.

• Jobs in a deprecated queue are not processed and may accumulate over time. These jobs are reflected in the monitor dashboard’s “Aqueduct queue depth” graph as an increase in resource_activity. In some cases, a buildup of unprocessed jobs can result in memory exhaustion. If you observe memory exhaustion on your instance and see a high metric for resource_activity, contact GitHub Support. [Updated: 2023-10-10]

• When an administrator uses the -p flag with the ghe-support-bundle utility to collect data for a specific number of hours, the utility erroneously collects more logs than necessary. [Updated: 2023-10-13]

• The settings for enabling scheduled reminders were added unintentionally to this release. Scheduled reminders are not officially supported. [Updated: 2023-10-17]

Enterprise Server 3.9.4Download GitHub Enterprise Server 3.9.4

August 24, 2023

📣 This is not the latest patch release of this release series, and this is not the latest release of Enterprise Server. Please use the latest release for the latest security, performance, and bug fixes.

Warning: A change to MySQL in GitHub Enterprise Server 3.9 and later may impact the performance of your instance. Before you upgrade, make sure you’ve read the “Known issues” section of these release notes.

3.9.4: Security fixes

• An authorization/sensitive information disclosure vulnerability was identified in GitHub Enterprise Server that allowed a fork to retain read access to an upstream repository after the fork’s visibility was changed to private. This vulnerability was reported via the GitHub Bug Bounty Program and assigned CVE-2023-23763. [Updated: 2023-09-01]

• Packages have been updated to the latest security versions.

3.9.4: Bug fixes

• On an instance with GitHub Actions enabled, scale sets configured at the enterprise level did not appear for use within the instance’s organizations or repositories.

• When an administrator tried to validate blob storage connection settings for GitHub Enterprise Importer in the Management Console using the Test storage settings button, the operation failed.

• syslog-ng configurations for containerized services caused errors for log forwarding services. The configurations have been removed.

• When an instance exhausted available memory, in some cases, the system’s out-of-memory killer (OOMK) killed the process for dockerd, causing Nomad to fail to recover after systemd restarted Docker.

• In some cases, when starting a new GitHub Enterprise Server instance, the preflight page indicated that there was no user disk of sufficient size attached.

• When running the ghe-migrator, certain error messages contained an invalid link to import documentation.

• On an instance with GitHub Actions enabled, due to mismatched values, users could not easily associate workflow job run IDs from the GitHub Enterprise Server APIs or webhooks with a job in the UI. Workflow job runs now use a new URL pattern of …actions/runs/job/{job_id}, and job_id matches values from APIs and webhook payloads.

• Administrators could not see or use the “Migrations” section in an instance’s Management Console, which prevented the configuration of blob storage for GitHub Enterprise Importer. [Updated: 2023-08-31]

3.9.4: Known issues

• Custom firewall rules are removed during the upgrade process.

• The GitHub Packages npm registry no longer returns a time value in metadata responses. This was done to allow for substantial performance improvements. We continue to have all the data necessary to return a time value as part of the metadata response and will resume returning this value in the future once we have solved the existing performance issues.

• During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

• If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see “Troubleshooting access to the Management Console.” [Updated: 2023-02-23]

• On an instance in a high-availability configuration, passive replica nodes accept Git client requests and forward the requests to the primary node.

• If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

• When running ghe-config-apply, the process may stall with the message Deployment is running pending automatic promotion.

• The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn’t required, and outputs an error instead of a warning.

• On an instance with subdomain isolation disabled, Mermaid diagrams in the web UI display an “Unable to render rich display” error and fail to render.

• When enabling CodeQL via default setup at scale, some checks related to GitHub Actions are omitted, potentially preventing the process from completing.

• After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see “Known issues with upgrades to your instance.” [Updated: 2023-08-11]

• After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9 or 3.10, I/O utilization will increase, and in some cases the instance’s performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see “Known issues with upgrades to your instance.”

• In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support. [Updated: 2023-09-04]

• On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes. [Updated: 2023-09-21]

• On an instance with GitHub Actions enabled, ephemeral self-hosted runners do not automatically update to the latest version. Users will need to manually update the runners to the latest version. [Updated: 2023-09-29]

• Jobs in a deprecated queue are not processed and may accumulate over time. These jobs are reflected in the monitor dashboard’s “Aqueduct queue depth” graph as an increase in resource_activity. In some cases, a buildup of unprocessed jobs can result in memory exhaustion. If you observe memory exhaustion on your instance and see a high metric for resource_activity, contact GitHub Support. [Updated: 2023-10-10]

• When an administrator uses the -p flag with the ghe-support-bundle utility to collect data for a specific number of hours, the utility erroneously collects more logs than necessary. [Updated: 2023-10-13]

• The settings for enabling scheduled reminders were added unintentionally to this release. Scheduled reminders are not officially supported. [Updated: 2023-10-17]

Enterprise Server 3.9.3Download GitHub Enterprise Server 3.9.3

August 10, 2023

📣 This is not the latest patch release of this release series, and this is not the latest release of Enterprise Server. Please use the latest release for the latest security, performance, and bug fixes.

Warning: A change to MySQL in GitHub Enterprise Server 3.9 and later may impact the performance of your instance. Before you upgrade, make sure you’ve read the “Known issues” section of these release notes.

3.9.3: Security fixes

• LOW: An incorrect comparison vulnerability was identified in GitHub Enterprise Server that allowed commit smuggling by displaying an incorrect diff in a reopened pull request. To exploit this vulnerability, an attacker would need write access to the repository. This vulnerability was reported via the GitHub Bug Bounty program and was assigned CVE-2023-23766. [Updated: 2023-09-22]

3.9.3: Bug fixes

• API results were incomplete, and ordering of results was incorrect if asc or desc appeared in lowercase within the API query.

• The checks in the merge box for a pull request did not always match the the checks for the most recent commit in the pull request.

• When a site administrator used GitHub Enterprise Importer on versions 3.7 and below to migrate repositories from GitHub Enterprise Server, the system backup size would increase after running many migrations due to storage files not being cleaned up.

• A collaborator with the “Set the social preview” permission inherited from the “Read” role could not upload the social preview image of a repository.

• The security settings page for a repository would return an error when enterprise-level runners were assigned to the repository.

• GitHub Enterprise Server was queuing zip jobs unnecessarily.

3.9.3: Changes

• On GitHub Enterprise Server 3.8 and above, a blob storage provider must be configured in the Management Console in order to use the GitHub Enterprise Importer CLI, “startRepositoryMigration” GraphQL API, or “Start an organization migration” REST API. The “Migrations” section in the Management Console was mistakenly removed and has been added back.

• Administrators can display all repositories in a network with spokesctl by using the repositories subcommand.

• The secondary abuse rate limits of the GraphQL API are now configurable in the Management Console. [Updated: 2023-09-01]

3.9.3: Known issues

• After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9 or 3.10, I/O utilization will increase, and in some cases the instance’s performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see “Known issues with upgrades to your instance.” [Updated: 2023-08-11]

• After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see “Known issues with upgrades to your instance.” [Updated: 2023-08-11]

• Custom firewall rules are removed during the upgrade process.

• The GitHub Packages npm registry no longer returns a time value in metadata responses. This was done to allow for substantial performance improvements. We continue to have all the data necessary to return a time value as part of the metadata response and will resume returning this value in the future once we have solved the existing performance issues.

• During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

• If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see “Troubleshooting access to the Management Console.” [Updated: 2023-02-23]

• On an instance in a high-availability configuration, passive replica nodes accept Git client requests and forward the requests to the primary node.

• If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

• When running ghe-config-apply, the process may stall with the message Deployment is running pending automatic promotion.

• The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn’t required, and outputs an error instead of a warning.

• On an instance with subdomain isolation disabled, Mermaid diagrams in the web UI display an “Unable to render rich display” error and fail to render.

• When enabling CodeQL via default setup at scale, some checks related to GitHub Actions are omitted, potentially preventing the process from completing.

• Administrators cannot set or update the instance’s blob storage settings in the Management Console using the “Migrations” settings tab. No matter what values an administrator provides, the following error message will appear: Unable to connect to migrations provider. Please check the form and try again. [Updated: 2023-08-18]

• In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support. [Updated: 2023-08-24]

• On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes. [Updated: 2023-09-21]

• On an instance with GitHub Actions enabled, ephemeral self-hosted runners do not automatically update to the latest version. Users will need to manually update the runners to the latest version. [Updated: 2023-09-29]

• Jobs in a deprecated queue are not processed and may accumulate over time. These jobs are reflected in the monitor dashboard’s “Aqueduct queue depth” graph as an increase in resource_activity. In some cases, a buildup of unprocessed jobs can result in memory exhaustion. If you observe memory exhaustion on your instance and see a high metric for resource_activity, contact GitHub Support. [Updated: 2023-10-10]

• When an administrator uses the -p flag with the ghe-support-bundle utility to collect data for a specific number of hours, the utility erroneously collects more logs than necessary. [Updated: 2023-10-13]

• The settings for enabling scheduled reminders were added unintentionally to this release. Scheduled reminders are not officially supported. [Updated: 2023-10-17]

Enterprise Server 3.9.2Download GitHub Enterprise Server 3.9.2

July 28, 2023

📣 This is not the latest patch release of this release series, and this is not the latest release of Enterprise Server. Please use the latest release for the latest security, performance, and bug fixes.

Warning: A change to MySQL in GitHub Enterprise Server 3.9 and later may impact the performance of your instance. Before you upgrade, make sure you’ve read the “Known issues” section of these release notes.

The known issues originally published on 2023-07-28 omitted a number of known issues that still existed. The Known issues section below was updated on 2023-08-08.

3.9.2: Changes

• Added a pre-upgrade check to validate the GHES version and MySQL configuration before allowing an upgrade to 3.9.

• Adjusted the timeout threshold for shutting down MySQL to prevent premature termination when upgrading to GHES 3.9.

3.9.2: Known issues

• After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9, I/O utilization will increase, and in some cases the instance’s performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see “Known issues with upgrades to your instance.”

• After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see “Known issues with upgrades to your instance.” [Updated: 2023-08-11]

• The Management Console may get stuck in a loop showing “Checking requirements…” when attempting to download an update. It is safe to proceed the update process manually via the command line.

• When enabling CodeQL via default setup at scale, some checks related to GitHub Actions are omitted, potentially preventing the process from completing.

• After restoration of a backup created using GitHub Enterprise Server Backup Utilities 3.7.0, 3.8.0, or 3.9.0, users may not be able to sign into the instance. To fix this issue, plus a bug that was preventing secret scanning encryption keys from being backed up, upgrade your backup host to use GitHub Enterprise Server Backup Utilities 3.9.1 and generate a new full backup using ghe-backup. For more information on using an existing backup, see “Known issues with backups for your instance.” [Updated: 2023-07-31]

• On an instance in a cluster configuration, after you upgrade nodes other than the primary MySQL node and before you upgrade the primary MySQL node, the following output may appear multiple times after you run ghe-config-apply.

  Error response from daemon: conflict: unable to delete IMAGE_ID (cannot be forced) - image is being used by running container CONTAINER_ID
  

  You can safely ignore this message.

• Custom firewall rules are removed during the upgrade process.

• The GitHub Packages npm registry no longer returns a time value in metadata responses. This was done to allow for substantial performance improvements. We continue to have all the data necessary to return a time value as part of the metadata response and will resume returning this value in the future once we have solved the existing performance issues.

• On an instance in a high-availability configuration, passive replica nodes accept Git client requests and forward the requests to the primary node.

• The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn’t required, and outputs an error instead of a warning.

• When using an outbound web proxy server, the ghe-btop command may fail in some circumstances with the error "Error querying allocation: Unexpected response code: 401".

• If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

• When running ghe-config-apply, the process may stall with the message Deployment is running pending automatic promotion.

• During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

• If the root site administrator is locked out of the Management Console after failed login attempts, the account will not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see “Troubleshooting access to the Management Console.”

• On an instance with subdomain isolation disabled, Mermaid diagrams in the web UI display an “Unable to render rich display” error and fail to render. [Updated: 2023-08-18]

• The “Migrations” section is missing from the Management Console, so it isn’t possible to enable, disable, or reconfigure blob storage credentials for migrations. [Updated: 2023-08-18]

• In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support. [Updated: 2023-08-24]

• On an instance with GitHub Actions enabled, if shared runner groups are configured for the enterprise, the enterprise security overview page may return a 500 error. You can avoid the issue by trying one of the following workarounds.

  • Add a runner scale set to the enterprise runner group shared with the repositories.
  • Remove access to the enterprise runner group from the affected repositories or organizations.

  [Updated: 2023-09-05]

• On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes. [Updated: 2023-09-21]

• On an instance with GitHub Actions enabled, ephemeral self-hosted runners do not automatically update to the latest version. Users will need to manually update the runners to the latest version. [Updated: 2023-09-29]

• Jobs in a deprecated queue are not processed and may accumulate over time. These jobs are reflected in the monitor dashboard’s “Aqueduct queue depth” graph as an increase in resource_activity. In some cases, a buildup of unprocessed jobs can result in memory exhaustion. If you observe memory exhaustion on your instance and see a high metric for resource_activity, contact GitHub Support. [Updated: 2023-10-10]

• When an administrator uses the -p flag with the ghe-support-bundle utility to collect data for a specific number of hours, the utility erroneously collects more logs than necessary. [Updated: 2023-10-13]

• The settings for enabling scheduled reminders were added unintentionally to this release. Scheduled reminders are not officially supported. [Updated: 2023-10-17]

Enterprise Server 3.9.1Download GitHub Enterprise Server 3.9.1

July 18, 2023

📣 This is not the latest patch release of this release series, and this is not the latest release of Enterprise Server. Please use the latest release for the latest security, performance, and bug fixes.

Warning: A change to MySQL in GitHub Enterprise Server 3.9 and later may impact the performance of your instance. Before you upgrade, make sure you’ve read the “Known issues” section of these release notes.

The known issues originally published on 2023-07-18 omitted a number of known issues that still existed. The Known issues section below was updated on 2023-08-08.

3.9.1: Security fixes

• MEDIUM: An attacker with write access to a repository could craft a pull request that would hide commits made in its source branch. This vulnerability was reported via the GitHub Bug Bounty Program and has been assigned CVE-2023-23764. [Updated: 2023-07-26]

• An attacker with access to the password hash of the root site administrator user for the instance’s Management Console could make requests to the password API endpoint from outside of the instance.

• Packages have been updated to the latest security versions.

• LOW: An incorrect comparison vulnerability was identified in GitHub Enterprise Server that allowed commit smuggling by displaying an incorrect diff in a re-opened Pull Request. To exploit this vulnerability, an attacker would need write access to the repository. This vulnerability was reported via the GitHub Bug Bounty Program and was assigned CVE-2023-23765.

3.9.1: Bug fixes

• If MinIO was configured for external blob storage on an instance with GitHub Actions enabled and MinIO was configured for bucket replication, the instance’s credential validation with MinIO would occasionally fail.

• Customers who use Azure Blob store as the remote blob provider to back GitHub Packages would have validation errors if the EndpointSuffix part of their Connection string was anything other than core.windows.net. Now all valid EndpointSuffix are accepted.

• When a user viewed a Jupyter notebook, GitHub Enterprise Server returned a 500 error code if the instance was configured with a self-signed TLS certificate.

• After creation of a blob object from the web UI, pre-receive hook events were missing from the instance’s audit log.

• On an instance with an outbound web proxy server configured, the proxy interfered with internal operations that used nomad alloc exec.

• On an instance in a cluster configuration, the ghe-cluster-balance behaved inconsistently when displaying status or managing jobs with more than one task group.

• .topojson files would not render correctly, but files that conformed to the TopoJSON spec that used a .geojson extension would render correctly.

• On an instance configured for LDAP authentication, if the LDAP server sent an empty string for the sshPublicKey attribute, LDAP user sync would fail.

• REST API endpoints for managing GitHub Enterprise Server are now functional. For more information, see “Manage GitHub Enterprise Server” in the REST API documentation.

• After creation of a new Management Console user, the Management Console did not display the button to copy the new users invitation.

• On an instance with Dependabot enabled, in some situations, Dependabot alerts were not updated when a user pushed to a repository.

• In some cases, pull requests with more than 25 rich-diff renderable files required that users toggle the diff type to correctly render the files over the 25-file limit.

• In rare circumstances, Git commits signed with SSH keys using the RSA algorithm would incorrectly indicate the signature was invalid.

• After a migration using GitHub Enterprise Importer, some repository autolink references were created with an incorrect format.

• In some cases on an instance without a GitHub Advanced Security license, Redis exceeded the maximum default memory allocation, causing 500 errors for the instance’s users.

• On an instance with many organizations, the enterprise security overview page returned a 500 error.

• On an instance that was not configured to deliver email notifications using SMTP, background jobs to deliver email were enqueued unnecessarily.

• Users were unable to configure a SSH certificate authority for an organization.

• An erroneous “Blocked Copilot Repositories” link was visible in site admin pages for organizations.

• On an instance with GitHub Actions enabled and a GitHub Advanced Security license, repository-level runner scale sets were not accounted for when determining whether default setup for code scanning could be used.

• Events related to repository notifications did not appear in the audit log.

• On an instance with a GitHub Advanced Security license and secret scanning enabled, in some cases, a committer would not receive an email notification for a secret scanning alert where push protections were bypassed.

• On an instance with a GitHub Advanced Security license, if a user filtered by a custom pattern on an organizations “Code & security analysis” page using an invalid query, the entire GitHub Advanced Security disappeared and an error reading “Sorry, something went wrong loading GitHub Advanced Security settings” appeared.

• On an instance with a GitHub Advanced Security license, if a user browsed to the alerts page for secret scanning without signing in, the instance responded with a 500 error.

• On an instance with a GitHub Advanced Security license and secret scanning enabled, output from Git for a push blocked by push protection always included an http:// link.

• On an instance with GitHub Actions enabled, links to http(s)://HOSTNAME/features/actions from the web UI returned a 500 error.

• If a user added a new item to a projects roadmap view, and the item was outside of the viewport, the view would crash and display "This project failed to load".

• The audit log reported the incorrect target repository for pre-receive hook failures.

• Users can add issues and pull requests from any organization to a project, and are no longer limited to the user or organization of the project.

• On an instance with GitHub Actions enabled and a GitHub Advanced Security license, enterprise-level runner scale sets with the code-scanning label were not sufficient to allow default setup for code scanning.

• On an instance in a high availability configuration, existing nodes with out-of-sync repositories prevented new nodes from replicating those repositories.

• On an instance with multiple nodes, ERROR-level “resolver failed” errors no longer appear in system logs when the instance is unable to resolve an offline fileserver. The messages are now DEBUG-level.

• On an instance with a GitHub Advanced Security license that was also configured for a timezone greater than UTC, the list of secret scanning alerts displayed a “Loading secrets failed” error if a user sorted secrets by date in descending order.

• Code Scanning workflow runs now only request the code-scanning label so that they can be used with runner scale sets.

3.9.1: Changes

• On an instance in a cluster configuration, the ghe-cluster-config-check command-line utility will return an affirmative message when no warnings or errors are detected. The affirmative message is “Configuration validation complete. No errors found.”

• During initialization of a cluster configuration, output from the ghe-cluster-config-init command-line utility is improved and simplified.

• The API endpoint for management of the GitHub Enterprise Server instance was unavailable prior to initial configuration of the instance.

• The Management Console displays a warning about unexpected consequences that may result from modification of the instance’s hostname after initial configuration.

• On an instance with multiple nodes, internal tooling to repair repositories now attempts to resolve problems within the entire repository network.

• To supplement a disaster recovery plan for a GitHub Enterprise Server instance in a cluster configuration, an administrator can configure a replica of an entire cluster in a separate datacenter, allowing the cluster to fail over to redundant nodes. For more information, see “Configuring high availability replication for a cluster.”

3.9.1: Known issues

• After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9 or 3.10, I/O utilization will increase, and in some cases the instance’s performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see “Known issues with upgrades to your instance.”

• After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see “Known issues with upgrades to your instance.” [Updated: 2023-08-11]

• The Management Console may get stuck in a loop showing “Checking requirements…” when attempting to download an update. It is safe to proceed the update process manually via the command line.

• When enabling CodeQL via default setup at scale, some checks related to GitHub Actions are omitted, potentially preventing the process from completing.

• After restoration of a backup created using GitHub Enterprise Server Backup Utilities 3.7.0, 3.8.0, or 3.9.0, users may not be able to sign into the instance. To fix this issue, plus a bug that was preventing secret scanning encryption keys from being backed up, upgrade your backup host to use GitHub Enterprise Server Backup Utilities 3.9.1 and generate a new full backup using ghe-backup. For more information on using an existing backup, see “Known issues with backups for your instance.” [Updated: 2023-07-31]

• On an instance in a cluster configuration, after you upgrade nodes other than the primary MySQL node and before you upgrade the primary MySQL node, the following output may appear multiple times after you run ghe-config-apply.

  Error response from daemon: conflict: unable to delete IMAGE_ID (cannot be forced) - image is being used by running container CONTAINER_ID
  

  You can safely ignore this message.

• Custom firewall rules are removed during the upgrade process.

• The GitHub Packages npm registry no longer returns a time value in metadata responses. This was done to allow for substantial performance improvements. We continue to have all the data necessary to return a time value as part of the metadata response and will resume returning this value in the future once we have solved the existing performance issues.

• On an instance in a high-availability configuration, passive replica nodes accept Git client requests and forward the requests to the primary node.

• The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn’t required, and outputs an error instead of a warning.

• When using an outbound web proxy server, the ghe-btop command may fail in some circumstances with the error "Error querying allocation: Unexpected response code: 401".

• If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

• When running ghe-config-apply, the process may stall with the message Deployment is running pending automatic promotion.

• During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

• If the root site administrator is locked out of the Management Console after failed login attempts, the account will not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see “Troubleshooting access to the Management Console.”

• On an instance with subdomain isolation disabled, Mermaid diagrams in the web UI display an “Unable to render rich display” error and fail to render. [Updated: 2023-08-18]

• The “Migrations” section is missing from the Management Console, so it isn’t possible to enable, disable, or reconfigure blob storage credentials for migrations. [Updated: 2023-08-18]

• In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support. [Updated: 2023-08-24]

• On an instance with GitHub Actions enabled, if shared runner groups are configured for the enterprise, the enterprise security overview page may return a 500 error. You can avoid the issue by trying one of the following workarounds.

  • Add a runner scale set to the enterprise runner group shared with the repositories.
  • Remove access to the enterprise runner group from the affected repositories or organizations.

  [Updated: 2023-09-05]

• On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes. [Updated: 2023-09-21]

• On an instance with GitHub Actions enabled, ephemeral self-hosted runners do not automatically update to the latest version. Users will need to manually update the runners to the latest version. [Updated: 2023-09-29]

• Jobs in a deprecated queue are not processed and may accumulate over time. These jobs are reflected in the monitor dashboard’s “Aqueduct queue depth” graph as an increase in resource_activity. In some cases, a buildup of unprocessed jobs can result in memory exhaustion. If you observe memory exhaustion on your instance and see a high metric for resource_activity, contact GitHub Support. [Updated: 2023-10-10]

• When an administrator uses the -p flag with the ghe-support-bundle utility to collect data for a specific number of hours, the utility erroneously collects more logs than necessary. [Updated: 2023-10-13]

• The settings for enabling scheduled reminders were added unintentionally to this release. Scheduled reminders are not officially supported. [Updated: 2023-10-17]

Enterprise Server 3.9.0Download GitHub Enterprise Server 3.9.0

June 08, 2023

📣 This is not the latest patch release of this release series, and this is not the latest release of Enterprise Server. Please use the latest release for the latest security, performance, and bug fixes.

For upgrade instructions, see “Upgrading GitHub Enterprise Server.”

Warning: A change to MySQL in GitHub Enterprise Server 3.9 and later may impact the performance of your instance. Before you upgrade, make sure you’ve read the “Known issues” section of these release notes.

3.9.0: Features

• Instance administration

  • To improve security posture and protect data from threats, enterprise owners can see user activity from the Management Console within the enterprise audit log, including events from the UI, API, and administrative SSH access. For more information, see “Audit log events for your enterprise.”

  • During an upgrade of an instance to a new release, people with administrative SSH access to the instance can monitor the progress of routine migrations using the ghe-migrations utility. For more information, see “Command-line utilities.”

  • On an instance with multiple nodes, site administrators can use the Manage GitHub Enterprise Server API to monitor the health of replication. For more information, see “Monitoring a high-availability configuration.”

  • On an instance in a cluster configuration, administrators can ensure a balanced distribution of jobs across nodes by using the ghe-cluster-rebalance utility. For more information, see “Rebalancing cluster workloads.”

  • On an instance in a cluster configuration, administrators can proactively monitor the health of individual nodes and control the reintroduction of unhealthy nodes into the cluster using Node Eligibility Service. For more information, see “Monitoring the health of your cluster nodes with Node Eligibility Service.”

• Identity and access management

  • On an instance configured for SAML SSO, enterprise owners can review information about the Identity Provider (IdP) configured for user authentication using the GraphQL API. The personal access token (PAT) used to authenticate requests to this API requires the read:enterprise scope. Previously, the PAT required the admin:enterprise scope. For more information, see “Objects” in the GraphQL API documentation.

• Authentication

  • For an instance or organization with 2FA enabled, users can configure a 2FA method to be a preferred method. Users can also update 2FA methods from http(s)://HOSTNAME/settings/security. For more information, see “Configuring two-factor authentication” and “Changing your preferred two-factor authentication method.”

• REST API

  • To provide API integrators a smooth migration path and time to update integrations after GitHub makes occasional breaking changes, the REST API now uses calendar-based versioning. GitHub Enterprise Server 3.9 provides version 2022-11-28 of the REST API. For more information, see “API Versions” in the REST API documentation.

• GitHub Connect

  • Enterprise owners who configure Server Statistics on an instance with GitHub Actions enabled will transmit usage metrics related to GitHub Actions. For more information, see “About Server Statistics.”

• GitHub Advanced Security

  • To more easily discover potential security or quality issues in code, users can configure code scanning directly through the web interface without adding a GitHub Actions workflow to the repository. This feature finds and sets up the best CodeQL configuration for the repository, detecting supported languages and enabling CodeQL analysis for every pull request and every push to the default branch and any protected branches. Analysis of JavaScript (including TypeScript), Python, and Ruby code, are currently supported. For more information, see “Configuring default setup for code scanning.”

  • To simplify the configuration of code scanning, organization owners can enable code scanning for all eligible repositories in an organization using a default configuration, either via the web interface or REST API. For more information, see “Configuring default setup for code scanning at scale” and “Organizations” in the REST API documentation.

  • To ensure that relevant alerts remain visible and actionable, users can manually remove stale alerts from code scanning. For more information, see “Managing code scanning alerts for your repository.”

  • To better understand the status of CodeQL and other code scanning tools for a repository, and to help troubleshoot, users can review the tool status page. For more information, see “About the tool status page for code scanning.”

  • To customize the behavior of code scanning on a per-repository basis, repository administrators can configure what severity levels for code scanning alerts will cause checks in a pull request to fail. For more information, see “Triaging code scanning alerts in pull requests.”

  • To protect repositories from pushes that contain custom secret scanning patterns defined at the enterprise, organization, or repository level, users can enable push protection for those patterns. For more information, see “Defining custom patterns for secret scanning.”

  • Organization owners can view the enablement status of security features for the organization’s repositories using the REST API. The endpoint provides details for GitHub Advanced Security, secret scanning, and push protection. For more information, see “Repositories” in the REST API documentation.

  • Repository administrators can programmatically enable code scanning with a default CodeQL configuration using the REST API. For more information, see the following documentation.

    • “Configuring default setup for code scanning”
    • “Get the code scanning default setup configuration” in the Code Scanning REST API documentation
    • “Update the code scanning default setup configuration” in the Code Scanning REST API documentation

• Dependabot

  • To improve the security of GitHub Actions workflows that pin references, Dependabot can update the versioning for calls to reusable workflows within workflow files. For more information, see “About Dependabot version updates.”

  • On an instance with GitHub Actions and the dependency graph enabled, as well as automatic access to GitHub.com actions using GitHub Connect, the web interface will suggest submission actions within a repository with supported languages. For more information, see the following documentation.

    • “Using the Dependency submission API”
    • “About GitHub Actions for enterprises”
    • “Enabling automatic access to GitHub.com actions using GitHub Connect”

    For repositories that use a language that has a submission action, when users with write access visit their dependency graph (this page), we will show them a prompt that directs them to the Marketplace to find an action that would help them.

  • To improve the security of projects that use npm v9, the dependency graph and Dependabot can parse and update package-lock.json files that specify lockfileVersion: 3. For more information, see “About the dependency graph,” “About Dependabot version updates,” and lockfileVersion in the npm documentation.

  • To improve the security of Gradle projects, the dependency graph and Dependabot can parse and update Gradle version catalogs in settings.gradle. For more information, see “About Dependabot version updates” and Sharing dependency versions between projects in the Gradle User Manual.

  • To ensure that users receive the most relevant and actionable alerts about dependency updates, repository administrators and organization owners can enable or disable Dependabot alerts for an individual repository or organization. For more information, see “Securing your repository” or “Securing your organization.”

  • If people with access to a repository do not interact with Dependabot security updates for over 90 days, Dependabot will pause automated pull request activity. For more information, see “About Dependabot security updates.”

  • To help users evaluate the stability risk of a dependency update, Dependabot can fetch release notes, changelogs, and commit history in pull requests that update Docker dependencies. For more information, see “Configuration options for the dependabot.yml file.”

  • To assist with software security and supply chain risk management, people with read access to a repository can export a software bill of materials (SBOM) for a repository’s dependency graph using the web interface or REST API. The SBOM adheres to the SPDX 2.3 specification. For more information, see “Using the Dependency submission API,” “Exporting a software bill of materials for your repository,” and The Software Package Data Exchange® (SPDX®) Specification Version 2.3 on the SPDX website.

  • The dependency graph can parse Python dependencies for pyproject.toml files that follow the PEP 621 standard. For more information, see “About the dependency graph” and PEP 621 – Storing project metadata in pyproject.toml in the Index of Python Enhancement Proposals.

  • Users can use the GraphQL API to review dependencies submitted using the Dependency submission API. For more information, see “Schema previews.”

• GitHub Actions

  • On instances in a cluster configuration, GitHub Actions is available as a private beta. Beta features are subject to change. For more information, and to enroll in the beta, contact your representative on GitHub’s Sales team.

  • Administrators of self-hosted runners for GitHub Actions can configure auto-scaling runners using Actions Runner Controller and runner scale sets. For more information, see “About Actions Runner Controller.”

  • Administrators can bypass all protection rules for a given environment and force the pending jobs referencing the environment to proceed. For more information, see “Using environments for deployment.”

  • Users who deploy with OIDC can define more advanced access policies by including additional custom claims within a token. To help uniquely verify the source of a workflow job, include the following claims.

    • actor_id
    • repository_id
    • repository_owner_id
    • workflow_ref
    • workflow_sha
    • job_workflow_sha

    For more information, see Security hardening your deployments.

  • To improve security for workflows that use GITHUB_TOKEN, the following defaults apply to new organizations and repositories.

    • New organizations that users create inherit permissions from the instance’s enterprise-level configuration. For more information, see “Disabling or limiting GitHub Actions for your organization.”
    • New repositories that users create within an organization inherit permissions from the organization. For more information, see “Managing GitHub Actions settings for a repository.”
    • New user-owned repositories have a read-only GITHUB_TOKEN. For more information, see “Managing GitHub Actions settings for a repository.”

  • To allow workflow authors to pin a required workflow file to a fully validated version, required workflows can be referenced using any branch, tag, or commit SHA from the repository containing the workflow file. For more information, see “Disabling or limiting GitHub Actions for your organization.”

  • To enforce required workflows throughout an organization, GitHub Enterprise Server blocks direct pushes to branches where required workflows are enforced. To allow direct pushes for a particular repository, remove the repository as a target for the required workflow. For more information, see “Disabling or limiting GitHub Actions for your organization.”

  • To improve performance for workflows that build Go, caching is enabled by default when using the setup-go action. For more information, see “Building and testing Go.”

• Organizations

  • Organization owners can improve security posture and protect data from threats by enabling the display of organization members’ IP addresses in audit log events. This feature is in beta and is subject to change. For more information, see “Displaying IP addresses in the audit log for your organization.”

  • To allow the management of branch protection rules without granting admin access, organization owners can create a custom role with the “Edit repository rules” permission. For more information, see “Managing custom repository roles for an organization.”

  • Users of the REST API can programmatically create and update least-privilege roles for repositories using the Custom Repository Roles REST API. The API is generally available, with a breaking change to the API’s endpoint paths. Previously, the API was accessible at /orgs/{org}/custom_roles, and is now accessible at /orgs/{org}/custom-repository-roles. The List custom repository roles in an organization will no longer be available in the next version of the REST API. For more information, see “About custom repository roles” and “Custom Repository Roles” in the REST API documentation.

  • Enterprise and organization owners can delete an organization and all of the organization’s repositories using the REST API. After deletion, organization names are locked for 90 days. For more information, see “Organizations” in the REST API documentation.

• Repositories

  • Within the “Insights” tab for a repository, the sidebar’s “Forks” tab provides more information about a project’s forks, including a sortable and filterable list of forks and more details about each fork.

  • Repository administrators can unarchive a repository using the REST API. For more information, see “Repositories” in the REST API documentation.

• Projects

  • To visualize a project at a high level and across a configurable timespan, users can apply a roadmap layout to any project view. For more information, see “Changing the layout of a view.”

  • To get started with a new project faster, users can copy an existing project, including the source project’s views, custom fields, and draft issues. For more information, see “Copying an existing project.”

  • To save time when adding items to a project, users can configure a workflow to automatically add new items from a repository as people create or update items that match specific criteria. For more information, see “Adding items automatically.”

  • To keep a long-lived project focused, users can define filters to automatically archive items. For more information, see “Archiving items automatically.”

  • To easily organize items within a project’s columns while using the board layout, users can sort the project by field values using the view configuration menu. For more information, see “Customizing the board layout.”

  • To quickly add a new issue to a project without changing context, users can create a new issue from a project’s omnibar by clicking +, then clicking Create new issue. For more information, see “Adding items to your project.”

  • To help people scan a project and take action, users can add a color and a text description to each value for a project’s single select fields. For more information, see “About single select fields.”

  • Users of the GitHub CLI can manage projects from the command line. For more information, see “About GitHub CLI” and the README for the github/gh-projects repository on GitHub.com.

  • For users who programmatically access projects using the GraphQL API, additional mutations are available. For more information, see “createProjectV2Field,” “deleteProjectV2Field,” and “deleteProjectV2” in the “Mutations” GraphQL documentation.

• GitHub Discussions

  • To indicate that a discussion is resolved, outdated, or a duplicate, users can close the discussion. For more information, see “Managing discussions.”

  • To encourage other users to include specific, structured information in discussions, users can create discussion category forms. For more information, see “Creating discussion category forms.”

  • After a user locks a discussion and disallows further comments, the user can permit emoji reactions on the discussion. For more information, see “Moderating discussions.”

• Pull requests

  • To provide feedback on an entire file, or a file that’s been deleted, users can comment on a file from a pull request’s “Files changed” tab. For more information, see “Commenting on a pull request.”

  • Users of the GraphQL API can revert a merged pull request by using the revertPullRequest mutation. For more information, see “Reverting a pull request” and “Mutations” in the GraphQL API documentation.

3.9.0: Changes

• Field names and destinations for some service logs on GitHub Enterprise Server have changed. If any tooling or processes in your environment rely on specific field names within logs, or log entries in specific files, the following changes may affect you.

  • level is now SeverityText.
  • log_message, msg, or message is now Body.
  • now is now Timestamp.
  • Custom field names such as gh.repo.id or graphql.operation.name use semantic names.
  • Log statements that the instance would previously write to auth.log, ldap.log, or ldap-sync.log now appear in containerized logs for github-unicorn if the statement originated from a web request, or in logs for github-resqued if the statement originated from a background job. For more information about containerized logs, see “About system logs.”

  For a full list of field mappings, download the OpenTelemetry attribute mapping CSV. This change is part of GitHub’s gradual migration to internal semantic conventions for OpenTelemetry, and additional field names will change in upcoming releases.

• On a configured instance, the name for the HAProxy service is now haproxy-frontend. Previously, the name was haproxy. Additionally, on an unconfigured instance, there is a new service named haproxy-pre-config. If your instance forwards logs to an external system, update your rules to reflect these changes. For more information, see “Log forwarding” article

• For an instance or organization with 2FA enabled, when a user sets up 2FA, GitHub Enterprise Server suggests an authenticator app (TOTP) by default.

• When a person with administrative SSH access to an instance submits a support bundle using either the ghe-support-bundle or ghe-cluster-support-bundle utility, a period for log collection specified with the -p or --period no longer requires quotes to enclose the date value. For more information, see “Command-line utilities.”

• To provide additional context within the web interface on an instance where Dependabot alerts are enabled, links to Dependabot alerts in an issue or pull request comment display an improved label and hovercard with alert details.

• On an instance with Dependabot alerts enabled, people with write or maintain access to a repository can view or act on Dependabot alerts by default. Custom roles, the security manager role, organization permissions, and notification settings are not affected.

• On an instance with a GitHub Advanced Security license and GitHub Connect enabled for the synchronization of actions from GitHub.com, CodeQL code scanning is up to 16% faster. For more information, see “Configuring code scanning for your appliance.”

• On an instance with a GitHub Advanced Security license and email configured for notifications, users can receive notifications for secret scanning alerts by watching a repository and choosing “All activity” or "Security alerts". To continue receiving notifications for secret scanning alerts in GitHub Enterprise Server 3.9 and later, users must enable email notifications in the web interface at http(s)://HOSTNAME/settings/notifications under “Watching” by choosing "Email".

• On an instance with a GitHub Advanced Security license, secret scanning alerts display whether detected tokens from GitHub are valid.

• On an instance with a GitHub Advanced Security license, the enterprise and organization audit logs now display an event when an owner enables or disables a push protection for a custom pattern for a repository, organization, or the enterprise. For more information, see “Reviewing the audit log for your organization” and “Audit log events for your enterprise.”

• Users can filter the lists of alerts for Dependabot, code scanning, and secret scanning by repository topic or team in the security overview for an organization. For more information, see “Filtering alerts in security overview.”

• In the security overview for an organization, the following improvements apply to the “Security coverage” view during feature enablement.

  • To provide insight into the number of GitHub Advanced Security licenses used, active committers for the repository are visible. For repositories where GitHub Advanced Security is not enabled, the number indicates the number of licenses required to enable the feature.
  • Unsaved changes are now labeled with a “Modified” tag, and the “Save security settings” button now displays the total number of changes to save.
  • While a security feature is being enabled, the “Security coverage” view shows a status of “Updating…” to inform you of the ongoing process.

  For more information, see “About security overview.”

• In the security overview’s “Security risk” and “Security coverage” views, when a user selects a team from the “Team” drop-down or filters by team, results appear for repositories where the team has write or administrative access or has been granted access to security alerts. Previously, users could only view results for repositories where the team had administrative access or had been granted access to security alerts.

• To provide more context within a project, users can share a deep link to a specific issue in a project to have the issue open in the project’s side panel.

• Organization owners can create up to five custom repository roles. Previously, the limit was three. For more information, see “About custom repository roles.”

• When transferring a repository, users can also rename the repository. For more information, see “Transferring a repository.”

• If a user archives a repository, responses from the GraphQL API that include information about the repository now include an archivedAt value with a timestamp representing the archival date.

3.9.0: Backups

• Before beginning a backup with GitHub Enterprise Server Backup Utilities 3.9.0 and later, the ghe-host-check utility will now perform a preflight check on the backup host to confirm the software version and disk space requirements. For more information, see the 3.9.0 release in the github/backup-utils repository on GitHub.com.

• GitHub Enterprise Server Backup Utilities 3.9.0 allows administrators to view the progress of backup and restoration operations on the backup host using the ghe-backup-progress utility. For more information, see “Configuring backups on your instance.”

3.9.0: Known issues

• After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9 or 3.10, I/O utilization will increase, and in some cases the instance’s performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see “Known issues with upgrades to your instance.”

• After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see “Known issues with upgrades to your instance.” [Updated: 2023-08-11]

• The Management Console may get stuck in a loop showing “Checking requirements…” when attempting to download an update. It is safe to proceed the update process manually via the command line.

• After restoration of a backup created using GitHub Enterprise Server Backup Utilities 3.7.0, 3.8.0, or 3.9.0, users may not be able to sign into the instance. To fix this issue, plus a bug that was preventing secret scanning encryption keys from being backed up, upgrade your backup host to use GitHub Enterprise Server Backup Utilities 3.9.1 and generate a new full backup using ghe-backup. For more information on using an existing backup, see “Known issues with backups for your instance.” [Updated: 2023-07-31]

• After upgrading an existing instance to GitHub Enterprise Server 3.9, the Manage GitHub Enterprise Server API is unavailable. To enable the API, SSH into the instance and run the following commands.

  Shell

  sudo mkdir -p /data/ghes-manage-gateway/current
  sudo chown -R ghes-manage-gateway:ghes-manage-gateway /data/ghes-manage-gateway/current
  sudo systemctl restart ghes-manage-gateway ghes-manage-gateway-consul
  

  For more information about the Manage GitHub Enterprise Server API, see “Manage GitHub Enterprise Server” in the REST API documentation. [Updated: 2023-06-22]

• On an instance in a cluster configuration, after you upgrade nodes other than the primary MySQL node and before you upgrade the primary MySQL node, the following output may appear multiple times after you run ghe-config-apply.

  Error response from daemon: conflict: unable to delete IMAGE_ID (cannot be forced) - image is being used by running container CONTAINER_ID
  

  You can safely ignore this message.

• Custom firewall rules are removed during the upgrade process.

• The GitHub Packages npm registry no longer returns a time value in metadata responses. This was done to allow for substantial performance improvements. We continue to have all the data necessary to return a time value as part of the metadata response and will resume returning this value in the future once we have solved the existing performance issues.

• On an instance in a high-availability configuration, passive replica nodes accept Git client requests and forward the requests to the primary node.

• The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn’t required, and outputs an error instead of a warning.

• When using an outbound web proxy server, the ghe-btop command may fail in some circumstances with the error "Error querying allocation: Unexpected response code: 401".

• If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

• When running ghe-config-apply, the process may stall with the message Deployment is running pending automatic promotion.

• During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

• If the root site administrator is locked out of the Management Console after failed login attempts, the account will not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see “Troubleshooting access to the Management Console.”

• On an instance with subdomain isolation disabled, Mermaid diagrams in the web UI display an “Unable to render rich display” error and fail to render. [Updated: 2023-08-18]

• In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support. [Updated: 2023-08-24]

• On an instance with GitHub Actions enabled, if shared runner groups are configured for the enterprise, the enterprise security overview page may return a 500 error. You can avoid the issue by trying one of the following workarounds.

  • Add a runner scale set to the enterprise runner group shared with the repositories.
  • Remove access to the enterprise runner group from the affected repositories or organizations.

  [Updated: 2023-09-05]

• On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes. [Updated: 2023-09-21]

• On an instance with GitHub Actions enabled, ephemeral self-hosted runners do not automatically update to the latest version. Users will need to manually update the runners to the latest version. [Updated: 2023-09-29]

• Jobs in a deprecated queue are not processed and may accumulate over time. These jobs are reflected in the monitor dashboard’s “Aqueduct queue depth” graph as an increase in resource_activity. In some cases, a buildup of unprocessed jobs can result in memory exhaustion. If you observe memory exhaustion on your instance and see a high metric for resource_activity, contact GitHub Support. [Updated: 2023-10-10]

• When an administrator uses the -p flag with the ghe-support-bundle utility to collect data for a specific number of hours, the utility erroneously collects more logs than necessary. [Updated: 2023-10-13]

• The settings for enabling scheduled reminders were added unintentionally to this release. Scheduled reminders are not officially supported. [Updated: 2023-10-17]

3.9.0: Deprecations

• Change to command-line utility for management of replication

  • On an instance with multiple nodes, people with administrative SSH access to the instance should use ghe-spokesctl for management of Git replication instead of ghe-spokes. For more information, see “Command-line utilities.”

• Dependency graph no longer ingests go.sum files

  • Because go.sum files are not lock files and may result in false positive Dependabot alerts, on an instance with the dependency graph enabled, the go.sum files are no longer ingested for users’ Go repositories. If Dependabot alerts are enabled, Dependabot will no longer alert users for vulnerabilities in a go.sum file’s dependencies. The dependency graph continues to support go.mod files, the recommended format for Go projects. Use Go 1.17 or higher to ensure your go.mod file contains a comprehensive view of all direct and transitive dependencies. For more information, see “About the dependency graph.”

• Only GitHub Actions can publish a GitHub Pages site if source includes symbolic links

  • To improve the security of an instance where users deploy sites using GitHub Pages, sites that contain symbolic links will no longer build outside of GitHub Actions. If a user’s site is affected and a site administrator has configured email for the instance, the user will receive an email with instructions about how to fix the error. To continue using symbolic links in the site’s source, the instance must be configured for GitHub Actions, and the user must write a GitHub Actions workflow to use as a publishing source. For more information, see “About GitHub Pages.”