A very large and top-rated government cybersecurity center, the German Federal Office for Information Security (BSI), has taken the lead in taking steps to fend off another CrowdStrike-style fiasco.
The BSI's actions should have big implications for DevOps, and not just at CrowdStrike and Microsoft.
The BSI has been in talks with both CrowdStrike and Microsoft about their DevOps following the July 19 global IT meltdown. As a result, BSI will be working with both companies to ensure that systems can be started in a minimum safe mode, even if serious errors occur.
BSI's long-term goal is to have new and resilient components designed and implemented offering the same functionality and level of protection as before, but which require less invasive permissions to operating systems. This aims to minimize the impact of software errors.
BSI has been in direct contact with Crowdstrike in Germany and in the USA since the incident on 19 July 2024. Following the immediate measures taken by the software vendor to prevent further incidents and the provision of an initial workaround for the affected systems, preliminary analysis reports on this incident were continually discussed between Crowdstrike and BSI and subsequently published. Based on the discussions, the evaluation of the available analyses and continued feedback from the vendor, BSI has initially developed the following measures:
Short-term measures until 15 August 2024
Impact analysis of the security incident in Germany
Continuous tracking of the recovery rate of affected systems (as of 25 July 2024 21:54 CEST and according to Crowdstrike, 97 percent of all systems with Windows sensors are already back online)
Merging already issued short-term warnings with expected incident-related CVEs based on the established CVD process
Medium-term measures until 30 September 2024
Evaluation of the upcoming detailed and final analysis report (root cause analysis)Review of the current and the improved test concept of Crowdstrike by BSI in coordination with other international partner agencies and discussion of necessary adjustments with Crowdstrike
Clarification of future measures to ensure a rapid rollout of business logic/signatures while strictly guaranteeing the operational stability of customer systems
Testing the effectiveness of the progressive and closely monitored update rollout process to customers as already announced by Crowdstrike with extended telemetry analyses by Crowdstrike for immediate detection of faults after installation of the updates
Raising the awareness of organizations using Crowdstrike products about fundamental operational risks (cf. https://www.crowdstrike.com/terms-and-conditions-de/) and creating sufficient operational redundancies for critical deployment scenarios
Long-term measures until 31 December 2024
Discussion of concrete possibilities for evaluating the vendor's software development processes by independent third parties based on announcements already made by Crowdstrike
Establishing a cooperation between BSI, Crowdstrike, and Microsoft with the objective to ensure booting of the system at least in a restricted mode, even in the event of serious malfunction of the EDR tool
Initial discussions with all relevant stakeholders on the architecture of EDR tools to increase their resilience
Further measures in 2025
Design and implementation of new, more resilient architectures for running EDR tools with the minimum required privileges while maintaining the same functionality and same level of protection
Involving all other software vendors in this product category, all relevant operating system platforms and, in general, providers of products with (currently still) high privileges
BSI is in continued contact with the vendor Crowdstrike and with Microsoft regarding the operational and strategic processing of the security incident expecting concrete results and solutions. In the meantime, Crowdstrike has published a large amount of additional information that already describes the initial implementation of the above measures.
These measures by the BSI are important and necessary. They will enhance DevOps, and should result in improved operability of systems and continuity, for businesses and consumers. They also work toward needed reform and improvement of not just DevOps but governmental guidance. They represent a growing trend toward inter-sector cooperation between state entities and private software companies.
Top comments (3)
Bitte sehr!
Thanks for the translation!
Bitte sehr!