Fireworks AI Breach Exposes Customer API Keys via Internal Credentials
Fireworks AI Discloses Breach of Customer Environment Variables
Related Entities
Other
Full Report
Executive Summary
Fireworks AI, a platform for developers using large language models (LLMs), has reported a security incident involving the compromise of internal company credentials. An unauthorized third party leveraged this access to view environment variables for a limited number of customer accounts. These variables contained sensitive secrets, including customer API keys for third-party services. The company detected the intrusion on October 6, 2026, began notifying affected customers on October 7, and has since taken steps to revoke compromised keys and rotate internal credentials. According to Fireworks AI, its investigation has found no evidence that customer models, data, or LLM prompts and outputs were accessed.
Threat Overview
The incident highlights a classic cloud security risk: the improper handling and exposure of secrets. The attacker, having obtained internal Fireworks AI credentials, was able to access a system that stored or had access to environment variables for customer evaluation jobs. This gave them a direct line to the secrets that customers had configured for use within the Fireworks platform.
Fireworks AI's response was swift upon detection. They initiated containment on the same day (October 6), began notifying the specific customers impacted the next day, and worked with an external cybersecurity firm to investigate. Subsequent actions included rotating all internal employee API keys and revoking user API keys and service-account tokens that were identified as potentially exposed.
Technical Analysis
The core of this incident is a failure in credential and secrets management. The attacker's initial vector for obtaining the internal credentials was not disclosed, but once obtained, they targeted a high-value resource: the environment variables used by customer workloads.
MITRE ATT&CK Techniques
T1078 - Valid Accounts: The attacker used legitimate but compromised internal Fireworks AI credentials to access the system. The initial vector for this compromise is unknown.T1552.005 - Cloud Instance Metadata API: (Assessed) A common way to access environment variables and secrets in cloud environments is by querying the instance metadata service. The attacker may have used their access to query this service from a compromised host.T1528 - Steal Application Access Token: The ultimate goal and result of the attack was the theft of customer API keys and other secrets stored as environment variables.T1098.004 - SSH Authorized Keys: (Assessed) The attacker may have used their initial access to plant SSH keys or other persistence mechanisms, although this was not specified in the report.
Impact Assessment
For the affected Fireworks AI customers, the primary impact is the exposure of their third-party credentials. If an API key for a service like OpenAI, AWS, or a database was exposed, the attacker could potentially use that key to access the customer's resources on that third-party platform, leading to data breaches or financial loss. Fireworks AI's quick notification was crucial in allowing customers to rotate these keys. The company's "Zero Data Retention" (ZDR) policy was a significant mitigating factor, as it meant that sensitive LLM prompts and outputs were not stored and therefore could not be stolen. The incident still causes reputational damage and highlights the immense trust customers place in platform providers to secure their secrets.
IOCs — Directly from Articles
No specific IOCs were provided in the source articles.
Cyber Observables — Hunting Hints
To detect similar incidents in a cloud platform environment, security teams should monitor for:
api_endpointGetSecretValue or GetParameterslog_sourceInstance Metadata Service Logsuser_account_patternInternal credentials used from external IPcommand_line_patternprintenv or envDetection & Response
- Secrets Detection: Continuously scan code repositories and running environments for hardcoded secrets. Any discovered secrets should be immediately revoked and replaced with a reference to a secrets manager.
- Audit Log Analysis: Ingest and analyze cloud audit logs (like AWS CloudTrail) to monitor all access to secrets management services. Alert on any unusual patterns, such as a single user accessing a large number of different secrets. This is an application of D3FEND Resource Access Pattern Analysis (D3-RAPA).
- Credential Hygiene: Enforce strict controls on internal credentials, including mandatory MFA, short-lived sessions, and regular rotation.
Mitigation
- Secrets Management as a Service: Avoid storing secrets in environment variables directly. Instead, use a dedicated secrets management service like AWS Secrets Manager, Azure Key Vault, or HashiCorp Vault. Applications should be granted IAM roles that allow them to retrieve secrets at runtime. This prevents secrets from being exposed in plaintext in environment configurations. This is a form of D3FEND Credential Hardening.
- Least Privilege for Workloads: The service or process that runs customer evaluation jobs should have narrowly scoped IAM permissions. It should only be able to access the specific secrets required for that single job, not a wide range of customer secrets. This limits the blast radius if the service is compromised.
- Zero Trust Network Access (ZTNA): Internal systems should not be trusted by default. Access to sensitive internal services that manage customer data or secrets should require strong authentication and authorization for every request, regardless of network location.
- Proactive Key Rotation: For affected customers, the primary mitigation is to immediately rotate any credentials that were stored as environment variables on the Fireworks AI platform. It is critical to assume they have been compromised.
Timeline of Events
MITRE ATT&CK Mitigations
Credential Access Protection
Using a dedicated secrets manager instead of environment variables is a key protection against this type of attack.
Multi-factor Authentication
Enforcing MFA on all internal employee accounts could have prevented the initial compromise of the credentials.
D3FEND Defensive Countermeasures
The root cause of this incident was the exposure of secrets in environment variables. The primary countermeasure is to stop this practice and use a dedicated secrets management system (e.g., AWS Secrets Manager, HashiCorp Vault). Instead of injecting plaintext secrets into environment variables, applications should be granted a temporary, narrowly-scoped IAM role that allows them to retrieve secrets from the vault at runtime. This ensures secrets are encrypted at rest and in transit, are never stored in plaintext on disk or in logs, and access can be centrally audited and revoked. This approach fundamentally hardens the credential lifecycle and prevents the type of exposure seen in the Fireworks AI incident.
Detecting the abuse of the compromised internal credential requires analyzing resource access patterns. A security team should baseline the normal behavior of internal accounts and services. For example, a specific service might only ever access secrets belonging to jobs running in a certain region. An alert should be triggered if that service's credentials are suddenly used to access secrets for jobs in a different region, or if an employee's account, which normally doesn't interact with the production secrets store, suddenly queries it. By analyzing logs from the secrets manager and cloud control plane, deviations from established patterns can be identified as indicators of compromise.
While the incident report doesn't state how the internal credentials were first compromised, enforcing MFA on all internal systems is a foundational defense. This applies not just to human users logging into dashboards, but also to programmatic access where possible. For privileged access to production control planes or systems managing customer secrets, access should require short-lived credentials obtained via an MFA-protected process. This would mean that even if an attacker stole a long-term static credential, it would be useless for accessing the most sensitive parts of the Fireworks AI infrastructure.
Timeline of Events
Fireworks AI identifies unauthorized activity and initiates containment.
Fireworks AI begins notifying affected customers.
The company rotates internal employee API keys.
Fireworks AI revokes additional user API keys and service-account tokens and updates its public bulletin.
Sources & References
Article Author
Jason Gomes
• Cybersecurity PractitionerCybersecurity professional with over 10 years of specialized experience in security operations, threat intelligence, incident response, and security automation. Expertise spans SOAR/XSOAR orchestration, threat intelligence platforms, SIEM/UEBA analytics, and building cyber fusion centers. Background includes technical enablement, solution architecture for enterprise and government clients, and implementing security automation workflows across IR, TIP, and SOC use cases.
Editorial Standards & Analyst Review
CyberNetSec.io uses automation to assist source monitoring, deduplication, observable extraction, and structured intelligence generation. Published analysis follows human-defined editorial standards and adds defensive context including MITRE ATT&CK, D3FEND, STIX, and Sigma where applicable. Read our editorial policy.
Tags
📢 Share This Article
Help others stay informed about cybersecurity threats
🎯 MITRE ATT&CK Mapped
Every tactic, technique, and sub-technique used in this threat has been identified and mapped to the MITRE ATT&CK framework for consistent, actionable threat language.
🧠 Enriched & Analyzed
Observables and indicators of compromise (IOCs) have been extracted and cataloged. Risk has been assessed and correlated with known threat actors and historical campaigns.
🛡️ Actionable Guidance
Detection rules, incident response steps, and D3FEND-aligned mitigation strategies are included so your team can act on this intelligence immediately.
🔗 STIX Visualizer
Structured threat data is packaged as a STIX 2.1 bundle and can be visualized as an interactive graph — relationships between actors, malware, techniques, and indicators.
⚡ Sigma Generator
Sigma detection rules are derived from the threat techniques in this article and can be converted for deployment across any major SIEM or EDR platform.