The PyTorch Lightning PyPI supply chain attack that was stopped in 42 minutes

Mr. Latte ∙ Software Development Expert

On 2026-04-30, malicious code was inserted into versions 2.6.2 and 2.6.3 of the lightning package on PyPI. The Lightning AI community took 42 minutes to spot the anomaly and respond. The official blog post leads with the number: “How the PyTorch Lightning Community Discovered a Supply Chain Attack and Fixed it in 42 Minutes.”

Forty-two minutes is fast. Still, any environment that ran pip install lightning in that window, or that pinned a range like lightning>=2.6, may have installed a malicious version. How many training pipelines, CI/CD systems and developer machines were actually affected has not been published.

Why supply chain attacks bite harder in ML

Software supply chain attacks are not new. Taking over a trusted package or distribution path and shipping malicious code to users has recurred across ecosystems.

A few structural traits make ML environments riskier.

1. Dependency versions are managed loosely. requirements.txt files often carry only a range, such as torch>=2.0. Every new environment then pulls whatever is newest on PyPI. Lock files and automated dependency checks are widespread in general application development, but experiment-driven data science environments still apply them inconsistently.

2. GPU nodes drift out of control easily. On-demand GPU instances are usually created by script and then configured with pip install -r requirements.txt. Unless the image and package versions are pinned, each new instance can end up with different dependencies. If the training cluster has open outbound network access, malicious code has an easy path for shipping credentials or data out.

3. Publishing accounts and tokens are attractive targets. The exact compromise path in the Lightning incident has not been confirmed by official investigation, so calling it a PyPI account takeover would be premature. That said, seizing a maintainer account, a long-lived API token or a release CI workflow is an efficient way to tamper with a widely used package.

Since January 2024, PyPI has required two-factor authentication for management actions and uploads. Two-factor authentication alone does not cover a leaked API token or a compromised release workflow. Where possible, Trusted Publishing, which issues short-lived OIDC credentials instead of long-lived tokens, is the safer arrangement.

4. There is a lot for malicious code to reach. An ML training environment concentrates raw data, model weights, cloud credentials, experiment tracking tokens and expensive GPU capacity in one place. Lightning AI’s security advisory states that the malicious versions contained functionality consistent with credential harvesting. Affected environments should not stop at removing the package. Rotate any credentials that may have been exposed and rebuild from a clean state (GitHub security advisory).

Dependency rules for an ML pipeline

Pin versions and hashes together

# risky
lightning>=2.6

# preferred
lightning==2.6.1

Packages used in production belong at exact versions. Use pip-compile from pip-tools, Poetry’s poetry.lock, or conda-lock.yml to lock transitive dependencies too, and commit the result.

Pinning versions is not sufficient on its own. An attacker can replace the distribution files for a given version, and an internal mirror can be poisoned.

pip install --require-hashes -r requirements.txt

pip-compile --generate-hashes produces a SHA-256 hash for each distribution file, so installation verifies file integrity.

Pin container images by digest as well

FROM pytorch/pytorch:2.3.1-cuda12.1-cudnn8-runtime@sha256:...

A latest tag can resolve to different images at different times from the same Dockerfile. Pin base images by digest, and manage the Python packages installed inside the image with lock files and hashes.

When the Docker layer cache hits, the install command does not run again. The exposure is when the cache is invalidated or the image is built in a fresh environment. A loose version specification lets a new package walk in unannounced.

Understand what vulnerability auditing cannot do

Tools like pip-audit are useful for finding dependencies with known vulnerabilities. Wire one into CI and it can fail the build when a vulnerable version appears.

What such tools cannot do is catch a malicious release that is not yet in a vulnerability database. In an incident like Lightning’s, where discovery and response fit inside 42 minutes, pinned versions, approved mirrors, network controls and runtime monitoring are more direct defenses than an audit tool.

Digital attestations under PEP 740 provide a basis for verifying package provenance and the release process. PEP 740 does not require installers to verify attestations for every package. The existence of a signature or an attestation and the enforcement of one in installation policy are separate questions.

Only let approved packages in

An organization of any size is better served by an internal registry such as Artifactory or Nexus than by installing directly from PyPI. New versions go through inspection and approval before they land in the internal repository.

Updates arrive somewhat more slowly, and in exchange a package published to a public repository does not walk straight into the training environment. The part that matters is separating pre-approval from post-approval repositories, not simply running a proxy cache.

Detecting a supply chain attack

What made the 42-minute response work was noticing the new version behaving abnormally. Inside an organization, several detection layers should overlap.

Block or log outbound network traffic during package installation and first execution. Alert when a connection reaches a host you never normally contact.
Issue only the credentials a training job needs, and only briefly. Do not hand over personal developer tokens or broad cloud permissions.
Verify checksums of model weights, datasets and container images before and after a run.
Put new package versions through static and behavioral analysis in an isolated environment. Treat YARA and malware scanning services as supporting measures.
Subscribe to PyPI releases and security advisories, but do not auto-update just because a notification arrived.

What is still unconfirmed

The official advisory states that 2.6.2 and 2.6.3 contained malicious functionality consistent with credential harvesting. The precise entry point of the compromise, its full behavior, actual download counts and the number of affected environments have not been published. The actor has not been identified either.

At this stage it is too early to assert that “a PyPI account was taken over” or that “a particular malware family was used.” As further investigation lands, the compromise path and response timeline should be revised accordingly.

The structural problem is still there

The PyTorch Lightning community responded quickly. Nothing guarantees the next attack is found in 42 minutes. Nightly packages whose contents change by the day, utilities maintained by a handful of people, an absent-minded !pip install inside a notebook, GPU clusters with unrestricted outbound access: all of it still adds up to a wide attack surface.

pip install downloads code and runs it with the current user’s privileges. Being a short and familiar command does not make it any safer.