Who Controls the Download Factory?

From Kicksecure
Jump to navigation Jump to search

Many Open Source users expect official downloads to be built by the project's own developers or infrastructure. For example, many users may expect official Firefox downloads to be built by Mozilla-controlled infrastructure. If official downloads are instead built on infrastructure controlled by companies such as Google, Microsoft, Amazon, GitHub, or similar providers, those providers become part of the trusted build path. This can create build-time backdoor risk.

Simple explanation

[edit]

This section is for non-technical readers. More technical details follow later.

Many users choose Open Source software because the source code can be inspected, shared, modified, and rebuilt.

However, most users do not build software themselves. They download the official program.

The important question is:

Who controlled the computer that created the official downloads?Plain-language build trust question.

Recipe analogy

[edit]

Imagine Open Source software as a recipe.

  • The source code is the recipe that people can read.
  • The official downloads is the finished meal that users actually receive.
  • The build computer is the kitchen that turns the recipe into the finished meal.

In software, this process is called a build.

The concern is simple:

The recipe may be public, but the finished meal still depends on the kitchen.Build trust analogy.

If someone else controls the kitchen, users must trust both the recipe author and that kitchen.

Two build models

[edit]

Project-controlled or developer-controlled build

The Open Source project, its developers, or its own infrastructure control the computers that create the official downloads.

Users still need to trust the project. This does not automatically make the download secure. It only means the project has not handed the build computers themselves to an outside build service.

Third-party-hosted build

The project publishes the source code, but another organization controls the computers that create the official downloads.

Examples include GitHub Actions, Google Cloud, Microsoft Azure, Amazon AWS, or another hosted CI/build provider.

Plain-language comparison
Question Project-controlled or developer-controlled build Third-party-hosted build
Who controls the build computer? The project, its developers, or its own infrastructure. Another company or outside service.
Does the user need to trust an extra build provider? Usually no. Yes.
Does this automatically prove the download is safe or unsafe? No. No.
Why might privacy and security users care? The official build path depends less on outside infrastructure. An outside build provider becomes part of the trusted path for the official downloads.

Why this matters

[edit]

If I choose Open Source to avoid Google, Microsoft, Amazon, or similar companies, the official downloads should not silently depend on those same companies as trusted build authorities.Paraphrased expectation of security and privacy enthusiasts.

That expectation is not part of the formal The Open Source Definition (OSD)archive.org iconarchive.today icon. The Open Source Definition is about source code freedoms. It does not require self-hosted builds, avoidance of proprietary infrastructure or reproducible builds.

Therefore, a project can be Open Source while still using third-party build infrastructure.

Build-time backdoor risk in plain language

[edit]

A build-time backdoor is a malicious change introduced while source code is being turned into an official download.

This is different from a source-code backdoor.

With a source-code backdoor, the malicious code is visible in the public source code.

With a build-time backdoor, the public source code may look clean, but the finished program could theoretically be changed during the build.

A malicious or compromised build environment could theoretically:

  • Inject a backdoor: Add malicious code during the build process that is not visible in the public source code.
  • Modify the application: Produce an official download that differs from what independent users would expect from the source code.
  • Steal signing secrets: Abuse build credentials, release tokens, or signing keys if these are available to the build system.
  • Target only some users: Serve or produce different binaries for specific users, regions, platforms, or release channels.
  • Hide the change: Leave the source code unchanged while modifying only the build output or release artifact.

This is a supply-chain risk.

Who creates official downloads?

[edit]

Comparison of who controls the computer that create official downloads for selected Open Source software projects.

Who creates the files users install?
Project Project-controlled or developer-controlled build? Build provider or location Plain explanation Evidence
Mozilla Firefox No, Third-party-hosted build Google Compute Engine (GCP) Trusted Firefox Linux build workers are configured to run on Google Cloud infrastructure. Mozilla says shippable builds were moved from AWS to GCP. Mozilla issue tracker and Firefox CI configuration provide additional technical evidence.[1]
Iceraven Browserarchive.org iconarchive.today icon No, Third-party-hosted build GitHub Actions, Microsoft Iceraven release automation uses GitHub Actions. GitHub is owned by Microsoft. Iceraven repository release automation statement.[2]
Flathub / Flatpak packages distributed through Flathub No, Third-party-hosted build Amazon AWS Flathub's build orchestrator uses RunsOn. RunsOn runs jobs on AWS. Flathub Vorarbeiter blog and RunsOn documentation. [3] See also chapter Flathub / Flatpak.
Kicksecure Yes, Project-controlled or developer-controlled build Developer machines Images and packages are built on developer machines. Kicksecure Trust wiki page.[4]
Qubes Unknown / needs documentation Unknown / needs documentation The build location needs further research. Related user discussion exists.[5]
Debian Distributed build infrastructure Various individuals and organizations Debian build machines are operated by various organizations and individual supporters. Debian machine database.[6]

Project notes

[edit]

Flathub / Flatpak

[edit]

Time of writing of this Flathub / Flatpak chapter: May 2026

Flathub's current public documentation says its build system uses GitHub Actions together with RunsOn, which provisions build machines on AWS, Amazon's cloud platform.

In simple terms, a "build" means turning source code and other declared inputs into the final Flatpak package that users install. A "reproducible build" means someone can repeat the build and get the same result.

Security and transparency concerns:

  • Amazon-backed build infrastructure: Flathub uses a build system called Vorarbeiterarchive.org iconarchive.today icon. Flathub describes Vorarbeiter as sitting between GitHub and GitHub Actions. Flathub later stated that it fully switched to RunsOnarchive.org iconarchive.today icon for all builds. RunsOn provides GitHub Actions runners using AWS infrastructure. AWS means Amazon Web Servicesarchive.org iconarchive.today icon, Amazon's cloud platform. Therefore, current public Flathub documentation shows that Flathub builds use GitHub Actions, RunsOn, and AWS-backed infrastructure.[7]
  • Build-time threat model: Because the documented build path uses GitHub Actions, RunsOn, and AWS-backed infrastructure, those systems should be treated as part of the trusted build path.
    • Trusted infrastructure path: It means that AWS-backed infrastructure is part of Flathub's documented build path, and that Amazon S3 may also be used for storing build-comparison reports.
    • Build-time attack surface: In the general threat model of this page, a compromised build runner, malicious insider, or legally compelled infrastructure provider could theoretically affect the finished Flatpak even if the public source code looked clean. This is the build-time backdoor concern described earlier on this page.
    • No allegation of actual tampering: This does not mean that Amazon, GitHub, RunsOn, or Flathub have inserted a backdoor, reviewed apps, or intentionally changed packages.
  • Build-from-source transparency: Flathub has a policy saying that source-available submissions should be built entirely from source code, including runtime dependencies included in the manifest, with possible exceptions. [8]
    • No clear per-app source/prebuilt-files indicator: Flathub does not currently prominently show ordinary users whether a specific Flatpak was fully built from source code or whether any already-built upstream files were used during the build process. A feature request to show this on flathub.org was closed as not planned. [9]
  • Reproducible builds: Flathub has reproducible build resultsarchive.org iconarchive.today icon, which is laudable. However, reproducible builds do not remove all trust questions.
    • Limited reproducibility scope: Flathub describes its reproducibility testing as testing x86_64 builds targeting the stable repository. [10]
    • Direct uploads and failures: Flathub states that direct uploads are not currently tested by its reproducibility system and that reproducibility failures are not currently acted on. [11]
    • Source versus packaging distinction: Reproducible packaging is not the same as proving that everything was compiled from original source code. It proves that the same recipe produced the same Flatpak again. If the recipe used a prebuilt upstream program as an input, reproducibility shows that the same program was packaged again in the same way.
    • Reporting-path trust: The reproducibility checker says it rebuilds Flatpak apps published on Flathub and compares the result using diffoscope. diffoscope is a tool that compares two build outputs and explains what differs. The checker documentation also says Amazon S3 upload support is optional and used for uploading diffoscope results. S3 means Amazon S3archive.org iconarchive.today icon, Amazon's cloud object storage service. S3 is storage, not a build machine. This means Amazon infrastructure may be used to store or serve detailed comparison reports. [12]
      • Reporting-path attack surface: A compromised or malicious reporting/storage path could theoretically hide, replace, remove, or misrepresent reproducibility artifacts shown to users.
      • Meaning of this limitation: This does not by itself prove that Amazon determines or manipulates the result; it means the hosted reporting path is another part of the trust model.
    • Hosted result trust: If users only read Flathub's own reproducibility page or Flathub-linked reports, they are trusting Flathub's reporting setup. This is not the same as an independent third-party rebuild. A malicious build path could theoretically affect the produced Flatpak, while a malicious or compromised reporting path could theoretically affect the reproducibility status or comparison artifacts that users see afterward.
    • Independent verification: Independent rebuilders matter because they let people check Flathub's claims without relying only on Flathub's own website. Volunteers can run Flathub's own flathub-repro-checkerarchive.org iconarchive.today icon, and at least one independent third-party tool exists: flathub-rebuilderarchive.org iconarchive.today icon. It is described as a command line tool to verify a Flatpak locally, recreate a Flatpak from Flathub by name, and compare results using diffoscope.
      • No independent public rebuilder dashboard found: No widely used independent public rebuilder dashboard for Flathub was found.
      • Flathub-hosted public results: The public reproducibility results at builds.flathub.org/reproduciblearchive.org iconarchive.today icon are Flathub-hosted.

Related:

Forum discussion:

Open Source Operating Systems

[edit]

What "building from source code" means in the context of building Open Source operating systems from source code is not well defined. See also Builds from Source Code versus Builds including Binary (pre-built) Packages and bootstrappable builds.

Related but different topics

[edit]

Binary means pre-built applications.

This page is related to reproducible builds, binary transparency, and bootstrappable builds, but it is not the same topic.

Reproducible builds

[edit]

Reproducible builds answer the question:

Can someone else independently produce the same program from the same source code?Reproducible builds question.

That is related, but separate.

A project may use a third-party build provider and still work toward reproducible builds.

A project may also build on its own machines but still fail to provide reproducible builds.

Reproducible builds are strongest when the rebuild is independent from the original build infrastructure.

In simple terms:

If the original meal was cooked in Amazon's kitchen, the strongest check is not another check controlled by the same kitchen.Reproducible builds and independent verification analogy.

For strongest verification, reproducible build checks should be possible outside the same cloud company or hosted build platform used for the official build.

Otherwise, users may still need to trust the same provider in two places:

  • Original build: The official program was built using third-party infrastructure.
  • Rebuild check: The reproducibility result or comparison report was also produced, stored, or served using third-party infrastructure.

This does not make reproducible builds useless.

Reproducible builds are still valuable because they make hidden changes harder.

But if the official build, the rebuild, and the reproducibility report all depend on large cloud providers or hosted platforms, then the verification is less independent than it appears.

The strongest model is:

  • Independent rebuilders: Other people or organizations can rebuild the software on separate infrastructure, and the project makes this practical and encouraged.
  • Independent reporting: Reproducibility results can be checked without relying only on the original project's website or the same cloud provider.
  • Public comparison: Differences between binaries can be inspected with tools such as diffoscope.
  • Multiple parties: More than one independent party can confirm the same result.

This page therefore treats reproducible builds as helpful, but not a complete answer to the build location question.

Binary transparency

[edit]

Binary transparency is about making published binaries (programs) visible and auditable over time.

This page is about who controls the infrastructure that creates the binary in the first place.

Bootstrappable builds

[edit]

Bootstrappable builds ask how much pre-existing binary software is needed to build the system from source code.

This page asks where the official binaries are built and who controls that infrastructure.

Technical terminology

[edit]

The technical version is:

Outsourced build trust is the security and privacy gap created when an Open Source project's official binaries are built, signed, stored, or published using infrastructure controlled by a third party, such as a proprietary CI provider or hyperscaler.

This makes the third party part of the trusted computing base, even when the source code itself is open.

Open Source users often assume that "open source" reduces dependency on proprietary vendors and opaque infrastructure. However, official binaries may be produced on hosted CI/CD systems or cloud infrastructure controlled by Microsoft, Google, Amazon, or other third parties. In that case, the source code may be open, but the binary build path still depends on a proprietary build trust boundary.

This is not solved merely by source availability. It requires transparent build provenance, independent rebuilds, reproducible or verifiable builds, controlled signing key custody, and disclosure of build infrastructure.

Security and privacy enthusiasts typically expect sovereign build infrastructure.

Many developers optimize for convenience, cost, and automation. Hyperscalers made it easy and fast to build releases, create digital software signatures, and publish software. For small and medium projects, running independent build infrastructure may be expensive and operationally difficult.

What this page documents

[edit]
  • Build locations: Where official software binaries are built, where that information is publicly known.
  • Build infrastructure: Whether official binaries are built on developer machines, self-hosted infrastructure, Google Cloud, Microsoft Azure, Amazon AWS, GitHub Actions, or other third-party infrastructure.
  • Trust relationships: The trust relationship created when the official program is built, signed, stored, or published through infrastructure controlled by someone other than the project itself.
  • Project comparisons: Multiple Open Source projects are compared. Kicksecure is included as one example, but the topic is general.

What this page does not claim

[edit]
  • No compromise claim: This page does not claim that every third-party build is compromised.
  • No automatic safety claim: This page does not claim that project-controlled builds are automatically safe.
  • No automatic insecurity claim: This page does not claim that third-party-hosted builds are automatically unsafe.
  • No source-code dismissal: This page does not claim that source code availability is useless.
  • No malware accusation: This page does not claim that the listed projects are malicious.
  • No complete supply-chain audit: This page does not attempt to fully audit every build system, signing key, dependency, maintainer, mirror, or hosting provider.

Related background

[edit]

This page does not attempt to explain every criticism of Google, Microsoft, Amazon, or similar companies. The narrower point of this page is build infrastructure.

For broader background, see:

Related arguments from other projects and organizations

[edit]

This concern is not unique to this page.

Other Free Software, Open Source, government, and software supply-chain security sources also treat build infrastructure, developer infrastructure, reproducible builds, and binary provenance as part of the software trust path.

Proprietary developer infrastructure

[edit]

The Software Freedom Conservancyarchive.org iconarchive.today icon argues that Free and Open Source Software projects should not normalize reliance on GitHub's proprietary developer infrastructure. This supports the broader point that the tools and platforms used to develop and publish Open Source software can matter, not only the source license. [13]

This does not by itself prove that GitHub-built binaries are malicious. It supports the narrower claim that proprietary developer infrastructure can be controversial for FOSS projects.

Sovereign source and binary infrastructure

[edit]

The Dutch government developer portal makes a direct integrity argument about code and binaries. It argues that hosting source code is critical infrastructure, that code or binaries in repositories must not be tampered with, and that the government needs full control over its Git forge. [14]

A later Dutch government page says code.overheid.nl is fully self-hosted and supports digital sovereignty. [15]

This is close to the concern discussed on this page: whoever controls the infrastructure can become part of the trust path for source code, binaries, and releases.

Build-system compromise and signed malware

[edit]

The Tor Project's deterministic builds article gives a strong build-time attack model. It argues that malware can attack software development and build processes themselves, and that deterministic, distributed builds are a defense against targeted attacks. [16]

In simpler terms: the danger is not only malicious source code. The danger can also be malicious infrastructure that turns clean source code into a malicious official binary.

Reproducible builds and independent verification

[edit]

The Reproducible Builds project describes reproducible builds as creating an independently verifiable path from source code to binary code. It also says reproducible builds help detect unauthorized changes to the build process and give users confidence that downloaded binaries match the untampered source code. [17]

This supports the point that source code visibility alone is not enough. Users also need ways to verify that the binary corresponds to the intended source and build process.

Build provenance and supply-chain threat models

[edit]

SLSA describes a build integrity threat as an adversary introducing behavior into an artifact without changing the source code. This directly matches the build-time backdoor concern discussed on this page. [18]

SLSA provenance also records details about the build, including the builder identity, build parameters, dependencies, and execution details. This supports documenting who built the binary, how it was built, and what infrastructure was involved. [19]

Binaries can differ from source code

[edit]

The Go project explains the source-to-binary problem in plain terms: Open Source code can be inspected, but most users download compiled binaries, and a supply-chain attacker could replace served binaries while leaving the source code unchanged. [20]

The Go article also gives a useful independence lesson: verifying a package without using the same software stack can be a stronger check than rebuilding inside the same environment that may have been compromised.

Multiple independent builders

[edit]

Gitian describes a software distribution method where binaries are verified by multiple builders. It says this removes the build and distribution process as a single point of failure. [21]

Bitcoin Optech similarly explains that reproducible builds mean no single person or computer needs to be trusted to produce the executable binaries most users run. [22]

This supports the positive model discussed on this page: independent rebuilders are stronger than relying on one build computer, one company, or one hosted platform.

Civil-society security framing

[edit]

The Open Technology Fund describes reproducible builds as creating an independently verifiable path from source to binary and preventing attacks against the complex systems that build shared digital infrastructure. [23]

This supports treating build systems as security-sensitive infrastructure, especially for privacy, security, and Internet freedom software.

User discussion

[edit]

These discussions show that some users notice and question connections to Google, cloud infrastructure, or opaque binary build and hosting paths.

Some of these discussions are about network connections or hosting, not build machines. They are included to show that some users do notice and question third-party infrastructure behind Open Source software.

Polls

[edit]

todo

[edit]

petition

flatpak bug report

document Amazon threat model or stop using Amazon

flatpak reddit

Footnotes

[edit]
  1. moving shippable builds and Android emulator tests from AWS over to GCPMozilla Discourse: Engineering Effectiveness Newsletter, August and September 2022archive.org iconarchive.today icon

    Mozilla issue tracker: Bug 1547111 - Migrate shippable builds from AWS to GCP r=ahal!,jmaher!archive.org iconarchive.today icon

  2. GitHubarchive.org iconarchive.today icon is owned by Microsoft. See also GitHub Actionsarchive.org iconarchive.today icon.

    Release automation We have now setup release automation so that Github actions automatically trigger a release build and publish a release when we push a tag to the repository.https://github.com/fork-maintainers/iceraven-browserarchive.org iconarchive.today icon

  3. Flatpak's build orchestrator Vorarbeiterarchive.org iconarchive.today icon according to a Flathub Vorarbeiter 2026 blog postarchive.org iconarchive.today icon uses RunsOnarchive.org iconarchive.today icon, which runs on AWS (Amazon).
  4. Kicksecure Trust wiki page, chapter Software Build Process Security
  5. related discussion: Qubes forums: update philosophy flawarchive.org iconarchive.today icon
  6. ...

  7. The new service is a middleman between GitHub and GitHub Actions.Flathub: Vorarbeiter is herearchive.org iconarchive.today icon

    Since then, we have fully switched to RunsOn for all builds.Flathub: What's new in Vorarbeiterarchive.org iconarchive.today icon

    Thank you RunsOn team and AWS for making this possible!Flathub: What's new in Vorarbeiterarchive.org iconarchive.today icon

  8. All source available submissions must be built entirely from source code.Flathub: Requirementsarchive.org iconarchive.today icon

    Exceptions can be granted on a case-by-case basis.Flathub: Requirementsarchive.org iconarchive.today icon

  9. Indicate on flathub.org Whether a Flatpak is Built from Source or Binary During Build Process on Flathub #5733archive.org iconarchive.today icon
  10. We have started testing binary reproducibility of x86_64 builds targetting the stable repository.Flathub: What's new in Vorarbeiterarchive.org iconarchive.today icon

  11. Failures are not currently acted on. When we collect more results, we may start to surface them to app maintainers for investigation. We also don't test direct uploads at the moment.Flathub: What's new in Vorarbeiterarchive.org iconarchive.today icon

  12. A tool to rebuild Flatpak apps published on Flathub and compare reproducibility using diffoscope.flathub-repro-checkerarchive.org iconarchive.today icon

    boto3 is optionally used to upload diffoscope results as a zip file to Amazon S3flathub-repro-checkerarchive.org iconarchive.today icon

  13. GitHub itself is the very opposite of FOSS.Software Freedom Conservancy: Give Up GitHubarchive.org iconarchive.today icon

    reject GitHub's proprietary servicesSoftware Freedom Conservancy: Give Up GitHubarchive.org iconarchive.today icon

  14. Het hosten van broncode is een kritiek onderdeel van de infrastructuurDeveloper Overheid: Aanbeveling voor de Git-werkplaats van de overheidarchive.org iconarchive.today icon

    volledige beschikking te hebben over een Git-forgeDeveloper Overheid: Aanbeveling voor de Git-werkplaats van de overheidarchive.org iconarchive.today icon

  15. fully self-hosted and supports digital sovereigntyDigital Government: Soft launch of open-source code platform for governmentarchive.org iconarchive.today icon

  16. malware that attacks the software development and build processesTor Project: Deterministic Builds Part Onearchive.org iconarchive.today icon

  17. independently-verifiable path from source to binary codeReproducible Buildsarchive.org iconarchive.today icon

    detect unauthorized changes to the build process earlyReproducible Buildsarchive.org iconarchive.today icon

  18. introduce behavior to an artifact without changing its source codeSLSA: Threats and mitigationsarchive.org iconarchive.today icon

  19. Details specific to this particular execution of the build.SLSA: Provenancearchive.org iconarchive.today icon

  20. leaving the source code unmodifiedGo Blog: Perfectly Reproducible, Verified Go Toolchainsarchive.org iconarchive.today icon

    posted binaries are free of hidden changesGo Blog: Perfectly Reproducible, Verified Go Toolchainsarchive.org iconarchive.today icon

  21. verified by multiple buildersGitianarchive.org iconarchive.today icon

    single point of failureGitianarchive.org iconarchive.today icon

  22. no one person or computer needs to be trustedBitcoin Optech: Reproducible buildsarchive.org iconarchive.today icon

  23. preventing attacks targeting the complex systemsOpen Technology Fund: Reproducible Buildsarchive.org iconarchive.today icon

Notification image

We believe security software like Kicksecure needs to remain Open Source and independent. Would you help sustain and grow the project? Learn more about our 14 year success story and maybe DONATE!