Onion Reachability

From Kicksecure
Jump to navigation Jump to search

Onion services can sometimes become unreachable even when the server configuration has not changed. Possible causes include Tor bugs, denial of service attacks, malformed service descriptors, problems with the local Tor daemon, and limitations in available debugging tools. This page documents known issues, investigation results, temporary recovery procedures, and related upstream bug reports.

User Documentation

[edit]

Non-functional Onion Services

Tor Upstream Unfixed Denial of Service Issue

[edit]

Tor upstream DoS issue attacker can force intro point rotation by ddosarchive.org iconarchive.today icon was reported by Tor developer Roger Dingledinearchive.org iconarchive.today icon. At the time of writing:

Malformed Hidden Service Descriptor Issues

[edit]

All were unresolved at the time of writing in July 2026.

General Bad State of Tor Onions

[edit]

Tor onions are generally in a bad state. This is unspecific to Kicksecure or Whonix.

Vanguards is a software project by The Tor Project (TPO) / Tor developer Mike Perry. Quote from The Vanguards Onion Service Addonarchive.org iconarchive.today icon:

Even after deployment of the new v3 onion service protocolarchive.org iconarchive.today icon, the attacks facing onion services are wide-ranging, and still require more extensive modifications to fix in Tor-core itself.

Because of this, we have decided to rapid-prototype these defenses in a controller addon in order to make them available ahead of their official Tor-core release, for onion services that require high security as soon as possible.

For details about the defenses themselves, please see README_TECHNICAL.mdarchive.org iconarchive.today icon.

For additional security information, please see README_SECURITY.mdarchive.org iconarchive.today icon.https://github.com/mikeperry-tor/vanguardsarchive.org iconarchive.today icon

A very comprehensive bug reportarchive.org iconarchive.today icon was submitted by an advanced user in 2022.

In the same year, 2022, Tor developer Mike Perry provided rare and detailed insights into the funding and development issues of Tor onions.

We have been seeing sustained attacks from multiple adversaries against onion services, and against the Tor network itself, for the past 3 months. These adversaries are continually adapting to our defenses.

For the past 3 years, we have attempted to get funding for DoS defenses for onion services from various funding sources. Every single one of these funding sources declined to fund onion service development or defenses. Because of the way our funding model works, this proposal-reject churn also delayed development on other tasks. Effectively a DoS attack in and of itself.

During this process, one of the two primary developers on onion services resigned. The other primary onion service developer is currently on leave. In fact, we will only have 2 developers directly devoted to all of C-Tor for the next 6-8 weeks or so, and about half of that developer time will be spent writing and revising other funding proposals during this period. The remainder will be spent performing other already funded work, and related administration tasks.

We do have pending funding for general DoS defenses that should be finalized soon, but that also does not cover onion service development. We will need to show other progress for that funder; any DoS defenses specifically for onion services will still have to be done "on the side".

Normally, I would not air our dirty laundry like this, but since it will have a direct impact on people like you, it seemed important to explain why we're going to be especially unresponsive to issues with onion services for the next while.https://github.com/mikeperry-tor/vanguards/issues/91#issuecomment-1228698338archive.org iconarchive.today icon

As per the Vanguards commit historyarchive.org iconarchive.today icon, the last substantial code change in Vanguards happened in 2021. There were some minor commits in 2023, but these were only related to Dependabotarchive.org iconarchive.today icon.

A Vanguards issuearchive.org iconarchive.today icon was reported to TPO in November 2023, was assigned, but stalled.

More tickets with zero replies:

There was a Tor Project fundraiser in 2023archive.org iconarchive.today icon.

The Tor Project published a roadmap for improving onion supportarchive.org iconarchive.today icon.

A number of open tickets are related to onion service stability:

The Tor Project's focus seems to have since shifted to onions in Arti.[1]

See Announcing Vanguards for Artiarchive.org iconarchive.today icon.

Arti's Vanguards implementation is based on the research behind Tor's Vanguards add-onarchive.org iconarchive.today icon, and offers similar protections to its Vanguards componentarchive.org iconarchive.today icon.https://blog.torproject.org/announcing-vanguards-for-arti/archive.org iconarchive.today icon

The other experimental defenses from the add-on are not yet implemented in Arti (but they are on our roadmap!)https://blog.torproject.org/announcing-vanguards-for-arti/archive.org iconarchive.today icon

Lack of Onion Reachability Debugging Tools

[edit]

A Tor v3 onion connection is two independent 3-hop circuits meeting at a rendezvous point (RP), split across two daemons on two hosts, with cached state and rate-limited observation. From the outside almost every failure looks identical: a connect just "hangs" or "fails". Each question below isolates a different failure stage, and the difficulty is nearly always the same shape: how do I observe one stage without the other stages masking what it is really doing?

Fault attribution: is it me or the service?

[edit]
Question Why it is hard Planned Tool(s)
Whose fault is a failed connect -- my client Tor, the network, or the service? The single hardest question: all three produce the same "it does not load". onion-preflight (isolates client-vs-onion via external control onions), tor-client-health
Can my local Tor reach relays at all, or is my guard set dead? A degraded or 0-up guard set is a client fault masquerading as an onion outage. tor-client-health (DEGRADED on 0 guards)
Is the break in the client's own 3 hops, or beyond the rendezvous point where I can never see? On the real internet each side sees only its own three hops. onion-connect-trace (client leg), onion-control-snapshot --paths (service leg), onion-e2e-path (stitches both on a same-host repro)
Which exact stage did the connect die at -- descriptor / intro / rend / service? "Failed" is not a stage. SERVICE = rendezvous done, far side refused at app/port level, a wholly different bug from "intro did not answer". onion-connect-trace (stage classifier), extended SOCKS 0xF0-0xF6

Descriptor problems (the stale-cache trap)

[edit]
Question Why it is hard Planned Tool(s)
Is the descriptor published on the hashring now, or am I clinging to a cached one? Two different truths, and the warm-client cache hides the difference. onion-desc-fetch (live HSFETCH), onion-desc-inspect (cached copy)
Am I hitting the tor#25882 refetch-stallarchive.org iconarchive.today icon (warm client, stale descriptor, dead intro points)? A fresh client works while the warm one fails, so it will not reproduce after a restart. onion-preflight REFETCH HEAL TEST (heals => stale-descriptor class; still fails => intro points genuinely dead)
How do I view the descriptor, its revision counter, and which dead intro relays the client held? The forensic state lives encrypted in the client's cache. onion-desc-inspect (revision counter, lifetime, single-onion, PoW, intro set; --ns marks NOT IN CONSENSUS)
Is my clock skewed outside the descriptor validity window? A silent skew makes a perfectly-published descriptor look absent. clock-drift-check (independent online clock vs local)

Split-brain (two Tor daemons, same identity)

[edit]
Question Why it is hard Planned Tool(s)
Are two Tor daemons publishing the same identity key and competing on the hashring? Intermittent by nature: a client hits a "wrong" HSDir only sometimes. onion-splitbrain-check (fans HSFETCH to every responsible dir)
Is it a REAL persistent split, or benign republish lag caught mid-flight? A normal republish makes old+new descriptors coexist transiently, so one sweep is never proof, and C-tor rate-limits per-HSDir requeries ~15 min. onion-splitbrain-corroborate (spaced sweeps: PERSISTENT vs FLAPPING vs UNDER-SAMPLED)

DoS / introduction flood

[edit]
Question Why it is hard Planned Tool(s)
Would we detect an introduction flood (tor#26294) before reachability degrades? The preflight path only points at the stats tool; it does not itself alarm, so a flood erodes reachability invisibly. onion-hs-stats (INTRODUCE2 rate, PoW verify-fails/replays, PoW-queue trim, est_hours_to_force_intro_rotation dropping below intro lifetime)
Is the service silently running on a degraded intro-point set? Fewer live intro points than configured shrinks reachability before it fully breaks. onion-intro-watchdog

Server-side intermittent disruption

[edit]
Question Why it is hard Planned Tool(s)
Is our own server's uplink intermittently killing OR-connections? Intermittent, gone by the time you look, and indistinguishable from a healthy circuit in any point-in-time snapshot. tor-orconn-watch --follow (live flap/reset/extend-fail timeline), tor-orconn-watch cron */5 (joined with host NIC/TCP counter deltas)

Reachable-but-wrong (the false green)

[edit]
Question Why it is hard Planned Tool(s)
Is a web onion reachable but serving the wrong thing -- a 502/503, the default vhost, or an empty reply after a good rendezvous? A completed rendezvous only proves the tunnel is up; nginx can still answer with an error page, the wrong server block, or nothing at all, and a plain reachability probe passes it clean. onion-preflight (HTTP end-to-end gate), onion-mail-probe --direct (local :74 HTTP stub vs the onion's :80)
Is a mail onion reachable but not actually replying correctly? Same trap for mail: a port can accept TCP yet give no valid greeting, or refuse at the app level after a good rendezvous. onion-mail-probe (POP3 +OK / IMAP * OK / SMTP 220; --direct separates "backend down" from "onion path broken")
Does any .onion request get redirected to the clearnet host? A correctness and deanonymization bug that a plain reachability check passes clean. onion-redirect-test

Planned Tools

[edit]

General tools:

  • onion-socks-probe -- raw SOCKS5 CONNECT reachability probe (real onion REP byte).
  • onion-desc-fetch -- is the descriptor PUBLISHED on the hashring right now (HSFETCH)?
  • onion-desc-inspect -- decrypt/print the client's CACHED descriptor (intro points, revision, PoW).
  • onion-connect-trace -- client-side sidecar timeline: stage reached + the client's own 3 hops.
  • tor-client-health -- client Tor health (bootstrap, guards, consensus validity window).
  • onion-splitbrain-check -- two daemons publishing the SAME identity key on the hashring?
  • onion-splitbrain-selftest -- mock-control-port selftest for the split-brain detector.
  • tor-orconn-watch -- server-side OR-connection disruption (live --follow or cron sample).
  • clock-drift-check -- online-vs-local clock delta (a skew breaks descriptor windows).
  • onion-control-snapshot -- service-side intro/guard/liveness snapshot (--paths: service's 3 hops).
  • onion-intro-watchdog -- detect a DEGRADED intro-point set (+ optional remediation).
  • onion-hs-stats -- onion health / anti-DoS stats (intro rate, PoW, upload freshness).
  • onion-hostname -- print an instance's .onion hostname.

Server specific tools:

  • onion-preflight -- staged reachability diagnosis for ONE onion (+ refetch heal test).
  • onion-e2e-path -- end-to-end hop-by-hop stitcher, both control ports (repro/same-host).
  • check-onions -- sweep EVERY instance, run the matching preflight; non-zero if any fail.
  • onion-probe-fresh -- probe via a FRESH, isolated Tor (onionprobe, ephemeral ports).
  • onion-fail-capture / -all -- on-failure forensic capture into the onion-diag log.
  • onion-rend-collect -- collect REND (degraded-descriptor) causes into the diag log.
  • onion-circuit-rate -- rendezvous-circuit request rate per onion (from nginx logs).
  • onion-redirect-test -- guard: a .onion request must never redirect to the clearnet host.
  • onion-ipv6-ab -- TEMPORARY: surface the IPv6 A/B circuit-test result.

Local Tor Client Stall Bug

[edit]

A relevant factor is the state of the Tor user's local Tor daemon.

It has been reproduced multiple times that an unreachable onion becomes reachable after restarting Tor. Similar reports can be found online.archive.org iconarchive.today icon

When restarting Tor resolves the issue, this can almost always be considered a bug in Tor, as Tor should be able to recover without requiring a restart.

onionprobe can be useful for this purpose.

onionprobe

[edit]

Whonix: onionprobe is incompatible with Whonix.[2]

Debian, Kicksecure, etc.:

sudo apt update

sudo apt install onionprobe

Kicksecure:

onionprobe -e w5j6stm77zs6652pgsij4awcjeel3eco7kvipheu6mtr623eyyehj4yd.onion

Whonix:

onionprobe -e dds6qkxpwdeubwucdiaord2xgbbeyds25rbsgr73tbfpqpt4a6vjwsyd.onion

Onion Domains by Other Projects

[edit]

Closes issues:

Commits:

Denial of Service Attacks against Tor and Onions Are Common

[edit]

At some points, the attacks impacted the network severely enough that users could not load pages or access onion services.https://blog.torproject.org/tor-network-ddos-attack/archive.org iconarchive.today icon

Non-Solutions

[edit]

OnionBalance

[edit]

As of December 2023, it does not support the Proof Of Work anti-DoS defense (PoWarchive.org iconarchive.today icon), but this is being plannedarchive.org iconarchive.today icon.https://onionservices.torproject.org/apps/web/checklist/#using-onionbalancearchive.org iconarchive.today icon

Non-Issues

[edit]

Docker

[edit]

download.kicksecure.com / download.whonix.org are simple, with no Docker at all.

Tor Onions Outdated Cryptography

[edit]

Does not support offline keys yet:

Footnotes

[edit]
  1. Keep an Eye on Arti Torarchive.org iconarchive.today icon
  2. Because onionprobe starts a temporary (throwaway) Tor daemon before probing an onion. Hence, onionprobe is not suitable to be run from within Whonix, and only from Debian, Kicksecure, or similar. onionprobe supports --no-launch_tor but requires Tor control protocol access. Writing an onion-graterarchive.org iconarchive.today icon profile might be possible, but it would be a lot of work for a non-ideal result using onionprobe differently from how it is designed to run by default.
Notification image

We believe security software like Kicksecure needs to remain Open Source and independent. Would you help sustain and grow the project? Learn more about our 14 year success story and maybe DONATE!