Urbit Developers
  • Hoon

    • Overview
    • Cheat Sheet
    • Irregular forms
    • Hoon Errors
    • Hoon Style Guide
    • Basic Types
    • Advanced Types
    • Auras
    • Arvo
    • Standard Library

      • 1a: Basic Arithmetic
      • 1b: Tree Addressing
      • 1c: Molds and Mold-Builders
      • 2a: Unit Logic
      • 2b: List Logic
      • 2c: Bit Arithmetic
      • 2d: Bit Logic
      • 2e: Insecure Hashing
      • 2f: Noun Ordering
      • 2g: Unsigned Powers
      • 2h: Set Logic
      • 2i: Map Logic
      • 2j: Jar and Jug Logic
      • 2k: Queue Logic
      • 2l: Container from Container
      • 2m: Container from Noun
      • 2n: Functional Hacks
      • 2o: Normalizing Containers
      • 2p: Serialization
      • 2q: Molds and Mold-Builders
      • 3a: Modular and Signed Ints
      • 3b: Floating Point
      • 3c: Urbit Time
      • 3d: SHA Hash Family
      • 3e: AES encryption (Removed)
      • 3f: Scrambling
      • 3g: Molds and Mold-Builders
      • 4a: Exotic Bases
      • 4b: Text Processing
      • 4c: Tank Printer
      • 4d: Parsing (Tracing)
      • 4e: Parsing (Combinators)
      • 4f: Parsing (Rule-Builders)
      • 4g: Parsing (Outside Caller)
      • 4h: Parsing (ASCII Glyphs)
      • 4i: Parsing (Useful Idioms)
      • 4j: Parsing (Bases and Base Digits)
      • 4k: Atom Printing
      • 4l: Atom Parsing
      • 4m: Formatting Functions
      • 4n: Virtualization
      • 4o: Molds
      • 5a: Compiler Utilities
      • 5b: Macro Expansion
      • 5c: Compiler Backend & Prettyprinter
      • 5d: Parser
      • 5e: Molds and mold builders
      • 5f: Profiling support
      • Runes

        • Constants (Atoms and Strings)
        • + lus · Arms
        • | bar · Cores
        • $ buc · Structures
        • % cen · Calls
        • : col · Cells
        • . dot · Nock
        • / fas · Imports
        • ^ ket · Casts
        • ; mic · Make
        • ~ sig · Hints
        • = tis · Subject
        • ? wut · Conditionals
        • ! zap · Wild
        • --, == Terminators
        • Limbs and wings

          • Limbs
          • Wings
          • Zuse

            • 2d(1-5): To JSON, Wains
            • 2d(6): From JSON
            • 2d(7): From JSON (unit)
            • 2e(2-3): Print & Parse JSON
            • 2m: Ordered Maps
          • Arvo

            • Overview
            • Ames

              • Overview
              • Cryptography
              • API Reference
              • Scry Reference
              • Data Types
              • Behn

                • Overview
                • API Reference
                • Scry Reference
                • Examples
                • Clay

                  • Overview
                  • Architecture
                  • Using Clay
                  • Data Types
                  • Scry Reference
                  • API Reference
                  • Examples
                  • Marks

                    • Overview
                    • Writing Marks
                    • Using Marks
                    • Examples
                  • Dill

                    • Overview
                    • API Reference
                    • Scry Reference
                    • Data Types
                    • Eyre

                      • Overview
                      • External API Reference
                      • Internal API Reference
                      • Scry Reference
                      • Data Types
                      • Guide
                      • Gall

                        • Overview
                        • API Reference
                        • Data Types
                        • Threads Reference
                        • Iris

                          • Overview
                          • API Reference
                          • Data Types
                          • Example
                          • Jael

                            • Overview
                            • API Reference
                            • Scry Reference
                            • Data Types
                            • Examples
                            • Khan

                              • Overview
                              • API Reference
                              • Data Types
                              • Example
                              • Threads

                                • Overview
                                • HTTP API
                                • Reference
                                • Gall

                                  • Start Thread
                                  • Take Result
                                  • Take Facts
                                  • Stop Thread
                                  • Poke Thread
                                  • Examples

                                    • Fetch JSON
                                    • Child Thread
                                    • Main-loop
                                    • Poke Agent
                                    • Scry
                                    • Take Fact
                                  • Concepts

                                    • Scries
                                    • Subscriptions
                                    • Tutorials

                                      • Move Trace
                                      • Reference

                                        • Cryptography
                                        • Filesystem Hierarchy
                                      • Nock

                                        • Nock Definition
                                        • Explanation
                                        • Example
                                        • Implementations
                                        • Vere

                                          • C Runtime System
                                          • Land of Nouns
                                          • API overview by prefix
                                          • C in Urbit
                                          • Writing Jets
                                          • Cryptography
                                          • Azimuth

                                            • Overview
                                            • Urbit HD Wallet
                                            • Azimuth Data Flow
                                            • Azimuth.eth
                                            • Ecliptic.eth
                                            • Advanced Azimuth Tools
                                            • Life and Rift
                                            • Layer 2

                                              • Layer 2 Overview
                                              • Layer 2 Actions
                                              • Transaction Format
                                              • Rollers
                                              • Roller HTTP RPC-API
                                              • Custom Roller Tutorial
                                            • Glossary

                                              • Agent
                                              • Ames
                                              • API
                                              • Aqua
                                              • Arm
                                              • Arvo
                                              • Atom
                                              • Aura
                                              • Aural ASCII
                                              • Azimuth
                                              • Battery
                                              • Behn
                                              • Bowl
                                              • Bridge
                                              • Bunt
                                              • Card
                                              • Case
                                              • Cask
                                              • Cell
                                              • Censures
                                              • Claims
                                              • Clay
                                              • Cold atom
                                              • Comet
                                              • Commit
                                              • Cons
                                              • Context
                                              • Cord
                                              • Core
                                              • Delegated Sending
                                              • Desk
                                              • Dill
                                              • Document Proposal
                                              • Dojo
                                              • Door
                                              • Double-boot
                                              • Dry Gate
                                              • Duct
                                              • Ecliptic
                                              • Entropy
                                              • Event Log
                                              • Eyre
                                              • Face
                                              • Fact
                                              • Foo Bar Baz
                                              • Galaxy
                                              • Gall
                                              • Garden
                                              • Gate
                                              • Generator
                                              • Gift
                                              • Glob
                                              • HD Wallet
                                              • Helm
                                              • Hood
                                              • Hoon
                                              • Invite Tree
                                              • Iris
                                              • Jael
                                              • Jet
                                              • Kelvin versioning
                                              • Kelvin versioning
                                              • Keyfile
                                              • Keywords
                                              • Khan
                                              • Kiln
                                              • Landscape
                                              • Leg
                                              • List
                                              • Loobean
                                              • Lull
                                              • Mark
                                              • Metals
                                              • Mold
                                              • Monad
                                              • Moon
                                              • Move
                                              • Nock
                                              • Noun
                                              • ~
                                              • OTA Updates
                                              • Path Prefix
                                              • Path
                                              • @p
                                              • Payload
                                              • Peek
                                              • pH
                                              • Pier
                                              • Pill
                                              • Planet
                                              • Poke
                                              • Proxies
                                              • Replay
                                              • Factory Reset
                                              • Roller
                                              • Naive rollups
                                              • Rune
                                              • Runtime
                                              • Sail
                                              • Sample
                                              • Scry
                                              • Senate
                                              • Ship
                                              • Slam
                                              • Spider
                                              • Sponsor
                                              • Star
                                              • String
                                              • Subject Oriented Programming
                                              • Subject
                                              • Subscription
                                              • Tape
                                              • Thread
                                              • Trap
                                              • Udon
                                              • Upgrade Proposal
                                              • Userspace
                                              • Vane
                                              • Vase
                                              • Vere
                                              • Voting
                                              • Warm atom
                                              • Wet Gate
                                              • Wing
                                              • Wire
                                              • Wrapper
                                              • ~zod
                                              • Zuse
                                              • Additional

                                                • Cryptography
                                                • Mips (Maps of Maps)
                                                • Software Distribution

                                                  • Overview
                                                  • Docket File
                                                  • Glob
                                                Urbit Developers
                                                • Hoon

                                                  • Overview
                                                  • Cheat Sheet
                                                  • Irregular forms
                                                  • Hoon Errors
                                                  • Hoon Style Guide
                                                  • Basic Types
                                                  • Advanced Types
                                                  • Auras
                                                  • Arvo
                                                  • Standard Library

                                                    • 1a: Basic Arithmetic
                                                    • 1b: Tree Addressing
                                                    • 1c: Molds and Mold-Builders
                                                    • 2a: Unit Logic
                                                    • 2b: List Logic
                                                    • 2c: Bit Arithmetic
                                                    • 2d: Bit Logic
                                                    • 2e: Insecure Hashing
                                                    • 2f: Noun Ordering
                                                    • 2g: Unsigned Powers
                                                    • 2h: Set Logic
                                                    • 2i: Map Logic
                                                    • 2j: Jar and Jug Logic
                                                    • 2k: Queue Logic
                                                    • 2l: Container from Container
                                                    • 2m: Container from Noun
                                                    • 2n: Functional Hacks
                                                    • 2o: Normalizing Containers
                                                    • 2p: Serialization
                                                    • 2q: Molds and Mold-Builders
                                                    • 3a: Modular and Signed Ints
                                                    • 3b: Floating Point
                                                    • 3c: Urbit Time
                                                    • 3d: SHA Hash Family
                                                    • 3e: AES encryption (Removed)
                                                    • 3f: Scrambling
                                                    • 3g: Molds and Mold-Builders
                                                    • 4a: Exotic Bases
                                                    • 4b: Text Processing
                                                    • 4c: Tank Printer
                                                    • 4d: Parsing (Tracing)
                                                    • 4e: Parsing (Combinators)
                                                    • 4f: Parsing (Rule-Builders)
                                                    • 4g: Parsing (Outside Caller)
                                                    • 4h: Parsing (ASCII Glyphs)
                                                    • 4i: Parsing (Useful Idioms)
                                                    • 4j: Parsing (Bases and Base Digits)
                                                    • 4k: Atom Printing
                                                    • 4l: Atom Parsing
                                                    • 4m: Formatting Functions
                                                    • 4n: Virtualization
                                                    • 4o: Molds
                                                    • 5a: Compiler Utilities
                                                    • 5b: Macro Expansion
                                                    • 5c: Compiler Backend & Prettyprinter
                                                    • 5d: Parser
                                                    • 5e: Molds and mold builders
                                                    • 5f: Profiling support
                                                    • Runes

                                                      • Constants (Atoms and Strings)
                                                      • + lus · Arms
                                                      • | bar · Cores
                                                      • $ buc · Structures
                                                      • % cen · Calls
                                                      • : col · Cells
                                                      • . dot · Nock
                                                      • / fas · Imports
                                                      • ^ ket · Casts
                                                      • ; mic · Make
                                                      • ~ sig · Hints
                                                      • = tis · Subject
                                                      • ? wut · Conditionals
                                                      • ! zap · Wild
                                                      • --, == Terminators
                                                      • Limbs and wings

                                                        • Limbs
                                                        • Wings
                                                        • Zuse

                                                          • 2d(1-5): To JSON, Wains
                                                          • 2d(6): From JSON
                                                          • 2d(7): From JSON (unit)
                                                          • 2e(2-3): Print & Parse JSON
                                                          • 2m: Ordered Maps
                                                        • Arvo

                                                          • Overview
                                                          • Ames

                                                            • Overview
                                                            • Cryptography
                                                            • API Reference
                                                            • Scry Reference
                                                            • Data Types
                                                            • Behn

                                                              • Overview
                                                              • API Reference
                                                              • Scry Reference
                                                              • Examples
                                                              • Clay

                                                                • Overview
                                                                • Architecture
                                                                • Using Clay
                                                                • Data Types
                                                                • Scry Reference
                                                                • API Reference
                                                                • Examples
                                                                • Marks

                                                                  • Overview
                                                                  • Writing Marks
                                                                  • Using Marks
                                                                  • Examples
                                                                • Dill

                                                                  • Overview
                                                                  • API Reference
                                                                  • Scry Reference
                                                                  • Data Types
                                                                  • Eyre

                                                                    • Overview
                                                                    • External API Reference
                                                                    • Internal API Reference
                                                                    • Scry Reference
                                                                    • Data Types
                                                                    • Guide
                                                                    • Gall

                                                                      • Overview
                                                                      • API Reference
                                                                      • Data Types
                                                                      • Threads Reference
                                                                      • Iris

                                                                        • Overview
                                                                        • API Reference
                                                                        • Data Types
                                                                        • Example
                                                                        • Jael

                                                                          • Overview
                                                                          • API Reference
                                                                          • Scry Reference
                                                                          • Data Types
                                                                          • Examples
                                                                          • Khan

                                                                            • Overview
                                                                            • API Reference
                                                                            • Data Types
                                                                            • Example
                                                                            • Threads

                                                                              • Overview
                                                                              • HTTP API
                                                                              • Reference
                                                                              • Gall

                                                                                • Start Thread
                                                                                • Take Result
                                                                                • Take Facts
                                                                                • Stop Thread
                                                                                • Poke Thread
                                                                                • Examples

                                                                                  • Fetch JSON
                                                                                  • Child Thread
                                                                                  • Main-loop
                                                                                  • Poke Agent
                                                                                  • Scry
                                                                                  • Take Fact
                                                                                • Concepts

                                                                                  • Scries
                                                                                  • Subscriptions
                                                                                  • Tutorials

                                                                                    • Move Trace
                                                                                    • Reference

                                                                                      • Cryptography
                                                                                      • Filesystem Hierarchy
                                                                                    • Nock

                                                                                      • Nock Definition
                                                                                      • Explanation
                                                                                      • Example
                                                                                      • Implementations
                                                                                      • Vere

                                                                                        • C Runtime System
                                                                                        • Land of Nouns
                                                                                        • API overview by prefix
                                                                                        • C in Urbit
                                                                                        • Writing Jets
                                                                                        • Cryptography
                                                                                        • Azimuth

                                                                                          • Overview
                                                                                          • Urbit HD Wallet
                                                                                          • Azimuth Data Flow
                                                                                          • Azimuth.eth
                                                                                          • Ecliptic.eth
                                                                                          • Advanced Azimuth Tools
                                                                                          • Life and Rift
                                                                                          • Layer 2

                                                                                            • Layer 2 Overview
                                                                                            • Layer 2 Actions
                                                                                            • Transaction Format
                                                                                            • Rollers
                                                                                            • Roller HTTP RPC-API
                                                                                            • Custom Roller Tutorial
                                                                                          • Glossary

                                                                                            • Agent
                                                                                            • Ames
                                                                                            • API
                                                                                            • Aqua
                                                                                            • Arm
                                                                                            • Arvo
                                                                                            • Atom
                                                                                            • Aura
                                                                                            • Aural ASCII
                                                                                            • Azimuth
                                                                                            • Battery
                                                                                            • Behn
                                                                                            • Bowl
                                                                                            • Bridge
                                                                                            • Bunt
                                                                                            • Card
                                                                                            • Case
                                                                                            • Cask
                                                                                            • Cell
                                                                                            • Censures
                                                                                            • Claims
                                                                                            • Clay
                                                                                            • Cold atom
                                                                                            • Comet
                                                                                            • Commit
                                                                                            • Cons
                                                                                            • Context
                                                                                            • Cord
                                                                                            • Core
                                                                                            • Delegated Sending
                                                                                            • Desk
                                                                                            • Dill
                                                                                            • Document Proposal
                                                                                            • Dojo
                                                                                            • Door
                                                                                            • Double-boot
                                                                                            • Dry Gate
                                                                                            • Duct
                                                                                            • Ecliptic
                                                                                            • Entropy
                                                                                            • Event Log
                                                                                            • Eyre
                                                                                            • Face
                                                                                            • Fact
                                                                                            • Foo Bar Baz
                                                                                            • Galaxy
                                                                                            • Gall
                                                                                            • Garden
                                                                                            • Gate
                                                                                            • Generator
                                                                                            • Gift
                                                                                            • Glob
                                                                                            • HD Wallet
                                                                                            • Helm
                                                                                            • Hood
                                                                                            • Hoon
                                                                                            • Invite Tree
                                                                                            • Iris
                                                                                            • Jael
                                                                                            • Jet
                                                                                            • Kelvin versioning
                                                                                            • Kelvin versioning
                                                                                            • Keyfile
                                                                                            • Keywords
                                                                                            • Khan
                                                                                            • Kiln
                                                                                            • Landscape
                                                                                            • Leg
                                                                                            • List
                                                                                            • Loobean
                                                                                            • Lull
                                                                                            • Mark
                                                                                            • Metals
                                                                                            • Mold
                                                                                            • Monad
                                                                                            • Moon
                                                                                            • Move
                                                                                            • Nock
                                                                                            • Noun
                                                                                            • ~
                                                                                            • OTA Updates
                                                                                            • Path Prefix
                                                                                            • Path
                                                                                            • @p
                                                                                            • Payload
                                                                                            • Peek
                                                                                            • pH
                                                                                            • Pier
                                                                                            • Pill
                                                                                            • Planet
                                                                                            • Poke
                                                                                            • Proxies
                                                                                            • Replay
                                                                                            • Factory Reset
                                                                                            • Roller
                                                                                            • Naive rollups
                                                                                            • Rune
                                                                                            • Runtime
                                                                                            • Sail
                                                                                            • Sample
                                                                                            • Scry
                                                                                            • Senate
                                                                                            • Ship
                                                                                            • Slam
                                                                                            • Spider
                                                                                            • Sponsor
                                                                                            • Star
                                                                                            • String
                                                                                            • Subject Oriented Programming
                                                                                            • Subject
                                                                                            • Subscription
                                                                                            • Tape
                                                                                            • Thread
                                                                                            • Trap
                                                                                            • Udon
                                                                                            • Upgrade Proposal
                                                                                            • Userspace
                                                                                            • Vane
                                                                                            • Vase
                                                                                            • Vere
                                                                                            • Voting
                                                                                            • Warm atom
                                                                                            • Wet Gate
                                                                                            • Wing
                                                                                            • Wire
                                                                                            • Wrapper
                                                                                            • ~zod
                                                                                            • Zuse
                                                                                            • Additional

                                                                                              • Cryptography
                                                                                              • Mips (Maps of Maps)
                                                                                              • Software Distribution

                                                                                                • Overview
                                                                                                • Docket File
                                                                                                • Glob
                                                                                              Reference/Arvo/Concepts

                                                                                              Subscriptions

                                                                                              Urbit relies heavily on subscriptions throughout the stack. We strongly prefer reactive data flow compared to querying or polling, which is why for example our equivalent to git natively supports subscriptions to its data.

                                                                                              We have many application-level styles of subscription, but we only have three true subscription types in Arvo, named for the vane they're implemented in: Clay subscriptions (filesystem), Gall subscriptions (applications), and Jael subscriptions (Urbit ID info). They're at three different points in the idea maze of subscriptions, and we'll encounter each of them here.

                                                                                              For our purposes, a subscription is a stream of events from a publisher to a subscriber, where (1) the subscriber has indicated interest in that stream, (2) any update sent to one subscriber is sent to all subscribers on the same stream, and (3) when the publisher has a new update ready, they send it immediately to their subscribers instead of waiting for a later request.

                                                                                              Basic properties

                                                                                              Exactly-once updates

                                                                                              Some subscriptions need the subscriber to receive each update separately instead of collapsing them into a single diff. Sometimes a subscription is really a sequence of commands, in which case there's no meaningful way to collapse them. Similarly, software updates may be expected to be applied in order.

                                                                                              Syncing data

                                                                                              Other times, it's more natural to think of a subscription as communicating a data structure, where the only goal is to keep the data structure on the subscriber's side up-to-date with what's on the publisher's side. In this case, it's perfectly fine to collapse several updates into one large update, or even to resend the entire data structure. Partial updates exist only as an optimization.

                                                                                              For this sort of subscription, we commonly use terms like "diff" instead of "message" or "update", and we say that we "apply a diff" instead of "process an update". We also often identify the subscription with the data structure itself.

                                                                                              Memory pressure

                                                                                              Naively implemented, subscriptions pose a memory leak risk if the subscriber can't receive updates as fast as the publisher wants to send them. This commonly happens when the subscriber goes offline for an extended period of time (possibly forever; especially for comets). It's sufficient to bound the amount of memory dedicated to any particular subscriber.

                                                                                              In a synchronous system, you don't have to worry about what happens if one side or the other ceases to be available. All of our subscriptions, though, can happen between ships.

                                                                                              Small data

                                                                                              For particularly low-volume subscriptions, it may be acceptable to ignore the problem. Jael uses this approach, since the only time it's used across ships is to update moon keys, which is very low-volume and less than 1KB per update. If these subscriptions are used for more, this strategy will have to be revisited.

                                                                                              Buffers

                                                                                              A common solution to this problem is the venerable buffer. If more than N updates need to be sent, start dropping updates. There are several options for what to do when the buffer is full and the publisher wishes to send another message:

                                                                                              • Drop the oldest message. Particularly suitable for systems where new messages obsolete old ones. For example, if you just need the latest ticker price for an asset, you could use a buffer size of one and replaced it any time you have a new price.

                                                                                              • Don't add the new message to the buffer. Semantically this is usually less helpful, but mechanically it can be simpler and reduce memory churn, since the buffer stops changing and new messages are simply ignored.

                                                                                              • Drop the newest message and close the subscription. Unlike the previous two, this is compatible with the principle of exactly-once messaging. Specifically, it preserves the guarantee that you won't receive two non-consecutive subscription updates, at the cost of making you resubscribe if you fall behind.

                                                                                              Gall subscriptions use the last of these. The buffer is bounded by a simple heuristic, and when a subscription exceeds that buffer, it is closed and the app is expected to resubscribe when they come back online. Note that even though the subscription is closed, messages in flight cannot be removed until they've been acknowledged since Ames guarantees exactly-once messaging, so those last few messages will be stuck in memory until the subscriber comes back online.

                                                                                              Buffers are commonly bounded by size, but this puts constraints on the types of messages may be sent. They may also be bounded in time: if the subscriber stops taking messages for at least X time, the buffer is full. Gall takes a hybrid approach; currently the heuristic is that the buffer counts as full if it has at least five messages and it has been 30 seconds since a message was acknowledged.

                                                                                              One at a time

                                                                                              Another scheme is to avoid multi-message subscriptions. Instead of allowing the subscriber to subscribe to all future updates on a path, allow them to "subscribe" to the next update only. When they get that update, they subscribe to the new next update, so that they ultimately get every message.

                                                                                              Commonly this is implemented where each update is given a version number, and every time you hear update N you request update N+1 (even though you won't get an answer for an indeterminate period of time).

                                                                                              This is near the theoretical ideal for memory usage, since you'll never queue up several messages to send to the subscriber.

                                                                                              It also encourages processing each update in order rather than zooming forward from time to time. It's possible (if a little awkward) to simulate this with buffers though, and it's possible to allow zooming forward in the one-at-a-time scheme by having the requests be "most recent after update N" instead of "next after update N".

                                                                                              This scheme is at its strongest, then, when you're already storing intermediate states or updates, or when you don't care about intermediate states and always send the most recent one. Otherwise, buffers are a convenient place to temporarily store updates that you don't want to store internally. Of course, if it's semantically important that you give each update separately, buffers won't save you from having to store the intermediate updates, since they may fill up.

                                                                                              Another conceptual advantage is that it has no notion of "connectivity". It operates forever in a single mode. By contrast, buffer-based solutions must keep track of whether the buffer is full and react to that. This increases complexity in the system, and complexity is the great evil.

                                                                                              The biggest disadvantage of this system is that it takes much longer to process many small updates. With buffers, the publisher could have a dozen small updates in flight at the same time, while in this scheme they would only send one at a time, waiting for a response between each one. Losing this pipelining can be quite significant for some applications, but for low-volume subscriptions (certainly anything that updates less than once per minute) this will have no noticeable effect.

                                                                                              Clay uses this scheme. It's low-volume, it stores all previous states anyway (since it's a revision-control system), and it's sometimes important to get updates one at a time (for example when downloading OTA software updates). Ships that send OTA updates often have many ships subscribed to them, and the updates themselves may be large, so it's important they don't queue up when ships inevitably disappear.

                                                                                              I tend to think most Gall subscriptions should follow this pattern as well. Our most latency-sensitive subscriptions are chat messages, and while I haven't tested it, I would guess that even those wouldn't be affected by this scheme.

                                                                                              <-

                                                                                              Scries

                                                                                              Edit this page on GitHub

                                                                                              Last modified March 17, 2023