This section will be reference documentation for the data types used by our filesystem.
These types are only used inside of Clay. These are only relevant if you're working directly on Clay itself, or trying to understand its inner workings.
The set of changes between the mergebase and one of the desks being merged.
+$ cane$: new=(map path lobe)cal=(map path lobe)can=(map path cage)old=(map path ~)==
newis the set of files in the new desk and not in the mergebase.
calis the set of changes in the new desk from the mergebase except for any that are also in the other new desk.
canis the set of changes in the new desk from the mergebase and that are also in the other new desk (potential conflicts).
oldis the set of files in the mergebase and not in the new desk.
State for ongoing
+$ melt [bas=beak con=(list [beak germ]) sto=(map beak (unit domo))]
conmaintains the ordering.
stostores the data needed to merge.
basis the base
beakfor the merge.
Domestic desk state
+$ dojo$: qyx=cult :: subscribersdom=dome :: desk stateper=regs :: read perms per pathpew=regs :: write perms per pathfiz=melt :: state for mega merges==
This is the all the data that is specific to a particular
desk on a domestic ship.
qyxis the set of subscribers to this
domis the data in the
mapof read permissions by path.
mapof write permissions by path.
fizis the state for ongoing
New desk data
+$ nako :: subscription state$: gar=(map aeon tako) :: new idslet=aeon :: next idlar=(set yaki) :: new commitsbar=~ :: new content== ::
Sent to other ships to update them about a particular desk. Includes a map of all new aeons to hashes of their commits, the most recent aeon, and sets of all new commits and data.
bar is always empty now because we expect you to request any data you don't have yet.
+$ raft :: filesystem$: rom=room :: domestichoy=(map ship rung) :: foreignran=rang :: hashesfad=flow :: ford cachemon=(map term beam) :: mount pointshez=(unit duct) :: sync ductcez=(map @ta crew) :: permission groupstyr=(set duct) :: app substur=rock:tire :: last tirepud=(unit [=desk =yoki]) :: pending updatebug=[veb=@ mas=@] :: verbosity== ::
This is the state of the vane. Anything that must be remembered between calls to Clay is stored in this state.
rom: the state for all local desks. It consists of a
ductto Dill and a collection of
hoy: the state for all foreign desks.
ran: the global, hash-addressed object store. It has maps of commit hashes to commits and content hashes to content.
fad: the global build cache. Each desk has its own fast-lookup index over this global cache.
mon: a collection of Unix mount points.
termis the mount point (relative to th pier) and
beamis a domestic Clay directory.
hez: the duct used to sync with Unix.
cez: a collection of named aermission groups.
tyr: app subscriptions.
tur: records whether apps are running and which kernel versions they're compatible with.
pud: an update that's waiting on a kernel upgrade.
bug: sets Clay's verbosity.
Unvalidated response to a request.
+$ rand :: unvalidated rant$: p=[p=care q=case r=@tas] :: clade release bookq=path :: spurr=page :: data== ::
$rant, but with a page of data rather than a cage of it.
Generic desk state
+$ rede :: universal project$: lim=@da :: complete toref=(unit rind) :: outgoing requestsqyx=cult :: subscribersdom=dome :: revision stateper=regs :: read perms per pathpew=regs :: write perms per pathfiz=melt :: domestic mega merges== ::
This is our knowledge of the state of a desk, either foreign or domestic.
lim: the most recent
@dafor which we're confident we have all the information for. For local
desks, this is always
now. For foriegn
desks, this is the last time we got a full update from the foreign ship.
ref: the request manager for the desk. For domestic
desks, this is null since we handle requests ourselves. For foreign
desks, this keeps track of all pending foriegn requests plus a cache of the responses to previous requests.
setof subscriptions to this desk, with listening
ducts. These subscriptions exist only until they've been filled. For domestic
desks, this is simply
qyx:dojo- all subscribers to the
desk. For foreign
desks this is all the subscribers from our ship to the foreign
dom: the data in the
mapof read permissions by path.
mapof write permissions by path.
fiz: the state for ongoing
Foreign request manager
+$ rind :: request manager$: nix=@ud :: request indexbom=(map @ud update-state) :: outstandingfod=(map duct @ud) :: current requestshaw=(map mood (unit cage)) :: simple cache== ::
When we send a request to a foreign ship, we keep track of it in here.
nix: request counter.
mapof request numbers to requests.
mapof requesters to request numbers.
haw: a simple cache of common
The list of agents that should be automatically started on a desk
+$ bill (list dude:gall)
State of outstanding foreign request
+$ update-state$: =duct=raveneed=(list lobe)nako=(qeu (unit nako))busy=_|==
update-state is used to represent the status of an outstanding request to a foreign
duct: the duct along which the request was made.
rave: the request itself.
need: a list of hashes yet to be acquired.
nako: a queue of data yet to be validated.
busy: tracks whether or not the request is currently being fulfilled.
Filesystem per domestic ship
+$ room :: fs per ship$: hun=duct :: terminal ductdos=(map desk dojo) :: native desk== ::
This is the representation of the filesystem of a ship on our pier.
hun: the duct we use to send messages to Dill to display notifications of filesystem changes. Only
%gifts should be produced along this
duct. This is set by the
dos: the set of
desks on this ship, mapped to their
Cached result of a request
+$ cach (unit (unit cage)) :: cached result
Stored source and request
+$ wove [for=(unit [=ship ver=@ud]) =rove] :: stored source + req
+$ rove :: stored request$% [%sing =mood] :: single request[%next =mood aeon=(unit aeon) =cach] :: next version of one$: %mult :: next version of any=mool :: original requestaeon=(unit aeon) :: checking for changeold-cach=(map [=care =path] cach) :: old versionnew-cach=(map [=care =path] cach) :: new version== ::[%many track=? =moat lobes=(map path lobe)] :: change range== ::
$rave but with caches of current versions for
%many. Generally used when we store a request in our state somewhere.
Foreign desk data
+$ rung$: rus=(map desk rede) :: neighbor desks==
This contains the filesystem of a neighbour ship. The keys to this
map are all the
desks we know about on their ship.
These types are defined in
lull.hoon, and are used in Clay's external interface.
Desk revision number
+$ aeon @ud
+$ beam [[p=ship q=desk r=case] s=path]
The full path to a file or directory.
+$ beak [p=ship q=desk r=case]
$beam sans the specific file path.
+$ cable$: face=(unit term)file-path=term==
+$ care$? %a %b %c %d %e %f%p %q %r %s %t %u%v %w %x %y %z==
This specifies what type of information is requested in a subscription or a scry.
%a: build a Hoon file at a
%b: build a dynamically typed
markby name (a
%c: build a dynamically typed
markconversion gate (a
$tube) by "from" and "to"
%d: returns a
(set desk)of the
desks that exist on your ship.
%e: builds a statically typed
markby name (a
%f: builds a statically typed mark converstion gate.
%p: produces the permissions for a directory, returned as a
%r: requests the file in the same fashion as
%x, but wraps the result in a
%s: has miscellaneous debug endpoints.
%t: produces a
(list path)of descendent
paths for a directory within a
%u: produces a
?depending on whether or not the specified file exists. It does not check any of its children.
%v: requests the entire
domefor a specified
deskat a particular
aeon. When used on a foreign
desk, this get us up-to-date to the requested version.
%w: requests the revision number and date of the specified path, returned as a
%x: requests the file at a specified path at the specified commit, returned as an
@. If there is no node at that path or if the node has no contents (that is, if
fil:ankhis null), then this crashes.
%y: requests an
archof the specfied commit at the specified path. It will return the bunt of an
archif the file or directory is not found.
%z: requests a recursive hash of a node and all its children, returned as a
See the scry reference for more details.
+$ cash$% [%tako p=tako]case==
Cases for revision
+$ cass [ud=@ud da=@da]
This is returned by a
+$ clue$% [%lobe =lobe][%all ~][%pick ~][%norm =ship =desk =norm][%worn =ship =desk =tako =norm][%seek =ship =desk =cash]==
%all: all safe tombstone targets.
%pick: collect garbage.
%norm: set default
normfor a specific commit.
%seek: backfill tombstoned commit.
+$ cone (map [ship desk] dome)
+$ dome$: let=aeonhit=(map aeon tako)lab=(map @tas aeon)tom=(map tako norm)nor=normmim=(map path mime)fod=fluewic=(map weft yoki)liv=zestren=rein==
dome is the state of a
desk and associated data.
letis the number of the most recently numbered commit. This is also the total number of numbered commits.
hitis a map of numerical IDs to commit hashes. These hashes are mapped into their associated commits in the
$rangof the the
$raftof Clay. In general, the keys of this map are exactly the numbers from 1 to
let, with no gaps. Of course, when there are no numbered commits,
letis 0, so
hitis null. Additionally, each of the commits is an ancestor of every commit numbered greater than this one. Thus, each is a descendant of every commit numbered less than this one. Since it is true that the date in each commit (
t.yaki) is no earlier than that of each of its parents, the numbered commits are totally ordered in the same way by both pedigree and date. If that sounds too complicated to you, don't worry about it. It basically behaves exactly as you would expect.
labis a map of textual labels to numbered commits. Labels must be unique across a desk.
tomcontains the tombstoning policies for all files in the desk.
noris the default tombstoning policy.
mimis a cache of the content in the directories that are mounted to Unix.
fodis the Ford cache, which keeps a cache of the results of builds performed at this
desk's current revision, including a full transitive closure of dependencies for each completed build.
wiccontains commits waiting for future kernel versions.
livsays whether agents on the desk are running or suspended.
renrecords which agents have been forced on or off, differing from the desk's
+$ crew (set ship)
+$ dict [src=path rul=real]
+$ dome$: let=@udhit=(map @ud tako)lab=(map @tas @ud)==
let: current revision number
hit: map of revisions to their commit hashes
lab: map of labels to their revision numbers
+$ germ :: merge style$? %init :: new desk%fine :: fast forward%meet :: orthogonal files%mate :: orthogonal changes%meld :: force merge%only-this :: ours with parents%only-that :: hers with parents%take-this :: ours unless absent%take-that :: hers unless absent%meet-this :: ours if conflict%meet-that :: hers if conflict== ::
See the Strategies section of "Using Clay" for further details of their meaning.
+$ lobe @uvI
This is a hash of a
page. These are most notably used in
lat.rang, where they are associated with the actual
page, and as the values in
paths are associated with their content hashes in a commit.
+$ miso$% [%del ~][%ins p=cage][%dif p=cage][%mut p=cage]==
There are four kinds of changes that may be made to a node in a
%del: deletes the node.
%ins: inserts a file given by
%dif: currently unimplemented. This may seem strange, so we remark that diffs for individual files are implemented using
marks. So for an
ankh, which may include both files and directories,
%difbeing unimplemented really just means that we do not yet have a formal concept of changes in directory structure.
%mut: mutates the file using raw data given by
+$ misu$% [%del ~][%ins p=cage][%dif p=lobe q=cage]==
%del: deletes the node.
%ins: inserts file
%dif: currently unimplemented.
+$ mizu [p=@u q=(map @ud tako) r=rang]
Normal change range
+$ moar [p=@ud q=@ud]
Range subscription request
+$ moat [from=case to=case =path]
This represents a request for all changes between
path. You will be notified when a change is made to the node referenced by the
path or to any of its children.
+$ mode (list [path (unit mime)])
This is used when there's a commit from the host system.
Single subscription request
+$ mood [=care =case =path]
This represents a request for data related to the state of the
desk at a particular commit, specfied by
care specifies what kind of information is desired, and the
path specifies the path we are requesting.
Requests in desk
+$ mool [=case paths=(set (pair care path))]
This is used in a
rave to specify the next version of multiple files with multiple cares.
+$ nori$% [%& p=soba][%| p=@tas q=(unit aeon)]==
This describes a change that we are asking Clay to make to the
desk. There are two kinds of changes that may be made: we can modify files or we can apply a label to a commit.
& case, we will apply the given changes. In the
| case, we will apply the given label to the commit specified in
q, or the current one if it's null.
+$ nuri$% [%& p=suba][%| p=@tas]==
+$ norm (axal ?)
axal is a recursive directory structure. For each file, a
? says whether it should be tombstoned or not.
+$ open $-(path vase)
This defines the type of a function for retrieving imported files.
A raw, unvalidated file.
+$ page ^page
This is just the
page defined in
arvo.hoon: a pair of a mark and a noun.
Ford build with content.
+$ pour$% [%file =path][%nave =mark][%dais =mark][%cast =mars][%tube =mars]:: leafs::[%vale =path =lobe][%arch =path =(map path lobe)]==
+$ rang$: hut=(map tako yaki)lat=(map lobe page)==
This is a data repository keyed by hash. Thus, this is where the "real" data is stored, but it is only meaningful if we know the hash of what we're looking for.
lat is a
map from content hashes (
lobes) to the actual content (
pages). We often get the hashes from a
yaki, which references this
map to get the data. There is no
yaki. They are only accessible through
+$ rant$: p=[p=care q=case r=desk]q=pathr=cage==
This is the data associated to the response to a request.
p.p: specifies the type of data that was requested (and is produced).
q.p: gives the specific version reported (since a range of versions may be requested in a subscription).
q: the path to the filesystem node.
r: is the data itself (in the format specified by
General subscription request
+$ rave$% [%sing =mood][%next =mood][%mult =mool][%many track=? =moat]==
This represents a subscription request for a
%sing: asks for data at single revision.
%next: asks to be notified the next time there’s a change to the specified file.
%mult: asks to be notified the next time there's a change to a specified set of files.
%many: asks to be notified on every change in a
deskfor a range of changes (including into the future).
+$ real$: mod=?(%black %white)who=(pair (set ship) (map @ta crew))==
mod: whether it's a blacklist or whitelist.
who: the ships who are blacklisted/whitelisted. It can have both individual ships as well as
Permission rules for paths
+$ regs (map path rule)
A map from file/directory paths to permission
Forced on/off apps
+$ rein (map dude:gall ?)
dude:gall is the name of a Gall agent and the
? is whether it's forced on or off. An app is forced when it's started despite not being on the
desk.bill manifest or stopped when it is on the manifest.
+$ riff [p=desk q=(unit rave)]
This represents a request for data about a particular
q contains a
rave, then this opens a subscription to the
desk for that data. If
q is null, then this tells Clay to cancel the subscription along this duct.
+$ riff-any$% [%1 =riff]==
Currently there's only one version.
+$ rite$% [%r red=(unit rule)][%w wit=(unit rule)][%rw red=(unit rule) wit=(unit rule)]==
%r: read permissions.
%w: write permissions.
%rw: both read and write permissions.
+$ riot (unit rant)
riot is a response to a subscription. If null, the subscription has been completed, and no more responses will be sent. Otherwise, the
rant is the produced data.
+$ rule [mod=?(%black %white) who=(set whom)]
mod: whether it's a blacklist or whitelist.
who: the ships or permission groups on the list.
+$ rump [p=care q=case r=@tas s=path]
Patch plus merge
+$ saba [p=ship q=@tas r=moar s=dome]
+$ soak$% [%cage =cage][%vase =vase][%arch dir=(map @ta vase)][%dais =dais][%tube =tube]==
The actual data in the Ford cache.
+$ soba (list [p=path q=miso])
This describes a
list of changes to make to a
paths to files to be changed, and the corresponding
miso value is a description of the change itself.
+$ suba (list [p=path q=misu])
+$ tako @uvI
This is a hash of a
yaki, a commit. These are most notably used as the keys in
hut.rang, where they are associated with the actual
yaki, and as the values in
hit.dome, where sequential numerical ids are associated with these.
+$ toro [p=@ta q=nori]
Change part of a list.
++ unce|* a=mold$% [%& p=@ud][%| p=(list a) q=(list a)]==
This is a single change in a list of elements of type
a. For example,
(unce @t) is a single change in lines of text.
%&: the next
plines are unchanged.
%|: the lines
phave changed to
++ urge |*(a=mold (list (unce a)))
This is a parametrized type for list changes. For example,
(urge @t) is a list change for lines of text.
+$ waft$^ [[%1 ~] p=(set weft)]weft
waft is the result of reading a
sys.kelvin file in a desk. It lists all the
wefts (kernel versions) a desk is compatible with. It may either be a single
[%zuse 417], or a range like:
[[%1 ~] (silt zuse+417 zuse+416 ~)]
Ship or named crew
+$ whom (each ship @ta)
Either a single ship or a set of ships in a
crew (permission group). This is used for read/write permissions.
+$ yoki (each yuki yaki)
+$ yuki$: p=(list tako)q=(map path (each page lobe))==
yuki is a proto-commit: a new, proposed commit that has not yet been finalized. This is in contrast to a
yaki. The main difference is that a
yuki may contain actual data, while a
yaki only contains
lobes ( content hashes used as references to data in the general object store).
p: commit references of any parents.
mapfrom file paths to either
+$ yaki$: p=(list tako)q=(map path lobe)r=takot=@da==
listof the hashes of the parents of this commit. In most cases, this will be a single commit, but in a merge there may be more parents.
q: is a
paths on a desk to the content hashes at that location. If you understand what a
pageis, then the type signature here tells the whole story.
r: is the hash associated with this commit.
t: is the date at which this commit was made.
+$ zest $~(%dead ?(%dead %live %held))
This represents the state of apps on the desk.
%held: suspended pending compatible system update.
+$ rock (map desk [=zest wic=(set weft)])
zestsays whether the desk is running or suspended.
wicis the set of kernel versions (
wefts) for which the desk has queued commits awaiting kernel updates.
App state changes
+$ wave$% [%wait =desk =weft][%warp =desk =weft][%zest =desk =zest]==
%wait: blocked commit by
%warp: unblocked commit by
%zest: desk is now running/suspended/held.
Ford cache key
+$ leak$~ [*pour ~]$: =pourdeps=(set leak)==
This includes all build inputs, including transitive dependencies, recursively.
Global Ford cache
+$ flow (map leak [refs=@ud =soak])
Refcount includes references from other items in the cache, and from
spills in each desk.
This is optimized for minimizing the number of rebuilds, and given that, minimizing the amount of memory used. It is relatively slow to lookup, because generating a cache key can be fairly slow (for files, it requires parsing; for
tubes, it even requires building the marks).
Per-desk build cache
+$ flue [spill=(set leak) sprig=(map mist [=leak =soak])]
spillis the set of "roots" we have into the global ford cache. We add a root for everything referenced directly or indirectly on a desk, then invalidate them on commit only if their dependencies change.
sprigis a fast-lookup index over the global ford cache. The only goal is to make cache hits fast.
Ford build without content
+$ mist$% [%file =path][%nave =mark][%dais =mark][%cast =mars][%tube =mars][%vale =path][%arch =path]==
This is used at the index of
Preprocessed hoon source file
+$ pile$: sur=(list taut)lib=(list taut)raw=(list [face=term =path])raz=(list [face=term =spec =path])maz=(list [face=term =mark])caz=(list [face=term =mars])bar=(list [face=term =mark =path])=hoon==
sur: surface imports from
lib: library imports from
raw: imports built hoon file at path (
raz: imports built hoon files from directory (
maz: imports mark definition from
caz: imports mark converter from
bar: unbuilt file imports, as mark (
hoon: the rest of the hoon file.
File import from
+$ taut [face=(unit term) pax=term]
Mark conversion request
+$ mars [a=mark b=mark]
Mark conversion gate
+$ tube $-(vase vase)
A gate that takes a
vase and produces a
vase. This is the type of mark convertion gate returned by
%c-care scries and read requests.