ventiaux 10 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504
  1. .TH VENTIAUX 8
  2. .SH NAME
  3. buildindex,
  4. checkarenas,
  5. checkindex,
  6. conf,
  7. copy,
  8. fmtarenas,
  9. fmtindex,
  10. fmtisect,
  11. rdarena,
  12. rdarenablocks,
  13. read,
  14. wrarenablocks,
  15. write \- Venti maintenance and debugging commands
  16. .SH SYNOPSIS
  17. .B venti/buildindex
  18. [
  19. .B -B
  20. .I blockcachesize
  21. ]
  22. [
  23. .B -Z
  24. ]
  25. .I venti.config
  26. .I tmp
  27. .PP
  28. .B venti/checkarenas
  29. [
  30. .B -afv
  31. ]
  32. .I file
  33. .PP
  34. .B venti/checkindex
  35. [
  36. .B -f
  37. ]
  38. [
  39. .B -B
  40. .I blockcachesize
  41. ]
  42. .I venti.config
  43. .I tmp
  44. .PP
  45. .B venti/conf
  46. [
  47. .B -w
  48. ]
  49. .I partition
  50. [
  51. .I configfile
  52. ]
  53. .PP
  54. .B venti/copy
  55. [
  56. .B -f
  57. ]
  58. .I src
  59. .I dst
  60. .I score
  61. [
  62. .I type
  63. ]
  64. .PP
  65. .B venti/fmtarenas
  66. [
  67. .B -Z
  68. ]
  69. [
  70. .B -a
  71. .I arenasize
  72. ]
  73. [
  74. .B -b
  75. .I blocksize
  76. ]
  77. .I name
  78. .I file
  79. .PP
  80. .B venti/fmtindex
  81. [
  82. .B -a
  83. ]
  84. .I venti.config
  85. .PP
  86. .B venti/fmtisect
  87. [
  88. .B -Z
  89. ]
  90. [
  91. .B -b
  92. .I blocksize
  93. ]
  94. .I name
  95. .I file
  96. .PP
  97. .B venti/rdarena
  98. [
  99. .B -v
  100. ]
  101. .I arenapart
  102. .I arenaname
  103. .PP
  104. .B venti/read
  105. [
  106. .B -h
  107. .I host
  108. ]
  109. .I score
  110. [
  111. .I type
  112. ]
  113. .PP
  114. .B venti/wrarena
  115. [
  116. .B -o
  117. .I fileoffset
  118. ]
  119. [
  120. .B -h
  121. .I host
  122. ]
  123. .I arenafile
  124. [
  125. .I clumpoffset
  126. ]
  127. .PP
  128. .B venti/write
  129. [
  130. .B -h
  131. .I host
  132. ]
  133. [
  134. .B -t
  135. .I type
  136. ]
  137. [
  138. .B -z
  139. ]
  140. .SH DESCRIPTION
  141. These commands aid in the setup, maintenance, and debugging of
  142. Venti servers.
  143. See
  144. .IR venti (8)
  145. and
  146. .IR venti.conf (6)
  147. for an overview of the data structures stored by Venti.
  148. .PP
  149. Note that the units for the various sizes in the following
  150. commands can be specified by appending
  151. .LR k ,
  152. .LR m ,
  153. or
  154. .LR g
  155. to indicate kilobytes, megabytes, or gigabytes respectively.
  156. .PP
  157. .I Buildindex
  158. populates the index for the Venti system described in
  159. .IR venti.config .
  160. The index must have previously been formatted using
  161. .IR fmtindex .
  162. This command is typically used to build a new index for a Venti
  163. system when the old index becomes too small, or to rebuild
  164. an index after media failure.
  165. Small errors in an index can usually be fixed with
  166. .IR checkindex .
  167. .PP
  168. The
  169. .I tmp
  170. file, usually a disk partition, must be large enough to store a copy of the index.
  171. This temporary space is used to perform a merge sort of index entries
  172. generated by reading the arenas.
  173. .PP
  174. Options to
  175. .I buildindex
  176. are:
  177. .TP
  178. .BI -B " blockcachesize
  179. The amount of memory, in bytes, to use for caching raw disk accesses while running
  180. .IR buildindex .
  181. (This is not a property of the created index.)
  182. The default is 8k.
  183. .TP
  184. .B -Z
  185. Do not zero the index.
  186. This option should only be used when it is known that the index was already zeroed.
  187. .PD
  188. .PP
  189. .I Checkarenas
  190. examines the Venti arenas contained in the given
  191. .IR file .
  192. The program detects various error conditions, and optionally attempts
  193. to fix any errors that are found.
  194. .PP
  195. Options to
  196. .I checkarenas
  197. are:
  198. .TP
  199. .B -a
  200. For each arena, scan the entire data section.
  201. If this option is omitted, only the end section of
  202. the arena is examined.
  203. .TP
  204. .B -f
  205. Attempt to fix any errors that are found.
  206. .TP
  207. .B -v
  208. Increase the verbosity of output.
  209. .PD
  210. .PP
  211. .I Checkindex
  212. examines the Venti index described in
  213. .IR venti.config .
  214. The program detects various error conditions including:
  215. blocks that are not indexed, index entries for blocks that do not exist,
  216. and duplicate index entries.
  217. If requested, an attempt can be made to fix errors that are found.
  218. .PP
  219. The
  220. .I tmp
  221. file, usually a disk partition, must be large enough to store a copy of the index.
  222. This temporary space is used to perform a merge sort of index entries
  223. generated by reading the arenas.
  224. .PP
  225. Options to
  226. .I checkindex
  227. are:
  228. .TP
  229. .BI -B " blockcachesize
  230. The amount of memory, in bytes, to use for caching raw disk accesses while running
  231. .IR checkindex .
  232. The default is 8k.
  233. .TP
  234. .B -f
  235. Attempt to fix any errors that are found.
  236. .PD
  237. .PP
  238. .I Fmtarenas
  239. formats the given
  240. .IR file ,
  241. typically a disk partition, into a number of
  242. Venti
  243. arenas.
  244. The arenas are given names of the form
  245. .IR name%d ,
  246. where
  247. .I %d
  248. is replaced with a sequential number starting at 0.
  249. .PP
  250. Options to
  251. .I fmtarenas
  252. are:
  253. .TP
  254. .BI -a " arenasize
  255. The arenas are of
  256. .I arenasize
  257. bytes. The default is 512 megabytes, which was selected to provide a balance
  258. between the number of arenas and the ability to copy an arena to external
  259. media such as recordable CDs and tapes.
  260. .TP
  261. .BI -b " blocksize
  262. The size, in bytes, for read and write operations to the file.
  263. The size is recorded in the file, and is used by applications that access the arenas.
  264. The default is 8k.
  265. .TP
  266. .B -Z
  267. Do not zero the data sections of the arenas.
  268. Using this option reduces the formatting time
  269. but should only be used when it is known that the file was already zeroed.
  270. .PD
  271. .I Fmtindex
  272. takes the
  273. .IR venti.conf (6)
  274. file
  275. .I venti.config
  276. and initializes the index sections to form a usable index structure.
  277. The arena files and index sections must have previously been formatted
  278. using
  279. .I fmtarenas
  280. and
  281. .I fmtisect
  282. respectively.
  283. .PP
  284. The function of a Venti index is to map a SHA1 fingerprint to a location
  285. in the data section of one of the arenas. The index is composed of
  286. blocks, each of which contains the mapping for a fixed range of possible
  287. fingerprint values.
  288. .I Fmtindex
  289. determines the mapping between SHA1 values and the blocks
  290. of the collection of index sections. Once this mapping has been determined,
  291. it cannot be changed without rebuilding the index.
  292. The basic assumption in the current implementation is that the index
  293. structure is sufficiently empty that individual blocks of the index will rarely
  294. overflow. The total size of the index should be about 2% to 10% of
  295. the total size of the arenas, but the exact depends both the index block size
  296. and the compressed size of block stored to Venti.
  297. .PP
  298. .I Fmtindex
  299. also computes a mapping between a linear address space and
  300. the data section of the collection of arenas. The
  301. .B -a
  302. option can be used to add additional arenas to an index.
  303. To use this feature,
  304. add the new arenas to
  305. .I venti.config
  306. after the existing arenas and then run
  307. .I fmtindex
  308. .BR -a .
  309. .PP
  310. A copy of the above mappings is stored in the header for each of the index sections.
  311. These copies enable
  312. .I buildindex
  313. to restore a single index section without rebuilding the entire index.
  314. .PP
  315. .I Fmtisect
  316. formats the given
  317. .IR file ,
  318. typically a disk partition, as a Venti index section with the specified
  319. .IR name .
  320. One or more formatted index sections are combined into a Venti
  321. index using
  322. .IR fmtindex .
  323. Each of the index sections within an index must have a unique name.
  324. .PP
  325. Options to
  326. .I fmtisect
  327. are:
  328. .TP
  329. .BI -b " blocksize
  330. The size, in bytes, for read and write operations to the file.
  331. All the index sections within a index must have the same block size.
  332. The default is 8k.
  333. .TP
  334. .B -Z
  335. Do not zero the index.
  336. Using this option reduces the formatting time
  337. but should only be used when it is known that the file was already zeroed.
  338. .PD
  339. .PP
  340. .I Rdarena
  341. extracts the named
  342. .I arena
  343. from the arena partition
  344. .I arenapart
  345. and writes this arena to standard output.
  346. This command is typically used to back up an arena to external media.
  347. The
  348. .B -v
  349. option generates more verbose output on standard error.
  350. .PP
  351. .I Wrarena
  352. writes the blocks contained in the arena
  353. .I arenafile
  354. (typically, the output of
  355. .IR rdarena )
  356. to a Venti server.
  357. It is typically used to reinitialize a Venti server from backups of the arenas.
  358. For example,
  359. .IP
  360. .EX
  361. venti/rdarena /dev/sdC0/arenas arena.0 >external.media
  362. venti/wrarena -h venti2 external.media
  363. .EE
  364. .LP
  365. writes the blocks contained in
  366. .B arena.0
  367. to the Venti server
  368. .B venti2
  369. (typically not the one using
  370. .BR /dev/sdC0/arenas ).
  371. .PP
  372. The
  373. .B -o
  374. option specifies that the arena starts at byte
  375. .I fileoffset
  376. (default
  377. .BR 0 )
  378. in
  379. .I arenafile .
  380. This is useful for reading directly from
  381. the Venti arena partition:
  382. .IP
  383. .EX
  384. venti/wrarena -h venti2 -o 335872 /dev/sdC0/arenas
  385. .EE
  386. .LP
  387. (In this example, 335872 is the offset shown in the Venti
  388. server's index list (344064) minus one block (8192).
  389. You will need to substitute your own arena offsets
  390. and block size.)
  391. .PP
  392. Finally, the optional
  393. .I offset
  394. argument specifies that the writing should begin with the
  395. clump starting at
  396. .I offset
  397. within the arena.
  398. .I Wrarena
  399. prints the offset it stopped at (because there were no more data blocks).
  400. This could be used to incrementally back up a Venti server
  401. to another Venti server:
  402. .IP
  403. .EX
  404. last=`{cat last}
  405. venti/wrarena -h venti2 -o 335872 /dev/sdC0/arenas $last >output
  406. awk '/^end offset/ { print $3 }' offset >last
  407. .EE
  408. .LP
  409. Of course, one would need to add wrapper code to keep track
  410. of which arenas have been processed.
  411. See
  412. .B /sys/src/cmd/venti/backup.example
  413. for a version that does this.
  414. .PP
  415. .I Read
  416. and
  417. .I write
  418. read and write blocks from a running Venti server.
  419. They are intended to ease debugging of the server.
  420. The default
  421. .I host
  422. is the environment variable
  423. .BR $venti ,
  424. followed by the network metaname
  425. .BR $venti .
  426. The
  427. .I type
  428. is the decimal type of block to be read or written.
  429. If no
  430. .I type
  431. is specified for
  432. .I read ,
  433. all types are tried, and a command-line is printed to
  434. show the type that eventually worked.
  435. If no
  436. .I type
  437. is specified for
  438. .I write ,
  439. .B VtDataType
  440. (13)
  441. is used.
  442. .I Read
  443. reads the block named by
  444. .I score
  445. (a SHA1 hash)
  446. from the Venti server and writes it to standard output.
  447. .I Write
  448. reads a block from standard input and attempts to write
  449. it to the Venti server.
  450. If successful, it prints the score of the block on the server.
  451. .PP
  452. .I Copy
  453. walks the entire tree of blocks rooted at
  454. .I score ,
  455. copying all the blocks visited during the walk from
  456. the Venti server at network address
  457. .I src
  458. to the Venti server at network address
  459. .I dst .
  460. If
  461. .I type
  462. (a decimal block type for
  463. .IR score )
  464. is omitted, all types will be tried in sequence
  465. until one is found that works.
  466. The
  467. .B -f
  468. flag runs the copy in ``fast'' mode: if a block is already on
  469. .IR dst ,
  470. the walk does not descend below it, on the assumption that all its
  471. children are also already on
  472. .IR dst .
  473. Without this flag, the copy often transfers many times more
  474. data than necessary.
  475. .PP
  476. To make it easier to bootstrap servers, the configuration
  477. file can be stored at the beginning of any Venti partitions using
  478. .IR conf .
  479. A partition so branded with a configuration file can
  480. be used in place of a configuration file when invoking any
  481. of the venti commands.
  482. By default,
  483. .I conf
  484. prints the configuration stored in
  485. .IR partition .
  486. When invoked with the
  487. .B -w
  488. flag,
  489. .I conf
  490. reads a configuration file from
  491. .I configfile
  492. (or else standard input)
  493. and stores it in
  494. .IR partition .
  495. .SH SOURCE
  496. .B /sys/src/cmd/venti
  497. .SH "SEE ALSO"
  498. .IR venti (8),
  499. .IR venti.conf (6)
  500. .SH BUGS
  501. .I Buildindex
  502. should allow an individual index section to be rebuilt.
  503. The merge sort could be performed in the space used to store the
  504. index rather than requiring a temporary file.