ventiaux 9.7 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475
  1. .TH VENTIAUX 8
  2. .SH NAME
  3. buildindex,
  4. checkarenas,
  5. checkindex,
  6. copy,
  7. fmtarenas,
  8. fmtindex,
  9. fmtisect,
  10. rdarena,
  11. rdarenablocks,
  12. read,
  13. wrarenablocks,
  14. write \- Venti maintenance and debugging commands
  15. .SH SYNOPSIS
  16. .B venti/buildindex
  17. [
  18. .B -B
  19. .I blockcachesize
  20. ]
  21. [
  22. .B -Z
  23. ]
  24. .I venti.config
  25. .I tmp
  26. .PP
  27. .B venti/checkarenas
  28. [
  29. .B -afv
  30. ]
  31. .I file
  32. .PP
  33. .B venti/checkindex
  34. [
  35. .B -f
  36. ]
  37. [
  38. .B -B
  39. .I blockcachesize
  40. ]
  41. .I venti.config
  42. .I tmp
  43. .PP
  44. .B venti/copy
  45. [
  46. .B -f
  47. ]
  48. .I src
  49. .I dst
  50. .I score
  51. [
  52. .I type
  53. ]
  54. .PP
  55. .B venti/fmtarenas
  56. [
  57. .B -Z
  58. ]
  59. [
  60. .B -a
  61. .I arenasize
  62. ]
  63. [
  64. .B -b
  65. .I blocksize
  66. ]
  67. .I name
  68. .I file
  69. .B venti/fmtindex
  70. [
  71. .B -a
  72. ]
  73. .I venti.config
  74. .PP
  75. .B venti/fmtisect
  76. [
  77. .B -Z
  78. ]
  79. [
  80. .B -b
  81. .I blocksize
  82. ]
  83. .I name
  84. .I file
  85. .PP
  86. .B venti/rdarena
  87. [
  88. .B -v
  89. ]
  90. .I arenapart
  91. .I arenaname
  92. .PP
  93. .B venti/read
  94. [
  95. .B -h
  96. .I host
  97. ]
  98. .I score
  99. [
  100. .I type
  101. ]
  102. .PP
  103. .B venti/wrarena
  104. [
  105. .B -o
  106. .I fileoffset
  107. ]
  108. [
  109. .B -h
  110. .I host
  111. ]
  112. .I arenafile
  113. [
  114. .I clumpoffset
  115. ]
  116. .PP
  117. .B venti/write
  118. [
  119. .B -h
  120. .I host
  121. ]
  122. [
  123. .B -t
  124. .I type
  125. ]
  126. [
  127. .B -z
  128. ]
  129. .SH DESCRIPTION
  130. These commands aid in the setup, maintenance, and debugging of
  131. Venti servers.
  132. See
  133. .IR venti (8)
  134. and
  135. .IR venti.conf (6)
  136. for an overview of the data structures stored by Venti.
  137. .PP
  138. Note that the units for the various sizes in the following
  139. commands can be specified by appending
  140. .LR k ,
  141. .LR m ,
  142. or
  143. .LR g
  144. to indicate kilobytes, megabytes, or gigabytes respectively.
  145. .PP
  146. .I Buildindex
  147. populates the index for the Venti system described in
  148. .IR venti.config .
  149. The index must have previously been formatted using
  150. .IR fmtindex .
  151. This command is typically used to build a new index for a Venti
  152. system when the old index becomes too small, or to rebuild
  153. an index after media failure.
  154. Small errors in an index can usually be fixed with
  155. .IR checkindex .
  156. .PP
  157. The
  158. .I tmp
  159. file, usually a disk partition, must be large enough to store a copy of the index.
  160. This temporary space is used to perform a merge sort of index entries
  161. generated by reading the arenas.
  162. .PP
  163. Options to
  164. .I buildindex
  165. are:
  166. .TP
  167. .BI -B " blockcachesize
  168. The amount of memory, in bytes, to use for caching raw disk accesses while running
  169. .IR buildindex .
  170. (This is not a property of the created index.)
  171. The default is 8k.
  172. .TP
  173. .B -Z
  174. Do not zero the index.
  175. This option should only be used when it is known that the index was already zeroed.
  176. .PD
  177. .PP
  178. .I Checkarenas
  179. examines the Venti arenas contained in the given
  180. .IR file .
  181. The program detects various error conditions, and optionally attempts
  182. to fix any errors that are found.
  183. .PP
  184. Options to
  185. .I checkarenas
  186. are:
  187. .TP
  188. .B -a
  189. For each arena, scan the entire data section.
  190. If this option is omitted, only the end section of
  191. the arena is examined.
  192. .TP
  193. .B -f
  194. Attempt to fix any errors that are found.
  195. .TP
  196. .B -v
  197. Increase the verbosity of output.
  198. .PD
  199. .PP
  200. .I Checkindex
  201. examines the Venti index described in
  202. .IR venti.config .
  203. The program detects various error conditions including:
  204. blocks that are not indexed, index entries for blocks that do not exist,
  205. and duplicate index entries.
  206. If requested, an attempt can be made to fix errors that are found.
  207. .PP
  208. The
  209. .I tmp
  210. file, usually a disk partition, must be large enough to store a copy of the index.
  211. This temporary space is used to perform a merge sort of index entries
  212. generated by reading the arenas.
  213. .PP
  214. Options to
  215. .I checkindex
  216. are:
  217. .TP
  218. .BI -B " blockcachesize
  219. The amount of memory, in bytes, to use for caching raw disk accesses while running
  220. .IR checkindex .
  221. The default is 8k.
  222. .TP
  223. .B -f
  224. Attempt to fix any errors that are found.
  225. .PD
  226. .PP
  227. .I Fmtarenas
  228. formats the given
  229. .IR file ,
  230. typically a disk partition, into a number of
  231. Venti
  232. arenas.
  233. The arenas are given names of the form
  234. .IR name%d ,
  235. where
  236. .I %d
  237. is replaced with a sequential number starting at 0.
  238. .PP
  239. Options to
  240. .I fmtarenas
  241. are:
  242. .TP
  243. .BI -a " arenasize
  244. The arenas are of
  245. .I arenasize
  246. bytes. The default is 512 megabytes, which was selected to provide a balance
  247. between the number of arenas and the ability to copy an arena to external
  248. media such as recordable CDs and tapes.
  249. .TP
  250. .BI -b " blocksize
  251. The size, in bytes, for read and write operations to the file.
  252. The size is recorded in the file, and is used by applications that access the arenas.
  253. The default is 8k.
  254. .TP
  255. .B -Z
  256. Do not zero the data sections of the arenas.
  257. Using this option reduces the formatting time
  258. but should only be used when it is known that the file was already zeroed.
  259. .PD
  260. .I Fmtindex
  261. takes the
  262. .IR venti.conf (6)
  263. file
  264. .I venti.config
  265. and initializes the index sections to form a usable index structure.
  266. The arena files and index sections must have previously been formatted
  267. using
  268. .I fmtarenas
  269. and
  270. .I fmtisect
  271. respectively.
  272. .PP
  273. The function of a Venti index is to map a SHA1 fingerprint to a location
  274. in the data section of one of the arenas. The index is composed of
  275. blocks, each of which contains the mapping for a fixed range of possible
  276. fingerprint values.
  277. .I Fmtindex
  278. determines the mapping between SHA1 values and the blocks
  279. of the collection of index sections. Once this mapping has been determined,
  280. it cannot be changed without rebuilding the index.
  281. The basic assumption in the current implementation is that the index
  282. structure is sufficiently empty that individual blocks of the index will rarely
  283. overflow. The total size of the index should be about 2% to 10% of
  284. the total size of the arenas, but the exact depends both the index block size
  285. and the compressed size of block stored to Venti.
  286. .PP
  287. .I Fmtindex
  288. also computes a mapping between a linear address space and
  289. the data section of the collection of arenas. The
  290. .B -a
  291. option can be used to add additional arenas to an index.
  292. To use this feature,
  293. add the new arenas to
  294. .I venti.config
  295. after the existing arenas and then run
  296. .I fmtindex
  297. .BR -a .
  298. .PP
  299. A copy of the above mappings is stored in the header for each of the index sections.
  300. These copies enable
  301. .I buildindex
  302. to restore a single index section without rebuilding the entire index.
  303. .PP
  304. .I Fmtisect
  305. formats the given
  306. .IR file ,
  307. typically a disk partition, as a Venti index section with the specified
  308. .IR name .
  309. One or more formatted index sections are combined into a Venti
  310. index using
  311. .IR fmtindex .
  312. Each of the index sections within an index must have a unique name.
  313. .PP
  314. Options to
  315. .I fmtisect
  316. are:
  317. .TP
  318. .BI -b " blocksize
  319. The size, in bytes, for read and write operations to the file.
  320. All the index sections within a index must have the same block size.
  321. The default is 8k.
  322. .TP
  323. .B -Z
  324. Do not zero the index.
  325. Using this option reduces the formatting time
  326. but should only be used when it is known that the file was already zeroed.
  327. .PD
  328. .PP
  329. .I Rdarena
  330. extracts the named
  331. .I arena
  332. from the Venti system described by the
  333. .IR venti.conf (6)
  334. file
  335. .I venti.config
  336. and writes this arena to standard output.
  337. This command is typically used to back up an arena to external media.
  338. The
  339. .B -v
  340. option generates more verbose output on standard error.
  341. .PP
  342. .I Wrarena
  343. writes the blocks contained in the arena
  344. .I arenafile
  345. (typically, the output of
  346. .IR rdarena )
  347. to a Venti server.
  348. It is typically used to reinitialize a Venti server from backups of the arenas.
  349. For example,
  350. .IP
  351. .EX
  352. venti/rdarena /dev/sdC0/arenas arena.0 >external.media
  353. venti/wrarena -h venti2 external.media
  354. .EE
  355. .LP
  356. writes the blocks contained in
  357. .B arena.0
  358. to the Venti server
  359. .B venti2
  360. (typically not the one using
  361. .BR /dev/sdC0/arenas ).
  362. .PP
  363. The
  364. .B -o
  365. option specifies that the arena starts at byte
  366. .I fileoffset
  367. (default
  368. .BR 0 )
  369. in
  370. .I arenafile .
  371. This is useful for reading directly from
  372. the Venti arena partition:
  373. .IP
  374. .EX
  375. venti/wrarena -h venti2 -o 335872 /dev/sdC0/arenas
  376. .EE
  377. .LP
  378. (In this example, 335872 is the offset shown in the Venti
  379. server's index list (344064) minus one block (8192).
  380. You will need to substitute your own arena offsets
  381. and block size.)
  382. .PP
  383. Finally, the optional
  384. .I offset
  385. argument specifies that the writing should begin with the
  386. clump starting at
  387. .I offset
  388. within the arena.
  389. .I Wrarena
  390. prints the offset it stopped at (because there were no more data blocks).
  391. This could be used to incrementally back up a Venti server
  392. to another Venti server:
  393. .IP
  394. .EX
  395. last=`{cat last}
  396. venti/wrarena -h venti2 -o 335872 /dev/sdC0/arenas $last >output
  397. awk '/^end offset/ { print $3 }' offset >last
  398. .EE
  399. .LP
  400. Of course, one would need to add wrapper code to keep track
  401. of which arenas have been processed.
  402. See
  403. .B /sys/src/cmd/venti/backup.example
  404. for a version that does this.
  405. .PP
  406. .I Read
  407. and
  408. .I write
  409. read and write blocks from a running Venti server.
  410. They are intended to ease debugging of the server.
  411. The default
  412. .I host
  413. is the environment variable
  414. .BR $venti ,
  415. followed by the network metaname
  416. .BR $venti .
  417. The
  418. .I type
  419. is the decimal type of block to be read or written.
  420. If no
  421. .I type
  422. is specified for
  423. .I read ,
  424. all types are tried, and a command-line is printed to
  425. show the type that eventually worked.
  426. If no
  427. .I type
  428. is specified for
  429. .I write ,
  430. .B VtDataType
  431. (13)
  432. is used.
  433. .I Read
  434. reads the block named by
  435. .I score
  436. (a SHA1 hash)
  437. from the Venti server and writes it to standard output.
  438. .I Write
  439. reads a block from standard input and attempts to write
  440. it to the Venti server.
  441. If successful, it prints the score of the block on the server.
  442. .PP
  443. .I Copy
  444. walks the entire tree of blocks rooted at
  445. .I score ,
  446. copying all the blocks visited during the walk from
  447. the Venti server at network address
  448. .I src
  449. to the Venti server at network address
  450. .I dst .
  451. If
  452. .I type
  453. (a decimal block type for
  454. .IR score )
  455. is omitted, all types will be tried in sequence
  456. until one is found that works.
  457. The
  458. .B -f
  459. flag runs the copy in ``fast'' mode: if a block is already on
  460. .IR dst ,
  461. the walk does not descend below it, on the assumption that all its
  462. children are also already on
  463. .IR dst .
  464. Without this flag, the copy often transfers many times more
  465. data than necessary.
  466. .SH SOURCE
  467. .B /sys/src/cmd/venti
  468. .SH "SEE ALSO"
  469. .IR venti (8),
  470. .IR venti.conf (6)
  471. .SH BUGS
  472. .I Buildindex
  473. should allow an individual index section to be rebuilt.
  474. The merge sort could be performed in the space used to store the
  475. index rather than requiring a temporary file.