venti-fmt 8.5 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408
  1. .TH VENTI-FMT 8
  2. .SH NAME
  3. buildindex,
  4. checkarenas,
  5. checkindex,
  6. conf,
  7. fmtarenas,
  8. fmtbloom,
  9. fmtindex,
  10. fmtisect,
  11. syncindex \- prepare and maintain a venti server
  12. .SH SYNOPSIS
  13. .PP
  14. .B venti/fmtarenas
  15. [
  16. .B -Z
  17. ]
  18. [
  19. .B -a
  20. .I arenasize
  21. ]
  22. [
  23. .B -b
  24. .I blocksize
  25. ]
  26. .I name
  27. .I file
  28. .PP
  29. .B venti/fmtisect
  30. [
  31. .B -1Z
  32. ]
  33. [
  34. .B -b
  35. .I blocksize
  36. ]
  37. .I name
  38. .I file
  39. .PP
  40. .B venti/fmtbloom
  41. [
  42. .B -n
  43. .I nblocks
  44. |
  45. .B -N
  46. .I nhash
  47. ]
  48. [
  49. .B -s
  50. .I size
  51. ]
  52. .I file
  53. .PP
  54. .B venti/fmtindex
  55. [
  56. .B -a
  57. ]
  58. .I venti.conf
  59. .PP
  60. .B venti/conf
  61. [
  62. .B -w
  63. ]
  64. .I partition
  65. [
  66. .I configfile
  67. ]
  68. .if t .sp 0.5
  69. .PP
  70. .B venti/buildindex
  71. [
  72. .B -bd
  73. ] [
  74. .B -i
  75. .I isect
  76. ] ... [
  77. .B -M
  78. .I imemsize
  79. ]
  80. .I venti.conf
  81. .PP
  82. .B venti/checkindex
  83. [
  84. .B -f
  85. ]
  86. [
  87. .B -B
  88. .I blockcachesize
  89. ]
  90. .I venti.conf
  91. .I tmp
  92. .PP
  93. .B venti/checkarenas
  94. [
  95. .B -afv
  96. ]
  97. .I file
  98. .SH DESCRIPTION
  99. These commands aid in the setup, maintenance, and debugging of
  100. venti servers.
  101. See
  102. .IR venti (6)
  103. for an overview of the venti system and
  104. .IR venti (8)
  105. for an overview of the data structures used by the venti server.
  106. .PP
  107. Note that the units for the various sizes in the following
  108. commands can be specified by appending
  109. .LR k ,
  110. .LR m ,
  111. or
  112. .LR g
  113. to indicate kilobytes, megabytes, or gigabytes respectively.
  114. .SS Formatting
  115. To prepare a server for its initial use, the arena partitions and
  116. the index sections must be formatted individually, with
  117. .I fmtarenas
  118. and
  119. .IR fmtisect .
  120. Then the
  121. collection of index sections must be combined into a venti
  122. index with
  123. .IR fmtindex .
  124. .PP
  125. .I Fmtarenas
  126. formats the given
  127. .IR file ,
  128. typically a disk partition, into an arena partition.
  129. The arenas in the partition are given names of the form
  130. .IR name%d ,
  131. where
  132. .I %d
  133. is replaced with a sequential number starting at 0.
  134. .PP
  135. Options to
  136. .I fmtarenas
  137. are:
  138. .TP
  139. .BI -a " arenasize
  140. The arenas are of
  141. .I arenasize
  142. bytes. The default is
  143. .BR 512M ,
  144. which was selected to provide a balance
  145. between the number of arenas and the ability to copy an arena to external
  146. media such as recordable CDs and tapes.
  147. .TP
  148. .BI -b " blocksize
  149. The size, in bytes, for read and write operations to the file.
  150. The size is recorded in the file, and is used by applications that access the arenas.
  151. The default is
  152. .BR 8k .
  153. .TP
  154. .B -4
  155. Create a `version 4' arena partition for backwards compatibility with old servers.
  156. The default is version 5, used by the current venti server.
  157. .TP
  158. .B -Z
  159. Do not zero the data sections of the arenas.
  160. Using this option reduces the formatting time
  161. but should only be used when it is known that the file was already zeroed.
  162. (Version 4 only; version 5 sections are not and do not need to be zeroed.)
  163. .PD
  164. .PP
  165. .I Fmtisect
  166. formats the given
  167. .IR file ,
  168. typically a disk partition, as a venti index section with the specified
  169. .IR name .
  170. Each of the index sections in a venti configuration must have a unique name.
  171. .PP
  172. Options to
  173. .I fmtisect
  174. are:
  175. .TP
  176. .BI -b " bucketsize
  177. The size of an index bucket, in bytes.
  178. All the index sections within a index must have the same bucket size.
  179. The default is
  180. .BR 8k .
  181. .TP
  182. .B -1
  183. Create a `version 1' index section for backwards compatibility with old servers.
  184. The default is version 2, used by the current venti server.
  185. .TP
  186. .B -Z
  187. Do not zero the index.
  188. Using this option reduces the formatting time
  189. but should only be used when it is known that the file was already zeroed.
  190. (Version 1 only; version 2 sections are not and do not need to be zeroed.)
  191. .PD
  192. .PP
  193. .I Fmtbloom
  194. formats the given
  195. .I file
  196. as a Bloom filter
  197. (see
  198. .IR venti (6)).
  199. The options are:
  200. .TF "\fL-s\fI size"
  201. .PD
  202. .TP
  203. .BI -n " nblock \fR| " -N " nhash
  204. The number of blocks expected to be indexed by the filter
  205. or the number of hash functions to use.
  206. If the
  207. .B -n
  208. option
  209. is given, it is used, along with the total size of the filter,
  210. to compute an appropriate
  211. .IR nhash .
  212. .TP
  213. .BI -s " size
  214. The size of the Bloom filter. The default is the total size of the file.
  215. In either case,
  216. .I size
  217. is rounded down to a power of two.
  218. .PD
  219. .PP
  220. The
  221. .I file
  222. argument in the commands above can be of the form
  223. .IB file : lo - hi
  224. to specify a range of the file.
  225. .I Lo
  226. and
  227. .I hi
  228. are specified in bytes but can have the usual
  229. .BI k ,
  230. .BI m ,
  231. or
  232. .B g
  233. suffixes.
  234. Either
  235. .I lo
  236. or
  237. .I hi
  238. may be omitted.
  239. This notation eliminates the need to
  240. partition raw disks on non-Plan 9 systems.
  241. .PP
  242. .I Fmtindex
  243. reads the configuration file
  244. .I venti.conf
  245. and initializes the index sections to form a usable index structure.
  246. The arena files and index sections must have previously been formatted
  247. using
  248. .I fmtarenas
  249. and
  250. .I fmtisect
  251. respectively.
  252. .PP
  253. The function of a venti index is to map a SHA1 fingerprint to a location
  254. in the data section of one of the arenas. The index is composed of
  255. blocks, each of which contains the mapping for a fixed range of possible
  256. fingerprint values.
  257. .I Fmtindex
  258. determines the mapping between SHA1 values and the blocks
  259. of the collection of index sections. Once this mapping has been determined,
  260. it cannot be changed without rebuilding the index.
  261. The basic assumption in the current implementation is that the index
  262. structure is sufficiently empty that individual blocks of the index will rarely
  263. overflow. The total size of the index should be about 2% to 10% of
  264. the total size of the arenas, but the exact percentage depends both on the
  265. index block size and the compressed size of blocks stored.
  266. See the discussion in
  267. .IR venti (8)
  268. for more.
  269. .PP
  270. .I Fmtindex
  271. also computes a mapping between a linear address space and
  272. the data section of the collection of arenas. The
  273. .B -a
  274. option can be used to add additional arenas to an index.
  275. To use this feature,
  276. add the new arenas to
  277. .I venti.conf
  278. after the existing arenas and then run
  279. .I fmtindex
  280. .BR -a .
  281. .PP
  282. A copy of the above mappings is stored in the header for each of the index sections.
  283. These copies enable
  284. .I buildindex
  285. to restore a single index section without rebuilding the entire index.
  286. .PP
  287. To make it easier to bootstrap servers, the configuration
  288. file can be stored in otherwise empty space
  289. at the beginning of any venti partitions using
  290. .IR conf .
  291. A partition so branded with a configuration file can
  292. be used in place of a configuration file when invoking any
  293. of the venti commands.
  294. By default,
  295. .I conf
  296. prints the configuration stored in
  297. .IR partition .
  298. When invoked with the
  299. .B -w
  300. flag,
  301. .I conf
  302. reads a configuration file from
  303. .I configfile
  304. (or else standard input)
  305. and stores it in
  306. .IR partition .
  307. .SS Checking and Rebuilding
  308. .PP
  309. .I Buildindex
  310. populates the index for the Venti system described in
  311. .IR venti.conf .
  312. The index must have previously been formatted using
  313. .IR fmtindex .
  314. This command is typically used to build a new index for a Venti
  315. system when the old index becomes too small, or to rebuild
  316. an index after media failure.
  317. Small errors in an index can usually be fixed with
  318. .IR checkindex ,
  319. but
  320. .I checkindex
  321. requires a large temporary workspace and
  322. .I buildindex
  323. does not.
  324. .PP
  325. Options to
  326. .I buildindex
  327. are:
  328. .TF "\fL-M\fI imemsize"
  329. .PD
  330. .TP
  331. .B -b
  332. Reinitialise the Bloom filter, if any.
  333. .TP
  334. .B -d
  335. `Dumb' mode; run all three passes.
  336. .TP
  337. .BI -i " isect
  338. Only rebuild index section
  339. .IR isect ;
  340. may be repeated to rebuild multiple sections.
  341. The name
  342. .L none
  343. is special and just reads the arenas.
  344. .TP
  345. .BI -M " imemsize
  346. The amount of memory, in bytes, to use for caching raw disk accesses while running
  347. .IR buildindex .
  348. (This is not a property of the created index.)
  349. The usual suffices apply.
  350. The default is 256M.
  351. .PD
  352. .PP
  353. .I Checkindex
  354. examines the Venti index described in
  355. .IR venti.conf .
  356. The program detects various error conditions including:
  357. blocks that are not indexed, index entries for blocks that do not exist,
  358. and duplicate index entries.
  359. If requested, an attempt can be made to fix errors that are found.
  360. .PP
  361. The
  362. .I tmp
  363. file, usually a disk partition, must be large enough to store a copy of the index.
  364. This temporary space is used to perform a merge sort of index entries
  365. generated by reading the arenas.
  366. .PP
  367. Options to
  368. .I checkindex
  369. are:
  370. .TP
  371. .BI -B " blockcachesize
  372. The amount of memory, in bytes, to use for caching raw disk accesses while running
  373. .IR checkindex .
  374. The default is 8k.
  375. .TP
  376. .B -f
  377. Attempt to fix any errors that are found.
  378. .PD
  379. .PP
  380. .I Checkarenas
  381. examines the Venti arenas contained in the given
  382. .IR file .
  383. The program detects various error conditions, and optionally attempts
  384. to fix any errors that are found.
  385. .PP
  386. Options to
  387. .I checkarenas
  388. are:
  389. .TP
  390. .B -a
  391. For each arena, scan the entire data section.
  392. If this option is omitted, only the end section of
  393. the arena is examined.
  394. .TP
  395. .B -f
  396. Attempt to fix any errors that are found.
  397. .TP
  398. .B -v
  399. Increase the verbosity of output.
  400. .PD
  401. .SH SOURCE
  402. .B /sys/src/cmd/venti/srv
  403. .SH SEE ALSO
  404. .IR venti (6),
  405. .IR venti (8)
  406. .SH BUGS
  407. .I Buildindex
  408. should allow an individual index section to be rebuilt.