README.design 24 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605
  1. Design document for the unified scheme data
  2. ===========================================
  3. How are things connected?
  4. -------------------------
  5. The unified scheme takes all its data from the build.info files seen
  6. throughout the source tree. These files hold the minimum information
  7. needed to build end product files from diverse sources. See the
  8. section on build.info files below.
  9. From the information in build.info files, Configure builds up an
  10. information database as a hash table called %unified_info, which is
  11. stored in configdata.pm, found at the top of the build tree (which may
  12. or may not be the same as the source tree).
  13. Configurations/common.tmpl uses the data from %unified_info to
  14. generate the rules for building end product files as well as
  15. intermediary files with the help of a few functions found in the
  16. build-file templates. See the section on build-file templates further
  17. down for more information.
  18. build.info files
  19. ----------------
  20. As mentioned earlier, build.info files are meant to hold the minimum
  21. information needed to build output files, and therefore only (with a
  22. few possible exceptions [1]) have information about end products (such
  23. as scripts, library files and programs) and source files (such as C
  24. files, C header files, assembler files, etc). Intermediate files such
  25. as object files are rarely directly referred to in build.info files (and
  26. when they are, it's always with the file name extension .o), they are
  27. inferred by Configure. By the same rule of minimalism, end product
  28. file name extensions (such as .so, .a, .exe, etc) are never mentioned
  29. in build.info. Their file name extensions will be inferred by the
  30. build-file templates, adapted for the platform they are meant for (see
  31. sections on %unified_info and build-file templates further down).
  32. The variables PROGRAMS, LIBS, ENGINES and SCRIPTS are used to declare
  33. end products.
  34. The variables SOURCE, DEPEND, INCLUDE and ORDINALS are indexed by a
  35. produced file, and their values are the source used to produce that
  36. particular produced file, extra dependencies, include directories
  37. needed, and ordinal files (explained further below.
  38. All their values in all the build.info throughout the source tree are
  39. collected together and form a set of programs, libraries, engines and
  40. scripts to be produced, source files, dependencies, etc etc etc.
  41. Let's have a pretend example, a very limited contraption of OpenSSL,
  42. composed of the program 'apps/openssl', the libraries 'libssl' and
  43. 'libcrypto', an engine 'engines/ossltest' and their sources and
  44. dependencies.
  45. # build.info
  46. LIBS=libcrypto libssl
  47. ORDINALS[libcrypto]=crypto
  48. ORDINALS[libssl]=ssl
  49. INCLUDE[libcrypto]=include
  50. INCLUDE[libssl]=include
  51. DEPEND[libssl]=libcrypto
  52. This is the top directory build.info file, and it tells us that two
  53. libraries are to be built, there are some ordinals to be used to
  54. declare what symbols in those libraries are seen as public, the
  55. include directory 'include/' shall be used throughout when building
  56. anything that will end up in each library, and that the library
  57. 'libssl' depend on the library 'libcrypto' to function properly.
  58. # apps/build.info
  59. PROGRAMS=openssl
  60. SOURCE[openssl]=openssl.c
  61. INCLUDE[openssl]=.. ../include
  62. DEPEND[openssl]=../libssl
  63. This is the build.info file in 'apps/', one may notice that all file
  64. paths mentioned are relative to the directory the build.info file is
  65. located in. This one tells us that there's a program to be built
  66. called 'apps/openssl' (the file name extension will depend on the
  67. platform and is therefore not mentioned in the build.info file). It's
  68. built from one source file, 'apps/openssl.c', and building it requires
  69. the use of '.' and 'include' include directories (both are declared
  70. from the point of view of the 'apps/' directory), and that the program
  71. depends on the library 'libssl' to function properly.
  72. # crypto/build.info
  73. LIBS=../libcrypto
  74. SOURCE[../libcrypto]=aes.c evp.c cversion.c
  75. DEPEND[cversion.o]=buildinf.h
  76. GENERATE[buildinf.h]=../util/mkbuildinf.pl "$(CC) $(CFLAGS)" "$(PLATFORM)"
  77. DEPEND[buildinf.h]=../Makefile
  78. DEPEND[../util/mkbuildinf.pl]=../util/Foo.pm
  79. This is the build.info file in 'crypto', and it tells us a little more
  80. about what's needed to produce 'libcrypto'. LIBS is used again to
  81. declare that 'libcrypto' is to be produced. This declaration is
  82. really unnecessary as it's already mentioned in the top build.info
  83. file, but can make the info file easier to understand. This is to
  84. show that duplicate information isn't an issue.
  85. This build.info file informs us that 'libcrypto' is built from a few
  86. source files, 'crypto/aes.c', 'crypto/evp.c' and 'crypto/cversion.c'.
  87. It also shows us that building the object file inferred from
  88. 'crypto/cversion.c' depends on 'crypto/buildinf.h'. Finally, it
  89. also shows the possibility to declare how some files are generated
  90. using some script, in this case a perl script, and how such scripts
  91. can be declared to depend on other files, in this case a perl module.
  92. Two things are worth an extra note:
  93. 'DEPEND[cversion.o]' mentions an object file. DEPEND indexes is the
  94. only location where it's valid to mention them
  95. Lines in 'BEGINRAW'..'ENDRAW' sections must always mention files as
  96. seen from the top directory, no exception.
  97. # ssl/build.info
  98. LIBS=../libssl
  99. SOURCE[../libssl]=tls.c
  100. This is the build.info file in 'ssl/', and it tells us that the
  101. library 'libssl' is built from the source file 'ssl/tls.c'.
  102. # engines/build.info
  103. ENGINES=libossltest
  104. SOURCE[libossltest]=e_ossltest.c
  105. DEPEND[libossltest]=../libcrypto
  106. INCLUDE[libossltest]=../include
  107. This is the build.info file in 'engines/', telling us that an engine
  108. called 'engines/libossltest' shall be built, that it's source is
  109. 'engines/e_ossltest.c' and that the include directory 'include/' may
  110. be used when building anything that will be part of this engine.
  111. Finally, the engine 'engines/libossltest' depends on the library
  112. 'libcrypto' to function properly.
  113. When Configure digests these build.info files, the accumulated
  114. information comes down to this:
  115. LIBS=libcrypto libssl
  116. ORDINALS[libcrypto]=crypto
  117. SOURCE[libcrypto]=crypto/aes.c crypto/evp.c crypto/cversion.c
  118. DEPEND[crypto/cversion.o]=crypto/buildinf.h
  119. INCLUDE[libcrypto]=include
  120. ORDINALS[libssl]=ssl
  121. SOURCE[libssl]=ssl/tls.c
  122. INCLUDE[libssl]=include
  123. DEPEND[libssl]=libcrypto
  124. PROGRAMS=apps/openssl
  125. SOURCE[apps/openssl]=apps/openssl.c
  126. INCLUDE[apps/openssl]=. include
  127. DEPEND[apps/openssl]=libssl
  128. ENGINES=engines/ossltest
  129. SOURCE[engines/ossltest]=engines/e_ossltest.c
  130. DEPEND[engines/ossltest]=libcrypto
  131. INCLUDE[engines/ossltest]=include
  132. GENERATE[crypto/buildinf.h]=util/mkbuildinf.pl "$(CC) $(CFLAGS)" "$(PLATFORM)"
  133. DEPEND[crypto/buildinf.h]=Makefile
  134. DEPEND[util/mkbuildinf.pl]=util/Foo.pm
  135. A few notes worth mentioning:
  136. LIBS may be used to declare routine libraries only.
  137. PROGRAMS may be used to declare programs only.
  138. ENGINES may be used to declare engines only.
  139. The indexes for SOURCE and ORDINALS must only be end product files,
  140. such as libraries, programs or engines. The values of SOURCE
  141. variables must only be source files (possibly generated)
  142. INCLUDE and DEPEND shows a relationship between different files
  143. (usually produced files) or between files and directories, such as a
  144. program depending on a library, or between an object file and some
  145. extra source file.
  146. When Configure processes the build.info files, it will take it as
  147. truth without question, and will therefore perform very few checks.
  148. If the build tree is separate from the source tree, it will assume
  149. that all built files and up in the build directory and that all source
  150. files are to be found in the source tree, if they can be found there.
  151. Configure will assume that source files that can't be found in the
  152. source tree (such as 'crypto/bildinf.h' in the example above) are
  153. generated and will be found in the build tree.
  154. The %unified_info database
  155. --------------------------
  156. The information in all the build.info get digested by Configure and
  157. collected into the %unified_info database, divided into the following
  158. indexes:
  159. depends => a hash table containing 'file' => [ 'dependency' ... ]
  160. pairs. These are directly inferred from the DEPEND
  161. variables in build.info files.
  162. engines => a list of engines. These are directly inferred from
  163. the ENGINES variable in build.info files.
  164. generate => a hash table containing 'file' => [ 'generator' ... ]
  165. pairs. These are directly inferred from the GENERATE
  166. variables in build.info files.
  167. includes => a hash table containing 'file' => [ 'include' ... ]
  168. pairs. These are directly inferred from the INCLUDE
  169. variables in build.info files.
  170. libraries => a list of libraries. These are directly inferred from
  171. the LIBS variable in build.info files.
  172. ordinals => a hash table containing 'file' => [ 'word', 'ordfile' ]
  173. pairs. 'file' and 'word' are directly inferred from
  174. the ORDINALS variables in build.info files, while the
  175. file 'ofile' comes from internal knowledge in
  176. Configure.
  177. programs => a list of programs. These are directly inferred from
  178. the PROGRAMS variable in build.info files.
  179. rawlines => a list of build-file lines. These are a direct copy of
  180. the BEGINRAW..ENDRAW lines in build.info files. Note:
  181. only the BEGINRAW..ENDRAW section for the current
  182. platform are copied, the rest are ignored.
  183. scripts => a list of scripts. There are directly inferred from
  184. the SCRIPTS variable in build.info files.
  185. sources => a hash table containing 'file' => [ 'sourcefile' ... ]
  186. pairs. These are indirectly inferred from the SOURCE
  187. variables in build.info files. Object files are
  188. mentioned in this hash table, with source files from
  189. SOURCE variables, and AS source files for programs and
  190. libraries.
  191. shared_sources =>
  192. a hash table just like 'sources', but only as source
  193. files (object files) for building shared libraries.
  194. As an example, here is how the build.info files example from the
  195. section above would be digested into a %unified_info table:
  196. our %unified_info = (
  197. "depends" =>
  198. {
  199. "apps/openssl" =>
  200. [
  201. "libssl",
  202. ],
  203. "crypto/buildinf.h" =>
  204. [
  205. "Makefile",
  206. ],
  207. "crypto/cversion.o" =>
  208. [
  209. "crypto/buildinf.h",
  210. ],
  211. "engines/ossltest" =>
  212. [
  213. "libcrypto",
  214. ],
  215. "libssl" =>
  216. [
  217. "libcrypto",
  218. ],
  219. "util/mkbuildinf.pl" =>
  220. [
  221. "util/Foo.pm",
  222. ],
  223. },
  224. "engines" =>
  225. [
  226. "engines/ossltest",
  227. ],
  228. "generate" =>
  229. {
  230. "crypto/buildinf.h" =>
  231. [
  232. "util/mkbuildinf.pl",
  233. "\"\$(CC)",
  234. "\$(CFLAGS)\"",
  235. "\"$(PLATFORM)\"",
  236. ],
  237. },
  238. "includes" =>
  239. {
  240. "apps/openssl" =>
  241. [
  242. ".",
  243. "include",
  244. ],
  245. "engines/ossltest" =>
  246. [
  247. "include"
  248. ],
  249. "libcrypto" =>
  250. [
  251. "include",
  252. ],
  253. "libssl" =>
  254. [
  255. "include",
  256. ],
  257. "util/mkbuildinf.pl" =>
  258. [
  259. "util",
  260. ],
  261. }
  262. "libraries" =>
  263. [
  264. "libcrypto",
  265. "libssl",
  266. ],
  267. "ordinals" =>
  268. {
  269. "libcrypto" =>
  270. [
  271. "crypto",
  272. "util/libcrypto.num",
  273. ],
  274. "libssl" =>
  275. [
  276. "ssl",
  277. "util/libssl.num",
  278. ],
  279. },
  280. "programs" =>
  281. [
  282. "apps/openssl",
  283. ],
  284. "rawlines" =>
  285. [
  286. ],
  287. "sources" =>
  288. {
  289. "apps/openssl" =>
  290. [
  291. "apps/openssl.o",
  292. ],
  293. "apps/openssl.o" =>
  294. [
  295. "apps/openssl.c",
  296. ],
  297. "crypto/aes.o" =>
  298. [
  299. "crypto/aes.c",
  300. ],
  301. "crypto/cversion.o" =>
  302. [
  303. "crypto/cversion.c",
  304. ],
  305. "crypto/evp.o" =>
  306. [
  307. "crypto/evp.c",
  308. ],
  309. "engines/e_ossltest.o" =>
  310. [
  311. "engines/e_ossltest.c",
  312. ],
  313. "engines/ossltest" =>
  314. [
  315. "engines/e_ossltest.o",
  316. ],
  317. "libcrypto" =>
  318. [
  319. "crypto/aes.c",
  320. "crypto/cversion.c",
  321. "crypto/evp.c",
  322. ],
  323. "libssl" =>
  324. [
  325. "ssl/tls.c",
  326. ],
  327. "ssl/tls.o" =>
  328. [
  329. "ssl/tls.c",
  330. ],
  331. },
  332. );
  333. As can be seen, everything in %unified_info is fairly simple suggest
  334. of information. Still, it tells us that to build all programs, we
  335. must build 'apps/openssl', and to build the latter, we will need to
  336. build all its sources ('apps/openssl.o' in this case) and all the
  337. other things it depends on (such as 'libssl'). All those dependencies
  338. need to be built as well, using the same logic, so to build 'libssl',
  339. we need to build 'ssl/tls.o' as well as 'libcrypto', and to build the
  340. latter...
  341. Build-file templates
  342. --------------------
  343. Build-file templates are essentially build-files (such as Makefile on
  344. Unix) with perl code fragments mixed in. Those perl code fragment
  345. will generate all the configuration dependent data, including all the
  346. rules needed to build end product files and intermediary files alike.
  347. At a minimum, there must be a perl code fragment that defines a set of
  348. functions that are used to generates specific build-file rules, to
  349. build static libraries from object files, to build shared libraries
  350. from static libraries, to programs from object files and libraries,
  351. etc.
  352. generatesrc - function that produces build file lines to generate
  353. a source file from some input.
  354. It's called like this:
  355. generatesrc(src => "PATH/TO/tobegenerated",
  356. generator => [ "generatingfile", ... ]
  357. generator_incs => [ "INCL/PATH", ... ]
  358. generator_deps => [ "dep1", ... ]
  359. incs => [ "INCL/PATH", ... ],
  360. deps => [ "dep1", ... ],
  361. intent => one of "libs", "dso", "bin" );
  362. 'src' has the name of the file to be generated.
  363. 'generator' is the command or part of command to
  364. generate the file, of which the first item is
  365. expected to be the file to generate from.
  366. generatesrc() is expected to analyse and figure out
  367. exactly how to apply that file and how to capture
  368. the result. 'generator_incs' and 'generator_deps'
  369. are include directories and files that the generator
  370. file itself depends on. 'incs' and 'deps' are
  371. include directories and files that are used if $(CC)
  372. is used as an intermediary step when generating the
  373. end product (the file indicated by 'src'). 'intent'
  374. indicates what the generated file is going to be
  375. used for.
  376. src2obj - function that produces build file lines to build an
  377. object file from source files and associated data.
  378. It's called like this:
  379. src2obj(obj => "PATH/TO/objectfile",
  380. srcs => [ "PATH/TO/sourcefile", ... ],
  381. deps => [ "dep1", ... ],
  382. incs => [ "INCL/PATH", ... ]
  383. intent => one of "lib", "dso", "bin" );
  384. 'obj' has the intended object file *without*
  385. extension, src2obj() is expected to add that.
  386. 'srcs' has the list of source files to build the
  387. object file, with the first item being the source
  388. file that directly corresponds to the object file.
  389. 'deps' is a list of explicit dependencies. 'incs'
  390. is a list of include file directories. Finally,
  391. 'intent' indicates what this object file is going
  392. to be used for.
  393. obj2lib - function that produces build file lines to build a
  394. static library file ("libfoo.a" in Unix terms) from
  395. object files.
  396. called like this:
  397. obj2lib(lib => "PATH/TO/libfile",
  398. objs => [ "PATH/TO/objectfile", ... ]);
  399. 'lib' has the intended library file name *without*
  400. extension, obj2lib is expected to add that. 'objs'
  401. has the list of object files (also *without*
  402. extension) to build this library.
  403. libobj2shlib - function that produces build file lines to build a
  404. shareable object library file ("libfoo.so" in Unix
  405. terms) from the corresponding static library file
  406. or object files.
  407. called like this:
  408. libobj2shlib(shlib => "PATH/TO/shlibfile",
  409. lib => "PATH/TO/libfile",
  410. objs => [ "PATH/TO/objectfile", ... ],
  411. deps => [ "PATH/TO/otherlibfile", ... ],
  412. ordinals => [ "word", "/PATH/TO/ordfile" ]);
  413. 'lib' has the intended library file name *without*
  414. extension, libobj2shlib is expected to add that.
  415. 'shlib' has the corresponding shared library name
  416. *without* extension. 'deps' has the list of other
  417. libraries (also *without* extension) this library
  418. needs to be linked with. 'objs' has the list of
  419. object files (also *without* extension) to build
  420. this library. 'ordinals' MAY be present, and when
  421. it is, its value is an array where the word is
  422. "crypto" or "ssl" and the file is one of the ordinal
  423. files util/libcrypto.num or util/libssl.num in the
  424. source directory.
  425. This function has a choice; it can use the
  426. corresponding static library as input to make the
  427. shared library, or the list of object files.
  428. obj2dynlib - function that produces build file lines to build a
  429. dynamically loadable library file ("libfoo.so" on
  430. Unix) from object files.
  431. called like this:
  432. obj2dynlib(lib => "PATH/TO/libfile",
  433. objs => [ "PATH/TO/objectfile", ... ],
  434. deps => [ "PATH/TO/otherlibfile",
  435. ... ]);
  436. This is almost the same as libobj2shlib, but the
  437. intent is to build a shareable library that can be
  438. loaded in runtime (a "plugin"...). The differences
  439. are subtle, one of the most visible ones is that the
  440. resulting shareable library is produced from object
  441. files only.
  442. obj2bin - function that produces build file lines to build an
  443. executable file from object files.
  444. called like this:
  445. obj2bin(bin => "PATH/TO/binfile",
  446. objs => [ "PATH/TO/objectfile", ... ],
  447. deps => [ "PATH/TO/libfile", ... ]);
  448. 'bin' has the intended executable file name
  449. *without* extension, obj2bin is expected to add
  450. that. 'objs' has the list of object files (also
  451. *without* extension) to build this library. 'deps'
  452. has the list of library files (also *without*
  453. extension) that the programs needs to be linked
  454. with.
  455. in2script - function that produces build file lines to build a
  456. script file from some input.
  457. called like this:
  458. in2script(script => "PATH/TO/scriptfile",
  459. sources => [ "PATH/TO/infile", ... ]);
  460. 'script' has the intended script file name.
  461. 'sources' has the list of source files to build the
  462. resulting script from.
  463. Along with the build-file templates is the driving engine
  464. Configurations/common.tmpl, which looks through all the information in
  465. %unified_info and generates all the rulesets to build libraries,
  466. programs and all intermediate files, using the rule generating
  467. functions defined in the build-file template.
  468. As an example with the smaller build.info set we've seen as an
  469. example, producing the rules to build 'libcrypto' would result in the
  470. following calls:
  471. # Note: libobj2shlib will only be called if shared libraries are
  472. # to be produced.
  473. # Note 2: libobj2shlib gets both the name of the static library
  474. # and the names of all the object files that go into it. It's up
  475. # to the implementation to decide which to use as input.
  476. # Note 3: common.tmpl peals off the ".o" extension from all object
  477. # files, as the platform at hand may have a different one.
  478. libobj2shlib(shlib => "libcrypto",
  479. lib => "libcrypto",
  480. objs => [ "crypto/aes", "crypto/evp", "crypto/cversion" ],
  481. deps => [ ]
  482. ordinals => [ "crypto", "util/libcrypto.num" ]);
  483. obj2lib(lib => "libcrypto"
  484. objs => [ "crypto/aes", "crypto/evp", "crypto/cversion" ]);
  485. src2obj(obj => "crypto/aes"
  486. srcs => [ "crypto/aes.c" ],
  487. deps => [ ],
  488. incs => [ "include" ],
  489. intent => "lib");
  490. src2obj(obj => "crypto/evp"
  491. srcs => [ "crypto/evp.c" ],
  492. deps => [ ],
  493. incs => [ "include" ],
  494. intent => "lib");
  495. src2obj(obj => "crypto/cversion"
  496. srcs => [ "crypto/cversion.c" ],
  497. deps => [ "crypto/buildinf.h" ],
  498. incs => [ "include" ],
  499. intent => "lib");
  500. generatesrc(src => "crypto/buildinf.h",
  501. generator => [ "util/mkbuildinf.pl", "\"$(CC)",
  502. "$(CFLAGS)\"", "\"$(PLATFORM)\"" ],
  503. generator_incs => [ "util" ],
  504. generator_deps => [ "util/Foo.pm" ],
  505. incs => [ ],
  506. deps => [ ],
  507. intent => "lib");
  508. The returned strings from all those calls are then concatenated
  509. together and written to the resulting build-file.