README.design 24 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608
  1. Design document for the unified scheme data
  2. ===========================================
  3. How are things connected?
  4. -------------------------
  5. The unified scheme takes all its data from the build.info files seen
  6. throughout the source tree. These files hold the minimum information
  7. needed to build end product files from diverse sources. See the
  8. section on build.info files below.
  9. From the information in build.info files, Configure builds up an
  10. information database as a hash table called %unified_info, which is
  11. stored in configdata.pm, found at the top of the build tree (which may
  12. or may not be the same as the source tree).
  13. Configurations/common.tmpl uses the data from %unified_info to
  14. generate the rules for building end product files as well as
  15. intermediary files with the help of a few functions found in the
  16. build-file templates. See the section on build-file templates further
  17. down for more information.
  18. build.info files
  19. ----------------
  20. As mentioned earlier, build.info files are meant to hold the minimum
  21. information needed to build output files, and therefore only (with a
  22. few possible exceptions [1]) have information about end products (such
  23. as scripts, library files and programs) and source files (such as C
  24. files, C header files, assembler files, etc). Intermediate files such
  25. as object files are rarely directly referred to in build.info files (and
  26. when they are, it's always with the file name extension .o), they are
  27. inferred by Configure. By the same rule of minimalism, end product
  28. file name extensions (such as .so, .a, .exe, etc) are never mentioned
  29. in build.info. Their file name extensions will be inferred by the
  30. build-file templates, adapted for the platform they are meant for (see
  31. sections on %unified_info and build-file templates further down).
  32. The variables PROGRAMS, LIBS, ENGINES and SCRIPTS are used to declare
  33. end products. There are variants for them with '_NO_INST' as suffix
  34. (PROGRAM_NO_INST etc) to specify end products that shouldn't get
  35. installed.
  36. The variables SOURCE, DEPEND, INCLUDE and DEFINE are indexed by a
  37. produced file, and their values are the source used to produce that
  38. particular produced file, extra dependencies, include directories
  39. needed, or C macros to be defined.
  40. All their values in all the build.info throughout the source tree are
  41. collected together and form a set of programs, libraries, engines and
  42. scripts to be produced, source files, dependencies, etc etc etc.
  43. Let's have a pretend example, a very limited contraption of OpenSSL,
  44. composed of the program 'apps/openssl', the libraries 'libssl' and
  45. 'libcrypto', an engine 'engines/ossltest' and their sources and
  46. dependencies.
  47. # build.info
  48. LIBS=libcrypto libssl
  49. INCLUDE[libcrypto]=include
  50. INCLUDE[libssl]=include
  51. DEPEND[libssl]=libcrypto
  52. This is the top directory build.info file, and it tells us that two
  53. libraries are to be built, the include directory 'include/' shall be
  54. used throughout when building anything that will end up in each
  55. library, and that the library 'libssl' depend on the library
  56. 'libcrypto' to function properly.
  57. # apps/build.info
  58. PROGRAMS=openssl
  59. SOURCE[openssl]=openssl.c
  60. INCLUDE[openssl]=.. ../include
  61. DEPEND[openssl]=../libssl
  62. This is the build.info file in 'apps/', one may notice that all file
  63. paths mentioned are relative to the directory the build.info file is
  64. located in. This one tells us that there's a program to be built
  65. called 'apps/openssl' (the file name extension will depend on the
  66. platform and is therefore not mentioned in the build.info file). It's
  67. built from one source file, 'apps/openssl.c', and building it requires
  68. the use of '.' and 'include' include directories (both are declared
  69. from the point of view of the 'apps/' directory), and that the program
  70. depends on the library 'libssl' to function properly.
  71. # crypto/build.info
  72. LIBS=../libcrypto
  73. SOURCE[../libcrypto]=aes.c evp.c cversion.c
  74. DEPEND[cversion.o]=buildinf.h
  75. GENERATE[buildinf.h]=../util/mkbuildinf.pl "$(CC) $(CFLAGS)" "$(PLATFORM)"
  76. DEPEND[buildinf.h]=../Makefile
  77. DEPEND[../util/mkbuildinf.pl]=../util/Foo.pm
  78. This is the build.info file in 'crypto', and it tells us a little more
  79. about what's needed to produce 'libcrypto'. LIBS is used again to
  80. declare that 'libcrypto' is to be produced. This declaration is
  81. really unnecessary as it's already mentioned in the top build.info
  82. file, but can make the info file easier to understand. This is to
  83. show that duplicate information isn't an issue.
  84. This build.info file informs us that 'libcrypto' is built from a few
  85. source files, 'crypto/aes.c', 'crypto/evp.c' and 'crypto/cversion.c'.
  86. It also shows us that building the object file inferred from
  87. 'crypto/cversion.c' depends on 'crypto/buildinf.h'. Finally, it
  88. also shows the possibility to declare how some files are generated
  89. using some script, in this case a perl script, and how such scripts
  90. can be declared to depend on other files, in this case a perl module.
  91. Two things are worth an extra note:
  92. 'DEPEND[cversion.o]' mentions an object file. DEPEND indexes is the
  93. only location where it's valid to mention them
  94. # ssl/build.info
  95. LIBS=../libssl
  96. SOURCE[../libssl]=tls.c
  97. This is the build.info file in 'ssl/', and it tells us that the
  98. library 'libssl' is built from the source file 'ssl/tls.c'.
  99. # engines/build.info
  100. ENGINES=dasync
  101. SOURCE[dasync]=e_dasync.c
  102. DEPEND[dasync]=../libcrypto
  103. INCLUDE[dasync]=../include
  104. ENGINES_NO_INST=ossltest
  105. SOURCE[ossltest]=e_ossltest.c
  106. DEPEND[ossltest]=../libcrypto.a
  107. INCLUDE[ossltest]=../include
  108. This is the build.info file in 'engines/', telling us that two engines
  109. called 'engines/dasync' and 'engines/ossltest' shall be built, that
  110. dasync's source is 'engines/e_dasync.c' and ossltest's source is
  111. 'engines/e_ossltest.c' and that the include directory 'include/' may
  112. be used when building anything that will be part of these engines.
  113. Also, both engines depend on the library 'libcrypto' to function
  114. properly. ossltest is explicitly linked with the static variant of
  115. the library 'libcrypto'. Finally, only dasync is being installed, as
  116. ossltest is only for internal testing.
  117. When Configure digests these build.info files, the accumulated
  118. information comes down to this:
  119. LIBS=libcrypto libssl
  120. SOURCE[libcrypto]=crypto/aes.c crypto/evp.c crypto/cversion.c
  121. DEPEND[crypto/cversion.o]=crypto/buildinf.h
  122. INCLUDE[libcrypto]=include
  123. SOURCE[libssl]=ssl/tls.c
  124. INCLUDE[libssl]=include
  125. DEPEND[libssl]=libcrypto
  126. PROGRAMS=apps/openssl
  127. SOURCE[apps/openssl]=apps/openssl.c
  128. INCLUDE[apps/openssl]=. include
  129. DEPEND[apps/openssl]=libssl
  130. ENGINES=engines/dasync
  131. SOURCE[engines/dasync]=engines/e_dasync.c
  132. DEPEND[engines/dasync]=libcrypto
  133. INCLUDE[engines/dasync]=include
  134. ENGINES_NO_INST=engines/ossltest
  135. SOURCE[engines/ossltest]=engines/e_ossltest.c
  136. DEPEND[engines/ossltest]=libcrypto.a
  137. INCLUDE[engines/ossltest]=include
  138. GENERATE[crypto/buildinf.h]=util/mkbuildinf.pl "$(CC) $(CFLAGS)" "$(PLATFORM)"
  139. DEPEND[crypto/buildinf.h]=Makefile
  140. DEPEND[util/mkbuildinf.pl]=util/Foo.pm
  141. A few notes worth mentioning:
  142. LIBS may be used to declare routine libraries only.
  143. PROGRAMS may be used to declare programs only.
  144. ENGINES may be used to declare engines only.
  145. The indexes for SOURCE must only be end product files, such as
  146. libraries, programs or engines. The values of SOURCE variables must
  147. only be source files (possibly generated).
  148. INCLUDE and DEPEND shows a relationship between different files
  149. (usually produced files) or between files and directories, such as a
  150. program depending on a library, or between an object file and some
  151. extra source file.
  152. When Configure processes the build.info files, it will take it as
  153. truth without question, and will therefore perform very few checks.
  154. If the build tree is separate from the source tree, it will assume
  155. that all built files and up in the build directory and that all source
  156. files are to be found in the source tree, if they can be found there.
  157. Configure will assume that source files that can't be found in the
  158. source tree (such as 'crypto/bildinf.h' in the example above) are
  159. generated and will be found in the build tree.
  160. The %unified_info database
  161. --------------------------
  162. The information in all the build.info get digested by Configure and
  163. collected into the %unified_info database, divided into the following
  164. indexes:
  165. depends => a hash table containing 'file' => [ 'dependency' ... ]
  166. pairs. These are directly inferred from the DEPEND
  167. variables in build.info files.
  168. engines => a list of engines. These are directly inferred from
  169. the ENGINES variable in build.info files.
  170. generate => a hash table containing 'file' => [ 'generator' ... ]
  171. pairs. These are directly inferred from the GENERATE
  172. variables in build.info files.
  173. includes => a hash table containing 'file' => [ 'include' ... ]
  174. pairs. These are directly inferred from the INCLUDE
  175. variables in build.info files.
  176. install => a hash table containing 'type' => [ 'file' ... ] pairs.
  177. The types are 'programs', 'libraries', 'engines' and
  178. 'scripts', and the array of files list the files of
  179. that type that should be installed.
  180. libraries => a list of libraries. These are directly inferred from
  181. the LIBS variable in build.info files.
  182. programs => a list of programs. These are directly inferred from
  183. the PROGRAMS variable in build.info files.
  184. scripts => a list of scripts. There are directly inferred from
  185. the SCRIPTS variable in build.info files.
  186. sources => a hash table containing 'file' => [ 'sourcefile' ... ]
  187. pairs. These are indirectly inferred from the SOURCE
  188. variables in build.info files. Object files are
  189. mentioned in this hash table, with source files from
  190. SOURCE variables, and AS source files for programs and
  191. libraries.
  192. shared_sources =>
  193. a hash table just like 'sources', but only as source
  194. files (object files) for building shared libraries.
  195. As an example, here is how the build.info files example from the
  196. section above would be digested into a %unified_info table:
  197. our %unified_info = (
  198. "depends" =>
  199. {
  200. "apps/openssl" =>
  201. [
  202. "libssl",
  203. ],
  204. "crypto/buildinf.h" =>
  205. [
  206. "Makefile",
  207. ],
  208. "crypto/cversion.o" =>
  209. [
  210. "crypto/buildinf.h",
  211. ],
  212. "engines/dasync" =>
  213. [
  214. "libcrypto",
  215. ],
  216. "engines/ossltest" =>
  217. [
  218. "libcrypto.a",
  219. ],
  220. "libssl" =>
  221. [
  222. "libcrypto",
  223. ],
  224. "util/mkbuildinf.pl" =>
  225. [
  226. "util/Foo.pm",
  227. ],
  228. },
  229. "engines" =>
  230. [
  231. "engines/dasync",
  232. "engines/ossltest",
  233. ],
  234. "generate" =>
  235. {
  236. "crypto/buildinf.h" =>
  237. [
  238. "util/mkbuildinf.pl",
  239. "\"\$(CC)",
  240. "\$(CFLAGS)\"",
  241. "\"$(PLATFORM)\"",
  242. ],
  243. },
  244. "includes" =>
  245. {
  246. "apps/openssl" =>
  247. [
  248. ".",
  249. "include",
  250. ],
  251. "engines/ossltest" =>
  252. [
  253. "include"
  254. ],
  255. "libcrypto" =>
  256. [
  257. "include",
  258. ],
  259. "libssl" =>
  260. [
  261. "include",
  262. ],
  263. "util/mkbuildinf.pl" =>
  264. [
  265. "util",
  266. ],
  267. }
  268. "install" =>
  269. {
  270. "engines" =>
  271. [
  272. "engines/dasync",
  273. ],
  274. "libraries" =>
  275. [
  276. "libcrypto",
  277. "libssl",
  278. ],
  279. "programs" =>
  280. [
  281. "apps/openssl",
  282. ],
  283. },
  284. "libraries" =>
  285. [
  286. "libcrypto",
  287. "libssl",
  288. ],
  289. "programs" =>
  290. [
  291. "apps/openssl",
  292. ],
  293. "sources" =>
  294. {
  295. "apps/openssl" =>
  296. [
  297. "apps/openssl.o",
  298. ],
  299. "apps/openssl.o" =>
  300. [
  301. "apps/openssl.c",
  302. ],
  303. "crypto/aes.o" =>
  304. [
  305. "crypto/aes.c",
  306. ],
  307. "crypto/cversion.o" =>
  308. [
  309. "crypto/cversion.c",
  310. ],
  311. "crypto/evp.o" =>
  312. [
  313. "crypto/evp.c",
  314. ],
  315. "engines/e_dasync.o" =>
  316. [
  317. "engines/e_dasync.c",
  318. ],
  319. "engines/dasync" =>
  320. [
  321. "engines/e_dasync.o",
  322. ],
  323. "engines/e_ossltest.o" =>
  324. [
  325. "engines/e_ossltest.c",
  326. ],
  327. "engines/ossltest" =>
  328. [
  329. "engines/e_ossltest.o",
  330. ],
  331. "libcrypto" =>
  332. [
  333. "crypto/aes.c",
  334. "crypto/cversion.c",
  335. "crypto/evp.c",
  336. ],
  337. "libssl" =>
  338. [
  339. "ssl/tls.c",
  340. ],
  341. "ssl/tls.o" =>
  342. [
  343. "ssl/tls.c",
  344. ],
  345. },
  346. );
  347. As can be seen, everything in %unified_info is fairly simple suggest
  348. of information. Still, it tells us that to build all programs, we
  349. must build 'apps/openssl', and to build the latter, we will need to
  350. build all its sources ('apps/openssl.o' in this case) and all the
  351. other things it depends on (such as 'libssl'). All those dependencies
  352. need to be built as well, using the same logic, so to build 'libssl',
  353. we need to build 'ssl/tls.o' as well as 'libcrypto', and to build the
  354. latter...
  355. Build-file templates
  356. --------------------
  357. Build-file templates are essentially build-files (such as Makefile on
  358. Unix) with perl code fragments mixed in. Those perl code fragment
  359. will generate all the configuration dependent data, including all the
  360. rules needed to build end product files and intermediary files alike.
  361. At a minimum, there must be a perl code fragment that defines a set of
  362. functions that are used to generates specific build-file rules, to
  363. build static libraries from object files, to build shared libraries
  364. from static libraries, to programs from object files and libraries,
  365. etc.
  366. generatesrc - function that produces build file lines to generate
  367. a source file from some input.
  368. It's called like this:
  369. generatesrc(src => "PATH/TO/tobegenerated",
  370. generator => [ "generatingfile", ... ]
  371. generator_incs => [ "INCL/PATH", ... ]
  372. generator_deps => [ "dep1", ... ]
  373. incs => [ "INCL/PATH", ... ],
  374. deps => [ "dep1", ... ],
  375. intent => one of "libs", "dso", "bin" );
  376. 'src' has the name of the file to be generated.
  377. 'generator' is the command or part of command to
  378. generate the file, of which the first item is
  379. expected to be the file to generate from.
  380. generatesrc() is expected to analyse and figure out
  381. exactly how to apply that file and how to capture
  382. the result. 'generator_incs' and 'generator_deps'
  383. are include directories and files that the generator
  384. file itself depends on. 'incs' and 'deps' are
  385. include directories and files that are used if $(CC)
  386. is used as an intermediary step when generating the
  387. end product (the file indicated by 'src'). 'intent'
  388. indicates what the generated file is going to be
  389. used for.
  390. src2obj - function that produces build file lines to build an
  391. object file from source files and associated data.
  392. It's called like this:
  393. src2obj(obj => "PATH/TO/objectfile",
  394. srcs => [ "PATH/TO/sourcefile", ... ],
  395. deps => [ "dep1", ... ],
  396. incs => [ "INCL/PATH", ... ]
  397. intent => one of "lib", "dso", "bin" );
  398. 'obj' has the intended object file with '.o'
  399. extension, src2obj() is expected to change it to
  400. something more suitable for the platform.
  401. 'srcs' has the list of source files to build the
  402. object file, with the first item being the source
  403. file that directly corresponds to the object file.
  404. 'deps' is a list of explicit dependencies. 'incs'
  405. is a list of include file directories. Finally,
  406. 'intent' indicates what this object file is going
  407. to be used for.
  408. obj2lib - function that produces build file lines to build a
  409. static library file ("libfoo.a" in Unix terms) from
  410. object files.
  411. called like this:
  412. obj2lib(lib => "PATH/TO/libfile",
  413. objs => [ "PATH/TO/objectfile", ... ]);
  414. 'lib' has the intended library file name *without*
  415. extension, obj2lib is expected to add that. 'objs'
  416. has the list of object files to build this library.
  417. libobj2shlib - backward compatibility function that's used the
  418. same way as obj2shlib (described next), and was
  419. expected to build the shared library from the
  420. corresponding static library when that was suitable.
  421. NOTE: building a shared library from a static
  422. library is now DEPRECATED, as they no longer share
  423. object files. Attempting to do this will fail.
  424. obj2shlib - function that produces build file lines to build a
  425. shareable object library file ("libfoo.so" in Unix
  426. terms) from the corresponding object files.
  427. called like this:
  428. obj2shlib(shlib => "PATH/TO/shlibfile",
  429. lib => "PATH/TO/libfile",
  430. objs => [ "PATH/TO/objectfile", ... ],
  431. deps => [ "PATH/TO/otherlibfile", ... ]);
  432. 'lib' has the base (static) library file name
  433. *without* extension. This is useful in case
  434. supporting files are needed (such as import
  435. libraries on Windows).
  436. 'shlib' has the corresponding shared library name
  437. *without* extension. 'deps' has the list of other
  438. libraries (also *without* extension) this library
  439. needs to be linked with. 'objs' has the list of
  440. object files to build this library.
  441. obj2dso - function that produces build file lines to build a
  442. dynamic shared object file from object files.
  443. called like this:
  444. obj2dso(lib => "PATH/TO/libfile",
  445. objs => [ "PATH/TO/objectfile", ... ],
  446. deps => [ "PATH/TO/otherlibfile",
  447. ... ]);
  448. This is almost the same as obj2shlib, but the
  449. intent is to build a shareable library that can be
  450. loaded in runtime (a "plugin"...).
  451. obj2bin - function that produces build file lines to build an
  452. executable file from object files.
  453. called like this:
  454. obj2bin(bin => "PATH/TO/binfile",
  455. objs => [ "PATH/TO/objectfile", ... ],
  456. deps => [ "PATH/TO/libfile", ... ]);
  457. 'bin' has the intended executable file name
  458. *without* extension, obj2bin is expected to add
  459. that. 'objs' has the list of object files to build
  460. this library. 'deps' has the list of library files
  461. (also *without* extension) that the programs needs
  462. to be linked with.
  463. in2script - function that produces build file lines to build a
  464. script file from some input.
  465. called like this:
  466. in2script(script => "PATH/TO/scriptfile",
  467. sources => [ "PATH/TO/infile", ... ]);
  468. 'script' has the intended script file name.
  469. 'sources' has the list of source files to build the
  470. resulting script from.
  471. Along with the build-file templates is the driving engine
  472. Configurations/common.tmpl, which looks through all the information in
  473. %unified_info and generates all the rulesets to build libraries,
  474. programs and all intermediate files, using the rule generating
  475. functions defined in the build-file template.
  476. As an example with the smaller build.info set we've seen as an
  477. example, producing the rules to build 'libcrypto' would result in the
  478. following calls:
  479. # Note: obj2shlib will only be called if shared libraries are
  480. # to be produced.
  481. # Note 2: obj2shlib must convert the '.o' extension to whatever
  482. # is suitable on the local platform.
  483. obj2shlib(shlib => "libcrypto",
  484. objs => [ "crypto/aes.o", "crypto/evp.o", "crypto/cversion.o" ],
  485. deps => [ ]);
  486. obj2lib(lib => "libcrypto"
  487. objs => [ "crypto/aes.o", "crypto/evp.o", "crypto/cversion.o" ]);
  488. src2obj(obj => "crypto/aes.o"
  489. srcs => [ "crypto/aes.c" ],
  490. deps => [ ],
  491. incs => [ "include" ],
  492. intent => "lib");
  493. src2obj(obj => "crypto/evp.o"
  494. srcs => [ "crypto/evp.c" ],
  495. deps => [ ],
  496. incs => [ "include" ],
  497. intent => "lib");
  498. src2obj(obj => "crypto/cversion.o"
  499. srcs => [ "crypto/cversion.c" ],
  500. deps => [ "crypto/buildinf.h" ],
  501. incs => [ "include" ],
  502. intent => "lib");
  503. generatesrc(src => "crypto/buildinf.h",
  504. generator => [ "util/mkbuildinf.pl", "\"$(CC)",
  505. "$(CFLAGS)\"", "\"$(PLATFORM)\"" ],
  506. generator_incs => [ "util" ],
  507. generator_deps => [ "util/Foo.pm" ],
  508. incs => [ ],
  509. deps => [ ],
  510. intent => "lib");
  511. The returned strings from all those calls are then concatenated
  512. together and written to the resulting build-file.