toolchaintechnotes.xml 17 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340
  1. <?xml version="1.0" encoding="ISO-8859-1"?>
  2. <!DOCTYPE sect1 PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN"
  3. "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [
  4. <!ENTITY % general-entities SYSTEM "../general.ent">
  5. %general-entities;
  6. ]>
  7. <sect1 id="ch-tools-toolchaintechnotes" xreflabel="Toolchain Technical Notes">
  8. <?dbhtml filename="toolchaintechnotes.html"?>
  9. <title>Toolchain Technical Notes</title>
  10. <para>This section explains some of the rationale and technical details
  11. behind the overall build method. It is not essential to immediately
  12. understand everything in this section. Most of this information will be
  13. clearer after performing an actual build. This section can be referred
  14. to at any time during the process.</para>
  15. <para>The overall goal of <xref linkend="chapter-cross-tools"/> and <xref
  16. linkend="chapter-temporary-tools"/> is to produce a temporary area that
  17. contains a known-good set of tools that can be isolated from the host system.
  18. By using <command>chroot</command>, the commands in the remaining chapters
  19. will be contained within that environment, ensuring a clean, trouble-free
  20. build of the target LFS system. The build process has been designed to
  21. minimize the risks for new readers and to provide the most educational value
  22. at the same time.</para>
  23. <para>The build process is based on the process of
  24. <emphasis>cross-compilation</emphasis>. Cross-compilation is normally used
  25. for building a compiler and its toolchain for a machine different from
  26. the one that is used for the build. This is not strictly needed for LFS,
  27. since the machine where the new system will run is the same as the one
  28. used for the build. But cross-compilation has the great advantage that
  29. anything that is cross-compiled cannot depend on the host environment.</para>
  30. <sect2 id="cross-compile" xreflabel="About Cross-Compilation">
  31. <title>About Cross-Compilation</title>
  32. <para>Cross-compilation involves some concepts that deserve a section on
  33. their own. Although this section may be omitted in a first reading, it
  34. is strongly suggested to come back to it later in order to get a full
  35. grasp of the build process.</para>
  36. <para>Let us first define some terms used in this context:</para>
  37. <variablelist>
  38. <varlistentry><term>build</term><listitem>
  39. <para>is the machine where we build programs. Note that this machine
  40. is referred to as the <quote>host</quote> in other
  41. sections.</para></listitem>
  42. </varlistentry>
  43. <varlistentry><term>host</term><listitem>
  44. <para>is the machine/system where the built programs will run. Note
  45. that this use of <quote>host</quote> is not the same as in other
  46. sections.</para></listitem>
  47. </varlistentry>
  48. <varlistentry><term>target</term><listitem>
  49. <para>is only used for compilers. It is the machine the compiler
  50. produces code for. It may be different from both build and
  51. host.</para></listitem>
  52. </varlistentry>
  53. </variablelist>
  54. <para>As an example, let us imagine the following scenario (sometimes
  55. referred to as <quote>Canadian Cross</quote>): we may have a
  56. compiler on a slow machine only, let's call the machine A, and the compiler
  57. ccA. We may have also a fast machine (B), but with no compiler, and we may
  58. want to produce code for another slow machine (C). To build a
  59. compiler for machine C, we would have three stages:</para>
  60. <informaltable align="center">
  61. <tgroup cols="5">
  62. <colspec colnum="1" align="center"/>
  63. <colspec colnum="2" align="center"/>
  64. <colspec colnum="3" align="center"/>
  65. <colspec colnum="4" align="center"/>
  66. <colspec colnum="5" align="left"/>
  67. <thead>
  68. <row><entry>Stage</entry><entry>Build</entry><entry>Host</entry>
  69. <entry>Target</entry><entry>Action</entry></row>
  70. </thead>
  71. <tbody>
  72. <row>
  73. <entry>1</entry><entry>A</entry><entry>A</entry><entry>B</entry>
  74. <entry>build cross-compiler cc1 using ccA on machine A</entry>
  75. </row>
  76. <row>
  77. <entry>2</entry><entry>A</entry><entry>B</entry><entry>C</entry>
  78. <entry>build cross-compiler cc2 using cc1 on machine A</entry>
  79. </row>
  80. <row>
  81. <entry>3</entry><entry>B</entry><entry>C</entry><entry>C</entry>
  82. <entry>build compiler ccC using cc2 on machine B</entry>
  83. </row>
  84. </tbody>
  85. </tgroup>
  86. </informaltable>
  87. <para>Then, all the other programs needed by machine C can be compiled
  88. using cc2 on the fast machine B. Note that unless B can run programs
  89. produced for C, there is no way to test the built programs until machine
  90. C itself is running. For example, for testing ccC, we may want to add a
  91. fourth stage:</para>
  92. <informaltable align="center">
  93. <tgroup cols="5">
  94. <colspec colnum="1" align="center"/>
  95. <colspec colnum="2" align="center"/>
  96. <colspec colnum="3" align="center"/>
  97. <colspec colnum="4" align="center"/>
  98. <colspec colnum="5" align="left"/>
  99. <thead>
  100. <row><entry>Stage</entry><entry>Build</entry><entry>Host</entry>
  101. <entry>Target</entry><entry>Action</entry></row>
  102. </thead>
  103. <tbody>
  104. <row>
  105. <entry>4</entry><entry>C</entry><entry>C</entry><entry>C</entry>
  106. <entry>rebuild and test ccC using itself on machine C</entry>
  107. </row>
  108. </tbody>
  109. </tgroup>
  110. </informaltable>
  111. <para>In the example above, only cc1 and cc2 are cross-compilers, that is,
  112. they produce code for a machine different from the one they are run on.
  113. The other compilers ccA and ccC produce code for the machine they are run
  114. on. Such compilers are called <emphasis>native</emphasis> compilers.</para>
  115. </sect2>
  116. <sect2 id="lfs-cross">
  117. <title>Implementation of Cross-Compilation for LFS</title>
  118. <note>
  119. <para>Almost all the build systems use names of the form
  120. cpu-vendor-kernel-os referred to as the machine triplet. An astute
  121. reader may wonder why a <quote>triplet</quote> refers to a four component
  122. name. The reason is history: initially, three component names were enough
  123. to designate unambiguously a machine, but with new machines and systems
  124. appearing, that proved insufficient. The word <quote>triplet</quote>
  125. remained. A simple way to determine your machine triplet is to run
  126. the <command>config.guess</command>
  127. script that comes with the source for many packages. Unpack the binutils
  128. sources and run the script: <userinput>./config.guess</userinput> and note
  129. the output. For example, for a 32-bit Intel processor the
  130. output will be <emphasis>i686-pc-linux-gnu</emphasis>. On a 64-bit
  131. system it will be <emphasis>x86_64-pc-linux-gnu</emphasis>.</para>
  132. <para>Also be aware of the name of the platform's dynamic linker, often
  133. referred to as the dynamic loader (not to be confused with the standard
  134. linker <command>ld</command> that is part of binutils). The dynamic linker
  135. provided by Glibc finds and loads the shared libraries needed by a
  136. program, prepares the program to run, and then runs it. The name of the
  137. dynamic linker for a 32-bit Intel machine will be <filename
  138. class="libraryfile">ld-linux.so.2</filename> (<filename
  139. class="libraryfile">ld-linux-x86-64.so.2</filename> for 64-bit systems). A
  140. sure-fire way to determine the name of the dynamic linker is to inspect a
  141. random binary from the host system by running: <userinput>readelf -l
  142. &lt;name of binary&gt; | grep interpreter</userinput> and noting the
  143. output. The authoritative reference covering all platforms is in the
  144. <filename>shlib-versions</filename> file in the root of the Glibc source
  145. tree.</para>
  146. </note>
  147. <para>In order to fake a cross compilation, the name of the host triplet
  148. is slightly adjusted by changing the &quot;vendor&quot; field in the
  149. <envar>LFS_TGT</envar> variable. We also use the
  150. <parameter>--with-sysroot</parameter> option when building the cross linker and
  151. cross compiler to tell them where to find the needed host files. This
  152. ensures that none of the other programs built in <xref
  153. linkend="chapter-temporary-tools"/> can link to libraries on the build
  154. machine. Only two stages are mandatory, and one more for tests:</para>
  155. <informaltable align="center">
  156. <tgroup cols="5">
  157. <colspec colnum="1" align="center"/>
  158. <colspec colnum="2" align="center"/>
  159. <colspec colnum="3" align="center"/>
  160. <colspec colnum="4" align="center"/>
  161. <colspec colnum="5" align="left"/>
  162. <thead>
  163. <row><entry>Stage</entry><entry>Build</entry><entry>Host</entry>
  164. <entry>Target</entry><entry>Action</entry></row>
  165. </thead>
  166. <tbody>
  167. <row>
  168. <entry>1</entry><entry>pc</entry><entry>pc</entry><entry>lfs</entry>
  169. <entry>build cross-compiler cc1 using cc-pc on pc</entry>
  170. </row>
  171. <row>
  172. <entry>2</entry><entry>pc</entry><entry>lfs</entry><entry>lfs</entry>
  173. <entry>build compiler cc-lfs using cc1 on pc</entry>
  174. </row>
  175. <row>
  176. <entry>3</entry><entry>lfs</entry><entry>lfs</entry><entry>lfs</entry>
  177. <entry>rebuild and test cc-lfs using itself on lfs</entry>
  178. </row>
  179. </tbody>
  180. </tgroup>
  181. </informaltable>
  182. <para>In the above table, <quote>on pc</quote> means the commands are run
  183. on a machine using the already installed distribution. <quote>On
  184. lfs</quote> means the commands are run in a chrooted environment.</para>
  185. <para>Now, there is more about cross-compiling: the C language is not
  186. just a compiler, but also defines a standard library. In this book, the
  187. GNU C library, named glibc, is used. This library must
  188. be compiled for the lfs machine, that is, using the cross compiler cc1.
  189. But the compiler itself uses an internal library implementing complex
  190. instructions not available in the assembler instruction set. This
  191. internal library is named libgcc, and must be linked to the glibc
  192. library to be fully functional! Furthermore, the standard library for
  193. C++ (libstdc++) also needs being linked to glibc. The solution to this
  194. chicken and egg problem is to first build a degraded cc1 based libgcc,
  195. lacking some functionalities such as threads and exception handling, then
  196. build glibc using this degraded compiler (glibc itself is not
  197. degraded), then build libstdc++. But this last library will lack the
  198. same functionalities as libgcc.</para>
  199. <para>This is not the end of the story: the conclusion of the preceding
  200. paragraph is that cc1 is unable to build a fully functional libstdc++, but
  201. this is the only compiler available for building the C/C++ libraries
  202. during stage 2! Of course, the compiler built during stage 2, cc-lfs,
  203. would be able to build those libraries, but (1) the build system of
  204. GCC does not know that it is usable on pc, and (2) using it on pc
  205. would be at risk of linking to the pc libraries, since cc-lfs is a native
  206. compiler. So we have to build libstdc++ later, in chroot.</para>
  207. </sect2>
  208. <sect2 id="other-details">
  209. <title>Other procedural details</title>
  210. <para>The cross-compiler will be installed in a separate <filename
  211. class="directory">$LFS/tools</filename> directory, since it will not
  212. be part of the final system.</para>
  213. <para>Binutils is installed first because the <command>configure</command>
  214. runs of both GCC and Glibc perform various feature tests on the assembler
  215. and linker to determine which software features to enable or disable. This
  216. is more important than one might first realize. An incorrectly configured
  217. GCC or Glibc can result in a subtly broken toolchain, where the impact of
  218. such breakage might not show up until near the end of the build of an
  219. entire distribution. A test suite failure will usually highlight this error
  220. before too much additional work is performed.</para>
  221. <para>Binutils installs its assembler and linker in two locations,
  222. <filename class="directory">$LFS/tools/bin</filename> and <filename
  223. class="directory">$LFS/tools/$LFS_TGT/bin</filename>. The tools in one
  224. location are hard linked to the other. An important facet of the linker is
  225. its library search order. Detailed information can be obtained from
  226. <command>ld</command> by passing it the <parameter>--verbose</parameter>
  227. flag. For example, <command>$LFS_TGT-ld --verbose | grep SEARCH</command>
  228. will illustrate the current search paths and their order. It shows which
  229. files are linked by <command>ld</command> by compiling a dummy program and
  230. passing the <parameter>--verbose</parameter> switch to the linker. For
  231. example,
  232. <command>$LFS_TGT-gcc dummy.c -Wl,--verbose 2&gt;&amp;1 | grep succeeded</command>
  233. will show all the files successfully opened during the linking.</para>
  234. <para>The next package installed is GCC. An example of what can be
  235. seen during its run of <command>configure</command> is:</para>
  236. <screen><computeroutput>checking what assembler to use... /mnt/lfs/tools/i686-lfs-linux-gnu/bin/as
  237. checking what linker to use... /mnt/lfs/tools/i686-lfs-linux-gnu/bin/ld</computeroutput></screen>
  238. <para>This is important for the reasons mentioned above. It also
  239. demonstrates that GCC's configure script does not search the PATH
  240. directories to find which tools to use. However, during the actual
  241. operation of <command>gcc</command> itself, the same search paths are not
  242. necessarily used. To find out which standard linker <command>gcc</command>
  243. will use, run: <command>$LFS_TGT-gcc -print-prog-name=ld</command>.</para>
  244. <para>Detailed information can be obtained from <command>gcc</command> by
  245. passing it the <parameter>-v</parameter> command line option while compiling
  246. a dummy program. For example, <command>gcc -v dummy.c</command> will show
  247. detailed information about the preprocessor, compilation, and assembly
  248. stages, including <command>gcc</command>'s included search paths and their
  249. order.</para>
  250. <para>Next installed are sanitized Linux API headers. These allow the
  251. standard C library (Glibc) to interface with features that the Linux
  252. kernel will provide.</para>
  253. <para>The next package installed is Glibc. The most important
  254. considerations for building Glibc are the compiler, binary tools, and
  255. kernel headers. The compiler is generally not an issue since Glibc will
  256. always use the compiler relating to the <parameter>--host</parameter>
  257. parameter passed to its configure script; e.g. in our case, the compiler
  258. will be <command>$LFS_TGT-gcc</command>. The binary tools and kernel
  259. headers can be a bit more complicated. Therefore, take no risks and use
  260. the available configure switches to enforce the correct selections. After
  261. the run of <command>configure</command>, check the contents of the
  262. <filename>config.make</filename> file in the <filename
  263. class="directory">build</filename> directory for all important details.
  264. Note the use of <parameter>CC="$LFS_TGT-gcc"</parameter> (with
  265. <envar>$LFS_TGT</envar> expanded) to control which binary tools are used
  266. and the use of the <parameter>-nostdinc</parameter> and
  267. <parameter>-isystem</parameter> flags to control the compiler's include
  268. search path. These items highlight an important aspect of the Glibc
  269. package&mdash;it is very self-sufficient in terms of its build machinery
  270. and generally does not rely on toolchain defaults.</para>
  271. <para>As said above, the standard C++ library is compiled next, followed in
  272. <xref linkend="chapter-temporary-tools"/> by all the programs that need
  273. themselves to be built. The install step of all those packages uses the
  274. <envar>DESTDIR</envar> variable to have the
  275. programs land into the LFS filesystem.</para>
  276. <para>At the end of <xref linkend="chapter-temporary-tools"/> the native
  277. lfs compiler is installed. First binutils-pass2 is built,
  278. with the same <envar>DESTDIR</envar> install as the other programs,
  279. then the second pass of GCC is constructed, omitting libstdc++
  280. and other non-important libraries. Due to some weird logic in GCC's
  281. configure script, <envar>CC_FOR_TARGET</envar> ends up as
  282. <command>cc</command> when the host is the same as the target, but is
  283. different from the build system. This is why
  284. <parameter>CC_FOR_TARGET=$LFS_TGT-gcc</parameter> is put explicitly into
  285. the configure options.</para>
  286. <para>Upon entering the chroot environment in <xref
  287. linkend="chapter-chroot-temporary-tools"/>, the first task is to install
  288. libstdc++. Then temporary installations of programs needed for the proper
  289. operation of the toolchain are performed. Programs needed for testing
  290. other programs are also built. From this point onwards, the
  291. core toolchain is self-contained and self-hosted. In
  292. <xref linkend="chapter-building-system"/>, final versions of all the
  293. packages needed for a fully functional system are built, tested and
  294. installed.</para>
  295. </sect2>
  296. </sect1>