Linux Guide:SMP Tips

From AnarchismToday.org Wiki

Utilizing Multiple Processors for Compiling

• Note that this requires multiple processors or a multi-core CPU of some sort to see any real benefit, as well as an SMP-enabled kernel running.

I noticed a file called "smp.txt" in the Documentation directory of a recent linux kernel. This file mentioned that you can use the "-j" parameter to cause a compile job to split in to multiple jobs. The recommendation from the file was to set the number to "number of CPUs + 1" though they also mentioned you may use no number at all, so long as you have sufficient memory, and encouraged experimentation as well as timing the compilations to show what sort of benefit you are seeing.

smp.txt:

To set up SMP Configure the kernel and answer Y to CONFIG_SMP. If you are using LILO, it is handy to have both SMP and non-SMP kernel images on hand. Edit /etc/lilo.conf to create an entry for another kernel image called "linux-smp" or something. The next time you compile the kernel, when running a SMP kernel, edit linux/Makefile and change "MAKE=make" to "MAKE=make -jN" (where N = number of CPU + 1, or if you have tons of memory/swap you can just use "-j" without a number). Feel free to experiment with this one. Of course you should time how long each build takes :) Example: make config time -v sh -c 'make clean install modules modules_install' If you are using some Compaq MP compliant machines you will need to set the operating system in the BIOS settings to "Unixware" - don't ask me why Compaqs don't work otherwise.

Here are 2 methods, a General Linux, Distribution Independent method, and a specific method for Debian kernel packaging.

Distribution Independent Method

For anything that uses make, or SCONS, use the -j parameter, followed by the number of cores or processors, plus one.

Rather than typing "make" to compile, I used "make -j3" for my dual-core processor.

The documentation for make provides further information:

-j [jobs], --jobs[=jobs] Specifies the number of jobs (commands) to run simultaneously. If there is more than one -j option, the last one is effective. If the -j option is given without an argument, make will not limit the number of jobs that can run simultaneously.

• NOTE: Using -j without specifying the number of jobs seems to be A Bad Idea.

-I ran out of memory after it went through my 2 GB of RAM and 2GB of swap. Maybe if you have some sort of freakish machine with 6-8 or more gigs of RAM you could pull it off. Increasing my swap size would probably just slow it down and defeat the purpose of splitting a compile into multiple jobs across multiple processors or cores in the first place.

Debian GNU/Linux Kernel Building and Packaging

Prerequisite:

• Linux Kernel source, properly configured.

If you don't have them already, you will need to install kernel-package, and the essential build packages:

aptitude install build-essential kernel-package

For make-kpkg to use extra cores, we need to set the environment variable "CONCURRENCY_LEVEL" to the number of cores, plus one. An example from my dual-core system, I enter this at the console:

CONCURRENCY_LEVEL=3

Then run make-kpkg to build the kernel and create the kernel package, for this example, we'll assume the kernel source is located in /usr/src/linux, and has been properly configured:

cd /usr/src/linux

make-kpkg --initrd --rev Custom.1 kernel_image

The kernel should begin building, utilizing the additional cores.