Idphix
From Digitalis
Contents |
Update
- Idphix now hosts a Nvidia K40 GPU along with the KNC Xeon Phi and a Infiniband host adapter.
- The OS is Grid'5000's Debian stable
If needed one can get the Centos OS back using kadeploy.
Information below are mostly unrelevant with the Debian OS.
Overview
Idphix is a single socket Sandy Bridge machine, equipped with 1 MIC accelerator.
- 1x Intel Xeon E5-2650 (8 cores)
- 64 GB RAM
- 1x coprocesseur Intel Xeon Phi 5110P
The setup is the following: We use an internal network on the host (idphix) for the Xeon Phi: 192.168.1.0/24.
This gives the following scheme:
======================================= | ------------- ----------- | || host | | Xeon Phi | | || idphix | |idphix-mic0| | Grid5000 network <---> eth0 mic0 <----> mic0 | | 172.16.21.6 192.168.1.254 192.168.1.1 | | || | | | | | ------------- ----------- | =======================================
How to experiment ?
For now (2013-09-13) the operating system is a Centos 6 (OS officially supported by Intel for the Xeon Phi).
Use the machine as follows:
Connect to digitalis/grid5000
workstation$ ssh digitalis.grenoble.g5k
In case of any issue (network issue, connection hanging...) please read more info here or ask on the Digitalis mailing list.
Reserve the machine/run a job
digitalis$ oarsub -p "machine='idphix'" -l walltime=2 -I
Setup your access to the Xeon Phi
(This might not be relevant if your are using some level of abstraction (COI, ...)
- WARNING
- there is currently a bug due to the last version of Intel MPSS, please report any issue to the mailing list.
Make sure you created your G5K ssh keys (keys without passephrase protection, dedicated to G5K and stored in your NFS home directory)
Run:
idphix$ sudo /usr/local/bin/mic-setup-my-user
If you get the error:
Error: Cannot get your SSH keys, please run 'chmod 755 ~/.ssh && chmod 644 ~/.ssh/{authorized_keys,id_*.pub}'
Please run:
idphix$ chmod 755 ~/.ssh && chmod 644 ~/.ssh/{authorized_keys,id_*.pub}
Output should be:
idphix$ sudo /usr/local/bin/mic-setup-my-user *** Configure user pneyron:users (10106:8000) on mic0 *** Copy user files to mic0 id_rsa.pub 100% 420 0.4KB/s 00:00 authorized_keys 100% 916 0.9KB/s 00:00 .profile 100% 62 0.1KB/s 00:00 *** Force creation of pneyron:users on mic0 and fix file ownership and permission Done
You shoud now be able to ssh to the coprocessor:
idphix$ ssh mic0 idphix-mic0$ uname -a Linux idphix-mic0.grenoble.grid5000.fr 2.6.38.8-g5f2543d #2 SMP Tue Apr 30 14:05:06 PDT 2013 k1om k1om k1om GNU/Linux
Intel compiler and other tools
Intel tools (icc & co) are installed in /opt/intel
It is configured to use Inria's license server (FlexLM).
This licence server provides 2 tokens which are shared among all Inria's users. Only 2 compilations can be performed at a time.
If icc complains about the license (error message talking about FlexLM), please report the issue to digitalis@lists.grid5000.fr
Rebooting the MIC
If needed, you can reboot the MIC from within a exclusive job (doing so in shared job session would risk annoying other users).
oarsub -t exclusive -p "machine='idphix'" ....
Then, you will be able to run
sudo /usr/local/bin/mic-reboot
(no password should be requested).
Rebooting the MIC takes some time. You can look at the output of dmesg
or miccrtl -s
meanwhile.
Data on the mic: no persistence
Please mind the fact that data pushed to the mic are not persistent. Your home directory on idphix-mic0 could be emptied for various reasons.
Privileged commands
Currently, the following commands can be run via sudo in exclusive jobs:
- sudo /usr/bin/whoami (provided for testing the mechanism, should return "root")
- sudo /sbin/reboot
- sudo /usr/bin/schedtool
- sudo /usr/local/bin/mic-reboot
- sudo /usr/local/bin/mic-pm
- NB
- sudo /usr/local/bin/mic-setup-my-user does not require an exclusive job.
- PM setting are not reset automatically. Please everyone check the PM settings (using micctl --pm or the mic-pm script) before running experiments
Redeploy and get full (root) access to the machine/MIC
One can be root on the machine using kadeploy.
To redeploy your own instance of idphix default system (Centos), add '-t redeploy
' to your oarsub command and run the following kadeploy command:
kadeploy -e idphix-default -m idphix -k
Then you should be able to do:
digitalis$ ssh root@idphix idphix$ ssh mic0 idphix-mic$ whoami root
At the end of the job, the machine will return to its default system environment.
Other experimental platforms with MICs
Grid'5000
4 nodes with 1 Xeon Phi each are available on Grid'5000 Nancy's site, in the Graphite cluster. See:
- https://www.grid5000.fr/mediawiki/index.php/Nancy:Hardware#Graphite
- https://www.grid5000.fr/mediawiki/index.php/Accelerators_on_Grid%275000
Irisa
Guillermo Andrade-barroso installed a Xeon Phi host server at INRIA Rennes. This server hosts 1 Intel® Xeon Phi™ 5110P (8 G, 1.053 GHz, 60 cores) under Debian 7.0 64 bits.
Access to this server is available for all INRIA members on demand.
The machine name is phileas.irisa.fr
Bordeaux - Plafrim
https://plafrim.bordeaux.inria.fr/doku.php?id=plateforme:configurations:mistral
System changelogs
2013-09-26
- System image
- idphix-default 20130926
- Centos 6
- Linux 2.6.32-358.el6.x86_64
- MPSS packages
intel-mic-flash-2.1.386-2.el6.x86_64 intel-mic-kmod-2.1.6720-13.2.6.32.358.el6.x86_64 intel-mic-sysmgmt-2.1.6720-13.el6.x86_64 intel-mic-gpl-2.1.6720-13.el6.x86_64 intel-mic-mpm-2.1.6720-13.el6.x86_64 intel-mic-2.1.6720-13.el6.x86_64 intel-mic-micmgmt-2.1.6720-13.2.6.32.358.el6.x86_64 intel-mic-perf-data-2.1.6720-13.el6.x86_64 intel-mic-gdb-2.1.6720-13.el6.x86_64 intel-mic-cdt-2.1.6720-13.el6.x86_64 intel-mic-perf-2.1.6720-13.el6.x86_64
- Intel Cluster Studio XE 2013
- MIC info
[root@idphix ~]# /opt/intel/mic/bin/micinfo MicInfo Utility Log Created Mon Dec 9 20:13:37 2013 System Info HOST OS : Linux OS Version : 2.6.32-358.el6.x86_64 Driver Version : 6720-13 MPSS Version : 2.1.6720-13 Host Physical Memory : 66077 MB Device No: 0, Device Name: mic0 Version Flash Version : 2.1.02.0386 SMC Firmware Version : 1.14.4616 SMC Boot Loader Version : 1.8.4326 uOS Version : 2.6.38.8-g5f2543d Device Serial Number : ADKC32100375 Board Vendor ID : 0x8086 Device ID : 0x2250 Subsystem ID : 0x2500 Coprocessor Stepping ID : 3 PCIe Width : x16 PCIe Speed : 5 GT/s PCIe Max payload size : 256 bytes PCIe Max read req size : 512 bytes Coprocessor Model : 0x01 Coprocessor Model Ext : 0x00 Coprocessor Type : 0x00 Coprocessor Family : 0x0b Coprocessor Family Ext : 0x00 Coprocessor Stepping : B1 Board SKU : INVALID SKU ECC Mode : Enabled SMC HW Revision : Product 225W Passive CS Cores Total No of Active Cores : 60 Voltage : 985000 uV Frequency : 1052631 kHz Thermal Fan Speed Control : N/A Fan RPM : N/A Fan PWM : N/A Die Temp : 35 C GDDR GDDR Vendor : Elpida GDDR Version : 0x1 GDDR Density : 2048 Mb GDDR Size : 7936 MB GDDR Technology : GDDR5 GDDR Speed : 5.000000 GT/s GDDR Frequency : 2500000 kHz GDDR Voltage : 1501000 uV
2013-12-09
- New system image
- idphix-default 20131209
- Updated Centos packages, except linux kernel
- Moved Intel Cluster Studio XE 2013 to NFS /applis/digitalis/intel
- No change to the MIC software (MPSS)
2013-12-13
- New system image
- idphix-default 20131213
- Activated Hyperthreading
[root@idphix ~]# lstopo --no-caches Machine (63GB) + Socket L#0 Core L#0 PU L#0 (P#0) PU L#1 (P#8) Core L#1 PU L#2 (P#1) PU L#3 (P#9) Core L#2 PU L#4 (P#2) PU L#5 (P#10) Core L#3 PU L#6 (P#3) PU L#7 (P#11) Core L#4 PU L#8 (P#4) PU L#9 (P#12) Core L#5 PU L#10 (P#5) PU L#11 (P#13) Core L#6 PU L#12 (P#6) PU L#13 (P#14) Core L#7 PU L#14 (P#7) PU L#15 (P#15)
- Updated MPSS 2.1 -> 3.1.1
[root@idphix ~]# rpm -qa | grep -e intel -e mpss mpss-miccheck-3.1.1-r1.glibc2.12.2.x86_64 mpss-sysmgmt-python-3.1.1-1.glibc2.12.2.x86_64 mpss-sciftutorials-3.1.1-1.glibc2.12.2.x86_64 mpss-eclipse-cdt-mpm-3.1.1-1.glibc2.12.2.x86_64 mpss-myo-doc-3.1.1-1.glibc2.12.2.x86_64 mpss-coi-doc-3.1.1-1.glibc2.12.2.x86_64 mpss-flash-3.1.1-1.glibc2.12.2.x86_64 mpss-myo-3.1.1-1.glibc2.12.2.x86_64 mpss-metadata-3.1.1-1.glibc2.12.2.x86_64 mpss-myo-dev-3.1.1-1.glibc2.12.2.x86_64 intel-composerxe-compat-k1om-3.1.1-1.x86_64 mpss-micmgmt-doc-3.1.1-1.glibc2.12.2.x86_64 mpss-sysmgmt-micsmc-gui-3.1.1-1.glibc2.12.2.x86_64 mpss-miccheck-bin-3.1.1-r1.glibc2.12.2.x86_64 mpss-sysmgmt-micras-3.1.1-1.glibc2.12.2.x86_64 mpss-modules-2.6.32-358.el6.x86_64-3.1.1-1.el6.x86_64 mpss-modules-dev-2.6.32-358.el6.x86_64-3.1.1-1.el6.x86_64 mpss-sciftutorials-doc-3.1.1-1.glibc2.12.2.x86_64 mpss-license-3.1.1-1.glibc2.12.2.x86_64 mpss-rasmm-kernel-3.1.1-1.glibc2.12.2.x86_64 mpss-micmgmt-python-3.1.1-1.glibc2.12.2.x86_64 mpss-coi-3.1.1-1.glibc2.12.2.x86_64 mpss-modules-headers-3.1.1-1.glibc2.12.2.x86_64 mpss-sdk-k1om-3.1.1-1.x86_64 mpss-metadata-dev-3.1.1-1.glibc2.12.2.x86_64 mpss-daemon-dev-3.1.1-1.glibc2.12.2.x86_64 mpss-coi-dev-3.1.1-1.glibc2.12.2.x86_64 mpss-micmgmt-3.1.1-1.glibc2.12.2.x86_64 mpss-mpm-doc-3.1.1-1.glibc2.12.2.x86_64 mpss-mpm-3.1.1-1.glibc2.12.2.x86_64 mpss-boot-files-3.1.1-1.glibc2.12.2.x86_64 mpss-daemon-3.1.1-1.glibc2.12.2.x86_64 mpss-modules-headers-dev-3.1.1-1.glibc2.12.2.x86_64
- Updated the MIC SMC and Flash (part of the MPSS upgrade)
[root@idphix ~]# micinfo MicInfo Utility Log Created Fri Dec 13 19:00:30 2013 System Info HOST OS : Linux OS Version : 2.6.32-358.el6.x86_64 Driver Version : 3.1.1-1 MPSS Version : 3.1.1 Host Physical Memory : 66076 MB Device No: 0, Device Name: mic0 Version Flash Version : 2.1.02.0390 SMC Firmware Version : 1.16.5078 SMC Boot Loader Version : 1.8.4326 uOS Version : 2.6.38.8+mpss3.1.1 Device Serial Number : ADKC32100375 Board Vendor ID : 0x8086 Device ID : 0x2250 Subsystem ID : 0x2500 Coprocessor Stepping ID : 3 PCIe Width : x16 PCIe Speed : 5 GT/s PCIe Max payload size : 256 bytes PCIe Max read req size : 512 bytes Coprocessor Model : 0x01 Coprocessor Model Ext : 0x00 Coprocessor Type : 0x00 Coprocessor Family : 0x0b Coprocessor Family Ext : 0x00 Coprocessor Stepping : B1 Board SKU : B1PRQ-5110P/5120D ECC Mode : Enabled SMC HW Revision : Product 225W Passive CS Cores Total No of Active Cores : 60 Voltage : 1021000 uV Frequency : 1052631 kHz Thermal Fan Speed Control : N/A Fan RPM : N/A Fan PWM : N/A Die Temp : 36 C GDDR GDDR Vendor : Elpida GDDR Version : 0x1 GDDR Density : 2048 Mb GDDR Size : 7936 MB GDDR Technology : GDDR5 GDDR Speed : 5.000000 GT/s GDDR Frequency : 2500000 kHz GDDR Voltage : 1501000 uV
2014-02-17
Installed Intel Cluster Studio XE 2013.1 in /grid5000/software/. It comes with icc 14. (/grid5000 is a NFS mount, commonly usable by all machines)
Vtune is part of Intel Cluster Studio XE 2013.1, but needs more work to be fully functional.
2014-02-19
- New system image
- idphix-default 20140219
- Fixed mic-setup-my-user which was broken since the upgrade to MPSS 3.1
- Added mount point for /grid5000
- Changed /opt/intel link to /grid5000/software/intel, which as of 2014-02-17 links to Intel Cluster Studio XE 2013.1
- Added packages:
- emacs-common-23.1-25.el6.x86_64
- emacs-nox-23.1-25.el6.x86_64
- autoconf-2.63-5.1.el6.noarch
- automake-1.11.1-4.el6.noarch
- automake16-1.6.3-18.el6.1.noarch
- automake15-1.5-27.el6.1.noarch
- automake14-1.4p6-19.2.el6.noarch
- autoconf213-2.13-20.1.el6.noarch
- autoconf-archive-2012.09.08-1.el6.noarch
- tcl-8.5.7-6.el6.x86_64
- expect-5.44.1.15-5.el6_4.x86_64
- dejagnu-1.4.4-17.el6.noarch
- tig-0.17-1.el6.x86_64
2014-02-26
- New system image
- idphix-default 20140226
- Changed /opt/intel to be a directory containing symlinks to /grid5000/software/intel/*, so that Intel OpenCL SDK can be installed.
- Installed rpms:
- svn
- ntp
- Intalled Intel OpenCL SDK (intel_sdk_for_ocl_applications_xe_2013_r3_sdk_3.2.1.16712_x64)
- install-cpu+mic.sh
2014-04-08
- New system image
- idphix-default 20140408
This image:
- upgrades the system to CentOS 6.5 (required to build Vtune kernel modules)
- installs VNC server + twm + xterm + xeyes
- brings support for Intel Vtune (disable nmi_watchdog + setup Vtune kernel modules)
- installs MPSS version 3.2
- adds script mic-pm to allow a user in an exclusive job to change the power-management settings of the mic
- yum logs for CentOS 6.5 upgrade
Apr 05 11:08:07 Installed: tigervnc-1.1.0-8.el6_5.x86_64 Apr 05 11:09:10 Installed: hal-libs-0.5.14-11.el6.x86_64 Apr 05 11:09:11 Installed: xkeyboard-config-2.6-6.el6.noarch Apr 05 11:09:13 Installed: mtdev-1.1.2-5.el6.x86_64 Apr 05 11:09:14 Installed: cryptsetup-luks-libs-1.2.0-7.el6.x86_64 Apr 05 11:09:14 Installed: cryptsetup-luks-1.2.0-7.el6.x86_64 Apr 05 11:09:14 Installed: ConsoleKit-libs-0.4.1-3.el6.x86_64 Apr 05 11:09:15 Installed: libxkbfile-1.0.6-1.1.el6.x86_64 Apr 05 11:09:15 Installed: xorg-x11-xkb-utils-7.7-4.el6.x86_64 Apr 05 11:09:15 Installed: xorg-x11-server-common-1.13.0-23.1.el6.centos.x86_64 Apr 05 11:09:16 Installed: eggdbus-0.6-3.el6.x86_64 Apr 05 11:09:16 Installed: ConsoleKit-0.4.1-3.el6.x86_64 Apr 05 11:09:16 Installed: polkit-0.96-5.el6_4.x86_64 Apr 05 11:09:17 Installed: hdparm-9.43-4.el6.x86_64 Apr 05 11:09:17 Installed: hal-info-20090716-3.1.el6.noarch Apr 05 11:09:17 Installed: pm-utils-1.2.5-10.el6.x86_64 Apr 05 11:09:18 Installed: hal-0.5.14-11.el6.x86_64 Apr 05 11:09:19 Installed: system-setup-keyboard-0.7-4.el6.x86_64 Apr 05 11:09:19 Installed: libXdmcp-1.1.1-3.el6.x86_64 Apr 05 11:09:19 Installed: xorg-x11-drv-evdev-2.7.3-5.el6.x86_64 Apr 05 11:09:20 Installed: xorg-x11-drv-void-1.4.0-3.el6.x86_64 Apr 05 11:09:20 Installed: xorg-x11-drv-vesa-2.3.2-4.el6.x86_64 Apr 05 11:09:20 Installed: xorg-x11-server-Xorg-1.13.0-23.1.el6.centos.x86_64 Apr 05 11:09:21 Installed: tigervnc-server-module-1.1.0-8.el6_5.x86_64 Apr 05 11:25:39 Updated: kernel-firmware-2.6.32-431.11.2.el6.noarch Apr 05 11:25:46 Installed: kernel-2.6.32-431.11.2.el6.x86_64 Apr 05 11:25:52 Installed: kernel-devel-2.6.32-431.11.2.el6.x86_64 Apr 05 11:25:53 Updated: kernel-headers-2.6.32-431.11.2.el6.x86_64 Apr 05 12:00:36 Installed: xorg-x11-fonts-misc-7.2-9.1.el6.noarch Apr 05 12:00:36 Installed: tigervnc-server-1.1.0-8.el6_5.x86_64 Apr 05 12:03:20 Installed: 1:xorg-x11-twm-1.0.3-5.1.el6.x86_64 Apr 05 12:05:03 Installed: libXpm-3.5.10-2.el6.x86_64 Apr 05 12:05:04 Installed: libXaw-1.0.11-2.el6.x86_64 Apr 05 12:05:04 Installed: xterm-253-1.el6.x86_64 Apr 05 12:15:45 Installed: xorg-x11-apps-7.6-6.el6.x86_64 Apr 05 13:28:12 Updated: nspr-4.10.2-1.el6_5.x86_64 Apr 05 13:28:13 Updated: nss-util-3.15.3-1.el6_5.x86_64 Apr 05 13:28:14 Updated: glib2-2.26.1-7.el6_5.x86_64 Apr 05 13:28:14 Updated: libjpeg-turbo-1.2.1-3.el6_5.x86_64 Apr 05 13:28:15 Updated: grep-2.6.3-4.el6_5.1.x86_64 Apr 05 13:28:15 Updated: coreutils-libs-8.4-31.el6_5.1.x86_64 Apr 05 13:28:17 Updated: coreutils-8.4-31.el6_5.1.x86_64 Apr 05 13:28:18 Updated: krb5-libs-1.10.3-15.el6_5.1.x86_64 Apr 05 13:28:18 Updated: p11-kit-0.18.5-2.el6_5.2.x86_64 Apr 05 13:28:19 Updated: p11-kit-trust-0.18.5-2.el6_5.2.x86_64 Apr 05 13:28:20 Updated: ca-certificates-2013.1.95-65.1.el6_5.noarch Apr 05 13:28:21 Updated: openssl-1.0.1e-16.el6_5.4.x86_64 Apr 05 13:28:22 Updated: 1:qt-4.6.2-28.el6_5.x86_64 Apr 05 13:28:23 Updated: 1:qt-sqlite-4.6.2-28.el6_5.x86_64 Apr 05 13:28:23 Updated: python-2.6.6-52.el6.x86_64 Apr 05 13:28:25 Updated: python-libs-2.6.6-52.el6.x86_64 Apr 05 13:28:26 Updated: yum-plugin-fastestmirror-1.1.30-17.el6_5.noarch Apr 05 13:28:26 Updated: yum-3.2.29-43.el6.centos.noarch Apr 05 13:28:27 Updated: mysql-libs-5.1.73-3.el6_5.x86_64 Apr 05 13:28:28 Updated: selinux-policy-3.7.19-231.el6_5.1.noarch Apr 05 13:28:29 Updated: nss-sysinit-3.15.3-6.el6_5.x86_64 Apr 05 13:28:29 Updated: nss-3.15.3-6.el6_5.x86_64 Apr 05 13:28:30 Updated: nss-tools-3.15.3-6.el6_5.x86_64 Apr 05 13:28:30 Updated: openldap-2.4.23-34.el6_5.1.x86_64 Apr 05 13:28:30 Updated: libtiff-3.9.4-10.el6_5.x86_64 Apr 05 13:28:31 Updated: 2:ethtool-3.5-1.2.el6_5.x86_64 Apr 05 13:28:32 Updated: tzdata-java-2014b-1.el6.noarch Apr 05 13:28:32 Updated: psmisc-22.6-19.el6_5.x86_64 Apr 05 13:28:33 Updated: upstart-0.6.5-13.el6_5.3.x86_64 Apr 05 13:28:34 Updated: initscripts-9.03.40-2.el6.centos.1.x86_64 Apr 05 13:28:34 Updated: mesa-dri-filesystem-9.2-0.5.el6_5.2.x86_64 Apr 05 13:28:36 Updated: mesa-dri-drivers-9.2-0.5.el6_5.2.x86_64 Apr 05 13:28:36 Updated: mesa-libGL-9.2-0.5.el6_5.2.x86_64 Apr 05 13:28:36 Updated: mesa-libGLU-9.2-0.5.el6_5.2.x86_64 Apr 05 13:28:37 Updated: 1:phonon-backend-gstreamer-4.6.2-28.el6_5.x86_64 Apr 05 13:28:39 Updated: 1:qt-x11-4.6.2-28.el6_5.x86_64 Apr 05 13:28:40 Updated: subversion-1.6.11-10.el6_5.x86_64 Apr 05 13:28:42 Updated: 2:postfix-2.6.6-6.el6_5.x86_64 Apr 05 13:28:49 Updated: 1:java-1.7.0-openjdk-1.7.0.51-2.4.4.1.el6_5.x86_64 Apr 05 13:28:49 Updated: openldap-clients-2.4.23-34.el6_5.1.x86_64 Apr 05 13:29:17 Updated: selinux-policy-targeted-3.7.19-231.el6_5.1.noarch Apr 05 13:29:18 Updated: wget-1.12-1.11.el6_5.x86_64 Apr 05 13:29:19 Updated: tzdata-2014b-1.el6.noarch Apr 05 13:29:20 Updated: pixman-0.26.2-5.1.el6_5.x86_64 Apr 05 13:29:20 Updated: openjpeg-libs-1.3-10.el6_5.x86_64 Apr 05 13:29:21 Updated: libtirpc-0.2.1-6.el6_5.1.x86_64 Apr 05 13:29:21 Updated: libXfont-1.4.5-3.el6_5.x86_64 Apr 05 13:29:22 Updated: gnutls-2.8.5-13.el6_5.x86_64 Apr 05 13:29:22 Updated: 1:dmidecode-2.12-5.el6_5.x86_64 Apr 05 13:29:23 Updated: etckeeper-1.11-1.el6.noarch
- Vtune installation
cp -a /opt/intel/vtune_amplifier_xe/sepdk /tmp cd /tmp/sepdk/src build-driver --install-dir=/tmp/sepdk/prebuilt copy /tmp/sepdk/prebuilt/* to /opt/intel/vtune_amplifier_xe/sepdk/prebuilt/ #(/!\ nfs is RO) boot-script -g users -d /opt/intel/vtune_amplifier_xe/sepdk/prebuilt/ echo kernel.nmi_watchdog = 0 >> /etc/sysctl.conf reboot
- MPSS 3.2 installation
tar xvf mpss-3.2-rhel-6.5.tar cd mpss-3.2 ./uninstall.sh rpm -ev opencl-1.2-intel-mic rpm -ivh *.rpm perf/*.rpm psm/*.rpm rpm -ivh src/mpss-modules-3.2-1.el6.src.rpm yum install rpm-build rpmlint rpmdevtools cd ~/rpmbuild && rpmbuild -ba SPECS/mpss-modules.spec rpm -ivh RPMS/mpss-modules*.rpm rpm -ivh opencl-1.2-intel-mic-3.2.1.16712-1.x86_64.rpm #(from intel_sdk_for_ocl_applications_xe_2013_r3_sdk_3.2.1.16712_x64) reboot
- MIC state snapshot
$ micctrl --pm mic0: cpufreq_on;corec6_off;pc3_on;pc6_off $ micinfo MicInfo Utility Log Copyright 2011-2013 Intel Corporation All Rights Reserved. Created Tue Apr 8 16:27:59 2014 System Info HOST OS : Linux OS Version : 2.6.32-431.11.2.el6.x86_64 Driver Version : 3.2-1 MPSS Version : 3.2 Host Physical Memory : 66076 MB Device No: 0, Device Name: mic0 Version Flash Version : 2.1.02.0390 SMC Firmware Version : 1.16.5078 SMC Boot Loader Version : 1.8.4326 uOS Version : 2.6.38.8+mpss3.2 Device Serial Number : ADKC32100375 Board Vendor ID : 0x8086 Device ID : 0x2250 Subsystem ID : 0x2500 Coprocessor Stepping ID : 3 PCIe Width : x16 PCIe Speed : 5 GT/s PCIe Max payload size : 256 bytes PCIe Max read req size : 512 bytes Coprocessor Model : 0x01 Coprocessor Model Ext : 0x00 Coprocessor Type : 0x00 Coprocessor Family : 0x0b Coprocessor Family Ext : 0x00 Coprocessor Stepping : B1 Board SKU : B1PRQ-5110P/5120D ECC Mode : Enabled SMC HW Revision : Product 225W Passive CS Cores Total No of Active Cores : 60 Voltage : 1021000 uV Frequency : 1052631 kHz Thermal Fan Speed Control : N/A Fan RPM : N/A Fan PWM : N/A Die Temp : 36 C GDDR GDDR Vendor : Elpida GDDR Version : 0x1 GDDR Density : 2048 Mb GDDR Size : 7936 MB GDDR Technology : GDDR5 GDDR Speed : 5.000000 GT/s GDDR Frequency : 2500000 kHz GDDR Voltage : 1501000 uV
2015-06-08
- New system image
- idphix-default 20150608
This image:
- upgrades the system to Centos 6.6
- use Linux kernel 2.6.32-431.11.2.el6.x86_64
- installs MPSS version 3.5.1
- yum logs for CentOS 6.6 upgrade
Updated: libgcc-4.4.7-11.el6.x86_64 Updated: xkeyboard-config-2.11-1.el6.noarch Updated: kernel-headers-2.6.32-504.16.2.el6.x86_64 Updated: centos-release-6-6.el6.centos.12.2.x86_64 Updated: tzdata-2015d-1.el6.noarch Updated: nss-softokn-freebl-3.14.3-22.el6_6.x86_64 Updated: glibc-common-2.12-1.149.el6_6.9.x86_64 Updated: glibc-2.12-1.149.el6_6.9.x86_64 Updated: bash-4.1.2-29.el6.x86_64 Updated: libselinux-2.0.94-5.8.el6.x86_64 Updated: 1:perl-Pod-Escapes-1.04-136.el6_6.1.x86_64 Updated: 3:perl-version-0.77-136.el6_6.1.x86_64 Updated: 1:perl-Module-Pluggable-3.90-136.el6_6.1.x86_64 Updated: 4:perl-libs-5.10.1-136.el6_6.1.x86_64 Updated: 1:perl-Pod-Simple-3.13-136.el6_6.1.x86_64 Updated: 4:perl-5.10.1-136.el6_6.1.x86_64 Updated: libstdc++-4.4.7-11.el6.x86_64 Updated: libcom_err-1.41.12-21.el6.x86_64 Updated: audit-libs-2.3.7-5.el6.x86_64 Updated: grep-2.6.3-6.el6.x86_64 Updated: coreutils-libs-8.4-37.el6.x86_64 Updated: coreutils-8.4-37.el6.x86_64 Updated: pam-1.1.1-20.el6.x86_64 Updated: nspr-4.10.8-1.el6_6.x86_64 Updated: 1:dbus-libs-1.2.24-8.el6_6.x86_64 Updated: 2:shadow-utils-4.1.4.2-19.el6_6.1.x86_64 Updated: nss-util-3.18.0-1.el6_6.x86_64 Updated: xz-libs-4.999.9-0.5.beta.20091007git.el6.x86_64 Updated: elfutils-libelf-0.158-3.2.el6.x86_64 Updated: file-libs-5.04-21.el6.x86_64 Updated: 1:dbus-1.2.24-8.el6_6.x86_64 Updated: freetype-2.3.11-15.el6_6.1.x86_64 Updated: fontconfig-2.8.0-5.el6.x86_64 Updated: libuuid-2.17.2-12.18.el6.x86_64 Updated: libblkid-2.17.2-12.18.el6.x86_64 Updated: pciutils-libs-3.1.10-4.el6.x86_64 Updated: libudev-147-2.57.el6.x86_64 Updated: cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 Updated: file-5.04-21.el6.x86_64 Updated: cpio-2.10-12.el6_5.x86_64 Updated: binutils-2.20.51.0.2-5.42.el6.x86_64 Updated: module-init-tools-3.9-24.el6.x86_64 Updated: hwdata-0.233-11.1.el6.noarch Updated: pixman-0.32.4-4.el6.x86_64 Updated: keyutils-libs-1.4-5.el6.x86_64 Updated: krb5-libs-1.10.3-37.el6_6.x86_64 Updated: libxml2-2.7.6-17.el6_6.1.x86_64 Updated: glib2-2.28.8-4.el6.x86_64 Updated: shared-mime-info-0.70-6.el6.x86_64 Updated: libpciaccess-0.13.3-0.1.el6.x86_64 Updated: libXfont-1.4.5-4.el6_6.x86_64 Updated: hal-libs-0.5.14-14.el6.x86_64 Updated: xz-4.999.9-0.5.beta.20091007git.el6.x86_64 Updated: nss-softokn-3.14.3-22.el6_6.x86_64 Updated: nss-3.18.0-5.3.el6_6.x86_64 Updated: nss-sysinit-3.18.0-5.3.el6_6.x86_64 Updated: avahi-libs-0.6.25-15.el6.x86_64 Updated: libproxy-python-0.3.0-10.el6.x86_64 Updated: libproxy-0.3.0-10.el6.x86_64 Updated: libproxy-bin-0.3.0-10.el6.x86_64 Updated: ca-certificates-2015.2.4-65.0.1.el6_6.noarch Updated: openssl-1.0.1e-30.el6.9.x86_64 Updated: libssh2-1.4.2-1.el6_6.1.x86_64 Updated: plymouth-scripts-0.8.3-27.el6.centos.1.x86_64 Updated: libselinux-utils-2.0.94-5.8.el6.x86_64 Updated: libgomp-4.4.7-11.el6.x86_64 Updated: libxcb-1.9.1-2.el6.x86_64 Updated: gettext-0.17-18.el6.x86_64 Installed: OpenIPMI-libs-2.0.16-14.el6.x86_64 Updated: nss-tools-3.18.0-5.3.el6_6.x86_64 Updated: libcurl-7.19.7-40.el6_6.4.x86_64 Updated: curl-7.19.7-40.el6_6.4.x86_64 Updated: rpm-libs-4.8.0-38.el6_6.x86_64 Updated: rpm-4.8.0-38.el6_6.x86_64 Updated: openldap-2.4.39-8.el6.x86_64 Updated: rpm-python-4.8.0-38.el6_6.x86_64 Updated: yum-plugin-fastestmirror-1.1.30-30.el6.noarch Updated: yum-3.2.29-60.el6.centos.noarch Updated: xz-lzma-compat-4.999.9-0.5.beta.20091007git.el6.x86_64 Updated: polkit-0.96-7.el6.x86_64 Installed: libIDL-0.8.13-2.1.el6.x86_64 Installed: ORBit2-2.14.17-5.el6.x86_64 Updated: grubby-7.0.15-7.el6.x86_64 Updated: keyutils-1.4-5.el6.x86_64 Updated: xorg-x11-server-common-1.15.0-26.el6.centos.x86_64 Updated: elfutils-libs-0.158-3.2.el6.x86_64 Updated: elfutils-0.158-3.2.el6.x86_64 Updated: nscd-2.12-1.149.el6_6.9.x86_64 Installed: sgml-common-0.6.3-33.el6.noarch Installed: GConf2-2.28.0-6.el6.x86_64 Updated: libss-1.41.12-21.el6.x86_64 Updated: e2fsprogs-libs-1.41.12-21.el6.x86_64 Updated: libstdc++-devel-4.4.7-11.el6.x86_64 Updated: mesa-private-llvm-3.4-3.el6.x86_64 Updated: procps-3.2.8-30.el6.x86_64 Updated: less-436-13.el6.x86_64 Updated: gzip-1.3.12-22.el6.x86_64 Updated: unzip-6.0-2.el6_6.x86_64 Updated: gdb-7.2-75.el6.x86_64 Updated: cpp-4.4.7-11.el6.x86_64 Updated: glibc-headers-2.12-1.149.el6_6.9.x86_64 Updated: glibc-devel-2.12-1.149.el6_6.9.x86_64 Updated: 1:perl-ExtUtils-ParseXS-2.2003.0-136.el6_6.1.x86_64 Updated: 4:perl-devel-5.10.1-136.el6_6.1.x86_64 Updated: perl-Test-Harness-3.17-136.el6_6.1.x86_64 Updated: perl-ExtUtils-MakeMaker-6.55-136.el6_6.1.x86_64 Updated: gcc-4.4.7-11.el6.x86_64 Updated: libtirpc-0.2.1-10.el6.x86_64 Updated: jasper-libs-1.900.1-16.el6_6.3.x86_64 Updated: libtasn1-2.3-6.el6_5.x86_64 Updated: gnutls-2.8.5-14.el6_5.x86_64 Updated: 1:cups-libs-1.4.2-67.el6.x86_64 Updated: libgfortran-4.4.7-11.el6.x86_64 Installed: lm_sensors-libs-3.1.1-17.el6.x86_64 Installed: 1:net-snmp-libs-5.5-50.el6_6.1.x86_64 Installed: OpenIPMI-2.0.16-14.el6.x86_64 Updated: numactl-2.0.9-2.el6.x86_64 Updated: plymouth-core-libs-0.8.3-27.el6.centos.1.x86_64 Updated: 2:ethtool-3.5-5.el6.x86_64 Updated: iptables-1.4.7-14.el6.x86_64 Updated: iproute-2.6.32-33.el6_6.x86_64 Updated: initscripts-9.03.46-1.el6.centos.1.x86_64 Updated: util-linux-ng-2.17.2-12.18.el6.x86_64 Updated: udev-147-2.57.el6.x86_64 Updated: policycoreutils-2.0.83-19.47.el6_6.1.x86_64 Updated: libdrm-2.4.52-4.el6.x86_64 Updated: openssh-5.3p1-104.el6_6.1.x86_64 Updated: plymouth-0.8.3-27.el6.centos.1.x86_64 Updated: dracut-004-356.el6_6.2.noarch Updated: dracut-kernel-004-356.el6_6.2.noarch Updated: selinux-policy-3.7.19-260.el6_6.3.noarch Updated: device-mapper-libs-1.02.90-2.el6_6.3.x86_64 Updated: device-mapper-1.02.90-2.el6_6.3.x86_64 Updated: cryptsetup-luks-libs-1.2.0-11.el6.x86_64 Updated: cryptsetup-luks-1.2.0-11.el6.x86_64 Updated: pm-utils-1.2.5-11.el6.x86_64 Updated: hal-0.5.14-14.el6.x86_64 Updated: nfs-utils-lib-1.1.5-9.el6.x86_64 Updated: 1:nfs-utils-1.2.3-54.el6.x86_64 Updated: ntpdate-4.2.6p5-3.el6.centos.x86_64 Updated: iptables-ipv6-1.4.7-14.el6.x86_64 Updated: mesa-dri-filesystem-10.1.2-2.el6.x86_64 Updated: mesa-dri-drivers-10.1.2-2.el6.x86_64 Updated: 12:dhcp-common-4.1.1-43.P1.el6.centos.1.x86_64 Updated: tzdata-java-2015d-1.el6.noarch Updated: libX11-common-1.6.0-2.2.el6.noarch Updated: libX11-1.6.0-2.2.el6.x86_64 Updated: libXext-1.3.2-2.1.el6.x86_64 Updated: libXrender-0.9.8-2.1.el6.x86_64 Updated: libXi-1.7.2-2.2.el6.x86_64 Updated: libXtst-1.2.2-2.1.el6.x86_64 Updated: libXfixes-5.0.1-2.1.el6.x86_64 Updated: libXcursor-1.1.14-2.1.el6.x86_64 Updated: pango-1.28.1-10.el6.x86_64 Updated: libXxf86vm-1.1.3-2.1.el6.x86_64 Updated: mesa-libGL-10.1.2-2.el6.x86_64 Updated: xorg-x11-server-Xorg-1.15.0-26.el6.centos.x86_64 Installed: gdk-pixbuf2-2.24.1-5.el6.x86_64 Updated: pulseaudio-libs-0.9.21-17.el6.x86_64 Updated: libXrandr-1.4.1-2.1.el6.x86_64 Updated: libXinerama-1.1.3-2.1.el6.x86_64 Updated: gtk2-2.24.23-6.el6.x86_64 Updated: libXt-1.1.4-6.1.el6.x86_64 Updated: kernel-firmware-2.6.32-504.16.2.el6.noarch Installed: kernel-2.6.32-504.16.2.el6.x86_64 Updated: xorg-x11-apps-7.7-6.el6.x86_64 Updated: 1:java-1.7.0-openjdk-1.7.0.79-2.5.5.3.el6_6.x86_64 Updated: xorg-x11-drv-void-1.4.0-23.el6.x86_64 Updated: xorg-x11-drv-vesa-2.3.2-15.el6.x86_64 Updated: xorg-x11-drv-evdev-2.8.2-4.el6.x86_64 Updated: tigervnc-server-module-1.1.0-16.el6.centos.x86_64 Updated: mesa-libGLU-10.1.2-2.el6.x86_64 Updated: tigervnc-server-1.1.0-16.el6.centos.x86_64 Updated: tigervnc-1.1.0-16.el6.centos.x86_64 Updated: libXv-1.0.9-2.1.el6.x86_64 Updated: hwloc-1.5-3.el6_5.x86_64 Updated: 12:dhclient-4.1.1-43.P1.el6.centos.1.x86_64 Updated: system-config-firewall-base-1.2.27-7.2.el6_6.noarch Updated: ntp-4.2.6p5-3.el6.centos.x86_64 Updated: parted-2.1-25.el6.x86_64 Updated: selinux-policy-targeted-3.7.19-260.el6_6.3.noarch Updated: openssh-server-5.3p1-104.el6_6.1.x86_64 Updated: openssh-clients-5.3p1-104.el6_6.1.x86_64 Updated: bfa-firmware-3.2.23.0-2.el6.noarch Updated: 1:cups-1.4.2-67.el6.x86_64 Updated: cyrus-sasl-2.1.23-15.el6_6.2.x86_64 Updated: rsyslog-5.8.10-10.el6_6.x86_64 Updated: subversion-1.6.11-12.el6_6.x86_64 Updated: nss-pam-ldapd-0.7.5-20.el6_6.3.x86_64 Updated: ipmitool-1.8.11-21.el6.x86_64 Updated: gcc-gfortran-4.4.7-11.el6.x86_64 Updated: gcc-c++-4.4.7-11.el6.x86_64 Updated: perl-Test-Simple-0.92-136.el6_6.1.x86_64 Updated: rpm-build-4.8.0-38.el6_6.x86_64 Updated: e2fsprogs-1.41.12-21.el6.x86_64 Updated: openldap-clients-2.4.39-8.el6.x86_64 Updated: gnupg2-2.0.14-8.el6.x86_64 Updated: sudo-1.8.6p3-15.el6.x86_64 Updated: cmake-2.8.12.2-4.el6.x86_64 Updated: wget-1.12-5.el6_6.1.x86_64 Updated: mailx-12.4-8.el6_6.x86_64 Updated: libgudev1-147-2.57.el6.x86_64 Updated: pciutils-3.1.10-4.el6.x86_64 Updated: python-magic-5.04-21.el6.x86_64 Updated: efibootmgr-0.5.4-12.el6.x86_64 Updated: ghostscript-fonts-5.50-23.2.el6.noarch Updated: at-3.1.10-44.el6_6.2.x86_64 Updated: authconfig-6.1.12-19.el6.x86_64 Updated: audit-2.3.7-5.el6.x86_64 Updated: tcsh-6.17-25.el6_6.x86_64 Updated: zsh-4.3.10-9.el6.x86_64 Updated: 1:grub-0.97-93.el6.x86_64 Updated: flac-1.2.1-7.el6_6.x86_64 Updated: etckeeper-1.18.1-2.el6.noarch Updated: 4:perl-Time-HiRes-1.9721-136.el6_6.1.x86_64 Updated: perl-CGI-3.51-136.el6_6.1.x86_64 Installed: kernel-devel-2.6.32-504.16.2.el6.x86_64 Updated: strace-4.5.19-1.19.el6.x86_64 Updated: libvisual-0.4.0-10.el6.x86_64 Updated: ncdu-1.11-1.el6.x86_64 Updated: rsync-3.0.6-12.el6.x86_64 Updated: xorg-x11-drv-ati-firmware-7.3.99-2.el6.noarch Updated: ql2400-firmware-7.03.00-1.el6_5.noarch Updated: xml-common-0.6.3-33.el6.noarch Updated: ql2500-firmware-7.03.00-1.el6_5.noarch Updated: nss-softokn-freebl-3.14.3-22.el6_6.i686 Updated: glibc-2.12-1.149.el6_6.9.i686 Updated: libgcc-4.4.7-11.el6.i686 Updated: libstdc++-4.4.7-11.el6.i686
- MPSS 3.5.1 installation
tar xvf mpss-3.5.1-linux.tar cd mpss-3.5.1 ./uninstall.sh rpm -ivh *.rpm perf/*.rpm psm/*.rpm rpm -ivh src/mpss-modules-3.5.1-1.src.rpm cd ~/rpmbuild && rpmbuild -ba SPECS/mpss-modules.spec rpm -ivh RPMS/x86_64/mpss-modules*.rpm reboot
- Vtune kernel module upgrade
cp -a /opt/intel/vtune_amplifier_xe/sepdk /tmp cd /tmp/sepdk/src build-driver --install-dir=/tmp/sepdk/prebuilt copy /tmp/sepdk/prebuilt/* to /opt/intel/vtune_amplifier_xe/sepdk/prebuilt/ #(/!\ nfs is RO) boot-script -g users -d /opt/intel/vtune_amplifier_xe/sepdk/prebuilt/ echo kernel.nmi_watchdog = 0 >> /etc/sysctl.conf reboot
- MIC state snapshot
[root@idphix ~]# micctrl --pm mic0: cpufreq_on;corec6_off;pc3_on;pc6_off
[root@idphix ~]# micinfo MicInfo Utility Log Created Mon Jun 8 22:20:34 2015 System Info HOST OS : Linux OS Version : 2.6.32-504.16.2.el6.x86_64 Driver Version : 3.5.1-1 MPSS Version : 3.5.1 Host Physical Memory : 66076 MB Device No: 0, Device Name: mic0 Version Flash Version : 2.1.02.0390 SMC Firmware Version : 1.16.5078 SMC Boot Loader Version : 1.8.4326 uOS Version : 2.6.38.8+mpss3.5.1 Device Serial Number : ADKC32100375 Board Vendor ID : 0x8086 Device ID : 0x2250 Subsystem ID : 0x2500 Coprocessor Stepping ID : 3 PCIe Width : x16 PCIe Speed : 5 GT/s PCIe Max payload size : 256 bytes PCIe Max read req size : 512 bytes Coprocessor Model : 0x01 Coprocessor Model Ext : 0x00 Coprocessor Type : 0x00 Coprocessor Family : 0x0b Coprocessor Family Ext : 0x00 Coprocessor Stepping : B1 Board SKU : B1PRQ-5110P/5120D ECC Mode : Enabled SMC HW Revision : Product 225W Passive CS Cores Total No of Active Cores : 60 Voltage : 1020000 uV Frequency : 1052631 kHz Thermal Fan Speed Control : N/A Fan RPM : N/A Fan PWM : N/A Die Temp : 42 C GDDR GDDR Vendor : Elpida GDDR Version : 0x1 GDDR Density : 2048 Mb GDDR Size : 7936 MB GDDR Technology : GDDR5 GDDR Speed : 5.000000 GT/s GDDR Frequency : 2500000 kHz GDDR Voltage : 1501000 uV
2015-06-09
- New system image
- idphix-default 20150609
This image:
- Upgrade to Intel Parallel Studio XE 2015 update 3
- Upgrade to Intel Vtune Amplifier XE 2015 update 4
(Parallel Studio XE 2015 update 3 comes with Vtune Amplifier XE 2015 update 3 -> Vtune is upgraded separatly)
$ icc -V Intel(R) C Intel(R) 64 Compiler XE for applications running on Intel(R) 64, Version 15.0.3.187 Build 20150407 Copyright (C) 1985-2015 Intel Corporation. All rights reserved. $ amplxe-cl -V Intel(R) VTune(TM) Amplifier XE 2015 Update 4 (build 410668) Command Line Tool Copyright (C) 2009-2015 Intel Corporation. All rights reserved.
References
- Main Intel resource page
- Documentation to install and configure the Xeon Phi
- Coding docs
- Intel webinars
- Directive-based programming extensions
- Intrisics on Xeon Phi
- Memory transfer
- Getting the highest bandwidth possible out of your Phi application:
- Documentation en francais
Acknolegment
The idphix machine was funded by the Mescal and Moais teams of LIG/Inria.
We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Tesla K40 GPU used for this research to Inria.