Idphix

From Digitalis

Revision as of 17:33, 2 March 2017 by Neyron (Talk | contribs)
(diff) ← Older revision | Current revision (diff) | Newer revision → (diff)
Jump to: navigation, search
| Introduction | Usage | Idfreeze | Idgraf | Idphix | Idbool | Idkat | Idcin | Idarm | Ppol | Grimage |

Contents

Update
  • Idphix now hosts a Nvidia K40 GPU along with the KNC Xeon Phi and a Infiniband host adapter.
  • The OS is Grid'5000's Debian stable

If needed one can get the Centos OS back using kadeploy.

Information below are mostly unrelevant with the Debian OS.

Overview

Idphix is a single socket Sandy Bridge machine, equipped with 1 MIC accelerator.

  • 1x Intel Xeon E5-2650 (8 cores)
  • 64 GB RAM
  • 1x coprocesseur Intel Xeon Phi 5110P

The setup is the following: We use an internal network on the host (idphix) for the Xeon Phi: 192.168.1.0/24.

This gives the following scheme:

                          =======================================
                        | -------------             -----------  |
                        ||    host     |           |  Xeon Phi | |
                        ||   idphix    |           |idphix-mic0| |
Grid5000 network <---> eth0           mic0 <----> mic0         | |
                    172.16.21.6 192.168.1.254  192.168.1.1     | |
                        ||             |           |           | |
                        | -------------             -----------  |
                          =======================================


How to experiment ?

For now (2013-09-13) the operating system is a Centos 6 (OS officially supported by Intel for the Xeon Phi).

Use the machine as follows:

Connect to digitalis/grid5000

workstation$ ssh digitalis.grenoble.g5k

In case of any issue (network issue, connection hanging...) please read more info here or ask on the Digitalis mailing list.

Reserve the machine/run a job

digitalis$ oarsub -p "machine='idphix'" -l walltime=2 -I

Setup your access to the Xeon Phi

(This might not be relevant if your are using some level of abstraction (COI, ...)

WARNING
there is currently a bug due to the last version of Intel MPSS, please report any issue to the mailing list.

Make sure you created your G5K ssh keys (keys without passephrase protection, dedicated to G5K and stored in your NFS home directory)

Run:

idphix$ sudo /usr/local/bin/mic-setup-my-user

If you get the error:

Error: Cannot get your SSH keys, please run 'chmod 755 ~/.ssh && chmod 644 ~/.ssh/{authorized_keys,id_*.pub}'

Please run:

idphix$ chmod 755 ~/.ssh && chmod 644 ~/.ssh/{authorized_keys,id_*.pub}

Output should be:

idphix$ sudo /usr/local/bin/mic-setup-my-user 
*** Configure user pneyron:users (10106:8000) on mic0
*** Copy user files to mic0
id_rsa.pub                                    100%  420     0.4KB/s   00:00    
authorized_keys                               100%  916     0.9KB/s   00:00    
.profile                                      100%   62     0.1KB/s   00:00    
*** Force creation of pneyron:users on mic0 and fix file ownership and permission
Done

You shoud now be able to ssh to the coprocessor:

idphix$ ssh mic0
idphix-mic0$ uname -a
Linux idphix-mic0.grenoble.grid5000.fr 2.6.38.8-g5f2543d #2 SMP Tue Apr 30 14:05:06 PDT 2013 k1om k1om k1om GNU/Linux

Intel compiler and other tools

Intel tools (icc & co) are installed in /opt/intel

It is configured to use Inria's license server (FlexLM).

This licence server provides 2 tokens which are shared among all Inria's users. Only 2 compilations can be performed at a time.

If icc complains about the license (error message talking about FlexLM), please report the issue to digitalis@lists.grid5000.fr


Rebooting the MIC

If needed, you can reboot the MIC from within a exclusive job (doing so in shared job session would risk annoying other users).

oarsub -t exclusive -p "machine='idphix'" ....

Then, you will be able to run

sudo /usr/local/bin/mic-reboot

(no password should be requested).

Rebooting the MIC takes some time. You can look at the output of dmesg or miccrtl -s meanwhile.

Data on the mic: no persistence

Please mind the fact that data pushed to the mic are not persistent. Your home directory on idphix-mic0 could be emptied for various reasons.

Privileged commands

Currently, the following commands can be run via sudo in exclusive jobs:

  • sudo /usr/bin/whoami (provided for testing the mechanism, should return "root")
  • sudo /sbin/reboot
  • sudo /usr/bin/schedtool
  • sudo /usr/local/bin/mic-reboot
  • sudo /usr/local/bin/mic-pm
NB
  • sudo /usr/local/bin/mic-setup-my-user does not require an exclusive job.
  • PM setting are not reset automatically. Please everyone check the PM settings (using micctl --pm or the mic-pm script) before running experiments

Redeploy and get full (root) access to the machine/MIC

One can be root on the machine using kadeploy.

To redeploy your own instance of idphix default system (Centos), add '-t redeploy' to your oarsub command and run the following kadeploy command:

kadeploy -e idphix-default -m idphix -k

Then you should be able to do:

digitalis$ ssh root@idphix
idphix$ ssh mic0
idphix-mic$ whoami
root

At the end of the job, the machine will return to its default system environment.

Other experimental platforms with MICs

Grid'5000

4 nodes with 1 Xeon Phi each are available on Grid'5000 Nancy's site, in the Graphite cluster. See:

Irisa

Guillermo Andrade-barroso installed a Xeon Phi host server at INRIA Rennes. This server hosts 1 Intel® Xeon Phi™ 5110P (8 G, 1.053 GHz, 60 cores) under Debian 7.0 64 bits.

Access to this server is available for all INRIA members on demand.

The machine name is phileas.irisa.fr

Bordeaux - Plafrim

https://plafrim.bordeaux.inria.fr/doku.php?id=plateforme:configurations:mistral

System changelogs

2013-09-26

System image
idphix-default 20130926
  • Centos 6
  • Linux 2.6.32-358.el6.x86_64
  • MPSS packages
intel-mic-flash-2.1.386-2.el6.x86_64
intel-mic-kmod-2.1.6720-13.2.6.32.358.el6.x86_64
intel-mic-sysmgmt-2.1.6720-13.el6.x86_64
intel-mic-gpl-2.1.6720-13.el6.x86_64
intel-mic-mpm-2.1.6720-13.el6.x86_64
intel-mic-2.1.6720-13.el6.x86_64
intel-mic-micmgmt-2.1.6720-13.2.6.32.358.el6.x86_64
intel-mic-perf-data-2.1.6720-13.el6.x86_64
intel-mic-gdb-2.1.6720-13.el6.x86_64
intel-mic-cdt-2.1.6720-13.el6.x86_64
intel-mic-perf-2.1.6720-13.el6.x86_64
  • Intel Cluster Studio XE 2013
  • MIC info
[root@idphix ~]# /opt/intel/mic/bin/micinfo 
MicInfo Utility Log

Created Mon Dec  9 20:13:37 2013


	System Info
		HOST OS			: Linux
		OS Version		: 2.6.32-358.el6.x86_64
		Driver Version		: 6720-13
		MPSS Version		: 2.1.6720-13
		Host Physical Memory	: 66077 MB

Device No: 0, Device Name: mic0

	Version
		Flash Version 		 : 2.1.02.0386
		SMC Firmware Version	 : 1.14.4616
		SMC Boot Loader Version	 : 1.8.4326
		uOS Version 		 : 2.6.38.8-g5f2543d
		Device Serial Number 	 : ADKC32100375

	Board
		Vendor ID 		 : 0x8086
		Device ID 		 : 0x2250
		Subsystem ID 		 : 0x2500
		Coprocessor Stepping ID	 : 3
		PCIe Width 		 : x16
		PCIe Speed 		 : 5 GT/s
		PCIe Max payload size	 : 256 bytes
		PCIe Max read req size	 : 512 bytes
		Coprocessor Model	 : 0x01
		Coprocessor Model Ext	 : 0x00
		Coprocessor Type	 : 0x00
		Coprocessor Family	 : 0x0b
		Coprocessor Family Ext	 : 0x00
		Coprocessor Stepping 	 : B1
		Board SKU 		 : INVALID SKU
		ECC Mode 		 : Enabled
		SMC HW Revision 	 : Product 225W Passive CS

	Cores
		Total No of Active Cores : 60
		Voltage 		 : 985000 uV
		Frequency		 : 1052631 kHz

	Thermal
		Fan Speed Control 	 : N/A
		Fan RPM 		 : N/A
		Fan PWM 		 : N/A
		Die Temp		 : 35 C

	GDDR
		GDDR Vendor		 : Elpida
		GDDR Version		 : 0x1
		GDDR Density		 : 2048 Mb
		GDDR Size		 : 7936 MB
		GDDR Technology		 : GDDR5 
		GDDR Speed		 : 5.000000 GT/s 
		GDDR Frequency		 : 2500000 kHz
		GDDR Voltage		 : 1501000 uV

2013-12-09

New system image
idphix-default 20131209
  • Updated Centos packages, except linux kernel
  • Moved Intel Cluster Studio XE 2013 to NFS /applis/digitalis/intel
  • No change to the MIC software (MPSS)

2013-12-13

New system image
idphix-default 20131213
  • Activated Hyperthreading
[root@idphix ~]# lstopo --no-caches
Machine (63GB) + Socket L#0
  Core L#0
    PU L#0 (P#0)
    PU L#1 (P#8)
  Core L#1
    PU L#2 (P#1)
    PU L#3 (P#9)
  Core L#2
    PU L#4 (P#2)
    PU L#5 (P#10)
  Core L#3
    PU L#6 (P#3)
    PU L#7 (P#11)
  Core L#4
    PU L#8 (P#4)
    PU L#9 (P#12)
  Core L#5
    PU L#10 (P#5)
    PU L#11 (P#13)
  Core L#6
    PU L#12 (P#6)
    PU L#13 (P#14)
  Core L#7
    PU L#14 (P#7)
    PU L#15 (P#15)
  • Updated MPSS 2.1 -> 3.1.1
[root@idphix ~]# rpm -qa | grep -e intel -e mpss
mpss-miccheck-3.1.1-r1.glibc2.12.2.x86_64
mpss-sysmgmt-python-3.1.1-1.glibc2.12.2.x86_64
mpss-sciftutorials-3.1.1-1.glibc2.12.2.x86_64
mpss-eclipse-cdt-mpm-3.1.1-1.glibc2.12.2.x86_64
mpss-myo-doc-3.1.1-1.glibc2.12.2.x86_64
mpss-coi-doc-3.1.1-1.glibc2.12.2.x86_64
mpss-flash-3.1.1-1.glibc2.12.2.x86_64
mpss-myo-3.1.1-1.glibc2.12.2.x86_64
mpss-metadata-3.1.1-1.glibc2.12.2.x86_64
mpss-myo-dev-3.1.1-1.glibc2.12.2.x86_64
intel-composerxe-compat-k1om-3.1.1-1.x86_64
mpss-micmgmt-doc-3.1.1-1.glibc2.12.2.x86_64
mpss-sysmgmt-micsmc-gui-3.1.1-1.glibc2.12.2.x86_64
mpss-miccheck-bin-3.1.1-r1.glibc2.12.2.x86_64
mpss-sysmgmt-micras-3.1.1-1.glibc2.12.2.x86_64
mpss-modules-2.6.32-358.el6.x86_64-3.1.1-1.el6.x86_64
mpss-modules-dev-2.6.32-358.el6.x86_64-3.1.1-1.el6.x86_64
mpss-sciftutorials-doc-3.1.1-1.glibc2.12.2.x86_64
mpss-license-3.1.1-1.glibc2.12.2.x86_64
mpss-rasmm-kernel-3.1.1-1.glibc2.12.2.x86_64
mpss-micmgmt-python-3.1.1-1.glibc2.12.2.x86_64
mpss-coi-3.1.1-1.glibc2.12.2.x86_64
mpss-modules-headers-3.1.1-1.glibc2.12.2.x86_64
mpss-sdk-k1om-3.1.1-1.x86_64
mpss-metadata-dev-3.1.1-1.glibc2.12.2.x86_64
mpss-daemon-dev-3.1.1-1.glibc2.12.2.x86_64
mpss-coi-dev-3.1.1-1.glibc2.12.2.x86_64
mpss-micmgmt-3.1.1-1.glibc2.12.2.x86_64
mpss-mpm-doc-3.1.1-1.glibc2.12.2.x86_64
mpss-mpm-3.1.1-1.glibc2.12.2.x86_64
mpss-boot-files-3.1.1-1.glibc2.12.2.x86_64
mpss-daemon-3.1.1-1.glibc2.12.2.x86_64
mpss-modules-headers-dev-3.1.1-1.glibc2.12.2.x86_64
  • Updated the MIC SMC and Flash (part of the MPSS upgrade)
[root@idphix ~]# micinfo 
MicInfo Utility Log

Created Fri Dec 13 19:00:30 2013


	System Info
		HOST OS			: Linux
		OS Version		: 2.6.32-358.el6.x86_64
		Driver Version		: 3.1.1-1
		MPSS Version		: 3.1.1
		Host Physical Memory	: 66076 MB

Device No: 0, Device Name: mic0

	Version
		Flash Version 		 : 2.1.02.0390
		SMC Firmware Version	 : 1.16.5078
		SMC Boot Loader Version	 : 1.8.4326
		uOS Version 		 : 2.6.38.8+mpss3.1.1
		Device Serial Number 	 : ADKC32100375

	Board
		Vendor ID 		 : 0x8086
		Device ID 		 : 0x2250
		Subsystem ID 		 : 0x2500
		Coprocessor Stepping ID	 : 3
		PCIe Width 		 : x16
		PCIe Speed 		 : 5 GT/s
		PCIe Max payload size	 : 256 bytes
		PCIe Max read req size	 : 512 bytes
		Coprocessor Model	 : 0x01
		Coprocessor Model Ext	 : 0x00
		Coprocessor Type	 : 0x00
		Coprocessor Family	 : 0x0b
		Coprocessor Family Ext	 : 0x00
		Coprocessor Stepping 	 : B1
		Board SKU 		 : B1PRQ-5110P/5120D
		ECC Mode 		 : Enabled
		SMC HW Revision 	 : Product 225W Passive CS

	Cores
		Total No of Active Cores : 60
		Voltage 		 : 1021000 uV
		Frequency		 : 1052631 kHz

	Thermal
		Fan Speed Control 	 : N/A
		Fan RPM 		 : N/A
		Fan PWM 		 : N/A
		Die Temp		 : 36 C

	GDDR
		GDDR Vendor		 : Elpida
		GDDR Version		 : 0x1
		GDDR Density		 : 2048 Mb
		GDDR Size		 : 7936 MB
		GDDR Technology		 : GDDR5 
		GDDR Speed		 : 5.000000 GT/s 
		GDDR Frequency		 : 2500000 kHz
		GDDR Voltage		 : 1501000 uV

2014-02-17

Installed Intel Cluster Studio XE 2013.1 in /grid5000/software/. It comes with icc 14. (/grid5000 is a NFS mount, commonly usable by all machines)

Vtune is part of Intel Cluster Studio XE 2013.1, but needs more work to be fully functional.

2014-02-19

New system image
idphix-default 20140219
  • Fixed mic-setup-my-user which was broken since the upgrade to MPSS 3.1
  • Added mount point for /grid5000
  • Changed /opt/intel link to /grid5000/software/intel, which as of 2014-02-17 links to Intel Cluster Studio XE 2013.1
  • Added packages:
    • emacs-common-23.1-25.el6.x86_64
    • emacs-nox-23.1-25.el6.x86_64
    • autoconf-2.63-5.1.el6.noarch
    • automake-1.11.1-4.el6.noarch
    • automake16-1.6.3-18.el6.1.noarch
    • automake15-1.5-27.el6.1.noarch
    • automake14-1.4p6-19.2.el6.noarch
    • autoconf213-2.13-20.1.el6.noarch
    • autoconf-archive-2012.09.08-1.el6.noarch
    • tcl-8.5.7-6.el6.x86_64
    • expect-5.44.1.15-5.el6_4.x86_64
    • dejagnu-1.4.4-17.el6.noarch
    • tig-0.17-1.el6.x86_64

2014-02-26

New system image
idphix-default 20140226

2014-04-08

New system image
idphix-default 20140408

This image:

  • upgrades the system to CentOS 6.5 (required to build Vtune kernel modules)
  • installs VNC server + twm + xterm + xeyes
  • brings support for Intel Vtune (disable nmi_watchdog + setup Vtune kernel modules)
  • installs MPSS version 3.2
  • adds script mic-pm to allow a user in an exclusive job to change the power-management settings of the mic
yum logs for CentOS 6.5 upgrade
Apr 05 11:08:07 Installed: tigervnc-1.1.0-8.el6_5.x86_64
Apr 05 11:09:10 Installed: hal-libs-0.5.14-11.el6.x86_64
Apr 05 11:09:11 Installed: xkeyboard-config-2.6-6.el6.noarch
Apr 05 11:09:13 Installed: mtdev-1.1.2-5.el6.x86_64
Apr 05 11:09:14 Installed: cryptsetup-luks-libs-1.2.0-7.el6.x86_64
Apr 05 11:09:14 Installed: cryptsetup-luks-1.2.0-7.el6.x86_64
Apr 05 11:09:14 Installed: ConsoleKit-libs-0.4.1-3.el6.x86_64
Apr 05 11:09:15 Installed: libxkbfile-1.0.6-1.1.el6.x86_64
Apr 05 11:09:15 Installed: xorg-x11-xkb-utils-7.7-4.el6.x86_64
Apr 05 11:09:15 Installed: xorg-x11-server-common-1.13.0-23.1.el6.centos.x86_64
Apr 05 11:09:16 Installed: eggdbus-0.6-3.el6.x86_64
Apr 05 11:09:16 Installed: ConsoleKit-0.4.1-3.el6.x86_64
Apr 05 11:09:16 Installed: polkit-0.96-5.el6_4.x86_64
Apr 05 11:09:17 Installed: hdparm-9.43-4.el6.x86_64
Apr 05 11:09:17 Installed: hal-info-20090716-3.1.el6.noarch
Apr 05 11:09:17 Installed: pm-utils-1.2.5-10.el6.x86_64
Apr 05 11:09:18 Installed: hal-0.5.14-11.el6.x86_64
Apr 05 11:09:19 Installed: system-setup-keyboard-0.7-4.el6.x86_64
Apr 05 11:09:19 Installed: libXdmcp-1.1.1-3.el6.x86_64
Apr 05 11:09:19 Installed: xorg-x11-drv-evdev-2.7.3-5.el6.x86_64
Apr 05 11:09:20 Installed: xorg-x11-drv-void-1.4.0-3.el6.x86_64
Apr 05 11:09:20 Installed: xorg-x11-drv-vesa-2.3.2-4.el6.x86_64
Apr 05 11:09:20 Installed: xorg-x11-server-Xorg-1.13.0-23.1.el6.centos.x86_64
Apr 05 11:09:21 Installed: tigervnc-server-module-1.1.0-8.el6_5.x86_64
Apr 05 11:25:39 Updated: kernel-firmware-2.6.32-431.11.2.el6.noarch
Apr 05 11:25:46 Installed: kernel-2.6.32-431.11.2.el6.x86_64
Apr 05 11:25:52 Installed: kernel-devel-2.6.32-431.11.2.el6.x86_64
Apr 05 11:25:53 Updated: kernel-headers-2.6.32-431.11.2.el6.x86_64
Apr 05 12:00:36 Installed: xorg-x11-fonts-misc-7.2-9.1.el6.noarch
Apr 05 12:00:36 Installed: tigervnc-server-1.1.0-8.el6_5.x86_64
Apr 05 12:03:20 Installed: 1:xorg-x11-twm-1.0.3-5.1.el6.x86_64
Apr 05 12:05:03 Installed: libXpm-3.5.10-2.el6.x86_64
Apr 05 12:05:04 Installed: libXaw-1.0.11-2.el6.x86_64
Apr 05 12:05:04 Installed: xterm-253-1.el6.x86_64
Apr 05 12:15:45 Installed: xorg-x11-apps-7.6-6.el6.x86_64
Apr 05 13:28:12 Updated: nspr-4.10.2-1.el6_5.x86_64
Apr 05 13:28:13 Updated: nss-util-3.15.3-1.el6_5.x86_64
Apr 05 13:28:14 Updated: glib2-2.26.1-7.el6_5.x86_64
Apr 05 13:28:14 Updated: libjpeg-turbo-1.2.1-3.el6_5.x86_64
Apr 05 13:28:15 Updated: grep-2.6.3-4.el6_5.1.x86_64
Apr 05 13:28:15 Updated: coreutils-libs-8.4-31.el6_5.1.x86_64
Apr 05 13:28:17 Updated: coreutils-8.4-31.el6_5.1.x86_64
Apr 05 13:28:18 Updated: krb5-libs-1.10.3-15.el6_5.1.x86_64
Apr 05 13:28:18 Updated: p11-kit-0.18.5-2.el6_5.2.x86_64
Apr 05 13:28:19 Updated: p11-kit-trust-0.18.5-2.el6_5.2.x86_64
Apr 05 13:28:20 Updated: ca-certificates-2013.1.95-65.1.el6_5.noarch
Apr 05 13:28:21 Updated: openssl-1.0.1e-16.el6_5.4.x86_64
Apr 05 13:28:22 Updated: 1:qt-4.6.2-28.el6_5.x86_64
Apr 05 13:28:23 Updated: 1:qt-sqlite-4.6.2-28.el6_5.x86_64
Apr 05 13:28:23 Updated: python-2.6.6-52.el6.x86_64
Apr 05 13:28:25 Updated: python-libs-2.6.6-52.el6.x86_64
Apr 05 13:28:26 Updated: yum-plugin-fastestmirror-1.1.30-17.el6_5.noarch
Apr 05 13:28:26 Updated: yum-3.2.29-43.el6.centos.noarch
Apr 05 13:28:27 Updated: mysql-libs-5.1.73-3.el6_5.x86_64
Apr 05 13:28:28 Updated: selinux-policy-3.7.19-231.el6_5.1.noarch
Apr 05 13:28:29 Updated: nss-sysinit-3.15.3-6.el6_5.x86_64
Apr 05 13:28:29 Updated: nss-3.15.3-6.el6_5.x86_64
Apr 05 13:28:30 Updated: nss-tools-3.15.3-6.el6_5.x86_64
Apr 05 13:28:30 Updated: openldap-2.4.23-34.el6_5.1.x86_64
Apr 05 13:28:30 Updated: libtiff-3.9.4-10.el6_5.x86_64
Apr 05 13:28:31 Updated: 2:ethtool-3.5-1.2.el6_5.x86_64
Apr 05 13:28:32 Updated: tzdata-java-2014b-1.el6.noarch
Apr 05 13:28:32 Updated: psmisc-22.6-19.el6_5.x86_64
Apr 05 13:28:33 Updated: upstart-0.6.5-13.el6_5.3.x86_64
Apr 05 13:28:34 Updated: initscripts-9.03.40-2.el6.centos.1.x86_64
Apr 05 13:28:34 Updated: mesa-dri-filesystem-9.2-0.5.el6_5.2.x86_64
Apr 05 13:28:36 Updated: mesa-dri-drivers-9.2-0.5.el6_5.2.x86_64
Apr 05 13:28:36 Updated: mesa-libGL-9.2-0.5.el6_5.2.x86_64
Apr 05 13:28:36 Updated: mesa-libGLU-9.2-0.5.el6_5.2.x86_64
Apr 05 13:28:37 Updated: 1:phonon-backend-gstreamer-4.6.2-28.el6_5.x86_64
Apr 05 13:28:39 Updated: 1:qt-x11-4.6.2-28.el6_5.x86_64
Apr 05 13:28:40 Updated: subversion-1.6.11-10.el6_5.x86_64
Apr 05 13:28:42 Updated: 2:postfix-2.6.6-6.el6_5.x86_64
Apr 05 13:28:49 Updated: 1:java-1.7.0-openjdk-1.7.0.51-2.4.4.1.el6_5.x86_64
Apr 05 13:28:49 Updated: openldap-clients-2.4.23-34.el6_5.1.x86_64
Apr 05 13:29:17 Updated: selinux-policy-targeted-3.7.19-231.el6_5.1.noarch
Apr 05 13:29:18 Updated: wget-1.12-1.11.el6_5.x86_64
Apr 05 13:29:19 Updated: tzdata-2014b-1.el6.noarch
Apr 05 13:29:20 Updated: pixman-0.26.2-5.1.el6_5.x86_64
Apr 05 13:29:20 Updated: openjpeg-libs-1.3-10.el6_5.x86_64
Apr 05 13:29:21 Updated: libtirpc-0.2.1-6.el6_5.1.x86_64
Apr 05 13:29:21 Updated: libXfont-1.4.5-3.el6_5.x86_64
Apr 05 13:29:22 Updated: gnutls-2.8.5-13.el6_5.x86_64
Apr 05 13:29:22 Updated: 1:dmidecode-2.12-5.el6_5.x86_64
Apr 05 13:29:23 Updated: etckeeper-1.11-1.el6.noarch
Vtune installation
cp -a /opt/intel/vtune_amplifier_xe/sepdk /tmp
cd /tmp/sepdk/src
build-driver --install-dir=/tmp/sepdk/prebuilt
copy /tmp/sepdk/prebuilt/* to /opt/intel/vtune_amplifier_xe/sepdk/prebuilt/ #(/!\ nfs is RO)
boot-script -g users -d /opt/intel/vtune_amplifier_xe/sepdk/prebuilt/
echo kernel.nmi_watchdog = 0 >> /etc/sysctl.conf
reboot
MPSS 3.2 installation
tar xvf mpss-3.2-rhel-6.5.tar
cd mpss-3.2
./uninstall.sh
rpm -ev opencl-1.2-intel-mic
rpm -ivh *.rpm perf/*.rpm psm/*.rpm
rpm -ivh src/mpss-modules-3.2-1.el6.src.rpm
yum install rpm-build rpmlint rpmdevtools
cd ~/rpmbuild && rpmbuild -ba SPECS/mpss-modules.spec
rpm -ivh RPMS/mpss-modules*.rpm
rpm -ivh opencl-1.2-intel-mic-3.2.1.16712-1.x86_64.rpm #(from intel_sdk_for_ocl_applications_xe_2013_r3_sdk_3.2.1.16712_x64)
reboot
MIC state snapshot
$ micctrl --pm
mic0: cpufreq_on;corec6_off;pc3_on;pc6_off

$ micinfo 
MicInfo Utility Log
Copyright 2011-2013 Intel Corporation All Rights Reserved.

Created Tue Apr  8 16:27:59 2014


	System Info
		HOST OS			: Linux
		OS Version		: 2.6.32-431.11.2.el6.x86_64
		Driver Version		: 3.2-1
		MPSS Version		: 3.2
		Host Physical Memory	: 66076 MB

Device No: 0, Device Name: mic0

	Version
		Flash Version 		 : 2.1.02.0390
		SMC Firmware Version	 : 1.16.5078
		SMC Boot Loader Version	 : 1.8.4326
		uOS Version 		 : 2.6.38.8+mpss3.2
		Device Serial Number 	 : ADKC32100375

	Board
		Vendor ID 		 : 0x8086
		Device ID 		 : 0x2250
		Subsystem ID 		 : 0x2500
		Coprocessor Stepping ID	 : 3
		PCIe Width 		 : x16
		PCIe Speed 		 : 5 GT/s
		PCIe Max payload size	 : 256 bytes
		PCIe Max read req size	 : 512 bytes
		Coprocessor Model	 : 0x01
		Coprocessor Model Ext	 : 0x00
		Coprocessor Type	 : 0x00
		Coprocessor Family	 : 0x0b
		Coprocessor Family Ext	 : 0x00
		Coprocessor Stepping 	 : B1
		Board SKU 		 : B1PRQ-5110P/5120D
		ECC Mode 		 : Enabled
		SMC HW Revision 	 : Product 225W Passive CS

	Cores
		Total No of Active Cores : 60
		Voltage 		 : 1021000 uV
		Frequency		 : 1052631 kHz

	Thermal
		Fan Speed Control 	 : N/A
		Fan RPM 		 : N/A
		Fan PWM 		 : N/A
		Die Temp		 : 36 C

	GDDR
		GDDR Vendor		 : Elpida
		GDDR Version		 : 0x1
		GDDR Density		 : 2048 Mb
		GDDR Size		 : 7936 MB
		GDDR Technology		 : GDDR5 
		GDDR Speed		 : 5.000000 GT/s 
		GDDR Frequency		 : 2500000 kHz
		GDDR Voltage		 : 1501000 uV

2015-06-08

New system image
idphix-default 20150608

This image:

  • upgrades the system to Centos 6.6
  • use Linux kernel 2.6.32-431.11.2.el6.x86_64
  • installs MPSS version 3.5.1
yum logs for CentOS 6.6 upgrade
Updated: libgcc-4.4.7-11.el6.x86_64
Updated: xkeyboard-config-2.11-1.el6.noarch
Updated: kernel-headers-2.6.32-504.16.2.el6.x86_64
Updated: centos-release-6-6.el6.centos.12.2.x86_64
Updated: tzdata-2015d-1.el6.noarch
Updated: nss-softokn-freebl-3.14.3-22.el6_6.x86_64
Updated: glibc-common-2.12-1.149.el6_6.9.x86_64
Updated: glibc-2.12-1.149.el6_6.9.x86_64
Updated: bash-4.1.2-29.el6.x86_64
Updated: libselinux-2.0.94-5.8.el6.x86_64
Updated: 1:perl-Pod-Escapes-1.04-136.el6_6.1.x86_64
Updated: 3:perl-version-0.77-136.el6_6.1.x86_64
Updated: 1:perl-Module-Pluggable-3.90-136.el6_6.1.x86_64
Updated: 4:perl-libs-5.10.1-136.el6_6.1.x86_64
Updated: 1:perl-Pod-Simple-3.13-136.el6_6.1.x86_64
Updated: 4:perl-5.10.1-136.el6_6.1.x86_64
Updated: libstdc++-4.4.7-11.el6.x86_64
Updated: libcom_err-1.41.12-21.el6.x86_64
Updated: audit-libs-2.3.7-5.el6.x86_64
Updated: grep-2.6.3-6.el6.x86_64
Updated: coreutils-libs-8.4-37.el6.x86_64
Updated: coreutils-8.4-37.el6.x86_64
Updated: pam-1.1.1-20.el6.x86_64
Updated: nspr-4.10.8-1.el6_6.x86_64
Updated: 1:dbus-libs-1.2.24-8.el6_6.x86_64
Updated: 2:shadow-utils-4.1.4.2-19.el6_6.1.x86_64
Updated: nss-util-3.18.0-1.el6_6.x86_64
Updated: xz-libs-4.999.9-0.5.beta.20091007git.el6.x86_64
Updated: elfutils-libelf-0.158-3.2.el6.x86_64
Updated: file-libs-5.04-21.el6.x86_64
Updated: 1:dbus-1.2.24-8.el6_6.x86_64
Updated: freetype-2.3.11-15.el6_6.1.x86_64
Updated: fontconfig-2.8.0-5.el6.x86_64
Updated: libuuid-2.17.2-12.18.el6.x86_64
Updated: libblkid-2.17.2-12.18.el6.x86_64
Updated: pciutils-libs-3.1.10-4.el6.x86_64
Updated: libudev-147-2.57.el6.x86_64
Updated: cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64
Updated: file-5.04-21.el6.x86_64
Updated: cpio-2.10-12.el6_5.x86_64
Updated: binutils-2.20.51.0.2-5.42.el6.x86_64
Updated: module-init-tools-3.9-24.el6.x86_64
Updated: hwdata-0.233-11.1.el6.noarch
Updated: pixman-0.32.4-4.el6.x86_64
Updated: keyutils-libs-1.4-5.el6.x86_64
Updated: krb5-libs-1.10.3-37.el6_6.x86_64
Updated: libxml2-2.7.6-17.el6_6.1.x86_64
Updated: glib2-2.28.8-4.el6.x86_64
Updated: shared-mime-info-0.70-6.el6.x86_64
Updated: libpciaccess-0.13.3-0.1.el6.x86_64
Updated: libXfont-1.4.5-4.el6_6.x86_64
Updated: hal-libs-0.5.14-14.el6.x86_64
Updated: xz-4.999.9-0.5.beta.20091007git.el6.x86_64
Updated: nss-softokn-3.14.3-22.el6_6.x86_64
Updated: nss-3.18.0-5.3.el6_6.x86_64
Updated: nss-sysinit-3.18.0-5.3.el6_6.x86_64
Updated: avahi-libs-0.6.25-15.el6.x86_64
Updated: libproxy-python-0.3.0-10.el6.x86_64
Updated: libproxy-0.3.0-10.el6.x86_64
Updated: libproxy-bin-0.3.0-10.el6.x86_64
Updated: ca-certificates-2015.2.4-65.0.1.el6_6.noarch
Updated: openssl-1.0.1e-30.el6.9.x86_64
Updated: libssh2-1.4.2-1.el6_6.1.x86_64
Updated: plymouth-scripts-0.8.3-27.el6.centos.1.x86_64
Updated: libselinux-utils-2.0.94-5.8.el6.x86_64
Updated: libgomp-4.4.7-11.el6.x86_64
Updated: libxcb-1.9.1-2.el6.x86_64
Updated: gettext-0.17-18.el6.x86_64
Installed: OpenIPMI-libs-2.0.16-14.el6.x86_64
Updated: nss-tools-3.18.0-5.3.el6_6.x86_64
Updated: libcurl-7.19.7-40.el6_6.4.x86_64
Updated: curl-7.19.7-40.el6_6.4.x86_64
Updated: rpm-libs-4.8.0-38.el6_6.x86_64
Updated: rpm-4.8.0-38.el6_6.x86_64
Updated: openldap-2.4.39-8.el6.x86_64
Updated: rpm-python-4.8.0-38.el6_6.x86_64
Updated: yum-plugin-fastestmirror-1.1.30-30.el6.noarch
Updated: yum-3.2.29-60.el6.centos.noarch
Updated: xz-lzma-compat-4.999.9-0.5.beta.20091007git.el6.x86_64
Updated: polkit-0.96-7.el6.x86_64
Installed: libIDL-0.8.13-2.1.el6.x86_64
Installed: ORBit2-2.14.17-5.el6.x86_64
Updated: grubby-7.0.15-7.el6.x86_64
Updated: keyutils-1.4-5.el6.x86_64
Updated: xorg-x11-server-common-1.15.0-26.el6.centos.x86_64
Updated: elfutils-libs-0.158-3.2.el6.x86_64
Updated: elfutils-0.158-3.2.el6.x86_64
Updated: nscd-2.12-1.149.el6_6.9.x86_64
Installed: sgml-common-0.6.3-33.el6.noarch
Installed: GConf2-2.28.0-6.el6.x86_64
Updated: libss-1.41.12-21.el6.x86_64
Updated: e2fsprogs-libs-1.41.12-21.el6.x86_64
Updated: libstdc++-devel-4.4.7-11.el6.x86_64
Updated: mesa-private-llvm-3.4-3.el6.x86_64
Updated: procps-3.2.8-30.el6.x86_64
Updated: less-436-13.el6.x86_64
Updated: gzip-1.3.12-22.el6.x86_64
Updated: unzip-6.0-2.el6_6.x86_64
Updated: gdb-7.2-75.el6.x86_64
Updated: cpp-4.4.7-11.el6.x86_64
Updated: glibc-headers-2.12-1.149.el6_6.9.x86_64
Updated: glibc-devel-2.12-1.149.el6_6.9.x86_64
Updated: 1:perl-ExtUtils-ParseXS-2.2003.0-136.el6_6.1.x86_64
Updated: 4:perl-devel-5.10.1-136.el6_6.1.x86_64
Updated: perl-Test-Harness-3.17-136.el6_6.1.x86_64
Updated: perl-ExtUtils-MakeMaker-6.55-136.el6_6.1.x86_64
Updated: gcc-4.4.7-11.el6.x86_64
Updated: libtirpc-0.2.1-10.el6.x86_64
Updated: jasper-libs-1.900.1-16.el6_6.3.x86_64
Updated: libtasn1-2.3-6.el6_5.x86_64
Updated: gnutls-2.8.5-14.el6_5.x86_64
Updated: 1:cups-libs-1.4.2-67.el6.x86_64
Updated: libgfortran-4.4.7-11.el6.x86_64
Installed: lm_sensors-libs-3.1.1-17.el6.x86_64
Installed: 1:net-snmp-libs-5.5-50.el6_6.1.x86_64
Installed: OpenIPMI-2.0.16-14.el6.x86_64
Updated: numactl-2.0.9-2.el6.x86_64
Updated: plymouth-core-libs-0.8.3-27.el6.centos.1.x86_64
Updated: 2:ethtool-3.5-5.el6.x86_64
Updated: iptables-1.4.7-14.el6.x86_64
Updated: iproute-2.6.32-33.el6_6.x86_64
Updated: initscripts-9.03.46-1.el6.centos.1.x86_64
Updated: util-linux-ng-2.17.2-12.18.el6.x86_64
Updated: udev-147-2.57.el6.x86_64
Updated: policycoreutils-2.0.83-19.47.el6_6.1.x86_64
Updated: libdrm-2.4.52-4.el6.x86_64
Updated: openssh-5.3p1-104.el6_6.1.x86_64
Updated: plymouth-0.8.3-27.el6.centos.1.x86_64
Updated: dracut-004-356.el6_6.2.noarch
Updated: dracut-kernel-004-356.el6_6.2.noarch
Updated: selinux-policy-3.7.19-260.el6_6.3.noarch
Updated: device-mapper-libs-1.02.90-2.el6_6.3.x86_64
Updated: device-mapper-1.02.90-2.el6_6.3.x86_64
Updated: cryptsetup-luks-libs-1.2.0-11.el6.x86_64
Updated: cryptsetup-luks-1.2.0-11.el6.x86_64
Updated: pm-utils-1.2.5-11.el6.x86_64
Updated: hal-0.5.14-14.el6.x86_64
Updated: nfs-utils-lib-1.1.5-9.el6.x86_64
Updated: 1:nfs-utils-1.2.3-54.el6.x86_64
Updated: ntpdate-4.2.6p5-3.el6.centos.x86_64
Updated: iptables-ipv6-1.4.7-14.el6.x86_64
Updated: mesa-dri-filesystem-10.1.2-2.el6.x86_64
Updated: mesa-dri-drivers-10.1.2-2.el6.x86_64
Updated: 12:dhcp-common-4.1.1-43.P1.el6.centos.1.x86_64
Updated: tzdata-java-2015d-1.el6.noarch
Updated: libX11-common-1.6.0-2.2.el6.noarch
Updated: libX11-1.6.0-2.2.el6.x86_64
Updated: libXext-1.3.2-2.1.el6.x86_64
Updated: libXrender-0.9.8-2.1.el6.x86_64
Updated: libXi-1.7.2-2.2.el6.x86_64
Updated: libXtst-1.2.2-2.1.el6.x86_64
Updated: libXfixes-5.0.1-2.1.el6.x86_64
Updated: libXcursor-1.1.14-2.1.el6.x86_64
Updated: pango-1.28.1-10.el6.x86_64
Updated: libXxf86vm-1.1.3-2.1.el6.x86_64
Updated: mesa-libGL-10.1.2-2.el6.x86_64
Updated: xorg-x11-server-Xorg-1.15.0-26.el6.centos.x86_64
Installed: gdk-pixbuf2-2.24.1-5.el6.x86_64
Updated: pulseaudio-libs-0.9.21-17.el6.x86_64
Updated: libXrandr-1.4.1-2.1.el6.x86_64
Updated: libXinerama-1.1.3-2.1.el6.x86_64
Updated: gtk2-2.24.23-6.el6.x86_64
Updated: libXt-1.1.4-6.1.el6.x86_64
Updated: kernel-firmware-2.6.32-504.16.2.el6.noarch
Installed: kernel-2.6.32-504.16.2.el6.x86_64
Updated: xorg-x11-apps-7.7-6.el6.x86_64
Updated: 1:java-1.7.0-openjdk-1.7.0.79-2.5.5.3.el6_6.x86_64
Updated: xorg-x11-drv-void-1.4.0-23.el6.x86_64
Updated: xorg-x11-drv-vesa-2.3.2-15.el6.x86_64
Updated: xorg-x11-drv-evdev-2.8.2-4.el6.x86_64
Updated: tigervnc-server-module-1.1.0-16.el6.centos.x86_64
Updated: mesa-libGLU-10.1.2-2.el6.x86_64
Updated: tigervnc-server-1.1.0-16.el6.centos.x86_64
Updated: tigervnc-1.1.0-16.el6.centos.x86_64
Updated: libXv-1.0.9-2.1.el6.x86_64
Updated: hwloc-1.5-3.el6_5.x86_64
Updated: 12:dhclient-4.1.1-43.P1.el6.centos.1.x86_64
Updated: system-config-firewall-base-1.2.27-7.2.el6_6.noarch
Updated: ntp-4.2.6p5-3.el6.centos.x86_64
Updated: parted-2.1-25.el6.x86_64
Updated: selinux-policy-targeted-3.7.19-260.el6_6.3.noarch
Updated: openssh-server-5.3p1-104.el6_6.1.x86_64
Updated: openssh-clients-5.3p1-104.el6_6.1.x86_64
Updated: bfa-firmware-3.2.23.0-2.el6.noarch
Updated: 1:cups-1.4.2-67.el6.x86_64
Updated: cyrus-sasl-2.1.23-15.el6_6.2.x86_64
Updated: rsyslog-5.8.10-10.el6_6.x86_64
Updated: subversion-1.6.11-12.el6_6.x86_64
Updated: nss-pam-ldapd-0.7.5-20.el6_6.3.x86_64
Updated: ipmitool-1.8.11-21.el6.x86_64
Updated: gcc-gfortran-4.4.7-11.el6.x86_64
Updated: gcc-c++-4.4.7-11.el6.x86_64
Updated: perl-Test-Simple-0.92-136.el6_6.1.x86_64
Updated: rpm-build-4.8.0-38.el6_6.x86_64
Updated: e2fsprogs-1.41.12-21.el6.x86_64
Updated: openldap-clients-2.4.39-8.el6.x86_64
Updated: gnupg2-2.0.14-8.el6.x86_64
Updated: sudo-1.8.6p3-15.el6.x86_64
Updated: cmake-2.8.12.2-4.el6.x86_64
Updated: wget-1.12-5.el6_6.1.x86_64
Updated: mailx-12.4-8.el6_6.x86_64
Updated: libgudev1-147-2.57.el6.x86_64
Updated: pciutils-3.1.10-4.el6.x86_64
Updated: python-magic-5.04-21.el6.x86_64
Updated: efibootmgr-0.5.4-12.el6.x86_64
Updated: ghostscript-fonts-5.50-23.2.el6.noarch
Updated: at-3.1.10-44.el6_6.2.x86_64
Updated: authconfig-6.1.12-19.el6.x86_64
Updated: audit-2.3.7-5.el6.x86_64
Updated: tcsh-6.17-25.el6_6.x86_64
Updated: zsh-4.3.10-9.el6.x86_64
Updated: 1:grub-0.97-93.el6.x86_64
Updated: flac-1.2.1-7.el6_6.x86_64
Updated: etckeeper-1.18.1-2.el6.noarch
Updated: 4:perl-Time-HiRes-1.9721-136.el6_6.1.x86_64
Updated: perl-CGI-3.51-136.el6_6.1.x86_64
Installed: kernel-devel-2.6.32-504.16.2.el6.x86_64
Updated: strace-4.5.19-1.19.el6.x86_64
Updated: libvisual-0.4.0-10.el6.x86_64
Updated: ncdu-1.11-1.el6.x86_64
Updated: rsync-3.0.6-12.el6.x86_64
Updated: xorg-x11-drv-ati-firmware-7.3.99-2.el6.noarch
Updated: ql2400-firmware-7.03.00-1.el6_5.noarch
Updated: xml-common-0.6.3-33.el6.noarch
Updated: ql2500-firmware-7.03.00-1.el6_5.noarch
Updated: nss-softokn-freebl-3.14.3-22.el6_6.i686
Updated: glibc-2.12-1.149.el6_6.9.i686
Updated: libgcc-4.4.7-11.el6.i686
Updated: libstdc++-4.4.7-11.el6.i686
MPSS 3.5.1 installation
tar xvf mpss-3.5.1-linux.tar
cd mpss-3.5.1
./uninstall.sh
rpm -ivh *.rpm perf/*.rpm psm/*.rpm
rpm -ivh src/mpss-modules-3.5.1-1.src.rpm
cd ~/rpmbuild && rpmbuild -ba SPECS/mpss-modules.spec
rpm -ivh RPMS/x86_64/mpss-modules*.rpm
reboot
Vtune kernel module upgrade
cp -a /opt/intel/vtune_amplifier_xe/sepdk /tmp
cd /tmp/sepdk/src
build-driver --install-dir=/tmp/sepdk/prebuilt
copy /tmp/sepdk/prebuilt/* to /opt/intel/vtune_amplifier_xe/sepdk/prebuilt/ #(/!\ nfs is RO)
boot-script -g users -d /opt/intel/vtune_amplifier_xe/sepdk/prebuilt/
echo kernel.nmi_watchdog = 0 >> /etc/sysctl.conf
reboot
MIC state snapshot
[root@idphix ~]# micctrl --pm

mic0: cpufreq_on;corec6_off;pc3_on;pc6_off
[root@idphix ~]# micinfo 
MicInfo Utility Log
Created Mon Jun  8 22:20:34 2015


	System Info
		HOST OS			: Linux
		OS Version		: 2.6.32-504.16.2.el6.x86_64
		Driver Version		: 3.5.1-1
		MPSS Version		: 3.5.1
		Host Physical Memory	: 66076 MB

Device No: 0, Device Name: mic0

	Version
		Flash Version 		 : 2.1.02.0390
		SMC Firmware Version	 : 1.16.5078
		SMC Boot Loader Version	 : 1.8.4326
		uOS Version 		 : 2.6.38.8+mpss3.5.1
		Device Serial Number 	 : ADKC32100375

	Board
		Vendor ID 		 : 0x8086
		Device ID 		 : 0x2250
		Subsystem ID 		 : 0x2500
		Coprocessor Stepping ID	 : 3
		PCIe Width 		 : x16
		PCIe Speed 		 : 5 GT/s
		PCIe Max payload size	 : 256 bytes
		PCIe Max read req size	 : 512 bytes
		Coprocessor Model	 : 0x01
		Coprocessor Model Ext	 : 0x00
		Coprocessor Type	 : 0x00
		Coprocessor Family	 : 0x0b
		Coprocessor Family Ext	 : 0x00
		Coprocessor Stepping 	 : B1
		Board SKU 		 : B1PRQ-5110P/5120D
		ECC Mode 		 : Enabled
		SMC HW Revision 	 : Product 225W Passive CS

	Cores
		Total No of Active Cores : 60
		Voltage 		 : 1020000 uV
		Frequency		 : 1052631 kHz

	Thermal
		Fan Speed Control 	 : N/A
		Fan RPM 		 : N/A
		Fan PWM 		 : N/A
		Die Temp		 : 42 C

	GDDR
		GDDR Vendor		 : Elpida
		GDDR Version		 : 0x1
		GDDR Density		 : 2048 Mb
		GDDR Size		 : 7936 MB
		GDDR Technology		 : GDDR5 
		GDDR Speed		 : 5.000000 GT/s 
		GDDR Frequency		 : 2500000 kHz
		GDDR Voltage		 : 1501000 uV

2015-06-09

New system image
idphix-default 20150609

This image:

  • Upgrade to Intel Parallel Studio XE 2015 update 3
  • Upgrade to Intel Vtune Amplifier XE 2015 update 4

(Parallel Studio XE 2015 update 3 comes with Vtune Amplifier XE 2015 update 3 -> Vtune is upgraded separatly)

$ icc -V
Intel(R) C Intel(R) 64 Compiler XE for applications running on Intel(R) 64, Version 15.0.3.187 Build 20150407
Copyright (C) 1985-2015 Intel Corporation.  All rights reserved.

$ amplxe-cl -V
Intel(R) VTune(TM) Amplifier XE 2015 Update 4 (build 410668) Command Line Tool
Copyright (C) 2009-2015 Intel Corporation. All rights reserved.

References

Acknolegment

The idphix machine was funded by the Mescal and Moais teams of LIG/Inria.

We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Tesla K40 GPU used for this research to Inria.

Personal tools
platforms