count
  • Supermicro SYS-7048GR-TR
  • (2) GTX 1080 FE
  • (2) GTX 1070 FE

Install the OS

Version:

  • Ubuntu 16.04.3-server-amd64

Install Options:

  • Standard System utilities
  • OpenSSH Server

Once the install has completed SSH into the server and update the OS to the latest packages:

$ sudo dpkg --remove-architecture i386
$ sudo apt-get update
$ sudo apt-get dist-upgrade
$ sudo reboot

When the server finishes rebooting, SSH back in and install required packages:

$ sudo apt-get install xserver-xorg-core xdm xserver-xorg-dev libgtk-3-0 libxnvctrl-dev p7zip-full pkg-config gcc --no-install-recommends

Stop the X display manager:

$ sudo systemctl stop xdm

Install NVidia Drivers

Check for the latest driver at http://www.nvidia.com/Download/Find.aspx, the link below was valid as of Aug 07, 2018.

$ wget http://us.download.nvidia.com/XFree86/Linux-x86_64/396.51/NVIDIA-Linux-x86_64-396.51.run

Make the installer executable and launch:

$ chmod +x NVIDIA-Linux-x86_64-396.51.run && sudo ./NVIDIA-Linux-x86_64-396.51.run
  • Select Accept license
  • Select NO to installing 32-bit libraries
  • Select NO to generate Xorg
  • Select OK to complete

Generate an Xorg config that enables GPU fan control:

$ sudo nvidia-xconfig -s -a --force-generate --allow-empty-initial-configuration \
--cool-bits=4 --registry-dwords="PowerMizerEnable=0x1; PerfLevelSrc=0x2222; PowerMizerDefaultAC=0x1" --no-sli \
--connected-monitor="DFP-0" -o /etc/X11/xorg.conf

Add Xorg configuration options to disable the NVidia driver watchdog.

$ sudo sh -c "sed -i '/^[[:blank:]]*Driver[[:blank:]]*\"nvidia/a\ \ \ \ Option         \"Interactive\" \"False\"' /etc/X11/xorg.conf"

Add settings required to control the GPU fans from terminal sessions (e.g SSH)

$ sudo sh -c "cat >> /etc/X11/xdm/Xsetup << EOF
export PATH=/bin:/usr/bin:/sbin
export HOME=/root
export DISPLAY=:0
xset -dpms
xset s off
xhost +
EOF"

Set the DISPLAY environment variable during login:

$ echo "export DISPLAY=:0" >> ~/.bashrc

Now reboot to verify all changes:

$ sudo reboot

When the reboot completes SSH back in and verify everything works:

$ nvidia-smi --query-gpu=gpu_name --format=csv,noheader

GeForce GTX 1080
GeForce GTX 1070
GeForce GTX 1080
GeForce GTX 1070

$ nvidia-settings -tq GPUCurrentFanSpeed

27
28
27
28

Install Hashcat

Download and unpack:

$ wget https://hashcat.net/files/hashcat-4.2.1.7z
$ 7z x hashcat-4.2.1.7z
$ cd hashcat-4.2.1.7

Verify that Hashcat recognizes the video cards:

$ ./hashcat64.bin -I

hashcat (v4.2.1) starting...

OpenCL Info:

Platform ID #1
  Vendor  : NVIDIA Corporation
  Name    : NVIDIA CUDA
  Version : OpenCL 1.2 CUDA 9.2.196

  Device ID #1
    Type           : GPU
    Vendor ID      : 32
    Vendor         : NVIDIA Corporation
    Name           : GeForce GTX 1080
    Version        : OpenCL 1.2 CUDA
    Processor(s)   : 20
    Clock          : 1733
    Memory         : 2029/8119 MB allocatable
    OpenCL Version : OpenCL C 1.2 
    Driver Version : 396.51
    <SNIP>

Run a simple benchmark to verify Hashcat is operational:

$ ./hashcat64.bin -m1000 -b

OpenCL Platform #1: NVIDIA Corporation
======================================
* Device #1: GeForce GTX 1080, 2029/8119 MB allocatable, 20MCU
* Device #2: GeForce GTX 1080, 2029/8119 MB allocatable, 20MCU
* Device #3: GeForce GTX 1070, 2029/8119 MB allocatable, 15MCU
* Device #4: GeForce GTX 1070, 2029/8119 MB allocatable, 15MCU

OpenCL Platform #2: Intel(R) Corporation
========================================
* Device #5: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz, skipped.

Benchmark relevant options:
===========================
* --optimized-kernel-enable

Hashmode: 1000 - NTLM

Speed.Dev.#1.....: 42997.9 MH/s (60.58ms) @ Accel:128 Loops:1024 Thr:1024 Vec:2
Speed.Dev.#2.....: 43085.5 MH/s (60.47ms) @ Accel:128 Loops:1024 Thr:1024 Vec:2
Speed.Dev.#3.....: 32656.8 MH/s (60.92ms) @ Accel:256 Loops:512 Thr:1024 Vec:2
Speed.Dev.#4.....: 32793.6 MH/s (60.69ms) @ Accel:256 Loops:512 Thr:1024 Vec:2
Speed.Dev.#*.....:   151.5 GH/s

NOTE: The ‘gpu-temp-retain’ option was removed from hashcat in v4.1.0 so we need to use the NVIDIA tools to manually set fan speeds.

Test GPU fan control via NVIDIA drivers:

$ nvidia-settings -a GPUFanControlState=1

  Attribute 'GPUFanControlState' (wopr:0[gpu:0]) assigned value 1.
  Attribute 'GPUFanControlState' (wopr:0[gpu:1]) assigned value 1.
  Attribute 'GPUFanControlState' (wopr:0[gpu:2]) assigned value 1.
  Attribute 'GPUFanControlState' (wopr:0[gpu:3]) assigned value 1.
  
$ nvidia-settings -a GPUTargetFanSpeed=75

  Attribute 'GPUTargetFanSpeed' (wopr:0[fan:0]) assigned value 75.
  Attribute 'GPUTargetFanSpeed' (wopr:0[fan:1]) assigned value 75.
  Attribute 'GPUTargetFanSpeed' (wopr:0[fan:2]) assigned value 75.
  Attribute 'GPUTargetFanSpeed' (wopr:0[fan:3]) assigned value 75.
  
$ nvidia-settings -tq GPUCurrentFanSpeed

75
75
75
75

(Optional) Install John the Ripper

$ sudo apt-get install libssl-dev librexgen
$ git clone -b bleeding-jumbo --single-branch https://github.com/magnumripper/JohnTheRipper.git
$ cd JohnTheRipper/src
$ ./configure
$

(Optional) Install OpenCL runtime for CPU cracking

$ sudo apt-get install clinfo ocl-icd-libopencl1 lsb-core --no-install-recommends

OpenCL™ Runtime for Intel® Core™ and Intel® Xeon® Processors

Check for the latest driver at https://software.intel.com/en-us/articles/opencl-drivers, the link below was valid as of Nov 14, 2017.

$ wget http://registrationcenter-download.intel.com/akdlm/irc_nas/9019/opencl_runtime_16.1.1_x64_ubuntu_6.4.0.25.tgz

Unpack and run the installer:

$ tar zxf opencl_runtime_16.1.1_x64_ubuntu_6.4.0.25.tgz
$ cd opencl_runtime_16.1.1_x64_ubuntu_6.4.0.25/
$ sudo ./install.sh

Comments