wiki:OsgVpbBuildEnvLinux

How to Setup OSG & VPB clustered Build Environment in Linux as Step-By-Step Tutorial

Basically it is possible to render virtualPlanetBuilder Databases on MS Windows and Linux. Unfortunately MS Windows is not stable enough (even with disabled anti-virus), maybe caused by NTFS and the heavy system load (RAM, CPU, HDD). To render large databases, it is absolutely recommended to use VPB on linux. This !tutorial is a guide how to setup a proper build environment.

System installation and basic preparation

  • Install Kubuntu 10.04 64 bit (64 bit is important to prevent your system to crash due to some ressource limits)
  • Activate proprietary video driver: Menu -> system -> driver -> reboot
  • Link shell to desktop
  • install basic packages
    aptitude update
    aptitude install vim
    

Setup Raid, if nessecary

  • Install package:
    aptitude install dmraid
    
  • Read in existing Windows or Linux raids and activate them
    dmraid -r
    dmraid -ay -v
    
  • Add raids an windows Drives to fstab
    mkdir /mnt/disk1
    mkdir /mnt/disk2
    
    vim /etc/fstab
    
    dev/mapper/isw_befejdgeeb_Volume01     /mnt/disk2      auto    defaults        0       0
    /dev/sda2                               /mnt/disk1      auto    defaults        0       0
    

lookup the device name in /dev/mapper after the last dmraid command

CUDA installation

Basically according to http://hdfpga.blogspot.com/2011/05/install-cuda-40-on-ubuntu-1104.html

In Short:

  • Stop X-Server with
    /etc/init.d/kdm stop
    
  • Uninstall the auto Nvidia driver provided by Kubuntu.
  • Download and install the newest Driver provided by Nvidia on its webpage.
  • Start X-Server with
    /etc/init.d/kdm start
    
  • Define GCC 4..4 as standart compiler for non-root:
    cd ~
    mkdir gcc44
    cd gcc4
    ln -s /usr/bin/cpp-4.4 cpp
    ln -s /usr/bin/gcc-4.4 gcc
    ln -s /usr/bin/g++-4.4 g++
    
  • Download and install CUDA:
    • Download the following files and install them as root
      CUDA Toolkit for Ubuntu Linux 10.10 (sudo sh ...) : cudatoolkit_4.0.17_linux_64_ubuntu10.10.run
      CUDA Tools SDK (sudo sh ...) : cudatools_4.0.17_linux_64.run
      
    • Download the following file and install it as non-root
      GPU Computing SDK code samples (no sudo) : gpucomputingsdk_4.0.17_linux.run
      
  • Announce CUDA to your build environment:
    • Edit /usr/local/cuda/bin/nvcc.profile to look in the gcc44 directory (append the full path):
      compiler-bindir =/home/your-non-root-user/gcc44
      
    • Append in /root/.bashrc and /home/your-non-root-user/.bashrc
      export CUDA_HOME="/usr/local/cuda"
      export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${CUDA_HOME}/lib64"
      export PATH=${CUDA_HOME}/bin:${PATH}
      
    • Take changes in effect:
      sudo source /root/.bashrc
      sudo source /home/your-non-root-user/.bashrc
      sudo ldconfig
      
  • Build GPU Computing SDK to check build CUDA client and sample programs
    • run make in NVIDIA_GPU_Computing_SDK/C
      cd ~/NVIDIA_GPU_Computing_SDK/C
      make
      
    • Verify CUDA 4.0 installation by running the deviceQuery sample in the NVIDIA_GPU_Computing_SDK/C
      bin/linux/release/deviceQuery
      

Compile and install Nvidia Texture Tools 2 (NVTT)

  • Download and install NVTT from svn (stable release 2.0.8.1 won't work with CUDA 4.0) as non-root:
    cd ~
    svn checkout http://nvidia-texture-tools.googlecode.com/svn/branches/2.0/ nvidia-texture-tools-read-only
    cd nvidia-texture-tools-read-only
    # Use CMake instead of ./configure to ensure CMake looks for cuda/lib64 instead of cuda/lib. Add CMake variable NVTT_SHARED with the boolean value 'true'. Configure CMake variable CMAKE_BUILD_TYPE as 'Release'
    make
    sudo make install
    

Compile and install OSG and its dependencies

Use this updated list which is adapted for Kubuntu 10.04:

aptitude update
aptitude install cmake subversion g++ libx11-dev nvidia-glx-185-dev libglu-dev valgrind
aptitude install libpng-dev libjpeg-dev libtiff-dev libfreetype6-dev libgdal-dev gdal-bin
aptitude install libcurl4-dev dcmtk libdcmtk1-dev libgtk2.0-dev libxul-dev libpoppler-glib-dev
aptitude install libvncserver-dev librsvg2-dev libsdl-dev libxml2-dev
aptitude install xxdiff libboost-regex-dev doxygen graphviz subversion-tools p7zip p7zip-full p7zip-rar

To use the new introduced resume function, you must use at least OSG 2.9.9 with VPB 0.9.11 Basically: VPB versions are bound to special OSG versions. (like OSG 3.0.0 is bound to VPB 1.0)

  • Compile OSG 3.0.0
    cd /tmp
    svn co http://www.openscenegraph.org/svn/osg/OpenSceneGraph/tags/OpenSceneGraph-3.0.0 OpenSceneGraph
    cd OpenSceneGraph
    # Use CMake instead of ./configure to ensure shared NVTT is used (should be nv*.so and not .a). Configure CMake variable CMAKE_BUILD_TYPE as 'Release'
    make -j 8 
    make install
    cd ..
    

To build OSG in debug mode, edit the configure file and change the build type to Debug, then repeat the last instructions.

  • ... or Compile newest OSG from SVN
    cd /tmp
    svn co http://www.openscenegraph.org/svn/osg/OpenSceneGraph/trunk OpenSceneGraph
    cd OpenSceneGraph
    # Use CMake instead of ./configure to ensure shared NVTT is used (should be nv*.so and not .a). Configure CMake variable CMAKE_BUILD_TYPE as 'Release'
    make -j 8 
    make install
    cd ..
    
  • Add OSG environment variables to make all osg binaries callable via interactive shell:
    vim /home/fsd/.bash_profile
    export PATH=$PATH:/home/fsd/OpenSceneGraph/bin
    export OSG_FILE_PATH=/home/fsd/sampledata/:/home/fsd/sampledata/Images
    
    vim /etc/bash.bashrc
    export LD_LIBRARY_PATH=/home/fsd/OpenSceneGraph/lib/:/home/fsd/VirtualPlanetBuilder/lib/
    

reboot to load changes and allow VPB compiling.

  • Compile VPB 1.0.0:
    svn checkout http://www.openscenegraph.org/svn/VirtualPlanetBuilder/tags/VirtualPlanetBuilder-1.0.0/ VirtualPlanetBuilder
    cd VirtualPlanetBuilder
    ./configure
    make -j 8
    make install
    
  • or Compile newest VPB from SVN:
    svn checkout http://www.openscenegraph.org/svn/VirtualPlanetBuilder/trunk VirtualPlanetBuilder
    cd VirtualPlanetBuilder
    ./configure
    make -j 8
    make install
    
  • Download Sampledata :
    wget http://download.osgvisual.org/sampleData.7z
    p7zip -d sampleData.7z
    mv Data sampledata
    
  • Configure ressource limits
    sudo vim /etc/security/limits.conf
    
    # End of file
    fsd  soft    nofile  65353
    fsd  hard    nofile  65353
    

If the limits are ignored or not set correctly according limits.conf, read this article how to enforce the limits: http://www.ubun2.com/question/433/how_set_ulimit_ubuntu_linux_getting_sudo_ulimit_command_not_found_error

Cluster preparation

Requirements for successful use of the cluster

  • Source data must be available on all nodes under the identical path (best solution: local source data on each node located at the same path)
  • Destination folder must be shared over any network file system and must be available on all nodes under the identical path. Easiest solution: sshfs)
  • Compile-script must be called from the destination directory (because all tasks contain the "run-path", this must be accessible by the executing node -> "run-path" must be the shared destination folder).

To fulfill this requirements, let's assume this setup:

  • One machine acts as server node and has the IP 192.168.0.55
  • Two machines act as client nodes and have the IPs 192.168.0.54 and 192.168.0.56
  • sshfs is used as network filesystem
  • the username on all nodes is fsd
  • The destination folder for the compiled database is /geodata
  • The local drive with geo source data is /localSourceData

To use vpbmaster clustered, all maschines must have access to the server node. Open a fresh shell and typ in as standart user (NOT root!):

ssh-keygen -t rsa
<if asked for passphrase: Press enter for no passphrase - otherwise the login would not be passwordless :)>

ssh-copy-id -i .ssh/id_rsa.pub fsd@192.168.0.55

Because remote ssh commands do not invoke the environment settings of an interactive shell, ssh must be prepared to provide the required environment variables also for ssh non-interactive-shell: vim /etc/ssh/sshd_config

PermitUserEnvironment yes

vim /home/fsd/.ssh/environment

LD_LIBRARY_PATH=/home/fsd/OpenSceneGraph/lib/:/home/fsd/VirtualPlanetBuilder/lib/
OSG_FILE_PATH=/home/fsd/sampledata/:/home/fsd/sampledata/Images

To share over network the destination folder to write the database in, install sshfs:

aptitude install sshfs

Create Folder for the local source data and shared destination data:

mkdir /mnt/disk3            # for source data
mkdir /geodata              # for destination data, shared over sshfs

Use Cluster

Preparation after boot of all clients

  • mount at the server node the destination harddrive into the appropriate folder:
    mount /dev/sde1 /geodata
    
  • mount on all client nodes the destination directory of the server node over sshfs.

Let's assume the server has the IP 192.168.0.56:

sshfs fsd@192.168.0.56:/geodata /geodata
<enter password of the server node>

Create the compile script

To allow easy recalls of the database build command, create a script: vim /etc/geodata/compile_BRD_Sued.sh

#!/bin/sh

vpbmaster --machines machinepool.txt\ 
 --geocentric\ 
 --terrain\ 
 --compressed\ 
 -d /localSourceData/srtm-V4.2-europa\ 
 -t /localSourceData/Muenchen_25cm\ 
 -t /localSourceData/geocontent/Deutschland_1m/Sued\ 
 -t /localSourceData/bluemarble\ 
 -o /geoData/BRD1m_MUC0.25m_srtmEU_BM/terrain.ive

Compile Database

If everything is setup correctly, the processing step ist very easy. Open an new shell on the server node and call as standard user (NOT ROOT!):

cd /geodata
./compile_BRD_SUED.sh

While the database is creating, it is possible to watch the actual progress in another console. Because files are written and modified, the viewer application will give warnings or will terminate in few cases. This happens due to the creation progress and does not affect or modify the database, please ignore it. After finished creation, this should not happen, of course ;)

Because system load during database creation is very high, on some systems the operation system or the compile process crash. If these crashes happens at the beginning of the rendering process, please check if your "openfile" limits are set correctly

Resume compiling database after crash

See http://www.openscenegraph.org/projects/VirtualPlanetBuilder/wiki/Resume

To resume the crashed compile process, execute on the server node:

cd /geodata
vpbmaster --machines machinepool.txt --tasks build_master.tasks

Take care NOT to call the compile script again, in this case all compiled tasks will be resetted to status "pending"

See Terrain Data for data sources.

Troubleshooting

If compiling fails at the beginning:

  • reboot
  • Please check if you use a 64 bit operating system
  • check your limits: {{limits -a}}

If compiling fails after many hours or days:

  • Maybe your OS runs out of any ressource, so reboot and resume (read above)
  • Check your hardware if resuming does not help.

If only local threads are executed, but no remote ones:

  • Check your ssh setup, if you can login passwordless from the server node to the client nodes
  • Check your ssh setup, if your environmental variables on the client nodes are available if you login to a client node
Last modified 9 years ago Last modified on Mar 29, 2015, 9:33:49 PM