Skip to content

GitLab

Last edited by Alexander Wiebel
Page history

Module_ExternalDevelopment

Out-Of-Source Development of Modules

Module development in OpenWalnut can be done in-source and externally (out-of-source). The in-source development is very easy to set-up and a perfect way to directly dive into OpenWalnut and its module development. A very good example on how to develop modules in-source is the template module found here: source:src/modules/template/. Basically, adding a new module in-source is as easy as creating a new directory, putting some sources into it and registering the new module in source:src/modules/modules-others.toolbox.

So, why should I switch to external (out-of-source) module development if the in-source development is that easy? Do you develop Qt applications inside the Qt source tree? Probably not. The main advantages of external module development in OpenWalnut are:

  • Development for a certain OpenWalnut version
    • Keep compatibility
    • Easier distribution in your department/group
    • Exclude problems caused by a bug in the current development version
  • More flexibility
    • Add the external libs you need without bloating OpenWalnut
    • Use your own and modified datatypes, datesets and structures throughout your module toolbox without conflicting with the OpenWalnut types
    • You can even create a very own GUI tailored for your special use case and module toolbox
  • Faster compilation

NOTE It is strongly recommended to develop modules out-of-source.

In this article, we show how to develop modules externally and how to setup your OpenWalnut to use these modules.

Prerequisites

What do you need to start? The answer is simple: OpenWalnut and its dependencies. This is very system specific and you should refer to the installation guide. Just make sure you install the developer packages for OpenWalnut and its dependencies.

Terminology

First, lets define some of the words you will read quite often in this article.

Library: A compiled piece of code that can be loaded during runtime (so,dll,dylib). Module: In OpenWalnut terms, a module is a single functional unit that can be plugged into a data flow network. It is a class derived from WModule. Toolbox: Historically, one library contained one module. So, in OpenWalnut, a library basically is a module. Today, you can have multiple modules (WModule-derived classes) in one library. But still OpenWalnut uses the term Module for these libraries. The term toolbox is only for distinguishing between libraries containing one and multiple modules.

Our Build System and Basic Usage

In this tutorial, we only introduce our CMake build scripts. By using our CMake scripts, you gain several advantages. Our build scripts handle nearly everything for you:

  • shaders
  • additional resources
  • documentation
  • unit testing
  • stylechecking
  • installation

However, if you decide to use your own build system, you can use pkg-config to find OpenWalnut (beginning with OpenWalnut 1.3 (and the current development code)). If this is the case, you can skip the remaining text.

Now, please download MyOpenWalnutToolbox.tar.gz. This package provides everything you will need to build modules using our CMake scripts. Unpack it to a directory of your choice. Lets have a look what's there.

  • build
    • this directory is empty and will contain your binary module
  • doc
    • this contains the configuration for building the developer doc using doxygen
  • resources
    • files managed as global resources
  • src
    • this contains the source and CMakeLists.txt
  • tools
    • in tools you will find the OpenWalnut CMake scripts and the stylechecker
  • VERSION
    • with this file you can utilize the integrated version management

In our first example, we concentrate on building your first external module. Before you can build the module, you need to find a way to tell the build system where to find OpenWalnut include files and the library. If you have installed OpenWalnut to a system location ( /usr, /usr/local, c:\Program Files\OpenWalnut or c:\Program Files (x86)\OpenWalnut), the build system will find everything on its own. If not, you need to point cmake to the locations manually. There are several ways to do this.

  1. Set OPENWALNUT_LIBDIR and OPENWALNUT_INCLUDEDIR as environment variables

    • Let them point to the directory containing the OpenWalnut sources (include files) and lib directory inside your build directory

  2. Issue cmake -DOPENWALNUT_LIBDIR=/My_OW/build/lib -DOPENWALNUT_INCLUDEDIR=/My_OW/src ../src

    • Let them point to the directory containing the OpenWalnut sources (include files) and lib directory inside your build directory

  3. Use a cmake GUI and set OPENWALNUT_INCLUDE_DIR and OPENWALNUT_LIBRARIES manually

    • Note that the variable OPENWALNUT_LIBRARIES does not contain the directory of the lib as OPENWALNUT_LIBDIR
    • Let OPENWALNUT_LIBRARIES directly point to the libopenwalnut.[so|dll|dylib]

In the first and second options, OPENWALNUT_LIBDIR needs to point to the directory containing the file libopenwalnut.[so|dll|dylib]. The value of OPENWALNUT_INCLUDEDIR is the directory containing the OpenWalnut headers. If you have build OpenWalnut from source, this can point to the src directory of your OpenWalnut checkout.

With this in mind, you can build the module as follows:

# tell where to find OpenWalnut
export OPENWALNUT_LIBDIR=~/My_OW/build/lib
export OPENWALNUT_INCLUDEDIR=~/My_OW/src
# go to build directory
cd MyOpenWalnutToolbox/build
# let cmake build the make files
cmake ../src
# let make build the module
make

This is it. You have build your first module for OpenWalnut! The next section will tell you how to actually use the module in your OpenWalnut installation.

Using the Module in OpenWalnut

You now need to tell OpenWalnut where to find your module. Unfortunately, this has been subject to several changes in the past.

In all version after 1.2.5 (including the current developer version):

In newer OpenWalnut versions you have three possibilities.

  1. Use the configuration dialog in OpenWalnut

    • Start OpenWalnut
    • Open the Menu called Settings->Configure Modules
    • Switch to the tab called Module Paths and click Add
    • Select the path to your module build directory
    • Save and restart OpenWalnut.

  2. Use $HOME/.OpenWalnut/modules

    • You can link or copy your module to this directory. It is searched automatically.
    • $HOME is system dependent
      • Linux and Mac: this is the $HOME environment variable
      • Windows: this is something like C:/Documents and Settings/Username
    • You need to link/copy the library itself and the share directory (only if you use shaders and resources)

  3. Use OW_MODULE_PATH as in 1.2.5

In OpenWalnut 1.2.5:

Set the environment variable OW_MODULE_PATH. This is a semicolon separated list of paths to search recursively. Add your complete build path to this variable:

export OW_MODULE_PATH=/home/someone/MyModuleToolbox/build

This is it. You now have a environment for developing OpenWalnut modules and you now know how to use them in OpenWalnut.

Build System Convenience

This section contains a short description of the addition features in our build system. They do not take up any additional setup effort.

Code Style

You hate ugly looking code? In OpenWalnut, we provide a coding guideline and coding standards. You utilize our style check mechanism too. If you have Python installed:

cd build
make stylecheck

In our example, this will (at least) complain that you do not have a comment header and copyright header in your files. The stylecheck target automatically checks your cpp,h and glsl files.

Code Documentation

If you have doxygen installed, you can utilize the automatically added doc target. This target uses the doxygen configuration doc/developer/doxygenConfig. It automatically created reference documentation for your code. If you, additionally, have the graphviz tools installed, it also created nice looking UML class diagram and call/caller diagrams.

cd build
make doc

This will also warn you about undocumented functions, members or classes. The built documentation can be found in doc/developer/html/index.html.

Installation of Modules

If you finally have your module done and you want to re-distribute it, nothing is more handy than a installation target in the Makefile. The OpenWalnut build scripts already provide this feature automatically for all modules you add via ADD_MODULE or SETUP_MODULE. This also works for all other convenience functions, like SETUP_RESOURCES and similar. As long as you rely on our OpenWalnut build scripts, you can rely on automatically created install targets. Simply call the install target:

cd build
make install

This will install your module to your system's default location (i.e. /usr/local). You can change the target directory by setting the CMake variable CMAKE_INSTALL_PREFIX, either using the CMake GUI or on command line:

cd build
cmake -DCMAKE_INSTALL_PREFIX=/my/install/path

Getting the Most out of the OpenWalnut Build System

In the last sections you have learned the basics on using our build scripts to build modules and how to use them in OpenWalnut. This section and its subsections concentrate on the more or less advanced details in our build scripts. You will learn how to add modules, add unit tests and make use of other integrated features. This section can be seen as a list of recipes. Combine them in your projects and utilize as much as possible from our build system. If you need to modify several things, please consider doing this in your CMakeLists.txt, for example, by overwriting options. Please keep in mind that we permanently extend and bug fix these scripts and if you like to profit from these improvements, you should avoid heavy modifications in the OpenWalnut*.cmake scripts.

If you do not know what CMake is or never have used it, this is a good opportunity to learn it. See http://www.cmake.org/.

The CMakeLists.txt

As in every project that uses CMake, you will a CMakeLists.txt in the src directory. This is the entry point, where you can begin modifying your module project. First, lets have a look at this file:

# CMake requires this
CMAKE_MINIMUM_REQUIRED(VERSION 2.8)
# How is your module/toolbox named?
PROJECT( MyOpenWalnutToolbox )

# where does cmake have to search the OpenWalnut.cmake script?
SET( OW_TOOLS_DIR ${PROJECT_SOURCE_DIR}/../tools )
LIST( APPEND CMAKE_MODULE_PATH ${OW_TOOLS_DIR}/cmake )
# use the OpenWalnut.cmake script
INCLUDE( OpenWalnut )

# IMPORTANT:
# put your own dependency find/include stuff here, AFTER the INCLUDE( OpenWalnut ) command!

# Add the complete src directory as one module toolbox.
SETUP_MODULE( 
    ${PROJECT_NAME}   # use project name as module(-toolbox) name
    "."               # where to find the sources
    ""                # no additonal dependencies
    ""                # no sources to exclude
)

This looks awesomely minimalistic doesn't it? If you are used to use CMake, you probably know that CMake files tent to explode in size once the project gets a little bit more complex. The OpenWalnut CMake script tries to hide as much as possible. On inclusion, this script initialized the typical compiler options, searches OpenWalnut and all the needed dependencies. The call SETUP_MODULE ensures that your complete src directory is used to build one library which then contains all the modules you add to src. In the following recipes, we use this basic file as reference.

IMPORTANT: the line INCLUDE( OpenWalnut ) must be called before any further operations you might want to add (find_package and similar).

Recipe: Adding own Dependencies

Assume you want to use other libraries in your code. How would you handle this? As usually done in CMake, you can utilize CMake's FIND_PACKAGE function. It will return a list of libs and the include directories. The include directory then needs to be add to the list of include directories and the library needs to be added as dependency to SETUP_MODULE:

.............

# search the libs
FIND_PACKAGE( JPEG REQUIRED )
FIND_PACKAGE( PNG REQUIRED )
# add the header directory to the include path
INCLUDE_DIRECTORIES( ${JPEG_INCLUDE_DIR} )
INCLUDE_DIRECTORIES( ${PNG_INCLUDE_DIRS} )

# Add the complete src directory as one module toolbox.
SETUP_MODULE( 
    ${PROJECT_NAME}   # use project name as module(-toolbox) name
    "."               # where to find the sources
    "${JPEG_LIBRARIES};${PNG_LIBRARIES}" # add the lib as dependency (;-separated)
    ""                # no sources to exclude
)

This example searches JPEG and PNG support and ensures these libs get linked to the final module.

Recipe: Adding External Code

Sometimes it is required to add other code to the project. The reasons are versatile. Let us assume you want to add two external libraries, residing in src/ext/. What you need to do is to write CMakeLists.txt for these libs. Maybe they authors already provide such a file. See http://www.cmake.org for details on how to do this. Let us assume you have written them. You then can add these externals to your CMake file and add the build libs as dependencies for your module. It is very similar to the above recipe.

.............

# build external libs
ADD_SUBDIRECTORY( ext/libA )
ADD_SUBDIRECTORY( ext/libB )

# add the header directory to the include path
INCLUDE_DIRECTORIES( ext/libA )
INCLUDE_DIRECTORIES( ext/libB )

# Add the complete src directory as one module toolbox.
SETUP_MODULE( 
    ${PROJECT_NAME}   # use project name as module(-toolbox) name
    "."               # where to find the sources
    "libA;libB"       # add the lib as dependency (;-separated)
    ""                # no sources to exclude
)

The same way you can add arbitrary other libs and executables. Just utilize CMake's tools to achieve this.

Recipe: Add Unit Tests

We encourage you to write tests for your data types, algorithms and other code. We already provide a ready-to-use integration of cxxtest (http://cxxtest.tigris.org/). When using SETUP_MODULE, unit tests that can be found somewhere inside the given source directory are automatically detected and compiled. Let us assume you want to add a test ANiceTest to test something in your AnotherModule directory.

cd src/AnotherModule
mkdir test
TEXTEDITOR ANiceTest_test.h

There are two naming rules to ensure the build system finds these tests: they must reside in a directory called test and the filename needs to end with _test.h.

Sometimes it is required to test against a certain file or dataset. The OpenWalnut build scripts are able to handle these so-called fixtures. Just create a subdirectory in your test directory and copy the files your need to this location:

cd src/AnotherModule/test
mkdir fixtures
cp /some_dir/my_test_data_file .

You need to allow CMake to find these tests now. This requires a re-run of cmake. After this, the tests can be compiled and executed:

cd build
cmake .
make test

Recipe: Adding Shaders

Have you ever written CMake code which handles non-compile files during build and installation? This can get very complex, which is why we already provide a automatic mechanism for you. Lets assume your module MyNewModule needs a shader. In your module directory src/MyNewModule, create a directory called "shaders". Put your shaders to this directory:

cd src/MyNewModule
mkdir shaders
TEXTEDITOR MyNewModule-vertex.glsl
TEXTEDITOR MyNewModule-fragment.glsl

You now need to inform CMake about the new files and let make copy them. After the make-command ran through, the shaders are placed in your module's share-directory.

cd build
cmake .
make
ls share/openwalnut/modules/MyOpenWalnutToolbox/shaders

A very nice developer feature is to link shaders instead of copying them. This is practical since OpenWalnut can reload shaders on-the-fly by pressing the "."-key (dot). To enable this, enable the option OW_LINK_SHADERS in CMake. This way, you can edit the shaders inside the src directory and directly use them in OpenWalnut. For more details on shaders in OpenWalnut, refer to our shader tutorial.

Recipe: Adding Resources

Sometimes, modules need additional files. Config-files, Textures, Lists, and so on. With our build system, you have two options. Provide global resource for the whole project or local resources for each toolbox library you build (remember: a toolbox is one lib and can contain multiple modules). Local resources are handy if they are only required by one toolbox, If several toolboxes in your project share the same resources, you should choose global resources.

Local Resources

To add automatically handled resources to a toolbox, create a directory called resources inside the source directory of the toolbox. Please remember that the source directory of a toolbox is the one you gave as second parameter to SETUP_MODULE.

The CMake function SETUP_MODULE allows you to add a complete part of your source tree to be compiled into one library (= toolbox, as we learned in the terminology section), regardless of how much WModule-derived classes are defined there. Assume the two project's CMakeLists.txt

................

# Create two libraries, each with another module:
SETUP_MODULE( 
    "toolboxA"
    "MyNewModule"     # where to find the sources (src/MyNewModule)
    ""                # no additonal dependencies
    ""                # no sources to exclude
)
SETUP_MODULE( 
    "toolboxB"
    "AnotherModule"   # where to find the sources (src/AnotherModule)
    ""                # no additonal dependencies
    ""                # no sources to exclude
)

And a second project building both modules into one toolbox:

# And create an all in one toolbox
SETUP_MODULE( 
    "toolboxAllInOne"
    "."               # where to find the sources (everything in src)
    ""                # no additonal dependencies
    ""                # no sources to exclude
)

This will create three libs:

  1. libtoolboxA

    • Contains only MyNewModule

  2. libtoolboxB

    • Contains only AnotherModule

  3. libtoolboxAllInOne

    • Contains both, MyNewModule and AnotherModule

Do you see the source directories given to the function? So, where to place local resources for:

  • toolboxA
    • src/MyNewModule/resources.
  • toolboxB
    • src/AnotherModule/resources.
  • toolboxAllInOne
    • src/resources

Conclusion: do not be shocked. It is simpler than it looks. Just copy your files to a directory called resources in the directory you specify in your SETUP_MODULE command.

After you have set this up, you need to re-run cmake and make as you are used to. The resource files will then be copied to share/openwalnut/modules/toolboxA/, and share/openwalnut/modules/toolboxB/ or share/openwalnut/modules/toolboxAllInOne/ respectively. But how to use these resources in your code? Every module has a protected member called m_localPath. This points to the directory which contains the module's local resources and can be used to construct the path to your resource. As the directory structure under your local resource directory will be retained, just add your path to m_localPath as in this example:

debugLog() << ( m_localPath / "mytexture.png" ).string();

Please remember that m_localPath is of type boost::filesystem::path. You should get used to this API beforehand as it can simplify most of your daily file handling needs.

Global Resources

Global resources are not setup automatically. You need to tell the OpenWalnut build system about them. Global resource file are very useful if you have multiple libraries in your project sharing the same files. First, let us create some resource files:

cd MyOpenWalnutToolbox
mkdir resources
mkdir resources/MyGlobalResources

Now, copy all the files including the wanted directory structure to the newly created directory. Now add the following line to your CMakeLists.txt

.........
SETUP_GLOBAL_RESOURCES( "MyGlobalResources" MyGlobalResourcesComponentName )

You can add this at an arbitrary point in your file (after including OpenWalnut of course). Now, run cmake and make. What do you see in your build directory? Right. The files from within resources/MyGlobalResources. As OpenWalnut tries to comply to the unix file system hierarchy standard, you should organize your global resources accordingly. Of course this might seem to be unimportant as long as you develop your module and never install it. But expect your module to be installed some day, somewhere! A good choice is to use resources/MyGlobalResources/share/openwalnut/modules/MyOpenWalnutToolbox/globals. This complies to the filesystem hierarchy standard and the OpenWalnut module standard. This way, you can simply pack your module into a tar/zip or distribution package and nobody messes up its system with messy resource file architectures flying around somewhere in the root filesystem.

Similar to local resources, the resources in resources/MyGlobalResources/share/openwalnut/modules/MyOpenWalnutToolbox/globals can be addressed from within your code using m_localPath. As we use a path that complies to OpenWalnut's standards (and actually seems like a local resource path for the module) you can use the resources this way:

debugLog() << ( m_localPath / "globals" / "mytexture.png" ).string();

Please remember that m_localPath is of type boost::filesystem::path. You should get used to this API beforehand as it can simplify most of your daily file handling needs.

Recipe: Platform Specific Resources

You have some additional files which are very system specific? No problem. Assume we have setup some global resources named MyGlobalResources, as in the above recipe. Besides the common files in resources/MyGlobalResources, which work on all platforms, you can add windows specific files:

  • resources/platformDependent/MyGlobalResources/linux
  • resources/platformDependent/MyGlobalResources/windows
  • resources/platformDependent/MyGlobalResources/macosx

Thats it. Simply put your files into the directory platformDependent/RESOURCE_NAME/OPERATING_SYSTEM.

Recipe: I Dislike the Resources Directory -> Setup Additional Files Manually

If you have certain files which do not perfectly match into the resource directory, you can setup additional file management manually. Assume you have a directory called examples in your toolbox root directory and you want this to be copied to share/MyOpenWalnutToolbox/examples. Add the following lines to your CMakeLists.txt, somewhere after OpenWalnut inclusion:

SETUP_ADDITIONAL_DIRECTORY( "share/MyOpenWalnutToolbox"         # target
                            ../examples/                        # source
                            "examples"                          # install component name
                            FALSE    # this denotes that the whole examples directory is copied to the target
                          )  
SETUP_ADDITIONAL_DIRECTORY( "share/MyOpenWalnutToolbox/help"    # target
                            ../doc/user/                        # source
                            "examples"                          # install component name
                            TRUE    # this denotes that the contents of doc/user is copied to the target
                          )

Please note that the difference between both calls is the fourth parameter! This influences what is copied.

Recipe: Adding further Documentation

To achieve this, you simply could use the SETUP_ADDITIONAL_DIRECTORY function from the above recipe. For your convenience, we also provide a special function for documentation files. Basically, it does the same as SETUP_ADDITIONAL_DIRECTORY but knows the proper target directory for you:

SETUP_COMMON_DOC( ../doc/helpfiles/                   # source
                   "userdoc"                          # install component name
                )

You should prefer this over the more general SETUP_ADDITIONAL_DIRECTORY for documentation.

Recipe Missing?

Is something missing you would like to read here? Contact us! We are very interested in comments, ideas, critics and recommendations.

Frequently Asked Questions

What is the difference between SETUP_MODULE and ADD_MODULE?

The function ADD_MODULE is a wrapper around SETUP_MODULE. It provides less options but is easier to use. See the documentation in tools/cmake/OpenWalnutUtils.cmake for details.

You often say "install component". What dose that mean?

The install component is an important parameter for many of our CMake functions. It defines to which component files and compiled libs/exe belong to. Why not using the same name for everything? The reason is obvious if you want to provide packages for your toolbox. This way, you can split installation of your lib from its documentation or examples. To install a single component only use:

cd build
cmake -DCOMPONENT=MyOpenWalnutToolbox_Global -P cmake_install.cmake

This installs the resources you associated with the name MyOpenWalnutToolbox_Global. You can even add convenience targets easily to your CMakeLists.txt:

ADD_CUSTOM_TARGET( install_globalresources
                    ${CMAKE_COMMAND}
                        -DCOMPONENT=MyOpenWalnutToolbox_Global
                        -P ${CMAKE_BINARY_DIR}/cmake_install.cmake
                 )

Now, you can use make install_globalresources.

Why is there no answer to my question?

You have a question? Simply ask it. Contact us and we will help you.