The authors have declared that no competing interests exist.
Conceived and designed the experiments: MC MB. Performed the experiments: ZL AT FDS CE MC MB. Analyzed the data: ZL AT FDS CE MC MB. Contributed reagents/materials/analysis tools: ZL AT FDS CE MC MB. Wrote the paper: ZL AT MC MB.
The video games industry develops ever more advanced technologies to improve rendering, image quality, ergonomics and user experience of their creations providing very simple to use tools to design new games. In the molecular sciences, only a small number of experts with specialized know-how are able to design interactive visualization applications, typically static computer programs that cannot easily be modified. Are there lessons to be learned from video games? Could their technology help us explore new molecular graphics ideas and render graphics developments accessible to non-specialists? This approach points to an extension of open computer programs, not only providing access to the source code, but also delivering an easily modifiable and extensible scientific research tool. In this work, we will explore these questions using the Unity3D game engine to develop and prototype a biological network and molecular visualization application for subsequent use in research or education. We have compared several routines to represent spheres and links between them, using either built-in Unity3D features or our own implementation. These developments resulted in a stand-alone viewer capable of displaying molecular structures, surfaces, animated electrostatic field lines and biological networks with powerful, artistic and illustrative rendering methods. We consider this work as a proof of principle demonstrating that the functionalities of classical viewers and more advanced novel features could be implemented in substantially less time and with less development effort. Our prototype is easily modifiable and extensible and may serve others as starting point and platform for their developments. A webserver example, standalone versions for MacOS X, Linux and Windows, source code, screen shots, videos and documentation are available at the address:
Biology currently undergoes a rapid expansion, calling for tools to visualize huge and complex systems, such as macromolecular structures, -omics networks or even organs and organisms
For many years, game developers have gathered routines and frameworks that can be reused for a wide range of games. These code blocks have been aggregated to provide toolkits, called game engines, embedding all the components required to create a game in one package
Here, we focus on molecular and network visualization using surfaces, spheres and links between them. With this test case we assess whether by using a game engine the functionality of classical viewers could be reproduced and extended in substantially less time and with less development effort than with common tools. Current needs include the ability to display hundreds of thousands of distinguishable elements in interactive time, using classical representation schemes. To provide a realistic scenario, inspired by current research, we have implemented our own visualization algorithm based on latest hardware capabilities such as programmable Graphics Processing Units (GPUs)
We have chosen to test the Unity3D game engine (
In this article, we implement and compare several approaches for ball-and-stick representations using built-in Unity3D features. We also implemented the HyperBalls representation
UnityMol can be deployed in two versions: a stand-alone application and a web applet running on top of the Unity3D web-plugin. We have tested several approaches to achieve the most efficient and appealing visualization of spheres, links and surfaces in order to balance graphical quality and display efficiency. We created a user interface to load and manipulate molecular structures, field lines, surface objects or biological network topologies.
For structural data we use the Protein Data Bank
We have tested graphical methods available in Unity3D as well as our HyperBalls representation based on ray casting
Unity3D provides an optimized set of graphical primitives for rendering. We use triangulated spheres, triangulated cubes and lines. Spheres and cubes display atoms (for molecular structures) or nodes (for networks); links between them are rendered using lines (see
A- Particle system depicting atom spheres. B- Unity3D triangulated spheres for atoms and lines for bonds. C-
Tessellated graphical primitives require a large number of triangles to represent smooth spheres and are not appropriate to represent molecules within a game engine. Instead, a single square (composed of two triangles only) always oriented perpendicular to the screen plane can be used to represent a single atom. An image of the atom sphere called sprite is pasted onto the square. This point-sprite method is very efficient to display many similar images and can be extended to a so-called particle system (see
We have developed a molecular representation called HyperBalls
The Unity3D framework enabled us to implement such a graphical method by integrating GPU code – so called shaders – to be run on the graphics card (as Cg or GLSL instructions). We implemented HyperBalls using built-in Cg functions. To create the representation, it is necessary to generate envelopes using cubes and parallelepipeds to define atom and bond primitives, respectively. Then, we represent spheres and hyperboloids within these envelopes using ray casting.
Starting from this previously published implementation, we were able to quickly add improvements to it, such as the lit spheres shading described below.
Electrostatic potential visualization was implemented in UnityMol either as isosurfaces or as animated particles following field lines. The latter representation is similar to the rendering described in a recent publication by Andrei
Once the line coordinates are loaded into memory, we reconstruct field lines using the line renderer of Unity3D. A dash-line effect depicting moving particles with a trail is achieved using a shader that animates a sinusoidal function along the field line according to a timer. The user can change the colour, the width, the length and the speed of the particles via the GUI (see
Molecular surfaces can be imported in Wavefront OBJ format from various molecular modelling programs such as MSMS
Alternatively, UnityMol is able to generate a Gaussian molecular surface. An atomic density grid is computed by adding Gaussian distributions originating at the atom positions as discussed in
All types of surfaces can be cut according to fixed or moving planes. The interior of the molecule is filled by a uniform colour to enhance the user’s impression of cutting a solid object. The implementation uses a shader hiding the pixels of the mesh located above the plane defined by the user. The colour, orientation and position of the cut plane can be changed interactively (see examples in the
For the moment, rendering of molecular surfaces is achieved by triangulation. The quality of such a surface is still limited because of memory restrictions. To improve this issue, it may be possible to build on recent works using the full capacity of modern graphical hardware to depict molecular surfaces combining a high quality of visualization and very good display performances
On a virtual 3D object, the orientation of one pixel relative to the camera is determined by its normal. All the visible normals for one viewport can be retrieved on a theoretical half-sphere located at the centre of the screen. Thus a lighted sphere, or lit sphere, depicted in 2D contains all the information required to construct a lighting model. Sloan
A few lit spheres are included as UnityMol program resources. The user can choose a texture for surfaces and HyperBalls representations using the GUI. In the standalone version, it is possible to import custom textures directly from the hard-drive.
Development, benchmarking and testing were carried out on a Mac Pro computer with a quad-core Intel Xeon 3 GHz processor, 8 GB of memory, an NVIDIA GTX 285 graphics card, running 64bit Mac OS X 10.6. Results presented in this article are mostly based on the 3.5 version of Unity3D. All source code is implemented using C# and Cg languages built into Unity3D and is available on the sourceforge project website
In this section, we evaluate several Unity3D features on a simple test case and go on to determine the efficiency of the most promising features on a wider range of molecular structures. We then compare these features to our HyperBalls implementation.
We used the molecule ferrocytochrome C
We evaluated four methods to render atoms (see
A – Rendering efficiency for ferrocytochrome C (PDB-id 1KX2) shown as atoms and bonds measured in Frames Per Second (FPS) for a 1344×1008 viewport. B - FPS values for molecules of increasing size rendered by a particle system, HyperBalls or Unity spheres. The red dashed line divides the graph in two parts: above this line the FPS step is 50, below this line the FPS step is 5.
For entirely fluid and interactive display manipulation, it is mandatory to achieve a rendering rate above 30 Frames Per Second (FPS).
We selected the particle system, Unity3D sphere and HyperBalls sphere methodologies to evaluate their respective efficiency limits using a set of molecules with sizes ranging from about 1 200 to 100 000 atoms (see
Molecule Name | PDB ID | Nb. of atoms | Reference |
Ferrocytochrome C | 1KX2 | 1 249 | Bartalesi, et al. |
Glutathione S-transferase | 1GLQ | 3 516 | Garcia-Saez, et al. |
Adhesin | 3SYJ | 7 968 | Meng, et al. |
Alpha-galactosidase | 2XN2 | 12 627 | Fredslund, et al. |
Acetylcholine binding protein | 2XNT | 16 550 | Akdemir, et al. |
DARPin bound to AcrB | 3NOC | 25 558 | Monroe, et al. |
Beta-galactosidase | 3MUY | 36 360 | Dugdale, et al. |
RNA polymerase | 2Y0S | 52 472 | Wojtas, et al. |
Hydrolase | 2B9V | 80 710 | Barends, et al. |
ATP synthase | 3OAA | 99 573 | Cingolani and Duncan |
To improve the overall reactivity of Unity3D, in particular for slower rendering modes such as HyperBalls spheres and bonds, we have implemented a so-called Level of Details (LoD) method. This functionality automatically switches from a specific representation that is assumed to render slowly to the fast particle system representation while the user is moving the molecule. This option is called “LoD Mode” in the UnityMol Graphical User Interface. LoD is commonly used in other generic data visualization packages such as Paraview (
Recently, Andrei
A – Depiction of the electrostatic potential of the GLIC ion channel (PDB-id 3EAM) by animated trails moving along the field lines. B – Use of a cut plane to reveal the buried channel. C - The built-in Unity3D editor was used to remove some field lines and focus on the ones lining the channel pore. D – Time series showing 3 snapshots of a moving trail. E – Electrostatic potential of ferrocytochrome C shown as iso-surfaces (red : contoured at −2, blue : contoured at +2) with a toon shading to enhance the volume perception.
In addition to dynamic field lines, classical static isosurface representations of electrostatic potentials can be rendered. The perception of the isosurface shapes may be enhanced using artistic rendering described below (see
Molecular surface rendering is a fairly standard feature in molecular viewers. The addition of cut planes enables the user to easily visualize hidden elements in the interior of a molecule. This feature is particularly useful for proteins with internal cavities or channels (see
Lit sphere shading
A - All possible lighting and shading components of an arbitrary object (right) can be represented on a 2D image of a sphere (left). Colored arrows depict locations of equivalent surface orientation. B - Different lit-spheres applied to a molecular surface creating different effects : hand drawing (left), green glass (center), cartoon shading (right). C – Illustrative rendering using lit spheres and Hyperballs. Van der Waals (left) and backbone (right) representation of the BLT2 GPCR receptor with hand drawn lit spheres.
Lit sphere shading is a non-demanding method to render complex lighting effects in real time, using simple square textures. No performance loss has been observed while applying lit spheres onto a mesh or HyperBalls. As the light orientation is encoded in the lit sphere, the lighting effect is view-dependent which is desirable to achieve a consistent rendering when rotating the camera. An intrinsic inconvenience may be encountered when adding external light sources originating from a different direction than the one in the lit sphere.
As a game engine, Unity3D provides pre-configured built-in features such as visual effects, a physics engine and access to peripherals (such as joypads). These functionalities can be reused for molecular visualization and interaction.
There are two kinds of special effects in Unity: image based methods working in 2D screen space and methods working in 3D space. Using screen space effects, it is possible to deform an image, add black lines to outline contours or add glow and blur effects simulating persistence of vision (see
A – Human hemoglobin (PDB id: 1HHO) rendered with a stylized blood effect. B – Binding of a DNA molecule on the DNA-binding domain of Myb (PDB id: 1MSE). The secondary structure object was imported from PyMol. C - Contour outlining combined with glow. D – Green fluorescent protein shown as glowing carbon alpha spline (PDB id: 1KYS). E - Artistic molecule rendering : leucotoxin S component from
A virtual environment may be created by adding a so-called Skybox surrounding the molecule and decorating it with landscape images (see
In Unity3D, a physics engine is used to model physical and mechanical behaviour of rigid 3D objects such as collision, dynamics, elastic joints, etc. … To illustrate the potential of such an engine, we have implemented an interaction with the molecule based on a very simple spring network. The user can grab atoms to deform the molecule while the structure is kept coherent (“Interactive Mode” in the UnityMol GUI; see
Unity3D provides routines to use game peripherals such as joypads and joysticks. These game controllers can assist the user in navigating the scene as illustrated in
In Unity3D, it is straightforward to create a GUI without using a dedicated and potentially complex toolkit. We used built-in mouse picking functions to highlight selected atoms and access information (such as atom type, residue, etc.) or calculate distances between atoms. The UnityMol GUI, main program features and general usage of the software are described in
A particular benefit of a game engine is its modularity and the reusability of the generated routines and scripts. The use of an integrated intuitive editor facilitates the development of applications without writing numerous lines of code. It is possible to position 3D objects in the scene and apply scripts and graphical functions directly via the Unity editor. To illustrate this feature, we have adapted our molecular viewer to represent generic biological networks in 3D. To implement this network viewer (embedded in the demo), we have adapted the file parser to read XGMML (extensible Graph Markup and Modelling Language) format obtained as output of the CytoScape program
A – Original cytoscape visualization used as test case. B - UnityMol visualization with improved visibility by adding depth and contours using a lit sphere.
We have gathered recently developed graphical tools in an easy to use distributable software that provides a unique combination of functionalities absent from other publicly available viewers. We have implemented the recently developed HyperBalls representation
We compared our approach with common molecular viewers such as VMD and PyMol (see
MacOSX 10.6 - 2.93 Ghz GTX 285 | Windows 7 - 3.0 Ghz GTX 480 | ||||||||
PdB id | Nb Atoms | UnityMol V3.5 | UnityMol V4 | VMD 1.9 glsl | Pymol 1.5 | UnityMol V4 | VMD 1.9 glsl | Pymol 1.5 | |
1KX2 | 1 249 | 44 | 52 | 198 | 135 | 35 | 272 | 330 | |
1GLQ | 3 516 | 14 | 16 | 83 | 111 | 12 | 108 | 328 | |
3SYJ | 7 968 | 7 | 8 | 37 | 109 | 5 | 51 | 325 | |
2XN2 | 12 627 | 4 | 4 | 24 | 70 | 3 | 32 | 321 | |
2XNT | 16 550 | 3 | 3 | 18 | 68 | 3 | 24 | 314 | |
3NOC | 25 558 | 3 | 3 | 11 | 51 | 3 | 16 | 299 | |
3MUY | 36 360 | 3 | 3 | 8 | 44 | 3 | 11 | 278 | |
ZY0S | 52 472 | 3 | 3 | 5 | 34 | 3 | 7 | 239 | |
2B9V | 80 710 | 3 | 3 | 4 | 26 | 3 | 4 | 197 | |
3OAA | 99 573 | 3 | 3 | 3 | 19 | 3 | 3 | 135 |
In addition to molecules, we have implemented a 3D visualization of biological networks from Cytoscape
UnityMol is different from classical pre-compiled molecular viewer executables. In addition to the standalone and webserver versions, it can be run directly within the freely available Unity3D game engine. The game engine environment provides the user with full control over any detail of UnityMol. Because the Unity user interface is very intuitive, many simple things can be changed by the click of a button. It is straightforward to extend the application. If a GUI menu entry is missing, it can be added in minutes with just a few lines, most often by copying existing examples in the code. Moreover, when running UnityMol from within Unity3D, one can switch from game to editor mode and hand-edit the scene. Because the elements in the scene are Unity3D primitives, they can be modified on-screen by the user. For example, when visualizing field lines, the editor mode enables the user to select and delete field lines that are unwanted and hence fine-tune the scene (as in
Thus, such a wide range of control by the user could be seen as a new paradigm for scientific software development and presents a clear rupture with current approaches used for molecular visualization tools. At best, common packages provide scripting access to selected functionality to the user, otherwise programming know-how is required to delve directly into the application source code – if it is available. In the context of open computer programs
We have described how to use a game engine such as Unity3D to create a viewer for molecular structures as well as for biological networks. During the development of these tools, it became obvious that a game engine facilitates the development of scientific applications: the editor window provides an interface to position 3D objects and assist with the attribution of functionalities, e.g. implemented by a script, by dragging an icon onto the target object. Thus, the amount of lines of code to be written is rather limited and can in large parts be hidden for non-experts. Development is further facilitated by support from a very active Unity3D user community (see
We only scratched the surface of many features that can be further extended to develop efficient programs. For example, the physics engine could be exploited to create spring networks, molecular dynamics and flexibility. Using this physics engine combined with joystick or Wiimote controllers, it would be straightforward to develop simple tools for interactive docking.
Our immediate plans are to extend UnityMol with practically important aspects for the biophysical community such as molecular dynamics trajectory loading and analysis. We started to experiment with the MDDriver library
The video game industry is now booming and outperforms the movie industry with revenues reaching $18.6 billion in the U.S. in 2010. Interactivity is a key advantage that can be achieved with this medium. Direct interaction with the game itself as well as the creation of common virtual worlds where players can interact with each other contribute to this success. Such games can be a platform for education
In the context of big data challenges, the development of citizen science – sometimes referred to as crowdsourcing - has been particularly welcomed and has proven its usefulness
(PDF)
We are indebted to Prof. Ge Chen for his kind assistance. M.C. would like to thank Dr J.P. Nominé for fruitful discussions.