REVIEW OF AUGMENTED REALITY AND VIRTUAL REALITY TECHNIQUES IN ROCK ART

The usage of augmented reality (AR) and virtual reality (VR) technologies began to grow when smartphones appeared. Until then, the number of portable devices capable of incorporating these technologies was reduced. Video games are the main field where these technologies are applied, but in other fields such as in archaeology, these technologies can offer many advantages. Ruins reconstruction, ancient life simulation, highly detailed 3D models visualisation of valuable objects from the past or even user free movement in missing places are just some examples found in literature. This paper reviews the latest visualisation technologies and their applicability to the rock art field. The main purpose is to disseminate rock art paintings through AR and VR applications. After the image-based three-dimensional (3D) modelling is obtained, an interactive visit to a shelter for displaying rock art paintings is presented. This is one of examples developed in this paper that pretends to apply the revised AR and VR techniques. In addition, an example of AR is developed that can be easily adapted to further applications displaying rock art paintings.


Introduction
The International Council on Monuments and Sites (ICOMOS) defines cultural heritage as follows: "Cultural Heritage is an expression of the ways of living developed by a community and passed on from generation to generation, including customs, practices, places, objects, artistic expression and values. Cultural Heritage is often expressed as either Intangible or Tangible Cultural Heritage" (ICOMOS 2002).
Tangible cultural heritage is formed by features such as buildings, structures, art works, documents, artefacts and rock art paintings.It is everyone's responsibility to understand, appreciate and conserve the cultural heritage.For this purpose, this paper describes new ways to understand and disseminate the cultural heritage, and specifically, rock art paintings, based on the fast development of both augmented reality (AR) and virtual reality (VR) technologies.These new technologies allow users to show highly accurate 3D models of tangible cultural heritage through realistic immersive environments.
AR is a visualisation technique in which virtual contents such as 3D models, text, video and other multimedia formats, are placed on top of the real world camera view (Kounavis et al. 2012).The user simultaneously can perceive the real environment and virtual elements overlaid in real time.AR technology provides a new way to show the cultural heritage.An example of this is Archeoguide (Vlahakis et al. 2002).It offers AR tours of archaeological sites and uses this technique to enhance information presentation, reconstruct ruined sites and simulate ancient life.
Nowadays, AR technology is used in many fields, such as entertainment, education or medicine but, it is also used in the tourism sector to enrich and enhance the user experience.There are several examples making use of this approach.An AR museum guide is presented by Miyashita et al. (2008).This AR guide provides the visitor a basic knowledge of some artworks and drives the visitor through the exhibition.Another example is described by Gutierrez et al. (2015) who presented an AR application to release information about historical graffiti in the Temple of Debod.
VR technology immerses completely users in a virtual world, a world created by computers.The difference between these two technologies is that while AR technology combines the virtual world with the real world, VR only shows virtual elements.VR offers the possibility to visit inaccessible sites or sites that no longer exist.The visualization process of the generated VR is carried out through head-mounted displays (HMD).These HMD have dramatically improved and nowadays are lightweight glasses.An example of this is the virtual reality glasses Oculus Rift, developed by Oculus VR.Although this last generation of HMD is focused on the field of gaming and entertainment, its application in other fields is worth mentioning.Specifically in the field of medicine, Draganov and Boumbarov (2015) investigate a medical assistive system for motor disabled patients through an Oculus Rift VR Display.Other research is focused on disseminating the cultural heritage content through the virtual reconstruction of a typical theatre of the time (Valtolina et al. 2005) or through a 3D virtual museum tour of the Santa Maria Della Scala using the Oculus Rift (Fineschi and Pozzebon 2015).
Both 3D visualisation techniques are quite similar.VR users go into a completely immersive virtual world outside of the real world, meanwhile AR users stay in a semi-immersive environment where they can interact with virtual objects around them in the real world.
In the 1950s appeared the first VR system, but until recently these technologies were rarely used.The user had to carry a computer and a HMD as well as a GPS, webcam, wireless LAN and other components.This hardware was complex and heavy.Besides hardly accessible for the majority of users.Nowadays, the breakthrough in mobile technologies and information technology (IT) infrastructure has allowed the growth of these technologies, especially with smartphone development.A smartphone is a small computer that is fully equipped with a variety of devices such as GPS, wireless internet, Bluetooth, high resolution video camera, accelerometers, gyroscopes, digital compass, barometer, proximity sensors, and other components.Due to hardware features, the smartphones have emerged as potential platforms for implementing mobile AR and VR (Kim et al., 2014).The user is acquainted with the use of a smartphone as a tool for its daily tasks.Furthermore, it is a friendly and portable device.For all these reasons, the smartphone is a perfect tool to implement these kinds of visualization technologies.
The following sections describe AR and VR technologies in detail, and also the current 3D frameworks and processes to develop an application.To test these solutions a smartphone app is developed that aims to teach and disseminate rock art paintings located in an inaccessible shelter.

AR technology
In the first AR systems, the user used to carry a computer and a HMD, and this entailed many limitations.Firstly, the hardware was uncomfortable and not very portable.Secondly, the HMD used to interrupt the user's natural view and prone to cause discomfort such as nausea or dizziness.
Finally, AR systems implementation for that particular hardware required much time and effort from the developers (Kim et al. 2014).
These limitations were improved with the appearance and fast evolution of the smartphones.Nowadays there are powerful smartphones on the market which work as a computer and HMD.As a consequence, the user does not have to carry special hardware and improves users' mobility.Therefore, at present there are many mobile AR applications (apps) on the market.
A mobile AR app consists in overlaying digital information onto the real world.The digital information is shown through the smartphone camera and is placed in a specific position.There are different ways to position information, such as mobile location.All current smartphones are equipped with GNSS localization systems and orientation sensors, among others.
The orientation sensor obtains information by processing the raw sensor data from the accelerometer and the geomagnetic field sensor.The orientation sensor provides the azimuth (degrees of rotation around the z axis), pitch (degrees of rotation around the x axis) and roll (degrees of rotation around the y axis) values.These values set out the direction of the device.
When a smartphone is placed in a location with known coordinates and orientation, the digital information is visualised on the camera.The disadvantage of this method is the accuracy of the GNSS positioning.In areas with tall buildings or with many trees, the signal bounces off and accuracy decreases.If it occurs, the digital information will be badly placed.Besides, this system does not work inside buildings.
Another way of showing the digital information is through the use of markers.A marker is a square frame made up of a light coloured usually white, surrounded by a dark coloured, usually black.Each marker is formed by a different pattern which makes them unique.For example, in the Vuforia markers the unique ID is encoded into a binary pattern along the border of the marker image (Fig. 1).In marker recognition process, first smartphone camera collects and processes the images in real time with aim of finding a pattern known.The image recognition techniques are based on finding patterns of known shapes, colour detection, geometries and repeated patterns.When a maker is detected, the virtual information overlaps on the camera.
Finally, using images or objects instead of markers is possible.This is known as Natural Feature Tracking (NFT).NFT allows users to recognise and track natural features on images and objects.This is very useful because it can be used for example, in advertising to augment the images of a brochure.The digital information can be of any type, such as 2D images, 3D models, text, animation, video, music, etc.This opens up the possibility of creating many and varied applications.Carmigniani et al. (2011) review the main AR apps.In advertising, it is used to promote new products online.In entertainment and education, AR is applied to develop cultural learning games, which are games with educational purposes.In medicine, AR apps are utilized to assist surgery achieving thus, a minimum invasive surgery with improved precision, dexterity, and visualisation.
Nowadays, AR development apps is easier than some years ago.An advantage is that everybody is able to create apps for the same hardware (smartphone).This enables to advance towards best app development due to large number growth of software frameworks.A framework generally provides some base functionality which can be used and extend to make more complex applications from.The main advantages for using frameworks are: they provide an overall application structure, they facilitate the collaboration between developers, and there are many available tools and libraries adapted to frameworks.
At present, there are many available frameworks to develop easily AR applications.The choice of one or another depends on application functionality and eventually the OS used.Kounavis et al. (2012) show some of the publicly available frameworks.In this paper, current free solutions have been analysed and are presented below: - ARToolkit is an open source AR library for iOS, Android, Linux, Windows, Mac OS X and it has a plug-in for Unity3D  1 summarizes current free features on AR frameworks.This list changes very fast since many frameworks stop being maintained and are outdated quickly or end up being bought by big companies such as Metaio SDK recently acquired by Apple.The most complete and easy to use tools are Vuforia and ARToolKit but if the goal is to make a location based app will have to choose another frameworks presented.Finally, and to conclude, AR apps are experiencing rapid development and their applications are endless.In addition, these enhance reality users' perception and surrounding environment.In this way, information becomes interactive and easily manipulated digitally.

VR technology
Although AR and VR are quite similar technologies, both represent different realities.While AR adds elements to the reality, VR creates a new reality that does not exist.
VR is an artificial environment created with software and presented to the user such as a real environment.The VR is represented in a 3D image that can be explored interactively, by means of gloves with sensors, controls such as a game console or a computer mouse so that image content moves in any direction or zooms in/out.
The VR applications (apps) require using a HMD, such as glasses, to visualise the virtual environment.The first systems of VR needed a complex HMD and the graphics quality was very low.Nowadays, thanks to the technological advances, the HMD has evolved considerably and there are currently powerful solutions in the market such as Oculus Rift (Oculus Rift 2016).
The Oculus Rift is a VR HMD developed by Oculus VR.
Its operation is based on the stereoscopic vision (imitating natural vision).The stereoscopic vision is any technique capable of collecting 3D visual information and creates the illusion of depth in an image.
In the natural stereo vision, the eyes, due to their separation (interpupillary distance), obtained two images with small differences between them.These differences are processed in the brain to produce depth perception.
The HMD projects a stereoscopic vision.The screen displays two images side by side, one for each eye.
A small controller on the basis of the display is used to adjust the interpupillary distance.The eyes separation varies between individuals and is an important element to give a realistic sense of stereoscopic vision.The HMD feature a virtual surround sound system and sensors that control the movements of the user's head and adjust the picture accordingly.For the time being, this device is intended to connect the computer and the smartphone.
This type of totally immersive 3D vision makes some users experience effects such as motion sickness and headaches.To avoid this, it is necessary to adjust the lens to each person vision, but even so, prolonged use might cause discomfort.
Visualising the VR with a HMD is sometimes a bit annoying.However, it is possible to create a semiimmersive app where the 3D model is shown in the mobile display and the user can interact with it through the touch screen.
There are some frameworks that allow users to load 3D models and interact with them.These frameworks are useful to create simple semi-immersive apps.However, the best solution is a game engine in order to work with 3D models and for rendering, creating animations, and so on.A game engine is a software framework designed for the creation and development of video games ready to be used on multiple platforms.The engine primary components are rendering, loading, animation, collision detection between objects, physics, inputs, GUI (Graphic User Interface) and AI (Artificial Intelligence).Besides, the game engine has other tools that make the actual game: characters and terrains, real world object behaviors, etc. (Paul et al. 2012)

Process to develop a VR app
The process of creating a virtual world requires more time than for one AR app as a complete environment must be created by the computer.
The process followed in this paper is divided into three phases: 1) Modelling and texturing the virtual element; 2) Environment creation; and 3) Interactive scripting.Unity was chosen to carry out these phases due to its compatibility with multiple platforms.Unity is a crossplatform used to develop games and interactive worlds in 3D and 2D for PC, consoles, mobile devices, VR and websites.
The main features of the Unity game engine are: -Dedicated tools for creating 2D and 3D content.
-Importation of models and animations made with other 3D applications such as Blender, Maya, 3DS Max and Cinema 4D.
-Easy programming through a variety of scripting languages and a comprehensive documentation.
-The inclusion of a physics engine which enables users to create a convincing physical behaviour.In this way, the objects in the scene will be affected by collisions, gravity and other forces.
Although Unity is focussed in the design and development of video games, it is a complete tool to create interactive 3D environments and smartphone apps.

Modelling and texturing the virtual elements
Any application that requires showing a real environment as a virtual tour, involves the generation of a realistic 3D model.Typical examples are ruins reconstruction (Vlahakis et al. 2002), the museum interior (Fineschi and Pozzebon 2015;Miyashita et al. 2008), an interesting cultural monuments or shelters rock art paintings.In order to create an accurate and detailed 3D model, it is recommended to use photogrammetry or terrestrial laser scanning techniques (Pavlidis et al. 2007).
Photogrammetry enables accurate measurements and 2D and 3D reconstructions of all types of existing objects using images at different scales.Through structure from motion (SfM) algorithms, it is possible to obtain accurate 3D models quickly and easily.Only overlapping images from multiple perspectives are needed and dense 3D point clouds with real texture are generated automatically in a few minutes (Furukawa and Ponce 2010;Micheletti et al. 2015).
Laser scanning is an efficient, high-precision and ultrafast active remote sensing technique used to acquire dense 3D point clouds (Lerma et al. 2008).These instruments can generate quickly vast amounts of 3D data.The usage of laser scanner is justified mainly for objects, monuments and sites with complex geometry.
Both photogrammetry and laser scanning techniques can be used to generate dense 3D point clouds from which real texture can be draped.In order to use these data in apps, it is necessary to convert the point clouds into more practical triangular meshes.This is possible with commercial as well as with open source software.
Another possibility to generate a simple 3D model is through CAD tools.CAD software is recommended in order to create simple 3D models since planar features are requested for simplified 3D modelling.The planes are imported into the software and all surfaces must be converted to solid objects as far as the extrusion command is available.Finally, a texture must be added to the surface of the object.

Environment creation
Once the 3D model is generated it should be placed in a real environment to add more realism.Unity includes several tools for this purpose such as terrains on which positioning the object, different lights to illuminate the scene, skyboxes and atmospheric effects.

Interactive scripting
In order to create an interactive scene, Unity provides programing scripts in either C#, JavaScript or Boo.In this phase, physical properties should be added to the objects with Unity physics engine tools.The most important is the "Mesh Colliders".This property ensures the player will not be able to walk through the created walls.Besides, the interaction with the elements of the scene (e.g.opening a door, moving an object...) must be developed through scripts.

Case studies
Two simple apps are developed in order to test both AR and VR technologies.In example 1, an AR app is developed to show an augmented globe.The goal is to know the steps to perform a complete AR app that might be expanded to disseminate rock art.In example 2, an interactive app is developed with the aim to show through a virtual tour, rock art paintings located in areas with limited access.

Example 1: The Augmented Globe
The application was developed using Vuforia SDK for Unity.The process followed to create an AR app is very simple.First, a frame marker is added to the scene.The app will use it to place the elements of AR.Then, the virtual elements (augmented elements) are added.On this application is used a simple object such as a globe.
The object geometry is very simple (a sphere); it can be created in Unity.Creating complex models with other image-based, range-based or CAD software and import the results into Unity is also possible.In order to improve the realism effect on the object, a texture is projected onto it.Finally, to visualize the object, a light is added to the scene.Furthermore, a simple script is developed to make the sphere turning around.With these steps, it is possible to visualise an object through smartphone camera pointing towards a frame marker (Fig. 2a) and then the augmented globe will be displayed (Fig 2b)  Using Vuforia SDK and Unity software, the steps to develop AR apps are pretty simple.This AR app can be adapted to another project after changing the virtual object to be shown.
In the rock art field, an app of this type can be used to teach and disseminate rock art paintings.This simple example can be used to create an application that, instead of displaying a 3D object, displays e.g.2D images of rock art motifs.Through a smartphone camera, the rock art motifs can be projected onto a wall in real size.This app makes possible to show in a comfortable way, rock art paintings that cannot be visited, or that are hardly visible.

Example 2: Interactive visit to a rock art shelter
Many interesting rock art paintings are located in rugged mountains and access to the public is either difficult or limited.In other cases, the rock art sites are closed to the public for protection and conservation, and the visitors are not allowed to visit them.For this reason, the aim of this example is to create an app where the user can make a complete virtual visit to a site.
For this example were tested two different frameworks: Unity game engine and min3d.Each can be used for a different purpose as already explained.In both cases, the first part of the process involves obtaining the 3D model.
The 3D model of a shelter was obtained from multiple images by means of SfM approaches.A point cloud of low density was created (Fig. 3a).Obtaining the 3D model using CAD tools is impossible due to complexity of its shape.On the other hand, the laser scanning is ruled out as it generates point clouds too big for a mobile application.An important limitation in this type of application is the generated 3D model size.Most of the times, it will be necessary to simplify the 3D model to reduce its size.
The triangular mesh has been created using MeshLab.
First normal vectors associated with each vertex are determined.Using the original point cloud with the computed normals, a surface reconstruction at the highest resolution is created.Finally, the texture from the colour images is generated after draping the images onto the 3D model.Following this way, a photorealistic 3D model of great detail and with texture is achieved (Fig. 3b).
The 3D model is imported to Unity to create a surrounding environment.The shelter is located on a rugged terrain on the hillside of a mountain.In Unity, with terrain tool, a similar terrain is created and the scene is illuminated with some lights.Finally, to give greater realism to the scene a sky is added through the skybox tool.The skybox is a complete wrapper around the scene which can be edited for changing the colour, exposure, adding clouds, etc.
The user has to move around the scene for the virtual tour.Unity has a standard asset (3D object motion script) for first person characters.Once added the first character into the scene, a person is be able to move in all directions within the project.It is possible to obtain the stereoscopic vision using the Oculus Rift SDK for Unity.Since not everyone has virtual reality glasses, a version for smartphones has been created through mind3d framework.In this version, the user will see on the smartphone screen the shelter 3D model and the rock art paintings.Besides the user is able to move in all directions and zoom with touching gestures.Figure 4 shows a app screenshot where the photorealistic model is visualised.The icon home returns home, and two icons allow users to zoom in/out.Additional icons are presented in other scenes to localise and how to reach the site, to get historical values as well as neat depictions of the rock art paintings.

Conclusions
This paper reviews the latest technologies for 3D models visualisation.These technologies provide the user real and enriching experiences for the visualisation of 3D objects.Besides, the amount of possible fields that can benefit from these technologies is very great.
Specifically, these technologies have been applied in the rock art field.
In future research, the user response to this type of applications will be discussed.These apps, particularly those involving the use of a VR glasses, can cause discomfort and inconvenience to some users.Therefore, some users may either prefer AR apps or display 3D models using the mobile screen.
On the other hand, the creation of photorealistic 3D model is just the first step.Creating complex and/or very large models can be approached with image-based photogrammetric techniques.Nevertheless, the point clouds obtained are so big that the hardware of a mobile phone does not support them.Therefore, optimisation of 3D models is crucial to create photorealistic models for smartphone apps.Enriching the apps with additional user friendly tools is a step that will be developed in the new future.

Figure 2 :
Figure 2: Smartphone screenshots: a) screenshot before the augmented globe; b) screenshot with the augmented globe.

Figure 3 :
Figure 3: The interior of a shelter: a) 3D point cloud; b) photorealistic 3D model.

Table
-BeyondAR is an open source AR framework based on mobile location for Android (BeyondAR 2016).-DroidAR is an open source AR framework for Android only, it features location based AR and recognition markers (DroidAR 2016).-Mixare is an open-source AR engine for Android and iPhone, which displays Wikipedia points of interest (POIs) of the surroundings and lets you add your own POIs database (Mixare 2016).

Table 1 :
Summary of AR frameworks.

Table 2 :
Summary of game engines.