MLF on aerial filming in 360°

From the beginning of MLF’s collaboration on Project Daedalus, they proposed to explore aerial 360° content, potentially live-streamed, for virtual reality. The aim of this combination of technologies (UAVs & 360º content) is to create an  out-of body experience where the user feels completely immersed in the viewpoint of the drone.

We share their research below for those creators wishing to experiment with similar techniques. This post is quite technical, and will need you to have some knowledge of film making for you to get the most out of it.

We conducted initial research (see download) in order to understand the  variety of technical capabilities of existing 360º  cameras. This helped us frame the best approach for filming aerially in 360º. Our  research was specifically being developed to explore content creation for virtual reality (VR) headsets – in order to give audiences anywhere, anytime, the ability to experience aerial perspectives in immersive ways.  Considerations were the resolution of cameras, weight (affects payload & flight-time of drones) and the camera’s field of view.

The first stage of working out which 360 model to pursue was to conduct desk based research.

Desk-based summary:

  • The simplest approach for creating a  3D VR experience (at the time of conducting research) is using a 180º camera setup. This means that no stitching needed, stitching means connecting the edges of different films together to create one seamless whole. To do this the films being taken need to be perfectly synchronised (taken at exactly the same time) and have very good depth perception. However, whilst this is working simply, it is only 180º, and we want 360º.
  • We have learnt that viewers don’t see 3D objects past 5 metres or so; therefore 3D is not necessarily important for aerial footage as the subjects filmed are likely beyond 5m. Tech note: Due to stereoscopic parallax flattening out at that distance.
  • As in 180, image syncing all cameras is essential in order to stitch a 360º image in post production. Tech note: A CCD (charge-coupled device) or CMOS (complementary metal-oxide semiconductor – a battery-powered memory chip) with global shutter is preferable but gets very expensive very quickly.
  • The use of mirrors is the seemingly the only way to get perfect stitch but not feasible for small setups due to expensive optics.
  • There are more and more 360º / panoramic stitching softwares available.
  • Resolution: The Theta approach (Ricoh’s two lens 360º camera) actually produces great results, but resolution is very low.
  • The Blackmagic Micro Studio camera is still much bigger than a GoPro (& therefore heavier and needs a larger copter to fly a rig) & not released until July 2015 (post r&d phase) – otherwise we would have used it for testing.
  • GoPro cameras are lightweight and allow lens adaptation for shooting wider angles.
  • GoPro have not yet released a sync kit (a sync kit allows for image synchronisation) for the Hero4 their higher resolution camera. When they do there will be many options for higher resolution 360º setups. Currently you can only sync using the Hero 3+.
  • To get perfect sync with multiple GoPro’s, right now, you have to use the Hero 3+.

Deciding which cameras to use:

Finding which camera to use was a process of elimination. We decided to use a GoPro, but had to make a decision between which model to use, the older Hero3+ and more recent Hero4. Right now (time of research March-May 2015), the only GoPro you can genlock is the Hero3+ based on the same technology as GoPro’s stereo kit which is only compatible with the Hero3+. GoPro have said they are working on a solution for the Hero4 (in form of a firmware update) but haven’t said when that will be released (at July 2015, this still hasn’t been released).

The downside with the Hero3+ is reduced resolution compares to the 4, so if you simply want the highest resolution possible then the Freedom360 mount with six Hero4’s would be best off-the-shelf solution. However, when filming from a drone and therefore constantly moving sync is important. Otherwise each camera might record a slightly different moment in time per frame and you will get shifts in the panorama. This will lead to low quality footage & a substandard immersive experience for the audience.

There are ways to reduce this effect in the Hero 4s without a sync cable – by using a higher frame rate (80fps vs 48fps).  However then you are using the same resolution as on the Hero3+ camera but without the ability to genlock the cameras & get a perfect sync.

The only way to take advantage of the Hero4’s higher resolution is to shoot 2.7k 4:3 at 30fps which will be hard to sync, you’ll have to a). switch on all camera’s at the same time using a script (to sync the internal clocks), b). start capture via the wifi remote and c). sync the video in post, but results won’t be perfect (15ms apart at worst) also, sync will shift over time so only works for relatively short durations (depends on the batch of GoPros you get as well).

So, we decided to use the GoPro Hero 3+ (black) cameras, a lower resolution but with the potential ability to achieve a perfect sync.

 

Sample video:

Test shoot with Hero3 Set-up

Our first test shoot was windy and the rig was swinging around a lot.  We successfully shot a small amount of footage, suitable to be stitched together. The image above is a still taken from the 360º video first shot by this camera.  You can see the aluminium rod we attached the camera with, this was not a suitable length as it creates too much movement. In post production we found that software stabilisation in Kolor seems very good, but strong movements will show up in the blended areas if exposure difference is too much.

Next Steps Post Test Shoot:

  • Test & build a few more mounting solutions
  • Build a gimbal
  • Add some extra weight half­way down the mounting pole on a cross which acts like two tightrope poles.

gimbal

Next Test Rig Details

Rig 2: 3 x Go Pro Hero 3+ (black)

Spec: 3 x Hero3+ shooting up (or down) on wide angle lens (194º lens on each camera) + one potential additional camera to cover top (get rid of drone from footage).

Resolution per cam: 1440p

Lens: 3-4 x 194º

Mount: Custom

Genlock? Yes

3D printed custom mount allows space for the arduino boards (which off the shelf options don’t).

Initial designs for the custom mount wanted to ensure a range of options for clustering the cameras:

Learning From Second Rig

We built the 3 camera rig pictured above, to explore the potential of hacking them and genlocking the 3 cameras. This would in allow us to sync the image capture and therefore have a much smaller task of stitching in post production. We knew this was possible with 2 GoPro’s, you use this kind of Hero 3 dual hero sync cable as a genlock generator.  And, we also know you can get hold of a genlock dongle.

After testing it we found that the 3 camera rig shows potential in terms of footage, with the lenses you can get a decent 360º image with enough overlap for a good stitch. If you add a 4th camera on top (something we have tried once) you also get enough overlap cover the drone in the film.

The genlock works well in terms of image sync, and fortuitously locks the exposure of the film too.  However, locking the exposure also means you can’t control the exposure. So conditions have to be perfect – on all cameras – on the auto exposure to not get issues from camera to camera (i.e. depending on the direction of the sun). Or you need to use neutral density lenses of an appropriate level for the scenes & conditions you are filming in.

Future Direction:

We are now testing a 7 camera GoPro Hero 3+ (black), with a custom mount, genlocked, with 194º lenses. A custom mount allows space for the arduino boards (which off the shelf options don’t) for the sync hack.  The extra camera (up from 6) allows full 360º with the lenses giving a wider field of view and therefore more overlap between the images & the ability to cut out the drone from the footage. It is being tested on DJI S900 drone. The weight of the 7 cameras plus mount means only the larger drones have the capable payload.

This video shows the 7 cameras in genlock (test)

7 cameras genlock

This is a first test of the rig being driven on a car (not 360º enabled video).  This is 6 cameras in genlock with exposure lock and ND filters .

 

Stitching in 360°

We tested the two main software packages used for 360 production, VideoStitch and Autopano by Kolor.

VideoStitch is very easy to use and most functions are automatic but motion based video synchronisation was very poor and didn’t work at all.  It automatically stitches the video without any extra software but if you want to tweak you need a license for PTgui Pro: https://www.ptgui.com/

Kolor workflow is very similar and requires two pieces of software, Autopano Giga for stitching and Autopano Video Pro for the rest.  They also have cheaper versions with limitations, i.e. no GPU support.

Both were pretty fast during rendering on my laptop so should be a breeze with a Titan.  Functionality is almost identical for both but Autopano feels more polished and was spot on at synchronising the videos even without any markers.  I’m working with trial versions atm.  Colour have a bundle for around £650.

Open software option:

https://en.wikipedia.org/wiki/Hugin_(software)

http://hugin.sourceforge.net/

Next page in toolkit: MLF on tracking and control with Piksi