Assignment 1, Computer Vision, 2017 Due: 11am, 17/4/17 Anton van den Hengel The University of Adelaide South Australia [email protected] Abstract The first assignment is to implement, describe, and test a method for creating mosaics from a set of smaller images. The submission will take the form of a conference paper. 1. Introduction There are a variety of situations in which stitching a set of images together into a single larger image is of interest, but one of the main motivations is the desire to be able to represent a wider field of view than is achievable using a normal camera. The image in Figure 1, for instance spans a horizontal angle of well over 180 degrees, which is difficult to achieve with a standard camera. The panoramas generated from images taken by the Mars rovers are a very well publicised example (see[4] for example). Figure 2 shows another mosaic which aims to represent more than could be captured in a single image, but this time by allowing significant deviation from the normal projection processes. The are a variety of other reasons for mosaicing images, however, such as the Photo Cubism approach developed by David Hockney1. Figures 3 4 show two of David Hockney's 'Joiners', which were constructed by hand from Polaroid images. 2. The method The method you will need to implement is based on that of Brown and Lowe, presented in [2]. The approach presented there has become one of the baseline methods in the area. You will need to understand the method in order to be able to write about it. The primary task in this assignment is to analyse the Brown and Lowe method. That means that you need to understand it, to come up with a hypothesis as to a limitation of the method, or a means of improving it. You then need 1See http://www.hockneypictures.com to devise a set of experiments which will test your hypothesis, and a means of reporting on the outcome of these tests. Critically, you need to analyse the outcome of your tests, in order to draw a useful conclusion. Whether or not your hypothesis turns out to be correct is a secondary consideration, what is important is that it shows insight, and that you have devised a sensible set of tests by which to evaluate it. You do not need to extend the method of Brown and Lowe, but it will improve your marks if you do. If you decide to extend their method then you need to describe in your paper why you chose your particular extension in addition to describing the details of it. The extension you have devised should thus be described in terms of the shortcoming in the original method that it addresses. You should then document your assessment of whether it has succeeded in doing so. Whether or not your improvement actually works better than the original method is thus a secondary consideration, the main issue is for you to be able to draw a sensible conclusion about it. In analysing the performance of the method it is not acceptable to simply present a set of examples and hope that the reader will do the analysis for you. I would suggest devising a metric test of performance, but if this is not possible then you need to devise another method for illustrating the effect that you wish to demonstrate. Sometimes, for instance, it is possible to devise a particularly informative set of test cases such that the deficiency in the method becomes obvious. The goal of the assignments is that you develop your ability to analyse methods, and to report on the outcome of your analysis. Every CVPR paper presents a hypothesis (typically that the proposed method is better than any of its competitors) and sets out to prove it. I would recommend reading a few to get an impression of the techniques used. 3Source: http://digitalimagingandphotography. blogspot.com.au/2011/07/cathedral-interior.html 1Figure 1. A panorama taken from the top of the Wren Library in Trinity College, Cambridge, by Anton van den Hengel Figure 2. A mosaic with an extremely skewed (and non-smooth) mapping from the original scene onto the image plane, but which captures more of the scene than would normally be visible in a photo.3 2.1. Implementation In order to be able to test the method you will need to implement it. Fortunately this is made significantly easier by the fact there are many existing implementations available. The OpenCV[1] version[5] is a good place to start, but there are others. If you use the OpenCV version it will simplify the process of making any changes you wish to implement, as OpenCV provides a variety of other tools and implementations that will help. The choice of language, platform, compiler, IDE, and similar is up to you. You are welcome to discuss these decisions on the forum. 3. Submission The submission takes the form of a conference paper, and your code. 3.1. Paper submission You need to write a paper such as might be submitted to a conference, and specifically a paper such as might be submitted to CVPR[3], which is one of the best conferences in Computer Vision. The paper must be in the CVPR format, and submitted as a pdf document. By far the easiest way to achieve that is to use L ATEX. L ATEXis a very powerful document formatting package, it's free, and it is the only way to generate well formatted documents that contain maths. It's also the easiest way to generate well formatted documents in general. All the information about the CVPR paper format is available on their web site[3]. The paper must be all your own work, with no text copied from any other document. I do, however, suggest that you study Brown and Lowe's paper[2], and some of the papers that have followed it, in order to see how such papers are written, and some of the methods used to assess the performance of these methods. You can find the papers that have sited Brown and Lowe's paper using Google Scholar. The purpose of the paper is to demonstrate that you understand the problem, and the solution. This means that your submission should have sections which broadly cover the following • An introduction, which describes the problem, and the method • A background section which describes competing approaches to the problem. Achieving this requires that you understand what the competing approaches do, how they do it, their advantages and shortcomings, and how they compare to the current approach. The methods you compare against here may well perform better than the method you are describing. The idea of this section is not that you show that yours is necessarily the best method available, but rather that you show that you understand enough about the literature in the area to be able to put it in context. • A description of your hypothesis. This will typically require explaining some part of the algorithm in detail, and providing examples illustrating its effects and deficiencies. If you propose an improvement then you should describe how your method works, in enough detail that a reasonably skilled person would be able to implement it. • Experimental Analysis. Describe the tests you have run, and your motivation for having run them. Report the results of the tests and the conclusions that you have drawn. Again, the goal is not to show that 2your method outperforms all comparators, but rather that you understand what your method aims to achieve, and can devise, execute, and report upon a set of tests which demonstrate whether it does so. If you have improved upon the base method then you have an opportunity here to show that your improvement is well motivated, and possibly even that it works. • Conclusion. Demonstrate that you have learned something worthwhile from the process, possibly including ideas about what you might do to improve the method you are reporting on. If you can come up with an interesting application, or evaluation of the method, then so much the better. The paper you submit must be in the format specified for CVPR 2017, which is specified as part of the author instructions[3]. The easiest way to achieve this is to download the L ATEXtemplate and use that. You can use some other means if you really want to, but your paper needs to conform to the CVPR style specification. The only exception is that I don't mind if you use a4 paper rather than their preference for letter paper (it's a US conference). 3.2. Code You will also need to submit your code through the svn server, but I will not be marking the quality of your code, only checking that it shows enough evidence that you wrote it yourself. 4. Assessment Your submission will be assessed primary in terms of how well it demonstrates that you • understand the problem, • understand Brown and Lowe's solution, and where it sits with respect to the relevant literature • have come up with a sensible hypothesis • can design and implement a suitable set of tests which will demonstrate (or disprove), • can interpret the results of these tests, and draw a sensible conclusion. 5. Conclusion If you have questions, please ask them on the forum, as they are likely to be of interest to others also. It is important to check the forum for the latest information about the assignment too. Figure 3. David Hockney pioneered a form of photography that he called 'Joiners', in which he manually combined multiple images into a larger piece. One of the goals of some of this work was to provide a single view of an object or scene which would describe its entire appearance. This is similar to one of the goals of Cubism, pioneered by Pablo Picasso, amongst others. Photo source: http://mjonesphotog.blogspot.com.au/2012/11/hockneysjoiners-and-cubism.html References [1] G. Bradski. The OpenCV library. Dr. Dobb's Journal of Software Tools, 2000. [2] M. Brown and D. G. Lowe. Automatic panoramic image stitching using invariant features. International Journal of Computer Vision, 74(1):59–73, 2007. [3] CVPR. Ieee computer society conference on computer vision and pattern recognition. See http://cvpr2017. thecvf.com/. [4] NASA. Billion-pixel view from curiosity at rocknest, 2012. http://mars.nasa.gov/multimedia/ interactives/billionpixel/. [5] OpenCV. The stitching pipeline. http://docs.opencv. org/modules/stitching/doc/introduction. html. 3Figure 4. A somewhat cubist Joiner by David Hockney (Source: http://thedelightsofseeing.blogspot.com.au/2011/03/cubismjoiners-and-multiple-viewpoint.html) 4