Testing and reviewing a lens

 

Lens testing: focusing in manual focus with live view at 10x.
 
Testing a lens can be fun an informative, but it is not that easy: you have to be very careful if you want to get reliable results, and you need a lot of patience. In this article, I have explained how I review a lens, why I give more importance to some aspects rather than others. This page is useful both to understand better my reviews, and to help you if you want to do your own tests!  
 
Even though the basis of my review process are always the same, in these years I have made many small improvements to make the reviews more precise, useful and informative. The information in this page describes the review system I use from late 2009.
 
 

Specifications

Specs are useful to get an overview of the lens. Some notes: there are mainly two types of AF, the "Ultrasonic AF motor" (very fast and silent) and the old "AF motor", that is slower and it does not have full time manual focus. In the comparison table, I always list the type of AF actually used by the lens, regardless of the manufacturer's claim: many manufacturers are not very honest about this aspect, and they call "Ultrasonic" even lenses that use the cheap "micro-USM" AF motor, that is identical to the old non-USM AF motor. For example, the Canon 300mm f/4 L IS USM has the true ultrasonic AF motor, while the Canon 70-300mm f/4-5.6 IS USM has the cheap non-USM AF motor.  
 
The production field lists the year of introduction of the lens and, in case of lens that are out of the market, the year of discontinuation. The date can be useful if you are evaluating the purchase of the lens, because old lenses are more likely to be replaced soon by a new version than recent lenses. The prices are in United States dollars and they are based on the B&H prices (www.bhphotovideo.com).
 
 

Description (built quality, autofocus and stabilization)

In this paragraph I describe the main aspect of the lens, and I compare its features and built quality with other similar lenses. I don't do in-depth tests of autofocus or image stabilization because it is nearly impossible to make a "scientific" evaluation of these aspects; my consideration are based on my experience with these lenses. (when I testing some lenses, I try to use them as much as possible, to get an idea of their AF and IS).
 
 

The postage stamps test: Sharpness, contrast, chromatic aberration

Sharpness is one of the most important aspect of image quality, for this reason I carefully test sharpness (and contrast) of every lens I test, and I publish 100% crops. These crops are useful even to evaluate chromatic aberration, too, but this is not as important: CA is one of those things you can correct with PS, while you can't do much for sharpness. (sharpening helps a bit, but it won't turn you 70-300 in a 300 2.8!)  
 
To test the sharpness, I use a 24x36 centimeters test target covered with postage stamps that have fine detail (for long lenses), while for wide-angles I prefer to use a much wider test target, that is about 3 x 4.5 meters. It does not make sense to test a wide angle with a small test target because the majority of times you will use it for wide views, so I think a 3 x 4.5 meters target provides a more accurate result. Standard zooms and tele instead are often used even for relatively small subjects (portraits, birds) so an approximately A4 test target is ok.  
 
Of course the test target must be flat and it must be perfectly parallel to the sensor! I focus carefully using live view and 10x magnification, and I take the photos using a stable tripod, mirror lock up and self timer, to avoid any vibration that may invalidate the results.  
 
I always repeat this test at least two times, to be sure there are not focus errors or other imprecisions. Currently I use two cameras for the tests: a Canon 1Ds III (fullframe, 21 megapixels) to review the majority of FF lenses except teles, and the Canon 7D (APS-C, 18 megapixels) to review APS-C lenses and teles. I prefer to use the 7D for long lenses because the majority of times these lenses will be used on a crop body, and usually when you take a photo with a tele the corner performance does not matter much, while the sharpness in the center is important.

In the majority of reviews, I compare the lens with other similar lens: it is very useful, because it allows to see how the lens compares with other lenses that have similar specification, so you can evaluate the alternatives. Other than that, many times it allows to see if a pro lens is really worth the money or if it is overpriced.
 
 

Distortion and vignetting

Sometimes (for wide-angles) I add a test of distortion, but personally I don't give much importance to this aspect, you can easily correct it with Photoshop. The same for vignetting: it is easily correctable with PS, so I don't give much importance to this aspect.
 
 

Flare

Flare is one of those things you can not correct with PS, so it may be important in the evaluation of a lens. While for teles it is never a problem, with standard zooms and wide-angles it can make a big difference, in particular if you take often photos that include the sun. That said, it is very difficult to make a good, scientific test of flare, so generally I don't include a test of flare in the reviews.

 

Sometimes my desk can get pretty busy :-)
 

The image samples

I always include two or more image samples in every lens review. You can download either the untouched photo (JPEG converted from RAW without any additional post processing; minimum contrast and saturation, no sharpening, no AC, distortion or vignetting correction) or the post processed version. The untouched photo is a good way to see the real image quality of the lens, and the post processed versions allow to see the final quality you can get with good post processing techniques.  
 
In samples taken with a wide aperture (f/1.4, f/2.8), it is very important to remember that the depth of field is extremely limited. Don't judge the corner sharpness in the wide open samples! The corners are likely to be very soft, but this softness may be due to lack of depth of field, not to poor image quality. If you want to see the image quality in the corners, look at the studio tests with the postage stamps, or at the stopped down samples. The wide open samples are provided mainly for judging the image quality in the in-focus areas and to give you an idea of the depth of field at these apertures.
 
 

Additional tests or considerations

Some lenses have peculiar characteristics that may not appear in the standard tests I do; for this reason, sometimes I include some more tests. For example, in the fisheyes review I have included a comparison of the angle of view; in the Canon 85mm f/1.2 review I have added some considerations about the difficulties of using this lens in the field, and so on.
 
 

Lens variation

While professional lenses tend to have always the same quality, sometimes in semi-pro or cheap lenses there may be noticeable differences between two samples of the same lenses. It does not happen often, in particular with Canon or Nikon lenses, while with other manufacturers it is sadly common. This means that your copy of the lens may be better or worse than the lens I have tested...there is not much to do about that, but when I test a lens that is know for its sample variation (for example the Sigma 12-24mm) I write it in the review so you know that you have to take the results with a grain of salt.
 
 

Conclusions

In the conclusion I write my opinion on the lens, based on the results of the tests and on other things (price, similar lenses, features). I don't pretend to say the last word on any lens: I try to be accurate and objective, but at the end you have to make your conclusions...the review provides plenty of tests and info so you can judge by yourself and you can evaluate if the lens can be a good choice for you! For example, I like to use long lenses for macro work, while other photographers prefer to use 60 or 100mm lenses: there is not a right or wrong, it depends by your personal tastes.
 
 

Lens sources and links

You may be wondering where I get all these lenses... I don't own them! I am not that rich and anyway I'd never buy so many lenses, I'd spend all the money for a BMW X6 or a Lamborghini Aventador :-) Nearly all the lenses I review had been loaned to me for one or two weeks by manufacturers, shops or friends. Currently I get the majority of lenses from friends and from DigitalFoto (an Italian camera dealer).  
 
Of course I am not paid by anyone for lens reviews.  
 
If you want to read other good lens reviews, I recommend these websites: www.the-digital-picture.com, www.photozone.de, www.dpreview.com.