Home | Alpha Telephone | Domain Names | Web Hosting | Get Traffic | xrEvidence | xrSoccer

United States Patent

Previous       Show 10       Next


United States Patent 6,429,895
Onuki August 6, 2002

Image sensing apparatus and method capable of merging function for obtaining high-precision image by synthesizing images and image stabilization function


Abstract

An image sensing apparatus, which performs pixel shifting operation for shifting an image formation position of an image on an image sensing device using a lens capable shifting a light path and senses images at respective image formation positions, has a vibration sensor for detecting vibration. The lens is moved so as to cancel the vibration detected by the vibration sensor when performing pixel shifting operation. Since the shifting characteristics of the lens slightly changes depending upon states of zooming and focusing lenses, the lens is moved by a shift amount which is adjusted in accordance with the states of the zooming and focusing lens.


Inventors: Onuki; Ichiro (Kawasaki, JP)
Assignee: Canon Kabushiki Kaisha (Tokyo, JP)
Appl. No.: 08/996,287
Filed: December 22, 1997

Foreign Application Priority Data

Dec 27, 1996 [JP] 8-349310
Dec 27, 1996 [JP] 8-349311

Current U.S. Class: 348/208.99
Field of Search: 348/207,218,219,200,222,208,358,335 382/284,299,312 396/54,55


References Cited

U.S. Patent Documents
5754226 May 1998 Yamada et al.
5889553 March 1999 Kino et al.
5930405 July 1999 Chida
6018363 January 2000 Horii
6130709 October 2000 Sekine et al.
Primary Examiner: Ho; Tuan
Attorney, Agent or Firm: Morgan & Finnegan, LLP

Claims



What is claimed is:

1. An image sensing apparatus comprising: image sensing means for converting an optical image into electric signals and outputting the electric signals as image signals; shifting means for shifting an image formation position of the optical image formed on said image sensing means to a plurality of different positions; first vibration detection means for detecting vibration of the image sensing apparatus and outputting vibration information; optical system state detection means for detecting a state of at least one of a zooming optical system and a focusing optical system, control means for controlling said shifting means on the basis of the vibration information outputted by said first vibration detection means, said control means comprises: pixel shifting control data generation means for generating pixel shifting control data for controlling said shifting means in pixel shifting operation; vibration compensation data generation means for generating vibration compensation data for compensating the vibration on the basis of the vibration information outputted by said first vibration detection means; control data synthesis means for synthesizing the pixel shifting control data and the vibration compensation data to generate a control signal for controlling said shifting means; data correction means for correcting at least one of the pixel shifting control data and the vibration compensation data in accordance with the state detected by said optical state detection means; image signal synthesis means for synthesizing image signals of a plurality of images outputted by said image sensing means to generate a single image; and storage means for storing first correction data for correcting the pixel shifting control data and second correction data for correcting the vibration compensation data, wherein said image sensing means converts the optical image into electric signals at each of the plurality of different image formation positions shifted by said shifting means and said data correction means reads at least one of the first and second correction data, corresponding to the state detected by said optical state detection means, stored in said storage means, and calculates the pixel shifting control data or the vibration compensation data on the basis of at least one of the read first and second data.

2. The image sensing apparatus according to claim 1, wherein said shifting means is a movable optical means.

3. The image sensing apparatus according to claim 1, wherein said shifting means is a plane parallel plate.

4. The image sensing apparatus according to claim 1, wherein said shifting means is a variable apical angle prism.

5. The image sensing apparatus according to claim 1, wherein said control means comprises change-over means for changing between a plurality of pixel shifting modes, and pixel shifting control data generation means for generating pixel shifting control data for controlling said shifting means in pixel shifting operation, wherein said change-over means changes the pixel shifting modes on the basis of the vibration information outputted by said first vibration detection means.

6. The image sensing apparatus according to claim 5, wherein the pixel shifting modes indicate number of the image formation positions to be shifted in pixel shifting operation, and said change-over means selects a first number of image formation positions when a degree of vibration indicated by the vibration information is equal or less than a first predetermined value; and selects a second number of image formation positions which is smaller than the first number when the degree of vibration is larger than the first predetermined value.

7. The image sensing apparatus according to claim 6, wherein said change-over means selects a third number of image formation positions which is smaller than the second number when the degree of vibration is larger than the second predetermined value which is larger than the first predetermined value.

8. The image sensing apparatus according to claim 5, further comprising optical system state detection means for detecting a state of at least one of a zooming optical system and a focusing optical system, wherein said control means further comprises data correction means for correcting the pixel shifting control data in accordance with the state detected by said optical state detection means.

9. The image sensing apparatus according to claim 8 further comprising storage means for storing correction data for correcting the pixel shifting control data, wherein said data correction means reads the correction data, corresponding to the state detected by said optical state detection means, stored in said storage means, and corrects the pixel shifting control data on the basis of the read correction data.

10. The image sensing apparatus according to claim 1, wherein said control means comprises change-over means for changing between a plurality of pixel shifting modes, and pixel shifting control data generation means for generating pixel shifting control data for controlling said shifting means in pixel shifting operation, wherein said change-over means changes between the pixel shifting modes in accordance with a user designation.

11. The image sensing apparatus according to claim 10 further comprising optical system state detection means for detecting a state of at least one of a zooming optical system and a focusing optical system, wherein said control means further comprises data correction means for correcting the pixel shifting control data in accordance with the state detected by said optical state detection means.

12. The image sensing apparatus according to claim 11 further comprising storage means for storing correction data for correcting the pixel shifting control data, wherein said data correction means reads the correction data, corresponding to the state detected by said optical state detection means, stored in said storage means, and corrects the pixel shifting control data on the basis of the read correction data.

13. The image sensing apparatus according to claim 1 further comprising second vibration detection means for detecting blurring in an image on the basis of the image signals outputted by said image sensing means and outputting blurring information, wherein said control means controls said shifting means on the basis of the vibration information outputted by said first vibration detection means and the blurring information outputted by said second vibration detection means.

14. The image sensing apparatus according to claim 13, wherein said first vibration detection means is a vibration-type gyro and said second vibration detection means is movement vector detection means.

15. The image sensing apparatus according to claim 1 further comprising photometry means and luminous exposure adjustment means for determining a luminous exposure on the basis of a result of the photometry performed by said photometry means and controlling said image sensing means to use the determined luminous exposure.

16. The image sensing apparatus according to claim 15, wherein, when the image formation position is shifted by said shifting means, said luminous exposure adjustment means determines a different luminous exposure and controls said image sensing means to use the different luminous exposure.

17. The image sensing apparatus according to claim 16, wherein said photometry means performs photometry on each of a plurality of divided areas of an image sensed by said image sensing means.

18. The image sensing apparatus according to claim 17, further comprising main object determination means for determining a divided area which includes a main object among the plurality of divided areas, wherein said luminous exposure adjustment means controls said image sensing means to use a first luminous exposure which is suitable for the divided area including the main object, which is determined by said main object determination means, and to use at least one of second and third luminous exposures where the second luminous exposure is larger than the first luminous exposure and the third exposure is smaller than the first luminous exposure.

19. The image sensing apparatus according to claim 18, wherein said control means comprises change-over means for changing between a plurality of pixel shifting modes, and pixel shifting control data generation means for generating pixel shifting control data for controlling said shifting means in pixel shifting operation, wherein said change-over means changes the pixel shifting modes on the basis of the vibration information outputted by said first vibration detection means, and said luminous exposure adjustment means determines the luminous exposure to be used in said image sensing device on the basis of the pixel shifting mode selected by said change-over means.

20. The image sensing apparatus according to claim 19, wherein the pixel shifting modes indicate number of the image formation positions to be shifted in pixel shifting operation, and said change-over means selects a first number of image formation positions when a degree of vibration indicated by the vibration information is equal or less than a first predetermined value; selects a second number of image formation positions which is smaller than the first number when the degree of vibration is larger than the first predetermined value and equal or less than a second predetermined value which is larger than the first predetermined value; and selects a third number of image formation positions which is smaller than the second number when the degree of vibration is larger than the second predetermined value.

21. The image sensing apparatus according to claim 18, wherein said control means comprises change-over means for changing between a plurality of pixel shifting modes, and pixel shifting control data generation means for generating pixel shifting control data for controlling said shifting means in pixel shifting operation, wherein said change-over means changes between the pixel shifting modes in accordance with a user designation and said luminous exposure adjustment means determines the luminous exposure to be used in said image sensing device on the basis of the pixel shifting mode selected by said change-over means.

22. The image sensing apparatus according to claim 15, wherein said luminous exposure adjustment means adjusts the luminous exposure by changing exposure time.

23. The image sensing apparatus according to claim 15, wherein said luminous exposure adjustment means adjusts the luminous exposure by changing aperture of an iris diaphragm.

24. The image sensing apparatus according to claim 15, wherein said luminous exposure adjustment means adjusts the luminous exposure using an electrochromic element.

25. The image sensing apparatus according to claim 1 further comprising blurring detection means for detecting relative blurring amount between two images out of a plurality of images obtained at the plurality of image formation positions shifted by said shifting means by comparing image signals of the two images, and outputting the blurring amount.

26. The image sensing apparatus according to claim 25 wherein said blurring detection means detects the relative blurring amount by comparing the image signals of the two images which are obtained in series out of the plurality of images obtained at the plurality of image formation positions.

27. The image sensing apparatus according to claim 25, wherein said blurring detection means detects the relative blurring amount by comparing the image signals of the two images which are obtained at the same image formation position in series out of the plurality of images obtained at the plurality of image formation positions.

28. The image sensing apparatus according to claim 25 further comprising disabling means for disabling the image signal synthesis means in a case where the blurring amount detected by said blurring detection means is greater than a predetermined value.

29. The image sensing apparatus according to claim 28 further comprising notification means for notifying a user of image synthesis being disabled in a case where said disabling means disabled said image signal synthesis means.

30. The image sensing apparatus according to claim 28, wherein said image signal synthesis means has a plurality of image synthesis modes, and selects one of the plurality of image synthesis modes in accordance with the blurring amount.

31. The image sensing apparatus according to claim 30, wherein the image synthesis modes indicate numbers of images to be synthesized out of the plurality of images sensed by said image sensing means, and said image signal synthesis means selects a first number of images when the blurring amount is equal or less than a first predetermined value; and selects a second number of images which is smaller than the first number when the blurring amount is larger than the first predetermined value.

32. The image sensing apparatus according to claim 31, wherein said image signal synthesis means selects a third number of images which is smaller than the second number when the blurring amount is larger than the second predetermined value which is larger than the first predetermined value.

33. The image sensing apparatus according to claim 31 further comprising notification means for notifying a user of the image sensing mode selected by said image signal synthesis means.

34. The image sensing apparatus according to claim 25, wherein said image signal synthesis means synthesizes the image signals of the plurality of images on the basis of the blurring amount detected by said blurring detection means so that blurring is compensated.

35. The image sensing apparatus according to claim 34, wherein said image signal synthesis means synthesizes the plurality of images after changing relative positions between the plurality of images on the basis of the blurring amounts between respective pairs of the plurality of images.

36. An image sensing apparatus comprising: an image sensing device for converting an optical image into electronic signals and outputting the electric signals as image signals; a shifting unit for shifting an image formation position of the optical image formed on said image sensing device to a plurality of different positions; an optical system state detection unit for detecting a state of a focusing optical system; a control unit for controlling said shifting unit on the basis of the state of the focusing optical system detected by said optical system state detection unit; and an image signal synthesis unit for synthesizing image signals of a plurality of images outputted by said image sensing device to generate a single image; wherein said image sensing device converts the optical image into electric signals at each of the plurality of different image formation positions shifted by said shifting unit.

37. The apparatus according to claim 36, wherein said focusing optical system is a rear focusing system.

38. The apparatus according to claim 36, wherein said control unit controls a shift amount of an image formation position by said shifting unit.

39. The apparatus according to claim 36, wherein said shifting unit moves in a substantially vertical direction with respect to an optical axis.

40. The apparatus according to claim 36, wherein said shifting unit is a plane parallel plate.

41. The apparatus according to claim 36, wherein said shifting unit is a variable apical angle prism.

42. An image sensing method comprising: an optical system state detection step of detecting a state of a focusing optical system; an image sensing step of, while shifting an image formation position of an optical image on an image sensing device to a plurality of different positions by a shifting unit on the basis of the state of the focusing optical system detected in said optical system state detection step, converting the optical image into electronic signals and outputting the electric signals as image signals at each image formation position; and an image signal synthesis step of synthesizing image signals of a plurality of images outputted by said image sensing device to generate a single image.

43. The method according to claim 42, wherein said focusing optical system is a rear focusing system.

44. The method according to claim 42, wherein a shift amount of the shifting unit is controlled upon shifting the image formation position.

45. The method according to claim 42, wherein the shifting unit moves in a substantially vertical direction with respect to an optical axis.

46. The method according to claim 42, wherein the shifting unit is a plane parallel plate.

47. The method according to claim 42, wherein the shifting unit is a variable apical angle prism.
Description



BACKGROUND OF THE INVENTION

The present invention relates to an image sensing apparatus, having an image stabilization function, capable of obtaining a high-resolution image by slightly shifting an image formation position by an optical system, on an image sensing device for performing photo-electric conversion on the image, and an image synthesis apparatus for synthesizing images obtained by the image sensing apparatuses.

An electronic still camera which uses a solid-state image sensing device, such as a charge coupled device (CCD), instead of a silver-halide film has been commercialized. An electronic still camera is superior to a camera using the silver-halide film in instantaneity, however, inferior in resolution and dynamic range.

In order to improve the resolution, which is one of the above defects of an electronic still camera, an image sensing apparatus adopting pixel shifting method has been proposed. In the pixel shifting method, a plurality of images are sensed while slightly shifting the image formation position of an image, incoming through an optical system, on an image sensing device for performing photo-electric conversion on the image, and the plurality of sensed images are synthesized using a predetermined method to obtain a single high-resolution image. As the prior arts of the pixel shifting method, there are following Japanese Patent Application Laid-Opens, for example.

In the Japanese Patent Application Laid-Open No. 60-27278, a wedge-shaped prism provided in front of the lens systems is rotated about the optical axis, thereby the light path of an optical image formed on an image sensing device is shifted in parallel to the optical axis of the lens system. Then, output images are synthesized to obtain a single high-resolution image.

In the Japanese Patent Application Laid-Open No. 60-91774, in an optical system configured with a magnification optical sub-system and a master optical system, a part of the lens of the master optical system is shifted in the vertical direction with respect to the optical axis, thereby the light path of the optical image formed on an image sensing device is shifted in parallel to an optical axis of the optical system. Then the obtained output images are synthesized to obtain a single high-resolution image.

Further, in the Japanese Patent Application Laid-Open No. 61-236282, a transparent plane parallel plate provided in front of an image sensing device is rotated about an axis which is perpendicular to the optical axis of the image sensing device, thereby the light path of the optical image formed on the image sensing device is shifted in parallel to the optical axis. Then the output images are synthesized to obtain a single high-resolution image.

In the Japanese Patent Application Laid-Open No. 7-287268 (U.S. patent application Ser. No. 08/339,407), a variable apical angle prism provided in front of an optical system is operated on the basis of a vibration signal and a pixel shifting signal, thereby shifting the light path of the optical image formed on an image sensing device in parallel to the optical axis. Accordingly, compensation of vibration caused by a user as well as improvement of resolution of an image are achieved simultaneously.

Note, in this specification, any unintentional undesired movement of camera is expressed as "vibration", and the movement is not limited to periodic motion.

However, in the aforesaid pixel shifting methods, it takes a long time from the first image signal until the last image signal are obtained, similarly to the case of performing multiple exposure in a still camera. Thus, when the electric still camera is vibrated, the quality of an image decreases, namely, a high-resolution image may not be obtained by performing pixel shifting. Thus, in order to overcome this problem, there are following Japanese Patent Application Laid-Opens, for example.

In the Japanese Patent Application Laid-Open No. 7-240932 (U.S. patent application Ser. No. 08/391,388), using a variable apical angle prism provided in front of the optical system or a moving lens system provided behind the optical system, both compensation of vibration and improvement of resolution of the image are achieved at the same time.

Further, according to the Japanese Patent Application Laid-Open No. 7-287268 (U.S. patent application Ser. No. 08/339,407), because resolution in pixel shifting operation decreases when the focal length of the optical system is larger than a predetermined value, pixel shifting is disabled in such a case.

Further, in order to widens dynamic range, which is the other defect of the electronic still camera, there are the following Japanese Patent Application Laid-Opens.

In the Japanese Patent Application Laid-Open No. 1-319370, an image sensing device is exposed a plurality of times with different luminous exposures, and a plurality of images obtained under this operation are synthesized to form a signal image of wide dynamic range.

In the Japanese Patent Application Laid-Open No. 7-264488, a plurality of image sensing devices having different sensibilities are used, and a plurality of images obtained by these image sensing devices are synthesized to form a single image of wide dynamic range.

Furthermore, as a technique for overcoming the aforesaid two problems at the same time, the Japanese Patent Application Laid-Open No. 8-37628 (U.S. patent application Ser. No. 08/505,608) discloses that at least one of a plurality of images obtained while performing pixel shifting is sensed in different luminous exposure from luminous exposure used for sensing other images, thereby obtaining an image of high resolution and wide dynamic range.

Further, in an image sensing apparatus having an image sensing device, it is possible to determine vibration of an image in advance to actually sensing the image by obtaining a movement vector of the image from time-sequential outputs from the image sensing device. Accordingly, in the Japanese Patent Application Laid-Open No. 2-57078 as a prior art in this field, a movement vector of an image is detected continuously, and, when the movement vector becomes the smallest, the image sensed at that time is decided as an image to be recorded, thereby reducing the effect of the vibration of an image sensing apparatus during exposure.

Furthermore, in the Japanese Patent Application Laid-Open No. 8-172568, movement vectors between a plurality of images sensed while performing pixel shifting are obtained, and component of vibration due to vibration of an image sensing apparatus or of an object are removed by performing interpolation, thereafter, the images are synthesized to form a single image of high resolution.

However, in the aforesaid conventional examples disclosed in the Japanese Patent Application Laid-Opens, there are following defeats.

In the methods disclosed in the Japanese Patent Application Laid-Open Nos. 60-27278, 60-91774, and 61-236282, no vibration correction mechanism is provided. Therefore, it is not possible to obtain a high-resolution image when blurring of an image caused by vibration of an image sensing apparatus is large. This is because the operation for obtaining a plurality of images in the pixel shifting method is the same as that of multiple exposure, as described above, and the time required in these operations, namely the time when the first image is sensed until the last image is sensed, is longer than the time required for performing a normal image sensing operation. As a result, effects of vibration on an image is greater when performing pixel shifting operation than when performing the normal image sensing operation vibration.

Further, according to the Japanese Patent Application Laid-Open No. 7-287268 (U.S. patent application Ser. No. 08/339,407), the variable apical angle prism, which is the light path shifting means used for vibration compensation and pixel shifting operation, is provided in front of the optical system, therefore, the coefficient (vibration compensation coefficient) for converting a vibration signal into a value for operating the variable apical angle prism does not change in response to zooming operation. Accordingly, when performing zooming operation, only the coefficient (pixel shifting coefficient) for converting a pixel shifting signal into a value for operating the variable apical angle prism needs to be changed. However, when the light path shifting means is provided in the middle of the optical system in order to down-sizing the optical system, it is necessary to convert both the vibration signal and the pixel shifting signal into values using specific coefficients when performing zooming operation, and operate the light path shifting means in accordance with the values, but there is no disclosure on such the conversion and operation in the Japanese Patent Application Laid-Open No. 7-287268 (U.S. patent application Ser. No. 08/339,407).

Furthermore, in the Japanese Patent Application Laid-Open No. 7-287268 (U.S. patent application Ser. No. 08/339,407), a pixel shifting mechanism is applied to a video camera (camcorder) for recording a moving image, therefore, the interval for taking images is fixed to the field frequency of a moving image. However, when the pixel shifting mechanism is applied to a so-called electronic still camera for recording a still image, it is advantageous to use an image sensing device whose image-taking interval can be changed on the basis of the charging period of the image sensing device (i.e., luminance of an object), since the camera can sense an object in a wide luminance range. In this case, as the image-taking interval becomes longer, effects of vibration on the image becomes stronger, therefore, delicate control of the pixel shifting operation in accordance with image sensing conditions is required. However, in the Japanese Patent Application Laid-Open No. 7-287268 (U.S. patent application Ser. No. 08/339,407), only change in pixel shifting operation in accordance with a focal length of the optical system is disclosed. In addition, there is no teaching on dynamic range expansion.

In the Japanese Patent Application Laid-Open No. 7-240932 (U.S. patent application Ser. No. 08/391,388), pixel shifting is performed even when the resolution of vibration compensation is not good or vibration compensation has failed. As a result, a high-resolution image is not obtained; on the contrary, the quality of the image obtained by performing pixel shifting operation would be lower than an image obtained without performing pixel shifting operation.

According to the Japanese Patent Application Laid-Open No. 7-287268 (U.S. patent application Ser. No. 08/339,407), under conditions in which it is predicted that resolution higher than a predetermined level can not be obtained by performing pixel shifting operation, the pixel shifting operation is disabled. However, the prediction is not performed on the basis of an actual effect of vibration on an image. Therefore, similarly to the cases of other references as explained above, when an image sensing apparatus vibrates by a large displacement amount, the obtained image would have a lower quality than an image obtained without performing pixel shifting operation.

Further, according to the Japanese Patent Application Laid-Open No. 60-91774, resolution of an image is increased, however, there is no teaching on dynamic range expansion.

On the contrary, in the Japanese Patent Application Laid-Open Nos. 1-319370 and 7-264488, dynamic range expansion is explained, however, how to increase resolution of an image is not discussed.

Whereas, in the Japanese Patent Application Laid-Open No. 8-37628 (U.S. patent application Ser. No. 08/505,608), methods for increasing resolution of an image and widening dynamic range are disclosed, however, there is no detailed description on method for determining luminous exposures to be used for sensing a plurality of images for dynamic range expansion. Therefore, the disclosed method is not possible to delicately cope with various scenes which have various luminous distributions. Further, pixel shifting operation requires longer time for performing exposing operation, similarly to a case of performing multiple exposure operation, as described above, therefore, it is necessary to cope with vibration problem. However, there is no teaching on any technique for overcoming the vibration problem.

Further, in the Japanese Patent Application Laid-Open No. 2-57078, there is no teaching on pixel shifting operation, therefore, only reduction of effect of vibration on an image sensed in a normal image sensing operation is achieved. Thus, improvement in resolution of an image is not expected. Furthermore, blurring due to a movement of an object can not be reduced.

Further, in a case of performing pixel shifting operation to increase resolution of an image, it is necessary to control the shift amount to be a predetermined amount based on the interval between pixels. However, there is no teaching on optical vibration compensation means using, e.g., a variable apical angle prism in the Japanese Patent Application Laid-Open No. 8-172568, therefore, blurring of an image while performing pixel shifting operation is large and occurs at random. Therefore, there is no guarantee that pixel shifting by the predetermined shift amount is always performed. In addition, even though blurring of an object is corrected by performing interpolation, possibility of obtaining a high resolution image is low. Furthermore, there is no teaching on warning a user of an image not being obtained in desired resolution, nor about an alternative suggestion to be followed for improving resolution of the image.

SUMMARY OF THE INVENTION

The present invention has been made in consideration of the above situation, and has as its object to provide an image sensing apparatus and method capable of performing image stabilization and pixel shifting simultaneously and at high resolution regardless of the configuration of the optical system and the configuration of image stabilization function.

According to the present invention, the foregoing object is attained by providing an image sensing apparatus comprising: image sensing means for converting an optical image into electric signals and outputting the electric signals as image signals; shifting means for shifting an image formation position of the optical image formed on the image sensing means to a plurality of different positions; first vibration detection means for detecting vibration of the image sensing apparatus and outputting vibration information; control means for controlling the shifting means on the basis of the vibration information outputted by the first vibration detection means; and image signal synthesis means for synthesizing image signals of a plurality of images outputted by the image sensing means to generate a single image, wherein the image sensing means converts the optical image into electric signals at each of the plurality of different image formation positions shifted by the shifting means.

It is another object of the present invention to provide an image sensing apparatus capable of reducing the effect of movement of an object while performing pixel shifting so as to obtain an image of high resolution.

According to the present invention, the foregoing object is attained by providing an image sensing apparatus comprising: image sensing means for converting an optical image into electric signals and outputting the electric signals as image signals; image sensing control means for controlling the image sensing means to sense a plurality of images within a predetermined period of time; image signal synthesis means for synthesizing image signals of the plurality of images outputted by the image sensing means to generate a single image; division means for dividing an image into a plurality of small areas; relationship determination means for determining relationship between the plurality of images by each of the plurality of small areas; and image synthesis control means for controlling image synthesis operation by the image signal synthesis means on the basis of the relationship between the plurality of images determined by the relationship determination means.

Other features and advantages of the present invention will be apparent from the following description taken in conjunction with the accompanying drawings, in which like reference characters designate the same or similar parts throughout the figures thereof.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

FIG. 1 is a block diagram illustrating a configuration of an image sensing apparatus according to an embodiment of the present invention;

FIGS. 2A and 2B are diagrams illustrating an example of an arrangement of lenses in a lens system according to the embodiment of the present invention;

FIGS. 3A and 3B are diagrams for explaining shifts of light paths by the lens system according to the embodiment of the present invention;

FIG. 4 is an explanatory view of a principle of pixel shifting;

FIG. 5 is a block diagram illustrating a detailed configuration of microcomputers of a camera main body and a lens and their peripheral units according to the embodiment of the present invention;

FIG. 6 is a flowchart of control processing in the camera main body according to first to third embodiments of the present invention;

FIG. 7 is a flowchart of control processing in the lens according to the first, fifth, eighth and eleventh embodiments of the present invention;

FIG. 8 is a timing chart of the control processing according to the first and eighth embodiments of the present invention;

FIGS. 9A and 9B are drawings for explaining pixel shifting method according to the first, fifth and eighth embodiments of the present invention;

FIGS. 10A to 10E are drawings for explaining image synthesis method according to the first, fifth, eighth and eleventh embodiments of the present invention;

FIG. 11 is a flowchart of control processing in the camera main body according to a second embodiment of the present invention;

FIGS. 12A and 12B are views for explaining pixel shifting method according to the second and sixth embodiments of the present invention;

FIG. 13A to 13C are drawings for explaining image synthesis method according to the second and sixth embodiments of the present invention;

FIG. 14 is a flowchart of control processing in the camera main body according to a third embodiment of the present invention;

FIG. 15 is a block diagram illustrating a detailed configuration of microcomputers of the camera main body and the lens and their peripheral units according to a fourth embodiment of the present invention;

FIG. 16 is a flowchart of control processing in the camera main body according to the fourth embodiment of the present invention;

FIG. 17 is a flowchart of control processing in the lens according to the fourth embodiment of the present invention;

FIG. 18 is a flowchart of control processing in the camera main body according to fifth to eighth, and tenth embodiments of the present invention;

FIG. 19 is a flowchart showing processes for setting luminous exposure performed in processing shown in FIG. 18;

FIG. 20 is a graph showing characteristics of a film and an image sensing device;

FIG. 21 is a view showing a detection area where focus state detection and the photometry are performed;

FIG. 22 is a timing chart of the control processing according to the fifth embodiments of the present invention;

FIG. 23 is a graph showing a feature of the fifth embodiment;

FIG. 24 is a flowchart of control processing in the camera main body according to a sixth embodiment of the present invention;

FIG. 25 is a flowchart of control processing in the camera main body according to a seventh embodiment of the present invention;

FIGS. 26A and 26B are graphs for explaining a principle for calculating a relative shifted amount of two images;

FIGS. 27A and 27B are graphs for explaining the principle for calculating a relative shifted amount of the two images;

FIG. 28 is a graph showing relationship between shift amount and correlation;

FIG. 29 is an explanatory drawing of a principle of pixel shifting according to an eighth embodiment;

FIG. 30 is a graph showing correlation between first and second image signals;

FIG. 31 is an explanatory view showing a case where large vibration was occurred while performing pixel shifting operation according to an eighth embodiment;

FIG. 32 is a graph showing correlation between first and second image signals;

FIG. 33 is a flowchart of control processing performed in the processing shown in FIG. 18;

FIG. 34 is an explanatory view showing a track of a given point of an image formed on an image sensing device according to a ninth embodiment of the present invention;

FIG. 35 is a flowchart of control processing in the camera main body according to the ninth embodiment of the present invention;

FIG. 36 is a flowchart of control processing in the camera main body according to the ninth embodiment of the present invention;

FIG. 37 is an explanatory view showing a track of a given point of an image on an image sensing device according to a tenth embodiment of the present invention;

FIG. 38 is a graph showing correlation between first and second image signals;

FIG. 39 is a flowchart of control processing in the camera main body according to the tenth embodiment of the present invention;

FIG. 40 is a flowchart of control processing in the camera main body according to an eleventh embodiment of the present invention;

FIG. 41 is an explanatory view showing an image of an object formed on a photo-sensing surface of an image sensing device;

FIG. 42 is an explanatory view showing a synthesized image of an object;

FIG. 43 is an explanatory view of divided areas of a photo-sensing surface of the image sensing device;

FIG. 44 is an explanatory view showing a determined result of blurring due to movement of the object;

FIG. 45 shows an image obtained by synthesizing a plurality of images according to the eleventh embodiment of the present invention;

FIG. 46 is a flowchart of blurring determination processing due to vibration performed in the processing shown in FIG. 40;

FIG. 47 is a flowchart of blurring determination processing due to movement of the object performed in the processing shown in FIG. 40;

FIG. 48 is a flowchart of image synthesis processing according to the eleventh embodiment;

FIG. 49 is a view for explaining image synthesis method according to a twelfth embodiment;

FIG. 50 is a view for explaining image synthesis method according to the twelfth embodiment;

FIG. 51 is a view for explaining image synthesis method according to the twelfth embodiment;

FIG. 52 is a flowchart of control processing in the camera main body according to the twelfth embodiment of the present invention;

FIG. 53 is a view for explaining image synthesis method according to a thirteenth embodiment;

FIG. 54 is a view for explaining image synthesis method according to the thirteenth embodiment;

FIG. 55 is a flowchart of control processing in the camera main body according to the thirteenth embodiment of the present invention; and

FIG. 56 is a flowchart of control processing in the camera main body according to a fourteenth embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Preferred embodiments of the present invention will be described in detail in accordance with the accompanying drawings.

As embodiments of the present invention, first to fourteenth embodiments are explained below. First, a configuration of a camera commonly used in the first to fourteenth embodiments is explained below.

<Configuration of Camera>

FIGS. 1 to 5 are common for the first to fourteenth embodiments.

FIG. 1 is a block diagram illustrating a configuration of an image sensing apparatus according to an embodiment of the present invention. The detail will be explained later.

FIGS. 2A and 2B are diagrams illustrating an example of an arrangement of lenses in a lens system. The lens system is a zoom lens system, and its focal length ranges between 10 mm to 30 mm, namely, third-power lens system. Especially, FIG. 2A shows the lenses in the wide-angle position (focal length=10 mm), and FIG. 2B shows the lenses in the telephoto position (focal length=30 mm).

This lens system is basically configured with four groups of lenses: when changing magnification, the fourth lens group is stationary, and the first, second and third lens groups move; further, when focusing, the second, third and fourth lens groups are stationary, and the first lens group moves. Pixel shifting and vibration compensation are performed by shifting the image formation position of an image on the imaging surface by shifting the second lens group in the vertical direction with respect to the optical axis of the lens system.

Next, an effect of the second lens group on shifting a light path is explained with reference to FIGS. 3A and 3B. FIGS. 3A and 3B are diagrams conceptually showing the four lens groups shown in FIGS. 2A and 2B. FIG. 3A shows a shift of the light path in the lens groups in the image space when the second lens group is shifted downward by a distance dL, and FIG. 3B shows the shift of the light path in the object space when the second lens group is shifted downward by the distance of dL.

First, FIG. 3A is explained. A light path of a ray incoming from the object space and traveling on the optical axis of the first lens group is shifted upward by the second lens group which is shifted downward, passes through the third and fourth lens groups and incidents on the imaging surface at a position which is above the optical axis of the first, third and fourth lens groups by the distance of d.sub.IM. If a ratio of the shifted distance of the second lens group, d.sub.L, to the maximum shifted distance on the imaging surface, d.sub.IM, is defined as decentering sensitivity, S.sub.d, then these three values have relationship expresses as,

The decentering sensitivity S.sub.d changes in accordance with the arrangement of the second to fourth lens groups, which means the decentering sensitivity Sd changes in accordance with zooming operation according to the embodiment of the present invention. Further, since a front focusing using the first lens group is adopted in the embodiment of the present invention, the decentering sensitivity S.sub.d does not change in response to focusing operation. However, when a rear focusing using the fourth lens group is adopted, the decentering sensitivity S.sub.d changes in response to focusing operation. Therefore, the decentering sensitivity S.sub.d is generally expressed as a function of the focal length f and a distance to an object R, namely, S.sub.d (f, R), thus, the equation (1) may be modified to,

Next, a shift amount of the second lens group when performing pixel shifting operation is explained. FIG. 4 is an explanatory view of a principle of pixel shifting and shows a magnified view of a photo-sensing surface of the image sensing device. Referring to FIG. 4, photo-sensing elements having a square shape, namely, pixels, are regularly arranged on the photo-sensing surface at intervals of W.sub.Y in the horizontal direction and W.sub.P in the vertical direction. The resolution of an image formed on the photo-sensing surface is determined by the intervals W.sub.Y and W.sub.P between the pixels. However, it is possible to increase the resolution of the image by taking a plurality of images while shifting relative positions between the pixels and the image, and synthesizing the images by following a predetermined rule to make a single image. For example, when a given object point of an image is formed at the center of the photo-sensing element, IM1, image signals of the image (entire image signals obtained from the area sensor) are read and stored as a first set of image signals. Next, the image is shifted so that the same object point of the image is shifted to a position IM2 on the photo-sensing surface, namely shifted to right by X.sub.Y =W.sub.Y /2, and there, a second set of image signals are read and stored. Similarly, while shifting the object point of the image to positions IM3 and IM4 on the photo-sensing surface, third and fourth sets of image signals are taken. Thereafter, the four sets of image signals, which provide four times more information on the image, are synthesized. Accordingly, spatial resolution of the image is doubled both in the horizontal and vertical directions.

In order to shift the image by X.sub.Y (=W.sub.Y /2) and/or X.sub.P (=W.sub.P /2) in the pixel shifting operation, the effect of the second lens group on shifting the light path shown in FIG. 3A is utilized. More specifically, in order to shift the image upward by X.sub.P, the second lens group is to be shifted downward by a distance of d.sub.L which is determined by,

in accordance with the equation (2). The shift amount X.sub.P of the image in the pixel shifting operation is fixed, however, the decentering sensitivity Sd(f, R) changes in accordance with zooming and focusing operations. Accordingly, the shift amount of the second lens group d.sub.L needs to be changed in accordance with the state of the lens system. In the present invention as described above, data on the decentering sensitivity Sd(f, R) corresponding to states of the lens system, such as zooming and focusing, is stored as a first coefficient in ROM of a microprocessor (CPU).

Next, FIG. 3B is explained. A ray traveling from the center of an image along an optical axis C of the lens system to the left in FIG. 3B passes through the fourth and third lens groups, then shifted upward by the second lens group which is shifted downward. Then, the ray which has passed through the first lens group is projected on the object tilted by an angle of .theta..sub.OB with respect to an axis C' which is parallel to the optical axis C. If the ratio of the shifted amount of the second lens group d.sub.L to the tilted angle .theta..sub.OB with respect to the optical axis is defined as the angle sensitivity, S.sub..theta., then these three values have relationship expressed as,

The angle sensitivity S.sub..theta. changes depending upon an arrangement of lenses in the upstream of the second lens group. Namely, in the embodiments of the present invention, the angle sensitivity S.sub..theta. changes in response to zooming and focusing operations. Since the angle sensitivity S.sub..theta. is expressed as a function of the focal length f and the distance to an object R, namely, S.sub..theta. (f, R), similarly to the decentering sensitivity S.sub.d, the equation (4) may be modified to,

.theta..sub.OB =S.sub..theta. (f, R).times.d.sub.L. (5)

Next, a shift amount of the second lens group for compensating vibration is explained. Assuming that a camera having the lens system and the image sensing device vibrates in the angular direction so that the optical axis of the lens system turns downward by an angle of .theta..sub.CAMERA, blurring of the image due to the angular vibration this time corresponds to a case where an object is shifted upward with respect to the camera by an angle of .theta..sub.OB (=.theta..sub.CAMERA). Referring to FIG. 3B, it is possible to compensate the shift of the object by shifting the second lens group by the direction of d.sub.L when the object is shifted upward by the angle of .theta..sub.OB. Therefore, the blurring of the image due to the angular vibration can be compensated by shifting the second lens group downward by the distance d.sub.L calculated on the basis of the following equation (6) on the basis of the shifted angle, .theta..sub.CAMERA, detected by a vibration sensor, and the equation (5),

Since the vibration angle .theta..sub.CAMERA changes with respect to time and the angle sensitivity S.sub..theta. (f, R) also changes in response to zooming and focus operations, the shift amount of the second lens group needs to be changed in accordance with the state of the lens system. Therefore, in the present invention, the angle sensitivity S.sub..theta. (f, R) which changes in response to zooming and focusing operations is also stored in connection with the states of the lens system in the ROM of the CPU as a second coefficient, similarly to the decentering sensitivity Sd(f, R).

FIG. 1 is the configuration of the image sensing apparatus according to the embodiment of the present invention. In FIG. 1, reference CMR denotes a camera main body; and LNS, a lens configured as an interchangeable lens, detachable from the camera main body CMR.

An explanation of the configuration of the camera main body CMR follows.

Reference CCPU denotes a one-chip microcomputer of the camera main body CMR, having ROM, RAM, and analog-digital and digital-analog conversion functions. The microcomputer CCPU performs a series of processes, such as automatic exposure (AE) control, automatic focusing (AF) control, and pixel shifting control, by executing a sequential program, for the camera, stored in the ROM. Thus, the microcomputer CCPU communicates with respective circuits of the camera main body CMR and the lens LNS in order to control the circuits and the lens LNS.

On a mount unit for connecting the camera main body CMR and the lens LNS, four pairs of connection terminals are provided. An internal battery BAT in the camera main body CMR provides electric power to the respective circuits in the camera main body CMR and to an actuator, as well as to the lens LNS via a line VCC.

Reference DCL denotes a signal line for transmitting a signal from the microcomputer CCPU of the camera main body CMR to a microcomputer LCPU (will be explained later) of the lens LNS, and DLC denotes a signal line for transmitting a signal from the microcomputer LCPU of the lens LNS to the microcomputer CCPU of the camera main body CMR. Further, the camera main body CMR and the lens LNS are both grounded via a line GND.

Reference IMS denotes an image sensing device, such as CCD, and reference IMDR denotes a driver for controlling charging of the image sensing device IMS and transference of the stored charges in the image sensing device IMS.

Further, reference MEM denotes a memory for recording/storing image data of a sensed image, and realized by a semi-conductor memory, a magnetic disk, and an optical disk, for instance; DISP, a display, such as a liquid crystal display, for displaying an image obtained by the image sensing device IMS as well as operation state of the camera; and BS, a beam splitter, configured with a half mirror, for leading a part of the luminous flux of an image to a sensor SNS. The sensor SNS has a focus state detection sensor for detecting the focus state of the lens system and a photometric sensor for detecting luminosity of the object.

Reference CNC denotes a connector for connecting to an external device, such as a desk-top computer, and the connector CNC is used for transmitting the contents of the memory MEM to the external device, and controlling the camera main body CMR from the external device using a signal from the external device.

Reference SWMN denotes a main switch, and when the main switch SWMN is turned on, then the microcomputer CCPU starts executing a predetermined program relating to image sensing operation.

SW1 and SW2 denote switches which operate in response to the operation of the release button, and SW1 is turned on when the release button is pressed halfway (half stroke) and the SW2 is turned on when full stroke is made.

SWSF denotes a pixel shifting mode selection switch which is used for selecting either permission or prohibition of pixel shifting operation, as well as selecting one of a plurality of predetermined pixel shifting modes.

SWIS denotes an image stabilization (IS) selection switch for selecting either permission or prohibition of image stabilization.

SWMOD denotes an image sensing mode selection switch, and when a user selects one of a predetermined image sensing modes, AE mode, AF mode, pixel shifting mode, and IS mode, corresponding to the selected image sensing mode, are automatically set.

Next, the configuration of the lens LNS is explained.

Reference LCPU denotes the microcomputer of the lens LNS, and configured as a one-chip microcomputer having ROM, RAM, and analog-digital and digital-analog conversion functions, similarly to the microcomputer CCPU of the camera main body CMR. The microcomputer LCPU controls a focusing actuator, a zooming actuator, a iris diaphragm actuator, and an IS actuator, all of which will be explained later, in accordance with instructions transmitted from the microcomputer CCPU via the signal line DCL. Further, the microcomputer LCPU transmits operation state of the lens LNS and parameters which are specific to the lens to the microcomputer CCPU via the signal line DLC.

Reference L1 to L4 denote lens groups corresponding to the first to fourth lens groups, respectively explained with reference to FIG. 2, which configure a zoom lens system, and an image of an object is formed on the image sensing device IMS via the zoom lens system.

FACT denotes the focusing actuator for moving the first lens group L1 in the back and forth directions along the optical axis to perform focus control. A focus encoder FENC detects the position of the first lens group L1, which corresponds to information on the distance to the object, then the obtained information is transmitted to the microcomputer LCPU.

Reference ZACT denotes the zooming actuator which performs zooming operation by moving the first to third lens groups L1 to L3 in the back and forth directions along the optical axis. A zoom encoder ZENC detects information on zooming operation, namely, the focal length of the zoom lens system, then transmits the information to the microcomputer LCPU.

DFM denotes an iris diaphragm, and DACT denotes an iris diaphragm actuator for driving the iris diaphragm DFM.

Further references GRP and GRY are vibration sensors, such as vibration-type gyroscopes, and two sensors of the same type are provided as the vibration sensors GRP and GRY for sensing the angular vibration in the vertical direction (pitch) and horizontal direction (yaw) of the camera. The detected results of vibration are sent to the microcomputer LCPU.

The second lens group L2 is provided so as to be movable individually in the plane perpendicular to the optical axis in the two-dimensional directions. The second lens group L2 is driven by a pitch actuator IACTP in the vertical direction with respect to the optical axis, i.e., in the direction for compensating pitch, and driven by a yaw actuator IACTY in the horizontal direction (in FIG. 1, the direction normal to the paper), namely, in the direction for compensating yaw. Note, the shift function is disclosed in the Japanese Patent Application Laid-Open 6-3727 by the same applicant.

FIG. 5 is a block diagram illustrating a detailed configuration of the microcomputers CCPU and LCPU of the camera main body CMR and the lens LNS and their peripheral units according to the embodiment of the present invention. Upper blocks enclosed by a two-dot-dash line are included in the microcomputer CCPU of the camera main body CMR, and lower blocks enclosed by another two-dot-dash line is included in the microcomputer LCPU of the lens LNS.

Reference numeral 11 denotes an image sensing condition setting circuit for setting operation modes, such as AE mode, AF mode, pixel shifting mode and IS mode; 12, a timing pulse generator for generating a trigger signal for controlling timing of operation of the lens system for pixel shifting operation and controlling timing for taking image signals from the image sensing device; 13, an image sensing device operation circuit for taking image signals at a predetermined timing under a predetermined charging condition in response to control signals generated by the image sensing condition setting circuit 11 and the timing pulse generator 12; 14, a temporary storage circuit for temporarily storing the obtained image signals until time for performing synthesis operation; 15, an image synthesis circuit for synthesizing plural sets of image signals obtained while performing pixel shifting operation to generate a single image of high resolution; and 16, a recording unit, corresponding to the memory MEM in FIG. 1, for recording the synthesized high-resolution image.

Further, reference numeral 21 denotes a pixel shifting signal generator for generating instruction signals (signals having reference waveforms shown in FIG. 8. Will be described later in detail) for shifting an image formation position for pixel shifting; and 22, a first coefficient generator for reading data corresponding to the decentering sensitivity Sd(f, R), which is explained above, from the ROM of the microcomputer LCPU in accordance with the focus and zoom information of the lens system, and calculating shift amount instruction values for the second lens group L2, so that the amounts of shifts of the image formation position in the yaw and pitch directions become X.sub.Y and X.sub.P, respectively. The shift amount instruction values can be obtained by multiplying the amplitude of the instruction signals generated by the pixel shifting signal generator 21 by the read data (decentering sensitivity Sd(f, R)).

Reference numeral 31 denotes a vibration sensor which corresponds to the vibration-type gyro GRP and GRY, explained above; 32, a vibration signal calculation circuit for performing filtering and accumulation on an angular velocity signal of the vibration detected by the vibration sensor 31, and calculating a vibration angle; and 33, a second coefficient generator for reading data corresponding to the angle sensitivity S.sub..theta. (f, R), explained above, from the ROM of the microcomputer LCPU in accordance with the focus and zoom information of the lens system, correcting the value of the vibration angle calculated by the vibration signal calculation circuit 32, and calculating shift amount instruction values (amplitudes of vibration compensation signals) for the second lens group L2 for image stabilization.

Further, reference numeral 41 denotes a synthesis circuit for adding the shift amount instruction value for the second lens group L2 for pixel shifting calculated by the first coefficient generator 22 and the shift amount instruction value for the second lens group L2 for image stabilization calculated by the second coefficient generator 33; 42, an image stabilization (IS) actuator controller for controlling the pitch actuator IACTP and the yaw actuator IACTY in FIG. 1, so that the second lens group L2 moves in accordance with the value obtained by the synthesis circuit 41; and 43, a block indicating that the second lens group L2 is shifted, in other words, the image formation position on the image sensing device 13 is shifted.

First Embodiment

FIGS. 6 and 7 are flowcharts of control processing by the microcomputer CCPU of the camera main body CMR and the microcomputer LCPU of the lens LNS, respectively, according to a first embodiment of the present invention.

First, a flow of the control processing by the microcomputer CCPU of the camera main body CMR is explained with reference to FIGS. 1 and 6.

When the main switch (power switch) SWMN of the camera main body CMR is turned on in step S101, electric power is supplied to the microcomputer CCPU, then the process proceeds to step S102 where operation of the camera starts.

In step S102, the state of the switch SW1, which is turned on in response to the half stroke of the release button, is detected. If the SW1 is off, then the process proceeds to step S103, where an instruction to stop image stabilization (IS) operation (IS stop instruction) is transmitted to the lens LNS.

The steps S102 and S103 are repeatedly performed until the switch SW1 is turned on or the main switch SWMN is turned off.

When the switch SW1 is turned on in step S102, the process proceeds to step S111. In step S111, the microcomputer CCPU transmits an instruction to start IS operation (IS start instruction) to the microcomputer LCPU via the signal line DCL.

Next in step S112, communication for obtaining parameters which are specific to the lens, such as F number and focal length of the lens, from the microcomputer LCPU is performed.

Then, in step S113, the luminance of the object is sensed by the sensor SNS, and the charging period of the image sensing device for obtaining image signals and the value for controlling the iris diaphragm are calculated in accordance with the predetermined exposure control program, and the microcomputer CCPU transmits the calculation results to the microcomputer LCPU.

The process proceeds to step S114 where the focus state is detected by the sensor SNS, and an instruction for operating the focus lens is transmitted to the microcomputer LCPU.

In step S115, the state of the pixel shifting mode selection switch SWSF is detected. Further, pixel shifting conditions, such as, whether or not the pixel shifting is to be performed and the number of image formation positions N.sub.SF on the image sensing device IMS in an image sensing operation (if it determined not to perform pixel shifting, N.sub.SF is set to 1, whereas if it is determined to perform pixel shifting, then N.sub.SF is set to at least 2), are set on the basis of the result of the photometry.

Thereafter, the process proceeds to step S116, where the state of the switch SW2 which is turned on in response to the full stroke of the release button is detected. If the switch SW2 is OFF, then the process returns to step S111, and steps S111 to S115 are repeated. Whereas, if it is detected that the switch SW2 is ON, then the process proceeds to step S117.

In step S117, a counter CNT for counting the number of image formation positions is initialized to 0.

Then, in step S118, a timing pulse which is a trigger signal for image sensing operation is generated, and transmitted to the microcomputer LCPU.

In step S119, the microcomputer CCPU controls the image sensing device IMS, via the driver IMDR, to charge, then transfer the charges in the image sensing device IMS.

In step S120, the image signals read at step S119 are temporarily stored in the RAM of the microcomputer CCPU.

In step S121, the counter CNT 1 is increased by 1.

In step S122, whether or not the counter CNT reaches the number of image formation positions N.sub.SF is determined. If it is not, then the process returns to step S118 and waits for the next timing pulse being generated, then pixel shifting operation is continued. If it is determined that the counter CNT has reached the number of image formation positions N.sub.SF, then the process proceeds to step S123.

In step S123, completion of pixel shifting operation (or completion of the storing of required image signals) is informed to the microcomputer LCPU.

In step S124, if image signals of a plurality of images are stored in the RAM, then they are synthesized to generate a single high-resolution image, then outputted. Whereas, if image signals of a single image are stored in the RAM, it is not possible to perform synthesis, therefore, the image signals are outputted.

In step S125, the image outputted in step S124 is stored in the memory MEM.

Accordingly, an image sensing operation is completed and the process returns to step S102. If the switch SW1 becomes ON in step S102, then the processes in step S111 and the subsequent steps are repeated, whereas, if the switch SW1 is OFF, then an instruction to stop IS operation is transmitted to the microcomputer LCPU in step S103.

FIG. 7 is the flowchart of control processing by the microcomputer LCPU of the lens LNS.

Referring to FIG. 7, when electric power is provided to the exchange lens, in step S131, in response to the ON operation of the main switch SWMN of the camera main body CMR, then the process proceeds to step S132.

In step S132, whether or not the IS start instruction is received or not is determined, and if the IS start instruction is not received from the camera main body CMR, then the process proceeds to step S133.

In step S133, whether or not the IS stop instruction is received from the camera main body CMR is determined, and if not, the process returns to step S132. If it is determined that the IS stop operation is received, then the process proceeds to step S134 where the IS operation is stopped. More specifically, the pitch and yaw actuators IACTP and IACTY are deactivated.

If the IS start instruction is received from the microcomputer CCPU while performing processes in steps S132 to S134, then the process proceeds from step S132 to step S141.

In step S141, the vibration sensors GRP and GRY are activated, and vibration signals in the pitch and yaw directions are inputted.

Step S142 corresponds to step S112 in FIG. 6, and in response to requests from the microcomputer CCPU, the microcomputer LCPU of the lens LNS transmits the parameters which are specific to the lens LNS to the camera main body CMR.

In step S143, the zoom encoder ZENC and the focus encoder FENC are checked in order to detect zooming and focus states of the lens system.

In step S144, on the basis of the detection result in step S143, the first coefficient for pixel shifting and the second coefficient for IS operation are read from a table stored in the ROM of the microcomputer LCPU.

In step S145, the pitch and yaw actuators IACTP and IACTY are operated on the basis of the vibration signal obtained in step S141 and the second coefficient obtained in step S144 to reduce blurring of an image due to vibration.

In step S146, the microcomputer LCPU operates the iris diaphragm DFM via the iris diaphragm actuator DACT on the basis of the information on a photometry result transmitted from the microcomputer CCPU to control the luminous exposure.

In step S147, the focusing actuator FACT is operated on the basis of the information on the focus state detection obtained from the microcomputer CCPU to adjust focus.

Next, in step S148, whether or not a timing pulse for triggering the pixel shifting operation is received is determined. If no timing pulse is received, the process returns to step S141, and the IS operation, the iris diaphragm control, and the focus adjustment are repeatedly performed. When it is determined in step S148 that the timing pulse is received, the process proceeds to step S149.

In step S149, signals having reference waveforms for driving the second lens group L2 in the pitch and yaw directions for pixel shifting operation are generated by the pixel shifting signal generator 21. Note, the amplitudes of the signals correspond to the distances X.sub.P and X.sub.Y, shown in FIG. 4, in the pitch and yaw directions, when it is assumed that there is no effect of the decentering sensitivity.

In step S150, the amplitudes of the signals having reference waveform, generated in step S149, are multiplied by the first coefficient read in step S144, thereby pixel shifting signals which compensates the effect of the decentering sensitivity of the second lens group L2 are generated. Thereafter, the generated pixel shifting signals are synthesized with signals for the IS operation (vibration compensation signals), generated by the second coefficient generator 33, in the synthesis circuit 41. By operating the pitch and yaw actuators IACTP and IACTY in accordance with the synthesized signals, the IS operation and the pixel shifting operation are performed simultaneously and precisely.

In step S151, whether or not a signal indicating completion of the pixel shifting operation is received from the microcomputer CCPU is determined, and if it is not, the process returns to step S148 because the pixel shifting has not been finished. Then, the process waits the next timing pulse. The processes in steps S148 to S150 are repeated for a predetermined number of times, and when the signal indicating completion of the pixel shifting operation is transmitted, the process returns from step S151 to step S132.

Then, if the IS start instruction is not received in step S132 and the IS stop instruction is detected in step S133, then the IS operation is stopped in step S134; more specifically, the pitch and yaw actuators IACTP and IACTY are deactivated, and a series of lens control operation relating to the image sensing operation is completed.

FIG. 8 is a timing chart for explaining operations of the camera main body CMR and the lens LNS shown in the flowcharts in FIGS. 6 and 7.

Signals A and B show states of the switches SW1 and SW2, respectively; a signal C is a timing signal for pixel shifting operation; a signal D is for charging in the image sensing device IMS; signals E and F have reference waveforms for pixel shifting operation (referred to as "pixel shifting reference signals" hereinafter) in the pitch direction and the yaw direction, respectively; and signals G and H are vibration signals, in the pitch and yaw directions, detected by the vibration sensors GRP and GRY, respectively. Here, vibration shift waveforms obtained by processing the detected signals by integration, for example, are shown. Further, signals I and J are operation signals for shifting the second lens group L2 in the pitch and yaw directions, respectively.

Next, overall operation shown in the flowcharts in FIGS. 6 and 7 is explained with reference to the timing chart shown in FIG. 8. When the switch SW1 is turned on at time t.sub.1, the vibration signals G and H are outputted. In turn, the second lens group L2 is operated as shown in the waveforms of the signals I and J in accordance with the vibration signals multiplied by the second coefficient.

Then the switch SW2 is turned on at time t.sub.2, and after a predetermined period of time elapses from the time t.sub.2, a timing pulse TP1 is generated at time t.sub.11. In turn, the photo-sensing elements of the image sensing device IMS are charged between time t.sub.12 and time t.sub.13 in accordance with a charging period calculated on the basis of the result of the photometry.

When the charging operation ends at the time t.sub.13, the charges are transferred and read. At the same time, the pixel shifting reference signal F in the yaw direction is generated. Accordingly, the second lens group L2 is operated in the yaw direction on the basis of instruction values shown by the signal J, obtained by adding the pixel shifting reference signal F multiplied by the first coefficient and the vibration signal H multiplied by the second coefficient.

After a predetermined period of time has passed since time t.sub.11, the second timing pulse TP2 is generated at time t.sub.21. Then, similarly to above, the photo-sensing elements of the image sensing device IMS are charged between time t.sub.22 and time t.sub.23. When the charging operation ends at time t.sub.23, the charges are transferred and read. At the same time, the pixel shifting reference signal E in the pitch direction is generated. Accordingly, the second lens group L2 is operated in the pitch direction on the basis of instruction values shown by the signal I, obtained by adding the pixel shifting reference signal E multiplied by the first coefficient and the vibration signal G multiplied by the second coefficient.

After a predetermined period of time has passed since time t.sub.21, the third timing pulse TP3 is generated at time t.sub.31. Then, similarly to above, the photo-sensing elements of the image sensing device IMS are charged between time t.sub.32 and time t.sub.33. When the charging operation ends at time t.sub.33, the charges are transferred and read. At the same time, the value of the pixel shifting reference signal F in the yaw direction is changed to the initial value. Accordingly, the second lens group L2 is operated in the yaw direction on the basis of instruction values shown by the signal J, corresponding to the vibration signal H multiplied by the second coefficient.

After a predetermined period of time has passed since time t.sub.31, the last timing pulse TP4 is generated at time t.sub.41. Then, similarly to above, the photo-sensing elements of the image sensing device IMS are charged between time t.sub.42 and time t.sub.43. When the charging operation ends at time t.sub.43, the charges are transferred and read. At the same time, the value of the pixel shifting reference signal E in the pitch direction is changed to the initial value. Accordingly, the second lens group L2 is operated in the pitch direction on the basis of instruction values shown by the signal I, corresponding to the vibration signal G multiplied by the second coefficient.

After the switch SW1 is turned off at time t.sub.5, the vibration detection and the operation of the second lens group L2 are stopped.

The given point of an image on the image sensing device IMS while performing the image stabilization and the pixel shifting operation at time t.sub.11, t.sub.21, t.sub.31, t.sub.41 and t.sub.5 are at IM1, IM2, IM3, IM4 and IM1 in FIG. 4, respectively, thus image formation positions of the image are shifted by a half pixel distance from each other in the vertical and horizontal directions.

Note, the reason for the pixel shifting reference waveform being a trapezoid shape rather than a square shape is to mitigate shock of sudden movement of the second lens group L2.

Next, the principle for generating an image signal of a single high-resolution image by synthesizing a plurality of images obtained while performing pixel shifting operation is explained with reference to FIGS. 9A, 9B and 10A to 10E.

FIGS. 9A and 9B are views for explaining relative position relationship between an image and the image sensing device IMS in pixel shifting operation. FIG. 9A corresponds to FIG. 4, and it shows that the position of the image shifts in the order of IM1, IM2, IM3, IM4, and IM1 with respect to the pixels of the image sensing device IMS fixed in the camera main body CMR. The above movement is equivalent to a case where the position of the image sensing device IMS moves in the order of IG1, IG2, IG3, IG4, and IG1 with respect to a fixed object. Now, let an output signal from each pixel when the image sensing device IMS is at the position, IG1, be IG1(i, j), where i and J are coordinates of the pixel. The image sensing device IMS is an area sensor having m.times.n pixels. Similarly, let output signals when image sensing device IMS is at the positions, IG2, IG3 and IG4, be IG2(i, j), IG3(i, j), and IG4(i, j), respectively.

FIGS. 10A to 10E show how to synthesize these four sets of image signals.

Let a new set of image signals representing 2m.times.2n pixels obtained by four sets of m.times.n pixels be denoted by IMG(u, v). The four left uppermost corner pixels of the image signals IMG(u, v) are obtained by synthesizing the respective left uppermost corner pixels of the four original images as shown in FIGS. 10A to 10E. When a method for synthesizing images is considered with reference to FIGS. 10A to 10E, it is possible to obtain an image signal of a single high-resolution image from the four original images by using the following four equations; ##EQU1##

Note, the aforesaid pixel shifting operation and the image synthesis method is used when image signals are obtained from a black-and-white image sensing device and a multiple-CCD type color image sensing device using a color separation prism. When image signals are obtained from a single CCD type color image sensing device covered with a mosaic color filter, although there are little differences in pixel shifting amount in the pixel shifting operation and the image synthesis method, the basic ideas of the pixel shifting operation and the image synthesis are the same.

According to the first embodiment as described above, (1) By changing a pixel shifting signal in accordance with the first coefficient, changing a vibration signal in accordance with the second coefficient, and operating an image stabilization lens system on the basis of the synthesized signal of the above two changed signals, it is possible to perform image stabilization operation and pixel shifting operation at the same time using only a single image shifting means, i.e., the image stabilization lens system. Accordingly, it is possible to obtain a high-resolution image, by pixel shifting operation, with less deterioration due to vibration. (2) By using the first and second coefficients selected in accordance with zooming and focus states, it is possible to always perform precise image stabilization operation and pixel shifting control even when the zooming and focus states are changed. (3) Since synthesis of images obtained while performing pixel shifting operation is performed within a camera, an image signal of a high-resolution image can be obtained without using an exclusive external device.

Second Embodiment

The first embodiment is for precisely and simultaneously performing image stabilization and pixel shifting operation. In the following second embodiment, an optimum pixel shifting mode is selected in accordance with the state of the camera.

FIG. 11 Is a flowchart showing control processing in the camera main body CMR according to the second embodiment; FIGS. 12A and 12B are views for explaining relative position relationship between an image and the image sensing device IMS in pixel shifting operation according to the second embodiment; and FIGS. 13A to 13C are views for explaining a principle of image synthesis in a second pixel shifting mode (will be explained later). The second embodiment will be explained with reference to accompanying drawings.

The control processing in the camera main body CMR in the second embodiment is basically the same as that shown in FIG. 6 explained in the first embodiment. However, the process performed in step S115, i.e., "to set a pixel shifting condition" is realized by a sub-routine as shown in FIG. 11, thereby the advantage of the second embodiment can be obtained. Below, the control processing according to the second embodiment will be explained with reference to FIGS. 6 and 11. Since the processes shown in FIG. 6 have been already explained in detail in the first embodiment, they are only briefly explained in the second embodiment.

Referring to FIG. 6, when it is determined in step S102 that the switch SW1 is ON, then the process proceeds to step S111. Thereafter, an IS start instruction is transmitted to the lens LNS in step S111 and parameters are received from the lens LNS in step S112. Next in steps S113 and S114, photometry and focus state detection are performed, and the obtained results are transmitted to the microcomputer LCPU of the lens LNS.

In the next step, S115, the processes shown in FIG. 11 are performed. In step S215 in FIG. 11, the microcomputer CCPU requests transmission of the peak value of vibration angular velocity, .omega..sub.peak, in a predetermined period of time to the microcomputer LCPU. In turn, the microcomputer LCPU transmits the peak value of the vibration angular velocity .omega..sub.peak occurring in a two second interval, for example, to the microcomputer CCPU.

In step S216, the maximum vibration value .delta. while exposing the image sensing device IMS when the IS function is not operated is calculated on the basis of the following equation,

where, f denotes a focal length of the optical system, and t.sub.exp is an exposure time, i.e., charging period, of the image sensing device determined on the basis of the result of photometry. The maximum vibration value .delta. obtained here is used in the subsequent steps as an index of vibration for determining whether or not pixel shifting operation should be performed.

In step S217, the value of the maximum vibration value .delta. is checked. If the maximum vibration value .delta. is equal or less than a predetermined value DEL1, then it is determined that effect of vibration is small, thus the quality of an image would improve by performing pixel shifting operation. Accordingly, the process proceeds to step S218 and the number of image formation positions N.sub.SF, is set to four. Here, the number of image formation positions, four, indicates the same pixel shifting operation and image synthesis explained in the first embodiment. This overall operation is referred to as "first pixel shifting mode" in the second embodiment.

Whereas, if it is determined in step S217 that the vibration value .delta. is larger than the value DEL1 and equal or less than a predetermined value DEL2 (DEL2>DEL1), then the process proceeds to step S219 where the number of image formation positions N.sub.SF is set to two. This is because deterioration of an image is expected to some degree even if the image stabilization is performed. Therefore, a mode which requires less image formation positions is selected (this mode is referred to as "second pixel shifting mode") so as to reduce deterioration of the image quality due to vibration and to realize an optimum improvement in image quality. Details of the second pixel shifting mode are explained later.

In step S217, when it is determined that the vibration value .delta. is greater than the predetermined value DEL2, then in step S220, the number of image formation positions N.sub.SF is set to 1 indicating that no pixel shifting operation is to be performed. The reason for setting the number of image formation positions N.sub.SF to 1 is that the effect of the vibration which causes deterioration of an image is stronger than the effect of pixel shifting operation which improves the quality of the image. Therefore, the pixel shifting is disabled.

After one of the processes in steps S218 to S220 is performed, the process proceeds to step S221.

In step S221, information on the determined pixel shifting mode is transmitted to the microcomputer LCPU of the lens LNS.

In step S222, the type of the pixel shifting mode is displayed on the display device DISP of the camera main body CMR to inform the user of which mode is used for photographing an image.

After step S222 is completed, the process returns to step S116.

In step S116, determination of the state of the switch SW2 is performed, and if SW2 is ON, then the process proceeds to step S117.

In steps S117 to S122, the pixel shifting is performed as explained in the first embodiment, however, in the second embodiment, the pixel shifting is performed in accordance with the selected pixel shifting mode determined in steps S215 to S222. Therefore, the pixel shifting operation is performed in the selected pixel shifting mode if either the first or second pixel shifting mode is selected. Whereas, if it is determined in the steps S215 to S222 not to perform pixel shifting operation, image formation position is not shifted.

Then, in step S123, the completion of the pixel shifting operation (or completion of the storing of required image signals) is informed to the microcomputer LCPU, and the process proceeds to step S124. In step S124, the image synthesis processing corresponding to the selected pixel shifting mode is performed if the first or second pixel shifting mode is selected. If the pixel shifting operation was not performed, there is no need to perform image synthesis, therefore, the photographed image is directly outputted.

Then, in step S125, the obtained image is recorded and the process returns to step S102.

FIGS. 12A and 12B are views for explaining pixel shifting method when the number of image formation positions is two. FIG. 12A shows shift of an image with respect to the image sensing device IMS. As shown in FIG. 12A, the image sensing device IMS is charged and read when a given point of an image is at a position IM21. Then, after the point of the image is shifted to the position IM22, the image sensing device is charged and read again. Thereafter, the image formation position is moved back to the initial position so that the point of the image is formed at the position IM21.

FIG. 12B shows a movement of the image sensing device IMS equivalent to the aforesaid movement of the image shown in FIG. 12A, and a pixel of the image sensing device IMS which is at an initial position IG21 is moved to the position IG22 by pixel shifting operation, then moved back to the initial position IG21.

FIG. 13 is a view for explaining an image synthesis method in the second pixel shifting mode. In the second pixel shifting mode, on the basis of image signals of two images, IG21(i, j) and IG22(i, j), namely, image signals of (2.times.m.times.n) pixels, an image IMG2(u, v) expressed with (4.times.m.times.n) pixels is obtained. For this reason, pixels expressed by black dots in IMG2(u, v) are the same values as pixels in the images IG21(i, j) and IG22(i, j), and the pixels shown by white dots are interpolated with averages of the values of the neighboring four pixels (at edge, two or three pixels, instead of four pixels).

In equations,

IMG2(u=2i-1, v=2j-1).rarw.{IG21(i, j)+IG22(i, j)+IG21(i, j-1)+IG22(i-1, j)}/4 (14)

According to the second embodiment as described above, in addition to the same effects as those of the first embodiment, (4) It is possible to perform the optimum pixel shifting operation in accordance with image sensing conditions since the optimum image sensing mode is selected in accordance with the vibration in consideration of possibility of deteriorating image quality due to vibration while performing the pixel shifting operation and effect of the pixel shifting operation on resolution of the image.

Third Embodiment

In the second embodiment, the optimum pixel shifting mode is selected in accordance with vibration. In the third embodiment, whether or not the pixel shifting operation is to be performed is determined in accordance with an image sensing mode set by a user, and, if it is determined to perform pixel shifting operation, the pixel shifting mode is changed in accordance with the image sensing mode.

The control processing in the camera main body CMR in the third embodiment is basically the same as that shown in FIG. 6 explained in the first embodiment, similarly to the second embodiment. However, the process performed in step S115, i.e., "to set a pixel shifting condition" is realized by a sub-routine as shown in FIG. 14, thereby the advantage of the third embodiment can be obtained. Below, the control processing according to the third embodiment will be explained with reference to FIGS. 6 and 14. Since the processes shown in FIG. 6 have been already explained in detail in the first embodiment, they are only briefly explained in the second embodiment.

Referring to FIG. 6, when it is determined in step S102 that the switch SW1 is ON, the process proceeds to step S111. Thereafter, an IS start instruction is transmitted to the lens LNS in step S111 and parameters are received from the lens LNS in step S112. Next in steps S113 and S114, photometry and focus state detection are performed, and the obtained results are transmitted to the microcomputer LCPU of the lens LNS.

In the next step, S115, the processes shown in FIG. 14 are performed. In step S315 in FIG. 14, the status of the image mode selection switch SWMOD (image sensing mode) provided in the camera main body CMR is determined, thereby image sensing conditions, such as exposure control mode, set by the user are determined.

In step S316, whether or not the image sensing mode selected by the user is a landscape mode is determined. Landscape mode is an exposure control mode in which field depth is deepened by setting a small iris diaphragm control value (large F number). When the landscape mode is set, it is expected that the object stands still, and the camera is held still, thus vibration would not occur in most cases. Accordingly, the process proceeds to step S319, where a "high-resolution" mode for sensing four images while shifting between four image formation positions in pixel shifting operation is set. When it is determined in step S316 that the set image sensing mode is not the landscape mode, then the process proceeds to step S317.

In step S317, whether or not the selected image sensing mode is a portrait mode is determined. Portrait mode is an exposure control mode in which field depth is narrowed by setting the iris diaphragm control value to near open (small F number). Since it is expected that the conditions for photographing in the portrait mode is similar to those of the landscape mode, the process proceeds to step S319. Whereas, if it is determined in step S317 that the set image sensing mode is not the portrait mode, then the process proceeds to step S318.

In step S318, whether or not the set image sensing mode is a sport mode is determined. Sport mode is an exposure control mode for photographing a moving object as if it is not moving by shortening exposure time. When the sport mode is selected, it is expected that the object is moving and the camera may be panning. In other words, the movement of the camera similarly to the vibration by a large displacement amount is expected. Further, blurring of the object due to the movement of the object while performing pixel shifting operation is expected. Accordingly, improvement of image quality is not anticipated; on the contrary, the obtained image by performing pixel shifting operation would be unnatural. Therefore, when the sport mode is set, the process proceeds to step S321 where N.sub.SF is set to one and the pixel shifting is disabled. If it is determined in step S318 that the sport mode is not set, namely, when the set image sensing mode is not any of the landscape, portrait, and sport modes, the process proceeds to step S320, and the number of image formation positions N.sub.SF is set to two.

After one of the processes in steps S319 to S321 is performed, the process proceeds to step S322.

In step S322, information on the determined pixel shifting mode is transmitted to the microcomputer LCPU of the lens LNS.

In step S323, the type of the pixel shifting mode is displayed on the display device DISP of the camera main body CMR to inform the user of which mode is used for photographing an image.

After step S323 is completed, the process returns to step S116.

In step S116, determination of the state of the switch SW2 is performed, and if SW2 is ON, then the process proceeds to step S117.

In steps S117 to S122, the pixel shifting is performed in accordance with the selected pixel shifting mode as explained in the second embodiment.

Then, in step S123, the completion of the pixel shifting operation (or completion of the storing of required image signals) is informed to the microcomputer LCPU, and the process proceeds to step S124. In step S124, the image synthesis processing corresponding to the selected pixel shifting mode is performed as described in the second embodiment.

In step S125, the obtained image is recorded and the process returns to step S102.

According to the third embodiment as described above, in addition to the same effects as those of the first embodiment, (5) It is possible to perform an optimum pixel shifting operation suitable for movements of both a camera and an object by determining whether or not it is appropriate to perform pixel shifting operation and changing pixel shifting modes, in accordance with an image sensing mode set by the user.

Further, an image sensing mode is selected on the basis of different photographing situations which require different exposure control, however, an image sensing mode may be selected on the basis of the result of focus state detection.

Fourth Embodiment

In the first embodiment, the pixel shifting operation and the image stabilization are achieved using one shifting device, namely, the second lens group L2, which is operated in accordance with the shift amounts adjusted by the first coefficient for the pixel shifting operation and the second coefficient for the image stabilization. It is possible to use the first coefficient in an image stabilization method which is different from the aforesaid image stabilization method in order to further improve image stabilization ability.

FIG. 15 is a block diagram illustrating a detailed configuration of microcomputers CCPU and LCPU of the camera main body CMR and the lens LNS and their peripheral units according to the fourth embodiment of the present invention. This is a modified version of the block diagram shown in FIG. 5. As shown in FIG. 15, a movement vector detector 17 is added to the microcomputer CCPU of the camera main body CMR and a third coefficient generator 22a is added to the microcomputer LCPU of the lens LNS.

The movement vector detector 17 is a known circuit for detecting shifted amount, due to vibration, between two images sensed at different times, on the basis of a spatial correlation of image signals of the two images, and used in, so-called, electronic image stabilization. The two images used for movement vector detection are those obtained while performing pixel shifting operation or those obtained periodically with no relation to pixel shifting operation. Alternately, a signal from the focus state detection sensor may be used. Note, image formation positions of the two images obtained in pixel shifting operation are naturally shifted by a predetermined amount, therefore, it is necessary to correct the shifted amount in consideration of the shift amount due to the pixel shifting operation for detecting a movement vector.

Since an image stabilization (IS) system including vibration detection sensors, such as vibration-type gyroscopes, utilizing inertia is provided to the camera in the fourth embodiment, while the IS system is operated, theoretically, blurring of an image is corrected, and thus the movement vector detector 17 does not detect vibration. However, the vibration detection sensors, such as the vibration-type gyro, utilizing inertia have a defect that it can not detect vibration of a very low frequency range because of direct-current offset and drift of an output signal, for instance. Therefore, the movement vector detector 17 detects low frequency vibration while the IS system is operated. Thus, the vibration signal detected by the movement vector detector 17 is converted by the third coefficient generator 22a, and the converted value, the shift amount instruction value, calculated by the first coefficient generator 22, for the second lens group L2 for pixel shifting operation, and the shift amount instruction value, calculated by the second coefficient generator 33, for the second lens group L2 for image stabilization are synthesized in the synthesis circuit 41. Then, by operating the IS actuator 42 on the basis of the synthesized signals, the IS system capable of compensating vibration in a wide frequency range, from a low frequency to a high frequency is realized, thereby image stabilization ability is improved. Accordingly, blurring of images obtained while performing pixel shifting operation is reduced, thereby contributing to improvement of resolution of an image.

FIGS. 16 and 17 are flowcharts of control processings by the microcomputers in the camera main body CMR and in the lens LNS, respectively, according to the fourth embodiment of the present invention.

First, a flow of the control processing by the microcomputer CCPU of the camera main body CMR is explained with reference to FIGS. 1 and 16.

When the main switch (power switch) SWMN of the camera main body CMR is turned on in step S401, electric power is supplied to the microcomputer CCPU, then the process proceeds to step S402 where operation of the camera starts.

In step S402, the state of the switch SW1, which is turned on in response to the half stroke of the release button, is detected. If the SW1 is off, then the process proceeds to step S403 where an instruction to stop image stabilization (IS) operation (IS stop instruction) is transmitted to the lens LNS.

The steps S402 and S403 are repeatedly performed until the switch SW1 is turned on or the main switch SWMN is turned off.

When the switch SW1 is turned on in step S402, the process proceeds to step S411. In step S411, the microcomputer CCPU transmits an instruction to start IS operation (IS start instruction) to the microcomputer LCPU via the signal line DCL.

Next in step S412, communication for obtaining parameters which are specific to the lens, such as F number and focal length of the lens, from the microcomputer LCPU is performed.

Then, in step S413, the luminance of the object is measured by the sensor SNS, and the charging period for obtaining image signals from the image sensing device and the iris diaphragm control value are calculated, and the microcomputer CCPU transmits the calculation result to the microcomputer LCPU.

In step S414, the focus state is detected by the sensor SNS, and the result is also transmitted to the microcomputer LCPU.

In step S415, the state of the pixel shifting mode selection switch SWSF is detected as well as pixel shifting conditions, e.g., whether or not to perform pixel shifting operation and the number of image formation positions, are set on the basis of the result of the photometry performed in step S413, for example.

In step S416, the microcomputer CCPU controls the image sensing device IMS, via the driver IMDR, to charge, then transfer the charges in the image sensing device IMS.

In step S417, the image signals read at step S416 are temporarily stored in the RAM of the microcomputer CCPU.

In step S418, a movement vector is determined from image signals of two images stored in the RAM. Note, when the process in step S418 is performed for the first time, image signals representing only one image are stored in the RAM; therefore, "0" is outputted as the movement vector.

In step S419, the movement vector determined in step 418 is transmitted to the microcomputer LCPU.

Thereafter, the process proceeds to step S420, where the state of the switch SW2 which is turned on in response to the full stroke of the release button is detected. If the switch SW2 is OFF, then the process returns to step S411, and steps S411 to S419 are repeated. Whereas, if it is detected that the switch SW2 is ON, then the process proceeds to step S421.

In step S421, the same pixel shifting control as that performed in steps S117 to S122 in FIG. 6 explained in the first embodiment is performed.

In step S422, completion of pixel shifting operation (or completion of the storing of required image signals) is informed to the microcomputer LCPU.

In step S423, if image signals of a plurality of images are stored in the RAM, then they are synthesized to generate a single high-resolution image, then outputted. Whereas, if image signals of a single image are stored in the RAM, it is not possible to perform synthesis, therefore, the image signals are outputted.

In step S424, the image outputted in step S423 is stored in the memory MEM.

Accordingly, an image sensing operation is completed and the process returns to step S402. If the switch SW1 becomes ON in step S402, then the processes in step S411 and the subsequent steps are repeated, whereas, if the switch SW1 is OFF, then an instruction to stop IS operation is transmitted to the microcomputer LCPU in step S403.

FIG. 17 is the flowchart of control processing by the microcomputer LCPU of the lens LNS.

Referring to FIG. 17, when electric power is provided to the exchange lens, in step S431, in response to the ON operation of the main switch SWMN of the camera main body CMR, then the process proceeds to step S432.

In step S432, whether or not the IS start instruction is received or not is determined, and if the IS start instruction is not received from the camera main body CMR, then the process proceeds to step S433.

In step S433, whether or not the IS stop instruction is received from the camera main body CMR is determined, and if not, the process returns to step S432. If it is determined that the IS stop operation is received, then the process proceeds to step S434 where the IS operation is stopped. More specifically, the pitch and yaw actuators IACTP and IACTY are deactivated.

If the IS start instruction is received from the microcomputer CCPU while performing processes in steps S432 to S434, then the process proceeds from step S432 to step S441.

In step S441, the vibration sensors GRP and GRY are activated, and vibration signals in the pitch and yaw directions are inputted.

Step S142 corresponds to step S412 in FIG. 16, and in response to requests from the microcomputer CCPU, the microcomputer LCPU of the lens LNS transmits the parameters which are specific to the lens LNS to the camera main body CMR.

In step S443, the zoom encoder ZENC and the focus encoder FENC are checked in order to detect zooming and focus states of the lens system.

In step S444, on the basis of the detection result in step S443, the first coefficient for pixel shifting and the second coefficient for IS operation are read from a table stored in the ROM of the microcomputer LCPU.

In step S445, the pitch and yaw actuators IACTP and IACTY are operated on the basis of the vibration signals obtained in step S441 and the second coefficient obtained in step S444 to reduce blurring of an image due to vibration.

In step S446, the microcomputer LCPU operates the iris diaphragm DFM via the iris diaphragm actuator DACT on the basis of the information on a photometry result transmitted from the microcomputer CCPU to control the luminous exposure.

In step S447, the focusing actuator FACT is operated on the basis of the information on the focus state detection obtained from the microcomputer CCPU to adjust focus.

In step S448, the movement vector transmitted in step S419 in FIG. 16 is received.

In step S449, shift amount instruction values for the second lens group L2 for compensating blurring are generated on the basis of the movement vector signal obtained in step S448 and the first coefficient obtained in step S444, and further added to shift amount instruction values obtained from the vibration sensors GRP and GRY and the second coefficient obtained in step S444. Then, the pitch and yaw actuators IACTP and IACTY are driven in accordance with the added signal, thereby realizing image stabilization in wide frequency range of vibration.

Next, in step S450, whether or not a timing pulse for triggering the pixel shifting operation is received is determined. If no timing pulse is received, the process returns to step S441, and the IS operation, the iris diaphragm control, the focus adjustment, and reception of the moving vector are repeatedly performed. When it is determined in step S450 that the timing pulse is received, the process proceeds to step S451.

In step S451, signals having reference waveforms for driving the second lens group L2 in the pitch and yaw directions for pixel shifting operation are generated by the pixel shifting signal generator 21.

In step S452, the amplitudes of the signals having reference waveform, generated in step S451, are multiplied by the first coefficient read in step S444, thereby pixel shifting signals which compensates the effect of the decentering sensitivity of the second lens group L2 are generated. Thereafter, the generated pixel shifting signals are synthesized with signals for the IS operation (vibration compensation signals), generated in step S449, in the synthesis circuit 41. By operating the pitch and yaw actuators IACTP and IACTY in accordance with the synthesized signals, the IS operation and the pixel shifting operation are performed simultaneously and precisely.

In step S453, whether or not a signal indicating completion of the pixel shifting operation is received from the microcomputer CCPU is determined, and if it is not, the process returns to step S450 because the pixel shifting is not finished. Then, the process waits the next timing pulse. The processes in steps S450 to S453 are repeated for a predetermined number of times, and when the signal indicating completion of the pixel shifting operation is transmitted, the process returns from step S453 to step S451.

Then, if the IS start instruction is not received in step S432 and the IS stop instruction is detected in step S433, then the IS operation is stopped in step S434; more specifically, the pitch and yaw actuators IACTP and IACTY are deactivated, and a series of lens control operation relating to the image sensing operation is completed.

According to the fourth embodiment as described above, in addition to the same effects as those of the first embodiment, (6) Correctable frequency range of vibration is widened by converting a pixel shifting signal using the first coefficient, converting a movement vector signal using the first coefficient, converting a vibration signal using the second coefficient, and driving the optical system for image stabilization in accordance with a synthesized signal of the above three converted signals. (7) Precise vibration compensation is realized even when zooming and focus states are changed by using the first and second coefficients corresponding to the zooming and focus states.

Note, the advantage of the fourth embodiment is obtained with or without pixel shifting function.

Modifications

In the first to fourth embodiments, a lens group in the optical system is moved in the orthogonal direction with respect to the optical axis of the optical system, thereby used as an image shifting means for realizing image stabilization and pixel shifting operation by utilizing optical shifting feature of the lens group. Alternately, it is possible to use a pair of transparent plates between which transparent liquid is fille d, so-called, a variable apical angle prism is used.

Further, any type of optical systems may be used as the optical system in the second and third embodiments. In addition, the features of the second and third embodiments are achieved regardless of the existence of image stabilization function.

Fifth Embodiment

FIG. 18 is a flowchart of control processing performed by the microcomputer CCPU of the camera main body according to a fifth embodiment of the present invention. The processes in FIG. 18 are same as those in FIG. 6 explained in the first embodiment except step S115.

In step S1115 in FIG. 18, the state of the pixel shifting mode selection switch SWSF is detected, and image sensing conditions, such as the type of pixel shifting operation and the type of luminous exposure control of the image sensing device, are set on the basis of results of photometry and focus state obtained in steps S113 and 114. The detail will be explained later.

Note, the control processing by the microcomputer LCPU of the lens LNS is the same as the one shown in FIG. 7 explained in the first embodiment.

Next, method for controlling luminous exposure according to the fifth embodiment will be explained in detail with reference to FIGS. 19 to 21. First, referring to FIG. 20, a typical photo-sensing characteristics of an image sensing device is explained.

Lines in a graph shown in FIG. 20 show characteristics of a film and an image sensing device, and the abscissa indicates luminous exposure and the ordinate indicates transparency of image recorded on the film or output voltage of the image sensing device. If an object is sensed with a fixed iris diaphragm control value at a fixed shutter speed, the abscissa can be considered as luminance of the object.

In FIG. 20, a broken line represents the characteristics of a silver-halide film, and a solid line represents the characteristics of the image sensing device, such as CCD. While the silver-halide film has a wide dynamic range, the dynamic range of the image sensing device is narrow, and the image sensing device can only reproduce an image of the object in a luminance range between log H1 and log H2.

FIG. 19 is a flowchart showing processes for setting luminous exposure, which shows detailed processes in step S1115 in FIG. 18.

In step S181, the image of the object whose luminance is measured in step S113 is divided into a plurality of blocks. Method for dividing the image is explained with reference to FIG. 21.

FIG. 21 is a view showing an area where focus state detection and the photometry, performed by the sensor SNS shown in FIG. 1, are performed. The detection area of the sensor SNS is roughly the same as the photo-sensing area of the image sensing device IMS, and is divided into 96 (=8.times.12) areas which are surrounded by thin lines in FIG. 21. Focus state and luminance can be individually detected in each divided area. Note, the sensor SNS can be realized by a pair of secondary focusing optical systems and focus state detection means, configured with two-dimensional image sensor, provided in respective secondary optical systems, adopting secondary phase difference detection method, for example.

On the sensor SNS, an image of an object is formed, as shown in FIG. 21. On the basis of the results of focus state detection and photometry performed on the image, the image is divided into five blocks, BK1 to BK5, surrounded by bold lines in FIG. 21, in this case. The division of the area is performed so that each block consists of the areas having roughly the same focus state and luminance levels out of the aforesaid 96 areas.

In step S182 in FIG. 19, a block including a main object is inferred from the divided blocks BK1 to BK5 on the basis of a predetermined algorithm. More specifically, the maim object can be inferred using the following principles: i) If an area on which focus state detection is to be performed is designated by a user, then an object in the area is the main object; ii) If the camera is set to a mode for automatically detecting a main object, then an object which is near the center of an image and at relatively short distance from the camera is the main object; iii) If an area on which photometry is to be performed is designated by the user, then an object included in the area is the main object; and iv) If a camera has a function for detecting the direction of the line of sight of the user, then an object on the line of sight is the main object.

In the fifth embodiment, one of the above principles is used, and it is assumed that a person included in the block BK3 is determined as the main object.

Then in step S183, on the basis of the divided blocks and the result of the main object inference, the blocks are made into groups. Since the main purpose of the fifth embodiment is luminous exposure control while performing pixel shifting operation, the number of luminous exposures used for sensing images in the pixel shifting operation should be equal or less than the number of image formation positions. Accordingly, the blocks are further collected into groups of a smaller number; more specifically, the number which is equal or less than that of the image formation positions in the pixel shifting operation. For example, the blocks BK1 and BK2 form a high luminance group GP1, and the block BK3, which includes the main object, forms a medium luminance group GP2, and the groups BK4 and BK5 form a low luminance group GP3. Namely,

BK1, BK2 .fwdarw. GP1 (high luminance group) BK3 .fwdarw. GP2 (medium luminance group) BK4, BK5 .fwdarw. GP3 (low luminance group)

Next, in step S184, an average luminance of each group determined in step S183 is calculated.

In step S185, proper luminous exposures E.sub.1 to E.sub.n (n=3, in this case) corresponding to respective average luminances are calculated. Then in step S186, iris diaphragm control values and exposure times t.sub.e1 to t.sub.en for obtaining the proper luminous exposures E.sub.1 to E.sub.n are calculated on the basis of a predetermined program.

After step S186, the process returns to step S116 in FIG. 18, and the pixel shifting operation and exposure (charging) of the image sensing device IMS are performed.

FIG. 22 is a timing chart for explaining operations of the camera main body CMR and the lens LNS shown in the flowcharts in FIGS. 18, 19 and 7.

Signal a and b show states of the switches SW1 and SW2, respectively; a signal c is a timing signal for pixel shifting operation; a signal d is for charging in the image sensing device IMS; signals e and f have reference waveforms for pixel shifting operation (referred to as "pixel shifting reference signals" hereinafter) in the pitch direction and the yaw direction, respectively; and signals g and h are vibration signals, in the pitch and yaw directions, detected by the vibration sensors GRP and GRY, respectively. Here, vibration shift waveforms obtained by processing the detected signals by integration, for example, are shown. Further, signals i and j are operation signals for shifting the second lens group L2 in the pitch and yaw directions, respectively.

Next, overall operation shown in the flowcharts in FIGS. 18, 19 and 7 is explained with reference to the timing chart shown in FIG. 22.

When the switch SW1 is turned on at time t.sub.1, the vibration signals g and h are outputted. In turn, the second lens group L2 is operated as shown in the waveforms of the signals i and j in accordance with the vibration signals multiplied by the second coefficient.

Then the switch SW2 is turned on at time t.sub.2, and after a predetermined period of time elapses from the time t.sub.2, a timing pulse TP1 is generated at time t.sub.11. In turn, the photo-sensing elements of the image sensing device IMS are charged between time t.sub.12 and time t.sub.13, namely, the exposure time t.sub.e2, calculated in step S186 in FIG. 19, which is suitable for the medium luminance group GP2.

When the charging operation ends at the time t.sub.13, the charges are transferred and read. At the same time, the pixel shifting reference signal f in the yaw direction is generated. Accordingly, the second lens group L2 is operated in the yaw direction on the basis of instruction values shown by the signal j, obtained by adding the pixel shifting reference signal f multiplied by the first coefficient and the vibration signal h, multiplied by the second coefficient.

After a predetermined period of time has passed since time t.sub.11, the second timing pulse TP2 is generated at time t.sub.21. Then, similarly to above, the photo-sensing elements of the image sensing device IMS are charged between time t.sub.22 and time t.sub.23. The exposure time used this time is the exposure time t.sub.e1, calculated in step S186 in FIG. 19, which is suitable for the high luminance group GP1.

When the charging operation ends at time t.sub.23, the charges are transferred and read. At the same time, the pixel shifting reference signal e in the pitch direction is generated. Accordingly, the second lens group L2 is operated in the pitch direction on the basis of instruction values shown by the signal i obtained by adding the pixel shifting reference signal e multiplied by the first coefficient and the vibration signal g multiplied by the second coefficient.

After a predetermined period of time has passed since time t.sub.21, the third timing pulse TP3 is generated at time t.sub.31. Then, similarly to above, the photo-sensing elements of the image sensing device IMS are charged between time t.sub.32 and time t.sub.33. The exposure time used this time is the exposure time t.sub.e2, calculated in step S186 in FIG. 19, which is suitable for the medium luminance group GP2.

When the charging operation ends at time t.sub.33, the charges are transferred and read. At the same time, the value of the pixel shifting reference signal f in the yaw direction is changed to the initial value. Accordingly, the second lens group L2 is operated in the yaw direction on the basis of instruction values shown by the signal j corresponding to the vibration signal h multiplied by the second coefficient.

After a predetermined period of time has passed since time t.sub.31, the last timing pulse TP4 is generated at time t.sub.41. Then, similarly to above, the photo-sensing elements of the image sensing device IMS are charged between time t.sub.42 and time t.sub.43. The exposure time used this time is the exposure time t.sub.e3, calculat