Apple Intelligence is once again helping photographers add to their images in Photos for iOS 27. So long as you’re fine with it guessing what you didn’t include in the original shot.
The Photos app has been the beneficiary of a number of generative AI-based tools that use Apple Intelligence. All to try and make your photographs look great.
It all started with Clean Up, which let you eliminate unwanted and unsightly elements from your images, and then intelligently fill in the missing pixels. This is also part of Spatial Reframe, which adds elements when you move the camera and show things that were not originally caught.
In another new AI-based tool, Extend, Apple uses the same concept but thinks bigger. It’s now generating elements that are outside of the frame.
Extending the crop
Editors working on images have a few choices when it comes to changing the composition of the shot. At its most basic level, this can be a simple crop of the image.
That is, lopping off edges from the shot so that a smaller part of the overall frame occupies more of the finished picture. Think of trimming the side of a photograph to get a thumb or an annoying bit of the photo off-frame.
Extending an image in Photos for iOS 27
However, if you felt that the image needed to be wider than what you actually shot, it’s a different type of problem. You could create a composite, using imagery from elsewhere to fill in the newly created and blank space.
Extend in Photos is the same concept. It’s just that it comes up with the imagery that fills that blank space.
Apple Intelligence looks at the surrounding pixels and the rest of the image to determine what goes into the empty space. It then creates its best guess and fills the void, using its knowledge base.
Educated AI guesses
The main point of extending the frame is to create something aesthetically pleasing to the user. It has to work out what could be there and create some form of realistic element to fit into the space that could plausibly exist.
This is not the same as generating pixels showing something that does actually exist in that space. Without looking at other reference material, no AI will be able to accomplish that without a high amount of luck.
Handily, most people who would use this will look for generations that are good enough to work. Not necessarily absolute reality.
Original shot [left], AI-extended version [right]
In a test image of a cat on a windowsill, it neatly created an extra curtain to one side, more dead plants to the other. It even made more window above the cat’s head, complete with condensation.
However, I know full well that the painted window frame on the left of the shot is completely incorrect. It should be plastic frames, but instead, it’s deteriorating painted wood.
This certainly doesn’t mean the resulting expansion of the image is ugly. Far from it.
Late night in Cardiff original [left] and expansion [right]. Note the left pole’s position and odd-shaped sign.
Another shot of a street corner and a castle late at night is decently enlarged too. Bollards, poles, and other traffic are all generated well, and most of it fits the scene.
That said, a visible back of a road sign seems off, in part because it seems like a slightly incorrect shape. The pole’s placement also doesn’t quite line up with the road layout either.
The original shot in an Apple Store in Rome [left], the extended version [middle], and a real shot of what’s actually in the expanded bit [right]
A shot of a real Apple Store staircase in Rome was used to try and compare the AI’s guesswork with reality. The resulting expansion simply added more stairs and glass to the shot, which again looked appropriate.
In reality, there were doors and no extra steps, but Photos didn’t know about that.
Hallucinations are possible
Don’t misunderstand the testing here. We are more interested in Extend coming up with a plausible way to add extra scenery and objects to a photograph.
Plausible and looking good without errors is the game here. And overall, it does quite well.
But even so, it is still susceptible to the occasional misstep.
An airport in Rome [left], and the generated expansion [right]. Note the floating truck…
In one shot of an airport in Rome, the initial result seems plausible. The buildings extend sensibly into the background, and at first glance, everything seems plausible.
That is, until you check out the yellow and red vehicle on the right-hand side. It appears that the AI decided that the weird tire and metal siding of something was an oddly short truck.
It would’ve been a good guess, had it not been rendered to be floating about a foot off the ground.
A close-up of the generated truck [left], and the real-world vehicle that was actually there [right]
Searching for other shots from the time reveals it is a movable conveyor used to load luggage on and off aircraft.
To be fair to it, the original image only had a small section of a vehicle showing, and it does look like the back of a truck if you ignore the shadows. It does have to generate based on what information it has available, no matter how small the item fragment is.
This does mean that, if you have unusual elements sticking in from the sides of your original shot, you may want to fix that first. You could use Clean Up to get rid of the oddity before extending the frame, otherwise you’re leaving it all up to chance.
Overall, Extend is a logical continuation of the generative AI tool that Apple had before, and one that works reasonably well.
It has the potential to go wrong, just like the nightmare fuel generated by Spatial Reframe if you’re not careful. So long as you aren’t expecting perfect reality and just want the elements to be “good enough,” then Extend is up to the task.
