91����

On the screen, a crisp image takes shape: a stone landscape worthy of Escher, with massive faces emerging from shadows and cracks.

91���� anthropologist Carl Lipo zooms in and more details come into view: the jutting noses and chins of the massive Easter Island statues known as moai, still in the quarry where they originated. There are no good maps of the nearly inaccessible volcanic crater, with its high, steep walls — that is, until now. A swipe of the finger, and the image changes angles — an interactive and nearly perfect three-dimensional map, worthy of science fiction.

“With a drone, we were able to systematically map this at 30-meter increments to create a centimeter-level, precise three-dimensional model. There are 22,000 photos stitched together with software,” Lipo explains. “In some ways, it’s actually better than being there.”

Early drones and satellites couldn’t produce the resolution needed for the project on remote Rapa Nui, Easter Island’s native name. To capture the data, the drones needed to fly methodically over the site, with 80% of each image overlapping with the next, providing the software with multiple points to create the model.

Drones, coupled with advances in chip manufacturing that have allowed for the miniaturization of computing technology and the advent of artificial intelligence, have transformed research. At 91����, drones are being used to detect landmines and agricultural pests, map lost battlefields, repair infrastructure — and even to find you a parking space.

“It’s a gateway to looking at things from different disciplinary perspectives,” says Adam Mathews, an associate professor of geography and Fulbright Fellow who recently worked with geologists in Ireland to map and examine peatlands.

The drone revolution

While affordable drones are a recent innovation, the value of a bird’s eye view is as old as time.

“From a conservation ethic, as an archaeologist, you want to impact the record as little as you possibly can. Every time you dig a hole, you’re destroying the thing that you’re trying to study,” explains Lipo, a professor of anthropology and associate dean for research and programs for Harpur College of Arts and Sciences.

In the early days, that meant leveraging aerial photography. The U.S. Department of Agriculture began systematically mapping North America farmland in the 1930s — which proved a boon to archaeologists and geographers.

Remote locations such as Rapa Nui, however, lack aerial photography and satellite imagery typically doesn’t have the fine detail needed for research. Early in his career, Lipo used kites and then small blimps, equipped with a digital camera and a radio-controlled mechanical hand to press the shutter button. There are no helium tanks on the island, so researchers ended up using hydrogen from the local weather station.

“We had this bomb flying above us and powerlines overhead,” Lipo remembers. “It was risky, although we were able to get some photos.”

Lipo began his research on the island in 2005; the first drones hit the consumer market in 2007. The early ones were home-built and fragile affairs that had to be flown manually. Jayson Boubin, an assistant professor of computer science at 91����, had his start with them in high school, when he built and flew quadcopters as a hobby.

The technological landscape changed when DJI released the Phantom 1, the first consumer drone. Unlike earlier drones, the next generation largely piloted itself. Camera and sensor technology developed in tandem, expanding to multispectral sensing, LIDAR and more. Technologies such as optics, processors and improved hyperspectral and multispectral cameras have allowed for more versatile applications.

“We’ve seen significant improvements in onboard computation,” explains Boubin, holding up a Raspberry Pi.

The palm-sized circuit board is essentially a small computer that easily fits onboard a drone. Add a tensor processing unit — which looks like a small silver box — and the same palm-sized circuit board will be a bit heavier, but much faster and capable of machine learning workloads.

“We’re able to do onboard, real-time three-dimensional SLAM, which we weren’t able to do 10 years ago at all,” says Boubin, using an acronym for simultaneous localization and mapping. “As the hardware is improving, the software is also improving.”

Digital cameras encode information as red, green and blue (RGB) pixels, which capture certain wavelengths of light while disregarding others. Because of low resolution, a typical RGB camera would make it difficult to determine whether a surface was composed of metal or cloth, for example. Hyperspectral imaging (HSI) cameras encode light on a much finer, more detailed scale. The problem: These cameras generate massive amounts of data, enough to fill a flash drive in around 30 seconds. To solve this, that data must be processed in real time — as quickly as it’s generated.

Boubin has been working on the problem for the past few years, and his lab is the only one in the world currently up to the task. Data from drone-mounted HSI cameras can detect harmful algal blooms in the Finger Lakes, landmines in conflict zones and even World War II battlefields hidden under the tree canopy.

“In the old days, we imagined what we could do but we could never get there,” Lipo muses. “And now you press a button and it goes up; the drone flies back and forth and lands itself. You download the data into the computer, which builds you a three-dimensional map.”

Maps and models

Three-dimensional maps and models are crucial to many kinds of research. Thomas Pingel, an associate professor of geography, uses them to gauge the effects of global warming in cities and the relationship between surface and air temperatures. Drones equipped with thermal cameras allow him to register those temperature gradations in high detail.

With an off-the-shelf commercial drone, Pingel and graduate student Sharifa Karwandyar use artificial intelligence to detect landmines in camera images, a strategy that may someday save lives around the world. Another project, spearheaded by graduate student Peter Vailakis, uses drones to track deer in the campus Nature Preserve. Deer blend in with the environment, so they are difficult to discern in a static image. Instead, researchers watch through the camera as the drone flies a pre-programmed route; when a deer is spotted, the drone switches to manual control and is steered closer to the target.

In collaboration with the Defense POW/MIA Accounting Agency, Lipo and Pingel have used drone-mounted LIDAR to map former World War II battlefields in Guam and the Solomon Islands to determine where fallen soldiers may still lie. In a further development, they are working with Boubin, Distinguished Professor of Chemistry Chuan-Jian “CJ” Zhong and Assistant Professor of Anthropology Laure Spake to build a “digital nose” that would allow drones to detect human remains.

Decomposing bodies produce organic molecules such as putrescene, which cadaver dogs can detect at low levels. These molecules persist over long periods of time and are detectable for hundreds of years under certain conditions. In addition to battlefields, this research could also contribute to forensic cases, helping locate victims of accidents or crimes.

“One of the reasons why people aren’t found is because they were lost in forested, rugged terrain,” Lipo says. “What we need are drones that can go into those landscapes and map underneath the vegetation.”

A drone flying under the canopy requires a top-notch sensor suite to determine obstacles, artificial intelligence to make complex navigational choices — and SLAM. While computationally heavy, simultaneous localization and mapping enables a drone to both determine and remember its location in respect to the objects around it.

In a video created by Zain Nasir ’22, MS ’24, today a machine learning engineer focused on cancer research, a drone flies over the courtyard outside the Engineering Building. But the black-and-white landscape is pocked with specks of light: The program is identifying structuring features of the landscape.

The light gathers on the corners of pavers and the canopy of a slender tree.

“A feature isn’t just an object. It’s a collection of gradients that are resident in the image that usually represent a corner of the object or some piece of it,” Boubin explains.

Outside of built environments, SLAM becomes more challenging; forests grow and change, and branches blow in the wind. Working with Pingel, Boubin is using LIDAR scans as an intermediate SLAM map to locate, for example, the largest trees in the Nature Preserve. This data then allows the drone to formulate an initial trajectory amid the trees, using SLAM processing to avoid collisions.

“Flying under the canopy is the next big frontier,” Pingel says.

Solving problems

Drones can do more than create better maps. Someday, they may repair solar panels and wind turbines, deliver packages, find parking spaces — or ticket you for parking in the wrong one.

Research from Yong Wang, an associate professor of systems science and industrial engineering, functions as a proof of concept, showcasing the possibilities of drone-based technology. Take parking, for example. In one scenario, a campus safety officer opens the trunk of their vehicle to release several drones, which methodically travel up and down campus lots, scanning license plates and checking them against the parking registry. If the number isn’t found, the system would alert a technician to issue a parking ticket.

In another application, drones scan the lot for open parking spaces, which are then relayed to drivers using an app. There are still some kinks to work out, Wang notes; the algorithm students tested to map those spaces failed to detect double-parked cars and sometimes mistook the sidewalk for a parking space.

His lab has also done work in hybrid delivery systems, in which drones work in concert with a driver to deliver packages. Back at the warehouse, drones can conduct inventories more accurately and efficiently than human beings, signaling the manager when supplies are running low. Drones are also more efficient at inspecting solar panels and wind turbines, using both color and thermal imaging to detect areas in need of repair.

“In this case, you can send the technician there; they don’t need to inspect each panel but focus on the ones with potential damage. You can increase your efficiency and reduce your labor workload,” Wang says.

Routing strategies need to consider a drone’s limited battery life; after around 30 minutes, a unit needs to recharge. Wang’s lab is looking at how routing strategies can accomplish the needed work, while taking the drone’s energy needs into account.

So is Kaiyan Yu, an associate professor of mechanical engineering who uses drones in a range of robotics-based projects. In one complex robotic operation to repair highway cracks, drones monitor traffic from above and signal ground-based robots that place signs or traffic cones. Another robot is then used to detect and fill the surface cracks.

Yu is working on a National Science Foundation proposal to use robot teams in agriculture in collaboration with Boubin, School of Computing Associate Professor Shiqi Zhang and local farmers. In this system, low-flying drones are deployed to monitor field health. When a problem is detected — say the crops need pesticide, fertilizer or water — the drone emits a signal. Ground-based robots may be summoned for a closer look at the problem area, or to apply the needed treatment.

“We’re trying to develop algorithms to use these tools efficiently so farmers can save energy and save money,” Yu says. “Eventually, we will be able to monitor crops at set periods of time. They won’t need to go into the field themselves to monitor the conditions.”

Yu’s role is to optimize the use of robots in decision-making and overcome logistical challenges — namely, the robot team’s timing and energy consumption. To address this, researchers need to optimize the planning of where and when robots should be deployed for field monitoring.

Also on the agricultural front, Boubin’s focus is the spotted lanternfly, an invasive species with the potential to devastate orchards and viticulture. Using drones to detect pests is challenging, he notes; as airborne vehicles, drones are loud and displace the air, which can disperse the insects they’re trying to detect.

Detecting the lanternflies as egg masses — before they hatch into short-lived adults — is a better solution. Researchers are using machine learning to identify these egg masses, with the help of farmers from around the country.

“A lot of the most interesting science is motivated by actual applications and by stakeholders. I can collect data and send it back to a farmer, or I can find landmines for removal or spotted lanternflies so someone’s vineyard doesn’t get infested,” Boubin says. “Interacting directly, seeing what the problems are and how to solve them — that’s what I want to do with my life.”