On Monday, October 9th, Greg Crutsinger woke to an ordinary sort of day. He was looking forward to the week. He was launching a new company called Drone Scholars. His wife had been promoted to an executive director in the non-profit she worked for. His daughter was on her way to being potty trained. Things were good until he heard about the fires that had happened in the night before just north of the San Francisco Bay area in California, not far from where he lived.
Fires are normal in California, part of the nature of the state. The plants of California are adapted to fire. The animals too. But these fires were different. They were hotter and moving faster than perhaps any fire before. So hot, that houses melted into confetti. So hot, cars crumbled and some were blown over. So fast, that whole neighborhoods in Sonoma county disappeared in minutes, leaving over 40 lives lost (still dozens unaccounted for) and 10s of thousands evacuated. As the fires spread over the next few days, Greg could smell their smoke. He saw the images on the news, but the fires affected his daily life only incidentally. His wife had to evacuate her staff out of the area office. His daughter’s daycare was canceled because of the bad air quality. It was on one of the days when he was at home watching his daughter that he got a call. The sheriff’s department was requesting assistance.
Greg operating a drone
I first met Greg when he was just beginning his work on goldenrod plants and the insects dependent on them at the University of Tennessee. He started his career as an ecologist. He was a clever graduate student, smugly clever even, out to make his mark. I was a postdoctoral researcher in the same lab. As a graduate student, Greg did a beautiful experiment in which he tested whether more genetically diverse clumps of plants tend to support a greater diversity of insects. Does diversity beget diversity? They do, it does. That experiment led him to discovery after discovery, a PhD after just four years, a prestigious Miller postdoc at UC Berkeley, and then, not long after, a job as an assistant professor at a top tier research institution in Canada. He was having the success scientists hope for, early success, tenure, great students, and the joys of revealing new truths. But as his career went on he found himself drawn to accomplishing real change rather than the abstraction that had come to dominate his work. He was drawn, in particular, to ways in which technology might help ecologists manage life, outside, on the ground. Also, he’d grown weary of some of the less pleasant parts of being an academic, the constant vying. He spent too much time in meetings listening to deans describe budget cuts and colleagues wax poetically about their own work. He spent too little time actually making discoveries, much less doing something useful. He decided to join a Silicon Valley startup working on developing new technologies to make drones more useful. He gave back the keys to his university office, let his Canadian visa lapse, and moved to California.
After a few years working with drones, Greg found himself on his third company. The industry was fast paced, which kept things fresh and new, but it was also, as a consequence, constantly in flux. Companies rose and fell. Greg switched from one job to another. His accomplishments felt more tangible than they had in academia, but the job was a hustle, a hustle to be the first, to get things done, to conquer markets. The work was good, but, once more, just like academia had been before, all enveloping. Then, just this year, Greg decided he wanted to spend more time at home with his family and less time on the road. So, he started his own drone consulting business. That’s when the sheriff’s office called, in precisely the moment in which he was ready to relax a little.
The officers wanted Greg to help them map a burned out mobile home park in which people had already died. At first, the work Greg did was like that done elsewhere by other people with other drones. At the mobile park, he directed the officers to fly specific drone routes for capturing the right overlap in photos and downloaded their data. He loaded it to his laptop that he plugged into a running generator on scene while search and rescue teams with dogs combed through what was left of the homes. The setting was new and horrible, but the steps were routine.
After about 90 minutes of processing, Greg presented high resolution 2D and 3D models of what had and had not burned to the sheriff (view below). The sheriff thanked Greg. Apart from the setting, the exchange was almost ordinary.
Greg wasn’t being paid for the work, but it felt important. The particular drone team he was working with was one of the most active in the U.S. for public safety. He had helped them before on some high profile fires, but nothing even close to this disaster. He knew the sheriff’s office could use the maps, with a resolution of roughly a square inch per pixel, to see all those things that you might ordinarily have to walk around to view first hand. The various agencies working the fires could use them to preserve the original scene and location of materials before recovery teams started moving stuff. They could also take detailed measurements from the 3D models and identify hazardous areas.
When the day was almost done, by which time Greg was feeling pretty spent, the commander of the sheriff’s UAV team told Greg to jump in his SUV. They’d head over to a larger neighborhood that had burned called Coffey Park. As he and the sheriff arrived in Coffey Park, it quickly became clear that the task ahead was of an entirely different scale. They were surrounded by acres and acres of burned out homes, a patchwork of ash where an entire community once stood. The color was sucked out of the landscape. The color had also left Greg’s face. He was exhausted. Yet, he knew that if he could map the site now, in the waning light, that it would be much more useful than if he did it the next day. The sheriff parked the car in an open place. The open space was, Greg soon realized, a cul-de-sac in which a kid’s basketball hoop and crooked chimneys stood as monuments to ordinary lives.
Photo by Greg Crutsinger
Mapping a patch the size of Coffey Park wouldn’t be a problem under normal circumstances. But these circumstances were far from normal. The whole fire zone was under a temporary flight restriction (called a TFR…similar to what happens in an area during the Superbowl or when the president might fly into an area). The sheriff’s office needed a special waiver from the Federal Aviation Administration to even fly. The permit was obtained, but the drone team could fly no higher than 100 ft because of the number of helicopters in the area. Typically, Greg would map a landscape from four times that height. Flying four times lower than normal meant it would take Greg four times as long to get the job done and that the job would require four times as many photos, thousands and thousands of photos.
Photo by Greg Crutsinger
Drone mapping has become fairly automated. Drone pilots now use apps on a smart phone or tablets ito make a flight plan. This mission is like a zig-zag grid or “lawnmower pattern”. Photos are triggered automatically. All the photos are then loaded into a photogrammetry program to be stitched together into a larger map. The software corrects the photos for perspective so everything is from the perspective of a bird flying by, bird’s eye view.
Greg sat down and got to work. He would load a mission to one drone and hand it off to an officer to go fly it, while planning another mission for the next sheriff’s drone. The group didn’t have a high tech command center. They had Greg, sitting on a brick circle on which a child’s baseketball had been bouncing just the day before. Greg had to program the routes for the drones manually, given what he could tell about the site and its geography by using his phone trying to zoom in on Google maps to see the streets, since the homes were all gone. There was a constant hum of a generator to charge batteries. Greg had the feeling that he was forgetting things, that he could easily miss something important, but he plugged on. Eventually, night came and they all went home only to return again at day break. In the few hours in between Greg had a chance to print out a map of the neighborhood streets on his printer at home. The printer was running out of ink, but had just enough. Greg then sharpied out the day’s new routes on the print out. It reminded him of being a scientist again checking off experimental plots one by one, and improvising when need be. Sometimes the glitchy apps would crash on the phone or the drone would fail to record photos, wasting a whole battery and a flight would need redone. Ultimately, it would take about two dozen flights (and as many batteries) to map the Coffey Park neighborhood….literally down to the sq inch.
During the ordeal, Greg noticed strange things about the fire. How it burned some things and left others untouched. How it must have been trash pickup day the day after the fire. There were small piles of melted glass bottles in front of each home with the plastic recycling bins having burned away. How all the figs remained roasted on a fig tree while the leaves were gone. In such moments, there are no right things to say, no perfect observations to make. Entire lives had been dissolved into ash, bits of plastic, and the vestiges here and there that remained where the fire, for whatever reason, had been a little less hot.
Even in this scene, Greg felt the power of human generosity, of what the kindness of strangers working together can yield. Ecologists and sheriffs don’t often work together and yet here they were. Also present were volunteers who drifted down the ashen streets bearing food, gatorade, and coffee. Breakfast burritos suddenly appeared, at one point, on the back of a police cruiser and, for that moment, spirits lifted, buoyed by beans. The disaster was bringing out the best in the community. The governor even drove by at one point in a long convoy of officials in SUVs. The convoy didn’t stop and the team just kept mapping…albeit with growing headaches from the fumes coming off the ashes, the fumes and exhaustion.
As Greg made map after map he had an idea. While he no longer works as an ecologist, he retains an ecologist’s sensibilities. He thinks constantly about data and about how a new perspective on a problem can reveal new truths. He’d wondered again and again about the possibility of linking the data taken from multiple drones into a single image, a single vision of the world, be that world a burned landscape or anything else. At least in theory, Greg knew, one could take many maps, link them digitally, and then, in having done so, create a drone map of an entire series of sites, not only from a bird’s eye perspective, but in three dimensions. It wouldn’t be hard, or at least it wouldn’t be hard thanks to Greg’s time in the drone industry. He could use off-the-shelf consumer drones, smartphones, and a clever app or two. He just hadn’t had a great use for it. Here, suddenly, surrounding him, was the use.
So, once more the scientist, Greg tried an experiment in the field, only instead of undergrads hired for the summer season, he had county law enforcement officials. Greg and one of the officers loaded up another app on their phones. This one was called Hangar 360. Greg reached out to some friends at Hangar Technology in Austin, TX. While the Hangar platform is designed for industrial data collection, Greg thought their Hangar 360 app could be the perfect tool to help him collect tons of imagery and fast.
With a simple one-button push on the phone, the officer sent a drone up to take a couple dozen pictures in a repeated circles with the camera pointed at 0, 45, and then 90 degrees straight down. The drone then landed, transferring the photos from drone to phone and then off to the cloud for processing. In less than 20 minutes, he had a web link ready to a 360 degree, high-resolution panoramic of the scene that he texted off to the commanding officer. It couldn’t get much easier than that. Here then was an image not from a bird’s eye view but instead from, well, something like the perspective of a hovering bird, a hawk say, pausing to search for a mouse from a hundred feet up. The very first one of these panoramic images, these hawk’s eye views, was both incredible and terrifying. To a much greater extent than the flat images, it showed the dimensions of the losses in Coffey Park.
Click the Hangar logo on the map to choose viewing locations and other options
Greg and the officer decided to continue throughout the neighborhood to collect more panoramics while the rest of the team finished the final few mapping flights. The panoramic images of the whole neighborhood took only a few hours (and a few batteries), compared to the days of dozens of batteries required by the traditional approach. At the end of it all, Greg had a computer filled with 10,000 photos, 50 Gigabytes in all, a disaster scene turned into pixels.
It took Greg another 2 days of processing (so much for his new relaxing career move) the thousands of photos and reaching the limits of his home laptop. He had never worked with such massive datasets before and had to call in technical assistance from the San Francisco office of Pix4D. With their help, the 2D dataset was done….all while additional panoramics kept rolling in from the officers still out in the field in other hard hit neighborhoods.
The next step was to visualize the data for Sonoma county office, who was leading the efforts on the ground. The problem was the end maps were several gigabytes and would need to be broken down into tiles. So, Greg called up the Mapbox office in San Francisco. He didn’t actually know anyone this office, but still they answered his plea for assistance and got right to work. They too wanted to help. Communities and their businesses were coming together. Within 24 hours, they had a map viewable online with streets and even the individual addresses of each home, or where each home had been. The Mapbox team was also able to overlay the panoramic images from the neighborhoods in which they had been taken.
You can see these maps here (camera icons depict 360 panoramics).
What they show is a series of images stitched together to show, in both 2D and 360 degree detail, the neighborhood in extreme high resolution. By connecting the maps, you (and all the public agencies) can see each burn in detail but also context. Ultimately, the maps were made public so that residents could look at their neighborhood before the road blocks opened up.
At the same time, the Hangar team out in Austin was working on a custom tool to help out the cause. In a day, they had created an interactive map of the panoramics taken by the officers out in the field, while Greg was back at home processing. They also might have inadvertently created the next big tool in disaster management. Their mapping approach is a simple, elegant design with pins dropped in the locations of each 360 panoramic. Moreover, as the viewer moves the scene, there is a field of view that changes on the topographic. The shifting field allows the user to easily locate the direction you are looking for clear frame of reference. It is often said that necessity is the mother of intervention, just as often it is hardship.
The end result, similar to Mapbox map, is a stunning perspective of the fire damage and loss of property. In fact, some of the imagery was so high quality the public agencies didn’t want it released until a few hours before the neighborhoods reopened because the fire safes of residents were clearly visible in some burned homes. The sheriffs were worried about looters seeing the safes.
In and of itself, what Greg did proved extraordinarily useful. He will be called again (there will be more emergencies) and he will be useful again. If he had any doubts about leaving academia with its passive aggressive brawls and anticlimactic victories, those doubts evaporated as he was sitting at the side of one of the fires with his computer. It evaporated when he saw the local agencies planning their responses with the data he helped gather. It wasn’t why he had started to work with drones, to help the community, and yet here he was, covered in sweat and breathing in smoke, absolutely fried from the persistent lack of sleep, and yet as sure as he had been in his career that he was doing the right thing.
But there was a turn in the story. When Greg finally handed off the work to the county GIS team, he did not sleep well. He was overtired. He had just started a business. He smelled like ash in a way that wouldn’t wash away. His mind was spinning. He was still trying to get his daughter potty trained. Ordinary life and this extraordinary moment needed to coexist. But there was more. His old life as an ecologist kept leaping up out of the dark recesses of his mind like a wild animal, a restlessness. What if, he imagined, he could pull together not just his own drone images onto a single map, but also other peoples’ drone images? What if he could coordinate people with drones around California, or around the world, to all take the kind of images he was taking and then to submit them to a central data site? Nearly a million drones are flying around the US alone right now. A MILLION! If even a tiny subset of those drones could be marshalled in some coordinated way they could see what is otherwise impossible to see. Then, he thought about Houston. In Houston, a huge part of the problem of the responding to the hurricane was being able to link up the images and drone footage from across the city. Many independent observations were being made, but they were herky jerky, the high resolution equivalent of a film flashing at the wrong speed and without space or narrative. They couldn’t be made to make sense. The same for Puerto Rico, only worse. And again for the recent shootings in Las Vegas. But this, he sat up and thought, could be solved. And it wouldn’t take machine learning or artificial intelligent or any other hot buzzwords floating around Silicon Valley venture capital firms right now. All it would take was an iPhone, an app and a DJI drone from the local Best Buy, or rather thousands and thousands of each of those. What it would take was a community.
Which is when he called me.
When Greg called he was on edge, that old edge of the discovery combined with the new edge of being able to help people in a way he never imagined. He wanted to join forces to combine what we do in my lab (to engage the public in citizen science in order to speed up discoveries around the world) with what he has begun to do. He wanted to use the public in a coordinated way to see. “Every major discovery,” I heard a fast talking Greg saying, has been about seeing more. “With drones we could see so much more. We could study global patterns in biological diversity. We could study the outbreak of pests.” We could study, I said in response, “the tides,” and “sea level rise.” “Or forest loss,” Greg said. Or forest recovery, one of said. Or deaths of trees due to beetles. We could see not like a bird flying, or a hawk hovering, but instead like thousands and thousands of hawks and what was more the ways in which we could see could be coordinated to particular moments.
Ultimately, what Greg hopes to do is to crowdsource the equivalent of Google Street view but from the sky. He wants to issue a call to action for drone pilots. These pilots could be public agencies, companies, or volunteer citizens depending on the circumstance (e.g., citizens shouldn’t fly in disaster zones). Locations could be randomly chosen or could be targeted for for specific questions and needs. While there have been other companies doing this for drone-mapping jobs (essentially; Uber for drones), their approach is commercial and elaborate, a for profit empire where a village might do. Such a village would be a huge aid in future fires, floods, and other disasters, but before the key moment when a team of drones would be needed in such an urgent way, before the emergency, one would want to test the team of citizen drones in a place that required coordination but was not life or death. And what better thing to test than to once again return to the ecological questions with which Greg started. A simple test that takes advantage of the data drones provide, data that are an order of magnitude in resolution than anything derived from satellites and less impacted by cloud cover. Greg and I decided, on the phone, to start with a simple, and yet important test. We would see if we could bring together the drone village to ask, how are leaves changing in the autumn?
It is actually surprisingly hard to capture the timing and details of the changing leaves of plants in the fall (or in the spring for that matter). Yet, knowing when the leaves turn, fall, and bud anew is key to understanding the impacts of climate change, urbanization and even biodiversity on the growth and future of forests and grasslands. The need for such data is real. Also, it being the middle of October, with Halloween coming up, the timing is perfect right now. We could summon the community of drones and with them map, all at once the status of the leaves across the United States, around the world.
And so we begin the Fly4Fall experiment. And you can help. You can take part in the test right now and for the next few weeks. If you have a DJI drone and want to participate click to see how you can be involved here at fly4fall.com.
Meanwhile, Greg has finally gone to sleep. His desk remains covered in thumbdrives from the mapping project. His daughter is mostly sleeping, though still not using the potty regularly (well, she’ll pee at least). Out not far from his house the fires are finally getting under control from the heroic efforts of fire crews. And even as all of this is happening, you can look at the images Greg compiled. You can see the fires, but also their scale. You can walk into each one. You can imagine the heat. You can imagine too the stories of the peoples’ whose houses have been lost, but also the story of the community that came together to help after the fires had gone through, a community assembled rapidly out of a need, a community that Greg hopes, that we now hope, can be built upon to react even faster and more effectively in the future. There will be more fires. There will be more floods. But there will also be, we have to have, an ever stronger community able to respond to them, a community in which the police, ecologists, and whoever else is needed use every tool at hand and some that, in those moments, are created anew, to respond.
Leave A Comment