# Beyond Reality Capture : Generating Value from your Data - Event: NestGen ’24 - Published: March 21, 2024 - Length: 31:38 - Watch: https://nestgen.org/library/watch/jm7FKC_kfSM/ > The transcript of a recorded NestGen session, for reading or for use with an AI tool. It comes from the video's > captions; automatic captions can mishear names and terms. ## About this session Embark on a captivating exploration of drone technology with Andy, Product manager at DroneDeploy. Join us as Andy delves into the revolutionary concept of reality capture, showcasing how drones redefine data acquisition and analysis. Discover the powerful capabilities of DroneDeploy's platform, from automation and high-accuracy mapping to intelligent insights. Through real-world use cases in construction, agriculture, and more, Andy illustrates the transformative impact of drones on operational efficiency and safety. To know more, visit: ===== Subscribe to FlytBase YouTube - Follow Us On - LinkedIn - Facebook - Twitter - Instagram - ## Transcript **[0:20]** hello hello everyone welcome to another very interesting session in the world of applications so before I get started why don't you let me know where you are all joining in from and how is the Saturday treating you so far by the way very cool fun fact it is already Sunday for our speaker as he's joining in all the way from New Zealand and I think it's 6:00 in the morning for him am I right Andy but so hold on to your seats because we are about to dive into the world where reality isn't just captured it is conquered in this industry if you haven't heard of them well you might have to check your satellite re ception with customers spanning across 190 countries and operations through drones Robotics and 360 degree camera data and with 9.4 million flights the Drone deploy platform is one of the widely used mapping platform out there and to give you a glimpse of the platform that enables you to go **[1:20]** beyond the capture of high accuracy data providing you with intelligent analysis deep insights and a single view of all your capture data please please welcome and Andy product manager of the robotic automation at drone deploy Andy is a mechanical engineer turn software product manager with over 13 years of experience with companies like Becca Trimble and now drone deployer he's most recently focused on automating reality capture and inspections to remove people arms way a very warm welcome to you Andy hello to you thank you but before I pass it over to you I would just like to run a few housekeeping rules so please feel free to type any questions that you might have during the session in the Q&A chat box uh the session is also getting recorded and we will publish it online soon and uh please visit the boots at the Expo section and **[2:22]** finally please remember to capture your favorite moments of the session and share it on Twitter and Linkedin by using the hashtag n a few lucky people might have some surprise in store finally I would like to thank our title sponsor DJI Enterprise and goal sponsor sunflower labs and with that over to you Andy I'll just stop sharing my screen and you can take over thank you bash Elli yeah welcome everyone thanks so much for the introduction bash Elli it's very very kind uh so today yeah I'll be talking about Beyond reality capture uh so you've already had a bit of an intro to me but yeah group product manager look after the robotic aut side hit drone deploy and we'll touch on a little bit of that today as well so a quick agenda uh what is reality capture we a quick overview of that it's a pretty short session so I won't go into too much detail uh some of the superpowers that drone deploy and reality capture give you which is exciting uh automation how we're trying to automate more of that with some of the the doc uh technology like Doc drones now uh high accuracy **[3:26]** and and kind of why that's important and then really what we want from we're capturing all this data really to answer answer questions and and get insights so how we can sort of start providing intelligent insights so at a high level I guess like reality capture we kind of say that we help digitize the physical world so giving you a a nice view of your physical sites and hopefully that's an up-to-date view as well uh when we talk about building the future of reality capture here at drone deploy we we talk about three three kind of key pillars being unified a and and intelligent so unified it's having all your data whether it's from the a or from the ground all in one location and not just like air and ground but also different types of data that you want to answer questions about orthos or uh 3D models uh lighter all of that together you can answer those questions in in one platform and share that amongst your your different stakeholders automated and it's a big part of what I'm doing at the moment we really **[4:27]** want that data to be scheduled by robot eventually get the technology coming along uh very rapidly which is awesome manual time to to to capture some of that data so the more now we can start to automate that and make it easier people to capture we can just have uh more more of that data and the next piece which is the intelligent insights so now collected all this amazing data really you just want ansers and insights and and ideally they started to be extracted automatically or you can ask questions about your data and I'll show a little bit of that as well so yeah to give a little bit of the the scale of drone deploy it's it's uh really really sort of insane now the amount of aerial flights and ground capture that's happening at the moment uh so this actually is a number from sort of last year so it's it's over this now but we hit the one billion images that have been processed by drone deploy and in the back the back of that map there you can see every single Blue Dot is an aerial flight uh across the globe and that's so it's **[5:29]** it's pretty much covering the entire the entire Globe which is which is really amazing amazing set of of data there and the red dots now as well which is becoming it's a bit more of a the more recent part of the solution is the ground capture as well um and really most of that that those Building images have been captured manually to some degree like someone to go out with a drone and capture it um still automated with with the Drone deploy flight planning but it's still quite a lot of manual effort to get out there and do that um and then on the ground side as well people walking around with 360 cameras and and and uh mobile phones but with with automation we really want a lot of this and a lot of these images the next billion to be to be more automated so all right so reality captures superpowers this what we kind of talk about at drone Deo to to explain I guess the value of reality capture um I guess as kids we always sort of a lot of us dreamed about having these sort of type of superpowers and now it's sort of coming to life so we talk about fly teleport time travel and X-ray and I'll just touch on on these briefly as well so flying I guess **[6:33]** you know probably going back to sort of the Phantom 4 uh model now when drones sort of became started becoming really reliable really really uh affordable you you could never get a picture like this of your site like 10 years ago it was super expensive to either hire a helicopter or have a fix wi plane fly over and capture something like this it was really hard to see your site like this um so having having democratized drones I guess and been able to to have anyone on site can can fly a drone and capture amazing data like this and make make really awesome U decisions is is really powerful uh and because you can do that yourself now you can fly more regularly it was no it was no it wasn't really that good having a a flight once every year or two years because that's all you could afford now you could have that every day if you want to so fly and that aerial view is is just like a very powerful thing to to give you context from um a higher level so as well as as been able to fly you can kind of teleport so now we've got this this unified platform we've got projects all across the world and so I can quickly jump in **[7:35]** teleport to any project anywhere in the world and have this high resolution data this is a a 360 degree Pano that we've stitched together from drone imagery um that we do in drone deploy and now you can kind of make decisions you can analyze you can measure you can you can you have really powerful you don't have to travel to site necessarily um and having that more upto-date view is just you know allows you to be like actually make decisions and not have to go out to site uh so often often and this was really important through Co times when it was super difficult to get to sites still really important now where you have uh not just construction sites but maybe energy sites or solar sites that are really remote uh and you're not on those sites so uh fulltime so rather than driving or flying hours uh you can you can jump in and see your site really at a high resolution and make measurements you need to is cool and then even inside your project we've got these things here we can just teleport uh between a Peno and a photo on a different side you know completely different side of the site so you can teleport across your own site as well which is which is exciting time travel is a sort of a big one now like being able to caption more **[8:38]** really gives you the ability about to go like what happened last week what happened last month uh and here you can actually compare earo imagery uh images orthos right on top of each other so slide that across and be like what happened last week compared to this week uh you know go back confirm that work's been done understand maybe subcontractors are doing something uh around last week confirmed that their work was done uh and where they're up to and so so just been enough to go back in time uh is really is really important for making a lot of decisions and uh having that data captured regularly is the only way you can really do that it's too late if you want to go back and you didn't capture that data um we can't quite unfortunately time travel to that that point yet and then the last one here we talk about x-ray uh so this this is a bit of a ground example but the same thing goes for for aiel for utility uh services or anything buried under the ground if you can capture this data a regular point you now know like what's in behind the walls you can you can x-ray **[9:38]** behind their wall and know exactly where the services were installed so that's a really powerful thing to know like a lot of lot of times now we got uh buildings or sites that are 30 40 years old um trying to dig up a utility line that you've got on an as build is is pretty much all you know often wrong so you're digging up the wrong side of the road or whatever to find these these utilities so having it and it's dangerous as well if you've got high press gas pipelines or anything so the ability to have those sort of as builts and be able to go back and know where where things are under the ground or behind walls is is is great and now I guess Switching gears a little bit moving to to automation um we really want to start to automate the capture of this and capture more regularly so we can get all of those those great superpowers um you only sort of get those by capturing the data at a at a reasonably regular interval so we've got's goal at drone deployed by 2027 we want 50% or more of the data collected in drone deploy to have been done by autonomous robot so whether that's a **[10:39]** dock drone or a Boston Dynamic spot there on the ground we're very agnostic to the to the robot type we want just the best the best robots and the best technology for the for the job but at the moment the percentage is is very small it's still a lot of manual work like I said so freeing up people's time um from capturing to actually you know doing the insights and and making decisions is is important to us and there I guess the advancement of of of robots and and Doc drones now is and the price point has come down so so much that it's really become starting to become a reality so we starting to see a lot lot of momentum in the space and yeah it's definitely an exciting time for the industry so drone deploy is done well traditionally I guess by just making drone flights and nonoc drone flights for a start super simple you just sort of pretty much have like what what is your outcome what are you trying to do are you trying to do a map are you trying to do a facade like a vertical face of a building do you want a panorama just to get like a uh 360 view of your site from a certain point and on that Panorama all you do is pick your location and the height and then drone **[11:42]** deploy automatically figures out how to stitch 26 photos take the 26 photos where to take them and it stitches it all all for you automatically so it makes it really accessible super easy anyone can can fly these drones now uh which which is great um and then so for a mapping example all you need to do is just literally draw an area on a satellite image around the area you want mapped you don't have to worry about like what overlap do I need or anything like that we automatically detect the Drone and do the optimal overlap to create the best 3D model the best Ortho that we can um it'll tell you how long it's going to take how many battery uh it'll take for the specific drone how many images it's going to take you can turn on 3D enhance 3D or not just little things like that are just really easy and it'll take um oblique images around the outside just to help produce a bit of 3D model so it's that's that's all you need to do you don't have to worry about like what does my overlap need to be what you know how do I fly this Grid or anything we do all that sort of sort of for you and we want to bring that ease of use in like simple UI to dock **[12:44]** drones as well so and I'll show just a Qui video soon so effectively now you would usually go to site with a controller fly a non- do drone like this um but then you have to take the data off the off the SD card upload it so there was still a couple of steps but now you can fly this right from the the uh drone deploy interface select the dock you want on your site it'll fly come back home and then upload the data and I'll I'll show that in a in a bit of a video form to to bring it to life quick video here so again this is on a a coal plant uh site here you just draw draw the area the Box around the the stockpile there that we we wanted to capture select the dock click Start flight you go through your pre-fight checklist start flight will take off and then it will fly that grid and you can watch that live from from anywhere in the world um obviously now with with the regulation starting to happen you can start to get BB Los wavers and do this completely remote depending on the different countries you can still have pilots and command on site but it just gives you a much **[13:44]** easier way to kick off kickoff missions uh and then when it comes to land you don't have to worry about charging the batteries anymore or swapping batteries out it charges itself so this is just an example of the DJI doc one there's a number of of dock Dr own Solutions out there the charges itself but now also you have to worry about the SD cards and all of that so the data automatically goes up into drone deploy 2 hours later you have a a 3D M map like this you can start to do the volume of that stockpile this with the doc one you've got an m3t uh drone in there so you can take thermal as well so you've really quickly got this amazing data a thermal scan of find any hot spots in that cold stock pile you can do a full volume of it you've got that zoom camera on that that dock drone to zoom in to cracks on a flare Sack or anything you've got this amazing sort of um ability and now that all that data goes up into your project automatically you don't have to worrying about uh pulling the SD card out and things like that or or even worse get into site and the batteries aren't charged or you've forgotten the SD card which which happens far more than it than it **[14:46]** should so with I guess at drone point now really a big push of ours is to try and create high accuracy Maps there just we say there's like 10x value or 10 times the amount of value in a high accy map versus a non a non High map uh and kind of why we say that is if you fly just with pure GPS if you fly a map one day and then fly the next day you're going to have a few meters whatever your GPS uh um accuracy is of difference they're not going to Overlay very nice it's it's very hard to compare them relatively it's really good you're still really powerful you can measure and and have a relative idea of what's going on but absolute becomes quite difficult so if you can have accurate maps and now with with some of the newer drones having rtk modules been enough to leverage rtk doc drones are great because they the majority have rtk built in just to help them with their High Precision landing and everything so you get that rtk we have ground control points and workflows as well to add in control points that you can uh on top of the rtk to to **[15:48]** to align even even more accurately um and when you do that you can see if you fly one day and you fly two weeks later you can see how close these like line over over top of each other so means you can start to change over time really accurately with that Earthworks analysis becomes very easy you can start through like what what's cut and fill and kind of do a heat map of that what's the volume that's been moved in or out of of dirt which is really important but if you don't have those Maps aligned you just you just can't do that um some the as built documentation so been able to bring in a 3D dxf um and know pretty accurately exactly where the pipes you put in the ground are uh is really important I'll show a bigger picture of that in a second and then when you export it into other tools if you're putting into CAD tools you're you're using this data somewhere else to do designs to do whatever if it's just relatively accurate anything think it's it doesn't line up with anything you bring into your other your other design files or anything like that so if you enough to bring into other tools and everything just lines up is is really powerful for for a user and you can kind of trust that data as well **[16:50]** then so yeah this is a pretty cool example of bringing in like a completely independent 3D dxf that's got its own GE um GE reference um information on it and you can see how of closely that aligns to the to the actual drone data and the map uh so that that just means that you can be sort of confident that before you bury this up and everything you've actually done this to the design it's looking like it's in the right place uh and again then having this data as well you now got the x-ray vision superpower that you can go back and and know what was done another I'll a quick video here um this is a a construction example but having that bringing in your your again independently GE reference 3D model you're not having to try Line This against a drone and like join them up you bring this in and you can see that your concrete levels of your flight match up really well to where you're about to um put all your steel work and everything are your Earthworks levels correct so again you've just got this amazing view to see like how you know like before you put something up and have some really costly rework you've got a really **[17:54]** good example of of that data and that's that's really why we're capturing this data is to about to Compu to design or do or answer some questions with it um it's not just sort of getting captured just for the for the sake of it and then another one here it's a bit more of a um like really useful for marketing everything but also tracking projects if you are capturing this is every month for the entire project so timelapse you can see the lies over top of each other pretty much bang on so you can see really powerful like how things are progressing you can do that month over month it's really great for updating uh SE senior stakeholders or your owners of the sites or anything from construction uh other other stakeholders need to need to be involved or updated can see the progress so again that high accuracy allows things like that which is great and then capturing all this data I guess like it and now with automation we we're capturing more and more data it's becoming like a little bit overwhelming like no one wants to get to go through every single one of those images or we're doing a **[18:55]** lot of work already to Stitch things together to make it easier to to to to get those insights out of this but uh there's still more work to do and that's what really we're pushing on now is is having these these smart reports with leveraging all the the hype around Ai and a lot of these models that have come a long way so dry itself has been in this space for quite a while with computer vision models and we've been doing this for for a lot of our customers but it's just making it easier and opening up more more opportunities so every every single minute in the US we have a a solar roof report or a drone up in the air taking a or uh an ortho of a of a residential roof um so we do over 300,000 of these reports a year and we and and we don't just give them an ortho we give them a 3D dxf of the roof so you can see all your slopes you know with the orientation of it so then we've got customers like Sunrun that will use that to design how many panels can they fit in the roof where what's the best part of the roof to to uh to put those panels on what sort of output will they get because we know where it's facing facing the sun um for how many hours **[19:57]** and everything so super powerful that and so that's before that this workflow they used to have someone go up onto the roof with a tape measure so really dangerous manual work go up measure the roof it would take about four hours five hours to do you might get one roof a day and now it's a f minute drone flight and and all the measurements are out very accurately you not have to worry about someone sketching it down on a on a notepad with with a tape measure to to um remember what what they sort of captured so just an amazing like value out of of of drones and reality capture yeah we do a bit in the egg space with corn and soy crops so uh early in the season when the planted you sort of want to know how many of actually how many sigings there actually are and how how well all your plantings worked um so before you would kind of I guess take like a sample set of of of your uh of the field and sort of just have to estimate out the rest is like that but now you can have a very accurate count of what areas have done really well what haven't **[20:58]** um compared to compared to other ones you can start to AB test things like fertilizer or or different different types of species of of um corn or soy or crops as well uh and these are like your count like see you can see here these are just like little areas with 50,000 100,000 like there's no chance you're ever going to have to count these manually so that's really powerful we do a lot of large property sort of inspections as well so uh big commercial to the buildings uh HVAC pulling of water damage to the roof uh big car parks and um pavement reports roof reports cracks and that and everything like that so that's that's that's really valuable so that sort of um yeah Property Management space and then constructions a big big part of what we do for sure and then things like stockpile reports and been able to automatically calculate some of the stockpiles it's really valuable to know what how much dirt's been moved in and out um and what type of material that was so we're sort of looking forward to this like we've got this these billion images now it's like how can we really add value **[22:00]** and and make it easier for customers to get those insights so we want it to be a bit more natural to talk about uh I guess yeah to talk like to your data to ask questions of your data the natural language type thing so can we measure the you know measure the stock piles and report by material uh can you count all the trees here and by type and and Export it to CSV to do something else so like can we really interact like we would kind of we're kind of asking the the data this as well but it's it's quite a manual task at the moment uh another one's like really important things that happen so before you pour concrete uh you have these these uh sleeves here that need to be in place they need to be in the right place your rebar obviously always all all needs to be in the right place as well but you really want to make sure that that's right before you pour concrete it's very expensive and very painful uh if you pour concrete and find out that you didn't have some of the sleeves on uh or or everything in the right place so uh we have this uh AI sort of um sleeve report **[23:02]** which we uh compare to your design so take a design file we got a high accuracy map so again the importance of high accuracy and we can we can with computer vision pick up all the sleeves with the green markers there you can see all the ones that match the design files so we turn this around within four hours um and it's often it needs to be done like that because you've only just got the sleeves and everything ready maybe a day if you're lucky before you pour the concrete so I've been have to scan fly this turn around a report within four hours and say You're missing two sleeves here or these sleeves are slightly out of Tolerance is a huge like huge value ad if you can if you can fix this now before you P concrete it's saving you hundreds of thousands of dollars uh so that's a really cool use of like that's what they want to know like they're flying the site to answer this exact question so can we just give them the answer uh and this is what we're we're excited about for for more and more of these kind of use cases and then solar construction is the same there's a huge huge push towards obviously renewable renewable energy which is great and there's especially in the US **[24:03]** now a lot of subsidies and a lot of a lot of solar construction being built but these are huge sites they huge on a scale these huge solar Farms how do how do you keep track of your progress how do you know what's going on how do you know if you've missed areas it's really hard to see that when you're on the ground it's it's obviously everything looks the same you're in these rows and blocks so now we have um some solar solar construction reports uh so we we take our high accuracy High see high resolution maps of that we already stitched together as part of drone deploy and you can sort of see here some panels in here some racks there some poles but now we can tell you within um a pretty short space of time exactly what how many poles are in there how many racks have been installed how many modules are there and the important bits there are like what parts have been missed like why why are there only poles here and no racks because once the crews have left these blocks it's very expensive to come back and do do this sort of stuff so you want to see at a at a high level glance like why why were these left is for for a certain reason or otherwise get crws while they're close to come back and finish that work uh and then **[25:05]** we can if you're doing this cap you can do this every day every every week the more you can capture this the better and then you can start to see day over day like the difference like polls and St poles coming in first then racks then then modules so you can track that over time and you can start to shren to see like you know is the project on on track or not which is which is super powerful and obviously not would take a hell of a long time to try and count through every single one of these poles or racks on these sites so to have that straight away and see that in a nice sort of a nice uh colorcoded way or heatmap way is is is we s a lot of value on that sort of report and then I'll finish here and try and get to a couple of questions as well uh on stockpiles so stockpiles traditionally and we've had this for a little while now you can you can already like put a 3D model in and manually go around and it's like you want to take a bit of K to make sure you get around the stock piles are always quite a difficult shape so you can see here it might take like 30 seconds to to go around still super **[26:06]** amazing before you'd have to get a survey to go out and try and Survey the stock pile now you can do it from drone imagery and get a pretty accurate volume if you've got a good uh ground surface um but now with all these great segmentation models the advancement of a lot of these AI tools now we can just click on it and knows exactly what this is a stock pile takes about a second you can click on another one so now you've almost gone like 10x or more um quicker about to get these stock piles but then like like why sto there we've got the technology now to to analyze that entire map um so the stage where at now is like can we uh you choose something that you want you slick something like a tree or a stockpile like this and then be like find me all the stock piles or or create a volume report for the entire thing so you can see here like now find all stock piles and this is a large map with a lot of stock piles so a great example of it imagine trying to draw around every one of those manually and figure out how much volumes moved there now if you can do this day over day you can have amazing amazing like comparison to what's been moved on site in and **[27:07]** out every single day um you can start to do things like detecting which type of stockpile now you know the cost of that material and what's been moved and anything you can Stu run numbers as well about what's happening so yeah hopefully hopefully that's a pretty quick session but uh hope give a bit of an overview or an insight into into drone deploy and and what kind of doing and we're yeah we're very excited to start capturing more and more data but actually give insights to to our users and customers thank you so much Andy for sharing this presentation it was really insightful starting with the map 1 billion images processed by drone deploy and you mentioned that it's still a very old data but this talks about you know volumes about how drones are being used today and the potential it holds also I love the Insight that you mentioned I just took a note uh that by 20 2027 50% of the data are going to be captured by autonomous robots beat airor ground **[28:08]** also love the platform the versatility the different kinds of applications it's being used for the concrete use case who would have thought about it yeah exactly yeah yeah right so all right I'll just take a few questions we are already running out of time uh just um one second I'll remove this for now yes so I have a few questions in the Q&A tab if anybody has any other questions please feel free to drop them in uh for one question is what kind of safety features are included for BV loss flights yeah yeah so we have like um return to home so about to return to home um obviously remotely sort of basic like manual control like that you can pause a mission so pause that and hover in place um we're leveraging a lot of the doc drone stuff the first DOC was supported I it's another question down there is the the DJI doc one and that's uh and obviously we're working on the the DJI doc 2 as well which is is coming out this year um so we leverage a little bit of that stuff as well so **[29:09]** they've got things like the the DJI air sense the ads adsb in um for other aircraft around there so at this stage it's sort of those those manual control the low latency video you have to watch that you have to pause it hover in place return to home if you need to um we're kind of finding like a lot of customers and to get their waivers anyway you kind of need to have a bit of a track record so we still have a lot of customers that are seeing a huge amount of value even without BBL loss just not having to charge batteries not having to wor about SD cards um and they can still have a pilot uh pilot and command or Pilot control on on the site there just sort of a backup with that controller but the flights are you know the whole flight path is automated the uploads automated so we're kind of working up towards more of those BV loss features um to have like the DOT cameras so you can check the the Drone before you take off everything that you need to do remotely so that's a lot of the stuff we're starting to to work on now as well I think uh moving on to the final question which would be um flight approvals is that something that drone deplo is providing as well yeah so **[30:10]** so flight approvals I guess we're kind of working on that now we're going through our own approval process um for some waivers in in the US and and some other uh like Australia as well um so that's something that we're trying to put together to help help some of our customers so yeah definitely get in touch we're we're trying to get all documentation around like what what what you need to to get through the waiver process and different different markets um and we have some great third party Partners as well that we work with across across the uh the world that that help that are really specialized in in those sort of BB loss waivers as well um so we try and do everything we can on the software side and everything uh and yeah to give you the the information you kind of need to apply for those waivers all right I I think yes that's the way forward that is going to happen for a while now and we'll all see more of pcc's and Pilots coming up and uh like you said the dock is going to be like a a unit which is going to be used for multiple applications be it uh especially the same uh dog can be used for security inspections across the same site so yes **[31:12]** exciting times ahead thank you again Andy for joining in and sharing this with us even though like I said it's 600 in the morning for you but um thank you once again and I hope you have a great um the we are already at the end of nesen we just away the last session is just starting so I would not hold everyone back here thank you again audience for joining in tuning in for I hope you had a great time awesome thanks so much