# Democratizing Reality Capture Video to 3D Models in 1 Button Press - Event: NestGen ’24 - Published: March 29, 2024 - Length: 30:54 - Watch: https://nestgen.org/library/watch/eStsHBmrZJg/ > The transcript of a recorded NestGen session, for reading or for use with an AI tool. It comes from the video's > captions; automatic captions can mishear names and terms. ## About this session Explore how this cutting-edge technology is democratizing access to advanced modeling capabilities, revolutionizing industries and workflows. Join us as we delve into the intricacies of this innovative solution, unlocking new possibilities for businesses and individuals alike. Learn how you can streamline your workflow, enhance efficiency, and elevate your projects to new heights with ease. Don't miss out on this transformative journey – watch now! To know more, visit: ===== Subscribe to FlytBase YouTube - Follow Us On - LinkedIn - Facebook - Twitter - Instagram - ## Transcript **[0:21]** hello and welcome to another session of n gen24 my name is zinkia and I'm excited to Welcome All of You today's session is like stepping into a sci-fi movie democratizing reality capture video to 3D models in one button press imagine turning video footage into detail 3D models with just a click we are diving into how Sky browse is re revolutionizing the way we capture and interact with the real world from crime scenes to construction sites it's like having a magic wand that turns videos into 3D models making complex data as easy to understand as your fav favorite comic book with that I would like to introduce our speaker Bobby from Sky Bobby is a public safety drone in instructor software de developer and inventor he has done software development and research on the air traffic incident Management Systems funded by US Department of uh US Department of Transportation back in 2016 various Federal contracts led to the invention of Sky browsers autonomous drone flight video **[1:23]** based 3D modeling and one tap Simplicity Bobby co-authored the patent proposal for for Sky browsers accent site reconstruction platform he also trains law enforcement in various training workshops on 3D modeling and Dron best practices before we begin we have some quick uh some quick housekeeping rules uh please feel free to type in any questions you might have during the session in the Q&A chat box the session is getting recorded and will be published soon we also have the booth in Expo section please make sure you visit it during networking and Booth visit slots last but not the least please remember to capture your favorite moments of the session and share them on Twitter and Linkedin by using hash nen I would also like to thank our respon title sponsor DJI Enterprise and our goal sponsor sunflower Labs with that over to you Bobby thanks for having me jinia uh great presentation as always thanks for the introduction hi everyone Bobby from skybrows I'm going to show you **[2:26]** just a brief uh overview of Sky brow some of our offerings and whatnot because we've got a lot of crazy updates floating around on the internet but this shows the whole picture of Sky brow so I'll go ahead and show everyone what we got so skybrows Sky brow started about eight years ago when I was a researcher at records University uh one sec I think can I put my nope where where basically uh I got into a motorcycle accident uh um took a turn a little too fast and uh LOE of my bike uh I was doing research on video based 3D model reconstruction which is turning videos into 3D models and after I got into the motorcycle accident I ended up in the hospital um few hours later I call up my friend to pick up the bike and he goes on scene and still sees police officers on scene doing accident reconstruction so I thought oh that's really inefficient what if we put this technology on drones and **[3:31]** made 3D models of car crashes and we did exactly that at the time it was just pure research code so no real world applications until the motorcycle accident and uh currently we have close to a thousand police departments using skybrows we'll make an announcement on that once it happens uh in California we do a lot of uh crime scene investigation as well um based off of some of our state agencies we've documented over 10,000 accidents and our models have gone through record so skybrows initially started off with just uh accident reconstruction so it was the premise that car crashes happen all the time every two seconds was a car crash and the method of ax investigation is really inefficient it takes hours just to document the scene using laser scanners or tape measurers or taking pictures and that shuts down traffic it's tedious nobody has a good time so we thought okay let's uh make 3D models of car crashes real quick so it'll take just 10 minutes to document entire scene from taking the Drone out of the um out of the patrol vehicle to making a 3D **[4:35]** model and then clearing the scene as OPP post a few hours right there so we started doing just that and uh we built an app that works with uh first only DJI drones on iOS U but then it became an app that works with just about all DJI drones autel drones and then in the summer of last year uh we announced Universal upload which allows you to upload any video into our uh system to make a 3D model so the app's really easy to use um it doesn't take hours to document the um Scene It just uh your drone flies for about two minutes and it's so easy to use here is a 5-year-old making a 3D model so this is Toby right here he's got the sky brows app connected to his drone all you have to do is Fly Above whatever you want to map out and press the start button and you'll see right here the Drone is Auto autonomously flying our flight app the sole purpose of the flight app is to just make the Drone fly autonomously **[5:36]** all of the processing is done through our Cloud so right now the Drone is flying its first orbit right there recording video not pictures Gathering Telemetry data from the Drone itself and then um pointing its gimbal directly at the center of whatever it's mapping out so you can see right here Toby is not a paid actor uh he's just a five-year-old but his dad is so Lewis is asking asking Toby's having a good time and Toby's like yeah I'm having a great time right there actually accidentally turns on the Drone right there at the end of it but you can see here's a 3D model that Toby created right here uh pretty straightforward to make a 3D model it's just you know upload a video get a 3D model back you don't have to take hundreds of pictures or whatnot drone doesn't have to fly that long it's just you know Plug and Play right there you got measurements and all the other stuff so I'll show that later on but uh going back to the app itself uh the app itself just controls the Drone it doesn't do any of the processing you just take the video go to our website upload a video and you get a 3D model um oh did I mention it's free uh you can make 3D models **[6:40]** for free unlimited 3D models for free and if you want measurements and whatnot uh you could either pay to unlock the measurements the tools or you could uh buy a subscription either a light subscription which is uh very very affordable starting at $49 a month or a premium subscription for our Enterprise uh user right there so the core component of Sky brows is video gretry we patented it uh where if you look at uh photogrametry which has been around for a century at this point photogrametry uh takes a bunch of pictures right here and it takes a very long time to capture the pictures you know your drone has to slow down from point A to point B otherwise you get a blurry picture so it takes very long time to capture all that data um you also end up with hundreds of pictures if not thousands for larger areas so you got large large file sizes clogging up your SD cards or your drawing in the boxes right there and also photogrametry you're not getting all the frames um between point a and point B so with video gretry right here the patent that I wrote uh basically we're **[7:44]** using every single video frame uh of the video itself to make a 3D model so it's very uh fast to capture all this data because drone doesn't have to slow down in fact um on our DJI non drone into box offerings it only we can make the Drone fly 34 miles an hour which is very maximum autonomous flight speed right there um of course the file sizes are a lot smaller compared to photogrametry right there so that's the difference between video gretry and photogrametry we believe that video gretry is going to take over um pretty soon and I think our 40,000 end users also agree with that right now so while we started off with just Public Safety uh we found out that there are a lot of other use cases out there so here's our Public Safety case right here where you know we've got a very secure Enterprise Cloud it's fed ramp impact level two secure it's hosted in the same Cloud cluster as the US Air Force and uh Ron as well um and also for Public Safety we maintain chain of custody so all the data is being **[8:48]** tracked and logged on who accesses the data so it's not you know misconstrued deleted or altered on that use case so those are some of our customers right there um some of them include uh National Transportation safety board they document plane crashes and train crashes throughout the United States um skybrows do a lot of stuff in California so you see a lot of California agencies some other state agencies uh on the federal side also Department of Homeland Security we're working with them for uh their Century their soft targets um initiative where we're mapping out University campuses because uh active Shooters are a reality in America so in this case we're able to document entire college campuses and University campuses so that they can use pull it up uh if a active shooter situation does occur right there so we started off with just Public Safety but we found a lot of other use cases you know going to conferences and stuff people were asking us about other use cases so **[9:50]** some of the other use cases include Dentistry you know taking a little camera putting in people's mouths and making a 3D model of their mouth for like for root canals and whole fillings and whatnot um on the insurancea use case you know mapping out natural disasters mapping out roofs uh using Sky brow just because you don't need the sky browse app to make a 3D model you can use any video out there so you can fly your drone record a video just fly your drone around a roof and it'll make a 3D model out of it and you can take measurements accur down to a centimeter right there uh one of our more recent um announcements was our iner mapping and that's been used for real estate as well as for other use cases so I'll play our interior mapping video right now and also for other tactical use **[10:58]** cases for **[12:23]** so in mapping has been real interesting because uh it targets two interesting segments uh that have nothing to do with each other Public Safety as well as real estate on the public safety side if you fly DJI Vada indoors and record a video you get a 3D model back in minutes so the the the general rule of thumb is it takes about a minute of processing per minute of video so it's real quick to generate a 3D model usually it takes less than a minute to record a video of a room so um in that case uh the latest 3D model that you saw it was like a five or six minute video so it took five or six minutes to get a 3D model back um and the use case for that is instead of sending people in you know busting down doors and everything for like a call out uh in this case you can actually get a 3D model of the entire floor plan you know it's essentially a pre-flight before people go in and it's a lot safer for SWAT officers uh going into any sort of call out right there and also on the and and you know with DJI vatas nobody's making 3D models using them uh except for Sky bras **[13:25]** because we can ingest video and djas have terrible cameras as well they're real small and grainy so it's very tough to actually use DJ of a using photogrametry take a bunch of pictures indoors and try to make a 3D model out of that in this case with video gamet you're just flying it inside and getting 3D models real quick so it's a fantastic use case right there um also with cell phone video 3D modeling for the real estate use case you can actually fly you can record a video a lot slower and get much better walls and whatnot to to get a nice real estate tour uh using Sky BRS right there so uh those are just some of the use cases um that we just released right there and of course our defense use case because we have multiple products we have Midnight Sun right now and I'll show you Midnight **[14:28]** Sun e **[15:28]** so Midnight Sun was based off of a video game Concept in Call of Duty the advanced UAV where you can see everything around you and we built out a prototype for Midnight Sun um for DOD as well as a Department of Homeland Security where you're able to generate a 3D model um either ahead of time or when you're on scene because so quick and then using the TAC browse app transmit your location within the 3D model itself and then uh drones running midnight sun which is basically any video feed protects objects and people and whatnot objects of interest and uh puts them into the 3D model so you can pull up the 3D model on your cell phone and see what's around you any given time so um on the defense side it also has a lot of use cases for being able to uh generate 3D models really really quickly right there so those are just some of the use cases but I'll go over some of the platform um itself targeted for uh in this case mostly Public Safety but go over some other use cases **[16:28]** as well so uh on the one of the things I didn't share was thermal mapping so I'll just switch to the video view for now uh here's the video of the uh structure fire that happened this is a live structure fire that happened um it was mapped out using Sky brow uh this was I think two years ago maybe three years ago so it's a little older the Drone flight is a little slower but um you can see there's live structure fire there's it's not training event there's water cannons going up right now and you can see all that smoke right there it's really really hard to see the RO of the smoke and see what's going on um underneath the scene so with thermal mapping we can also injust any thermal video and you get a 3D model out of it pretty quickly you can see it loads pretty quickly as well right here and just the thermal mapping takes about um half a minute or so to generate uh this is using the FL boson sensor 640 sensor with the white hot isotherm you can actually see the hot spots around the area so the white aspects of it uh show that it's hotter and the darker it is um the **[17:34]** colder it is so you know exactly where the hot spots are and interestingly enough as you see the water cannons uh trickling down right here you can actually see how hot the water is as well nearly as hot as up here so um really interesting use case for detecting say anomalies and solar panels but also for live structure fires where you can get this so so quick another use case is 3D modeling in the dark uh we got lighting as well so um just put some stationary scene lighting over there and you can get a 3D model kind of like this where uh the 3D model itself shows up yeah here it is and with uh this is a mock scene right here showing accuracy and whatnot um using nighttime scene lighting so it's a lot crisper right here uh than any other photogrametry Solutions out there and of course this video right here uh was a minute and a half video created by a student who's never flown a drone before and created this type of 3D model so um making good 3D models with Sky BRS is not too hard um we do have a **[18:37]** lot of training on it online as well if you want to check out our training which is also pretty beneficial for for making a good 3D model But ultimately the gist of the training is you know record a good video try not to try not to point your camera directly at the sky and you'll get a good 3D model um you can also annotate stuff so if I wanted to like put down a car I can as a car like here's a toy gun right here put a gun right here and take measurements just like that measurements are accurate down to a centimeter uh and if you're using Universal upload suppose you know you're not using Geor referenced video so uh with DJI and aels you can enable subtitle files and those subtitle files when you upload them actually contain Telemetry data so that's how we're able to generate uh Geo reference 30 models right there but if uh for instance you don't have that um and you know for inance since this is uh say a foot then you can use the no measurement tool quick no measurement right here say this is one feet and now if I take that same measurement you'll see that it's a **[19:44]** foot throughout the rest of the 3D model right here so you can also scale your 3D model pretty easily using the scale tool delete those layers right there and uh some other cool stuff uh you can take area measurements just click few times get area uh you can get angle measurements as well just gets your angle like that and I'll show over to a different 3D model for other use cases let's see let's do a bigger area so in this case we mapped out NFL stadium back in 2020 I believe uh This was done in a f minute drone flight to get this 3D model so it's pretty big right here and you can do a lot with this on top of the area this area is I think 50 acres or so so it's a lot it's a it's a massive area here I'll take area measurement so you can see just do a real quick and dirty area measurement like that and of course you can switch it from feet to say meters for instance for our non-americans out **[20:48]** there switch back to feet right here and you can do a lot with the 3D model one is you know sharing it and making it nice and easy uh to share view on your phone or a computer but you can also turn to a heat map right here which gives you elevation profiles so the brighter the color the higher it is the darker the color the lower it is and you can see right here this uh roadway actually slopes downwards which is really hard to see if I'm just looking at a regular 3D model kind of tough to see if it's sloping downwards right here so for HazMat we're looking at angle elevation change you can do that for a quick and dirty way or you can use our slope map tool which takes the highest points of the 3D model and presents a slope map right here so now you've got uh distance and height anchored down to a centimeter taking slope profiles which updates in real time right here on top of that for larger 3D models you also have the POV tool where for instance if I wanted to see what's the best seat in the stadium for the game going on next weekend well I can use the POV tool **[21:48]** click into it and it zooms me into the 3D model and now I can see you know essentially what the game is going on right here or if I wanted to position cctvs for a security purpose Bo I'm on top of the stadium now looking down at the stadium getting that point of view and make it nice and big and see everything around you just like that so that's another really interesting use case both on the Tactical side as well as the planning side of things and I'll go over one more thing as my time is coming up we now sketch diagrams right here so one of the things we noticed uh with Public Safety is that they're making 3D models of car crashes using skybrows and then they're taking the 3 models and putting into like a drawing software to to create their reports we're like oh that's inefficient they're they're using like clip art cars and like dragging and drop them then drawing like Lan lines and whatnot so we're like okay well what if we just did that ourselves so we did exactly that with Sketch diagrams right here um I think these are non- Geo reference let's see **[22:51]** so just put it in and you get a sketch drawing out of it right here where it gives you the entire Lane lines and everything and we also have another sketch tool as well which actually gives you uh measurements and whatnot so Sky browse generates a 3D model but also a 2d map and a texture so in this case let's see if it's going to load right here you can actually get I think the video shows it right here actually get scale bars right here as well as head positional heading so it'll not only give you the sketch drawing but also you know the distance and whatnot let's see Pop this up so this is the same exact uh example I showed previously but now if I zoom in you can see it actually gives you you know one inch equals N 9 N and a quarter feet right here it gives the scale bar right here showing you the scale of the sketch drawing and of course um positional heading if you want more detail you can click that it'll generate more details for you or less details if you **[23:53]** don't want that they report and remove a lot of that pretty quickly so sket drawings right here pretty neat uh also just based off of our uh use cases that we realized oh well let's let's integrate this um further out so U you can also you know download this currently as a I think a geotiff and also a PNG file but we'll look at other uh formats as well maybe a dxf file in the future we'll see for the construction aec so yeah uh here's my contact info if you want to uh reach out I and always around so happy to answer any emails questions and I'll skip on over to the Q&A section see if there's any questions about Sky brow perfect thanks a lot Bobby that was very insightful and informative uh the various use cases that you walked us through and how digital models or 3D models uh can be used and can increase the efficiency that that's really helpful we have a lot of questions coming **[24:55]** in so I I'll take the first one uh a question is are there any settings in your Cloud processing model to decrease or increase the resolution of the rendered model yeah uh the primary setting is uh subscribing to Sky brow light or Sky brow premium so you'll get better looking 3D models that way right uh we have another question uh can Sky brow work with thermal video yes yep we can adjust any video so MP4 videos um mov transport stream so you can record a video on your phone whatever format it might be you can upload it um any video out there got it and what file types are exported models good question great question so with the uh models you you can export the point Cloud you can export the texture and the 2D map both in jpeg as well as H geotiff format if it's Geor referenced so the point cloud is La format it's open **[25:57]** source unversal format so we're not locking it down to our system only you can take it and Export it into other 3D model viewers same thing with the texture files obj files you can export it out turn it like 3D print it if you want convert to stdl and 3D print it or you could uh you can toss it into other 3D model viewers you know Unity um any sort of game engines or whatnot it's uh Universal so um as long as you have an obj file you can open it up pretty easily got it thanks thanks for that answer uh which drones you support for indoor navigation I mean oh clarify on interior mapping uh interior mapping is manual flight so it's all manual flight you're just uh flying indoors and recording a video and then turning that video into 3D model by uploading it into our system so uh primarily we're seeing DJI of autos because a lot of Public Safety habits but uh We've also made 3D models with DJI minis airs as well um I think the airs are pretty stable indoors **[26:58]** same thing with the minis the later versions right and for the outdoors are there any specific Mission patterns that needs to be flown the Drone needs to be flown in for to generate a a basically high quality Digital model or a 3D twin yeah so for 3D models uh you can either generate it through an orbit which is flying a circle around the object um you can do it manually or through the sky brows app if we support that drone uh then it it'll just be one button press instead of having to manually fly your drone it's slow easier using the sky brows app but of course you can still manually fly the Drone just do a circle around it and if you're looking to do larger swads of land we also have wide brows which is our back and forth flight pattern it does a large lawn mower grid back and forth um with I think 70% overlap or something on horizontal overlap and you'll be able to get a you nice big 3D model but it's more like a two and a half V model because the gimbal is pointed downwards you're not getting too much of the sides of the object itself so what we generally do for for training is um if you're looking to map out a large area but you have **[28:02]** find Details inside of it using the large area you want to get a wide browse the 2 and a half D model a sky browse which still captures up to like an acre um of the of the uh 3D model if you want really good details use Universal upload get a cell phone and walk around say for instance you're uh you're documenting a car crash um then use your cell phone to record video so that you're really close to the the vehicle itself and then you know merge them all together so you get a really nice looking 3D model so merging is part of our training as well got it got it uh there's one question I'll just read it out I recently saw a video on LinkedIn wherein you created a 3D render using video provided by open AI open a SORA using Sky browse and on on that line the question is do you think such a thing can be implemented in real use case so is there any real real life application to that you can think of or share with us yeah yeah for Vision Pro if you want to like uh create like 3D **[29:04]** environments Al together because one of the misconceptions is that Sora is a 3D engine itself it's not it's generating uh it's it's using a temporal uh algorithm to look at where an object should be in space but you can't interact with that at all it's only generating one Viewpoint so it's Sky browse you're making a 3D Viewpoint so You' be able to see multiple angles and whatnot and interact with the 3D model itself so if you want to Stage like a house if you want to plan a new house that you're building um then you can just generate a 3D model of it walk around in it see what you like see what you don't like and then edit it over time so that's uh one of the really exciting use cases we're looking to see so uh stay tuned for announcements of our partnership with open AI stay tuned for that great uh excited to hear about it soon so again Bobby we out of time but any closing thoughts from your and uh yeah just reach out to us uh ideally just email me Bobby sky.com **[30:04]** I'll be around um if you reach out through our contact us Forum we are currently on a massive backlog a lot of people have been emailing us with all the new marketing and releases and whatnot so um if you reach out through contract us uh I apologize in advance if it takes a little longer our support team's been pretty pretty busy uh working with our current customers as well as all the inbound um inbound contacts y so thanks a lot Bobby for your time today and and if anybody has any questions like Bobby mentioned you can reach out to him directly uh and we also have networking uh Booth so if if you want to talk to Bobby during the event you can reach out to them as well so thanks a lot everyone from for joining in and thank you again Bobby for your time today yep thanks for having me appreciate it