{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Example of EGMS toolkit in Python environment\n", "*Date: 26 Jan. 2024 by Alexis Hrysiewicz*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Import the required modules\n", "\n", "As a Python package, we import the **EGMS-toolkit** sub-packages. For this example, (and for a complete processing), **EGMS-toolkit** requires 4 modules: \n", "\n", "- *EGMSS1burstIDapi*: management of the S1-Burst-ID map (from the *classes* modules);\n", "- *EGMSS1ROIapi*: process the selection of datasets based on a ROI (from the *classes* modules);\n", "- *EGMSdownloaderapi*: downloader of EGMS data (from the *classes* modules);\n", "- *egmsdatatools*: some supplementary tools (from the *functions* modules). " ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "from EGMStoolkit.classes import EGMSS1burstIDapi\n", "from EGMStoolkit.classes import EGMSS1ROIapi \n", "from EGMStoolkit.classes import EGMSdownloaderapi\n", "from EGMStoolkit.functions import egmsdatatools" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Of course, it also is possible to directly import EGMStoolkit: \n", "\n", "```python\n", "import EGMStoolkit\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Manage the Sentinel-1-Burst-ID map\n", "\n", "The first step is the creation of the *S1burstIDmap* object to manage the Sentinel-1-Burst-ID map. \n", "\n", "> **_NOTE:_** To improve the clarity of this example, the verbose mode will be actived for some selected processing step. \n", "\n", "### Create the python variable \n", "\n", "We call the class with the verbose and logging modes deactived. " ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "info = EGMSS1burstIDapi.S1burstIDmap(verbose=False,log=None)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Of each **EGMS-toolkit** class, the *print()* method can be used to visualise its attributes. " ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "date_str_init: 29/05/2022, dirmap: /Users/alexis_hrysiewicz/Work/UCD_InSAR_Scripts/EGMStoolkit/src/EGMStoolkit/3rdparty, pathIDmap: /Users/alexis_hrysiewicz/Work/UCD_InSAR_Scripts/EGMStoolkit/src/EGMStoolkit/3rdparty/S1_burstid_20220530, list_date: ['20220530', '20220531', '20220601', '20220602', '20220603', '20220604', '20220605', '20220606', '20220607', '20220608', '20220609', '20220610', '20220611', '20220612', '20220613', '20220614', '20220615', '20220616', '20220617', '20220618', '20220619', '20220620', '20220621', '20220622', '20220623', '20220624', '20220625', '20220626', '20220627', '20220628', '20220629', '20220630', '20220701', '20220702', '20220703', '20220704', '20220705', '20220706', '20220707', '20220708', '20220709', '20220710', '20220711', '20220712', '20220713', '20220714', '20220715', '20220716', '20220717', '20220718', '20220719', '20220720', '20220721', '20220722', '20220723', '20220724', '20220725', '20220726', '20220727', '20220728', '20220729', '20220730', '20220731', '20220801', '20220802', '20220803', '20220804', '20220805', '20220806', '20220807', '20220808', '20220809', '20220810', '20220811', '20220812', '20220813', '20220814', '20220815', '20220816', '20220817', '20220818', '20220819', '20220820', '20220821', '20220822', '20220823', '20220824', '20220825', '20220826', '20220827', '20220828', '20220829', '20220830', '20220831', '20220901', '20220902', '20220903', '20220904', '20220905', '20220906', '20220907', '20220908', '20220909', '20220910', '20220911', '20220912', '20220913', '20220914', '20220915', '20220916', '20220917', '20220918', '20220919', '20220920', '20220921', '20220922', '20220923', '20220924', '20220925', '20220926', '20220927', '20220928', '20220929', '20220930', '20221001', '20221002', '20221003', '20221004', '20221005', '20221006', '20221007', '20221008', '20221009', '20221010', '20221011', '20221012', '20221013', '20221014', '20221015', '20221016', '20221017', '20221018', '20221019', '20221020', '20221021', '20221022', '20221023', '20221024', '20221025', '20221026', '20221027', '20221028', '20221029', '20221030', '20221031', '20221101', '20221102', '20221103', '20221104', '20221105', '20221106', '20221107', '20221108', '20221109', '20221110', '20221111', '20221112', '20221113', '20221114', '20221115', '20221116', '20221117', '20221118', '20221119', '20221120', '20221121', '20221122', '20221123', '20221124', '20221125', '20221126', '20221127', '20221128', '20221129', '20221130', '20221201', '20221202', '20221203', '20221204', '20221205', '20221206', '20221207', '20221208', '20221209', '20221210', '20221211', '20221212', '20221213', '20221214', '20221215', '20221216', '20221217', '20221218', '20221219', '20221220', '20221221', '20221222', '20221223', '20221224', '20221225', '20221226', '20221227', '20221228', '20221229', '20221230', '20221231', '20230101', '20230102', '20230103', '20230104', '20230105', '20230106', '20230107', '20230108', '20230109', '20230110', '20230111', '20230112', '20230113', '20230114', '20230115', '20230116', '20230117', '20230118', '20230119', '20230120', '20230121', '20230122', '20230123', '20230124', '20230125', '20230126', '20230127', '20230128', '20230129', '20230130', '20230131', '20230201', '20230202', '20230203', '20230204', '20230205', '20230206', '20230207', '20230208', '20230209', '20230210', '20230211', '20230212', '20230213', '20230214', '20230215', '20230216', '20230217', '20230218', '20230219', '20230220', '20230221', '20230222', '20230223', '20230224', '20230225', '20230226', '20230227', '20230228', '20230301', '20230302', '20230303', '20230304', '20230305', '20230306', '20230307', '20230308', '20230309', '20230310', '20230311', '20230312', '20230313', '20230314', '20230315', '20230316', '20230317', '20230318', '20230319', '20230320', '20230321', '20230322', '20230323', '20230324', '20230325', '20230326', '20230327', '20230328', '20230329', '20230330', '20230331', '20230401', '20230402', '20230403', '20230404', '20230405', '20230406', '20230407', '20230408', '20230409', '20230410', '20230411', '20230412', '20230413', '20230414', '20230415', '20230416', '20230417', '20230418', '20230419', '20230420', '20230421', '20230422', '20230423', '20230424', '20230425', '20230426', '20230427', '20230428', '20230429', '20230430', '20230501', '20230502', '20230503', '20230504', '20230505', '20230506', '20230507', '20230508', '20230509', '20230510', '20230511', '20230512', '20230513', '20230514', '20230515', '20230516', '20230517', '20230518', '20230519', '20230520', '20230521', '20230522', '20230523', '20230524', '20230525', '20230526', '20230527', '20230528', '20230529', '20230530', '20230531', '20230601', '20230602', '20230603', '20230604', '20230605', '20230606', '20230607', '20230608', '20230609', '20230610', '20230611', '20230612', '20230613', '20230614', '20230615', '20230616', '20230617', '20230618', '20230619', '20230620', '20230621', '20230622', '20230623', '20230624', '20230625', '20230626', '20230627', '20230628', '20230629', '20230630', '20230701', '20230702', '20230703', '20230704', '20230705', '20230706', '20230707', '20230708', '20230709', '20230710', '20230711', '20230712', '20230713', '20230714', '20230715', '20230716', '20230717', '20230718', '20230719', '20230720', '20230721', '20230722', '20230723', '20230724', '20230725', '20230726', '20230727', '20230728', '20230729', '20230730', '20230731', '20230801', '20230802', '20230803', '20230804', '20230805', '20230806', '20230807', '20230808', '20230809', '20230810', '20230811', '20230812', '20230813', '20230814', '20230815', '20230816', '20230817', '20230818', '20230819', '20230820', '20230821', '20230822', '20230823', '20230824', '20230825', '20230826', '20230827', '20230828', '20230829', '20230830', '20230831', '20230901', '20230902', '20230903', '20230904', '20230905', '20230906', '20230907', '20230908', '20230909', '20230910', '20230911', '20230912', '20230913', '20230914', '20230915', '20230916', '20230917', '20230918', '20230919', '20230920', '20230921', '20230922', '20230923', '20230924', '20230925', '20230926', '20230927', '20230928', '20230929', '20230930', '20231001', '20231002', '20231003', '20231004', '20231005', '20231006', '20231007', '20231008', '20231009', '20231010', '20231011', '20231012', '20231013', '20231014', '20231015', '20231016', '20231017', '20231018', '20231019', '20231020', '20231021', '20231022', '20231023', '20231024', '20231025', '20231026', '20231027', '20231028', '20231029', '20231030', '20231031', '20231101', '20231102', '20231103', '20231104', '20231105', '20231106', '20231107', '20231108', '20231109', '20231110', '20231111', '20231112', '20231113', '20231114', '20231115', '20231116', '20231117', '20231118', '20231119', '20231120', '20231121', '20231122', '20231123', '20231124', '20231125', '20231126', '20231127', '20231128', '20231129', '20231130', '20231201', '20231202', '20231203', '20231204', '20231205', '20231206', '20231207', '20231208', '20231209', '20231210', '20231211', '20231212', '20231213', '20231214', '20231215', '20231216', '20231217', '20231218', '20231219', '20231220', '20231221', '20231222', '20231223', '20231224', '20231225', '20231226', '20231227', '20231228', '20231229', '20231230', '20231231', '20240101', '20240102', '20240103', '20240104', '20240105', '20240106', '20240107', '20240108', '20240109', '20240110', '20240111', '20240112', '20240113', '20240114', '20240115', '20240116', '20240117', '20240118', '20240119', '20240120', '20240121', '20240122', '20240123', '20240124', '20240125', '20240126', '20240127', '20240128', '20240129', '20240130', '20240131', '20240201', '20240202'], verbose: False, log: None\n" ] } ], "source": [ "info.print()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Dowloading of the Sentinel-1-Burst-ID map\n", "\n", "**EGMS-toolkit** requires the Sentinel-1-Burst-ID map to detect the datasets. **EGMS-toolkit** can check the latest version and download the file by using the following method. \n", "\n", "By default, the file will be stored in the *3rdparty* directory. However, the user can change the directory by modifying the *dirmap* attributes of the class. " ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "info.downloadfile(verbose=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Check the tile/bursts available according the user inputs\n", "\n", "From the Sentinel-1-Burst-ID map, **EGMS-toolkit** is now able to detect the associated EGMS datasets. Of course, the user can give some parameters to control the search. \n", "\n", "### Create the Python variable. \n", "\n", "We call the class. " ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "ROIpara = EGMSS1ROIapi.S1ROIparameter(verbose=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Definition of some parameters\n", "\n", "#### For the EGMS level\n", "\n", "The user can define the level of EGMS datasets: i.e., L2a, L2b or L3. " ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "ROIpara.egmslevel = 'L2b' # Level of EGMS data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### For the ROI (or bbox)\n", "\n", "The user must define the ROI. There are three ways to define this parameters: \n", "\n", "- by a list of EPSG:4326 coordinates [W,S,E,N]; \n", "- by a country name (i.e., GB, FR, etc.); \n", "- by a shapefile in EPSG:4326 coordinates (multiline string format). " ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "ROIpara.bbox = [-6.427059639290446,53.2606655698541,-6.0952332730202095,53.41811986118854]\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### For the release\n", "\n", "Currently, there are two available releases: \n", "- 2015-2021; \n", "- 2018-2022. \n", "\n", "The user can select one of these releases. " ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "ROIpara.release = '2018_2022'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Conversion of the bbox to a ROI\n", "\n", "**EGMS-toolkit** requires a conversion of the bbox to a ROI. The toolkit will create a shapefile to store the ROI. The default value is *bbox.shp*. " ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ROIpara.createROI(verbose=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Detection of the EGMS datasets\n", "\n", "Now, **EGMS-toolkit** has the required information to detect the EGMS datasets. \n", "\n", "By default, all relative orbits and passes will be selected. Of course, the user can used two options to define the relative orbits and pass of Sentinel-1 satellites. \n", "\n", "> **_NOTE:_** Lists of tracks and passes can be given for multiples searches. " ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "################################################################################\n", "EGMStoolkit 0.2.1 Beta\n", "\n", "--------------------------------------------------------------------------------\n", "EGMStoolkit.classes.EGMSS1ROIapi.EGMStoolkit.classes.EGMSS1ROIapi:\n", "\tDetect the data regarding the burst IDs\n", "\n", "\tScript: /Users/alexis_hrysiewicz/Work/UCD_InSAR_Scripts/EGMStoolkit/src/EGMStoolkit/classes/EGMSS1ROIapi.py\n", "\t\tCopyright 2024, UCD / iCRAG\n", "################################################################################\n", "\n", "WARNING in EGMS-toolkit processing: in EGMStoolkit.classes.EGMSS1ROIapi.checkfile\n", "\t/Users/alexis_hrysiewicz/Work/UCD_InSAR_Scripts/EGMStoolkit/src/EGMStoolkit/classes/EGMSS1ROIapi.py\n", "\t\t--> The use of the S1 burst ID map is less accurate than the use of .xml S1 files.\n", "For the L2a and L2b levels\n", "|████████████████████████████████████████| 1127661/1127661 [100%] in 45.1s (24992.20/s) \n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ROIpara.detectfromIDmap(info,Track_user=1,Pass_user='Ascending',verbose=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Save and load results from detection\n", "\n", "**EGMS-toolkit** can save and load the results of a detection. " ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Save the burst ID list\n", "ROIpara.saveIDlistL2()\n", "# Load the burst ID list\n", "ROIpara.loadIDlistL2()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "By default, the file name is *egmslist.pkl* but the user can change it with the *input* option. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Display a map \n", "\n", "To visualise the results, **EGMS-toolkit** can display the expected datasets. " ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "ROIpara.displaymap(output='fig_search.jpg',use_folium=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If the *ouput* option is not given, the figure will be displayed and not saved. \n", "\n", "![](fig_search.jpg)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Download the EGMS data\n", "\n", "The next step therefore is the downloading of the datasets. However, the user needs to create the links. \n", "\n", "### Create the python variable \n", "\n", "We call the class. " ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "WARNING in EGMS-toolkit processing: in EGMStoolkit.classes.EGMSdownloaderapi.checkparameter\n", "\t/Users/alexis_hrysiewicz/Work/UCD_InSAR_Scripts/EGMStoolkit/src/EGMStoolkit/classes/EGMSdownloaderapi.py\n", "\t\t--> The user token is not correct.\n" ] } ], "source": [ "downloadpara = EGMSdownloaderapi.egmsdownloader(verbose=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The **WARNING** is due to a incorrect token. The user will modify it later. \n", "\n", "### Update the list of datasets (and create the links)\n", "\n", "Now, the user can concatenate their search to create a list of downloading links. " ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "WARNING in EGMS-toolkit processing: in EGMStoolkit.classes.EGMSdownloaderapi.checkparameter\n", "\t/Users/alexis_hrysiewicz/Work/UCD_InSAR_Scripts/EGMStoolkit/src/EGMStoolkit/classes/EGMSdownloaderapi.py\n", "\t\t--> The user token is not correct.\n", "WARNING in EGMS-toolkit processing: in EGMStoolkit.classes.EGMSdownloaderapi.checkparameter\n", "\t/Users/alexis_hrysiewicz/Work/UCD_InSAR_Scripts/EGMStoolkit/src/EGMStoolkit/classes/EGMSdownloaderapi.py\n", "\t\t--> The user token is not correct.\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "downloadpara.updatelist(infoS1ROIparameter=ROIpara)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And the user can displayed the lists of datasets." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "WARNING in EGMS-toolkit processing: in EGMStoolkit.classes.EGMSdownloaderapi.checkparameter\n", "\t/Users/alexis_hrysiewicz/Work/UCD_InSAR_Scripts/EGMStoolkit/src/EGMStoolkit/classes/EGMSdownloaderapi.py\n", "\t\t--> The user token is not correct.\n", "################################################################################\n", "EGMStoolkit 0.2.1 Beta\n", "\n", "--------------------------------------------------------------------------------\n", "EGMStoolkit.classes.EGMSdownloaderapi.EGMStoolkit.classes.EGMSdownloaderapi:\n", "\tPrint the list(s) of EGMS files\n", "\n", "\tScript: /Users/alexis_hrysiewicz/Work/UCD_InSAR_Scripts/EGMStoolkit/src/EGMStoolkit/classes/EGMSdownloaderapi.py\n", "\t\tCopyright 2024, UCD / iCRAG\n", "################################################################################\n", "\n", "For the EGMS data: L2b\n", "\t File 1: EGMS_L2b_001_0314_IW2_VV_2018_2022_1.zip (Release 2018_2022)\n", "\t File 2: EGMS_L2b_001_0314_IW3_VV_2018_2022_1.zip (Release 2018_2022)\n", "\t File 3: EGMS_L2b_001_0315_IW2_VV_2018_2022_1.zip (Release 2018_2022)\n", "\t File 4: EGMS_L2b_001_0315_IW3_VV_2018_2022_1.zip (Release 2018_2022)\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "downloadpara.printlist(verbose=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Concatenate other searches\n", "\n", "Often, the user would like to download datasets with different parameters (i.e., tracks). **EGMS-toolkit** can concatenate other searches to optimise the downloading time. \n", "\n", "Firstly, the next lines will add the search for L3 vertical EGMS datasets of the 2015-2021 release. " ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "WARNING in EGMS-toolkit processing: in EGMStoolkit.classes.EGMSS1ROIapi.checkfile\n", "\t/Users/alexis_hrysiewicz/Work/UCD_InSAR_Scripts/EGMStoolkit/src/EGMStoolkit/classes/EGMSS1ROIapi.py\n", "\t\t--> The use of the S1 burst ID map is less accurate than the use of .xml S1 files.\n", "WARNING in EGMS-toolkit processing: in EGMStoolkit.classes.EGMSdownloaderapi.checkparameter\n", "\t/Users/alexis_hrysiewicz/Work/UCD_InSAR_Scripts/EGMStoolkit/src/EGMStoolkit/classes/EGMSdownloaderapi.py\n", "\t\t--> The user token is not correct.\n", "WARNING in EGMS-toolkit processing: in EGMStoolkit.classes.EGMSdownloaderapi.checkparameter\n", "\t/Users/alexis_hrysiewicz/Work/UCD_InSAR_Scripts/EGMStoolkit/src/EGMStoolkit/classes/EGMSdownloaderapi.py\n", "\t\t--> The user token is not correct.\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ROIpara.egmslevel = 'L3'\n", "ROIpara.egmsL3component = 'UD'\n", "ROIpara.release = '2015_2021'\n", "ROIpara.detectfromIDmap(infoburstID=info,verbose=False)\n", "downloadpara.updatelist(infoS1ROIparameter=ROIpara,verbose=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Secondly, we will also add an other search for L3 horizontal EGMS datasets of the 2018-2022 release." ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "WARNING in EGMS-toolkit processing: in EGMStoolkit.classes.EGMSS1ROIapi.checkfile\n", "\t/Users/alexis_hrysiewicz/Work/UCD_InSAR_Scripts/EGMStoolkit/src/EGMStoolkit/classes/EGMSS1ROIapi.py\n", "\t\t--> The use of the S1 burst ID map is less accurate than the use of .xml S1 files.\n", "WARNING in EGMS-toolkit processing: in EGMStoolkit.classes.EGMSdownloaderapi.checkparameter\n", "\t/Users/alexis_hrysiewicz/Work/UCD_InSAR_Scripts/EGMStoolkit/src/EGMStoolkit/classes/EGMSdownloaderapi.py\n", "\t\t--> The user token is not correct.\n", "WARNING in EGMS-toolkit processing: in EGMStoolkit.classes.EGMSdownloaderapi.checkparameter\n", "\t/Users/alexis_hrysiewicz/Work/UCD_InSAR_Scripts/EGMStoolkit/src/EGMStoolkit/classes/EGMSdownloaderapi.py\n", "\t\t--> The user token is not correct.\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ROIpara.egmslevel = 'L3'\n", "ROIpara.egmsL3component = 'EW'\n", "ROIpara.release = '2018_2022'\n", "downloadpara.updatelist(infoS1ROIparameter=ROIpara.detectfromIDmap(infoburstID=info))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And the user can print the new list. " ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "WARNING in EGMS-toolkit processing: in EGMStoolkit.classes.EGMSdownloaderapi.checkparameter\n", "\t/Users/alexis_hrysiewicz/Work/UCD_InSAR_Scripts/EGMStoolkit/src/EGMStoolkit/classes/EGMSdownloaderapi.py\n", "\t\t--> The user token is not correct.\n", "################################################################################\n", "EGMStoolkit 0.2.1 Beta\n", "\n", "--------------------------------------------------------------------------------\n", "EGMStoolkit.classes.EGMSdownloaderapi.EGMStoolkit.classes.EGMSdownloaderapi:\n", "\tPrint the list(s) of EGMS files\n", "\n", "\tScript: /Users/alexis_hrysiewicz/Work/UCD_InSAR_Scripts/EGMStoolkit/src/EGMStoolkit/classes/EGMSdownloaderapi.py\n", "\t\tCopyright 2024, UCD / iCRAG\n", "################################################################################\n", "\n", "For the EGMS data: L2b\n", "\t File 1: EGMS_L2b_001_0314_IW2_VV_2018_2022_1.zip (Release 2018_2022)\n", "\t File 2: EGMS_L2b_001_0314_IW3_VV_2018_2022_1.zip (Release 2018_2022)\n", "\t File 3: EGMS_L2b_001_0315_IW2_VV_2018_2022_1.zip (Release 2018_2022)\n", "\t File 4: EGMS_L2b_001_0315_IW3_VV_2018_2022_1.zip (Release 2018_2022)\n", "For the EGMS data: L3UD\n", "\t File 1: EGMS_L3_E32N34_100km_U.zip (Release 2015_2021)\n", "For the EGMS data: L3EW\n", "\t File 1: EGMS_L3_E32N34_100km_E_2018_2022_1.zip (Release 2018_2022)\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "downloadpara.printlist(verbose=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Change the user token \n", "\n", "In order to dowload the EGMS data, the user requires to be authorised on the EGMS servers. The hack is to use the user token which can be found, online, at the end of downloading links. \n", "\n", "**This token is time-limited.**" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [], "source": [ "downloadpara.token = 'XXXXXXXXXXX'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Download the EGMS data\n", "\n", "Now, it is possible to download the data. All the used options are optionnal and are defined by default. " ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "downloadpara.download(outputdir='./Output',unzipmode=False,cleanmode=False) " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Unzip the files\n", "\n", "The downlaoded files are .zip archives. The following method will unzip the archives. \n", "\n", "**However, it is possible to unzip the file directly with the *download* method by using the *unzipmode* option.** \n", "\n", "*nbworker* option enables the parallel unzipping (number of workers)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "################################################################################\n", "EGMStoolkit 0.2.1 Beta\n", "\n", "--------------------------------------------------------------------------------\n", "EGMStoolkit.classes.EGMSdownloaderapi.EGMStoolkit.classes.EGMSdownloaderapi:\n", "\tUnzip the EGMS files\n", "\n", "\tScript: /Users/alexis_hrysiewicz/Work/UCD_InSAR_Scripts/EGMStoolkit/src/EGMStoolkit/classes/EGMSdownloaderapi.py\n", "\t\tCopyright 2024, UCD / iCRAG\n", "################################################################################\n", "\n", "1 / 6 files: Unzip the file: EGMS_L3_E32N34_100km_E_2018_2022_1.zip\n", "2 / 6 files: Unzip the file: EGMS_L3_E32N34_100km_U.zip\n", "3 / 6 files: Unzip the file: EGMS_L2b_001_0314_IW3_VV_2018_2022_1.zip\n", "4 / 6 files: Unzip the file: EGMS_L2b_001_0315_IW3_VV_2018_2022_1.zip\n", "5 / 6 files: Unzip the file: EGMS_L2b_001_0315_IW2_VV_2018_2022_1.zip\n", "6 / 6 files: Unzip the file: EGMS_L2b_001_0314_IW2_VV_2018_2022_1.zip\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "downloadpara.unzipfile(outputdir='./Output',unzipmode=True,nbworker=2,cleanmode=True,verbose=True) " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Clean the unused files\n", "\n", "Then, the user can delete the files that are not in the lists. " ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "downloadpara.clean(outputdir='./Output',verbose=False) " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Post-processing of the EGMS data\n", "\n", "The files are in the *output* directory: \n", "\n", "
\n",
    "├── Output\n",
    "│   ├── L2a\n",
    "│   ├── L2b\n",
    "│   │   ├── 2018_2022\n",
    "│   │   │   ├── EGMS_L2b_001_0314_IW2_VV_2018_2022_1\n",
    "│   │   │   │   ├── EGMS_L2b_001_0314_IW2_VV_2018_2022_1.csv\n",
    "│   │   │   │   ├── EGMS_L2b_001_0314_IW2_VV_2018_2022_1.xml\n",
    "│   │   │   ├── EGMS_L2b_001_0315_IW2_VV_2018_2022_1\n",
    "│   │   │   │   ├── EGMS_L2b_001_0315_IW2_VV_2018_2022_1.csv\n",
    "│   │   │   │   ├── EGMS_L2b_001_0315_IW2_VV_2018_2022_1.xml\n",
    "│   │   │   ├── EGMS_L2b_001_0314_IW3_VV_2018_2022_1\n",
    "│   │   │   │   ├── EGMS_L2b_001_0314_IW3_VV_2018_2022_1.csv\n",
    "│   │   │   │   ├── EGMS_L2b_001_0314_IW3_VV_2018_2022_1.xml\n",
    "│   │   │   ├── EGMS_L2b_001_0315_IW3_VV_2018_2022_1\n",
    "│   │   │   │   ├── EGMS_L2b_001_0315_IW3_VV_2018_2022_1.csv\n",
    "│   │   │   │   ├── EGMS_L2b_001_0316_IW3_VV_2018_2022_1.xml\n",
    "│   ├── L3EW\n",
    "│   │   ├── 2018_2022\n",
    "│   │   │   ├── EGMS_L3_E32N34_100km_E_2018_2022_1\n",
    "│   │   │   │   ├── EGMS_L3_E32N34_100km_E_2018_2022_1.csv\n",
    "│   │   │   │   ├── EGMS_L3_E32N34_100km_E_2018_2022_1.tiff\n",
    "│   │   │   │   ├── EGMS_L3_E32N34_100km_E_2018_2022_1.xml\n",
    "│   ├── L3UD\n",
    "│   │   ├── 2015_2021\n",
    "│   │   │   ├── EGMS_L3_E32N34_100km_U\n",
    "│   │   │   │   ├── EGMS_L3_E32N34_100km_U.csv\n",
    "│   │   │   │   ├── EGMS_L3_E32N34_100km_U.tiff\n",
    "│   │   │   │   ├── EGMS_L3_E32N34_100km_U.xml\n",
    "
\n", "\n", "It therefore is smarter to post-process the files. \n", "\n", "**Of course, all steps are optional.**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Merge the .csv files\n", "\n", "The first post-processing is the merging of .csv. The user can define several options: \n", "- *outputdir* the ouput directory; \n", "- *inputdir* the input directory (where are stored the EGMS .csv files); \n", "- *verbose* the verbose mode\n", "- *paratosave* parameter regarding the EGMS names ['all' or string value]. ['latitude', 'longitude', 'easting', 'northing', 'height', 'height_wgs84'] which will always be saved.\n", "\n", "**The last option is *mode*. Indeed, the scripts can find the .csv files based on the search lists (onlist mode) or on the stored files (onfiles mode). For the last mode, the *downloadpara* class is required.**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Three other options are available: \n", "\n", "* __removeduplicate__ (bool): Reduce the duplicate points [Default: True]\n", "* __length_threshold__ (int): Length for the concave hull [Default: 1000]\n", "* __usevrtmerging__ (bool): Use the vrt for merging [Default: False]" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "################################################################################\n", "EGMStoolkit 0.2.1 Beta\n", "\n", "--------------------------------------------------------------------------------\n", "EGMStoolkit.functions.egmsdatatools.EGMStoolkit.functions.egmsdatatools:\n", "\tMerge the datasets in csv format\n", "\n", "\tScript: /Users/alexis_hrysiewicz/Work/UCD_InSAR_Scripts/EGMStoolkit/src/EGMStoolkit/functions/egmsdatatools.py\n", "\t\tCopyright 2024, UCD / iCRAG\n", "################################################################################\n", "\n", "\tOutput Directory: ./Output\n", "\tInput Directory: ./Output\n", "\tSelected parameters: all\n", "\tMode: onlist\n", "Merging for EGMS_L3_2015_2021_UD...\n", "Merging for EGMS_L2b_001_VV_2018_2022_1...\n", "Merging for EGMS_L3_2018_2022_1_EW...\n" ] } ], "source": [ "egmsdatatools.datamergingcsv(infoEGMSdownloader=downloadpara,inputdir='./Output',outputdir='./Output',mode='onlist',verbose=True,paratosave='all')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Merge the .tiff files (only for the vertical and horizontal displacements)\n", "\n", "The .tiff can also be merged. The options are similar. \n", "\n", "**The *paratosave* option does not exist here.**" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "################################################################################\n", "EGMStoolkit 0.2.1 Beta\n", "\n", "--------------------------------------------------------------------------------\n", "EGMStoolkit.functions.egmsdatatools.EGMStoolkit.functions.egmsdatatools:\n", "\tMerge the datasets in tiff format (only for L3 level)\n", "\n", "\tScript: /Users/alexis_hrysiewicz/Work/UCD_InSAR_Scripts/EGMStoolkit/src/EGMStoolkit/functions/egmsdatatools.py\n", "\t\tCopyright 2024, UCD / iCRAG\n", "################################################################################\n", "\n", "\tOutput Directory: ./Output\n", "\tInput Directory: ./Output\n", "\tSelected parameters: all\n", "\tMode: onlist\n", "Merging for EGMS_L3_2015_2021_UD...\n", "Used command: gdal_merge.py -o ./Output/EGMS_L3_2015_2021_UD.tiff -n -9999 -a_nodata -9999 ./Output/L3UD/2015_2021/EGMS_L3_E32N34_100km_U/EGMS_L3_E32N34_100km_U.tiff\n", "0...10...20...30...40...50...60...70...80...90...100 - done.\n", "Merging for EGMS_L3_2018_2022_1_EW...\n", "Used command: gdal_merge.py -o ./Output/EGMS_L3_2018_2022_1_EW.tiff -n -9999 -a_nodata -9999 ./Output/L3EW/2018_2022/EGMS_L3_E32N34_100km_E_2018_2022_1/EGMS_L3_E32N34_100km_E_2018_2022_1.tiff\n", "0...10...20...30...40...50...60...70...80...90...100 - done.\n" ] } ], "source": [ "egmsdatatools.datamergingtiff(infoEGMSdownloader=downloadpara,inputdir='./Output',outputdir='./Output',mode='onlist',verbose=True) " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Clip/crop the data\n", "\n", "The user can clip or crop the files regarding the ROI (or another shapefile).\n", "\n", "The new parameter *namefile* allows to define the file. By default, 'all' will clip/crop all the files. The option __clipuseogr2ogr__ (bool, optional) can be used to force the use of ogr2ogr for clipping (default is False)." ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "################################################################################\n", "EGMStoolkit 0.2.1 Beta\n", "\n", "--------------------------------------------------------------------------------\n", "EGMStoolkit.functions.egmsdatatools.EGMStoolkit.functions.egmsdatatools:\n", "\tClip the dataset(s)\n", "\n", "\tScript: /Users/alexis_hrysiewicz/Work/UCD_InSAR_Scripts/EGMStoolkit/src/EGMStoolkit/functions/egmsdatatools.py\n", "\t\tCopyright 2024, UCD / iCRAG\n", "################################################################################\n", "\n", "\tThe file name is: all\n", "\tInput Directory: ./Output\n", "\tOutput Directory: ./Output\n", "\tShapefile: bbox.shp\n", "\t1 / 5 file(s): Clip the file ./Output/EGMS_L3_2015_2021_UD.csv to ./Output/EGMS_L3_2015_2021_UD_clipped.csv...\n", "\t2 / 5 file(s): Clip the file ./Output/EGMS_L3_2018_2022_1_EW.csv to ./Output/EGMS_L3_2018_2022_1_EW_clipped.csv...\n", "\t3 / 5 file(s): Clip the file ./Output/EGMS_L2b_001_VV_2018_2022_1.csv to ./Output/EGMS_L2b_001_VV_2018_2022_1_clipped.csv...\n", "\t4 / 5 file(s): Crop the file ./Output/EGMS_L3_2015_2021_UD.tiff to ./Output/EGMS_L3_2015_2021_UD_cropped.tiff...\n", "\t5 / 5 file(s): Crop the file ./Output/EGMS_L3_2018_2022_1_EW.tiff to ./Output/EGMS_L3_2018_2022_1_EW_cropped.tiff...\n" ] } ], "source": [ "egmsdatatools.dataclipping(inputdir='./Output',outputdir='./Output',namefile='all',shapefile='bbox.shp',verbose=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Clean the raw files and directories\n", "\n", "To save space, the raw files and directories can be deleted." ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "################################################################################\n", "EGMStoolkit 0.2.1 Beta\n", "\n", "--------------------------------------------------------------------------------\n", "EGMStoolkit.functions.egmsdatatools.EGMStoolkit.functions.egmsdatatools:\n", "\tDelete the raw dataset(s) from EGMS\n", "\n", "\tScript: /Users/alexis_hrysiewicz/Work/UCD_InSAR_Scripts/EGMStoolkit/src/EGMStoolkit/functions/egmsdatatools.py\n", "\t\tCopyright 2024, UCD / iCRAG\n", "################################################################################\n", "\n", "\tInput Directory: ./Output\n", "Deleting...\n", "Deleting...\n" ] } ], "source": [ "egmsdatatools.removerawdata(inputdir='./Output',verbose=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Post-processing for advanced users\n", "\n", "The next sections show some other commands to post-process the data. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Interpolation of point data (.csv) into a .tif raster file \n", "\n", "The first command is the possibility to interpolate the point data (from .csv files) into a .tif raster file. Of course, this step can be can be done before the cropping/clipping step. \n", "\n", "#### Creation of the dict. for the gridding parameters\n", "\n", "The user requires to define a dictionary with the gridding parameter. " ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [], "source": [ "paragrid = dict()\n", "paragrid['Xmin'] = 2896000 # Minimal X coordinate in EPGS:3035\n", "paragrid['Ymin'] = 3317250 # Minimal Y coordinate in EPGS:3035\n", "paragrid['Xmax'] = 3359000 # Maximal X coordinate in EPGS:3035\n", "paragrid['Ymax'] = 3745500 # Maximal Y coordinate in EPGS:3035\n", "paragrid['xres'] = 50 # X spatial resolution in EPGS:3035\n", "paragrid['yres'] = 50 # Y spatial resolution in EPGS:3035\n", "paragrid['algo'] = 'average:radius1=50:radius2=50:angle=0.0:nodata=-9999' # Alfgorithm used and options\n", "paragrid['variable'] = 'mean_velocity,mean_velocity_std,acceleration,acceleration_std,seasonality,seasonality_std'\n", "# paragrid['algo'] = 'invdist:power=2.0:smoothing=0.0:radius1=0.0:radius2=0.0:angle=0.0:max_points=0:min_points=0:nodata=0.0'\n", "# paragrid['algo'] = 'invdistnn:power=2.0:radius=1.0:max_points=12:min_points=0:nodata=0'\n", "# paragrid['algo'] = 'average:radius1=0.0:radius2=0.0:angle=0.0:min_points=0:nodata=0.0'\n", "# paragrid['algo'] = 'nearest:radius1=0.0:radius2=0.0:angle=0.0:nodata=0.0'\n", "# paragrid['algo'] = 'linear:radius=-1.0:nodata=0.0'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And the user can run the interpolation. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "egmsdatatools.datagridding(inputdir='./Output',outputdir='./Output',namefile='all',verbose=True,paragrid=paragrid)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Conversion of .csv files to other vector format\n", "\n", "The last command will convert the .csv files to other vector formats. The available formats are: \n", "\n", "- ESRI Shapefile (default); \n", "- GPKG; \n", "- GeoJSON." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "egmsdatatools.convertcsv(verbose=False)" ] } ], "metadata": { "kernelspec": { "display_name": "EGMStookit", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.13" } }, "nbformat": 4, "nbformat_minor": 2 }