Check number of products available in portals (colhub, colhub AOI, scihub, colhub ESA)

Aim: check that colhub portals at MET have the same number of products as scihub

Use sentinelAPI to request each dhus FE instance, to check how many products are available for any sensing date.

Uses outputs from script nbs@ppi:/home/nbs/colhub/monitoring_portals_cron.sh that is run every night with cron.

If outputs look suspiscious (zeros for all products for a FE, ...), one should re-run the script on PPI, using either:

• the cron script nbs@ppi:/home/nbs/colhub/monitoring_portals_cron.sh if one wants to re-run all data (do not modify this script)
• its 'twin' nbs@ppi:/home/nbs/colhub/monitoring_portals_twin.sh if only a subset of products or FE needs to be re-computed (full freedom to modify this script)

It is not a problem to re-run the scripts again and again on the same dates. The script below only plots the latest data for each date. So, when there has been a colhub incident, the query script and notebook can be used to check that missing data have been synchronized.

Read products available in all backends & frontends

Read Frontends data

Read Backends data

Merge FE and BE

Plot statistics in global frontends: colhub, colhub ESA, scihub

>

Zoom on the last 30 days

>

	colhub_global	scihub	esahub_global	BE
sensing_date
2022-09-26	2303	2277	2277.0	2277
2022-09-27	2234	2187	2187.0	2187
2022-09-28	2084	2046	2050.0	2058

>

Zoom on the last 30 days

>

	colhub_global	scihub	esahub_global	BE
sensing_date
2022-09-23	12282	12282	12282.0	12282
2022-09-24	12443	12443	12443.0	12443
2022-09-25	12838	12838	12838.0	12838
2022-09-26	11999	11999	11999.0	11999
2022-09-27	12332	12332	12332.0	12332
2022-09-28	11950	11950	11950.0	11950

>

Zoom on the last 30 days

>

	colhub_global	scihub	esahub_global	BE
sensing_date
2022-09-26	12000	12000	12000.0	12000
2022-09-27	12332	12332	12332.0	12332
2022-09-28	11161	11339	11338.0	11352

>

Zoom on the last 30 days

>

	colhub_global	scihub	esahub_global	BE
sensing_date
2022-09-26	8465	8465	8465.0	8465
2022-09-27	7862	7899	7897.0	7899
2022-09-28	4775	4796	4779.0	4779

>

Zoom on the last 30 days

>

	colhub_global	scihub	BE
sensing_date
2022-09-19	1907	1907	1907
2022-09-20	1916	1916	1916
2022-09-21	1752	1752	1752
2022-09-22	1871	1871	1871
2022-09-23	1927	1927	1927
2022-09-24	1799	1799	1799
2022-09-25	1907	1907	1907
2022-09-26	1880	1880	1880
2022-09-27	1762	1764	1764
2022-09-28	1719	1723	1723

Plot statistics in AOI frontend and colhub/scihub only over AOI

>

Zoom on the last 30 days

>

	colhub_global	scihub	colhub_AOI	BE
sensing_date
2022-09-26	256	230	255.0	255
2022-09-27	292	245	287.0	287
2022-09-28	267	229	265.0	265

>

Zoom on the last 30 days

>

	colhub_global	scihub	colhub_AOI	BE
sensing_date
2022-09-19	1246	1246	1240.0	1240
2022-09-20	1286	1286	1276.0	1276
2022-09-21	1204	1204	1196.0	1196
2022-09-22	1206	1206	1195.0	1195
2022-09-23	1123	1123	1116.0	1116
2022-09-24	1177	1177	1169.0	1169
2022-09-25	1112	1112	1105.0	1105
2022-09-26	1075	1075	1067.0	1067
2022-09-27	1031	1031	1019.0	1019
2022-09-28	1075	1075	1069.0	1069

>

Zoom on the last 30 days

>

	colhub_global	scihub	colhub_AOI	BE
sensing_date
2022-09-19	1246	1246	1240.0	1240
2022-09-20	1286	1286	1276.0	1276
2022-09-21	1204	1204	1196.0	1196
2022-09-22	1206	1206	1195.0	1195
2022-09-23	1123	1123	1116.0	1116
2022-09-24	1177	1177	1169.0	1169
2022-09-25	1112	1112	1105.0	1105
2022-09-26	1075	1075	1067.0	1067
2022-09-27	1031	1031	1019.0	1019
2022-09-28	1075	1075	1069.0	1069

>

Zoom on the last 30 days

>

	colhub_global	scihub	colhub_AOI	BE
sensing_date
2022-09-19	1007	1018	986.0	986
2022-09-20	1002	1022	991.0	991
2022-09-21	989	998	980.0	980
2022-09-22	970	978	965.0	965
2022-09-23	923	923	919.0	919
2022-09-24	914	914	897.0	897
2022-09-25	911	911	901.0	901
2022-09-26	895	895	892.0	892
2022-09-27	803	824	798.0	801
2022-09-28	438	438	433.0	433

>

Zoom on the last 30 days

>

	colhub_global	scihub	colhub_AOI	BE
sensing_date
2022-09-26	577	577	577.0	577
2022-09-27	460	462	460.0	460
2022-09-28	328	332	331.0	332