Alert refactor merge (#29)

* New feed facade and celery task improvements

* Update main_cap-aggregator(dev).yml

* added separate configurations for each cap source format

* added region data

* reset migrations

* removed region usage before declaration

* fixed variation scope issue

* created migration

* injecting regions

* injecting countries (#7)

* matched CAP alerts to country polygons

* Integrated Celery and RabbitMQ

* Preparation for region data injection (#6)

* Update main_cap-aggregator(dev).yml

* added separate configurations for each cap source format

* added region data

* reset migrations

* removed region usage before declaration

* fixed variation scope issue

* Add or update the Azure App Service build and deployment workflow config

* add celery tasks

---------

Co-authored-by: HangZhou01 <[email protected]>

* added celery startup commands

* changed startup.sh

* added filter for unexpired alerts, fixed cors policy

* enabled github workflow tests for all branches, whitenoise package version bump

* fixed cors policy origin path

* disabled celery for deployment

* reverted cors policy

* New feed facade and celery task improvements

* Add or update the Azure App Service build and deployment workflow config

* add celery tasks

* merge the develop branch

* Polling Feeds at Different Rate

* Automatically Polling Alerts when a New Feed is created

* Automatically Deleting or Updating Task when there is a Feed gets deleted or updated

* Fix bugs on Automatically Deleting Updating Task when there is a Feed gets deleted or updated

* fixed minor bugs

* refactored celery code, fixed lots of bugs

---------

Co-authored-by: HangZhou01 <[email protected]>

* fixed celery task name issue

* fixed naming issue

* resolved conflicts between develop and main

* fixed merge issue

---------

Co-authored-by: HangZhou01 <[email protected]>

* New country geodata, feed facade redesign, graphql filters

* Update main_cap-aggregator(dev).yml

* added separate configurations for each cap source format

* added region data

* reset migrations

* removed region usage before declaration

* fixed variation scope issue

* created migration

* injecting regions

* injecting countries (#7)

* matched CAP alerts to country polygons

* Integrated Celery and RabbitMQ

* Preparation for region data injection (#6)

* Update main_cap-aggregator(dev).yml

* added separate configurations for each cap source format

* added region data

* reset migrations

* removed region usage before declaration

* fixed variation scope issue

* Add or update the Azure App Service build and deployment workflow config

* add celery tasks

---------

Co-authored-by: HangZhou01 <[email protected]>

* added celery startup commands

* changed startup.sh

* added filter for unexpired alerts, fixed cors policy

* enabled github workflow tests for all branches, whitenoise package version bump

* fixed cors policy origin path

* disabled celery for deployment

* reverted cors policy

* New feed facade and celery task improvements

* Add or update the Azure App Service build and deployment workflow config

* add celery tasks

* merge the develop branch

* Polling Feeds at Different Rate

* Automatically Polling Alerts when a New Feed is created

* Automatically Deleting or Updating Task when there is a Feed gets deleted or updated

* Fix bugs on Automatically Deleting Updating Task when there is a Feed gets deleted or updated

* fixed minor bugs

* refactored celery code, fixed lots of bugs

---------

Co-authored-by: HangZhou01 <[email protected]>

* fixed celery task name issue

* fixed naming issue

* resolved conflicts between develop and main

* fixed merge issue

* Added continents, changed country polygon data source

* New feed facade and celery task improvements

* Update main_cap-aggregator(dev).yml

* added separate configurations for each cap source format

* added region data

* reset migrations

* removed region usage before declaration

* fixed variation scope issue

* created migration

* injecting regions

* injecting countries (#7)

* matched CAP alerts to country polygons

* Integrated Celery and RabbitMQ

* Preparation for region data injection (#6)

* Update main_cap-aggregator(dev).yml

* added separate configurations for each cap source format

* added region data

* reset migrations

* removed region usage before declaration

* fixed variation scope issue

* Add or update the Azure App Service build and deployment workflow config

* add celery tasks

---------

Co-authored-by: HangZhou01 <[email protected]>

* added celery startup commands

* changed startup.sh

* added filter for unexpired alerts, fixed cors policy

* enabled github workflow tests for all branches, whitenoise package version bump

* fixed cors policy origin path

* disabled celery for deployment

* reverted cors policy

* New feed facade and celery task improvements

* Add or update the Azure App Service build and deployment workflow config

* add celery tasks

* merge the develop branch

* Polling Feeds at Different Rate

* Automatically Polling Alerts when a New Feed is created

* Automatically Deleting or Updating Task when there is a Feed gets deleted or updated

* Fix bugs on Automatically Deleting Updating Task when there is a Feed gets deleted or updated

* fixed minor bugs

* refactored celery code, fixed lots of bugs

---------

Co-authored-by: HangZhou01 <[email protected]>

* fixed celery task name issue

* fixed naming issue

* resolved conflicts between develop and main

* fixed merge issue

---------

Co-authored-by: HangZhou01 <[email protected]>

* added continents, changed country polygon data source

* merge continents

---------

Co-authored-by: HangZhou01 <[email protected]>

* fixed merge duplication

* fixed inconsistencies

* -enabled geo injection

* fixed graphql schema

* deployment fix: inject continents only

* deployment fix: changed celery task name

* deployment fix: inject continents

* deployment fix: removed save to database

* deployment fix: try except inject continents

* deployment fix: try except error message

* deployment fix: try except print error

* deployment fix: changed file path

* deployment fix: changed file path

* deployment fix: replaced filename dashes with underscores

* deployment fix: changed relative paths

* multipolygons added for countries

* improved admin fields, filters, and displays, added cors, fixed celery tasks bug, fixed duplicate iso3 sources, updated settings.py celery format

* added description, senderName, source CAP alert fields, added more sources, standardised [long, lat] coordinates, added more admin filters, displays

* added graphql filters for alerts and countries

---------

Co-authored-by: HangZhou01 <[email protected]>

* Update README.md

Fixed documentation mistake

* Alert model changed to comply with CAP 1.2, changed format structure to be more modular to allow for complex alert parsing

* added meteoalarm, aws, nws_us alert feed advanced formats

* fixed broken tests from alert model change

* fixed feed format NoneType errors, tidied feed facade

* updated graphql for new Alert and AlertInfo structures

* revised README.md

* removed old file

* fixed merge issues

---------

Co-authored-by: HangZhou01 <[email protected]>

README.md

@@ -1,6 +1,6 @@
 # IFRC/UCL Alert Hub - CAP Aggregator
 
-The CAP Aggregator is an alert aggregation service built for IFRC's Alert Hub. Public alerts use the Common Alerting Protocol (CAP) international standard.
+The CAP Aggregator is an alert aggregation service built for IFRC's Alert Hub. Public alerts use the Common Alerting Protocol (CAP) Version 1.2 standard.
 
 This is a Python web app using the Django framework and the Azure Database for PostgreSQL relational database service. The Django app is hosted in a fully managed Azure App Service. Requests to hundreds of publicly available alert sources are managed by Celery and Redis. Aggregated alerts are then made available to the Alert Hub via a GraphQL API.
 
@@ -37,26 +37,24 @@ The allocation of countries into regions and continents is necessary for easier
 ## Alert Aggregation Process
 *Alerts are retrieved and processed before they are handed off to be displayed on the Alert Hub and alert subscription system.*
 
-New alert sources are added from the Feed Facade and polling intervals are used to adjust the frequency of requests to the alerting source. Alert sources with the same polling interval are automatically grouped into the same periodic task used by the *Celery Beat* scheduler. These tasks are handed off to *Redis* and *Celery* workers for asynchronous background processing.
+New alert sources are added by an admin from the Feed Facade and polling intervals are used to adjust the frequency of requests to the alerting source. Alert sources with the same polling interval are automatically grouped into the same periodic task used by the *Celery Beat* scheduler. These tasks are handed off to *Redis* and *Celery* workers for asynchronous background processing.
 
 When processing the CAP feed of alerting sources, a processing template or format is used to interpret the different xml formats. For example, Meteoalarm represents a network of public weather services across Europe, and these European countries encode their cap alerts in the same xml format. The 'meteoalarm' format can therefore be selected when adding MeteoAlarm alerting sources in the feed facade, but a different format would need to be used to interpret alerts from the Algerian Meteorological Office.
 
-Formats are very convenient for admin users but have an inevitable drawback — they have to be created manually by developers and updated if alerting sources make changes to their CAP alerts. The number of available formats is currently limited, but we intend to greatly expand on this number as we complete other prioritised tasks such as work relating to *Django Channels*.
+Formats are very convenient for admin users and can guarantee alerts are processed correctly. But they inevitably have to be manually created by developers and updated if alerting sources make changes to their alert feed format.
 
-CAP alerts including all kinds of metadata specific to each format is interpreted and saved to the Postgresql database. Dates and times are standardised across the system using the UTC timezone. Some alerting sources keep outdated alerts on their alert feeds, so expired alerts are identified and are not saved.
+The CAP-aggregator processes alerts according to the CAP-V1.2 specification which details alert elements and sub-elements such as *info*. Dates and times are standardised across the system using the UTC timezone. Some alerting sources keep outdated alerts on their alert feeds, so expired alerts are identified and are not saved.
 
-Another periodic task for removing expired alerts also runs continously in the background. This task is responsible for identifying and removing alerts which have expired since being saved to the database.
+Another periodic task for removing expired alerts also runs continously in the background. This task is responsible for identifying and removing alerts which have expired since being saved to the database. However, the alert expiry date and time is contained in the *info* element according to CAP V-1.2. Therefore it is theoretically possible for multiple *info* elements to have different expiry times. Expired *info* elements are automatically removed, and the *alert* element (the actual alert itself) will be removed if all *info* elements have expired or been removed.
 
 Alerts are aggregated by countries, regions, and continents. Using filtered queries with GraphQL, the Alert Hub and other broadcasters can easily fetch only the relevant alerts, reducing unnecessary strain on the system.
 
-> During development, we have discovered that a significant number of alerts are issued but then retracted by alerting sources before the stated expiry time. This may be a problem for alert subscriptions and we are currently discussing on our strategy for this issue.
-
 ## Feed Facade
 *Admin users can manage countries, regions, continents, sources, and each individual alert using the Feed Facade.*
 
 Each alerting source and their alerts belong to a country, and each country belongs to a particular region and continent. Therefore, it is necessary for regions and continents to exist first before countries can be added (although all regions and continents already exist in the system). Similarly, a new country needs be created by an admin user before a new alerting source can be added for that country.
 
-Deleting a region or continent would delete all countries belonging to them. In a chain reaction, all alerts and sources belonging to the deleted countries would also be deleted. This current behaviour is clearly unsafe and possibly undesirable but is convenient for development. This can be changed after some discussion. Lastly, deleting an alerting source does not delete existing alerts from the same country. This is because alerts belong to the country of the alerting source, but not to the sources themselves.
+Deleting a region or continent would delete all countries belonging to them. In a chain reaction, all alerts and sources belonging to the deleted countries would also be deleted. This current behaviour is possibly unsafe and undesirable but is convenient for development. Lastly, deleting an alerting source also deletes existing alerts from the same country.
 
 Search functions, filters and sortable columns are available when they would be relevant. For example, an admin user could filter sources by format (e.g., meteoalarm) or search for sources belonging to a particular country using the search bar on the same page.
 
@@ -83,4 +81,4 @@ Start celery worker and sceduler for local development:
 ```
 celery -A capaggregator worker --pool=solo -l info
 celery -A capaggregator beat -l info
-```
+```

cap_feed/admin.py

@@ -1,13 +1,30 @@
 from django.contrib import admin
-from .models import Alert, Continent, Region, Country, Source
+from .models import Alert, AlertInfo, Continent, Region, Country, Source
 from django_celery_beat.models import CrontabSchedule, ClockedSchedule, SolarSchedule, IntervalSchedule
 from django_celery_results.models import GroupResult
 
 
 
+class AlertInfoAdmin(admin.ModelAdmin):
+    list_display = ["alert", "language"]
+    list_filter = ["alert__source_feed"]
+    fieldsets = [
+        ("Administration", {"fields": ["alert"]}),
+        ("Alert Info" , {"fields": ["language", "category", "event", "response_type", "urgency", "severity", "certainty", "audience", "event_code", "effective", "onset", "expires", "sender_name", "headline", "description", "instruction", "web", "contact", "parameter"]}),
+    ]
+
+class AlertInfoInline(admin.StackedInline):
+    model = AlertInfo
+    extra = 0
+
 class AlertAdmin(admin.ModelAdmin):
-    list_display = ["id", "source", "urgency", "severity", "certainty", "sent", "effective", "expires"]
-    list_filter = ["source", "urgency", "severity", "certainty", "sent", "effective", "expires"]
+    list_display = ["id", "source_feed", "sent"]
+    list_filter = ["source_feed", "country"]
+    fieldsets = [
+        ("Administration", {"fields": ["country", "source_feed"]}),
+        ("Alert Header" , {"fields": ["identifier", "sender", "sent", "status", "msg_type", "source", "scope", "restriction", "addresses", "code", "note", "references", "incidents"]}),
+    ]
+    inlines = [AlertInfoInline]
 
 class CountryAdmin(admin.ModelAdmin):
     list_display = ["name", "iso3", "region", "continent"]
@@ -22,6 +39,7 @@ class SourceAdmin(admin.ModelAdmin):
 
 
 admin.site.register(Alert, AlertAdmin)
+admin.site.register(AlertInfo, AlertInfoAdmin)
 admin.site.register(Continent)
 admin.site.register(Region)
 admin.site.register(Country, CountryAdmin)