Eurostat imports: EuroStat_Pop_Age_Sex_Edu_Attainment_Level and EuroStat_Pop_Broad_Age_Group_Sex

statvar_imports/eurostat/pop_age_sex_edu_attainment_level/README.md

@@ -0,0 +1,87 @@
+# Eurostat Population on 1 January by Age, Sex, and Educational Attainment Level Import
+
+## Overview
+This dataset contains annual population data on January 1st, broken down by age groups, sex, and highest level of education completed, sourced from Eurostat.
+
+**type of place:** Country, NUTS Regions
+**years:** 2007 to 2025
+**place_resolution:** Resolved to DCIDs (e.g., dcid:country/FRA, dcid:nuts/AT11)
+
+## Data Source
+**Source URL:**
+https://ec.europa.eu/eurostat/databrowser/view/DEMO_PJANEDU/default/table?lang=en
+
+**Provenance Description:**
+The data is provided by Eurostat, the statistical office of the European Union. It is part of the "Demography and migration" database, specifically the "Population on 1 January by age, sex and educational attainment level" (DEMO_PJANEDU) dataset.
+
+## Refresh Type
+Automatic Refresh
+
+The refresh is automated using the provided `run.sh` script, which handles both data download and processing.
+
+## How To Run Import
+To execute the complete import process (download and processing), run:
+```bash
+./run.sh
+```
+
+### Script Details:
+- **Download**: Uses `curl` to fetch the latest SDMX-CSV data from Eurostat's dissemination API.
+- **Processing**: Uses `stat_var_processor.py` to map raw data to Data Commons StatVarObservations using the PV map and metadata configuration.
+
+## Processing Instructions
+To process the Population on 1 January by Age, Sex, and Educational Attainment Level Import and generate statistical variables, use the following commands from your current import data directory:
+
+Download input file
+
+```bash
+mkdir -p input_files
+curl -L --retry 3 "https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/data/DEMO_PJANEDU/?format=SDMX-CSV&compressed=false" -o ./input_files/pop_age_sex_edu_attainment_level_input.csv
+```
+
+For Test Data Run
+
+```bash
+python3 ../../../tools/statvar_importer/stat_var_processor.py \
+  "--input_data=./test_data/pop_age_sex_edu_attainment_level_input.csv" \
+  "--pv_map=./pop_age_sex_edu_attainment_level_pvmap.csv" \
+  "--output_path=./test_data/pop_age_sex_edu_attainment_level_output" \
+  "--config_file=./pop_age_sex_edu_attainment_level_metadata.csv" \
+  "--existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf"
+```
+
+For Main data processing run
+
+```bash
+python3 ../../../tools/statvar_importer/stat_var_processor.py \
+  "--input_data=./input_files/*.csv" \
+  "--pv_map=./pop_age_sex_edu_attainment_level_pvmap.csv" \
+  "--config_file=./pop_age_sex_edu_attainment_level_metadata.csv" \
+  "--generate_statvar_name=True" \
+  "--skip_constant_csv_columns=False" \
+  "--output_columns=observationDate,observationAbout,variableMeasured,value,observationPeriod,unit" \
+  "--output_path=./pop_age_sex_edu_attainment_level_output" \
+  "--places_resolved_csv=./places_resolved.csv" \
+  "--existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf"
+```
+
+## Key Files
+- `run.sh`: Main execution script for download and processing.
+- `pop_age_sex_edu_attainment_level_pvmap.csv`: Property-Value mapping for StatVar definitions and dimensions.
+- `pop_age_sex_edu_attainment_level_metadata.csv`: Configuration parameters for the processor.
+- `places_resolved.csv`: Mapping of place codes to Data Commons DCIDs.
+- `pop_age_sex_edu_attainment_level_output.csv`: Processed statistical observations.
+- `pop_age_sex_edu_attainment_level_output.tmcf`: Template MCF mapping the CSV columns to Data Commons schema.
+
+## Validation
+To validate the generated data, use the Data Commons import tool (lint mode):
+```bash
+java -jar datacommons-import-tool.jar lint pop_age_sex_edu_attainment_level_output.csv pop_age_sex_edu_attainment_level_output.tmcf
+```
+The resulting reports (`report.json`, `summary_report.html`) in `dc_generated/` provide detailed insights into data quality and validation status.
+
+## Testing
+Testing is performed using the `test_data` directory:
+- Raw Input: `test_data/pop_age_sex_edu_attainment_level_input.csv`
+- Expected Output: `test_data/pop_age_sex_edu_attainment_level_output.csv`
+- Expected TMCF: `test_data/pop_age_sex_edu_attainment_level_output.tmcf`

statvar_imports/eurostat/pop_age_sex_edu_attainment_level/manifest.json

@@ -0,0 +1,34 @@
+{
+  "import_specifications": [
+    {
+      "import_name": "EuroStat_Pop_Age_Sex_Edu_Attainment_Level",
+      "curator_emails": [
+        "support@datacommons.org"
+      ],
+      "provenance_url": "https://ec.europa.eu/eurostat/databrowser/view/DEMO_PJANEDU/default/table?lang=en",
+      "provenance_description": "Annual tracking of the total population on January 1st, by age groups, sex, and highest level of education completed",
+      "scripts": [
+        "run.sh"
+      ],
+      "import_inputs": [
+        {
+          "template_mcf": "pop_age_sex_edu_attainment_level_output.tmcf",
+          "cleaned_csv": "pop_age_sex_edu_attainment_level_output.csv"
+        }
+      ],
+      "source_files": [
+        "input_files/*.csv"
+      ],
+      "cron_schedule": "5 1 1,15 * *",
+      "resource_limits": {"cpu": 4, "memory": 8, "disk":100},
+      "config_override": {
+         "invoke_import_validation": true,
+         "invoke_import_tool": true,
+         "invoke_differ_tool": true,
+         "skip_input_upload": false,
+         "skip_gcs_upload": false,
+         "cleanup_gcs_volume_mount": false
+        }
+      }
+    ]
+}

statvar_imports/eurostat/pop_age_sex_edu_attainment_level/places_resolved.csv

@@ -0,0 +1,265 @@
+"place_name","dcid","placeId","wikidataId","name","alternateName","typeOf","containedInPlace"
+"AD","dcid:country/AND","","","","","",""
+"AE","dcid:country/ARE","","","","","",""
+"AF","dcid:country/AFG","","","","","",""
+"AG","dcid:country/ATG","","","","","",""
+"AI","dcid:country/AIA","","","","","",""
+"AL","dcid:country/ALB","","","","","",""
+"AM","dcid:country/ARM","","","","","",""
+"AN","dcid:country/ANT","","","","","",""
+"AO","dcid:country/AGO","","","","","",""
+"AQ","dcid:country/ATA","","","","","",""
+"AR","dcid:country/ARG","","","","","",""
+"AS","dcid:country/ASM","","","","","",""
+"AT","dcid:country/AUT","","","","","",""
+"AU","dcid:country/AUS","","","","","",""
+"AW","dcid:country/ABW","","","","","",""
+"AZ","dcid:country/AZE","","","","","",""
+"BA","dcid:country/BIH","","","","","",""
+"BB","dcid:country/BRB","","","","","",""
+"BD","dcid:country/BGD","","","","","",""
+"BE","dcid:country/BEL","","","","","",""
+"BF","dcid:country/BFA","","","","","",""
+"BG","dcid:country/BGR","","","","","",""
+"BH","dcid:country/BHR","","","","","",""
+"BI","dcid:country/BDI","","","","","",""
+"BJ","dcid:country/BEN","","","","","",""
+"BL","dcid:country/BLM","","","","","",""
+"BM","dcid:country/BMU","","","","","",""
+"BN","dcid:country/BRN","","","","","",""
+"BO","dcid:country/BOL","","","","","",""
+"BR","dcid:country/BRA","","","","","",""
+"BS","dcid:country/BHS","","","","","",""
+"BT","dcid:country/BTN","","","","","",""
+"BV","dcid:country/BVT","","","","","",""
+"BW","dcid:country/BWA","","","","","",""
+"BY","dcid:country/BLR","","","","","",""
+"BZ","dcid:country/BLZ","","","","","",""
+"CA","dcid:country/CAN","","","","","",""
+"CC","dcid:country/CCK","","","","","",""
+"CD","dcid:country/COD","","","","","",""
+"CF","dcid:country/CAF","","","","","",""
+"CG","dcid:country/COG","","","","","",""
+"CH","dcid:country/CHE","","","","","",""
+"CI","dcid:country/CIV","","","","","",""
+"CK","dcid:country/COK","","","","","",""
+"CL","dcid:country/CHL","","","","","",""
+"CM","dcid:country/CMR","","","","","",""
+"CN","dcid:country/CHN","","","","","",""
+"CO","dcid:country/COL","","","","","",""
+"CR","dcid:country/CRI","","","","","",""
+"CU","dcid:country/CUB","","","","","",""
+"CV","dcid:country/CPV","","","","","",""
+"CX","dcid:country/CXR","","","","","",""
+"CY","dcid:country/CYP","","","","","",""
+"CZ","dcid:country/CZE","","","","","",""
+"DE","dcid:country/DEU","","","","","",""
+"DJ","dcid:country/DJI","","","","","",""
+"DK","dcid:country/DNK","","","","","",""
+"DM","dcid:country/DMA","","","","","",""
+"DO","dcid:country/DOM","","","","","",""
+"DZ","dcid:country/DZA","","","","","",""
+"EC","dcid:country/ECU","","","","","",""
+"EE","dcid:country/EST","","","","","",""
+"EG","dcid:country/EGY","","","","","",""
+"EH","dcid:country/ESH","","","","","",""
+"EL","dcid:country/GRC","","","","","",""
+"ER","dcid:country/ERI","","","","","",""
+"ES","dcid:country/ESP","","","","","",""
+"ET","dcid:country/ETH","","","","","",""
+"FI","dcid:country/FIN","","","","","",""
+"FJ","dcid:country/FJI","","","","","",""
+"FK","dcid:country/FLK","","","","","",""
+"FM","dcid:country/FSM","","","","","",""
+"FO","dcid:country/FRO","","","","","",""
+"FR","dcid:country/FRA","","","","","",""
+"FX","dcid:country/FXX","","","","","",""
+"GA","dcid:country/GAB","","","","","",""
+"UK","dcid:country/GBR","","","","","",""
+"GD","dcid:country/GRD","","","","","",""
+"GE","dcid:country/GEO","","","","","",""
+"GF","dcid:country/GUF","","","","","",""
+"GG","dcid:country/GGY","","","","","",""
+"GH","dcid:country/GHA","","","","","",""
+"GI","dcid:country/GIB","","","","","",""
+"GL","dcid:country/GRL","","","","","",""
+"GM","dcid:country/GMB","","","","","",""
+"GN","dcid:country/GIN","","","","","",""
+"GQ","dcid:country/GNQ","","","","","",""
+"GS","dcid:country/SGS","","","","","",""
+"GT","dcid:country/GTM","","","","","",""
+"GU","dcid:country/GUM","","","","","",""
+"GW","dcid:country/GNB","","","","","",""
+"GY","dcid:country/GUY","","","","","",""
+"HK","dcid:country/HKG","","","","","",""
+"HM","dcid:country/HMD","","","","","",""
+"HN","dcid:country/HND","","","","","",""
+"HR","dcid:country/HRV","","","","","",""
+"HT","dcid:country/HTI","","","","","",""
+"HU","dcid:country/HUN","","","","","",""
+"ID","dcid:country/IDN","","","","","",""
+"IE","dcid:country/IRL","","","","","",""
+"IL","dcid:country/ISR","","","","","",""
+"IM","dcid:country/IMN","","","","","",""
+"IN","dcid:country/IND","","","","","",""
+"IO","dcid:country/IOT","","","","","",""
+"IQ","dcid:country/IRQ","","","","","",""
+"IR","dcid:country/IRN","","","","","",""
+"IS","dcid:country/ISL","","","","","",""
+"IT","dcid:country/ITA","","","","","",""
+"JE","dcid:country/JEY","","","","","",""
+"JM","dcid:country/JAM","","","","","",""
+"JO","dcid:country/JOR","","","","","",""
+"JP","dcid:country/JPN","","","","","",""
+"KE","dcid:country/KEN","","","","","",""
+"KG","dcid:country/KGZ","","","","","",""
+"KH","dcid:country/KHM","","","","","",""
+"KI","dcid:country/KIR","","","","","",""
+"KM","dcid:country/COM","","","","","",""
+"KN","dcid:country/KNA","","","","","",""
+"KP","dcid:country/PRK","","","","","",""
+"KR","dcid:country/KOR","","","","","",""
+"KW","dcid:country/KWT","","","","","",""
+"KY","dcid:country/CYM","","","","","",""
+"KZ","dcid:country/KAZ","","","","","",""
+"LA","dcid:country/LAO","","","","","",""
+"LB","dcid:country/LBN","","","","","",""
+"LC","dcid:country/LCA","","","","","",""
+"LI","dcid:country/LIE","","","","","",""
+"LK","dcid:country/LKA","","","","","",""
+"LR","dcid:country/LBR","","","","","",""
+"LS","dcid:country/LSO","","","","","",""
+"LT","dcid:country/LTU","","","","","",""
+"LU","dcid:country/LUX","","","","","",""
+"LV","dcid:country/LVA","","","","","",""
+"LY","dcid:country/LBY","","","","","",""
+"MA","dcid:country/MAR","","","","","",""
+"MC","dcid:country/MCO","","","","","",""
+"MD","dcid:country/MDA","","","","","",""
+"ME","dcid:country/MNE","","","","","",""
+"MF","dcid:country/MAF","","","","","",""
+"MG","dcid:country/MDG","","","","","",""
+"MH","dcid:country/MHL","","","","","",""
+"MK","dcid:country/MKD","","","","","",""
+"ML","dcid:country/MLI","","","","","",""
+"MM","dcid:country/MMR","","","","","",""
+"MN","dcid:country/MNG","","","","","",""
+"MO","dcid:country/MAC","","","","","",""
+"MP","dcid:country/MNP","","","","","",""
+"MQ","dcid:country/MTQ","","","","","",""
+"MR","dcid:country/MRT","","","","","",""
+"MS","dcid:country/MSR","","","","","",""
+"MT","dcid:country/MLT","","","","","",""
+"MU","dcid:country/MUS","","","","","",""
+"MV","dcid:country/MDV","","","","","",""
+"MW","dcid:country/MWI","","","","","",""
+"MX","dcid:country/MEX","","","","","",""
+"MY","dcid:country/MYS","","","","","",""
+"MZ","dcid:country/MOZ","","","","","",""
+"NA","dcid:country/NAM","","","","","",""
+"NC","dcid:country/NCL","","","","","",""
+"NE","dcid:country/NER","","","","","",""
+"NF","dcid:country/NFK","","","","","",""
+"NG","dcid:country/NGA","","","","","",""
+"NI","dcid:country/NIC","","","","","",""
+"NL","dcid:country/NLD","","","","","",""
+"NO","dcid:country/NOR","","","","","",""
+"NP","dcid:country/NPL","","","","","",""
+"NR","dcid:country/NRU","","","","","",""
+"NU","dcid:country/NIU","","","","","",""
+"NZ","dcid:country/NZL","","","","","",""
+"OM","dcid:country/OMN","","","","","",""
+"PA","dcid:country/PAN","","","","","",""
+"PE","dcid:country/PER","","","","","",""
+"PF","dcid:country/PYF","","","","","",""
+"PG","dcid:country/PNG","","","","","",""
+"PH","dcid:country/PHL","","","","","",""
+"PK","dcid:country/PAK","","","","","",""
+"PL","dcid:country/POL","","","","","",""
+"PM","dcid:country/SPM","","","","","",""
+"PN","dcid:country/PCN","","","","","",""
+"PR","dcid:country/PRI","","","","","",""
+"PS","dcid:country/PSE","","","","","",""
+"PT","dcid:country/PRT","","","","","",""
+"PW","dcid:country/PLW","","","","","",""
+"PY","dcid:country/PRY","","","","","",""
+"QA","dcid:country/QAT","","","","","",""
+"RE","dcid:country/REU","","","","","",""
+"RO","dcid:country/ROU","","","","","",""
+"RS","dcid:country/SRB","","","","","",""
+"RU","dcid:country/RUS","","","","","",""
+"RW","dcid:country/RWA","","","","","",""
+"SA","dcid:country/SAU","","","","","",""
+"SB","dcid:country/SLB","","","","","",""
+"SC","dcid:country/SYC","","","","","",""
+"SD","dcid:country/SDN","","","","","",""
+"SE","dcid:country/SWE","","","","","",""
+"SG","dcid:country/SGP","","","","","",""
+"SH","dcid:country/SHN","","","","","",""
+"SI","dcid:country/SVN","","","","","",""
+"SJ","dcid:country/SJM","","","","","",""
+"SK","dcid:country/SVK","","","","","",""
+"SL","dcid:country/SLE","","","","","",""
+"SM","dcid:country/SMR","","","","","",""
+"SN","dcid:country/SEN","","","","","",""
+"SO","dcid:country/SOM","","","","","",""
+"SR","dcid:country/SUR","","","","","",""
+"ST","dcid:country/STP","","","","","",""
+"SV","dcid:country/SLV","","","","","",""
+"SX","dcid:country/SXM","","","","","",""
+"SY","dcid:country/SYR","","","","","",""
+"SZ","dcid:country/SWZ","","","","","",""
+"TC","dcid:country/TCA","","","","","",""
+"TD","dcid:country/TCD","","","","","",""
+"TF","dcid:country/ATF","","","","","",""
+"TG","dcid:country/TGO","","","","","",""
+"TH","dcid:country/THA","","","","","",""
+"TJ","dcid:country/TJK","","","","","",""
+"TK","dcid:country/TKL","","","","","",""
+"TL","dcid:country/TLS","","","","","",""
+"TM","dcid:country/TKM","","","","","",""
+"TN","dcid:country/TUN","","","","","",""
+"TO","dcid:country/TON","","","","","",""
+"TR","dcid:country/TUR","","","","","",""
+"TT","dcid:country/TTO","","","","","",""
+"TV","dcid:country/TUV","","","","","",""
+"TW","dcid:country/TWN","","","","","",""
+"TZ","dcid:country/TZA","","","","","",""
+"UA","dcid:country/UKR","","","","","",""
+"UG","dcid:country/UGA","","","","","",""
+"UM","dcid:country/UMI","","","","","",""
+"US","dcid:country/USA","","","","","",""
+"UY","dcid:country/URY","","","","","",""
+"UZ","dcid:country/UZB","","","","","",""
+"VA","dcid:country/VAT","","","","","",""
+"VC","dcid:country/VCT","","","","","",""
+"VE","dcid:country/VEN","","","","","",""
+"VG","dcid:country/VGB","","","","","",""
+"VI","dcid:country/VIR","","","","","",""
+"VN","dcid:country/VNM","","","","","",""
+"VU","dcid:country/VUT","","","","","",""
+"WF","dcid:country/WLF","","","","","",""
+"WS","dcid:country/WSM","","","","","",""
+"XK","dcid:country/XKS","","","","","",""
+"YE","dcid:country/YEM","","","","","",""
+"YT","dcid:country/MYT","","","","","",""
+"ZA","dcid:country/ZAF","","","","","",""
+"ZM","dcid:country/ZMB","","","","","",""
+"ZW","dcid:country/ZWE","","","","","",""
+"DE_TOT","dcid:nuts/DE_TOT","","","","","",""
+"EA","dcid:nuts/EA","","","","","",""
+"EA12","dcid:nuts/EA12","","","","","",""
+"EA18","dcid:nuts/EA18","","","","","",""
+"EA19","dcid:nuts/EA19","","","","","",""
+"EEA","dcid:nuts/EEA","","","","","",""
+"EEA30_2007","dcid:nuts/EEA30_2007","","","","","",""
+"EEA31","dcid:nuts/EEA31","","","","","",""
+"EFTA","dcid:nuts/EFTA","","","","","",""
+"EU","dcid:nuts/EU","","","","","",""
+"EU27","dcid:nuts/EU27","","","","","",""
+"EU27_2007","dcid:nuts/EU27_2007","","","","","",""
+"EU27_2020","dcid:nuts/EU27_2020","","","","","",""
+"EA20","dcid:nuts/EA20","","","","","",""
+"EA21","dcid:nuts/EA21","","","","","",""
+"EU28","dcid:nuts/EU28","","","","","",""
+"EEA28","dcid:nuts/EEA28","","","","","",""

statvar_imports/eurostat/pop_age_sex_edu_attainment_level/pop_age_sex_edu_attainment_level_metadata.csv

@@ -0,0 +1,2 @@
+parameter,value
+header_rows,1