diff --git a/statvar_imports/eurostat/pop_age_sex_edu_attainment_level/README.md b/statvar_imports/eurostat/pop_age_sex_edu_attainment_level/README.md new file mode 100755 index 0000000000..ce92e4cbb0 --- /dev/null +++ b/statvar_imports/eurostat/pop_age_sex_edu_attainment_level/README.md @@ -0,0 +1,87 @@ +# Eurostat Population on 1 January by Age, Sex, and Educational Attainment Level Import + +## Overview +This dataset contains annual population data on January 1st, broken down by age groups, sex, and highest level of education completed, sourced from Eurostat. + +**type of place:** Country, NUTS Regions +**years:** 2007 to 2025 +**place_resolution:** Resolved to DCIDs (e.g., dcid:country/FRA, dcid:nuts/AT11) + +## Data Source +**Source URL:** +https://ec.europa.eu/eurostat/databrowser/view/DEMO_PJANEDU/default/table?lang=en + +**Provenance Description:** +The data is provided by Eurostat, the statistical office of the European Union. It is part of the "Demography and migration" database, specifically the "Population on 1 January by age, sex and educational attainment level" (DEMO_PJANEDU) dataset. + +## Refresh Type +Automatic Refresh + +The refresh is automated using the provided `run.sh` script, which handles both data download and processing. + +## How To Run Import +To execute the complete import process (download and processing), run: +```bash +./run.sh +``` + +### Script Details: +- **Download**: Uses `curl` to fetch the latest SDMX-CSV data from Eurostat's dissemination API. +- **Processing**: Uses `stat_var_processor.py` to map raw data to Data Commons StatVarObservations using the PV map and metadata configuration. + +## Processing Instructions +To process the Population on 1 January by Age, Sex, and Educational Attainment Level Import and generate statistical variables, use the following commands from your current import data directory: + +Download input file + +```bash +mkdir -p input_files +curl -L --retry 3 "https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/data/DEMO_PJANEDU/?format=SDMX-CSV&compressed=false" -o ./input_files/pop_age_sex_edu_attainment_level_input.csv +``` + +For Test Data Run + +```bash +python3 ../../../tools/statvar_importer/stat_var_processor.py \ + "--input_data=./test_data/pop_age_sex_edu_attainment_level_input.csv" \ + "--pv_map=./pop_age_sex_edu_attainment_level_pvmap.csv" \ + "--output_path=./test_data/pop_age_sex_edu_attainment_level_output" \ + "--config_file=./pop_age_sex_edu_attainment_level_metadata.csv" \ + "--existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf" +``` + +For Main data processing run + +```bash +python3 ../../../tools/statvar_importer/stat_var_processor.py \ + "--input_data=./input_files/*.csv" \ + "--pv_map=./pop_age_sex_edu_attainment_level_pvmap.csv" \ + "--config_file=./pop_age_sex_edu_attainment_level_metadata.csv" \ + "--generate_statvar_name=True" \ + "--skip_constant_csv_columns=False" \ + "--output_columns=observationDate,observationAbout,variableMeasured,value,observationPeriod,unit" \ + "--output_path=./pop_age_sex_edu_attainment_level_output" \ + "--places_resolved_csv=./places_resolved.csv" \ + "--existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf" +``` + +## Key Files +- `run.sh`: Main execution script for download and processing. +- `pop_age_sex_edu_attainment_level_pvmap.csv`: Property-Value mapping for StatVar definitions and dimensions. +- `pop_age_sex_edu_attainment_level_metadata.csv`: Configuration parameters for the processor. +- `places_resolved.csv`: Mapping of place codes to Data Commons DCIDs. +- `pop_age_sex_edu_attainment_level_output.csv`: Processed statistical observations. +- `pop_age_sex_edu_attainment_level_output.tmcf`: Template MCF mapping the CSV columns to Data Commons schema. + +## Validation +To validate the generated data, use the Data Commons import tool (lint mode): +```bash +java -jar datacommons-import-tool.jar lint pop_age_sex_edu_attainment_level_output.csv pop_age_sex_edu_attainment_level_output.tmcf +``` +The resulting reports (`report.json`, `summary_report.html`) in `dc_generated/` provide detailed insights into data quality and validation status. + +## Testing +Testing is performed using the `test_data` directory: +- Raw Input: `test_data/pop_age_sex_edu_attainment_level_input.csv` +- Expected Output: `test_data/pop_age_sex_edu_attainment_level_output.csv` +- Expected TMCF: `test_data/pop_age_sex_edu_attainment_level_output.tmcf` diff --git a/statvar_imports/eurostat/pop_age_sex_edu_attainment_level/manifest.json b/statvar_imports/eurostat/pop_age_sex_edu_attainment_level/manifest.json new file mode 100755 index 0000000000..725d3f76a6 --- /dev/null +++ b/statvar_imports/eurostat/pop_age_sex_edu_attainment_level/manifest.json @@ -0,0 +1,34 @@ +{ + "import_specifications": [ + { + "import_name": "EuroStat_Pop_Age_Sex_Edu_Attainment_Level", + "curator_emails": [ + "support@datacommons.org" + ], + "provenance_url": "https://ec.europa.eu/eurostat/databrowser/view/DEMO_PJANEDU/default/table?lang=en", + "provenance_description": "Annual tracking of the total population on January 1st, by age groups, sex, and highest level of education completed", + "scripts": [ + "run.sh" + ], + "import_inputs": [ + { + "template_mcf": "pop_age_sex_edu_attainment_level_output.tmcf", + "cleaned_csv": "pop_age_sex_edu_attainment_level_output.csv" + } + ], + "source_files": [ + "input_files/*.csv" + ], + "cron_schedule": "5 1 1,15 * *", + "resource_limits": {"cpu": 4, "memory": 8, "disk":100}, + "config_override": { + "invoke_import_validation": true, + "invoke_import_tool": true, + "invoke_differ_tool": true, + "skip_input_upload": false, + "skip_gcs_upload": false, + "cleanup_gcs_volume_mount": false + } + } + ] +} diff --git a/statvar_imports/eurostat/pop_age_sex_edu_attainment_level/places_resolved.csv b/statvar_imports/eurostat/pop_age_sex_edu_attainment_level/places_resolved.csv new file mode 100644 index 0000000000..acc65e505e --- /dev/null +++ b/statvar_imports/eurostat/pop_age_sex_edu_attainment_level/places_resolved.csv @@ -0,0 +1,265 @@ +"place_name","dcid","placeId","wikidataId","name","alternateName","typeOf","containedInPlace" +"AD","dcid:country/AND","","","","","","" +"AE","dcid:country/ARE","","","","","","" +"AF","dcid:country/AFG","","","","","","" +"AG","dcid:country/ATG","","","","","","" +"AI","dcid:country/AIA","","","","","","" +"AL","dcid:country/ALB","","","","","","" +"AM","dcid:country/ARM","","","","","","" +"AN","dcid:country/ANT","","","","","","" +"AO","dcid:country/AGO","","","","","","" +"AQ","dcid:country/ATA","","","","","","" +"AR","dcid:country/ARG","","","","","","" +"AS","dcid:country/ASM","","","","","","" +"AT","dcid:country/AUT","","","","","","" +"AU","dcid:country/AUS","","","","","","" +"AW","dcid:country/ABW","","","","","","" +"AZ","dcid:country/AZE","","","","","","" +"BA","dcid:country/BIH","","","","","","" +"BB","dcid:country/BRB","","","","","","" +"BD","dcid:country/BGD","","","","","","" +"BE","dcid:country/BEL","","","","","","" +"BF","dcid:country/BFA","","","","","","" +"BG","dcid:country/BGR","","","","","","" +"BH","dcid:country/BHR","","","","","","" +"BI","dcid:country/BDI","","","","","","" +"BJ","dcid:country/BEN","","","","","","" +"BL","dcid:country/BLM","","","","","","" +"BM","dcid:country/BMU","","","","","","" +"BN","dcid:country/BRN","","","","","","" +"BO","dcid:country/BOL","","","","","","" +"BR","dcid:country/BRA","","","","","","" +"BS","dcid:country/BHS","","","","","","" +"BT","dcid:country/BTN","","","","","","" +"BV","dcid:country/BVT","","","","","","" +"BW","dcid:country/BWA","","","","","","" +"BY","dcid:country/BLR","","","","","","" +"BZ","dcid:country/BLZ","","","","","","" +"CA","dcid:country/CAN","","","","","","" +"CC","dcid:country/CCK","","","","","","" +"CD","dcid:country/COD","","","","","","" +"CF","dcid:country/CAF","","","","","","" +"CG","dcid:country/COG","","","","","","" +"CH","dcid:country/CHE","","","","","","" +"CI","dcid:country/CIV","","","","","","" +"CK","dcid:country/COK","","","","","","" +"CL","dcid:country/CHL","","","","","","" +"CM","dcid:country/CMR","","","","","","" +"CN","dcid:country/CHN","","","","","","" +"CO","dcid:country/COL","","","","","","" +"CR","dcid:country/CRI","","","","","","" +"CU","dcid:country/CUB","","","","","","" +"CV","dcid:country/CPV","","","","","","" +"CX","dcid:country/CXR","","","","","","" +"CY","dcid:country/CYP","","","","","","" +"CZ","dcid:country/CZE","","","","","","" +"DE","dcid:country/DEU","","","","","","" +"DJ","dcid:country/DJI","","","","","","" +"DK","dcid:country/DNK","","","","","","" +"DM","dcid:country/DMA","","","","","","" +"DO","dcid:country/DOM","","","","","","" +"DZ","dcid:country/DZA","","","","","","" +"EC","dcid:country/ECU","","","","","","" +"EE","dcid:country/EST","","","","","","" +"EG","dcid:country/EGY","","","","","","" +"EH","dcid:country/ESH","","","","","","" +"EL","dcid:country/GRC","","","","","","" +"ER","dcid:country/ERI","","","","","","" +"ES","dcid:country/ESP","","","","","","" +"ET","dcid:country/ETH","","","","","","" +"FI","dcid:country/FIN","","","","","","" +"FJ","dcid:country/FJI","","","","","","" +"FK","dcid:country/FLK","","","","","","" +"FM","dcid:country/FSM","","","","","","" +"FO","dcid:country/FRO","","","","","","" +"FR","dcid:country/FRA","","","","","","" +"FX","dcid:country/FXX","","","","","","" +"GA","dcid:country/GAB","","","","","","" +"UK","dcid:country/GBR","","","","","","" +"GD","dcid:country/GRD","","","","","","" +"GE","dcid:country/GEO","","","","","","" +"GF","dcid:country/GUF","","","","","","" +"GG","dcid:country/GGY","","","","","","" +"GH","dcid:country/GHA","","","","","","" +"GI","dcid:country/GIB","","","","","","" +"GL","dcid:country/GRL","","","","","","" +"GM","dcid:country/GMB","","","","","","" +"GN","dcid:country/GIN","","","","","","" +"GQ","dcid:country/GNQ","","","","","","" +"GS","dcid:country/SGS","","","","","","" +"GT","dcid:country/GTM","","","","","","" +"GU","dcid:country/GUM","","","","","","" +"GW","dcid:country/GNB","","","","","","" +"GY","dcid:country/GUY","","","","","","" +"HK","dcid:country/HKG","","","","","","" +"HM","dcid:country/HMD","","","","","","" +"HN","dcid:country/HND","","","","","","" +"HR","dcid:country/HRV","","","","","","" +"HT","dcid:country/HTI","","","","","","" +"HU","dcid:country/HUN","","","","","","" +"ID","dcid:country/IDN","","","","","","" +"IE","dcid:country/IRL","","","","","","" +"IL","dcid:country/ISR","","","","","","" +"IM","dcid:country/IMN","","","","","","" +"IN","dcid:country/IND","","","","","","" +"IO","dcid:country/IOT","","","","","","" +"IQ","dcid:country/IRQ","","","","","","" +"IR","dcid:country/IRN","","","","","","" +"IS","dcid:country/ISL","","","","","","" +"IT","dcid:country/ITA","","","","","","" +"JE","dcid:country/JEY","","","","","","" +"JM","dcid:country/JAM","","","","","","" +"JO","dcid:country/JOR","","","","","","" +"JP","dcid:country/JPN","","","","","","" +"KE","dcid:country/KEN","","","","","","" +"KG","dcid:country/KGZ","","","","","","" +"KH","dcid:country/KHM","","","","","","" +"KI","dcid:country/KIR","","","","","","" +"KM","dcid:country/COM","","","","","","" +"KN","dcid:country/KNA","","","","","","" +"KP","dcid:country/PRK","","","","","","" +"KR","dcid:country/KOR","","","","","","" +"KW","dcid:country/KWT","","","","","","" +"KY","dcid:country/CYM","","","","","","" +"KZ","dcid:country/KAZ","","","","","","" +"LA","dcid:country/LAO","","","","","","" +"LB","dcid:country/LBN","","","","","","" +"LC","dcid:country/LCA","","","","","","" +"LI","dcid:country/LIE","","","","","","" +"LK","dcid:country/LKA","","","","","","" +"LR","dcid:country/LBR","","","","","","" +"LS","dcid:country/LSO","","","","","","" +"LT","dcid:country/LTU","","","","","","" +"LU","dcid:country/LUX","","","","","","" +"LV","dcid:country/LVA","","","","","","" +"LY","dcid:country/LBY","","","","","","" +"MA","dcid:country/MAR","","","","","","" +"MC","dcid:country/MCO","","","","","","" +"MD","dcid:country/MDA","","","","","","" +"ME","dcid:country/MNE","","","","","","" +"MF","dcid:country/MAF","","","","","","" +"MG","dcid:country/MDG","","","","","","" +"MH","dcid:country/MHL","","","","","","" +"MK","dcid:country/MKD","","","","","","" +"ML","dcid:country/MLI","","","","","","" +"MM","dcid:country/MMR","","","","","","" +"MN","dcid:country/MNG","","","","","","" +"MO","dcid:country/MAC","","","","","","" +"MP","dcid:country/MNP","","","","","","" +"MQ","dcid:country/MTQ","","","","","","" +"MR","dcid:country/MRT","","","","","","" +"MS","dcid:country/MSR","","","","","","" +"MT","dcid:country/MLT","","","","","","" +"MU","dcid:country/MUS","","","","","","" +"MV","dcid:country/MDV","","","","","","" +"MW","dcid:country/MWI","","","","","","" +"MX","dcid:country/MEX","","","","","","" +"MY","dcid:country/MYS","","","","","","" +"MZ","dcid:country/MOZ","","","","","","" +"NA","dcid:country/NAM","","","","","","" +"NC","dcid:country/NCL","","","","","","" +"NE","dcid:country/NER","","","","","","" +"NF","dcid:country/NFK","","","","","","" +"NG","dcid:country/NGA","","","","","","" +"NI","dcid:country/NIC","","","","","","" +"NL","dcid:country/NLD","","","","","","" +"NO","dcid:country/NOR","","","","","","" +"NP","dcid:country/NPL","","","","","","" +"NR","dcid:country/NRU","","","","","","" +"NU","dcid:country/NIU","","","","","","" +"NZ","dcid:country/NZL","","","","","","" +"OM","dcid:country/OMN","","","","","","" +"PA","dcid:country/PAN","","","","","","" +"PE","dcid:country/PER","","","","","","" +"PF","dcid:country/PYF","","","","","","" +"PG","dcid:country/PNG","","","","","","" +"PH","dcid:country/PHL","","","","","","" +"PK","dcid:country/PAK","","","","","","" +"PL","dcid:country/POL","","","","","","" +"PM","dcid:country/SPM","","","","","","" +"PN","dcid:country/PCN","","","","","","" +"PR","dcid:country/PRI","","","","","","" +"PS","dcid:country/PSE","","","","","","" +"PT","dcid:country/PRT","","","","","","" +"PW","dcid:country/PLW","","","","","","" +"PY","dcid:country/PRY","","","","","","" +"QA","dcid:country/QAT","","","","","","" +"RE","dcid:country/REU","","","","","","" +"RO","dcid:country/ROU","","","","","","" +"RS","dcid:country/SRB","","","","","","" +"RU","dcid:country/RUS","","","","","","" +"RW","dcid:country/RWA","","","","","","" +"SA","dcid:country/SAU","","","","","","" +"SB","dcid:country/SLB","","","","","","" +"SC","dcid:country/SYC","","","","","","" +"SD","dcid:country/SDN","","","","","","" +"SE","dcid:country/SWE","","","","","","" +"SG","dcid:country/SGP","","","","","","" +"SH","dcid:country/SHN","","","","","","" +"SI","dcid:country/SVN","","","","","","" +"SJ","dcid:country/SJM","","","","","","" +"SK","dcid:country/SVK","","","","","","" +"SL","dcid:country/SLE","","","","","","" +"SM","dcid:country/SMR","","","","","","" +"SN","dcid:country/SEN","","","","","","" +"SO","dcid:country/SOM","","","","","","" +"SR","dcid:country/SUR","","","","","","" +"ST","dcid:country/STP","","","","","","" +"SV","dcid:country/SLV","","","","","","" +"SX","dcid:country/SXM","","","","","","" +"SY","dcid:country/SYR","","","","","","" +"SZ","dcid:country/SWZ","","","","","","" +"TC","dcid:country/TCA","","","","","","" +"TD","dcid:country/TCD","","","","","","" +"TF","dcid:country/ATF","","","","","","" +"TG","dcid:country/TGO","","","","","","" +"TH","dcid:country/THA","","","","","","" +"TJ","dcid:country/TJK","","","","","","" +"TK","dcid:country/TKL","","","","","","" +"TL","dcid:country/TLS","","","","","","" +"TM","dcid:country/TKM","","","","","","" +"TN","dcid:country/TUN","","","","","","" +"TO","dcid:country/TON","","","","","","" +"TR","dcid:country/TUR","","","","","","" +"TT","dcid:country/TTO","","","","","","" +"TV","dcid:country/TUV","","","","","","" +"TW","dcid:country/TWN","","","","","","" +"TZ","dcid:country/TZA","","","","","","" +"UA","dcid:country/UKR","","","","","","" +"UG","dcid:country/UGA","","","","","","" +"UM","dcid:country/UMI","","","","","","" +"US","dcid:country/USA","","","","","","" +"UY","dcid:country/URY","","","","","","" +"UZ","dcid:country/UZB","","","","","","" +"VA","dcid:country/VAT","","","","","","" +"VC","dcid:country/VCT","","","","","","" +"VE","dcid:country/VEN","","","","","","" +"VG","dcid:country/VGB","","","","","","" +"VI","dcid:country/VIR","","","","","","" +"VN","dcid:country/VNM","","","","","","" +"VU","dcid:country/VUT","","","","","","" +"WF","dcid:country/WLF","","","","","","" +"WS","dcid:country/WSM","","","","","","" +"XK","dcid:country/XKS","","","","","","" +"YE","dcid:country/YEM","","","","","","" +"YT","dcid:country/MYT","","","","","","" +"ZA","dcid:country/ZAF","","","","","","" +"ZM","dcid:country/ZMB","","","","","","" +"ZW","dcid:country/ZWE","","","","","","" +"DE_TOT","dcid:nuts/DE_TOT","","","","","","" +"EA","dcid:nuts/EA","","","","","","" +"EA12","dcid:nuts/EA12","","","","","","" +"EA18","dcid:nuts/EA18","","","","","","" +"EA19","dcid:nuts/EA19","","","","","","" +"EEA","dcid:nuts/EEA","","","","","","" +"EEA30_2007","dcid:nuts/EEA30_2007","","","","","","" +"EEA31","dcid:nuts/EEA31","","","","","","" +"EFTA","dcid:nuts/EFTA","","","","","","" +"EU","dcid:nuts/EU","","","","","","" +"EU27","dcid:nuts/EU27","","","","","","" +"EU27_2007","dcid:nuts/EU27_2007","","","","","","" +"EU27_2020","dcid:nuts/EU27_2020","","","","","","" +"EA20","dcid:nuts/EA20","","","","","","" +"EA21","dcid:nuts/EA21","","","","","","" +"EU28","dcid:nuts/EU28","","","","","","" +"EEA28","dcid:nuts/EEA28","","","","","","" \ No newline at end of file diff --git a/statvar_imports/eurostat/pop_age_sex_edu_attainment_level/pop_age_sex_edu_attainment_level_metadata.csv b/statvar_imports/eurostat/pop_age_sex_edu_attainment_level/pop_age_sex_edu_attainment_level_metadata.csv new file mode 100644 index 0000000000..9499660fe2 --- /dev/null +++ b/statvar_imports/eurostat/pop_age_sex_edu_attainment_level/pop_age_sex_edu_attainment_level_metadata.csv @@ -0,0 +1,2 @@ +parameter,value +header_rows,1 diff --git a/statvar_imports/eurostat/pop_age_sex_edu_attainment_level/pop_age_sex_edu_attainment_level_pvmap.csv b/statvar_imports/eurostat/pop_age_sex_edu_attainment_level/pop_age_sex_edu_attainment_level_pvmap.csv new file mode 100644 index 0000000000..4580635da4 --- /dev/null +++ b/statvar_imports/eurostat/pop_age_sex_edu_attainment_level/pop_age_sex_edu_attainment_level_pvmap.csv @@ -0,0 +1,142 @@ +key,property1,value1,property2,value2,property3,value3 + +# --- AUTOMATED CLEANUP: Ignored Columns --- +DATAFLOW,#ignore,"",,,, +LAST UPDATE,#ignore,"",,,, +CONF_STATUS,#ignore,"",,,, + +# Global properties from DATAFLOW +ESTAT:DEMO_PJANEDU(1.0),populationType,dcs:Person,measuredProperty,dcs:count,statType,dcs:measuredValue +# Frequency +freq:A,observationPeriod,P1Y +# Observation Date and Value +TIME_PERIOD,observationDate,{Data} +OBS_VALUE,value,{Number} +# Sex +sex:F,gender,dcs:Female +sex:M,gender,dcs:Male +sex:T,gender,"" + +# --- Age Group Normalization (Explicit dcs: Mapping) --- +age:TOTAL,age,"",,, +age:UNK,age,dcs:USC_AgeNotStated,,, +age:Y_LT1,age,dcs:Years0,,, +age:Y_OPEN,age,dcs:Years100Onwards,,, +age:Y1,age,dcs:Years1,,, +age:Y2,age,dcs:Years2,,, +age:Y3,age,dcs:Years3,,, +age:Y4,age,dcs:Years4,,, +age:Y5,age,dcs:Years5,,, +age:Y6,age,dcs:Years6,,, +age:Y7,age,dcs:Years7,,, +age:Y8,age,dcs:Years8,,, +age:Y9,age,dcs:Years9,,, +age:Y10,age,dcs:Years10,,, +age:Y11,age,dcs:Years11,,, +age:Y12,age,dcs:Years12,,, +age:Y13,age,dcs:Years13,,, +age:Y14,age,dcs:Years14,,, +age:Y15,age,dcs:Years15,,, +age:Y16,age,dcs:Years16,,, +age:Y17,age,dcs:Years17,,, +age:Y18,age,dcs:Years18,,, +age:Y19,age,dcs:Years19,,, +age:Y20,age,dcs:Years20,,, +age:Y21,age,dcs:Years21,,, +age:Y22,age,dcs:Years22,,, +age:Y23,age,dcs:Years23,,, +age:Y24,age,dcs:Years24,,, +age:Y25,age,dcs:Years25,,, +age:Y26,age,dcs:Years26,,, +age:Y27,age,dcs:Years27,,, +age:Y28,age,dcs:Years28,,, +age:Y29,age,dcs:Years29,,, +age:Y30,age,dcs:Years30,,, +age:Y31,age,dcs:Years31,,, +age:Y32,age,dcs:Years32,,, +age:Y33,age,dcs:Years33,,, +age:Y34,age,dcs:Years34,,, +age:Y35,age,dcs:Years35,,, +age:Y36,age,dcs:Years36,,, +age:Y37,age,dcs:Years37,,, +age:Y38,age,dcs:Years38,,, +age:Y39,age,dcs:Years39,,, +age:Y40,age,dcs:Years40,,, +age:Y41,age,dcs:Years41,,, +age:Y42,age,dcs:Years42,,, +age:Y43,age,dcs:Years43,,, +age:Y44,age,dcs:Years44,,, +age:Y45,age,dcs:Years45,,, +age:Y46,age,dcs:Years46,,, +age:Y47,age,dcs:Years47,,, +age:Y48,age,dcs:Years48,,, +age:Y49,age,dcs:Years49,,, +age:Y50,age,dcs:Years50,,, +age:Y51,age,dcs:Years51,,, +age:Y52,age,dcs:Years52,,, +age:Y53,age,dcs:Years53,,, +age:Y54,age,dcs:Years54,,, +age:Y55,age,dcs:Years55,,, +age:Y56,age,dcs:Years56,,, +age:Y57,age,dcs:Years57,,, +age:Y58,age,dcs:Years58,,, +age:Y59,age,dcs:Years59,,, +age:Y60,age,dcs:Years60,,, +age:Y61,age,dcs:Years61,,, +age:Y62,age,dcs:Years62,,, +age:Y63,age,dcs:Years63,,, +age:Y64,age,dcs:Years64,,, +age:Y65,age,dcs:Years65,,, +age:Y66,age,dcs:Years66,,, +age:Y67,age,dcs:Years67,,, +age:Y68,age,dcs:Years68,,, +age:Y69,age,dcs:Years69,,, +age:Y70,age,dcs:Years70,,, +age:Y71,age,dcs:Years71,,, +age:Y72,age,dcs:Years72,,, +age:Y73,age,dcs:Years73,,, +age:Y74,age,dcs:Years74,,, +age:Y75,age,dcs:Years75,,, +age:Y76,age,dcs:Years76,,, +age:Y77,age,dcs:Years77,,, +age:Y78,age,dcs:Years78,,, +age:Y79,age,dcs:Years79,,, +age:Y80,age,dcs:Years80,,, +age:Y81,age,dcs:Years81,,, +age:Y82,age,dcs:Years82,,, +age:Y83,age,dcs:Years83,,, +age:Y84,age,dcs:Years84,,, +age:Y85,age,dcs:Years85,,, +age:Y86,age,dcs:Years86,,, +age:Y87,age,dcs:Years87,,, +age:Y88,age,dcs:Years88,,, +age:Y89,age,dcs:Years89,,, +age:Y90,age,dcs:Years90,,, +age:Y91,age,dcs:Years91,,, +age:Y92,age,dcs:Years92,,, +age:Y93,age,dcs:Years93,,, +age:Y94,age,dcs:Years94,,, +age:Y95,age,dcs:Years95,,, +age:Y96,age,dcs:Years96,,, +age:Y97,age,dcs:Years97,,, +age:Y98,age,dcs:Years98,,, +age:Y99,age,dcs:Years99,,, + +# ISCED Educational Attainment +isced11:TOTAL,educationalAttainment,"" +isced11:ED0-2,educationalAttainment,dcs:LessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation +isced11:ED3_4,educationalAttainment,dcs:UpperSecondaryEducationOrPostSecondaryNonTertiaryEducation +isced11:ED5-8,educationalAttainment,dcs:TertiaryEducation +isced11:NAP,#ignore,Not Applicable +isced11:UNK,educationalAttainment,dcs:UnknownEducationalAttainment +# Geographic Areas (Countries) +# Fallback for other geo codes +geo,observationAbout,{Data} +# Unit +unit:NR,unit,dcs:Person +# Flags +OBS_FLAG:p,#ignore,"",,,,,, +OBS_FLAG:e,#ignore,"",,,,,, +OBS_FLAG:b,#ignore,"",,,,,, +OBS_FLAG:ep,#ignore,"",,,,,, +OBS_FLAG:bep,#ignore,"",,,,,, diff --git a/statvar_imports/eurostat/pop_age_sex_edu_attainment_level/run.sh b/statvar_imports/eurostat/pop_age_sex_edu_attainment_level/run.sh new file mode 100755 index 0000000000..7f5adc1c47 --- /dev/null +++ b/statvar_imports/eurostat/pop_age_sex_edu_attainment_level/run.sh @@ -0,0 +1,17 @@ +#!/bin/bash + +# Step 1: Data Download +mkdir -p input_files +curl -L --retry 3 "https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/data/DEMO_PJANEDU/?format=SDMX-CSV&compressed=false" -o ./input_files/pop_age_sex_edu_attainment_level_input.csv + +# Step 2: Full Data Processing +python3 ../../../tools/statvar_importer/stat_var_processor.py \ + "--input_data=./input_files/*.csv" \ + "--pv_map=./pop_age_sex_edu_attainment_level_pvmap.csv" \ + "--config_file=./pop_age_sex_edu_attainment_level_metadata.csv" \ + "--generate_statvar_name=True" \ + "--skip_constant_csv_columns=False" \ + "--output_columns=observationDate,observationAbout,variableMeasured,value,observationPeriod,unit" \ + "--output_path=./pop_age_sex_edu_attainment_level_output" \ + "--places_resolved_csv=./places_resolved.csv" \ + "--existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf" diff --git a/statvar_imports/eurostat/pop_age_sex_edu_attainment_level/test_data/pop_age_sex_edu_attainment_level_input.csv b/statvar_imports/eurostat/pop_age_sex_edu_attainment_level/test_data/pop_age_sex_edu_attainment_level_input.csv new file mode 100644 index 0000000000..0fdfa33ef0 --- /dev/null +++ b/statvar_imports/eurostat/pop_age_sex_edu_attainment_level/test_data/pop_age_sex_edu_attainment_level_input.csv @@ -0,0 +1,100 @@ +DATAFLOW,LAST UPDATE,freq,unit,sex,age,isced11,geo,TIME_PERIOD,OBS_VALUE,OBS_FLAG,CONF_STATUS +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,AZ,2017,960700,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,DK,2007,825500,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,DK,2008,812505,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,DK,2009,848660,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,DK,2010,838701,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,DK,2011,830792,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,DK,2012,818731,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,DK,2013,805064,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,DK,2014,790435,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,DK,2015,775082,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,DK,2016,759269,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,DK,2017,744634,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,DK,2018,743654,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,DK,2019,729912,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,DK,2020,720404,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,DK,2022,696936,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,EE,2014,124763,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,EE,2015,120721,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,EE,2016,116094,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,EE,2017,112589,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,EE,2018,109573,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,EE,2019,106860,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,EE,2020,104452,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,EE,2023,100571,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,EE,2024,100157,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,EE,2025,99766,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,FI,2007,1245919,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,FI,2008,1230033,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,FI,2009,1214997,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,FI,2010,1199551,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,FI,2011,1183690,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,FI,2012,1168382,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,FI,2013,1152672,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,FI,2014,709152,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,FI,2015,692778,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,FI,2016,681145,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,FI,2017,651120,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,FI,2018,631941,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,FI,2019,603326,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,FI,2020,591904,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,FI,2021,581339,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,FI,2022,572559,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,FI,2023,473590,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,FI,2024,458684,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,FI,2025,445083,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,NO,2007,588801,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,NO,2008,574247,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,NO,2009,562618,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,NO,2010,567428,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,NO,2011,577214,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,NO,2012,573964,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,NO,2013,571533,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,NO,2014,566344,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,NO,2015,560317,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,NO,2016,553300,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,NO,2017,545903,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,NO,2018,538285,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,NO,2019,529778,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,NO,2020,519085,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,NO,2021,511755,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,NO,2022,502143,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,NO,2023,493740,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SE,2007,720542,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SE,2008,710593,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SE,2009,984365,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SE,2010,963336,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SE,2011,941142,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SE,2012,913755,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SE,2013,884152,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SE,2014,1117553,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SE,2015,1098966,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SE,2016,1078871,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SE,2017,1065696,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SE,2018,1051083,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SE,2019,1039204,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SE,2020,1026598,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SE,2023,971717,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SE,2024,952517,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SE,2025,931891,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SI,2011,299674,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SI,2012,291657,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SI,2014,277966,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SI,2015,271182,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SI,2016,264173,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SI,2017,257060,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SI,2018,250177,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SI,2019,242915,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SI,2020,238531,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SI,2022,228680,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SI,2023,226072,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SI,2024,223725,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SI,2025,221692,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SK,2011,935546,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SK,2012,922000,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SK,2013,911327,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SK,2014,893135,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SK,2015,878647,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SK,2016,860204,, +ESTAT:DEMO_PJANEDU(1.0),08/05/26 11:00:00,A,NR,F,TOTAL,ED0-2,SK,2017,851604,, diff --git a/statvar_imports/eurostat/pop_age_sex_edu_attainment_level/test_data/pop_age_sex_edu_attainment_level_output.csv b/statvar_imports/eurostat/pop_age_sex_edu_attainment_level/test_data/pop_age_sex_edu_attainment_level_output.csv new file mode 100644 index 0000000000..c138aae386 --- /dev/null +++ b/statvar_imports/eurostat/pop_age_sex_edu_attainment_level/test_data/pop_age_sex_edu_attainment_level_output.csv @@ -0,0 +1,100 @@ +observationDate,observationAbout,variableMeasured,value,observationPeriod,unit +2017,dcid:country/AZE,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,960700,P1Y,dcs:Person +2007,dcid:country/DNK,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,825500,P1Y,dcs:Person +2008,dcid:country/DNK,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,812505,P1Y,dcs:Person +2009,dcid:country/DNK,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,848660,P1Y,dcs:Person +2010,dcid:country/DNK,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,838701,P1Y,dcs:Person +2011,dcid:country/DNK,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,830792,P1Y,dcs:Person +2012,dcid:country/DNK,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,818731,P1Y,dcs:Person +2013,dcid:country/DNK,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,805064,P1Y,dcs:Person +2014,dcid:country/DNK,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,790435,P1Y,dcs:Person +2015,dcid:country/DNK,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,775082,P1Y,dcs:Person +2016,dcid:country/DNK,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,759269,P1Y,dcs:Person +2017,dcid:country/DNK,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,744634,P1Y,dcs:Person +2018,dcid:country/DNK,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,743654,P1Y,dcs:Person +2019,dcid:country/DNK,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,729912,P1Y,dcs:Person +2020,dcid:country/DNK,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,720404,P1Y,dcs:Person +2022,dcid:country/DNK,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,696936,P1Y,dcs:Person +2014,dcid:country/EST,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,124763,P1Y,dcs:Person +2015,dcid:country/EST,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,120721,P1Y,dcs:Person +2016,dcid:country/EST,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,116094,P1Y,dcs:Person +2017,dcid:country/EST,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,112589,P1Y,dcs:Person +2018,dcid:country/EST,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,109573,P1Y,dcs:Person +2019,dcid:country/EST,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,106860,P1Y,dcs:Person +2020,dcid:country/EST,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,104452,P1Y,dcs:Person +2023,dcid:country/EST,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,100571,P1Y,dcs:Person +2024,dcid:country/EST,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,100157,P1Y,dcs:Person +2025,dcid:country/EST,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,99766,P1Y,dcs:Person +2007,dcid:country/FIN,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,1245919,P1Y,dcs:Person +2008,dcid:country/FIN,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,1230033,P1Y,dcs:Person +2009,dcid:country/FIN,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,1214997,P1Y,dcs:Person +2010,dcid:country/FIN,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,1199551,P1Y,dcs:Person +2011,dcid:country/FIN,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,1183690,P1Y,dcs:Person +2012,dcid:country/FIN,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,1168382,P1Y,dcs:Person +2013,dcid:country/FIN,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,1152672,P1Y,dcs:Person +2014,dcid:country/FIN,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,709152,P1Y,dcs:Person +2015,dcid:country/FIN,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,692778,P1Y,dcs:Person +2016,dcid:country/FIN,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,681145,P1Y,dcs:Person +2017,dcid:country/FIN,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,651120,P1Y,dcs:Person +2018,dcid:country/FIN,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,631941,P1Y,dcs:Person +2019,dcid:country/FIN,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,603326,P1Y,dcs:Person +2020,dcid:country/FIN,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,591904,P1Y,dcs:Person +2021,dcid:country/FIN,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,581339,P1Y,dcs:Person +2022,dcid:country/FIN,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,572559,P1Y,dcs:Person +2023,dcid:country/FIN,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,473590,P1Y,dcs:Person +2024,dcid:country/FIN,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,458684,P1Y,dcs:Person +2025,dcid:country/FIN,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,445083,P1Y,dcs:Person +2007,dcid:country/NOR,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,588801,P1Y,dcs:Person +2008,dcid:country/NOR,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,574247,P1Y,dcs:Person +2009,dcid:country/NOR,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,562618,P1Y,dcs:Person +2010,dcid:country/NOR,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,567428,P1Y,dcs:Person +2011,dcid:country/NOR,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,577214,P1Y,dcs:Person +2012,dcid:country/NOR,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,573964,P1Y,dcs:Person +2013,dcid:country/NOR,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,571533,P1Y,dcs:Person +2014,dcid:country/NOR,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,566344,P1Y,dcs:Person +2015,dcid:country/NOR,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,560317,P1Y,dcs:Person +2016,dcid:country/NOR,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,553300,P1Y,dcs:Person +2017,dcid:country/NOR,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,545903,P1Y,dcs:Person +2018,dcid:country/NOR,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,538285,P1Y,dcs:Person +2019,dcid:country/NOR,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,529778,P1Y,dcs:Person +2020,dcid:country/NOR,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,519085,P1Y,dcs:Person +2021,dcid:country/NOR,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,511755,P1Y,dcs:Person +2022,dcid:country/NOR,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,502143,P1Y,dcs:Person +2023,dcid:country/NOR,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,493740,P1Y,dcs:Person +2007,dcid:country/SWE,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,720542,P1Y,dcs:Person +2008,dcid:country/SWE,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,710593,P1Y,dcs:Person +2009,dcid:country/SWE,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,984365,P1Y,dcs:Person +2010,dcid:country/SWE,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,963336,P1Y,dcs:Person +2011,dcid:country/SWE,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,941142,P1Y,dcs:Person +2012,dcid:country/SWE,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,913755,P1Y,dcs:Person +2013,dcid:country/SWE,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,884152,P1Y,dcs:Person +2014,dcid:country/SWE,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,1117553,P1Y,dcs:Person +2015,dcid:country/SWE,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,1098966,P1Y,dcs:Person +2016,dcid:country/SWE,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,1078871,P1Y,dcs:Person +2017,dcid:country/SWE,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,1065696,P1Y,dcs:Person +2018,dcid:country/SWE,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,1051083,P1Y,dcs:Person +2019,dcid:country/SWE,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,1039204,P1Y,dcs:Person +2020,dcid:country/SWE,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,1026598,P1Y,dcs:Person +2023,dcid:country/SWE,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,971717,P1Y,dcs:Person +2024,dcid:country/SWE,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,952517,P1Y,dcs:Person +2025,dcid:country/SWE,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,931891,P1Y,dcs:Person +2011,dcid:country/SVN,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,299674,P1Y,dcs:Person +2012,dcid:country/SVN,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,291657,P1Y,dcs:Person +2014,dcid:country/SVN,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,277966,P1Y,dcs:Person +2015,dcid:country/SVN,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,271182,P1Y,dcs:Person +2016,dcid:country/SVN,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,264173,P1Y,dcs:Person +2017,dcid:country/SVN,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,257060,P1Y,dcs:Person +2018,dcid:country/SVN,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,250177,P1Y,dcs:Person +2019,dcid:country/SVN,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,242915,P1Y,dcs:Person +2020,dcid:country/SVN,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,238531,P1Y,dcs:Person +2022,dcid:country/SVN,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,228680,P1Y,dcs:Person +2023,dcid:country/SVN,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,226072,P1Y,dcs:Person +2024,dcid:country/SVN,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,223725,P1Y,dcs:Person +2025,dcid:country/SVN,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,221692,P1Y,dcs:Person +2011,dcid:country/SVK,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,935546,P1Y,dcs:Person +2012,dcid:country/SVK,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,922000,P1Y,dcs:Person +2013,dcid:country/SVK,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,911327,P1Y,dcs:Person +2014,dcid:country/SVK,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,893135,P1Y,dcs:Person +2015,dcid:country/SVK,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,878647,P1Y,dcs:Person +2016,dcid:country/SVK,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,860204,P1Y,dcs:Person +2017,dcid:country/SVK,dcid:Count_Person_EducationalAttainmentLessThanPrimaryEducationOrPrimaryEducationOrLowerSecondaryEducation_Female,851604,P1Y,dcs:Person diff --git a/statvar_imports/eurostat/pop_age_sex_edu_attainment_level/test_data/pop_age_sex_edu_attainment_level_output.tmcf b/statvar_imports/eurostat/pop_age_sex_edu_attainment_level/test_data/pop_age_sex_edu_attainment_level_output.tmcf new file mode 100644 index 0000000000..39a28d1ba4 --- /dev/null +++ b/statvar_imports/eurostat/pop_age_sex_edu_attainment_level/test_data/pop_age_sex_edu_attainment_level_output.tmcf @@ -0,0 +1,8 @@ +Node: E:output->E0 +observationDate: C:output->observationDate +observationAbout: C:output->observationAbout +variableMeasured: C:output->variableMeasured +value: C:output->value +observationPeriod: C:output->observationPeriod +unit: C:output->unit +typeOf: dcs:StatVarObservation diff --git a/statvar_imports/eurostat/pop_broad_age_group_sex/README.md b/statvar_imports/eurostat/pop_broad_age_group_sex/README.md new file mode 100755 index 0000000000..5d4a7bb567 --- /dev/null +++ b/statvar_imports/eurostat/pop_broad_age_group_sex/README.md @@ -0,0 +1,87 @@ +# Eurostat Population on 1 January by Broad Age Group and Sex Import + +## Overview +This dataset contains annual tracking of the total population on January 1st, broken down simultaneously by broad age groups, sex, and geographic region, sourced from Eurostat. + +**type of place:** Country, NUTS Regions +**years:** 1960 to 2025 +**place_resolution:** Resolved to DCIDs (e.g., dcid:country/FRA, dcid:nuts/AT11) + +## Data Source +**Source URL:** +https://ec.europa.eu/eurostat/databrowser/view/DEMO_PJANBROAD/default/table?lang=en + +**Provenance Description:** +The data is provided by Eurostat, the statistical office of the European Union. It is part of the "Demography and migration" database, specifically the "Population on 1 January by broad age groups and sex" (DEMO_PJANBROAD) dataset. + +## Refresh Type +Automatic Refresh + +The refresh is automated using the provided `run.sh` script, which handles both data download and processing. + +## How To Run Import +To execute the complete import process (download and processing), run: +```bash +./run.sh +``` + +### Script Details: +- **Download**: Uses `curl` to fetch the latest SDMX-CSV data from Eurostat's dissemination API. +- **Processing**: Uses `stat_var_processor.py` to map raw data to Data Commons StatVarObservations using the PV map and metadata configuration. + +## Processing Instructions +To process the Population on 1 January by Broad Age Group and Sex Import and generate statistical variables, use the following commands from your current import data directory: + +Download input file + +```bash +mkdir -p input_files +curl -L --retry 3 "https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/data/DEMO_PJANBROAD/?format=SDMX-CSV&compressed=false" -o input_files/pop_broad_age_group_sex_input.csv +``` + +For Test Data Run + +```bash +python3 ../../../tools/statvar_importer/stat_var_processor.py \ + "--input_data=./test_data/pop_broad_age_group_sex_input.csv" \ + "--pv_map=./pop_broad_age_group_sex_pvmap.csv" \ + "--output_path=./test_data/pop_broad_age_group_sex_output" \ + "--config_file=./pop_broad_age_group_sex_metadata.csv" \ + "--existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf" +``` + +For Main data processing run + +```bash +python3 ../../../tools/statvar_importer/stat_var_processor.py \ + "--input_data=./input_files/*.csv" \ + "--pv_map=./pop_broad_age_group_sex_pvmap.csv" \ + "--config_file=./pop_broad_age_group_sex_metadata.csv" \ + "--generate_statvar_name=True" \ + "--skip_constant_csv_columns=False" \ + "--output_columns=observationDate,observationAbout,variableMeasured,value,observationPeriod,unit" \ + "--output_path=./pop_broad_age_group_sex_output" \ + "--places_resolved_csv=./places_resolved.csv" \ + "--existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf" +``` + +## Key Files +- `run.sh`: Main execution script for download and processing. +- `pop_broad_age_group_sex_pvmap.csv`: Property-Value mapping for StatVar definitions and dimensions. +- `pop_broad_age_group_sex_metadata.csv`: Configuration parameters for the processor. +- `places_resolved.csv`: Mapping of place codes to Data Commons DCIDs. +- `pop_broad_age_group_sex_output.csv`: Processed statistical observations. +- `pop_broad_age_group_sex_output.tmcf`: Template MCF mapping the CSV columns to Data Commons schema. + +## Validation +To validate the generated data, use the Data Commons import tool (lint mode): +```bash +java -jar datacommons-import-tool.jar lint pop_broad_age_group_sex_output.csv pop_broad_age_group_sex_output.tmcf +``` +The resulting reports (`report.json`, `summary_report.html`) in `dc_generated/` provide detailed insights into data quality and validation status. + +## Testing +Testing is performed using the `test_data` directory: +- Raw Input: `test_data/pop_broad_age_group_sex_input.csv` +- Expected Output: `test_data/pop_broad_age_group_sex_output.csv` +- Expected TMCF: `test_data/pop_broad_age_group_sex_output.tmcf` diff --git a/statvar_imports/eurostat/pop_broad_age_group_sex/final_output/pop_broad_age_group_sex_input.csv b/statvar_imports/eurostat/pop_broad_age_group_sex/final_output/pop_broad_age_group_sex_input.csv new file mode 100644 index 0000000000..bc235311e0 --- /dev/null +++ b/statvar_imports/eurostat/pop_broad_age_group_sex/final_output/pop_broad_age_group_sex_input.csv @@ -0,0 +1,100 @@ +DATAFLOW,LAST UPDATE,freq,unit,age,sex,geo,TIME_PERIOD,OBS_VALUE,OBS_FLAG,CONF_STATUS +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AD,1986,20898,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AD,1987,22148,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AD,1988,22814,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AD,1989,23813,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AD,1990,24171,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AD,1991,25542,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AD,1992,27572,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AD,1993,28864,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AD,1994,30406,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AD,1995,30228,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AD,1996,30200,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AD,1997,30627,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AD,1998,31025,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AD,1999,31384,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AD,2000,31538,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AD,2001,31576,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AD,2002,31915,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AD,2003,32354,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AD,2004,34738,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AD,2005,36790.0,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AD,2006,37575.0,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AD,2007,38705.0,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AD,2008,39622.0,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AD,2009,40303.0,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AD,2010,40296.0,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AD,2011,38252,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AD,2012,38252,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AD,2013,37408,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AD,2019,37388,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AL,2000,1526762,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AL,2001,1535822,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AL,2002,1532563,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AL,2003,1526180,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AL,2004,1520481,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AL,2005,1512745.0,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AL,2006,1503969.0,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AL,2007,1492439.0,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AL,2008,1480358.0,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AL,2009,1468935.0,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AL,2010,1459025.0,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AL,2011,1451691,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AL,2012,1444234,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AL,2013,1437193,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AL,2014,1430827,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AL,2015,1424597,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AL,2016,1417141,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AL,2017,1423050,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AL,2018,1431715,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AL,2019,1432833,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AL,2020,1425342,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AL,2021,1419759,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AL,2022,1406532,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AL,2023,1394864,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AL,2025,1194597,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AM,2000,1681200,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AM,2001,1672200,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AM,2002,1670500,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AM,2003,1666700,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AM,2004,1665500,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AM,2005,1665200.0,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AM,2006,1665123.0,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AM,2007,1665554.0,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AM,2008,1667529.0,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AM,2009,1669711.0,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AM,2010,1673656.0,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AM,2011,1679066,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AM,2012,1684000,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AM,2015,1571450,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AM,2016,1569535,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AM,2017,1567380,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AM,2018,1564533,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AM,2019,1563538,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AM,2020,1562689,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AM,2021,1565144,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AM,2023,1571757,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AM,2024,1577675,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AT,1960,3757167,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AT,1961,3773097,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AT,1962,3794130,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AT,1963,3814191,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AT,1964,3836415,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AT,1965,3857760,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AT,1966,3876559,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AT,1967,3899799,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AT,1968,3922359,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AT,1969,3932691,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AT,1970,3944298,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AT,1971,3953827,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AT,1972,3972070,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AT,1973,3990665,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AT,1974,4006377,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AT,1975,4003303,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AT,1976,3992886,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AT,1977,3992437,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AT,1978,3994255,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AT,1979,3985363,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AT,1980,3981359,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AT,1981,3983154,, +ESTAT:DEMO_PJANBROAD(1.0),08/05/26 11:00:00,A,NR,TOTAL,F,AT,1982,3993808,, diff --git a/statvar_imports/eurostat/pop_broad_age_group_sex/final_output/pop_broad_age_group_sex_output.csv b/statvar_imports/eurostat/pop_broad_age_group_sex/final_output/pop_broad_age_group_sex_output.csv new file mode 100644 index 0000000000..a6ea96c608 --- /dev/null +++ b/statvar_imports/eurostat/pop_broad_age_group_sex/final_output/pop_broad_age_group_sex_output.csv @@ -0,0 +1,100 @@ +observationDate,observationAbout,variableMeasured,value,observationPeriod,unit +1986,dcid:country/AND,dcid:Count_Person_Female,20898,P1Y,dcs:Person +1987,dcid:country/AND,dcid:Count_Person_Female,22148,P1Y,dcs:Person +1988,dcid:country/AND,dcid:Count_Person_Female,22814,P1Y,dcs:Person +1989,dcid:country/AND,dcid:Count_Person_Female,23813,P1Y,dcs:Person +1990,dcid:country/AND,dcid:Count_Person_Female,24171,P1Y,dcs:Person +1991,dcid:country/AND,dcid:Count_Person_Female,25542,P1Y,dcs:Person +1992,dcid:country/AND,dcid:Count_Person_Female,27572,P1Y,dcs:Person +1993,dcid:country/AND,dcid:Count_Person_Female,28864,P1Y,dcs:Person +1994,dcid:country/AND,dcid:Count_Person_Female,30406,P1Y,dcs:Person +1995,dcid:country/AND,dcid:Count_Person_Female,30228,P1Y,dcs:Person +1996,dcid:country/AND,dcid:Count_Person_Female,30200,P1Y,dcs:Person +1997,dcid:country/AND,dcid:Count_Person_Female,30627,P1Y,dcs:Person +1998,dcid:country/AND,dcid:Count_Person_Female,31025,P1Y,dcs:Person +1999,dcid:country/AND,dcid:Count_Person_Female,31384,P1Y,dcs:Person +2000,dcid:country/AND,dcid:Count_Person_Female,31538,P1Y,dcs:Person +2001,dcid:country/AND,dcid:Count_Person_Female,31576,P1Y,dcs:Person +2002,dcid:country/AND,dcid:Count_Person_Female,31915,P1Y,dcs:Person +2003,dcid:country/AND,dcid:Count_Person_Female,32354,P1Y,dcs:Person +2004,dcid:country/AND,dcid:Count_Person_Female,34738,P1Y,dcs:Person +2005,dcid:country/AND,dcid:Count_Person_Female,36790,P1Y,dcs:Person +2006,dcid:country/AND,dcid:Count_Person_Female,37575,P1Y,dcs:Person +2007,dcid:country/AND,dcid:Count_Person_Female,38705,P1Y,dcs:Person +2008,dcid:country/AND,dcid:Count_Person_Female,39622,P1Y,dcs:Person +2009,dcid:country/AND,dcid:Count_Person_Female,40303,P1Y,dcs:Person +2010,dcid:country/AND,dcid:Count_Person_Female,40296,P1Y,dcs:Person +2011,dcid:country/AND,dcid:Count_Person_Female,38252,P1Y,dcs:Person +2012,dcid:country/AND,dcid:Count_Person_Female,38252,P1Y,dcs:Person +2013,dcid:country/AND,dcid:Count_Person_Female,37408,P1Y,dcs:Person +2019,dcid:country/AND,dcid:Count_Person_Female,37388,P1Y,dcs:Person +2000,dcid:country/ALB,dcid:Count_Person_Female,1526762,P1Y,dcs:Person +2001,dcid:country/ALB,dcid:Count_Person_Female,1535822,P1Y,dcs:Person +2002,dcid:country/ALB,dcid:Count_Person_Female,1532563,P1Y,dcs:Person +2003,dcid:country/ALB,dcid:Count_Person_Female,1526180,P1Y,dcs:Person +2004,dcid:country/ALB,dcid:Count_Person_Female,1520481,P1Y,dcs:Person +2005,dcid:country/ALB,dcid:Count_Person_Female,1512745,P1Y,dcs:Person +2006,dcid:country/ALB,dcid:Count_Person_Female,1503969,P1Y,dcs:Person +2007,dcid:country/ALB,dcid:Count_Person_Female,1492439,P1Y,dcs:Person +2008,dcid:country/ALB,dcid:Count_Person_Female,1480358,P1Y,dcs:Person +2009,dcid:country/ALB,dcid:Count_Person_Female,1468935,P1Y,dcs:Person +2010,dcid:country/ALB,dcid:Count_Person_Female,1459025,P1Y,dcs:Person +2011,dcid:country/ALB,dcid:Count_Person_Female,1451691,P1Y,dcs:Person +2012,dcid:country/ALB,dcid:Count_Person_Female,1444234,P1Y,dcs:Person +2013,dcid:country/ALB,dcid:Count_Person_Female,1437193,P1Y,dcs:Person +2014,dcid:country/ALB,dcid:Count_Person_Female,1430827,P1Y,dcs:Person +2015,dcid:country/ALB,dcid:Count_Person_Female,1424597,P1Y,dcs:Person +2016,dcid:country/ALB,dcid:Count_Person_Female,1417141,P1Y,dcs:Person +2017,dcid:country/ALB,dcid:Count_Person_Female,1423050,P1Y,dcs:Person +2018,dcid:country/ALB,dcid:Count_Person_Female,1431715,P1Y,dcs:Person +2019,dcid:country/ALB,dcid:Count_Person_Female,1432833,P1Y,dcs:Person +2020,dcid:country/ALB,dcid:Count_Person_Female,1425342,P1Y,dcs:Person +2021,dcid:country/ALB,dcid:Count_Person_Female,1419759,P1Y,dcs:Person +2022,dcid:country/ALB,dcid:Count_Person_Female,1406532,P1Y,dcs:Person +2023,dcid:country/ALB,dcid:Count_Person_Female,1394864,P1Y,dcs:Person +2025,dcid:country/ALB,dcid:Count_Person_Female,1194597,P1Y,dcs:Person +2000,dcid:country/ARM,dcid:Count_Person_Female,1681200,P1Y,dcs:Person +2001,dcid:country/ARM,dcid:Count_Person_Female,1672200,P1Y,dcs:Person +2002,dcid:country/ARM,dcid:Count_Person_Female,1670500,P1Y,dcs:Person +2003,dcid:country/ARM,dcid:Count_Person_Female,1666700,P1Y,dcs:Person +2004,dcid:country/ARM,dcid:Count_Person_Female,1665500,P1Y,dcs:Person +2005,dcid:country/ARM,dcid:Count_Person_Female,1665200,P1Y,dcs:Person +2006,dcid:country/ARM,dcid:Count_Person_Female,1665123,P1Y,dcs:Person +2007,dcid:country/ARM,dcid:Count_Person_Female,1665554,P1Y,dcs:Person +2008,dcid:country/ARM,dcid:Count_Person_Female,1667529,P1Y,dcs:Person +2009,dcid:country/ARM,dcid:Count_Person_Female,1669711,P1Y,dcs:Person +2010,dcid:country/ARM,dcid:Count_Person_Female,1673656,P1Y,dcs:Person +2011,dcid:country/ARM,dcid:Count_Person_Female,1679066,P1Y,dcs:Person +2012,dcid:country/ARM,dcid:Count_Person_Female,1684000,P1Y,dcs:Person +2015,dcid:country/ARM,dcid:Count_Person_Female,1571450,P1Y,dcs:Person +2016,dcid:country/ARM,dcid:Count_Person_Female,1569535,P1Y,dcs:Person +2017,dcid:country/ARM,dcid:Count_Person_Female,1567380,P1Y,dcs:Person +2018,dcid:country/ARM,dcid:Count_Person_Female,1564533,P1Y,dcs:Person +2019,dcid:country/ARM,dcid:Count_Person_Female,1563538,P1Y,dcs:Person +2020,dcid:country/ARM,dcid:Count_Person_Female,1562689,P1Y,dcs:Person +2021,dcid:country/ARM,dcid:Count_Person_Female,1565144,P1Y,dcs:Person +2023,dcid:country/ARM,dcid:Count_Person_Female,1571757,P1Y,dcs:Person +2024,dcid:country/ARM,dcid:Count_Person_Female,1577675,P1Y,dcs:Person +1960,dcid:country/AUT,dcid:Count_Person_Female,3757167,P1Y,dcs:Person +1961,dcid:country/AUT,dcid:Count_Person_Female,3773097,P1Y,dcs:Person +1962,dcid:country/AUT,dcid:Count_Person_Female,3794130,P1Y,dcs:Person +1963,dcid:country/AUT,dcid:Count_Person_Female,3814191,P1Y,dcs:Person +1964,dcid:country/AUT,dcid:Count_Person_Female,3836415,P1Y,dcs:Person +1965,dcid:country/AUT,dcid:Count_Person_Female,3857760,P1Y,dcs:Person +1966,dcid:country/AUT,dcid:Count_Person_Female,3876559,P1Y,dcs:Person +1967,dcid:country/AUT,dcid:Count_Person_Female,3899799,P1Y,dcs:Person +1968,dcid:country/AUT,dcid:Count_Person_Female,3922359,P1Y,dcs:Person +1969,dcid:country/AUT,dcid:Count_Person_Female,3932691,P1Y,dcs:Person +1970,dcid:country/AUT,dcid:Count_Person_Female,3944298,P1Y,dcs:Person +1971,dcid:country/AUT,dcid:Count_Person_Female,3953827,P1Y,dcs:Person +1972,dcid:country/AUT,dcid:Count_Person_Female,3972070,P1Y,dcs:Person +1973,dcid:country/AUT,dcid:Count_Person_Female,3990665,P1Y,dcs:Person +1974,dcid:country/AUT,dcid:Count_Person_Female,4006377,P1Y,dcs:Person +1975,dcid:country/AUT,dcid:Count_Person_Female,4003303,P1Y,dcs:Person +1976,dcid:country/AUT,dcid:Count_Person_Female,3992886,P1Y,dcs:Person +1977,dcid:country/AUT,dcid:Count_Person_Female,3992437,P1Y,dcs:Person +1978,dcid:country/AUT,dcid:Count_Person_Female,3994255,P1Y,dcs:Person +1979,dcid:country/AUT,dcid:Count_Person_Female,3985363,P1Y,dcs:Person +1980,dcid:country/AUT,dcid:Count_Person_Female,3981359,P1Y,dcs:Person +1981,dcid:country/AUT,dcid:Count_Person_Female,3983154,P1Y,dcs:Person +1982,dcid:country/AUT,dcid:Count_Person_Female,3993808,P1Y,dcs:Person diff --git a/statvar_imports/eurostat/pop_broad_age_group_sex/final_output/pop_broad_age_group_sex_output.tmcf b/statvar_imports/eurostat/pop_broad_age_group_sex/final_output/pop_broad_age_group_sex_output.tmcf new file mode 100644 index 0000000000..39a28d1ba4 --- /dev/null +++ b/statvar_imports/eurostat/pop_broad_age_group_sex/final_output/pop_broad_age_group_sex_output.tmcf @@ -0,0 +1,8 @@ +Node: E:output->E0 +observationDate: C:output->observationDate +observationAbout: C:output->observationAbout +variableMeasured: C:output->variableMeasured +value: C:output->value +observationPeriod: C:output->observationPeriod +unit: C:output->unit +typeOf: dcs:StatVarObservation diff --git a/statvar_imports/eurostat/pop_broad_age_group_sex/manifest.json b/statvar_imports/eurostat/pop_broad_age_group_sex/manifest.json new file mode 100755 index 0000000000..6a5333524d --- /dev/null +++ b/statvar_imports/eurostat/pop_broad_age_group_sex/manifest.json @@ -0,0 +1,34 @@ +{ + "import_specifications": [ + { + "import_name": "EuroStat_Pop_Broad_Age_Group_Sex", + "curator_emails": [ + "support@datacommons.org" + ], + "provenance_url": "https://ec.europa.eu/eurostat/databrowser/view/DEMO_PJANBROAD/default/table?lang=en", + "provenance_description": "Annual tracking of the total population on January 1st, broken down simultaneously by broad age groups, sex, and geographic region", + "scripts": [ + "run.sh" + ], + "import_inputs": [ + { + "template_mcf": "pop_broad_age_group_sex_output.tmcf", + "cleaned_csv": "pop_broad_age_group_sex_output.csv" + } + ], + "source_files": [ + "input_files/*.csv" + ], + "cron_schedule": "5 1 1,15 * *", + "resource_limits": {"cpu": 4, "memory": 8, "disk":100}, + "config_override": { + "invoke_import_validation": true, + "invoke_import_tool": true, + "invoke_differ_tool": true, + "skip_input_upload": false, + "skip_gcs_upload": false, + "cleanup_gcs_volume_mount": false + } + } + ] +} diff --git a/statvar_imports/eurostat/pop_broad_age_group_sex/places_resolved.csv b/statvar_imports/eurostat/pop_broad_age_group_sex/places_resolved.csv new file mode 100644 index 0000000000..acc65e505e --- /dev/null +++ b/statvar_imports/eurostat/pop_broad_age_group_sex/places_resolved.csv @@ -0,0 +1,265 @@ +"place_name","dcid","placeId","wikidataId","name","alternateName","typeOf","containedInPlace" +"AD","dcid:country/AND","","","","","","" +"AE","dcid:country/ARE","","","","","","" +"AF","dcid:country/AFG","","","","","","" +"AG","dcid:country/ATG","","","","","","" +"AI","dcid:country/AIA","","","","","","" +"AL","dcid:country/ALB","","","","","","" +"AM","dcid:country/ARM","","","","","","" +"AN","dcid:country/ANT","","","","","","" +"AO","dcid:country/AGO","","","","","","" +"AQ","dcid:country/ATA","","","","","","" +"AR","dcid:country/ARG","","","","","","" +"AS","dcid:country/ASM","","","","","","" +"AT","dcid:country/AUT","","","","","","" +"AU","dcid:country/AUS","","","","","","" +"AW","dcid:country/ABW","","","","","","" +"AZ","dcid:country/AZE","","","","","","" +"BA","dcid:country/BIH","","","","","","" +"BB","dcid:country/BRB","","","","","","" +"BD","dcid:country/BGD","","","","","","" +"BE","dcid:country/BEL","","","","","","" +"BF","dcid:country/BFA","","","","","","" +"BG","dcid:country/BGR","","","","","","" +"BH","dcid:country/BHR","","","","","","" +"BI","dcid:country/BDI","","","","","","" +"BJ","dcid:country/BEN","","","","","","" +"BL","dcid:country/BLM","","","","","","" +"BM","dcid:country/BMU","","","","","","" +"BN","dcid:country/BRN","","","","","","" +"BO","dcid:country/BOL","","","","","","" +"BR","dcid:country/BRA","","","","","","" +"BS","dcid:country/BHS","","","","","","" +"BT","dcid:country/BTN","","","","","","" +"BV","dcid:country/BVT","","","","","","" +"BW","dcid:country/BWA","","","","","","" +"BY","dcid:country/BLR","","","","","","" +"BZ","dcid:country/BLZ","","","","","","" +"CA","dcid:country/CAN","","","","","","" +"CC","dcid:country/CCK","","","","","","" +"CD","dcid:country/COD","","","","","","" +"CF","dcid:country/CAF","","","","","","" +"CG","dcid:country/COG","","","","","","" +"CH","dcid:country/CHE","","","","","","" +"CI","dcid:country/CIV","","","","","","" +"CK","dcid:country/COK","","","","","","" +"CL","dcid:country/CHL","","","","","","" +"CM","dcid:country/CMR","","","","","","" +"CN","dcid:country/CHN","","","","","","" +"CO","dcid:country/COL","","","","","","" +"CR","dcid:country/CRI","","","","","","" +"CU","dcid:country/CUB","","","","","","" +"CV","dcid:country/CPV","","","","","","" +"CX","dcid:country/CXR","","","","","","" +"CY","dcid:country/CYP","","","","","","" +"CZ","dcid:country/CZE","","","","","","" +"DE","dcid:country/DEU","","","","","","" +"DJ","dcid:country/DJI","","","","","","" +"DK","dcid:country/DNK","","","","","","" +"DM","dcid:country/DMA","","","","","","" +"DO","dcid:country/DOM","","","","","","" +"DZ","dcid:country/DZA","","","","","","" +"EC","dcid:country/ECU","","","","","","" +"EE","dcid:country/EST","","","","","","" +"EG","dcid:country/EGY","","","","","","" +"EH","dcid:country/ESH","","","","","","" +"EL","dcid:country/GRC","","","","","","" +"ER","dcid:country/ERI","","","","","","" +"ES","dcid:country/ESP","","","","","","" +"ET","dcid:country/ETH","","","","","","" +"FI","dcid:country/FIN","","","","","","" +"FJ","dcid:country/FJI","","","","","","" +"FK","dcid:country/FLK","","","","","","" +"FM","dcid:country/FSM","","","","","","" +"FO","dcid:country/FRO","","","","","","" +"FR","dcid:country/FRA","","","","","","" +"FX","dcid:country/FXX","","","","","","" +"GA","dcid:country/GAB","","","","","","" +"UK","dcid:country/GBR","","","","","","" +"GD","dcid:country/GRD","","","","","","" +"GE","dcid:country/GEO","","","","","","" +"GF","dcid:country/GUF","","","","","","" +"GG","dcid:country/GGY","","","","","","" +"GH","dcid:country/GHA","","","","","","" +"GI","dcid:country/GIB","","","","","","" +"GL","dcid:country/GRL","","","","","","" +"GM","dcid:country/GMB","","","","","","" +"GN","dcid:country/GIN","","","","","","" +"GQ","dcid:country/GNQ","","","","","","" +"GS","dcid:country/SGS","","","","","","" +"GT","dcid:country/GTM","","","","","","" +"GU","dcid:country/GUM","","","","","","" +"GW","dcid:country/GNB","","","","","","" +"GY","dcid:country/GUY","","","","","","" +"HK","dcid:country/HKG","","","","","","" +"HM","dcid:country/HMD","","","","","","" +"HN","dcid:country/HND","","","","","","" +"HR","dcid:country/HRV","","","","","","" +"HT","dcid:country/HTI","","","","","","" +"HU","dcid:country/HUN","","","","","","" +"ID","dcid:country/IDN","","","","","","" +"IE","dcid:country/IRL","","","","","","" +"IL","dcid:country/ISR","","","","","","" +"IM","dcid:country/IMN","","","","","","" +"IN","dcid:country/IND","","","","","","" +"IO","dcid:country/IOT","","","","","","" +"IQ","dcid:country/IRQ","","","","","","" +"IR","dcid:country/IRN","","","","","","" +"IS","dcid:country/ISL","","","","","","" +"IT","dcid:country/ITA","","","","","","" +"JE","dcid:country/JEY","","","","","","" +"JM","dcid:country/JAM","","","","","","" +"JO","dcid:country/JOR","","","","","","" +"JP","dcid:country/JPN","","","","","","" +"KE","dcid:country/KEN","","","","","","" +"KG","dcid:country/KGZ","","","","","","" +"KH","dcid:country/KHM","","","","","","" +"KI","dcid:country/KIR","","","","","","" +"KM","dcid:country/COM","","","","","","" +"KN","dcid:country/KNA","","","","","","" +"KP","dcid:country/PRK","","","","","","" +"KR","dcid:country/KOR","","","","","","" +"KW","dcid:country/KWT","","","","","","" +"KY","dcid:country/CYM","","","","","","" +"KZ","dcid:country/KAZ","","","","","","" +"LA","dcid:country/LAO","","","","","","" +"LB","dcid:country/LBN","","","","","","" +"LC","dcid:country/LCA","","","","","","" +"LI","dcid:country/LIE","","","","","","" +"LK","dcid:country/LKA","","","","","","" +"LR","dcid:country/LBR","","","","","","" +"LS","dcid:country/LSO","","","","","","" +"LT","dcid:country/LTU","","","","","","" +"LU","dcid:country/LUX","","","","","","" +"LV","dcid:country/LVA","","","","","","" +"LY","dcid:country/LBY","","","","","","" +"MA","dcid:country/MAR","","","","","","" +"MC","dcid:country/MCO","","","","","","" +"MD","dcid:country/MDA","","","","","","" +"ME","dcid:country/MNE","","","","","","" +"MF","dcid:country/MAF","","","","","","" +"MG","dcid:country/MDG","","","","","","" +"MH","dcid:country/MHL","","","","","","" +"MK","dcid:country/MKD","","","","","","" +"ML","dcid:country/MLI","","","","","","" +"MM","dcid:country/MMR","","","","","","" +"MN","dcid:country/MNG","","","","","","" +"MO","dcid:country/MAC","","","","","","" +"MP","dcid:country/MNP","","","","","","" +"MQ","dcid:country/MTQ","","","","","","" +"MR","dcid:country/MRT","","","","","","" +"MS","dcid:country/MSR","","","","","","" +"MT","dcid:country/MLT","","","","","","" +"MU","dcid:country/MUS","","","","","","" +"MV","dcid:country/MDV","","","","","","" +"MW","dcid:country/MWI","","","","","","" +"MX","dcid:country/MEX","","","","","","" +"MY","dcid:country/MYS","","","","","","" +"MZ","dcid:country/MOZ","","","","","","" +"NA","dcid:country/NAM","","","","","","" +"NC","dcid:country/NCL","","","","","","" +"NE","dcid:country/NER","","","","","","" +"NF","dcid:country/NFK","","","","","","" +"NG","dcid:country/NGA","","","","","","" +"NI","dcid:country/NIC","","","","","","" +"NL","dcid:country/NLD","","","","","","" +"NO","dcid:country/NOR","","","","","","" +"NP","dcid:country/NPL","","","","","","" +"NR","dcid:country/NRU","","","","","","" +"NU","dcid:country/NIU","","","","","","" +"NZ","dcid:country/NZL","","","","","","" +"OM","dcid:country/OMN","","","","","","" +"PA","dcid:country/PAN","","","","","","" +"PE","dcid:country/PER","","","","","","" +"PF","dcid:country/PYF","","","","","","" +"PG","dcid:country/PNG","","","","","","" +"PH","dcid:country/PHL","","","","","","" +"PK","dcid:country/PAK","","","","","","" +"PL","dcid:country/POL","","","","","","" +"PM","dcid:country/SPM","","","","","","" +"PN","dcid:country/PCN","","","","","","" +"PR","dcid:country/PRI","","","","","","" +"PS","dcid:country/PSE","","","","","","" +"PT","dcid:country/PRT","","","","","","" +"PW","dcid:country/PLW","","","","","","" +"PY","dcid:country/PRY","","","","","","" +"QA","dcid:country/QAT","","","","","","" +"RE","dcid:country/REU","","","","","","" +"RO","dcid:country/ROU","","","","","","" +"RS","dcid:country/SRB","","","","","","" +"RU","dcid:country/RUS","","","","","","" +"RW","dcid:country/RWA","","","","","","" +"SA","dcid:country/SAU","","","","","","" +"SB","dcid:country/SLB","","","","","","" +"SC","dcid:country/SYC","","","","","","" +"SD","dcid:country/SDN","","","","","","" +"SE","dcid:country/SWE","","","","","","" +"SG","dcid:country/SGP","","","","","","" +"SH","dcid:country/SHN","","","","","","" +"SI","dcid:country/SVN","","","","","","" +"SJ","dcid:country/SJM","","","","","","" +"SK","dcid:country/SVK","","","","","","" +"SL","dcid:country/SLE","","","","","","" +"SM","dcid:country/SMR","","","","","","" +"SN","dcid:country/SEN","","","","","","" +"SO","dcid:country/SOM","","","","","","" +"SR","dcid:country/SUR","","","","","","" +"ST","dcid:country/STP","","","","","","" +"SV","dcid:country/SLV","","","","","","" +"SX","dcid:country/SXM","","","","","","" +"SY","dcid:country/SYR","","","","","","" +"SZ","dcid:country/SWZ","","","","","","" +"TC","dcid:country/TCA","","","","","","" +"TD","dcid:country/TCD","","","","","","" +"TF","dcid:country/ATF","","","","","","" +"TG","dcid:country/TGO","","","","","","" +"TH","dcid:country/THA","","","","","","" +"TJ","dcid:country/TJK","","","","","","" +"TK","dcid:country/TKL","","","","","","" +"TL","dcid:country/TLS","","","","","","" +"TM","dcid:country/TKM","","","","","","" +"TN","dcid:country/TUN","","","","","","" +"TO","dcid:country/TON","","","","","","" +"TR","dcid:country/TUR","","","","","","" +"TT","dcid:country/TTO","","","","","","" +"TV","dcid:country/TUV","","","","","","" +"TW","dcid:country/TWN","","","","","","" +"TZ","dcid:country/TZA","","","","","","" +"UA","dcid:country/UKR","","","","","","" +"UG","dcid:country/UGA","","","","","","" +"UM","dcid:country/UMI","","","","","","" +"US","dcid:country/USA","","","","","","" +"UY","dcid:country/URY","","","","","","" +"UZ","dcid:country/UZB","","","","","","" +"VA","dcid:country/VAT","","","","","","" +"VC","dcid:country/VCT","","","","","","" +"VE","dcid:country/VEN","","","","","","" +"VG","dcid:country/VGB","","","","","","" +"VI","dcid:country/VIR","","","","","","" +"VN","dcid:country/VNM","","","","","","" +"VU","dcid:country/VUT","","","","","","" +"WF","dcid:country/WLF","","","","","","" +"WS","dcid:country/WSM","","","","","","" +"XK","dcid:country/XKS","","","","","","" +"YE","dcid:country/YEM","","","","","","" +"YT","dcid:country/MYT","","","","","","" +"ZA","dcid:country/ZAF","","","","","","" +"ZM","dcid:country/ZMB","","","","","","" +"ZW","dcid:country/ZWE","","","","","","" +"DE_TOT","dcid:nuts/DE_TOT","","","","","","" +"EA","dcid:nuts/EA","","","","","","" +"EA12","dcid:nuts/EA12","","","","","","" +"EA18","dcid:nuts/EA18","","","","","","" +"EA19","dcid:nuts/EA19","","","","","","" +"EEA","dcid:nuts/EEA","","","","","","" +"EEA30_2007","dcid:nuts/EEA30_2007","","","","","","" +"EEA31","dcid:nuts/EEA31","","","","","","" +"EFTA","dcid:nuts/EFTA","","","","","","" +"EU","dcid:nuts/EU","","","","","","" +"EU27","dcid:nuts/EU27","","","","","","" +"EU27_2007","dcid:nuts/EU27_2007","","","","","","" +"EU27_2020","dcid:nuts/EU27_2020","","","","","","" +"EA20","dcid:nuts/EA20","","","","","","" +"EA21","dcid:nuts/EA21","","","","","","" +"EU28","dcid:nuts/EU28","","","","","","" +"EEA28","dcid:nuts/EEA28","","","","","","" \ No newline at end of file diff --git a/statvar_imports/eurostat/pop_broad_age_group_sex/pop_broad_age_group_sex_metadata.csv b/statvar_imports/eurostat/pop_broad_age_group_sex/pop_broad_age_group_sex_metadata.csv new file mode 100644 index 0000000000..5b1b0e1bbe --- /dev/null +++ b/statvar_imports/eurostat/pop_broad_age_group_sex/pop_broad_age_group_sex_metadata.csv @@ -0,0 +1,4 @@ +parameter,value +header_rows,1 +resolve_places,False +#input_rows,20 \ No newline at end of file diff --git a/statvar_imports/eurostat/pop_broad_age_group_sex/pop_broad_age_group_sex_pvmap.csv b/statvar_imports/eurostat/pop_broad_age_group_sex/pop_broad_age_group_sex_pvmap.csv new file mode 100644 index 0000000000..c41955e1ca --- /dev/null +++ b/statvar_imports/eurostat/pop_broad_age_group_sex/pop_broad_age_group_sex_pvmap.csv @@ -0,0 +1,41 @@ +key,p1,v1,p2,v2,p3,v3,p4,v4,p5,v5 + +# --- AUTOMATED CLEANUP: Ignored Columns --- +DATAFLOW,#ignore,"",,,,,,,, +LAST UPDATE,#ignore,"",,,,,,,, +CONF_STATUS,#ignore,"",,,,,,,, + +# Frequency dimension (P1Y for Annual series) +freq:A,observationPeriod,P1Y,,,,,,,, + +# Global observation properties mapped on unit +unit:NR,populationType,dcs:Person,measuredProperty,dcs:count,statType,dcs:measuredValue,unit,dcs:Person,, + +# Age mappings (Corrected dcs:Years0To14 to dcs:YearsUpto14) +age:TOTAL,age,"",,,,,,,, +age:Y15-64,age,dcs:Years15To64,,,,,,,, +age:Y_LT15,age,dcs:YearsUpto14,,,,,,,, +age:Y_GE65,age,dcs:Years65Onwards,,,,,,,, +age:UNK,age,dcs:USC_AgeNotStated,,,,,,,, + +# Sex / Gender dimension +sex:T,gender,"",,,,,,,, +sex:M,gender,dcs:Male,,,,,,,, +sex:F,gender,dcs:Female,,,,,,,, + +# --- Geopolitical Entity Translation (Bypasses offline API call failures) --- +geo,observationAbout,{Data},,,,,,,, + +# Time and Value mappings +TIME_PERIOD,observationDate,{Data},,,,,,,, +OBS_VALUE,value,{Number},,,,,,,, + +# Safely Ignoring Transient and Composite Status Flags to Prevent Ingestion Warnings +OBS_FLAG,#ignore,"",,,,,,,, +OBS_FLAG:b,#ignore,"",,,,,,,, +OBS_FLAG:e,#ignore,"",,,,,,,, +OBS_FLAG:p,#ignore,"",,,,,,,, +OBS_FLAG:be,#ignore,"",,,,,,,, +OBS_FLAG:bp,#ignore,"",,,,,,,, +OBS_FLAG:ep,#ignore,"",,,,,,,, +OBS_FLAG:bep,#ignore,"",,,,,,,, \ No newline at end of file diff --git a/statvar_imports/eurostat/pop_broad_age_group_sex/run.sh b/statvar_imports/eurostat/pop_broad_age_group_sex/run.sh new file mode 100755 index 0000000000..c6fd32af2a --- /dev/null +++ b/statvar_imports/eurostat/pop_broad_age_group_sex/run.sh @@ -0,0 +1,17 @@ +#!/bin/bash + +# Step 1: Data Download +mkdir -p input_files +curl -L --retry 3 "https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/data/DEMO_PJANBROAD/?format=SDMX-CSV&compressed=false" -o input_files/pop_broad_age_group_sex_input.csv + +# Step 2: Full Data Processing (Using Clean Local Map) +python3 ../../../tools/statvar_importer/stat_var_processor.py \ + "--input_data=./input_files/*.csv" \ + "--pv_map=./pop_broad_age_group_sex_pvmap.csv" \ + "--config_file=./pop_broad_age_group_sex_metadata.csv" \ + "--generate_statvar_name=True" \ + "--skip_constant_csv_columns=False" \ + "--output_columns=observationDate,observationAbout,variableMeasured,value,observationPeriod,unit" \ + "--output_path=./pop_broad_age_group_sex_output" \ + "--places_resolved_csv=./places_resolved.csv" \ + "--existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf"