9054d3be

Can Machine Learning Predict Olive Oil Prices

Fri, 13 Mar 2026 21:20:47 GMT

Here's What I Found.

The Question

Extra virgin olive oil has had a turbulent few years. Prices roughly doubled between 2022 and 2024, driven by catastrophic harvests in Spain and Italy. As someone with a background in data analysis and machine learning, I wanted to know: could an ML model have seen this coming? And more practically, can ML predict where prices are going next?

I built a forecasting system to find out. The short answer is: not at weekly frequency. But the journey to that conclusion taught me something genuinely interesting about how commodity markets work.

The Data

I used weekly wholesale prices for Extra Vergine di Oliva (acidity ≤ 0.4%) from the Camera di Commercio di Bari — the Bari Chamber of Commerce. Bari is the primary reference market for Italian olive oil, and the Chamber publishes a weekly price list that represents genuine wholesale transactions between producers, brokers and buyers in Puglia, Italy's largest producing region.

The dataset covers September 2020 to March 2026 — 289 weekly observations. This period is unusually rich: it captures a cyclical low in 2020, a steady climb through 2022, an extraordinary price spike peaking in early 2024 driven by drought across the Mediterranean, and a partial normalisation thereafter.

The target variable is the weekly percentage price change — calculated from the midpoint of the published minimum and maximum prices each week.

The Features

I assembled features in two layers.

Layer 1 — price history only:

Lagged weekly price changes (1, 2, 3, 4 and 8 weeks)
A price level signal (z-score relative to 52-week rolling mean)
Seasonality (week of year, harvest season flag)

Layer 2 — external signals:

EUR/USD exchange rate (FRED API)
Brent crude oil price (FRED API)
Weekly weather data for Puglia and Andalusia: rainfall, temperature, evapotranspiration, water balance, and 4-week and 13-week rolling aggregates (Open-Meteo API)
Google Trends search interest for "olio extravergine" and "olive oil price"
Annual harvest production figures for Italy and Spain (International Olive Council)

The final augmented dataset contained 41 features across 238 usable weekly observations.

The Models

I tested four models at four forecast horizons (1, 2, 4 and 8 weeks ahead):

Naive baseline — predict zero change every week
Linear regression — autoregression on lagged price changes
Random Forest — price history only
Random Forest — full augmented feature set

Performance was measured on a held-out test set covering January 2024 to March 2026, using three metrics: MAE, RMSE, and directional accuracy.

The Results

ModelMAE h=1RMSE h=1Dir% h=1Naive (predict zero)0.7082.08879.8%Linear regression0.9382.1409.6%Random Forest (prices only)1.0442.06910.5%Random Forest (augmented)1.1752.3146.1%

The naive model won on every metric at every horizon.

A model that simply predicts "prices won't change this week" outperformed every ML approach tested — including one with 41 features drawn from weather data, financial markets, search trends and harvest forecasts.

Why?

Three reasons, in order of importance.

1. The market is efficient at short horizons. Weekly price changes in the Bari wholesale market are essentially unpredictable from publicly available information. This is the weak form of the Efficient Market Hypothesis applied to a commodity market: by the time a signal is observable, the market has already incorporated it into prices. Traders and brokers in Bari are reading the same weather forecasts and harvest reports we used as features — they've already acted on them.

2. Most weeks nothing happens. Over 73% of weekly observations showed a price change of less than 0.5% in either direction. A model that always predicts zero is correct three weeks out of four by construction. This makes the naive benchmark exceptionally hard to beat and means directional accuracy is a misleading metric when applied to all weeks indiscriminately.

3. The real signals operate at the wrong timescale. The most important features identified by the Random Forest were precip_mm_13w (13-week rolling rainfall in Puglia), eurusd , and precip_esp_13w (13-week rolling rainfall in Andalusia). These are genuine, meaningful signals — but they drive prices over months, not weeks. A drought in Andalusia in July doesn't move the Bari price list the following Tuesday. It moves prices gradually over the following harvest season as the supply deficit becomes real and undeniable.

What the Models Did Learn

Despite failing to beat naive, the models consistently agreed on which features carry the most information:

13-week rolling rainfall in both Puglia and Andalusia — accumulated water stress over a quarter, not a single week
EUR/USD exchange rate — EVOO is heavily exported to dollar-denominated markets; a stronger euro compresses export margins and softens domestic prices
Week of year and price level relative to history — the market has a seasonal rhythm and a tendency to mean-revert from extremes

These findings are not noise. They are a description of how the EVOO market actually works — driven by structural fundamentals at seasonal frequency, not by short-term momentum or technical signals.

The Honest Conclusion

Weekly EVOO wholesale prices are not reliably predictable from the signals tested. This is not a failure of the models — it is a finding about the market.

The more interesting and tractable question is whether these signals can predict the direction of the seasonal price trend — whether prices will be materially higher or lower in 3 to 6 months. That is a question the harvest data, accumulated weather signals and exchange rate are much better positioned to answer, and it is the subject of the next phase of this project.

Data source: Camera di Commercio di Bari. External signals: FRED, Open-Meteo, Google Trends, International Olive Council. Code and dataset available on GitHub.

Was the EU’s Soft Power Ever Real?

Sun, 01 Mar 2026 14:30:03 GMT

And does it still matter?

For much of the past five decades, the European Union was described as a soft power: an international actor that shaped behaviour not through force, but through rules, norms, and persuasion. This idea was never about military strength. It was about how influence was exercised , and whether that influence could be considered legitimate.

From a republican perspective—one concerned less with outcomes than with procedures—this claim made sense. EU external action relied heavily on consent, institutional alignment, and voluntary rule adoption. Enlargement policy, neighbourhood conditionality, and regulatory diffusion all operated through processes that avoided overt coercion. On those terms, the EU could plausibly claim that when it exercised power, it did so in a non-dominating way.

Crucially, legitimacy and effectiveness were never the same thing. The EU was not a great power in the traditional sense, but that was not the standard it set for itself.

What has changed over the past decade is not the internal logic of this argument, but the world in which it operated.

Russia’s increasingly revisionist use of hard power, combined with the erosion of US security guarantees and a more isolationist turn in American politics, has exposed a structural weakness in Europe’s approach. Soft power presupposes a relatively stable international environment, shared minimum rules, and—often implicitly—a hard-power backstop provided by others. As those conditions have deteriorated, the limits of EU influence have become harder to ignore.

Even in its strongest domain—trade—the EU’s soft power has proven more constrained than often assumed. US retrenchment into protectionist policies has revealed fractures in Europe’s ability to shape outcomes beyond its own and neighbouring markets. While threats of tariffs have frequently remained rhetorical rather than fully enacted, their political effect has been real: encouraging accommodation, delay, and strategic caution rather than confident rule-setting. This suggests that EU economic influence is not immune to external pressure, even when formal coercion is absent.

At the same time, the EU’s use of economic conditionality appears to have been most effective where attraction already existed. Access to the single market has functioned as a powerful incentive primarily for countries that were geographically closer, economically interdependent, and at earlier stages of development. In these cases—particularly among neighbouring states and emerging economies—alignment with EU rules offered tangible growth prospects and a credible path toward deeper integration. Elsewhere, where market access was less central or alternative partners were available, conditionality has been markedly less persuasive.

This pattern complicates the image of EU soft power as globally diffuse. Rather than radiating outward evenly, it has tended to operate selectively , reinforcing convergence among those already oriented toward Europe while struggling to reshape preferences beyond its immediate economic orbit.

This helps explain why critiques such as those voiced by Mario Draghi resonate. His argument is not that the EU lacks influence altogether, but that its influence is uneven—strongest where markets and regulation dominate, and far weaker where strategic competition, speed, and coercive leverage matter most.

That does not mean the EU’s soft power project was misguided or illusory. Rather, it suggests that it was structurally dependent on favourable conditions : a rules-based international order, limited great-power rivalry, and partners already inclined toward economic and institutional convergence. As those conditions erode, soft power becomes harder to sustain—not because it lacks normative appeal, but because its scope is narrower than once assumed.

Seen this way, the problem facing the EU today is not simply one of weakness, but of misalignment between tools and environment . The challenge is no longer whether non-coercive influence can be legitimate, but whether it can remain effective in a world where attraction alone no longer suffices.

School Attendance in London

Sun, 08 Feb 2026 23:17:22 GMT

Patterns, inequalities, and the role of transport costs

Introduction

School attendance has become a central concern for policymakers, schools, and communities in the years following the COVID-19 pandemic. While recent national statistics suggest some short-term improvement, absence rates remain high by historical standards, and persistent absence continues to affect a substantial proportion of pupils.

In London, these challenges are particularly pronounced. Attendance patterns reflect a complex mix of health, social, and economic pressures, and vary significantly by age and socio-economic background. This briefing brings together recent attendance data for London with wider evidence on the drivers of absence, and explores the extent to which transport costs may play a role as one contributing factor.

The analysis draws primarily on administrative datasets published by the Department for Education (DfE), complemented by national transport data from the Department for Transport (DfT). While these sources allow for a detailed descriptive picture, important limitations remain, particularly around causal inference and data coverage. The focus of this briefing is therefore on patterns, trends, and plausible mechanisms, rather than definitive estimates of policy impact.

Attendance levels and recent trends

Using term-level local authority attendance data aggregated across recent academic years, average absence rates in London are around 9.4% overall . Absence is substantially lower in primary schools ( 6.4% ) and markedly higher in secondary schools ( 8.8% ), reflecting patterns seen nationally.

Looking across the last three academic years, absence rates in London appear to have been gradually declining since 2022/23 . This downward trend is visible for both disadvantaged and non-disadvantaged pupils and suggests a slow recovery from pandemic-related disruption. However, improvements have been incremental rather than dramatic, and absence remains well above pre-pandemic norms.

National Department for Education statistics provide important context. Recent figures indicate overall absence rates of around 6–7% in early 2026, alongside persistent absence affecting around 19% of pupils so far in the 2025/26 academic year. Although these national figures are not directly comparable to the term-averaged London estimates presented here, they suggest that short-term improvements have not yet translated into a resolution of deeper attendance challenges.

Inequalities in attendance

One of the most striking features of attendance in London is the scale of inequality by socio-economic disadvantage. Pupils eligible for free school meals (FSM) have an average absence rate of 11.0%, compared with 7.8% for non-FSM pupils, a gap of 3.2 percentage points. This difference is highly statistically significant and indicates a persistent and substantial disparity in attendance.

These inequalities are visible across school phases but are particularly pronounced in secondary education, where absence rates are highest overall. While the existence of an FSM attendance gap is well established, its size in recent London data underlines the continuing challenge of improving attendance for disadvantaged pupils, even as overall absence rates show signs of improvement.

Drivers of absence

A substantial body of evidence points to a set of overlapping factors that drive pupil absence. Illness remains the most frequently recorded reason for absence, accounting for a significant share of missed sessions nationally. Alongside this, mental health challenges-including anxiety, depression, and school-related stress-have become increasingly prominent and are strongly associated with persistent absence.

School experiences also play an important role. Disengagement from learning, a lack of belonging, feeling unsafe in the classroom, or negative relationships with peers or teachers can all contribute to non-attendance. Bullying, both in-person and online, can make school environments feel unsafe and lead pupils to avoid attending altogether.

Children with Special Educational Needs and Disabilities (SEND) are overrepresented in absence figures, often reflecting unmet or inadequately supported needs. Family and external factors, including term-time holidays, caring responsibilities, and wider socio-economic pressures such as poverty, further shape attendance patterns. These drivers rarely operate in isolation and tend to compound one another, particularly for disadvantaged pupils.

Within this wider set of factors, the cost and accessibility of transport h as been identified in previous research as a potential barrier to attendance, particularly for older pupils and those from lower-income households. This raises the question of whether changes in transport affordability can meaningfully influence attendance outcomes.

The role of transport costs: exploratory evidence

To explore whether transport costs are associated with changes in attendance, an exploratory analysis examined attendance patterns before, during, and after the introduction of the £2 bus fare cap in England (2023 and 2024). The analysis combined administrative attendance data from the Department for Education with national transport data from the Department for Transport.

Given the widespread coverage of the policy and limitations in available data, the analysis focused on changes within the same areas over time, rather than direct comparisons between different parts of the country. This approach does not allow firm conclusions about cause and effect, and the findings should be interpreted as descriptive and indicative rather than definitive.

The analysis suggests a small improvement in attendance during the period when the bus fare cap was in place . However, this improvement did not continue once the policy ended , with attendance levels rising again in the following period. In other words, while lower transport costs may have helped some pupils attend school more regularly in the short term, there is no clear evidence that this led to a lasting improvement in attendance .

This pattern is consistent with wider research showing that transport affordability can ease immediate barriers to attendance, but is unlikely on its own to address the deeper drivers of persistent absence.

Implications for policy

Taken together, the findings in this briefing point to the limits of single-policy interventions in addressing complex attendance challenges. Attendance outcomes are shaped by a combination of health, social, educational, and economic factors, many of which interact and reinforce one another.

Policies aimed at reducing the cost of travel to school may play a useful supporting role, particularly during periods of acute cost pressure. However, lasting improvements in attendance are likely to require coordinated action across transport, education, health, and social support , with particular attention to secondary schools, disadvantaged pupils, and those with additional needs.

In this context, transport affordability should be seen as one component of a broader strategy , rather than a standalone solution. Addressing the underlying drivers of absence-mental health, SEND provision, school engagement, and family circumstances-remains essential if attendance gaps in London are to narrow in a sustained way.

Notes on data and interpretation

This analysis is descriptive rather than causal. It draws on term-level local authority attendance data published by the Department for Education and national transport data from the Department for Transport. Due to data coverage and structural limitations, national-level geographic mapping and causal policy evaluation were not feasible. The focus is therefore on trends, inequalities, and exploratory evidence to inform policy discussion rather than definitive impact estimates.

What role should polling play in modern democracy?

Thu, 11 Dec 2025 16:48:38 GMT

Challenges faced by the industry today

Recently, while browsing social media, I came across the following question, posted by a social scientist: “If the modern polling industry had existed in 1939, would Britain have declared war on Germany over Poland?” The ensuing debate was engaging, as it raised the issue of the growing influence of public opinion—and polling—on public policy. The tension between leadership and populism, while strikingly current, is not new. It was the 19th-century French radical Alexandre Auguste Ledru-Rollin who, allegedly, said, “There go the people. I must find out where they are going so I can lead them!”

If politicians merely chase public sentiment rather than lead with vision and principles, are they truly governing? This highlights a key dilemma in democratic theory: should leaders act as delegates, executing the direct will of the people, or as trustees, making informed decisions on their behalf? Polling sharpens this tension by providing instant snapshots of public opinion, forcing politicians to navigate between responsiveness and responsibility.

Polling plays a fundamental role in modern democracy by keeping governments accountable and ensuring they remain in touch with public sentiment. It provides valuable insight into attitudes on policies, especially between elections, when direct democratic input is limited. However, it can also turn politics into a popularity contest, encouraging leaders to seek short-term approval rather than pursue long-term objectives. Moreover, polling can be manipulated to drive polarization, with political actors selectively using data to justify decisions rather than genuinely listening to the public. When polling is transparent and widely available, it strengthens democracy by amplifying public voices. But when selectively used, distorted, or overemphasized, it risks tilting democracy toward elite control, where “public opinion” becomes a curated tool rather than a genuine expression of the people’s will.

Beyond its political implications, the polling industry faces significant methodological and technological challenges. Declining response rates make it harder to obtain representative samples, and in an era of fragmented media and digital communication, traditional polling methods struggle to keep up. Online surveys, while faster and cheaper, often rely on self-selected respondents, increasing the risk of bias. Additionally, the rise of AI-generated responses and bot activity further threatens polling accuracy, making it harder to distinguish genuine opinion from manipulation.

To remain relevant and credible, the polling industry must adapt to these challenges by refining its methodologies, improving transparency, and embracing new technologies without compromising accuracy. Ultimately, polling should inform democratic decision-making—but not dictate it. The balance between public feedback and principled leadership remains central to ensuring that democracy is both responsive and effective. Had modern polling existed in 19th-century France, Alexandre Auguste Ledru-Rollin might have been more than just a footnote in history.