measuring skills shortages in real time

Measuring skills shortages in real time

By Fabio Manca.

Blog31.01The past decades have seen important shifts in the skills required by employers and the labour market. These shifts have been brought about by what have become known as the “mega trends”: changes in the global division of labour and a growing dependence of domestic jobs on economic globalisation; changes in the way firms are organised; demographic change.

While these changes can provide opportunities for growth and innovation, employers are often not able to take advantage of them due to difficulties finding the skills they are looking for in the labour market. This is so despite high levels of over-qualification across workers or high unemployment rates. As an example, unemployment in Europe rose from 7% in 2008 to 10.8% in 2013. Yet, in 2013, there were around two million vacancies available in the EU and four out of ten employers reported difficulties in finding employees with the right skills. The reasons behind this are multiple but the lack of information about evolving skill needs is at the root of the problem. Without this knowledge it is hard to even start to adapt education and training systems to the new labour market requirements.

This is where technology – the very reason behind changing skill requirements – can play another key role by enabling the collection of a great deal of information precisely on those changes…and quickly.

It is no mystery that, nowadays, most of job vacancies are advertised online. New platforms like LinkedIn are on the rise as these enable sharing not only business contacts but also information about job-vacancies in real time across the globe. Websites specialised in distributing information on job offers such as Monster have created global platforms containing real-time job vacancies in all fields and sectors. Similarly, other platforms such as Glassdoor permit to retrieve information on salaries across a multitude of companies and countries, allowing job-seekers to take more informed decisions on their career paths.

All this information is not only useful for job-seekers but it can also be exploited to rapidly measure skills pressures on the labour market (e.g. the emergence of skill shortages) creating new opportunities for policy makers to track the evolution of skills needs and for education systems to plan training so as to meet the needs of the labour market in a timely fashion.

The use of vacancy data and of real-time job market information is, therefore, playing an increasingly important role within the available set of tools to monitor and measure skills shortages in the labour market. Web-scraping technologies (e.g. automated software programmes that simulate human searches in the internet for specific contents, in this case: job-vacancies) allow to collect information on job-postings on a daily basis. Burning Glass, a rapidly growing company based in the U.S. collects, for instance, job advertisements from over 40,000 websites every day. Their automated web-scraping routine and text mining algorithms allow to parse each posting to extract information such as the job title, occupation, skills requirements, certifications as well as the salary of job-adverts and then place them within the Standard Occupational Classification (SOC).

The collection of real-time labour market information has drawn the attention of the European Union too. The EDUWORKS project is a European initiative sponsored under the Marie-Curie Initial Training Network programme that pools together researchers from several different institutions across 5 disciplines (Labour Economics, Sociology of Occupations, Human Resource Management, Lifelong Learning and Knowledge Management). Its aim, among others, is to extract ICT skills from web job advertisements to allow for cross-country/region comparisons and to uncover trends in skills demands across different labour markets.

So, what is the use of this information more concretely? Job-vacancy data can be used to tell what occupations are more likely to suffer from skills shortages. Economists, for instance, usually interpret long vacancy periods (information that can now be collected rapidly) as signalling skills shortages and difficulties for the employers in finding the adequate skills in the labour market.  Analysing real-time information about the growth in vacancies across occupations or the evolution of salaries (e.g. wage pressure analysis) can also allow to pin down what skills are in need in the labour market and provide extremely useful input to policy makers.

On a similar line, vacancy data from Burning Glass (Figure 1), has been used, for instance, to compute the growth in job-posting requiring digital competencies. Maybe surprisingly, health care activities have been found among those occupations that increasingly require digital skills. This information, and other of this kind, can play a fundamental role in adjusting training and education paths to meet the needs of labour market’s trends, eventually fostering a better match between the skills of the workforce and those required by employers.


But this is not all. Big-(jobs)-data can be used to learn (better) what businesses do across and within sectors. The recent Tech Nation 2016 report published by NESTA in the UK makes use of the data collected by GrowthIntel to study the sector in which a business operates (e.g. ‘financial services’) and to determine if its “core capabilities” are digital.  Figure 2, for instance, shows the proportion of ‘digital’ businesses in traditionally ‘non-digital’ sectors. Results highlight how competences that were once exclusively required by a specific sector are now spreading onto others. Training programmes should be adjusted to fill the potentially emerging skill gaps due to these new trends.


All in all, the use of big data and the collection of real-time labour market information are going to represent an invaluable tool to anticipate what skills will be needed in the future and to inform education and employment policies to spur a better match between workers’ skills and those required by a fast-changing labour market. The OECD is already working on this and interesting news are going to be coming soon. Follow us!