How Are Enterprises Linking Data Talent Gap?

Bridging the Talent Gap Using AI, Data & Analytics

Add bookmark

AI, Data & Analytics Network (ADA) Editorial Team
11/09/2022

Welcome to the Great Resignation

In many ways, 2022 has been an exercise in dealing with the reeling of the Covid-19 pandemic. All over the world, supply chain issues have led to a myriad of challenges for enterprises, leading to deep shifts in how we do business and unravelling many of our assumptions about how society operates.

This upending is seemingly never ending, with businesses now having to come to grips with the Great Resignation — the name given to the dearth in talent unfolding across all industries in wake of the Covid-19 pandemic.

While the Great Resignation was in many ways a long time coming – according to a ManpowerGroup survey, talent shortages in the U.S. have more than tripled in the last ten years with 69% of employers struggling to fill positions (up from just 14% in 2010) - this surge in empty positions has been amplified profusely by the pandemic.

This is of course being noticed already with Harvey Nash and KPMG, who surveyed over 3,000 technology leaders for their annual CIO Survey, finding that 65% said that hiring challenges are hurting the industry.

Indeed, O’Reilly’s 2021 AI Adoption in the Enterprise report, which surveyed more than 3,500 business leaders, found that a lack of skilled people and difficulty hiring topped the list of challenges in AI, with 19% of respondents citing it as a “significant” barrier — revealing how persistent the talent gap might be.

This in turn is leading to further problems in terms of adoption, with IT executives stating that the talent shortage is the most significant adoption barrier to 64% of emerging technologies, compared with just 4% in 2020 according to a September 2021 survey by Gartner.

ALSO READ: Advanced Analytics 2021 Survey

It doesn’t look like this issue will solve itself anytime soon, with an October 2021 survey conducted by TalentLMS finding that of the 1,200 IT workers polled and found that 72% said they are thinking of quitting their jobs in the next 12 months. In addition, 75% agree that their employers are focusing more on hiring new staff rather than investing in the existing workforce.

All in all, the outlook doesn’t look good. The global consulting firm Korn Ferry reported that over the next few years the US alone could lose out on $162 billion worth of revenues per year unless it finds more high-tech workers.

Here at the AI, Data and Analytics Network we decided to take stock of the situation even further, asking hundreds of enterprises about the current make-up of their data expert and examining their short term and long-term goals in this year’s talent survey: Bridging the Data Talent Gap.

Data Science Team Structure

To begin, we asked respondents about how their current data science teams are structured, asking them how many data scientists, analysts, engineers, etc. their organizations currently employ and if they expected their data science team to grow over the next year.

In terms of composition, a majority of respondents stated that their data science team was centralized, with 34% of respondents stating this to be the case. This was followed closely by 30% stating that their teams were decentralized, with a similar figure of 31% stating that it was a hybrid of the two.

How is your data science team structured?

How many data scientists, analysts, engineers, etc. does your organization currently employ?

Do you expect your data science team to grow over the next year?

Yes, I expect it to grow- 78%
No change- 16%
Decrease- 1%
I don’t know- 5%

Such a similar finding is to be expected, as 46% of respondents stated that their data teams consisted of 1 to 10 employees in total (in contrast, 8% stated their data science teams were 1,000+ in total). This in part explains why 79% of our respondents thought that over the next year, their data teams would be likely to grow. Larger organizations are often ones with more centralized control of their teams, and this then naturally includes their data science teams also. Conversely, smaller teams tend to be decentralized and contained within the business units that they serve. Whether centralized, decentralized, or hybrid, your data science teams will change depending on your business needs and analytical aspirations.

Budgets: 2022 & Beyond

In terms of budgeting for changes in the data talent landscape, 74% of respondents stated that they were expecting to increase their team’s budget for hiring, training and engaging data talent to increase over the next year. This correlated with the fact that presently 41% of respondents stated that their budget for hiring, training and engaging data talent was under $500,000 per annum, with 10% stating that they had no budget at all.

Do you expect your team’s budget for hiring, training and engaging data talent to increase over the next year?

What is your annual budget for hiring, training and engaging data talent?

Turnover

In terms of turnover, 37% of our respondents stated that it was under 10% of their data teams, whilst some 27% stated that it was somewhere between 11 and 25%.

What is your annual turnover rate for data-related roles?

While higher turnover may raise alarms for individual enterprises, Katie Bardaro, lead economist at PayScale recently told Business Insider that it isn’t always a bad signal for the wider market.

“Workers might be job hopping more than before this means that the industry is hot, and the economy is improving,” she said. “Some of the firms on [the high turnover] list, they’re there because they’re a hot market.”

“If employees have more options and can easily move, they’ll do it,” she added. “You’ll see it happening first with top performers in this environment, companies will need to evaluate what causes employees to leave and improve these areas, such as pay, work environment, [and] vacation policies.”

In this, Barbardo states that changes in employee loyalty are the wider causes, as well as the fact that many of these employees lived through the 2008 Financial Crash, wherein they witnessed their wider families and networks being laid off and losing their pensions.

Talent Search Challenges

More widely, when asked what the top three challenges are when it comes to successfully recruiting, training, and retaining appropriately educated and skilled employees, finding the people with the right technical skills scored highest at 56%, followed by finding people with the adequate domain or industry knowledge at 40%. The third most pertinent challenge was keeping talent motivated, with 32% of respondents stating that keeping existing employees inspired and engaged was a significant issue.

The lack of data experts in the data science field has been tracked for a while at this point. As Dr. Robert P. Eyle told careers-search website Zippia last year: “We will see more jobs out there for data science and analysis across multiple subfields (AI programming, digital marketing, process automation, information/cyber security). Many of these trends were already in motion pre-pandemic and are now hastened due to the dynamic marketplace online and on our phones. We will also see risk management specialists hired as businesses come back to work slowly but surely.”

When it comes to successfully recruiting, training, and retaining appropriately educated and skilled employees, what are your top 3 challenges?

Talent Shortage Breakdown

When we asked respondents in which areas, they were seeing a direct talent shortage, 36% stated they were lacking staff in AI development. AI development is the process of training artificial intelligence systems with relevant and sufficient data to enable them to carry out intelligent tasks almost as effectively as humans (or even more). This is important considering the fact that there was a 270% increase in businesses adopting AI systems over the past 4 years according to Gartner, highlighting the increased importance of AI systems (and therefore their development) on how enterprise works in 2022.

Other notable shortages included AI engineering and research (31% and 29%), data analysis (33%) and data scientists at 34%. These areas were also expected to see an increase in the skills gap supply of talent vs. demand) over the next 5 years, with AI development once again seeing the highest expectation at 32%.

In what areas is your organization experiencing talent shortages? Do you expect these talent gaps to increase over the next 5 years?

Solving this increasing issue and closing the data talent gap was reported as somewhat of a priority by 58% of those surveyed, with a quarter stating that it was a major priority for them going forward.

Is closing the data talent gap a priority for your organization?

Not a priority- 17%
Somewhat a priority – 58%
A major priority – 25%

The ways in which enterprises are solving this problem are manyfold. Almost 50% of those surveyed said they were reskilling existing tech workers in order to mitigate the data talent crisis, with 34% saying they are upskilling employees in non-data related roles.

Further solutions came in the form of expanding remote work options (29%), together with the outsourcing or offshoring of more roles (28%) and more generally increasing external hires (32%). Many respondents stated they are building strategic partnerships with universities and associations (28%).

In total, 19% said that they were increasing salaries, a stat which correlates with findings that in some metro areas, tech salaries have spiked by as much as 14% in just one year.

How is your organization working to close the data talent gap?

Beating Burnout

Post-pandemic has seen a surge in employee burnout. A survey of 1000 workers conducted by Indeed found that a startling 52% of employees were feeling burned out by their work, with 67% of respondents stating that it had gotten worse over the course of the pandemic. When we asked respondents about their enterprises, the results followed this wider trend. Some 54% stated that it was somewhat of a problem, with 16% stating it was a major issue. 32% conversely, stated that it was not an issue in their data science organization.

The problem is seemingly understood by our surveyants, who were tackling the problem with a myriad of solutions. 44% said they were actively working to mitigate overwhelming workloads, with a third offering mental health services. Almost half (49%) said they were training managers to actively identify and address burnout, whilst 48% stated they were revisiting workplace policies to create more flexibility for employees.

Is employee burnout a major problem for your data science organization?

How is your organization addressing employee burnout?

This correlates with research by Pew which shows that, since the early days of the pandemic, flexible working has shifted from a matter of survival to a matter of preference. In a recent study, Pew writes that, “Today, more workers say they are doing this by choice rather than necessity,” and that, “Among those who have a workplace outside of their home, 61% now say they are choosing not to go into their workplace, while 38% say they’re working from home because their workplace is closed or unavailable to them.”

Collaboration Ignited

The rise of hybrid working and working from home has led to a torrent of collaboration tools entering the market. When we asked our surveyants which they used the most, the open-source software application GitLab came top at 70% with a further 30% stating they were considering using it in the future.

While this is interesting and shows the growing commitment to software development, what is perhaps more enlightening is that when taking into account weighted averages Troop Messenger, an office chat tool, came top at 1.8.

This was by no means the only messaging app that respondents stated they were considering for future use, with apps like Chanty and Blink also scoring high when it came to weighted averages. What this shows is that as the market continues to evolve, so too will the tools that enterprises use in order to solve their individual problems.

ALSO READ: Democratizing Data & Analytics

However, it would seem that those already adopted have a first-mover advantage, with 71% of respondents stating that their current collaboration tools were satisfactory.

Do you currently use or plan to use any of the following workplace collaboration tools?

In terms of video collaboration tools, it would seem that the pandemic has cemented them as common across all enterprises, with 93% of those surveyed stating they use Microsoft Teams and 90% using Zoom.

Do you currently use or plan to use any of the following VIDEO collaboration tools?

These tools have paved the way for further considerations for immersive tools going forward, with 90% of respondents stating that they were considering VR tools such as Somnium Space in the future.

Do you currently use or plan to use any of the following IMMERSIVE collaboration tools?

How much of this expected adoption is due to the metaverse dominating the headlines for most of this past year rather than actual intention is less clear. However, it seems that businesses are betting that immersive technology – which grows cheaper and more viable by the day – is likely to see mass adoption over the coming year. Statista for example reported that while the global augmented reality (AR), virtual reality (VR), and mixed reality (MR) market reached 28 billion U.S. dollars in 2021, it is expected to rise to over 250 billion U.S. dollars by 2028.

It is unclear how many will afford these new tools in the future however, with only 45% of our respondents stating that collaboration tools are included in their budgets.

How would you rate your current collaboration tools?

Are collaboration tools included in your budget?

Respondent Demographics

The 150+ Data, AI and Analytics leaders we surveyed come from all across the globe and represent a wide range of functions, industries and maturity levels.