Data Analytics: Use cases and Challenges for SMEs

With advances in digital technologies, increased internet penetration, and the rapid growth of devices connected to the internet, the volume, speed, and detail of data on the internet is unprecedented. As such, web data extraction and integration (WDEI) is now more than ever, important in unlocking the value of digital data by turning it into actionable insights for businesses.

Sometime last year, I had a lengthy conversation with two top executives who had unsuccessfully ventured into WDEI initiatives in their companies. One was in the travel and tourism industry and the other was in real estate. The failure of such data projects is something I am hearing quite often as I interact with small and medium-sized businesses.

That being said, I also know of a large Kenyan firm in extracting hundreds of thousands of pages a day. With their enormous data extraction capabilities, they turn digital ‘breadcrumbs’ from different sources into valuable real-time insights to inform business decisions. Having learned about these contrasting experiences, I felt it was necessary to contribute to the conversation about data analytics, especially for small and midsized enterprises (SMEs).

First, let me talk a bit about some of the use cases for data extraction and analytics for SMEs. Even after spending three years consulting for clients in the banking, retail, and event industry, I am still intrigued by the wide and endless applications of web data analytics. For now, I will just highlight four broad applications. 

One big area of application for web data analytics is competitive analysis. There are at least 2 billion websites in the digital space, so businesses can easily and quickly find out about who their competitors are, and what sort of products and services they are offering. In a world where most products and services are being sold online, data analytics, for instance, can enable real-time price comparison to ensure businesses offer competitive prices. That is important in retaining customers who are always shopping for the best deals in a competitive eCommerce space. Without a doubt, failure to adapt the pricing strategy by matching competition, offering cheaper prices, or greater value can be disastrous for businesses in today’s marketplace.

Another application WDEI is what I would call “health scores”. There are tons of data on blogs and social media that can be extracted to make sense of what customers are saying about a brand, product, or service. Using techniques like topic modeling and sentiment analysis, these qualitative data can provide feedback for businesses, insights into the perception of the brand, and help anticipate a business crisis. 

Relatedly, web data analytics can lend perspective to the effectiveness of marketing efforts. For example, data from blogs and social media can shed light on the geographical reach of a brand or product. Combined with insights from topic modeling and sentiment analysis, aggregated data on customer engagement from multiple platforms (e.g., call centers, company emails, Facebook, Twitter, YouTube, etc.) can also tell a business whether their marketing campaigns are paying off.

Lastly, WDEI can help businesses understand and segment customers. By extracting and aggregating dozens of data points, it is possible to uncover habits and traits that are valuable when it comes to customer-centered product design and marketing. Segmentation can help SMEs to better serve the needs of customers by understanding demographic and locational traits. In the next few years, I also expect to see the continued growth of behavioral segmentation, which has to do with grouping customers based on their unique personas, needs, expectations, journeys, and their buying patterns.

My sense is that SMEs aspiring to create and sustain a winning edge over their peers must understand their customers, design products that meet their needs, and optimize their marketing efforts. That can be done through ethnographic research and data analytics.  

The failure of the two projects is indicative of the complexity of data analytics for SMEs. At first, it had seemed to them like a straightforward routine involving free python libraries to extract and analyze data on their markets. They had both hired relatively experienced data scientists for the tasks, but the initiatives quickly ran into problems and were shut down.

The sources and data structures were changing too rapidly, and they were unable to scale their projects beyond a few static web pages.  And of course, the fragmented nature of digital data required complex codes and different paid software to effectively extract and transform data into business insights.

It is not just the cost of setting up that presents a big problem for SMEs. The cost of maintaining human capital and an analytical database begin to pile up when businesses start scaling their data operations. For example, one of the two companies needed to spend at least Sh8.5 million to scale their data project, an average of Sh708,333 per month. As trendy as data analytics seems, the cost implications of adopting it for small businesses cannot be brushed aside.

Perhaps the least talked about yet most important consideration for any business venturing into web data analytics are the legal and ethical concerns. The Facebook-Cambridge Analytica scandal, and more recently, a lawsuit pitting Microsoft-owned LinkedIn against HiQ brought data rights and privacy to the forefront of data science conversations. Like in many emerging areas of research, there's still a lack of clarity on the do’s and dont's. 

Late last year, the US Court of Appeals denied LinkedIn’s request to bar the start-up from scrapping its platform making publicly available data “technically” legal in the US. Although some realms of data extraction are illegal, the historic decision is bound to set precedence for data regulations worldwide. Therefore, extracting and using publicly available data requires careful consideration of legal and ethical issues around personal privacy, rights, and the extent to which web extraction violates the rights of other companies. Navigating these legal and ethical issues around WDEI takes research experience that may not be available to most SMEs.

But how can SMEs leverage the vast amount of web data? Part of the answer to the first question is in the analysis of the demand for data engineering and scientist occupations across the world.

The 2020 edition of Dice’s Salary Report revealed significant demand and salary growth for data skills. Salaries for data engineers and scientists grew by 9.3 percent and 11.4 percent respectively between 2019 to 2020 in the US, compared to software developers (4.7 percent) and project managers (4.6 percent). In Kenya, the trend is quite similar. The data science community is growing rapidly, and data skills are now required in almost every industry.

In other words, companies are hiring data people and paying lucrative salaries more than any other time. For the two executives, the lack of internal skills to handle the complex projects, and the cost of building highly experienced and effective data teams became too much to bear.  

Obviously, internal expertise would have some sort of continuity, but the complexities of the WDEI projects necessitated a change of approach. As companies think about the risks around data regulations and realize they may not have internal skills and resources to scale the projects, they start exploring the option of working with experienced and specialized external providers. In the next few years, we are certainly going to see a shift from internal data initiatives to consulting models for SMEs as companies readjust their spending due to the COVID 19 pandemic.

The use cases I have highlighted represent just a tiny sample from the plethora of applications for WDEI. I should also point out that the opportunities will continue to grow across different industries. Businesses will also try to find new ways of using digital data to make informed and effective decisions. But to find value in web data, companies must be willing to build and sustain talented data science teams. Then again, SMEs can also benefit more from the multidisciplinary experience and “out of the box” thinking that comes with working with consultants.