type: Academic topic: Science and Technology program: Data Science

Data Scientist, Data Engineer, or Technology Manager: Which Job Is Right for You?

Hal Varian, the chief economist at Google, is famous for saying in 2010 that “the sexy job in the next 10 years will be statisticians.” Building on this idea, Thomas Davenport and D.J. Patil wrote "Data Scientist: The Sexiest Job of the 21st Century."

Should you seek training in data science because the job is “sexy”? Probably not. You want to know that the job is in demand. Data scientists are in demand and so are data engineers.

Data scientists continue to be one of the fastest-growing occupations in the United States. According to the Bureau of Labor Statistics, the data scientist role boasts the 4th highest projected growth rate at 36% over the next decade (2023-2033), with an annual median pay of $108,020 in 2023.

There is also a high demand for data engineers. Indeed has included data engineers in their 2024 List of 10 Best Jobs in the US, with an average annual salary of $130,135.

The rise of generative AI and large language models has fundamentally transformed the technology landscape. Data scientist and data engineer roles are more critical than ever, particularly as organizations increasingly invest in these technologies. A 2024 Forrester survey reveals that 67% of AI decision-makers plan to increase generative AI investments in the coming year. This surge underscores the critical need for highly adaptable data scientists and data engineers with specialized skills in AI.

The Work of Data Science and Data Engineering

To remain competitive in today’s world, companies need to be informed by data and to use data in their day-to-day operations. That does not happen unless there are data engineers and technical professionals (software engineers, database administrators, cloud architects, and the like). These are people who can translate research results and data science models into systems that work.

The role of the data scientist is evolving with the emergence of capable generative AI and large language models. In addition to building new models from scratch, data scientists are integrating prebuilt models into the data pipeline. Many of these prebuilt models come from large technology firms. Building integrated, intelligent systems is the work of data engineering.

Data engineering can be seen as the marriage of data science and software engineering. Data engineers need to implement prebuilt models in production and to ensure that the systems they build are robust, scalable, and portable across operating systems and cloud environments.

Looking to the future, data scientists will continue to address management questions relating to business tactics and strategy. They will collect relevant data and identify appropriate methods for analyzing data. Data scientists will remain in high demand.

In addition to people with data analysis and modeling skills, however, there is a growing need for people who can build intelligent systems with advanced tools. These are the data engineers.

In his presentation Putting Data Science into Practice, Tom Miller, faculty director of Northwestern’s data science program, used this figure to distinguish between the roles of data scientists and data engineers:

Diagram: Roles of data scientists and engineers

Activities in blue in the figure are associated with data science. These are research, measurement, and modeling activities. Data scientists translate management questions into research questions, identifying relevant data sources and sampling frames, defining appropriate measures (or what is sometimes called feature engineering), and building and testing models of the data.

Activities in green in the figure are associated with data engineering. Data engineers have a key role to play at the beginning of every data science project. Data scientists depend on data engineers to gather and prepare data for analysis. Without data, there are no analyses, no models to build and test.

Some analytics and modeling projects end with a written report to management or a display of results in a dashboard or presentation. Research findings guide management decisions.

Analysis and modeling projects need not end with a report. Many models are put into practice. They become the way a company conducts its business. Data engineers have key roles to play in building data science applications and implementing information systems.

Putting Data Science into Practice Requires Data Engineering

Data engineers are essential to putting data science into practice. Examples of data engineering abound. Here are a few data science applications that you may encounter day to day:

Recommendation Engines

An online retailer provides product recommendations as you shop. The model that drives the recommendation engine may have been defined by a data scientist, but the system that delivers the recommendations was the work of data and software engineers.

Target Marketing

You receive an email message touting a new product. The targeting method, building on information about market segments, consumer preferences, and willingness to buy may have been devised by a data scientist. But implementing the method is the work of data and software engineers.

Dynamic Pricing

While scheduling your next trip by air, you notice how ticket prices vary greatly depending on the day and time of flight. Those prices are the work of scheduling models designed for to ensure that airplane seats are occupied in response to demand. The model serving an airline’s objective of profit maximization may have been developed by a data scientist specializing in operations research. The system delivering the price quotations is again the work of data and software engineers.

Fraud Detection

Your bank alerts you about a potentially fraudulent transaction, or your credit provider asks you to verify a purchase before executing a charge against your account. A data scientist was probably involved in training and testing an anomaly detection model for these financial institutions. Implementing the system, ensuring it works in near-real-time, as it must in order serve the needs of fraud detection, is the work of data and software engineers.

More About the Need for Data Engineers

Industry experts agree that data engineering skills are essential to addressing the data problems of business. Consider this article by Guillaume Moutier, Senior Principal Data Engineering Architect at Red Hat: The real issue behind the data science skills gap isn’t what you may think.

Data Science Specializations

Putting data science into practice requires data engineers, people who understand how to build end-to-end information systems. Northwestern recognizes the importance of data engineering by offering a Data Engineering specialization.

Seven courses in the Master of Science in Data Science (MSDS) program are closely aligned with the Data Engineering specialization:

Furthermore, Northwestern’s data science program is the first to feature the Go programming language, along with Python and R. Go is an excellent systems programming language for implementing data science applications, as described by Tom Miller in Data Science and the Go Programming Language.

The Northwestern data science program offers four other specializations. Analytics and Modeling and Artificial Intelligence are most closely aligned with the data scientist role. And students seeking technology management roles may select Analytics Management or Technology Entrepreneurship.

Choosing the Right Job for You

Data scientists are like chameleons, changing their colors to match the business context. There are data scientists who could just as easily be called marketing researchers, financial analysts, or competitive intelligence professionals, depending on the work that needs to be done.

Analytics and modeling lie at the heart of data science, and many data scientists think of themselves as applied statisticians.

Data science is an eclectic discipline, drawing on many fields of study, as shown in this figure:

Data science—the eclectic discipline

To serve the needs of today’s data-driven, data-intensive world, data scientists need to be multilingual, speaking the languages of information technology and business, as well as analytics and modeling.

Students interested in data science application development and systems implementation can specialize in data engineering and seek various information technology positions. Data engineers could just as easily be called software engineers, systems engineers, cloud architects, computer scientists, machine learning engineers, AI engineers, or development and operations (DevOpps) professionals. Job titles may vary from one company or one industry to the next, but the job opportunities are plentiful.

Reading job postings for data scientists and data engineers, you will find considerable overlap in the desired skillsets.

Data Scientist skills:

Programming
Cloud computing
Database management
Data visualization
Probability and statistics
Multivariate calculus and linear algebra
Machine learning & deep learning

Data Engineer skills:

Programming
Extraction Transformation and Loading (ETL)
System architecture
Database design and configuration
Interface and sensor configuration

Technology Management

Moving from the technology side to the management side of data science, presents additional opportunities. Organizations need people who can understand data science and data engineering, but also speak the language of business. This figure illustrates technology management roles:

Diagram: Roles of technology management

As with data science and data engineering, there are many jobs for technology managers. Someone needs to build and manage teams of data scientists and data engineers. Someone needs make decisions about information infrastructure.

Communication skills are essential to technology management. There is great need for people who can understand business problems as well as technical solutions to problems. There is great need for people who can translate technical jargon into language that non-technologists can understand.

Whatever role is best for you—data scientist, data engineer, or technology manager— Northwestern's Master of Science in Data Science program will help you to prepare for the jobs of today and the jobs of the future.

This story was updated on Dec. 13, 2024. It was originally published on Feb. 18, 2022.

Northwestern University School of Professional Studies offers many degree and certificate programs, with evening and online options available. To learn more about how Northwestern University's Master of Science in Data Science program prepares graduates to advance their careers, fill out the form below and we will be in touch with you soon.

tags:

February 18, 2022