source: marktechpost.com

Home » Data Science

Data Science

Data Science: Definition

Data science continues to evolve as one of the most promising and in-demand career paths for skilled professionals. Today, successful data professionals understand that they must advance past the traditional skills of analyzing large amounts of data, data mining, and programming skills. In order to uncover useful intelligence for their organizations, data scientists must master the full spectrum of the data science life cycle and possess a level of flexibility and understanding to maximize returns at each phase of the process.

source: ischoolonline.berkeley.edu

Data Science: Quick glossary

Hadoop Performance

MapReduce

Predictive modeling

Predictive modeling, a component of predictive analysis, is a statistical process used to predict future outcomes or events using historical or real-time data. Businesses often use predictive modeling to forecast sales, understand customer behavior and mitigate market risks. It is also used to determine what historical events are likely to occur again in the future. Predictive modeling solutions frequently use data mining technologies to analyze large sets of data. Common steps in the predictive modeling process include gathering data, performing statistical analysis, making predictions, and validating or revising the model. These processes are repeated if additional input data becomes available.

source: techrepublic.com

zero trust

Grounded in the principle of “never trust, always verify,” zero trust is designed as a response to the outdated assumption that everything inside of an organization’s network can be implicitly trusted. Traditional layers of security assume users and data are always operating within the confines of the enterprise walls and data centers — like a physical store. But today’s enterprises have users and partners working from anywhere and accessing applications and data deployed across data centers and external clouds — like an online store.

source: techrepublic.com

Address Verification

Address Verification definition

Address verification cleans and standardizes address data against an authoritative address database. Address verification software corrects spelling errors, formats information, and adds missing ZIP codes to incomplete or inaccurate physical addresses.

reference: dataladder.com

Address validation vs. verication

Address validation checks a postal address against a database to determine

whether it’s a deliverable address. Address validation tools typically use authoritative databases such as the United States Postal Service (USPS) or Canada Post to run these comparisons.

How does our address verification tool work?

What if you could make it easier for your customers or staff to enter a valid

address, improve the quality of your data and increase conversions? Well, you can with Loqate. Watch our short video to learn more about international address validation.

What is the purpose of address verification?

The Address Verification Service (AVS) is a fraud prevention system that, when

used effectively, can help to limit fraud and charge-backs. AVS works to verify that the billing address entered by the customer is the same as the one associated with the cardholder’s credit card account.

Benefits

An address is a vital identification attribute to verify a person’s identity. Proo

of Address is often requested when opening a bank account or other government account to confirm residence, help Know Your Customer (KYC) compliance and prevent fraudulent activities.

reference:
g2.com
loqate.com

Data Cleansing

Data Cleansing: Definition

Data cleansing is a process by which a computer program detects, records, and corrects inconsistencies and errors within a collection of data.
Data cleansing, also referred to as data scrubbing, is the process of removing duplicate, corrupted, incorrect, incomplete and incorrectly formatted data from within a dataset. The process of data cleansing involves identifying, removing, updating and changing data to fix it. The objective of data cleansing is to make reliable, consistent and accurate data available throughout the data lifecycle.

Benefits of data cleansing

Data cleansing processes have moved from a “nice to have” to a “must have” for effective data-driven operations, especially as businesses growl increasingly reliant on data for decision-making. If data is not cleansed, it can lead to flawed business planning and missed opportunities, which can result in reduced revenue and increased costs. It can also compromise the ability of an organization to leverage their data analytics technologies.

source: techrepublic.com

Steps to performing data cleansing

Step 1: Remove irrelevant and duplicate data
Step 2: Fix formatting and structural errors
Step 3: Filter outliers
Step 4: Address missing data
Step 5: Validate data
Step 6: Report results to appropriate stakeholders

Data cleansing tools

One of the challenges of data cleansing is that it can be time-consuming, especially when pinpointing issues across disparate data systems. One of the best ways to make data cleansing more efficient is to use data cleansing tools.

There are a variety of data cleansing tools available in the market, including open source applications and commercial software. These tools include a variety of functions to help identify and fix data errors and missing information. Vendors, such as WinPure and DataLadder, offer specialized tools that focus solely on data cleansing tasks. And some data quality management tools, such as Datactics and Precisely, also offer helpful features for data cleansing.

The core features of data cleansing tools include data profiling, batch matching, data verification and data standardization. Some data cleansing tools also offer advanced data quality checks that monitor and report errors while processing data. There are also workflow automation features offered by some data cleansing tools that automate the profiling of incoming data, data validation and data loading.

source: techrepublic.com

We use in most case
community.talend.com

reference: dataladder.com

Data Deduplication

Data Deduplication: Definition

Deduplication refers to a method of eliminating a dataset’s redundant data. In a secure data deduplication process, a deduplication assessment tool identifies extra copies of data and deletes them, so a single instance can then be stored.
Data deduplication software analyzes data to identify duplicate byte patterns. In this way, the deduplication software ensures the single-byte pattern is correct and valid, then uses that stored byte pattern as a reference. Any further requests to store the same byte pattern will result in an additional pointer to the previously stored byte pattern.

source: druva.com
reference: dataladder.com

Data Literacy

Data literacy: why is it essential for success?

Data literacy refers to the ability to read, understand, communicate, analyze and derive information from data, all while putting it into proper context. Forbes defines data literacy as using “data effectively everywhere for business actions and outcomes.”

Data literacy is a set of skills and knowledge used to find, understand, evaluate and create data.

With data literacy skills, employees better understand how company data works and how they can use it, allowing them to be more effective and streamline processes for the organization.

With the growing importance of data literacy in organizations and the abundance of data, there is increased emphasis on establishing data literacy training programs and appointing chief data officers to continuously assess and improve data literacy in the organization.

source: techrepublic.com

Why is data literacy important for your business?

Data literacy skills are not only required by the analytics or the IT team; all departments and roles within an organization can benefit from data literacy skills. Data literacy enables employees to ask the right questions, gather the right data and connect the right data points to derive meaningful and actionable business insights. It also ensures that all employees understand how to manage and use data in ways that are ethical and compliant.

Data literacy examples and use cases

The following data management frameworks and tasks work best when the entire organization is made up of data-literate staff:

Data ecosystems. Data literacy is useful in establishing and maintaining a reliable data ecosystem, which can include physical infrastructures such as cloud storage or service space and non-physical components, such as software and data sources.
Data governance.Organizations use data governance to manage their data assets so that they are complete, accurate and secure. Data governance is not the sole responsibility of any particular team; the entire workforce must have the appropriate data literacy levels to contribute to its success.
Many organizations have a data policy that all employees must understand and adhere to. This includes how to access sensitive data, how to ensure data remains secure and other data processes.
Data wrangling. is the process of converting raw data into a more structured and usable format. Data wrangling helps reduce errors in the data. An organization might have individuals or automated software for data wrangling, but every employee that works with any form of data also plays a role in keeping data in an acceptable format.
Data visualization. Creating a visual representation of data, such as a chart or graph, allows data professionals to more effectively communicate insights derived from data. Visualization can include infographics, tables, videos, charts, and maps. Both the creators of these visualizations and the stakeholders to whom they are presented need at least baseline levels of data literacy to understand the implications of the data in front of them.

source: techrepublic.com

Data Masking

Data Masking: Definition

Data masking is a way to create a fake, but a realistic version of your organizational data. The goal is to protect sensitive data, while providing a functional alternative when real data is not needed—for example, in user training, sales demos, or software testing.

Data masking processes change the values of the data while using the same format. The goal is to create a version that cannot be deciphered or reverse engineered. There are several ways to alter the data, including character shuffling, word or character substitution, and encryption.

How data masking works

Why is Data Masking Important?

Data masking solves several critical threats – data loss, data exfiltration, insider threats or account compromise, and insecure interfaces with third party systems.

Reduces data risks associated with cloud adoption.
Makes data useless to an attacker, while maintaining many of its inherent functional properties.
Allows sharing data with authorized users, such as testers and developers, without exposing production data.
Can be used for data sanitization – normal file deletion still leaves traces of data in storage media, while sanitization replaces the old values with masked ones.

Data Masking Types

There are several types of data masking types commonly used to secure sensitive data.

Static Data Masking (SDM) Static data masking processes can help you create a sanitized copy of the database. The process alters all sensitive data until a copy of the database can be safely shared. Typically, the process involves creating a backup copy of a database in production, loading it to a separate environment, eliminating any unnecessary data, and then masking data while it is in stasis. The masked copy can then be pushed to the target location.
Deterministic Data Masking Involves mapping two sets of data that have the same type of data, in such a way that one value is always replaced by another value. For example, the name “John Smith” is always replaced with “Jim Jameson”, everywhere it appears in a database. This method is convenient for many scenarios but is inherently less secure.
On-the-Fly Data Masking Masking data while it is transferred from production systems to test or development systems before the data is saved to disk. Organizations that deploy software frequently cannot create a backup copy of the source database and apply masking—they need a way to continuously stream data from production to multiple test environments. On the fly, masking sends smaller subsets of masked data when it is required. Each subset of masked data is stored in the dev/test environment for use by the non-production system. It is important to apply on-the-fly masking to any feed from a production system to a development environment, at the very beginning of a development project, to prevent compliance and security issues.
Dynamic Data Masking Similar to on-the-fly masking, but data is never stored in a secondary data store in the dev/test environment. Rather, it is streamed directly from the production system and consumed by another system in the dev/test environment.

Data Masking Techniques

Let’s review a few common ways organizations apply masking to sensitive data. When protecting data, IT professionals can use a variety of techniques.

Data Encryption When data is encrypted, it becomes useless unless the viewer has the decryption key. Essentially, data is masked by the encryption algorithm. This is the most secure form of data masking but is also complex to implement because it requires a technology to perform ongoing data encryption, and mechanisms to manage and share encryption keys
Data Scrambling Characters are reorganized in random order, replacing the original content. For example, an ID number such as 76498 in a production database, could be replaced by 84967 in a test database. This method is very simple to implement, but can only be applied to some types of data, and is less secure.
Nulling Out Data appears missing or “null” when viewed by an unauthorized user. This makes the data less useful for development and testing purposes.
Value Variance Original data values are replaced by a function, such as the difference between the lowest and highest value in a series. For example, if a customer purchased several products, the purchase price can be replaced with a range between the highest and lowest price paid. This can provide useful data for many purposes, without disclosing the original dataset.
Data Substitution Data values are substituted with fake, but realistic, alternative values. For example, real customer names are replaced by a random selection of names from a phonebook.
Data Shuffling Similar to substitution, except data values are switched within the same dataset. Data is rearranged in each column using a random sequence; for example, switching between real customer names across multiple customer records. The output set looks like real data, but it doesn’t show the real information for each individual or data record.
Pseudonymisation According to the EU General Data Protection Regulation (GDPR), a new term has been introduced to cover processes like data masking, encryption, and hashing to protect personal data: pseudonymization. Pseudonymization, as defined in the GDPR, is any method that ensures data cannot be used for personal identification. It requires removing direct identifiers, and, preferably, avoiding multiple identifiers that, when combined, can identify a person.

Data Masking Best Practices

Determine the Project Scope In order to effectively perform data masking, companies should know what information needs to be protected, who is authorized to see it, which applications use the data, and where it resides, both in production and non-production domains. While this may seem easy on paper, due to the complexity of operations and multiple lines of business, this process may require a substantial effort and must be planned as a separate stage of the project.
Ensure Referential Integrity Referential integrity means that each “type” of information coming from a business application must be masked using the same algorithm.
Secure the Data Masking Algorithms It is critical to consider how to protect the data making algorithms, as well as alternative data sets or dictionaries used to scramble the data. Because only authorized users should have access to the real data, these algorithms should be considered extremely sensitive. If someone learns which repeatable masking algorithms are being used, they can reverse engineer large blocks of sensitive information.

source:
magedata.ai
imperva.com

Static Data Masking (SDM) definition:

“The act of permanently replacing sensitive data at rest with a realistic fictional equivalent for the purpose of protecting data from unwanted disclosure.”
Industry analysts characterize SDM as a must-have data protection layer capable of protecting large swaths of data within an organization.

How organizations use static data masking
Copying sensitive data into misconfigured or unsecured testing environments happens more frequently than organizations would like to admit.

By using SDM, organizations can provide high-quality fictional data for the development and testing of applications without disclosing sensitive information. The more realistic the SDM tool can make the sensitive data, the more effective development and testing teams can be in identifying defects earlier in their development cycle. SDM facilitates cloud adoption because DevOps workloads are among the first that organizations migrate to the cloud. Masking data on-premises prior to uploading it to the cloud reduces the risk for organizations concerned with cloud-based data disclosure.

Organizations also use SDM to anonymize data they use in analytics and training as well as to facilitate compliance with standards and regulations (such as GDPR, PCI, HIPAA, etc.) that require limits on sensitive data that reveals personally identifiable information (PII).

In practical terms, SDM makes sensitive data unsensitive because it applies data transformations as it makes a realistic looking database copy. If an attacker compromises a non-production, statically masked database, the sensitive data might look like real data, but it isn’t. SDM does not slow down or change the way an application using the data will work because it applies SDM on all data up-front, so there is no impact once the masked database is made available to the various functions. SDM dramatically simplifies the securing of non-production data, because all sensitive data has been replaced, so there is no need to implement fine-grained object-level security.

source: imperva.com

DDM, Dynamic Data Masking Definition

Applies to: SQL Server 2016 (13.x) and later Azure SQL Database Azure SQL Managed Instance Azure Synapse Analytics

source: learn.microsoft.com

What tool do we use for Data Masking?

we use Microsoft & Talend visual studio
learn.microsoft.com

help.talend.com

Other players
imperva.com
magedata.ai

Data Presentation

Presenting Data: Definition

In the field of math, data presentation is the method by which people summarize, organize and communicate information using a variety of tools, such as diagrams, distribution charts, histograms and graphs. The methods used to present mathematical data vary widely. Common presentation modes including

data catalog. A Data Catalog is a collection of metadata, combined with data management and search tools, that helps analysts and other data users to find the data that they need, serves as an inventory of available data, and provides information to evaluate fitness of data for intended uses.
data analysis,
- Descriptive analysis.
- Exploratory analysis.
- Inferential analysis.
- Predictive analysis.
- Causal analysis.
- Mechanistic analysis.
source: builtin.com
reference
The Best Data Analytics Software of 2023
- Microsoft Power BI: Best for Data Visualization
- Tableau: Best for Business Intelligence
- Qlik Sense: Best for Machine Learning
- Looker: Best for Data Exploration
- Klipfolio: Best for Instant Metrics
- Zoho Analytics: Best for Robust Insights
- Domo: Best for Streamlining Workflows
source: forbes.com

drawing diagrams,
Top 8 Diagramming Software
- Visio.
- SmartDraw.
- Gliffy.
- Creately.
- Sketch.
- Whimsical.
- Draw.io.
- Cacoo.
source: g2.com

boxplots,boxplot is a standardized way of displaying the distribution of data based on a five number summary (“minimum”, first quartile [Q1], median, third quartile [Q3] and “maximum”). Here’s an example.

Boxplots can tell you about your outliers and what their values are. It can also tell you if your data is symmetrical, how tightly your data is grouped and if and how your data is skewed.

WHAT IS A BOXPLOT?

A boxplot is a standardized way of displaying the distribution of data based on a five number summary (“minimum”, first quartile [Q1], median, third quartile [Q3] and “maximum”). It can tell you about your outliers and what their values are. Boxplots can also tell you if your data is symmetrical, how tightly your data is grouped and if and how your data is skewed.

source: builtin.com

tables, pie charts, histograms.
Data visualization

Tableau Is Business Intelligence Software that Helps People See and Understand Their Data. Your Eureka Moments are Waiting. Visualize Your Data in Minutes with Tableau. Business Analytics. Drag & Drop Reporting. Data Analysis. Business Intelligence. Data Discovery. Visual Analytics. Business Dashboards. Self-Service-BI. Mobile An

source: tableau.com

source: reference.com

Data Quality

Data Quality:Definition

Data quality is a measurement of company data that looks at metrics such as consistency, reliability, completeness and accuracy. The highest levels of data quality are achieved when data is accessible and relevant to what business users are working on.

Evaluation criteria:

Accuracy: Is the data valid? Does it possess sufficient details to be useful?
Completeness: Is all relevant data present in the data set? Is it sufficiently comprehensive? Are there any gaps or inconsistencies?
Reliability: Can the data be trusted for business decision-making? Are there any contradictions in the data set that cause you to question its reliability?
Relevance: Can the data be applied to all relevant business needs and concerns?
Timeliness: Is the data up-to-date? Can it be used to make real-time decisions?

source: techrepublic.com

Top 10 benefits of data quality

Raised data confidence
Confidence is critical in an organization but difficult to attain, especially because the people who use data to make decisions are rarely involved in its collection and preparation for consumption.
Take the example of a CEO who has to make tough decisions based on data provided by his technical teams. If the CEO has historically received inaccurate data that has led to poor business decisions, they may have misgivings about future data, hesitate to rely on it and seek to validate it.
When an organization has a robust data quality strategy and processes that everyone trusts, it gives the CEO and other decision-makers the confidence they need to rely on the data for decisions.
Better decision-making
Data quality directly impacts an organization’s bottom line because it affects the accuracy of decisions. When data is complete, accurate and timely, organizations can make sound decisions that lead to positive outcomes.
Poor data quality, on the other hand, can lead to false conclusions and bad decisions that can harm the bottom line.
As an example, if a bank or financial services company makes decisions based on incomplete or inaccurate data, they are at risk of making poor lending decisions, which can lead to defaults and losses. An insurance company that relies on flawed data to price its products can end up overcharging or undercharging customers, which can hurt the company’s reputation and competitive edge in the industry.
Increased scalability
As organizations grow, their data needs change and evolve. Good data quality is essential for ensuring that an organization’s data scales for new business use cases and opportunities.
Poor data quality can impede an organization’s ability to scale effectively and efficiently. For example, an e-commerce company that uses data to personalize the customer experience for each visitor to its website will need a robust and scalable data infrastructure to support this personalized experience at scale. If the company’s data quality is poor, it will be difficult to scale personalized experiences to a large number of visitors without major errors or workforce inefficiencies.
Improved consistency
High data quality is essential for ensuring consistency across an organization’s processes and procedures. In many companies, different people may need to see the same sales numbers but could be looking at totally different data sources for those numbers. Incongruency across systems and reporting can harm decision-making and cross-departmental initiatives. A consistent data quality strategy ensures that data is harmonious throughout the organization.
Readiness to deal with changes in the business environment
Organizations with good data quality are often better prepared to deal with changes in the business environment and can adapt to change more quickly and efficiently. Poor data quality can hamper an organization’s ability to change and adapt to new technologies, processes and operational needs, leading to stagnation and decline.
Lower costs
Higher data quality can also help to reduce costs within an organization. When data is accurate and complete, organizations spend less time and money on things like reprinting product documents or re-running reports after initial errors. Additionally, high-quality data can help organizations avoid regulatory fines or penalties for non-compliance.
Time savings
Inaccurate data can lead to operational inefficiencies and wasted time and resources, especially if certain members of your team spend all of their working hours on quality testing data.
A company that doesn’t maintain a complete dataset of its customers may, for example, send marketing materials to the wrong addresses or call customers by the wrong name. These kinds of mistakes not only annoy customers and damage a company’s reputation but also waste time and resources that could have been used more effectively.
Boosted productivity
The byproduct of lower operational costs and time savings is increased productivity. When an organization runs more efficiently, its employees can be more productive and focus on more strategic tasks rather than tactical data maintenance tasks.
Improved compliance
Organizations that maintain high data quality standards are also more likely to comply with the laws and regulations that govern their industry. This is because accurate and complete data makes it easier for organizations to meet their reporting requirements and avoid penalties for non-compliance.

source: techrepublic.com

5 tips for improving data quality for unstructured data

Use system and performance monitoring tools Data quality can only be as good as the environments where data resides. To ensure that your data platforms and storage systems are performing optimally, utilize comprehensive monitoring and alerting controls for all relevant environments. Consistent, real-time monitoring of these data-storing systems ensures the availability, reliability and security of the data assets in question. APM monitoring and data observability tools are some of the best options on the market to support this kind of data monitoring
Make data quality fixes in real time whenever possible It’s a good idea to incorporate real-time data validation and verification across your data operations. This will help you to avoid harnessing unnecessary, incomplete or incorrect information, which will detract from business efforts to obtain value from the data.
Cleanse data regularly Utilize comprehensive data cleansing and scrubbing methods to remove irrelevant, obsolete or redundant data. Removing excess data makes it much easier to sort through and assess the relevant information in your systems. It may be worth investing in a data cleansing tool that helps you to automate and simplify this process.
Research and apply new data quality management techniques It’s important to conduct routine analysis of your existing data quality improvement techniques and to look at new technologies and techniques as they emerge. Especially be on the lookout for data collection and storage improvements, developing data standards, and new governance and compliance requirements.

What is unstructured data?
Unstructured data is a heterogeneous set of different data types that are stored in native formats across multiple environments or systems. Email and instant messaging communications, Microsoft Office documents, social media and blog entries, IoT data, server logs and other “standalone” information repositories are common examples of unstructured data.

Unstructured data may sound like a complicated scattering of unrelated information, not to mention a nightmare to analyze and manage, and it does take data science expertise and specialized tools to make use of this information, but despite the complexity of working with and making sense of unstructured data, this data type offers some significant advantages to companies that learn how to use it.

How to analyze unstructured data

Before you can start analyzing your unstructured data effectively, it’s important to set goals regarding what data you want to analyze and for which intended outcomes. Depending on your business and its data goals, you may be looking at unstructured data to understand anything from customer shopping trends to seasonal real estate purchases and geographic-based spending. Knowing the type of data you want to analyze and what it needs to communicate to your users is an important first move in data quality management.
Next, you should identify where the necessary data resides, how it should be collected and analyzed, and which methodologies will work best with this data type. It’s important to ensure you have a secure and reliable method for collecting this information and feeding it into data analysis tools. Factor in mobile or portable devices and how you will need to keep them linked during the data collection process as well.
Throughout your unstructured data analysis, plan to utilize metadata — or data about data — for better performance. You should also determine whether artificial intelligence and machine learning techniques can or should come into play for automated workflows and real-time data management requirements.

source: techrepublic.com

Data Quality KPIs

Typical Data Quality by Attribute

Consumer Names
50-75% of input records do not have gender captured
20-40% of records have customer name values parsed incorrectly
Postal Address
10-40% of input addresses are invalid
Email Address
10-30% of input emails are invalid
Phone Numbers
10-30% of input phone numbers are invalid
50-90% have no phone type (mobile or land-line)
Duplication
10-40% of customer records are duplicates

Data Validation KPMs

Valid Rate (Valid=Verified/CorrectedData)
Invalid Rate (Invalid=Known/ConfirmedBadData) Don’t forget Indeterminate Data (unknown whether it is valid or invalid.
False Positive Rate
False Negative Rate

source: globalz.com

Key data quality metrics to consider

Accuracy is often considered the most critical metric for data quality. Accuracy should be measured through source documentation or independent confirmation techniques. This metric also refers to data status changes as they happen in real-time.
Different instances of the same data must be consistent across all systems where that data is stored and used. While consistency does not necessarily imply correctness, having a single source of truth for data is vital.
Incomplete information is data that fails to provide the insights necessary to draw needed business conclusions. Completeness can be measured by determining whether or not each data entry is a “full” data entry. In many cases, this is a subjective measurement that must be performed by a data professional rather than a data quality tool.
Known as data validation, data integrity ensures that data complies with business procedures and excels in structural data testing. Data transformation error rates — when data is taken from one format to another and successfully migrated — can be used to measure integrity.
Out-of-date data almost always leads to poor data quality scores. For example, leaving old client contact data without updates can significantly impact marketing campaigns and sales initiatives. Outdated data can also affect your supply chain or shipping. It’s important for all data to be updated so that it meets accessibility and availability standards.
Data may be of high quality in other ways but irrelevant to the purpose for which a company needs to use it. For example, customer data is relevant for sales but not for all top-level internal decisions. The most important way to ensure the relevancy of data is to confirm that the right people have access to the right datasets and systems.

source: techrepublic.com

The Cost of Poor Data Quality

Only 3% of Companies’ Data Meets Basic Quality Standards
On average, 47% of newly-created data records have at least one critical (e.g., work-impacting) error.
Only 3% of the DQ scores in our study can be rated “acceptable” using the loosest-possible standard.

source: hbr.org

What are the business costs or risks of poor data quality?

Poor external data quality can lead to missed opportunities, loss of revenue, reduced efficiency and neglected customer experiences.
Poor internal data quality is also responsible for ineffective supply chains—an issue that has been breaking news constantly in the past year. The same factor is one of the main drivers of the Great Resignation, as HR departments operating with poor data are challenged to understand their workers in order to retain talent.
Additionally, there are severe immediate risks that companies need to address, and they can only do that by tackling data quality. The cybersecurity and threat landscape continues to increase in size and complexity and thrives when poor data quality management policies prevail.
Companies that work with data and fail to meet data, financial and privacy regulations risk reputation damages, lawsuits, fines and other consequences linked to lack of compliance.

source: techrepublic.com

Gartner estimates that the average financial impact of poor data quality on organizations is $9.7 million annually. At the same time, IBM says that in the U.S. alone, businesses lose $3.1 trillion annually due to insufficient data quality.
Gartner explains the importance of data quality metrics well, revealing that poor data quality costs organizations an average of $12.9 million every year. Beyond revenue losses, poor data quality complicates operations and data ecosystems and leads to poor decision-making, which further affects performance and your bottom line.
McKinsey says that companies should think of data as a product, managing their data to create “data products” across the organization. When data is managed like a product, quality is guaranteed because the data is ready to use, consume and sell. The quality of this data is unique. It is verified, reliable, consistent and secure. Like a finished product your company sells, it is double-checked for quality.
Gartner explains that to address data quality issues, businesses must align data policies and quality processes with business goals and missions. Executives must understand the connection between their business priorities and the challenges they face and take on a data quality approach that solves real-world problems.
McKinsey explains that teams using data should not have to waste their time searching for it, processing it, cleaning it or making sure it is ready for use. It proposes an integral data architecture to deal with data quality and assures its model can accelerate business use cases by 90%, reduce data-associated costs by 30% and keep companies free from data governance risks.
McKinsey warns that neither the grass-root approach, in which individual teams piece together data, nor the big-bang data strategy, where a centralized team responds to all the processes, will reap good results.

source: gartner.com

Talend: The tools we use

What is Talend Data Quality?

Talend Data Quality is one of the most popular solutions for business data quality purposes. It offers a wide range of data profiling, cleaning, standardization, matching and deduplication features. The solution is part of the greater Talend Data Fabric product, with a suite of solutions that also includes data integration, data integrity and governance, and application and API integration services.

Key features of Talend Data Quality

Real-time data profiling, cleaning and masking
Talend Data Quality is a powerful data quality management tool that uses machine learning to automatically profile data in real time. Its machine learning algorithm examines the data as it flows through your systems and identifies anomalies that suggest the need for data cleansing or masking. Talend’s machine learning algorithm also makes recommendations for addressing data quality issues, such as identifying invalid or duplicate records.
In addition, the tool uses machine learning-enabled deduplication, validation and standardization to cleanse incoming data so business analysts can focus on other core business tasks.
This feature ensures that data is of the highest quality, and Talend’s recommendations help you address any data quality issues that may arise.

Self-service interface
Summary statistics and visualizations
Talend Trust Score
Robust security and compliance

Pros of Talend Data Quality

Good market reputation
Gentle learning curve
Flexible deployment
Integrations
Talend Open Studio
Support and services

Cons of Talend Data Quality

There are customer complaints about Talend Data Quality processing speeds. These concerns mostly stem from slow speeds during data profiling and cleansing on large data sets.
In addition, there is room for improvement with nested options, or features nested within each other. Currently, users must follow a tedious series of steps every time they want to create new data cleansing rules.

source: techrepublic.com

Data Integration

Data Integration: Definition

What is data integration?
Data integration is a framework that combines data from siloed sources to provide users with a unified vision. Data integration benefits include better data governance and quality, increased visualization, better decision-making, and better performance. Standardizing data is essential for data integration to be successful, as multiple teams — some of which may not have advanced IT technical knowledge and skills — need to access and use the data system.

When combined with tools like machine learning and predictive analysis, unified data insights can significantly impact a company’s operations, allowing it to detect risks in advance, meet compliance across the board, boost sales and detect new growth opportunities. Data integration aims to create a single access point of data storage that is available and has good quality standards. But to move data from one system to another and meet big data challenges with excellence requires a data integration strategy.

source: techrepublic.com

Benefits of Data Integration

1) Make Smarter Business Decisions
2) Deliver Improved Customer Experiences
3) Cost Reduction
4) Increased Revenue Potential
5) Increased Innovation
6) Improved Security
7) Improved Collaboration

Data Integration Use-cases in Real World
1) Retail
Data integration can be used to track and manage inventory levels across multiple retail locations or channels, such as online and brick-and-mortar stores. This can help retailers ensure that they have the right products in stock at the right time and place, reducing the risk of lost sales due to out-of-stock items.

2) Marketing
Data integration can be used to combine customer data from various sources, such as social media interactions, website activity, and email campaigns, to create more detailed and accurate customer profiles. This can help marketers segment their audience more effectively and target their campaigns to specific groups of customers.

3) Finance
Data integration can help financial institutions improve risk management, fraud detection, and compliance efforts. Accordingly, this can help financial institutions identify new business opportunities and optimize their products and services according to the pricing plans.

4) Telecommunications
By integrating data from various sources, such as customer interactions, demographic information, and usage data, telecommunications companies can gain a 360-degree understanding of deeper understanding of their customers and their needs. This can help them to tailor their products and services to better meet the needs of their customers, leading to improved customer satisfaction.

5) Rating Services
Data integration can help rating services improve the accuracy and timeliness of their ratings by integrating data from multiple sources. For example, by integrating data from financial statements, market data, and news articles, rating services can gain a more complete view of a company’s financial performance and risk profile, which can help them provide more accurate and timely ratings.

6) Healthcare
Integrating patient data from various sources can help healthcare providers make more informed treatment decisions and improve patient care and outcomes. By having access to a more comprehensive view of a patient’s medical history, including past treatments and diagnoses, allergies, and medications, healthcare providers can make more informed treatment decisions and avoid potential adverse reactions or complications.

source: hevodata.com

What happen without data Integration?

With massive amounts of data being generated, the rapid pace of tech innovation, the costs of change, growing sprawl of application and data silos, and a plethora of available data management and analytics tools to choose from – it’s easy to see why so many businesses wrestle with trying to effectively manage and glean real value from their data.

Data that is not integrated remains siloed in the places it resides. It takes a lot of time and effort to write code and manually gather and integrate data from each system or application, copy the data, reformat it, cleanse it, and then ultimately analyze it. Because it takes so long to do this, and skilled resources in IT who can do it are scarce, the data itself may easily be outdated and rendered useless by the time the analysis is complete. Businesses don’t have time to wait anymore.

source: snaplogic.com

Types of data integration

Data migration
Data replication involves replicating data from where it is generated – such as a POS system or a warehouse inventory record from a particular region – to where it needs to be analyzed for planning, forecasting, and insights.
There are different types of data replication, such as:
- Bring it closer to other data assets that are similar so that they can be combined to get useful insights y Reduce the cost of data storage
- Improve data access performance
- Improve data availability

Data replication
Data replication involves replicating data from where it is generated – such as a POS system or a warehouse inventory record from a particular region – to where it needs to be analyzed for planning, forecasting, and insights. There are different types of data replication, such as:
- A full table replication copies data from source table to the destination in its entirety. It is time- consuming and requires significant network bandwidth
- Incremental replication, which can be key-based or log-based, identifies changes in the source data and propagates them to the destination

Data synchronization Data synchronization is the process of synchronizing similar objects or data structures (schemas) across different data stores or applications. There are two ways of looking at data synchronization:
- Incremental data replication can be viewed as data synchronization from a source to a destination
- Data synchronization can also be between two different applications – for example, a CRM system (such as Salesforce, Microsoft Dynamics CRM) and a Service Management system (such as ServiceNow, Zendesk) both hold important customer records. But the data in the CRM system will often be viewed as the master record. In that case, customer details in a Service Management system need to synchronized periodically with the CRM system

Data sharing across enterprises Organizations deal with many external entities such as suppliers, distributors, customers, and partners as part of normal business operations. Data sharing across enterprises include systems such as Electronic Data Interchange (EDI) that enable business partners to agree on data formats, acquire data, exchange messages, collaborate, and execute end-to-end business processes such as catalog discovery, procure-to- pay, order-to-cash, transportation of goods, and more. Effective data sharing across enterprises provides the following benefits:
- Reduces time-to-market
- Reduces manual errors and improves productivity
- Improves revenue and earnings by selling goods and services through more channels and partners

Data transformation Data integration tools are often known as Extract, Transform, and Load (ETL) tools, and the key functionality they provide is the ability to transform data. Data transformation includes but is not limited to changing data formats, combining data across multiple data sources, filtering or excluding certain data entries from the combined data set, summarizing values across data sets, and so on. Data transformation can be done using code, SQL scripts, or visually. Data transformations are so fundamental to any data integration flow that the ability to transform data with ease is a crucial differentiator for any data integration tool.

Data governance and management
Data governance and management is a broad capability that consists of the ability to audit, control access to, profile, govern, share, and monitor data. It encompasses areas such as:
- Data Catalogs allow organizations to create an orderly list of data assets in an organization. Data catalogs use metadata associated with data assets to uncover context with various repositories of data. This metadata is then used for data discovery and to uncover data relationships
- Data Virtualization allows users and applications to access and manipulate data without any knowledge of how the data is structured, or where it is located
- Data governance tools enable organizations to:
- Build trust in data
Maintain data privacy by controlling access
Provide audit capability so that organizations can proactively comply with regulations
Allow collaboration between users so that collectively they can make the most of the data

Enterprise application integration
What is enterprise application integration (EAI)? Just like the name suggests, EAI is all about creating interoperability among applications and systems. This is where you get Salesforce, Workday, Microsoft Dynamics, ServiceNow, and NetSuite to play nice with each other. Traditionally, organizations managed their EAI tools distinct from their data integration tools. But now organizations are increasingly using a single unified platform for both application and data integration. EAI is critical to creating omnichannel customer experiences, streamlining workflows and processes, and creating seamless experiences for customers, partners, and employees.
These are the most common aspects of data integration. Next, we turn to the role artificial intelligence and machine learning play in data integration.

source: snaplogic.com

Data Integration methods

So, how do you do data integration? There are several approaches ranging from manual integration to low- code data integration platforms:

- Code it manually. This is a time-consuming and resource-intensive method where integrations are manually coded from a source to a destination and must be monitored and continually maintained by IT
- Use middleware. Middleware data integration serves as a mediator between data that needs to be normalized
- Let an integration platform as a service (iPaaS) simplify it for you. An iPaaS, such as the SnapLogic Intelligent Integration Platform, provides out-of-the-box connectivity to thousands of data and application endpoints, simplifies data transformations, and makes it easy to manage and govern that data

source: snaplogic.com

Where do you start with a data integration strategy?

Developing a strategy for integrating data across your organization helps ensure that everyone has access to the most up-to-date data in a secure way. This article provides an example of a strategy you can use to develop your own.

The first step in data integration is not to acquire the tools and tech from vendors but to plan the company’s strategy. Data integration is not about data and technology — these are just tools that serve a purpose. Data integration is about collaboration between people, teams and your entire workforce.

Every company has its objectives and goals and must understand which data will help them achieve them. Aligned with a company’s mission, values and data governance strategy, data leaders must lead the data integration strategy.

Once organizations have answered what business goals their data integration needs to support, they can turn to other questions. Access and availability need to be clear and transparent. While executives and critical stakeholders might need full access and visibility on all unified data, other departments require restricted access. Additionally, roles and responsibilities should also be set.

Ideally, organizations should aim to integrate independent systems into one master data warehouse. In order to accomplish this task, leaders need to ask what data needs to be place: on the cloud, on-premises or hybrid. integrated, who will make up the data integration team and where will the data integration take place: on the cloud, on-premises or hybrid.

sources & references: techrepublic.com

Data integration vs ETL: What are the differences?

If you’re considering using a data integration platform to build your ETL process, you may be confused by the terms data integration vs. ETL. Here’s what you need to know about these two processes.
Businesses have a wealth of data at their disposal, but it is often spread out among different systems. This scenario makes it challenging to get a clear picture of what’s happening in the business.

That’s where data integration and ETL — or Extract, Transform and Load — come in to support greater data visibility and usability. Although these two concepts are closely related, data integration and ETL serve distinct purposes in the data management lifecycle.
How are data integration and ETL similar?
However, it’s important to note that not all data integration solutions use ETL tools or concepts. In some cases, it’s possible to use alternative methods such as data replication, data virtualization, application programming interfaces or web services to combine data from multiple sources. It all depends on the specific needs of the organization if ETL will be the most useful form of data integration or not.
How are data integration and ETL different?
The main difference between data integration and ETL is that data integration is a broader process. It can be used for more than just moving data from one system to another. It often includes:
Data quality: Ensuring the data is accurate, complete and timely.
Defining master reference data: Creating a single source of truth for things like product names and codes and customer IDs. This gives context to business transactions.

source: techrepublic.com

The future of data integration

In the past, data integration was primarily done using ETL tools. But, in recent years, the rise of big data has led to a shift towards ELT — extract, load and transform tools. ELT is a shorter workflow that is more analyst-centric and that can be implemented using scalable, multicloud data integration solutions.

These solutions have distinct advantages over ETL tools. Third-party providers can produce general extract-and-load solutions for all users; data engineers are relieved of time-consuming, complicated and problematic projects; and when you combine ETL with other cloud-based business applications, there is broader access to common analytics sets across the entire organization.

In the age of big data, data integration needs to be scalable and compatible with multicloud. Managed services are also becoming the standard for data integration, because they provide the flexibility and scalability that organizations need to keep up with changing big data use cases. Regardless of how you approach your data integration strategy, make sure you have capable ETL/data warehouse developers and other data professionals on staff who can use data integration and ETL tools effectively.

Data integration trends to watch in 2022
The proliferation of remote and online work maximizes workforce potential for companies but spreads data thin from platform to platform. From customer relationship management software to cloud services, data for your business could be hosted in multiple locations, leading to disorganization, data set errors and poor decision-making.

Companies are increasingly recognizing the problems that come with disparate data platforms and are leaning into data integration solutions. In this article, we cover some of the top data integration trends we’re seeing today and where they could lead in the future.

Getting started with an integration platform as a service solution
Leveraging real-time integrations
Enabling machine learning and AI
Eliminating data silos
Establishing a metadata management strategy

source: techrepublic.com

Top Data Integration Tools

talend.com

Talend helps organizations deliver healthy data to correctly inform their decision-making. The company provides a unified platform to support data needs without limits in scale or complexity. With Talend, organizations can execute workloads seamlessly across cloud providers or with on-premises data. Talend’s data integration solutions enable users to connect all their data sources into a clean, comprehensive and compliant source of truth.

Key differentiators

1,000+ connectors and components
Flexibility and intuitiveness
Embedded data quality
Management of larger data sets

Con: Talend may suffer from performance and memory management issues.

hevodata.com

Hevo is an end-to-end data pipeline platform that enables users to effortlessly leverage data. Hevo can pull data from multiple sources into warehouses, carry out transformations and offer operational intelligence to business tools. It is purpose-built for the ETL, ELT and Reverse ETL needs of today and helps data teams to streamline and automate data flows. Common benefits of working with Hevo include hours of time saved per week, accelerated reporting, and optimized analytics and decision-making.

Key differentiators

No-code user interface
Fault-tolerant architecture
Automated pipelines

Con: The tool could benefit from more detailed documentation for first-time users to smoothen the learning curve.

informatica.com

Informatica Cloud Data Integration provides a fast and reliable way to integrate and deliver data and analytics to businesses. It is an intelligent data platform that continuously assesses the performance of processing engines and workload variations, all while enabling users to identify the correct data integration pattern for their use cases. With Informatica Cloud Data Integration, users can connect hundreds of applications and data sources on-premises and integrate data sources at scale in the cloud.

Key differentiators

Rich set of connectors for all major data sources
Advanced data transformation functionality
Codeless integration

Con: Users cannot store metadata locally with Informatica Cloud Data Integration.

oracle.com

Oracle Data Integrator is a thorough data integration platform that covers data integration requirements like high-volume, high-performance batch loads; SOA-enabled data services; and event-driven, trickle-feed integration processes. Oracle also provides advanced data integration capabilities to users that seek to implement seamless and responsive Big Data Management platforms through Oracle Data Integrator for Big Data.

Key differentiators

Out-of-the-box integrations
Heterogeneous support
Knowledge module framework for extensibility
Rich ETL

Con: Oracle’s solution involves a complex development experience in comparison to competitors.

dataddo.com

Dataddo is a no-code platform for data integration, transformation and automation that seeks to provide users with complete control and access to their data. The platform works with many online data services, including existing data architectures users already have. Dataddo syncs dashboarding applications, data lakes, data warehouses and cloud-based services. It also visualizes, centralizes, distributes and activates data by automating its transfer from source to destination.

Key differentiators

Data to dashboards
Data anywhere
Headless data integration

Con: The platform can be quite confusing to new users.

integrate.io

Integrate.io is a low-code data warehouse integration platform that supports informed decision-making for data-driven growth. Its platform offers organizations capabilities to integrate, process and prepare data for analytics on the cloud. Integrate.io’s platform is scalable to make sure that organizations can make the most of big data opportunities without a hefty investment in software, hardware and staff. The platform gives companies the chance to enjoy instant connectivity to multiple data sources and a rich set of out-of-the-box data transformations.

Key differentiators

ETL and reverse ETL
ELT and CDC
API generation

Con: The tool could benefit from more advanced features and customization.

iri.com

IRI Voracity is an end-to-end data lifecycle management platform that leverages technology to tackle speed, complexity and cost issues in the data integration market. It is an integration platform as a service (IPaaS) data integration tool that is ideal for quick and affordable ETL operations. IRI Voracity also offers data quality, masking, federation and MDM integrations.

Key differentiators

Multiple source connections
Data mapping
Hadoop transforms

Con: The IRI product names may be confusing to users.

zigiwave.com
Zigiwave is a company that aims to disrupt the integrations industry, particularly where connections between applications are characterized by lines of code and a lot of time invested. Zigiwave’s product, ZigiOps, is a highly flexible no-code integration platform that creates powerful integrations in a handful of minutes. ZigiOps empowers non-technical users to carry out integration tasks in a few clicks without having to add scripts.

Key differentiators

No code integrations
Flexibility and scalability
Integration templates
Recovery and data security features

Con: Since the company is based in Europe, the support time coverage may prove to be a challenge for users outside of Europe.

source: techrepublic.com

We use talend.com

Data Lakes

Data Lake: Definition

A data lake is a set of unstructured information that you assemble for analysis. Deciding which information to put in the lake, how to store it, and what to make of it are the hard parts.
The concept of a data lake is perhaps the most challenging aspect of information management to understand. A data lake can be thought of not as something you buy, but as something you do. “Data lake” sounds like a noun, but it works like a verb.

James Dixon, chief technology officer of Hitachi-owned Pentaho, is credited with coining the term data lake in 2008. Dixon said he was looking for a way to explain unstructured data.

Data mart and data warehouse were existing terms; the former is generally defined as a department-level concept where information is actually used, and the latter is more of a storage concept. He began to think about metaphors with water: thirsty people get bottles from a mart, the mart gets cases from a warehouse, and the warehouse obtains and bottles it from the wild source — the lake.

source: techrepublic.com

Who does Data Lakes affect?

Nik Rouda said the most common mistake in data lake projects is that companies don’t have the right people to manage it. Database administrators may not understand how to apply their knowledge to unstructured information, while storage managers typically focus on nuts and bolts. The people most affected by a data lake are probably those who pull the purse strings, because a company will need to budget for hiring analytic experts or outsourcing that job to a professional services organization.

source: techrepublic.com

Data Migration

Data Migration: Definition

Moving data from one location to another is the simple concept behind data migration. It is described as a shift of data from one system to another, characterized by a change in database, application or storage. Data migration may result from a need to modernize databases, build new data warehouses and/or merge new data from sources, among other reasons.

Key features for data migration

Functionality: The functionality of a tool should involve plans, scheduling jobs, organizing workflows, data mapping and profiling, ETL tools and post-migration audits.
Handling of data sources and target systems: A data migration tool should be compatible with a user’s desired data source or data type.
Performance and flexibility: A good data migration tool can transfer data in a short time frame without compromising data quality. Cloud tools offer greater flexibility and scalability than on-premises tools in this area, as on-premises tools are subject to hardware parameters.
Intuitiveness and ease of use: Intuitive and easy-to-use solutions save time for users. Users should consider solutions that are not only intuitive and easy to use but also backed up with exceptional technical support.

Popular data migration tools include:

source: techrepublic.com

we use talend.com

top 5 data migration trends

Shifting toward data lakehouses
Arguably one of the biggest innovations in data migration has been the data lakehouse, first introduced by Databricks, which largely does away with the need to migrate data. Rather than moving data from a data lake to a data warehouse with expensive and time- basically transforms a data lake into a data warehouse. consuming extract, transform and load tools, the data lakehouse As Databricks executive, David Meyer, explained in an interview: “Data lakes were great in a lot of ways … but they didn’t have a lot of characteristics that you’d want to do data and AI at scale.” He went on to describe some of the weaknesses of data lakes, including their lack of governance, ACID compliance and transactional features. By adding a layer like the open source Delta Lake that Databricks uses, companies can leverage massive quantities of data for things like machine learning applications without necessarily having to move or migrate that data.
Avoiding data loss and expanding capacity through cloud migration
This second trend is far more obvious because, quite literally, almost everybody’s doing it. That is migrating data to the cloud. Though cloud spending remains relatively small compared to the overall IT market — less than 10%, according to IDC and Gartner data — it’s growing much faster than other areas.
By moving data to the cloud, enterprises can not only handle a wider array of data types, but they can also ingest more data much faster. They also get the benefit of seemingly infinite capacity, something they definitely don’t have with on-premises deployments, where the surge in data is maxing out companies’ ability to store it all.
Fortunately, each of the major cloud providers offers a variety of services to enable more seamless data migration. There are also a host of system integrators with expertise in assisting companies as they move their data to different cloud providers’ storage, database and related systems. It’s never been easier to migrate data to the cloud.
Uniting legacy on-premises data with cloud customer data
Part of the drive to migrate data to the cloud is that so much new data already lives there, and that data is in fact born in the cloud. This isn’t so much a trend in data migration as it is the reason for data migration.
Indeed, much of a company’s most valuable data, at least as it relates to customers, is cloud data, which has sparked data migration projects to move more hitherto on-premises data to those same cloud environments. This includes migrating data lakes and data lakehouses to the cloud. Through this process, organizations develop a richer, more holistic view of their customer data.
Using data migration resources to get the most out of unstructured data
An increasing percentage of the customer-related data mentioned in Trend #3 is semi-structured or unstructured data. Examples of these types of data include geospatial, sensor and social media data.
This data doesn’t easily fit within a relational database, and increasingly finds a home in so-called NoSQL databases. Whether unstructured data is stored in a NoSQL database, a data lake or elsewhere, enterprises are looking to data migration strategies and tools to move, cleanse and transform this data to make it easier to analyze.
Migration begets modernization
When companies begin to think about data migration, application modernization often isn’t far behind. If a company is considering moving data to the cloud, rather than starting with a basic lift- and-shift approach, many companies are now starting the process by moving from a self- managed, on-premises database to a fully managed database service. For example, these organizations may move from self-hosting MySQL to running Amazon RDS for MySQL.
The pre-migration stage is also when enterprises are increasingly choosing to rearchitect an application to use an entirely different database. Perhaps they are switching from a relational database to a document or key-value store, or perhaps they are moving in the opposite direction if they are starting with a wide-column database for a particular application and think a relational approach would be a better fit.
The minute you start to think about migrating data, it’s worth considering if it’s time to make other major changes to your data storage, management and infrastructure. You may also want to consider hiring data professionals who specialize in this kind of data modernization and migration work.

source: techrepublic.com

Etc

Data Consistency

Data Consistency: Definition

Data consistency means that each user sees a consistent view of the data, including visible changes made by the user’s own transactions and transactions of other users.

What is data consistency?Data consistency is one of ten dimensions of data quality. Data is considered consistent if two or more values in different locations are identical. Ask yourself: Is the data internally consistent? If there are redundant data values, do they have the same value? Or, if values are aggregations of each other, are the values consistent with each other?

What are some examples of inconsistent data?Imagine you’re a lead analytics engineer at Rainforest, an ecommerce company that sells hydroponic aquariums to high-end restaurants. Your data would be considered inconsistent if the engineering team records aquarium models that don’t match the models recorded by the sales team. Another example would be if the monthly profit number is not consistent with the monthly revenue and cost numbers.

How do you measure data consistency?
To test your any data quality dimension, you must measure, track, and assess a relevant data quality metric. In the case of data consistency, you can measure the number of passed checks to track the uniqueness of values, uniqueness of entities, corroboration within the system, or whether referential integrity is maintained. Codd’s Referential Integrity constraint is one example of a consistency check.

How to ensure data consistency
One way to ensure data consistency is through anomaly detection, sometimes called outlier analysis, which helps you to identify unexpected values or events in a data set.

Using the example of two numbers that are inconsistent with one another, anomaly detection software would notify you instantly when data you expect to match doesn’t. The software knows it’s unusual because its machine learning model learns from your historical metadata.

source: metaplane.dev

Data Encryption

Data Encryption Best practices

Introduction
With data breaches on the rise, encryption has never been more important for protecting companies against hackers and cyberattacks.
In a poll of 1,000 business professionals and software developers, nearly 45% say their company Nasdaq show that the number of data breaches grew by more than 68% in 2021, and this number is bound to grow.

Data encryption best practices

Build a unified data security policy
Implement access control
Use an identity and access management solution

Zero trust is a framework for securing infrastructure and data. The security framework assumes that the organization’s network is always at risk so it requires that all users — whether within or outside an organization — be authorized and authenticated before they are granted access to data and applications.

source: techrepublic.com

Data Estate

Data Estate Definition

“A data estate is simply the infrastructure to help companies systematically manage all of their corporate data. A data estate can be developed on-premises, in the cloud or a combination of both (hybrid). From here, organizations can store, manage and leverage their analytics data, business applications, social data, customer relationship systems, functional business and departmental data, internet of things (IoT) and more.”

source: Forbes states in the article ‘ Why The Modern-Day Corporation Should Consider A Data Estate

forbes.com

What is data estate migration and modernization?
A data estate refers to all the data an organization owns, regardless of where it is stored.

source: learn.microsoft.com

8 Strategic Steps To Building A Modern Data Estate
This tells us that the abilities of an organization towards capturing data, data storage, data analysis and searching, sharing, transferring, visualizing, querying, updating of data, as well as compliance and data privacy are no longer a want-to-have.

From A Data Warehouse To A Data Estate

Define your goal in terms of data and analytics maturity
Define the business needs of today and the future
Describe the core business and data processes
How will the data be accessed and by whom?
Define your architectur
Data lake
Data warehouse
Data mart
Cloud, on premise or hybrid
Selecting your construction partnersData management and automation software
Deployment and maintenance partner
Think big, start small and act agile

source: timextender.com

link & references: learn.microsoft.com

Data Fabric

Data Fabric Definition

“A Data Fabric orchestrates disparate data sources intelligently and securely in a self-service manner, leveraging data platforms such as data lakes, data warehouses, NoSQL, translytical, and others to deliver a unified, trusted, and comprehensive view of customer and business data across the enterprise to support applications and insights.”

Properties

A modern Data Fabric comprises multiple layers that work together to meet these needs:

Data management
Data ingestion and streaming
Data processing and persistence
Data orchestration
Data discovery
Global data access

source: cloudera.com

Data Governance

Data Governance: Definition

Data governance is a data management discipline. It ensures that the data managed by an organization is available, usable, consistent, trusted and secure. In a majority of companies, IT is the principal steward of data. It is responsible for data governance. But do companies understand the full meaning of data governance? The answer is no.

How to best utilize data governance

Ensure data availability
Keep the data usable
Maintain data consistency
Clean the data
Bolster data security

What data governance isn’t

Data governance is not data architecture or data management
Data governance is not just IT’s responsibility

Data governance best practices

Think big picture, but start small
Build a solid business case
Select and focus on the right metrics
Engage in transparent communication about data roles and responsibilities

How to implement data governance best practices in your organization

Evaluating past data management practices
Finding the right data professionals and tools for success

source: techrepublic.com

Data Gravity

Data Gravity Definition

Data gravity is the observed characteristic of large datasets that describes their tendency to attract smaller datasets, as well as relevant services and applications. It also speaks to the difficulty of moving a large, “heavy” dataset.

Think of a large body of data, such as a data lake, as a planet, and services and applications being moons. The larger the data becomes, the greater its gravity. The greater the gravity, the more satellites (services, applications, and data) the data will pull into its orbit.

Large datasets are attractive because of the diversity of data available. They are also attractive (i.e. have gravity) because the technologies used to store such large datasets — such as cloud services — are available with various configurations that allow for more choices on how data is processed and used.

The concept of data gravity is also used to indicate the size of a dataset and discuss its relative permanence. Large datasets are “heavy,” and difficult to move. This has implications for how the data can be used and what kind of resources would be required to merge or migrate it.

As business data continues to become an ever increasing commodity, it is essential that data gravity be taken into consideration when designing solutions that will use that data. One must consider not only current data gravity, but its potential growth. Data gravity will only increase over time, and in turn will attract more applications and services.

source: talend.com

How data gravity affects the enterprise

Data must be managed effectively to ensure that the information it is providing is accurate, up-to-date, and useful. Data gravity comes into play with any body of data, and as a part of data management and governance, the enterprise must take the data’s influence into account.

Without proper policies, procedures, and rules of engagement, the sheer amount of data in a warehouse, lake, or other dataset can become overwhelming. Worse yet, it can become underutilized. Application owners may revert to using only the data they own to make decisions, leading to incongruous decisions made about a single, multi-owned application.

Data integration is greatly affected by the idea of data gravity — especially the drive to unify systems and decrease the resources wasted by errors or the need to rework solutions. Placing data in one central arena means that data gravity will not collect slowly over time, but rather increase significantly in a short time.

Understanding how the new data gravity will affect the enterprise will ensure that contingencies are in place to handle the data’s rapidly increasing influence on the system. For example, consider how data gravity affects data analysis. Moving massive datasets into analytic clusters is an ineffective — not to mention expensive — process. The enterprise will need to develop better storage optimization that allows for greater data maneuverability.

The problem with data gravity
Data gravity presents data managers with two issues: latency and data non-portability.

Latency. By its very nature, a large dataset requires the applications that use it to be close, in its orbit, or suffer latency. This is because the closer the applications are to the data, the better the workload performance.
Speed is critical to successful business operations, and increasing latency as the data’s gravity increases is simply not an option. The enterprise will need to ensure that throughput and workload balance grows with the data’s gravity. This means moving applications to the same arena as the data in order to prevent latency and increase throughput. A good example of how to combat the latency issue is Amazon QuickSight; it was developed to rest directly on cloud data to optimize performance.

Non-portability. Data gravity increases with the size of the dataset, and the larger the dataset, the more difficult the dataset is to move. After all, moving a planet would be quite a feat. Moving vast quantities of data is slow and ties up resources in the process.
Data gravity has to be taken into account any time the data needs to be migrated. Due to the dataset’s continual growth, the enterprise would need to develop their migration plans based on requirements that account for the size of the dataset as it will be, rather than its actual, current size.
Data gravity is the likelihood of how many services, applications, and/or additional data will be attracted to the dataset, and should be considered when determining future size. Migration will require a specialized, often creative, plan in order to be successful.

source: talend.com

Data Integrity

Data Integrity Definition

Data integrity is the consistency and correctness of data across its entire life cycle. Learn more about data integrity with the help of this guide.
Clean, healthy data can be a major competitive advantage, especially for businesses that invest the appropriate time and resources into their data management strategies. In the age of Big Data, organizations that harness data effectively and promote data integrity can make better data-driven decisions, improve data quality, and reduce the risk of data loss or corruption.

What is data integrity?
At its most basic level, data integrity is the accuracy and consistency of data across its entire life cycle, from when it is captured and stored to when it is processed, analyzed and used.

Data integrity management means ensuring data is complete and accurate, free from errors or anomalies that could compromise data quality.

Data that has been accurately and consistently recorded and stored will retain its integrity, while data that has been distorted or corrupted cannot be trusted or relied upon for business use.

source: techrepublic.com

Why is data integrity important?

Data integrity is important for a number of reasons. However, its importance is best explained with a practical example.

Imagine you are a project manager who is running clinical trials for a new revolutionary drug that will be a game changer in the fight against cancer. You have conducted human trials over the past five years and are convinced you’re ready to move into production.

However, while going through regulatory protocols with the FDA to get your drug to market, they find data integrity issues within the data from your trials — some crucial quality control data is missing.

source: techrepublic.com

The risks associated with data integrity

Data integrity is a complex and multifaceted issue. Data professionals must be vigilant about the various risks that can compromise data integrity and quality. These include the following:

Human error
Misconfigurations and security errors
Compromised hardware
Unintended transfer errors
Malware, insider threats and cyberattacks

source: techrepublic.com

Types of data integrity

Physical integrity
Logical integrity
Entity integrity
Referential integrity
Domain integrity
User-defined integrity

source: techrepublic.com

Data Lab

Data Lab: Definition

A data lab is a designated data science system that is intended to uncover all that your data has to offer. As a space that facilitates data science and accelerates data experimentation, data labs uncover which questions businesses should ask, then help to find the answer.

The business value of a data lab
Data labs offer advantages that can improve operations and uncover valuable business information. These advantages include:

Managing multiple data science projects: Since data labs are designated systems separate from a data lake, center, or warehouse, they have a larger capacity for maintaining various data projects at once.
Data labs, especially when used in conjunction with the cloud, allow for easy scaling up or down depending on the needs of the company. So all projects get done, no matter the workload.
Generating reliable, refined outputs: Executives can enjoy greater confidence in business intelligence with the sophistication offered by a data lab’s outputs. Data labs create processes that generate stable outputs to better inform business decisions.
Positioning the business as a thought leader in its industry: Data labs yield innovative data insights that help to push the envelope in developing business sectors and that keep companies on the cutting edge in their fields. With a data lab, you can tackle the large problems and come up with unprecedented solutions to drive your business to the top.

Though the distinct business advantages will vary between companies, it’s clear that data labs add certain business value to operations as a whole.

source: talend.com

Data Matching

Data matching Definition

Data matching is the process of comparing data values and calculating the degree to which they are similar. This process is helpful in eliminating record duplicates that usually form over time, especially in databases that do not contain unique identifiers or appropriate primary and foreign keys.

In such cases, a combination of non-unique attributes (such as last name, company name, or street address) is used to match data and find the probability of two records being similar.

Benefits

Execute custom data matching
Weigh in the nature of your data and choose the right matching fields, algorithms and confidence levels to attain the best match results.
Reduce computational complexity
Eliminate duplicate records present in databases and free up storage space to attain quick and timely query results
Increase operational efficiency
Reduce manual labor, level up data quality, and optimize business processes with automatic data matching technology.
Facilitate any use case
Whether you want to clean mailing lists, detect fraudulent behavior, or match patient records, data matching software can help you out.
Ensure data compliance
Ensure that the records in your databases follow data compliance standards, such as GDPR, HIPAA, CCPA, etc.
Enrich data for deeper insights
Efficiently match organizational data present at different data stores and determine the next best move for your business.

source: dataladder.com

Data Modeling

What does a data modeler do?

Data modeling is the process of creating and using a data model to represent and store data. A data model is a representation — in diagrammatic or tabular form — of the entities that are involved in some aspect of an application, the relationships between those entities and their attributes.

The three most common types of data models are.

Relational model
The most popular database model format is relational, which stores data in fixed-format records and organizes it into tables with rows and columns. The most basic data model has two components: measures and dimensions. Raw data can be a measure or a dimension.
- Measures: These numerical values are used in mathematical calculations, such as sum or average.
- Dimensions: Text or numerical values. They aren’t used in calculations and include locations or descriptions.
In relational database design, “relations,” “attributes,” “tuples” and “domains” are some of the most frequently used terms. Additional terms and structural criteria also define a relational database, but the significance of relationships within that structure is what matters. Key data elements (or keys) connect tables and data sets together. Explicit relationships such as parent-child or one-to-one/many connections can also be established.
Dimensional model
A dimensional model is a type of data model that is less rigid and structured than other types of models. It is best for a contextual data structure that is more related to the business use or context. Dimensional models are optimized for online queries and data warehousing tools.
Crucial data points, such as transaction quantity, are called “facts.” Alongside these facts are reference pieces of information known as “dimensions,” which can include things like product ID, unit price and transaction price.
A fact table is a dimensional model’s primary table. Retrieval can be quick and effective because data for a specific activity is kept together. However, the absence of linkages can make analytical retrieval and data usage difficult.
Entity-relationship (ER) model
The entity-relationship model is a graphical representation of a business’s data structure. It contains boxes with various shapes and lines to represent activities, functions or “entities” and associations, dependencies or “relationships,” respectively.
The ER model provides a framework for understanding, analyzing and designing databases. This type of data model is used most often to design relational databases.
In an ER diagram, entities are represented by rectangles, and relationships are represented by diamonds. An entity is anything that can be identified as distinct from other things. A relationship is an association between two or more entities. Attributes are the properties or characteristics of an entity or a relationship.
ER diagrams can be categorized into three types: One-to-one, one-to-many, and many-to-many relationships.
- One-to-one relationship: An example of a one-to-one relationship would be a Social Security Number (SSN) and a person. Each SSN can only be assigned to one person and each person can only have one SSN.
- One-to-many relationship: An example of a one-to-many relationship would be a company and employees. A company can have many employees, but each employee typically only works for one company.
- Many-to-many relationship: An example of a many-to-many relationship would be students and classes. A student can take many classes and a class can have many students enrolled in it.

source: techrepublic.com

Top data modeling tools of 2022

IDERA ER/Studio is a data modeling software suite for business analysts, architects and developers. It allows them to create data models for various applications and provides several components such as business data objects, shapes, text blocks and data dictionary tables. IDERA ER/Studio is an intuitive tool that is capable of easily integrating different enterprise systems, giving users full control over their data management process.
erwin Data Modeler by Quest is a cloud-based enterprise data modeling tool for finding, visualizing, designing, deploying and standardizing enterprise data assets. It provides logical and physical modeling and schema engineering features to assist with the modeling process.
erwin is a complete solution for modeling complex data and has an easy drag-and-drop interface for creating and modifying structures, tables and relationships. In addition, this tool provides centralized management dashboards for administrators to view conceptual, logical and physical models.

IBM InfoSphere Data Architect is a data modeling tool that supports business intelligence, analytics, master data management and service-oriented architecture initiatives.
This tool allows users to align processes, services, applications and data architectures.

Moon Modeler is a data modeling solution for visualizing MongoDB and Mongoose ODM objects. It also supports MariaDB, PostgreSQL and GraphQL. This tool allows users to draw diagrams, reverse engineer, create reports and generate scripts to map object types to the appropriate databases in the right format.
DbSchema Pro is an all-in-one database modeling solution that allows you to easily design, visualize and maintain your databases. It has many features to help you manage and optimize your data, including a graphical query builder, schema comparer, schema documentation, schema synchronization and data explorer. It can be used with many relational and NoSQL databases like MongoDB, MySQL, PostgreSQL, SQLite, Microsoft SQL Server and MariaDB.
Oracle SQL Developer Data Modeler is a free graphical tool that enables users to create data models with an intuitive drag-and-drop interface. It can create, browse and edit logical, relational, physical, multi-dimensional and data-type models. As a result, the software streamlines the data modeling development process and improves collaboration between data architects, database administrators, application developers and end users.
Archi (Archimate modeling is an open-source solution for analyzing, describing and visualizing architecture within and across various industries. It’s hosted by The Open Group and aligns with TOGAF. The tool is designed for enterprise architects, modelers and associated stakeholders to promote the development of an information model that can be used to describe the current or future state of an organization’s environment.
MagicDraw a business process, architecture, software and system modeling tool that enables all aspects of model building. It provides a rich set of graphical notations to model data in all its complexities, from entities to tables. Its intuitive interface provides wizards for the most common types of models, including Entity Relationship Diagrams (ERD), Business Process Models and Notation (BPMN), and Object-Oriented Design Models (OO). In addition, MagicDraw supports round-trip engineering with Unified Modeling Language (UML).
Lucidchart is an intuitive and intelligent diagramming application that makes it easy to make professional-looking flowcharts, org charts, wireframes, UML diagrams and conceptual drawings. This tool allows administrators to visualize their team’s processes, systems and organizational structure. It also enables developers to create UI mockups in a few clicks.
It has a drag-and-drop interface which simplifies the process of creating these diagrams. It also integrates with other business applications like Google Drive, Jira and Slack, which helps users to complete project work faster.
ConceptDraw is a diagramming solution that enables users to create diagrams or download and use premade ones. The data modeling tools include: ‘Table Designer,’ ‘Database Diagrams’ and ‘Data Flow Diagram.’ Users can also create flowcharts, UML diagrams, ERD diagrams, mind maps and process charts with this solution.

source: techrepublic.com

Data Munging

Data Munging: Definition

data munging is the initial process of refining raw data into content or formats better-suited for consumption by downstream systems and users.

The term ‘Mung’ was coined in the late 60s as a somewhat derogatory term for actions and transformations which progressively degrade a dataset, and quickly became tied to the backronym “Mash Until No Good” (or, recursively, “Mung Until No Good”).

But as the diversity, expertise, and specialization of data practitioners grew in the internet age, ‘munging’ and ‘wrangling’ became more useful generic terms, used analogously to ‘coding’ for software engineers.

With the rise of cloud computing and storage, and more sophisticated analytics, these terms evolved further, and today refer specifically to the initial collection, preparation, and refinement of raw data.

source: talend.com

The data munging process: An overview

With the wide variety of verticals, use-cases, types of users, and systems utilizing enterprise data today, the specifics of munging can take on myriad forms.

Data exploration: Munging usually begins with data exploration. Whether an analyst is merely peeking at completely new data in initial data analysis (IDA), or a data scientist begins the search for novel associations in existing records in exploratory data analysis (EDA), munging always begins with some degree of data discovery.
Data transformation: Once a sense of the raw data’s contents and structure have been established, it must be transformed to new formats appropriate for downstream processing. This step involves the pure data scientist, for example un-nesting hierarchical JSON data, denormalizing disparate tables so relevant information can be accessed from one place, or reshaping and aggregating time series data to the dimensions and spans of interest.
Data enrichment: Optionally, once data is ready for consumption, data mungers might choose to perform additional enrichment steps. This involves finding external sources of information to expand the scope or content of existing records. For example, using an open-source weather data set to add daily temperature to an ice-cream shop’s sales figures.
Data validation: The final, perhaps most important, munging step is validation. At this point, the data is ready to be used, but certain common-sense or sanity checks are critical if one wishes to trust the processed data. This step allows users to discover typos, incorrect mappings, problems with transformation steps, even the rare corruption caused by computational failure or error.

source: talend.com

The cloud and the future of data munging

Cloud computing and cloud data warehouses have generally contributed to a massive expansion of enterprise data’s role throughout organizations, and across markets. Data munging is only a relevant term today thanks to the importance of fast, flexible, but carefully governed information, all of which have been the primary benefits of modern cloud data platforms.

Now, concepts such as the data lake and NoSQL technologies have exploded the prevalence, and utility, of self-service data and analytics. Individual users everywhere have access to vast raw data, and are increasingly trusted to transform and analyze that data effectively. These specialists must know how to clean, transform, and verify all of this information themselves.

Whether in modernizing existing systems like data warehouses for better reliability and security, or empowering users such as data scientists to work on enterprise information end-to-end, data munging have never been more relevant concepts.

source: talend.com

Data Observability

Data observability: Definition

Data observability refers to an organization’s ability to understand the health of data throughout the data lifestyle. It helps companies connect the data tools and applications to better manage and monitor data across the full tech stack.

One of the core objectives of data observability is to be able to resolve real-time data issues, such as data downtime, which refers to periods where data is missing, incomplete or erroneous. Such issues with data can be extremely costly for an organization as it can lead to compromised decision-making ability, corrupted data sets, disrupted daily operations and other serious problems.

It is a common misconception that the scope of data observability is only limited to monitoring data quality. That might have been true a few years ago, however, with the increasing complexity of IT systems, the scope of data observantly now includes the entire data value chain.

source: techrepublic.com

Benefits of data observability

Data observability is a must-have for an organization that seeks to accelerate innovation, improve operational efficiency and gain a competitive advantage. The benefits of data observability include better data accessibility, which means the organization has access to uninterrupted data, which is needed for various operational processes and business decision-making.

Another key benefit of data observability is that it allows an organization to discover problems with data before they have a significant negative impact on the business. The real-time data monitoring and alerting can easily be scaled as the organization grows larger or has an increase in workload.

An organization can also benefit from improved collaboration among data engineers, business analysts and data scientists using data observability. The trust in data is also enhanced by data observability, so an organization can be confident in making data-driven business decisions.

Drawbacks to data observability
Data observability has several advantages for an organization, but there are also some downsides and risks. One of the major challenges of data observability is that it is not a plug-and-play solution, which means it requires an organization-level effort for its proper implementation and use. Data observability won’t work with data silos, so there needs to be an effort to integrate all the systems across the organization. This may require all data sources to abide by the same standards.

Another downside of data observability is that it requires a skilled team to get the maximum value from data observability. This means an organization needs to dedicate resources that have the capacity, experience and skills to observe the data. Several data observability tools, provided by various companies, can help but eventually it will be the responsibility of the data engineers to interpret the information, make decisions and determine the root cause of any data-related issues.

There has been significant progress in using machine learning and artificial intelligence to automate some of the data observer roles and responsibilities, however, there is still a long way to go before data observability can be automated

source: techrepublic.com

Data Preparation

Data preparation: Definition

Data preparation can be complicated. Get an overview of common data preparation tasks like transforming data, splitting datasets and merging multiple data sources.

Data preparation is a critical step in the data management process, as it can help to ensure that data is accurate, consistent and ready for modeling. In this guide, we explain more about how data preparation works and best practices.

Data preparation defined
Data preparation is the process of cleaning, transforming and restructuring data so that users can use it for analysis, business intelligence and visualization. In the era of big data, it is often a lengthy task for data engineers or users, but it is essential to put data in context. This process turns data into insights and eliminates errors and bias resulting from poor data quality.

Data preparation can involve a variety of tasks, such as the following:

Data cleaning: Removing invalid or missing values.
Data transformation: Converting data from one format to another.
Data restructuring: Aggregating data or creating new features.

While data preparation can be time-consuming, it is essential to the process of building accurate predictive models.

source: techrepublic.com

Why is data preparation important?

Data scientists spend most of their time preparing data. According to a recent study by Anaconda, data scientists spend at least 37% of their time preparing and cleaning data.

The amount of time spent on menial data preparation tasks makes many data scientists feel that data preparation is the worst part of their jobs, but accurate insights can only be gained from data that has been prepared well. Here are some of the key reasons why data preparation is important:

Delivers reliable results from analytics applications. Analytics applications can only provide reliable results if data is cleansed, transformed and structured correctly. Invalid data can lead to inaccurate results and cause data scientists to waste time trying to fix issues with the data.
Data preparation can help identify errors in data that would otherwise go undetected. These errors can be corrected before they impact the results of analytics applications.
Supports better decision-making. The data preparation process can help to improve the quality of data, leading to better decision-making across departments and projects.
Reduces data management and analytics costs
Organizations can reduce the costs associated with data management and analytics by automating data preparation tasks.
Avoids duplication of effort
Data preparation can help to avoid duplication of effort by ensuring that data is consistent and accurate. This saves time and resources that would otherwise be spent on data cleansing and data transformation.
Leads to higher ROI from BI and analytics initiatives
A well-executed data preparation process can improve the accuracy of insights, which can lead to a higher ROI from BI and analytics initiatives.

source: techrepublic.com

Data Profiling

Data Profiling: Definition

The health of your data depends on how well you profile it. Data quality assessments have revealed that only about 3% of data meets quality standards. That means poorly managed data costs companies millions of dollars in wasted time, money, and untapped potential.

Healthy data is easily discoverable, understandable, and of value to the people who need to use it; and it’s something every organization should strive for. Data profiling helps your team organize and analyze your data so it can yield its maximum value and give you a clear, competitive advantage in the marketplace. In this article, we explore the process of data profiling and look at the ways it can help you turn raw data into business intelligence and actionable insights.

Basics of data profiling
Data profiling is the process of examining, analyzing, and creating useful summaries of data. The process yields a high-level overview which aids in the discovery of data quality issues, risks, and overall trends. Data profiling produces critical insights into data that companies can then leverage to their advantage.

More specifically, data profiling sifts through data to determine its legitimacy and quality. Analytical algorithms detect dataset characteristics such as mean, minimum, maximum, percentile, and frequency to examine data in minute detail. It then performs analyses to uncover metadata, including frequency distributions, key relationships, foreign key candidates, and functional dependencies. Finally, it uses all of this information to expose how those factors align with your business’s standards and goals.

Data profiling can eliminate costly errors that are common in customer databases. These errors include null values (unknown or missing values), values that shouldn’t be included, values with unusually high or low frequency, values that don’t follow expected patterns, and values outside the normal range.

reference:
dataladder.com
talend.com

Four benefits of data profiling

Bad data can cost businesses 30% or more of their revenue. For many companies that means millions of dollars wasted, strategies that must be recalculated, and tarnished reputations. So how do data quality problems arise?

Often the culprit is oversight. Companies can become so busy collecting data and managing operations that the efficacy and quality of data becomes compromised. That could mean lost productivity, missed sales opportunities, and missed chances to improve the bottom line. That’s where a data profiling tool comes in.

Once a data profiling application is engaged, it continually analyzes, cleans, and updates data in order to provide critical insights that are available right from your laptop. Specifically, data profiling provides:

Better data quality and credibility
Once data has been analyzed, the application can help eliminate duplications or anomalies. It can determine useful information that could affect business choices, identify quality problems that exist within an organization’s system, and be used to draw certain conclusions about future health of a company.
Predictive decision making
Profiled information can be used to stop small mistakes from becoming big problems. It can also reveal possible outcomes for new scenarios. Data profiling helps create an accurate snapshot of a company’s health to better inform the decision-making process.
Proactive crisis management
Data profiling can help quickly identify and address problems, often before they arise.
Organized sorting
Most databases interact with a diverse set of data that could include blogs, social media, and other big data markets. Profiling can trace back to the original data source and ensure proper encryption for safety. A data profiler can then analyze those different databases, source applications, or tables, and ensure that the data meets standard statistical measures and specific business rules.
Understanding the relationship between available data, missing data, and required data helps an organization chart its future strategy and determine long-term goals. Access to a data profiling application can streamline these efforts.

reference: talend.com

Types of data profiling

In general, data profiling applications analyze a database by organizing and collecting information about it. This involves data profiling techniques such as column profiling, cross-column profiling, and cross-table profiling. Almost all of these profiling techniques can be categorized in one of three ways:

Structure discovery — Structure discovery (or analysis) helps determine whether your data is consistent and formatted correctly. It uses basic statistics to provide information about the validity of data.
Content discovery — Content discovery focuses on data quality. Data needs to be processed for formatting and standardization, and then properly integrated with existing data in a timely and efficient manner. For example, if a street address or phone number is incorrectly formatted it could mean that certain customers can’t be reached, or a delivery is misplaced.
Relationship discovery — Relationship discovery identifies connections between different datasets.

reference: talend.com

Data Purge

Data Purging: Definition

Data purging is the process of permanently removing obsolete data from a specific storage location when it is no longer required.

Common criteria for data purges include the advanced age of the data or the type of data in question. When a copy of the purged data is saved in another storage location, the copy is referred to as an archive.

The purging process allows an administrator to permanently remove data from its primary storage location, yet still retrieve and restore the data from the archive copy should there ever be a need. In contrast, the delete process also removes data permanently from a storage location, but doesn’t keep a backup.

In enterprise IT, the compound term purging and archiving is used to describe the removal of large amounts of data, while the term delete is used to refer to the permanent removal of small, insignificant amounts of data. In this context, the term deletion is often associated with data quality and data hygiene, whereas the term purging is associated with freeing up storage space for other uses.

Strategies for data purging are often based on specific industry and legal requirements. When carried out automatically through business rules, purging policies can help an organization run more efficiently and reduce the total cost of data storage both on-premises and in the cloud.

reference:
dataladder.com
techopedia.com

Data Recovery

Disaster Recovery Plan:Definition

A disaster recovery plan forms a crucial part of any business continuity plan. It is a formal document that outlines in detail how an organization will respond to disasters impacting the business’ IT operations, including:

Natural disasters
Power outages
Cyberattacks
Human error

What are the benefits of a disaster recovery plan?
A disaster recovery plan aims to:

Minimize disruption to business operations
Minimize the extent of any disruption and damage caused
Limit the economic impact of the downtime
Plan and facilitate alternative operation methods in advance
Ensure the relevant team members are familiar with the emergency processes and procedures
Facilitate the fast restoration of service.

What should a Disaster Recovery Plan include?

Business impact analysis
Robustness analysis
Communications information
Software application profile
Inventory profile
Disaster recovery procedures
For potential risk, you’ll need to include details of the disaster recovery procedures that have been prepared. This should cover:

- The scenario
- Possible causes
- IT services and data at risk
- Potential impact
- Preventative measures
- Plan of action
- Key contacts
Revision history

source: ontrack.com

Data Recovery: best seller tools

References: Put all in the same wrapped line

cleverfiles.com
download.cnet.com
cgsecurity.org
ccleaner.com
crashplan.com
dmde.com
easeus.com
minitool.com
ontrack.com
stellarinfo.com
seriousbit.com
techtarget.com

comparison chart: Put all in the same wrapped line
cleverfiles.com
handyrecovery.com

Writing a disaster recovery plan template for your small business

The practice of preparing for downtime is called disaster recovery (DR) planning.

A disaster recovery plan consists of the policies and procedures that your business will follow when IT services are disrupted. The basic idea is to restore the affected business processes as quickly as possible, whether by bringing disrupted services back online or by switching to a contingency system.

Your disaster recovery plan should take into account the following:

IT services: Which business processes are supported by which systems? What are the risks?
People: Who are the stakeholders, on both the business and IT side, in a given DR process?
Suppliers: Which external suppliers would you need to contact in the event of an IT outage? Your data recovery provider, for example.
Locations: Where will you work if your standard premises are rendered inaccessible?
Testing: How will you test the DR plan?
Training: What training and documentation will you provide to end-users?

At the centre of most DR plans are two all-important KPIs, which are typically applied individually to different IT services: recovery point objective (RPO) and recovery time objective (RTO). Don’t be confused by the jargon, because they’re very simple:

RPO: The maximum age of a backup before it ceases to be useful. If you can afford to lose a day’s worth of data in a given system, you set an RPO of 24 hours.
RTO: The maximum amount of time that should be allowed to elapse before the backup is implemented and normal services are resumed.

Structuring the perfect disaster recovery plan
Even a small business DR plan can be a lengthy and complex document. However, most follow a similar structure, encompassing definitions, duties, step-by-step response procedures and maintenance activities. In our template, we’ve used the following outline:

Introduction: A summary of the objectives and scope of the plan, including IT services and locations covered, RPOs and RTOs for different services, and testing and maintenance activities. Also includes a revision history to track changes.
Roles and responsibilities: A list of the internal and external stakeholders involved in each DR process covered, complete with their contact details and a description of their duties.
Incident response: When should the DR plan be triggered, and how and when should employees, management, partners and customers be notified?
DR procedures: Once the DR plan is triggered, the stakeholders can start to action a DR process for each affected IT service. In this section, those procedures are set out step-by-step.
Appendices: A collection of any other lists, forms and documents relevant to the DR plan, such as details on alternate work locations, insurance policies, and the storage and distribution of DR resources.

Do not forget these suggestions:

Keeping your disaster recovery plan alive
Test, test, test!

What is a disaster recovery plan, and why do you need one?

A step-by-step guide to disaster recovery planning

If you’re interested in ensuring Ontrack is part of your disaster recovery plan, talk to one of our experts today.

source: resources.ontrack.com

reference:
Put all in the same wrapped line
ontrack.com
iso-docs.com
disasterrecoveryplantemplate.org
microfocus.com
easeus.com
ibm.com
solutionsreview.com

Data Silos

Data Silos: Definition

A data silo is a collection of data held by one group that is not easily or fully accessible by other groups in the same organization. Finance, administration, HR, marketing teams, and other departments need different information to do their work. Those different departments tend to store their data in separate locations known as data or information silos, after the structures farmers use to store different types of grain. As the quantity and diversity of data assets grow, data silos also grow.

Data silos may seem harmless, but siloed data creates barriers to information sharing and collaboration across departments. Due to inconsistencies in data that may overlap across silos, data quality often suffers. When data is siloed, it’s also hard for leaders to get a holistic view of company data.

In short, siloed data is not healthy data. Data is healthy when it’s accessible and easily understood across your organization. If data isn’t easy to find and use in a timely fashion, or can’t be trusted when it is found, it isn’t adding value to analyses and decision-making processes. An organization that digitizes without breaking down data silos won’t access the full benefits of digital transformation. To become truly data-driven, organizations need to provide decision-makers with a 360-degree view of data that’s relevant to their analyses. 

Data analysis of enterprise-wide data supports fully informed decision-making, and a more holistic view of hidden opportunities — or threats! Plus, siloed data is itself a risk. Data that is siloed makes data governance impossible to manage on an organization-wide scale, impeding regulatory compliance and opening the door to misuse of sensitive data.

To better understand if data silos are holding back your potential for holistic data analysis, you’ll need to learn more about where data silos come from, how they hinder getting the full benefit of data, and your options for data integration to get rid of data silos.

source: talend.com

Why do data silos occur?

Data silos occur naturally over time, mirroring organizational structures. As each department collects and stores its own data for its own purposes, it creates its own data silo. Most businesses can trace the problem to these causes of data silos:

Siloed organizational structure
Company culture
Technology

4 ways data silos are silently killing your business
Each department exists to support a common goal. While departments operate separately, they are also interdependent. At least some of the internal data that the finance department creates and manages, for example, is relevant for analysis by administration and other departments.

Here are four common ways data silos hurt businesses:

Data silos limit the view of data
Data silos threaten data integrity
Data silos waste resources
Data silos discourage collaborative work

source: talend.com

How to break down data silos in 4 steps

The solutions to silos are technological and organizational. Centralizing data for analysis has become much faster and easier in the cloud. Cloud-based tools streamline the process of gathering data into a common pool and format for efficient analysis. What once took weeks, months, or years can now be accomplished in days or hours.

Change management
Develop a way to centralize data
Integrate data
Integrating data efficiently and accurately is a guaranteed method to preventing future data silos. Organizations integrate data using one of several methods:
- Scripting
- On-premises ETL tools
- Cloud-based ETL
Establish governed self-service access

The cloud and the future of data storage

The cloud has emerged as a natural way to centralize data from diverse sources to make it easily accessible from the office, at home, on the road, or by branch operations.
Cloud data solutions help eliminate the technology barriers to collaboration and offer a ready solution for connecting siloed data. Using an established ETL process to strip away irrelevant data and eliminate duplication, organizations can quickly add new and updated data to a cloud data warehouse. This enables different departments to work collaboratively with fresh, clean, and timely data in a single, accessible platform that scales to meet demand.
Cloud technology and cloud data warehouses connect disparate business units into a cohesive ecosystem. Data analysts get a better view of how their work affects the whole organization, and how everyone’s work affects each other. Access to enterprise-wide data gives analysts a 360-degree view of the organization.

Tearing down data silos
Data silos undermine productivity, hinder insights, and obstruct collaboration. But silos cease to be a barrier when data is centralized and optimized for analysis. Cloud technology has been optimized to make centralization practical.

source: talend.com

Data Stewardship

Data stewardship: Definition

Data stewardship is the implementation of the procedures, roles, policies and rules set by the data governance framework. This includes people, technology and processes. Data stewards or a team of data stewards are tasked with the responsibility of protecting data assets of the entire organization, department, business unit or a small set of data. They are also tasked with the implementation of data governance initiatives, improving the adoption of data policies and procedures, and ensuring users are held accountable for the data in their care.

What are the similarities and differences between data stewardship and data governance?

As data stewardship is effectively a branch of data governance, they share some common goals of protecting data, making it more manageable and getting the maximum value from it. The ultimate goal of data governance and data stewardship is to have fully governed data assets.

Although these two terms are used interchangeably, there are distinct differences. While data governance deals with policies, processes and procedures, data stewardship is only concerned with the procedures. This means that data stewards are not responsible for creating or writing policies or processes, their job is to interpret and implement them on a day-to-day basis. This requires data stewards to have technical familiarity with the data and the systems that use the data, and business acumen to understand integration of data with business processes and outcomes.

Data stewardship best practices

Encourage adoption of data governance
Regularly verify data quality
Establish a data stewardship committee

source: techrepublic.com

Data Transformation

Data Transformation: Definition

Data transformation is the process of converting, cleansing, and structuring data into a usable format that can be analyzed to support decision making processes, and to propel the growth of an organization

Data transformation is used when data needs to be converted to match that of the destination system. This can occur at two places of the data pipeline. First, organizations with on-site data storage use an extract, transform, load, with the data transformation taking place during the middle ‘transform’ step.

Source: tibco.com

Data Vault

The Data Vault: Definition and why do we need it

The Data Vault is a hybrid data modeling methodology providing historical data representation from multiple sources designed to be resilient to environmental changes. Originally conceived in 1990 and released in 2000 as a public domain modeling methodology, Dan Linstedt, its creator, describes a resulting Data Vault database as:

“A detail oriented, historical tracking and uniquely linked set of normalized tables that support one or more functional areas of business. It is a hybrid approach encompassing the best of breed between 3NF and Star Schemas. The design is flexible, scalable, consistent and adaptable to the needs of the enterprise.”

(en.wikipedia.org)

Focused on the business process, the Data Vault as a data integration architecture, has robust standards and definitional methods which unite information in order to make sense if it. The Data Vault model is comprised of three basic table types:

HUB (blue): containing a list of unique business keys having its own surrogate key. Metadata describing the origin of the business key, or record ‘source’ is also stored to track where and when the data originated.
LNK (red): establishing relationships between business keys (typically hubs, but links can link to other links); essentially describing a many-to-many relationship. Links are often used to deal with changes in data granularity reducing the impact of adding a new business key to a linked Hub.
SAT (yellow): holding descriptive attributes that can change over time (similar to a Kimball Type II slowly changing dimension). Where Hubs and Links form the structure of the data model, Satellites contain temporal and descriptive attributes including metadata linking them to their parent Hub or Link tables. Metadata attributes within a Satellite table containing a date the record became valid and a date it expired provide powerful historical capabilities enabling queries that can go ‘back-in-time’.

There are several key advantages to the Data Vault approach:

Simplifies the data ingestion process
Removes the cleansing requirement of a Star Schema
Instantly provides auditability for HIPPA and other regulations
Puts the focus on the real problem instead of programming around it
Easily allows for the addition of new data sources without disruption to existing schema

Simply put, the Data Vault is both a data modeling technique and methodology which accommodates historical data, auditing, and tracking of data.

“The Data Vault is the optimal choice for modeling the EDW in the DW 2.0 framework”

Conclusion

It adapts to a changing business environment
It supports very large data sets
It simplifies the EDW/BI design complexities
It increases usability by business users because it is modeled after the business domain
It allows for new data sources to be added without impacting the existing design

source: talend.com

Data Wrangling

Data Wrangling: Definition

Data wrangling is the process of removing errors and combining complex data sets to make them more accessible and easier to analyze.

A data wrangling process, also known as a data munging process, consists of reorganizing, transforming and mapping data from one “raw” form into another in order to make it more usable and valuable for a variety of downstream uses including analytics.

Data wrangling can be defined as the process of cleaning, organizing, and transforming raw data into the desired format for analysts to use for prompt decision-making. Also known as data cleaning or data munging, data wrangling enables businesses to tackle more complex data in less time, produce more accurate results, and make better decisions. The exact methods vary from project to project depending upon your data and the goal you are trying to achieve. More and more organizations are increasingly relying on data wrangling tools to make data ready for downstream analytics.

Data has the potential to change the world. But before it does, it has to go through a fair amount of processing to be ready for analysis. A critical step in this processing is data wrangling. Data wrangling is a non-technical term used to describe the crucial cleaning and sorting step of data analysis. Specifically, data wrangling is a process that manually transforms and maps raw data into various formats based on specific use cases.

While the process is not considered glamorous, data wrangling is the backbone behind understanding data. Without it, a business’s data is nothing more than an unorganized mess — difficult to read, impossible to access, and unlikely to be analyzed in a useful manner. It’s no surprise, then, that Data scientists dedicate 80% of their time to data wrangling.

Data Science

In Dept

Data Science

Data Science: Definition

Data Science: Quick glossary

Address Verification

Address Verification definition

Data Cleansing

Data Cleansing: Definition

Steps to performing data cleansing

Data cleansing tools

Data Deduplication

Data Deduplication: Definition

Data Literacy

Data literacy: why is it essential for success?

Why is data literacy important for your business?

Data Masking

Data Masking: Definition

Static Data Masking (SDM) definition:

DDM, Dynamic Data Masking Definition

What tool do we use for Data Masking?

Data Presentation

Presenting Data: Definition

Data Quality

Data Quality:Definition

Top 10 benefits of data quality

5 tips for improving data quality for unstructured data

Data Quality KPIs

The Cost of Poor Data Quality

Talend: The tools we use

Data Integration

Data Integration: Definition

Benefits of Data Integration

What happen without data Integration?

Types of data integration

Data Integration methods

Where do you start with a data integration strategy?

Data integration vs ETL: What are the differences?

The future of data integration

Top Data Integration Tools

Data Lakes

Data Lake: Definition

Who does Data Lakes affect?

Data Migration

Data Migration: Definition

top 5 data migration trends

Etc

Data Consistency

Data Consistency: Definition

Data Encryption

Data Encryption Best practices

Data Estate

Data Estate Definition

Data Fabric

Data Fabric Definition

Data Governance

Data Governance: Definition

Data Gravity

Data Gravity Definition

How data gravity affects the enterprise

Data Integrity

Data Integrity Definition

Why is data integrity important?

The risks associated with data integrity

Types of data integrity

Data Lab

Data Lab: Definition

Data Matching

Data matching Definition

Data Modeling

What does a data modeler do?

Top data modeling tools of 2022

Data Munging

Data Munging: Definition

The data munging process: An overview

The cloud and the future of data munging

Data Observability

Data observability: Definition

Benefits of data observability

Data Preparation

Why do data silos occur?

How to break down data silos in 4 steps