Implications of big data for individuals and society

Limitations of predictive analytics

With big data analytics and machine learning technologies it is becoming possible to very accurately predict trends and patterns in many areas but also in regards to human behaviour. So far this has been applied in most part to retail market and online shopping.

There has been discussions of future systems that while technologically possible are controversial. Crime prediction. Big data and even real time big data analytics combined with machine learning technology can be very accurate. This could possibly palce authorities in advantaged position in order to prevent crime by for example dispatching appropriate resources to an area with high risk potential in real time.

Discussions also tend to steer into more of a science fiction scenarios taken from movies like Minority Report where police uses technology to predict crime and make arrests based on that prediction.

Obvious controversy is in human belief in destiny and predefined path, but this could also be a discussion about religion. there will always be pros and cons, believers and non believers. Its a free will problem and I believe that human beings should never be judged on an assumption or a prediction.

Theoretically there could be benefits of preventing crime from happening and while technologically possible I believe that scenario should be left in the entertainment industry in the hands of movie makers.

Implications of big data for individuals

Implications of big data analytics are a constant topic of a debate. There are many ways that individuals can be affected. The positive implications can greatly improve a life of an individual.

Those include an idea of driver-less cars that utilise real time big data analytics.

Car journeys could become:

safer due to elimination of a human error
faster due to big data real time traffic analysis
more efficient and economical due to big data route, traffic and even optimisation for the most economic driving style

Another positive implication of utilisation of big data for individuals lays in healthcare. With previously mentioned examples of ability to considerably cut diagnosis time as well costs. Use of dedicated healthcare smart wearable devices could mean early detection of health problems for individuals even before symptoms occur which could potentially have life saving implications. Healthcare is an area where bid data can make an impact which could include:

Extension of life
Early health problem detection
Diagnosis time reduced
Reduced costs of diagnosis
Ability to treat more patients due to time saved

While there are many more benefits of how big data analytics can affect and individuals life there are many negative effects that I think we need to be aware of. Big data analytics brings more automation to almost every sector that it is applied to. It comes with territory.

First example of possible negative implications is something that is already happening and more and more companies are starting to use automated profiling of prospective employees. This could greatly reduce chances of employment even for individuals with the right skills and expertise solely due to being less skilled / technical about formatting their resumes to a specific formats that an algorithm will be able to “read” or “understand”.

Similar automated profiling techniques can cause individuals not the be awarded place at educational establishments or being regarded as inadequate students during their time at university solely due to inner workings of an algorithm used to asses the entry applications or academic progress.

In line with the above techniques and their implications there may be a fear of loss of personal liberty where choices are made for an individual and solely up to inner workings of algorithms that asses and analyse data.

Negative implications include:

Profiling of job applicants,
Profiling of students,
Profiling of credit applicants
Loss of personal liberty
Loss of privacy

Implications of big data for society

Implications of big data analytics for society can overlap with previously mentioned implications for individuals.

Main positive implications include:

Great improvement in healthcare

With great potential of reducing diagnosis time and effectively its cost there can be less drain on healthcare secedes. This could enable faster and more effective appointment systems. Patients with rare ore more complicated conditions could be seen by specialists much sooner.

Extension of life could also be possible due to faster diagnosis ans well a increased rate of early health problems detection could lead to life saved.

Use of big data analytics in healthcare can have a positive economical effect for the society where costs of diagnosis were saved.

Near instant court decisions for crime sentencing

While controversial, another possible positive implication of use of big data analytics are arguably near instant court decisions for crime sentencing based on hard facts and eliminating factors like bias or likeability .

Cutting time and effectively costs of court time could have a positive impact on economy for the benefit of the society.

This example is widely debated and controversial and I will also present an opposed view in a negative implications of that particular use of big data analytics.

Main negative implications include:

Potential loss of jobs

Most discussed and debated implication of big data analytics (or progress technological progress in general) for the society were always topics of job loss and replacement of manual labour with automated systems or robots.

While not that long ago this may have been a chapter from William Gibson’s science fiction book it poses a real life implications today for low skilled jobs in certain industries that could affect teachers, nurses or any kind of manual labour jobs.

Topic of job loss due to advancement technology is not new. It was widely debated during each major technological advancement and there are always two sides to the medal.

Some argue that technolog is not at fault here but the problem lies in how our current societies and governmental systems are structured.

A quote from New York congressional representative Alexandria Ocasio-Cortez shine some light on that side that is usually left out from the debate:

“We should not be haunted by the specter of being automated out of work,” “We should be excited by that. But the reason we’re not excited by it is because we live in a society where if you don’t have a job, you are left to die. And that is, at its core, our problem.”

Alexandria Ocasio-Cortez (2019)

Human rights implications – mass surveillance

To power data driven services from driver-less cars, traffic automation to life saving medical wearable devices a massive about of dat needs to be collected on daily basis. To enable the above debated scenarios and technologies to benefit individuals and the society more data gathering deceives and technologies need to be deployed.

This comes at cost to personal privacy and can have human rights implications.

A laud case of Edward Snowden, a former Central Intelligence Agency employee who with help of professional journalists disclosed use of mass surveillance technologies by the government agency on unsuspecting citizens.

With large percent of our lives shifting into the online space and sharing data and information on each one of us this can be possible cost to the benefits we gain or a scary scenario where personal privacy is abolished heavily implicating human rights and affecting whole societies.

Near instant court decisions for crime sentencing – counterargument

There is also a counter argument for use of big data analytics in the workings of court proceedings and criminal sentencing. The argument being that basing the sentencing solely on data analytics does not take a human factor and most importantly the context which often needs to be assessed on case by case basis due to often unique nature of each case and its circumstances.

Strategies for limiting the negative effects of big data

Creation of rules and laws
Enforcement
Opt Out – data sharing
Misinformation

With big data analytics the privacy of individuals comes very close to being non existent. With that said and the previously mentioned negative effects on society and individuals there are many ways to limit or in some cases eliminate those negative effects.

In the public and official sectors like councils or banking where data must be accurate the only way to limit the negative effects for an individual is creation and enforcement of new rules and laws that would protect an individual from being exploited.

As an individual we also have few limited choices. In many cases we can op out from online data sharing and most of institution are already obliged to clearly state how our personal data is handled.

In regards to already mentioned possible surveillance and protection of individual privacy there are ways to use the same technologies in counteract. This method was known for hundreds of years and it is misinformation.

There could be a market for those privacy-conscious for an applications which would constantly generate and share randomly generated data (for example search queries) from our devices. This in turn would hide our actual online presence in plain sight effectively rending big data’s Veracity useless every time.

Applications of Big Data

In this post I will summarise applications of big data in business, science and society as well as possible future applications of big data.

I will also summarise the technological requirements of big data and how it is gathered, stored, processed and visualised.

Applications of big data in business

Big data helps businesses and organisations to use data to their advantage and turn it to identification of new opportunities.

This improves business decision making, making operations more efficient, turns higher profits and making customers happier with their product / service.

Big data and systems like cloud based analytics or Hadoop makes it possible to reduce costs fore organisations. This mostly applies to the costs of storing large amounts of data.

With the efficiency ans speed of systems like Hadoop better and faster decision making is now possible. This is due to ability to analyse new sources of data, in-memory analytics. Organisations are now able to analyse information much faster or even immediately which in turn allows to make business decisions based of what they have learned much faster.

Organisations with the ability to measure customer needs and their satisfaction with analytics gives organisations the ability to give the customers what they want.

More and more businesses are able to create new products that come from big data analytics.

In media and entertainment industry that revolves around user-generated data through various social media platforms, businesses were able to leveraged that data in order to predict the interests of their audience, getting insights into customers reviews and behavioural patterns.

Spotify, an on-demand music streaming service uses big data to to serve targeted music recommendations to individual users.

Similar use od big data is employed at Amazon Prime that offers e-books, video streaming as well as music streaming.

Applications of big data in science

There are wast benefits of application of big data in healthcare.

Healthcare analytics have great potential in areas like improving quality of life, prediction of potential outbreaks of epidemics but also significantly reducing costs of treatments ant time of diagnosis.

Big data application in healthcare means that doctors now can make more informed decisions as they have access to wider range of data.

There are many healthcare data types created in hospitals or clinics those include:

Clinical data: doctors motes, prescriptions, reports from medical imaging, laboratory data, pharmacy data
Machine generated data: generated from monitoring vital signs, emergency care data, medical journals
Patient data that includes electronic patient records

Another example for the use of big data in healthcare is predictive analytics that use data like pre-existing conditions, habit patterns, which enable to foresee how vulnerable an individual is to cancer and in turn allows for early treatment.

There are many more applications of big data in healthcare and those include:

Tracking Patient Vitals

It is much easier to monitor patients in emergency rooms plugged to monitoring devices because any change in patterns can be quickly alerted to doctors in hospitals that often do not have enough staffing.

Other uses of big data usage in healthcare include:

Improved hospital administration
Fraud prevention and detection

Applications of big data in society

With big data already having an impact on business world and healthcare does it have any application in society?

The short answer is yes but lets fins some examples.

The main use of gig data that has impact on society are transformation of cities into smart cities. Using variety of sensors, tracking movement of public transport and that data that is generated enable cities to be more efficient and provide citizens with better and more personalised services while cutting down unnecessary costs and greatly reducing waste of resources.

Cities transportation systems that utilises big data analytics can be greatly beneficial.

Enabling data streaming gathered from variety of sensors, on-vehicle devices and smart traffic lights, processing in real time and communicating traffic information with drivers directly to their smart phones.

Recognition of traffic patterns by analysing real time data, reduction of roads congestion by traffic prediction and adjusting traffic controls accordingly.

Future applications of big data

Smart Cities

There are many possible uses of big data in the future. Previously mentioned evolving smart-cities could turn into fully automates and even semi-autonomous cities providing public with with benefits of automated and personalised transportation systems.

There are many speculations on topics of ethics and security but from purely technological point of view the potential of transformations of cities into smart cities driven by big data analytics are huge.

Smart transportation is one of the areas that can transform how we commute.

Recognition of traffic patterns in real time, solving road congestion problems, utilising smart traffic lights systems and even driver-less cars that utilise real-time big data analytics.

Smart Environment

Environment data gathered from large amount of sensors and processed can provide weather information that will lead to improving agriculture, better energy management and informing the public about hazardous conditions.

Healthcare

Healthcare is another area where future use of big data analytics can have life changing effects. We all know the current technology we use on daily basis, our smartphones, smart watches and sport / fitness trackers. Ability to gather data in real time from all kinds of smart wearable devices can save your live one day.

I had a pleasure to attend Glasgow TEDx Talks where Dr Jack Kreindler talked about the future of healthcare, data and technology.

Dr Jack presenting a Bio Sensor to the audience

In the above picture Dr Jack Kreindler holds a bio-sensor that when attached to patients body is able to measure and transmit hart rate, ECG, body temperature, posture (standing, sitting, falling), stress, breathing rate.

To do this 10 years ago (for the drivers of formula 1) it costed about 10000 dollars.

Today a cost os such device it about $1 a day.

With devices like this and with many people using them, for a $1 a day (including all of the machine intelligence to compute the data) it is believed that the crisis that healthcare is facing with chronic diseases can be greatly reduced.

Among many different tests the results have shown that such device with big data analytics and cloud computing can alert patients of medical problems even before symptoms occur.

Technological requirements of Big Data

I have already mentioned some of the technological requirements that are very much linked to big data characteristics and especially the volume of it.

Storage

As mentioned previously traditional storage solution are not well suited for big data due to its volume and large variety of types of data.

Number of organisation who already have data storing capabilities in-house. Those organisations that are exploring the option of using big data analytics may need to reevaluate and opt for storage types that are more efficient for big data analytics such as cloud computing or the use of flash storage that may be more suited due to its performance advantages.

The largest companies such as Facebook for best efficiency and performance use. Those may consist of clusters of severs with direst-attached storage and using tools like Hadoop while often using PCIe flash based storage for improved latency.

Processing

Processing bid data and performing analytics operations require powerful computing resources.

While companies like Google can afford to build custom mage infrastructure of server clusters with powerful processing power smaller companies may opt for cost effective cloud computing (renting) that can be used on demand saving them large investments that would come from the purchase, set-up and maintenance of of their oven big data analytics server infrastructure. Smaller companies are also able tu use technologies like previously mentioned Hadoop.

Gathering data

There are many sources of big data. Most companies collect data about their customers, business operations, cities and councils collect data from sensors, big stores collect in-store customer traffic data.

Depending on the type of organisation the type of data may vary but all of those who collect big data have one thing in common, storage of gathered data. This is closely tied to the above mentioned storage requirements and will depend on type and size of the organisations.

There are meny technological choices including own big data analytics sever infrastructure, cloud computing or distributed storage systems that store data that is gathered from company customers that are interacting with their websites, shops, services.

Data is also gathered from loyalty cards, social media websites and apps, satellites and so on.

Big data Visualisation

Visualisation can be thought of as another characteristic of big data and currently it is a most challenging aspect that data scientist face.

Enormous amounts of data, event in a processed and analysed state is often not very useful insight for a human being. It may not be easily comprehended and auctioned on.

While traditional graphs did a good job for traditional data analysis in the past with big data there are simple too many data points (billions) to plot which usually fails due to in-memory limitations and poor scalability which is one of the important requirements in big data. When it comes to big date scalability is a must.

Solutions to those problems are constantly being developed. Some include the use of data clustering, tree maps, parallel coordinates or circular network diagrams.