Traditionally Technical Architects design the logical, physical and logical, physical and processes for IT systems. This design is needed for consistent IT strategy and to ensure solutions meet the needs of the business, rather than the desires of its vendors.
However Big Data is different to traditional projects and requires a new approach to be successful – something called the User Orientated Architecture (UOA).
So why is a radical rethink required for Big Data projects?
Skills are the main constraint for Big Data
Legacy data technology provided a ‘pre-canned’ solution with rigid data schemas, user-friendly interfaces and easy to write queries. But this came with issues of large costs, limited scaleability and rigidity.
Big Data addresses these issues but still has its own limits, critically the developer and administrator skills available to the enterprise. Think of Big Data as a racing car – it has very high performance but winning requires good driving and a team of mechanics. So when designing a Big Data system, Architects must account for skill and resource constraints as the number one priority.
The need to make new money
A Big Data tenet is to provide better answers to existing questions and fresh answers to new questions. Simple to say, not easy to do – this is quite different from standard IT systems with known inputs and outputs.
Architects need to align their design with the financial needs of the business, so that when the Finance Director requires answers these can be given quickly and provide new and fresh insights. Future-proofed solutions recognise that answers evolve based on insights from data exploration and analysis, and flex with the needs of the business. The value of the system will be judged by the quality and speed of answers produced.
Ever tried using two hammers simultaneously?
Businesses choosing to use the Big Data ecosystem should fully recognise that a game-changing tool has arrived. Legacy infrastucture strategy and choices should not stop the most fit for purpose tools being used – design should provide the best quality of commercial answers with the skill-sets available.
What does UOA mean for Big Data projects?
Putting a successful service in place – not just infrastructure components
Successful architectures will focus on maximising personal productivity for users.
“Walking the walk” requires architects to be expert users of the system so that they have real empathy for developers and administrators by experiencing for themselves the pain that can come from bad design decisions.
Being commercially focused
Big Data technology is mature enough to prove that it works, so succesful architectures can focus on delivering commercially orientated proofs of concepts (POCs) to establish value before large spend is incurred. To be meaningful and successful, these POCs need to identify and address the right commercial questions which requires architects to step outside of their traditional roles and
- become an active business partner in commercial discussions
- have strong understanding and empathy of the data itself
- understand what is algorithmically possible
- establish analytics plans with suitable computing resources and tools.
Creating a flexible system
Big Data systems are inherently more open than traditonal data environments – trying to lock them down “old school” will destroy the gains that agile developers can extract. This requires fresh thinking on
- how to manage internal and external data and its security
- using different clusters for different purposes
- use of the cloud.
UOA’s need for commercially focused agility means companies should consider measuring design success with a balanced scorecard across:
- the value of financial benefits realised
- speed of creating the code asset generating these benefits
- user feedback
- developing new IT security models that do not shackle results
See here how UOA’s philosophy of achieving quality by helping users is being proven by one of the world’s largest companies.