Content management error: Header Banners should not be placed in the Navigation placeholder!
Content management error: Generic Content Banners should not be placed in the Navigation placeholder!
It’s all about data ownership and control
Every organisation wants better control over their incoming data from external suppliers and the distribution there of to various systems within the enterprise. This is especially case in situations where the destination systems are 3rd party owned.
In this example, a Modis customer in the Public Sector in Australia approached us with a data challenge.
The client receives files from numerous sources ever month, which all go through intensive manual processes to then load into a core bespoke developed business application. This application then performs various evaluations and calculation on the data.
They found that this core system and supporting manual processes were aging and expensive to manage and maintain. They looked at cost effect alternatives and suppliers of similar systems around the world and soon realized they had a need to exercise greater control and ownership over this data as they started looking with the external 3rd party solutions.
The data comes in various flat file formats (position notation files, CSV files, XML, JSON) and unstructured in the form of PDF or image files. The data were transported/transferred in files using old methods such as email, or dropping files into shared file stores, which both have security issues.
The client was not only looking for an ETL tool, but rather something that could work as a hub to take data from any approved relevant source, then validate and transform and distribute it to various systems throughout the organisation.
Through working with Modis, our client now has a highly configurable, scalable, cost-effective solution comprising of 10 Lambda functions (and growing). This solution breaks the business logic into smaller SOA components within one part of the overall business function. The bigger solution can be broken into three distinctive area:
This part of the application deals with user registration only. It allows user partially self-register. Internal users can register their intent to use the application through using their AD FS single sign credentials.
Notes:
Once a user is approved and begins to use the application as a supplier of data, they will be able to upload files for extraction and processing. They only execute high responsive restful services that will present a positive and informative experience.
Once the file has been uploaded, SQS queued Lambdas are woken up and an extraction, validation and if required a transformation is performed before its dataset is posted to a submitted state. The validation lambda scales horizontally to validate sets of 1,000 records defined via the SQS queueing mechanism.
Admin users will be able to monitor and manually amend data, as well as manage various aspects of their supplier users.
Notes on diagrams:
This is a completely serverless solution. This part of the solution sends data of files to the predetermined locations. Destinations for this data currently can be one of the following: file store, a restful web service, a database.
All the intelligence of what propagation actions are to be taken against a dataset are defined in a set of RDS table and are used as lookups by each Lambda involved. These definitions allow a single dataset to have multiple propagation definitions containing destinations, extracted values and data format (csv, xml, json).
This solution resulted in an efficient and cost-effective processing service, that required minimal management overhead. Our maintenance and management will be to update the Lambda Runtime(s) and our code base (Node.JS, Java) when language updates require modification, and manage our client-side JavaScript library dependencies such as Angular.
The delivery was driven from CloudFormation Templates, stored in CodeCommit, and deployed across environments from development to production using DevOps approaches. Therefore, maintenance and updates over time can be repeatedly and reliably re-deployed in the future.
Modis is a global IT consulting company, driving innovation and digital transformation solutions to enable a more sustainable tomorrow. Comprising of three key business lines - Tech Consulting, Tech Academy, Tech Talent Services – we connect people, technology, and businesses to the opportunities they need to thrive in a rapidly advancing world. With passion for technology and talent, Modis drives innovation and enables digital transformation for a smart and sustainable tomorrow. Modis is a Global Business Unit of the Adecco Group, a Fortune Global 500 company. #EngineeringaSmarterFuture
Content management error: Generic Content Banners should not be placed in the Navigation placeholder!
Content management error: Generic Content Banners should not be placed in the Navigation placeholder!
Modis developed and continues to manage a fleet of ESRI ArcGIS geospatial servers for a commercial operator in the lands industry. The ArcGIS applicat...
An electrical energy retailer has started a migration into AWS for its VM based business applications, with Modis helping prepare solid, evidence-base...
Modis implemented a fully managed cloud firewall to either restrict or inspect and permit data egressing from a customer’s Cloud network as a central ...
A state government customer has a number of microservices and applications that are logging to individual Cloud Watch log streams and investigating in...
Content management error: Generic Content Banners should not be placed in the Navigation placeholder!