I would like to provide a data architecture summary briefly. Additionally, it reflects my long-term working practices in this area.
Data Architecture and Governance:
- Data governance involves strategies, policies, processes and procedures, collaboration, tools, audit and metrics, etc. For detail info, please refer to Seven Steps to Effective Data Governance.
- Data quality includes data validation, foreign keys, constraints and data cleansing. Please refer to 10 Best Practices in Database Schema Design.
- Metadata management involves schema design, data linage, API contracts, etc. Building a Database-driven Application Framework includes good references.
- Toolbox: MS Purview, Ataccama, Denodo
Data Modeling:
- UML: describe the relationship among objects, no emphasize on normalization. Core concepts are composition relationship, generalization relationship, dependency, aggregation and association. Please refer to Database Modeling in UML. Spatial data modeling aligns UML data modeling on Visio, Rational Rose and Sparx EA, etc. Toolbox for UML data modeling: Visio, Visual Paradigm, Sparx EA
- ERD: describe the relationship among entities, tables, columns, PK, FK, normalization, cardinality, etc. Modeling the world from conceptual model, logical model to physical model. It requires ORM while programming. Typical ORM tool is Hibernate. Toolbox for ERD data modeling: Visio, Visual Paradigm, ERWin, Toad Data Modeler
- Note Visual Paradigm is a tool which allows UML and ERD two-way conversion.
- Please refer to 10 Best Practices in Database Schema Design for data modeling goals and best practices.
Data Migration:
- From source datastore to target. Database export and import command tools are essential, plus DDL and DML.
- Data loading includes bulk and incremental.
- Toolbox: Toad, Oracle SQL Developer, DBVisualizer, Oracle Data Integrator (ODI), Informatica, Talend, Pentaho, Azure Data Factory, etc. Spatial data migration may utilize FME by Safe.com, LiveSync Automation.
- For large volume data migration in limited period, please refer to: How to Improve Data Replication Performance Significantly and How to Implement Near Zero Downtime Data Migration?
Data Integration:
- API via REST or SOAP, using Swagger and RAML for design. Most popular way, may involve API management.
- File-based / SFTP
- Streaming via Apache Kafka: event-driven, high throughput, near real-time ingestion, publish-subscription mode. Please refer to: What Confluent / Kafka can do for Data Synchronization.
Modern data architecture involves data warehouse, BI, data lake, non-relational data store, big data and machine learning, data architecture summary.
Lionsgate Software consultant has over two decades experience on database design, development and data migration. Should you have any questions, please feel free to contact us.
Leave a reply
You must be logged in to post a comment.